Compliance e-learning: completion versus real competence

The effectiveness of compliance e-learning cannot be read from its completion rate. A very high rate shows that almost everyone reached the last screen. It does not say that employees would know what to do in a real situation. Completion measures a path, competence measures a behavior.

This article explains why the two are so often confused. It summarizes what memory research allows us to say, then proposes measuring something other than the click. It is part of our guide to mandatory compliance training.

Why the completion rate is so appealing

Completion is easy to obtain. The LMS provides it effortlessly, it fits in a dashboard and it can be compared from one year to the next. For a compliance officer who must report to management or to a regulator, it is a reassuring number.

The problem lies in what it is made to say. This figure proves that the training was delivered and followed. It says nothing about what employees retained from it. It says even less about what they will do on a Tuesday morning, on the phone, with an upset customer.

The 2021 recommendations of the French anti-corruption agency (AFA) go further than attendance. They call for tools to check that training has been properly understood, for example a test at the end of the session or after a delay. They also suggest tracking the coverage rate of the target audience and the hours devoted.

What we know about forgetting

A 2015 experiment, published in PLoS ONE by Murre and Dros, reproduced the forgetting curve described by Ebbinghaus in 1880. Its results are close to the original data: we forget quickly after learning, then more slowly. This experiment concerns lists of syllables, not compliance rules, but it recalls the principle.

Training seen once, without review, therefore leaves few traces after a few weeks. Employees are not careless. They do what every human memory does when information is not reused.

The work of Roediger and Karpicke, published in Psychological Science in 2006, points the same way. Students who tested themselves on a text retained it better, at two days and at one week, than those who had reread it. Testing yourself, that is, retrieving the information yourself, strengthens memory more than simple rereading.

Knowing is not knowing how to do

Even an employee who remembers the rule may not apply it. Between the rule and the action, there is a conversation. Take a customer service rep who must ask a due-diligence question of a customer in a hurry, or a manager who must cut short an inappropriate remark. Both know. Few have already done it.

Three obstacles recur:

• social pressure, which makes the rule harder to hold in front of a human than on a screen;
• the lack of words: without a ready sentence, we improvise, or we stay silent;
• the absence of feedback: nobody tells the employee whether their answer was good.

E-learning, even well designed, handles these three points poorly. It may contain a quiz, but a quiz checks recognition, not spoken production. For a comparable analysis on the cybersecurity side, see our article on the limits of e-learning for security reflexes.

What should you measure instead of completion?

A good compliance dashboard keeps completion, because it remains useful as proof of delivery. It adds measures of competence. Here are the most accessible ones.

1. The score on a delayed test. Asking three or four questions again a month later shows what remains.
2. Decisions in a scenario. Faced with a case, does the learner choose the right conduct, and justify it?
3. The quality of the conversation. Did they ask the right question, refuse clearly, avoid overcommitting?
4. Field signals. Referrals to the contact person, reports, questions escalated, errors caught in controls.
5. Progress from one attempt to the next. If the learner improves when replaying, the training is having an effect.

Our article on measuring the effectiveness of security awareness training takes up this logic on the cyber side.

What a spoken role-play brings

A role-play makes people produce an answer. The learner must speak, choose their words, hold their position, and does so in front of a counterpart who reacts. This is the form of retrieval effort described by research on testing, applied to a professional action.

It also produces a richer record. Instead of a score, you have what the person said and what they avoided. You also see their progress from one session to the next. A trainer can reread these elements and decide where to focus their support.

Three levels of evidence to distinguish

To steer a compliance program honestly, it is useful to separate three levels, which the dashboard often mixes.

The level of delivery

Who received the training, when, for how long? This is completion and attendance. This level answers the question “did we do what was planned?”.

The level of understanding

Did participants retain the essentials? A test at the end of the path, then a delayed test, answers this question. It stays in the register of knowing, which is already more informative than attendance.

The level of behavior

Do participants do the right thing when the situation arises? Only role-play and field observation answer this. It is the hardest level to reach, and the most valuable for reducing a real risk.

A serious program reports on all three levels, without hiding one. It admits that the third is the least well covered, and plans how to improve it.

How to move from a module to a practice

There is no need to redo everything. A gradual path is enough.

1. Keep the existing module for the information common to all.
2. Choose the two or three situations where the gap between knowing and doing costs the most.
3. Add a short role-play for each situation, played a few weeks after the module.
4. Compare the first and second attempts, then the result from one quarter to the next.
5. Adjust the module based on what practice reveals as poorly understood.

This loop gives a second benefit: practice shows up the weak points of the content. If all learners stumble on the same question, the problem often lies with the module, not the learners.

Using role-play as a measure of competence

A role-play does not replace the information module: it comes after it, to check what holds. With Face Up, the trainer creates a scenario from their real cases (a customer who gets frustrated by a due-diligence question, for example). The learner talks to a persona played by an AI, then reads their debrief: objectives met or not, strengths, points to improve, quotes, action plan. They can replay.

Face Up integrates with LMSs through LTI 1.3. The online module therefore stays in your platform, and practice is added to it. For the principle, see our page on role-plays.

One last tip on method: tell employees from the start that the role-play is for learning, not for grading. If the exercise is felt as an exam, participants play it safe and hide their hesitations. If it is felt as training, they try, miss, replay and improve.

Common mistakes when steering compliance

• Taking completion for the result. It proves delivery, nothing more.
• Making the final quiz too easy. A quiz everyone passes teaches the person running the program nothing.
• Measuring only straight after. Satisfaction on the day says nothing about the effect a month later.
• Multiplying indicators. Three well-chosen measures are worth more than fifteen.
• Hiding poor results. A team that fails in a role-play is useful information, not a failure to hide.

Frequently asked questions

Should compliance e-learning be abandoned?

No. It suits spreading information common to everyone and keeping a record. It becomes insufficient when you expect a behavior. The ideal is to keep it for knowledge and add practice for the action.

What should you say to an auditor who asks for evidence of effectiveness?

Show completion, but also delayed test results, scenario assessments and field indicators. Our article on proving training in an audit details the documents to keep.

How do you stop employees clicking without reading?

By changing what you ask of them: producing an answer rather than approving a screen. The subject of fatigue is covered in our article on fatigue with annual training.

To see how a role-play can complement your modules and measure something other than the click, book a demo.