Measure awareness effectiveness beyond clicks

To measure the effectiveness of security awareness training, you need to look beyond the completion rate and the click rate. The useful indicators are behavioral: reporting rate, speed and quality of reports, success in spoken exercises and progress from one session to the next. Here is how to choose and read them.

The central idea: you do not measure what employees know, but what they would do. That is the link with the guide to security awareness training and vishing.

Why the completion rate is not enough

The completion rate shows that the module was opened and finished. It is a distribution indicator, useful for an audit, but it says nothing about future behavior. An employee can click on the right answers to finish faster.

The click rate of fake email campaigns is better, because it observes a real act. It remains limited to email, as our article on the limits of phishing simulation against voice explains.

These two figures answer the question “have we rolled it out?”, not “would the teams react well?”. You need others.

The four levels of evaluation, applied to cybersecurity

Kirkpatrick's model distinguishes four levels of training evaluation: reaction, learning, behavior and results. It offers a simple framework for sorting your indicators.

• Reaction: was the training judged useful and clear? Easy to measure, not very informative.
• Learning: are the rules known? A quiz is enough, with its limits.
• Behavior: do employees apply the rule in a situation? This is the decisive level.
• Results: incidents avoided, losses reduced, detection times shortened.

Most programs stop at the first two levels. The value lies in the next two. For the general method, see our methodology for measuring the impact of a training program.

Which behavioral indicators should you track?

The reporting rate

This is the proportion of employees who report a suspicious message or call, relative to the number of real or simulated situations. A growing rate indicates that the reporting culture is progressing. A zero rate can also mean that nobody dares speak.

The reporting delay

The earlier the security team is warned, the more it can act: block a transfer, warn other colleagues. Measure the time between the suspicious call and the report, in hours or minutes.

The quality of the report

A useful report contains the name given, the department, the number, the requests and what was disclosed. Our grid for reporting a security incident serves as the reference. You can score each report from 0 to 5 according to the number of elements present.

Success in verification exercises

It is measured on observable criteria. Did the learner ask for a case number? Did they offer to call back on the official number? Did they avoid giving a code? These points can be checked in the transcript of a role-play.

The false alarm rate

An employee who reports everything and anything drowns the security team. Track the share of relevant reports. The goal is the right alert, not volume.

Measuring progress in role-play

A spoken exercise produces richer data than a quiz. Each session can be assessed against the same list of criteria, which makes it possible to follow the progress of a person or a group.

With Face Up, the learner receives a debrief after each session. It shows the objectives met or missed, their strengths, their points to improve, an action plan and quotes of what they said. The trainer can rely on these elements to compare a first attempt and a replay.

Imagine a fake advisor scenario where the share of participants who offer a callback rises markedly between the first and the third session. This progress, observable in the transcripts, is more telling than a module score. The construction method is detailed in our article on vishing simulation through AI role-play.

How to score the quality of a report

A quality report is scored with a short grid, applied in the same way to everyone. A score out of 5 works well: one point per element present.

1. The name the caller gave.
2. The department or company announced.
3. The number displayed or provided.
4. What was requested (code, transfer, access, information).
5. What the person said or gave, including “nothing”.

Tracking this score over time shows whether teams know what to retain during a call. This skill can be trained. In the debrief led by the trainer after a role-play, everyone recounts their call as if they were reporting it. The group fills in what is missing.

Setting a starting point before launching the program

A measurement only makes sense by comparison. Before the first exercise, record the starting situation: reporting rate over the last twelve months, average quality of existing reports, success in a first spoken scenario.

Do not look for a flattering figure. A low starting point is good news for what follows, because it leaves a visible margin for progress. A team that reports little today can become, within a few months, one of those that report best, if it is trained and reassured.

Also decide, before starting, what you will do with the results. An indicator with no decision attached ends up ignored. For example: if the reporting delay exceeds a threshold in a team, an additional exercise is planned for it.

Presenting results to management and the audit committee

Management wants to know whether the risk is falling. Auditors want proof that the training exists and is followed. These two expectations can be met with the same data, presented differently.

• For management: three behavior indicators, their trend and a decided action.
• For the audit: the list of sessions held, the criteria used and the trace of debriefs.
• For the teams: their own progress, with concrete examples of well-said sentences.

For regulated organizations, proof of practice usefully complements proof of rollout. Our article on vishing simulation through AI role-play explains what these sessions can leave as a trace.

Building a simple dashboard

Four to six indicators are enough. Beyond that, nobody reads them. Here is a reusable base:

1. Reporting rate (by team, by quarter).
2. Median reporting delay.
3. Average quality of reports.
4. Share of participants who meet the key criteria of a verification exercise.
5. Change between the first session and the last, for the same population.
6. Click rate in phishing campaigns, as a complement.

For each indicator, set a starting point, a realistic target and a frequency of readings. Present results by team and never by person, to avoid a climate of surveillance.

Pitfalls to avoid

• Rewarding a low click rate alone. It may reflect a fake message that was too easy.
• Punishing mistakes. A person who fears sanction stops reporting.
• Comparing campaigns of different difficulty. Note the difficulty each time.
• Waiting for the incident to judge. Real incidents are rare, exercises are frequent.
• Forgetting the control group. When possible, compare a trained team with one that is not yet trained.

Frequently asked questions

What is the best indicator of good awareness training?

There is no single one. The reporting rate, combined with its quality and its delay, gives the best picture of behavior. Add spoken exercises for the telephone.

How do you measure vishing without surprise fake calls?

Through announced exercises, played as role-plays, with observable criteria. You get comparable data without trapping anyone and without legal stakes.

Can you measure a return on investment?

You can estimate avoided costs with caution, but rare incidents make the calculation fragile. It is better to track behavior and delay indicators, which are more stable from one year to the next. See also the before-and-after evaluation to prove progress.

How often should indicators be taken?

Once a quarter is a good rhythm for the dashboard. Exercises can be more frequent and shorter, which feeds the readings.

To follow your teams' progress on criteria your trainers define themselves, book a Face Up demo.