When do we know whether awareness is working, and what do we measure?

Start

You need to know whether awareness changes what people do

Completion shows that a course was delivered, never that it worked. The programme is judged on behaviour, with completion shown beside it. You can tell whether it works when each practice email closes, each month when the measures are read, and once a year when each theme is judged.

• Then: Set five behaviour measures, with completion and compromises beside

Step

Set five behaviour measures, with completion and compromises beside

The pack's defaults: reporting rate for a practice email, 40% at the group's grade; time to first report, 10 minutes or less; repeat clickers as a group share, 2% or fewer; incidents with a human factor, 2 or fewer a quarter; role training done by its due date, all. Beside them, training completion (95%) and phishing compromises (amber at 2, red at 4 incidents in 3 months), with the metrics pack's definitions unchanged.

• Then: Is completion the only figure you report today?

Decision

Is completion the only figure you report today?

Many programmes report one figure: the share of staff who finished the course. It says people took the course. It cannot show whether anyone reports sooner, gives fewer passwords to fake pages or sends fewer files to the wrong person.

• Yes, or not sure: Add the behaviour measures from sources you already hold
• No: Each month, show every value with its target and status in words

Step

Add the behaviour measures from sources you already hold

Reporting figures come from the simulation platform and the reporting route. Human factors come from the incident log, judged from each incident's root cause as what happened, never who, with the control that should have stopped it beside it. Role training comes from the training register. Count the real suspicious emails staff report each month as context.

• Then: Each month, show every value with its target and status in words

Step

Each month, show every value with its target and status in words

The status words are: on target; below target, within tolerance; outside tolerance. Never colour alone, never a combined index. Never carry last month's value forward: a month with no campaign shows 'not available'. A small organisation with too few events in a month can report quarterly.

• Then: Did the grade, or a measure's definition, change since last time?

Decision

Did the grade, or a measure's definition, change since last time?

Practice-email figures are compared only at the same grade. A change of grade, or of a measure's definition, source or thresholds, is a break in series. If you are not sure, treat it as a break.

• Yes, or not sure: Mark the break, and say 'not comparable' in place of a direction
• No: Is each behaviour measure on target?

Step

Mark the break, and say 'not comparable' in place of a direction

Show the new value without a direction. In the pack's example, September's 28% reporting at Grade 2 is not compared with August's 26% at Grade 1.

• Then: Is each behaviour measure on target?

Decision

Is each behaviour measure on target?

Read the reporting measures first: reporting is the behaviour the programme exists to build. The click rate is context only. It moves with the grade and the lure.

• On target: Are phishing compromises, or another people-risk KRI, amber or red?
• Below target, within tolerance: Owner writes the plan to reach target, with its date
• Outside tolerance: Owner says what is decided; executive management sees it

Step

Owner writes the plan to reach target, with its date

A measure short of target needs its owner's commentary, giving the plan and the date. In the pack's example, completion at 92% against 95% comes with a plan to run the course on every warehouse night shift by a set date.

• Then: Are phishing compromises, or another people-risk KRI, amber or red?

Step

Owner says what is decided; executive management sees it

Beyond the tolerance limit, the owner's commentary says what has been decided. Measures short of target or outside green go to executive management every month, with the plan's progress.

• Then: Are phishing compromises, or another people-risk KRI, amber or red?

Decision

Are phishing compromises, or another people-risk KRI, amber or red?

Phishing compromises, the metrics pack's indicator, counts incidents in the last three calendar months whose root cause was a phishing email; read any other indicator on a people risk the same way. Amber means the linked risk may be above appetite, and its owner reviews it. Red means it may be past tolerance: executive management is told and the risk is reassessed.

• Yes: Escalate to the risk owner, and re-plan within 10 working days
• No: Does any warning signal show?

Outcome

Escalate to the risk owner, and re-plan within 10 working days

Escalate as the metrics method's escalation table sets, counting from the date the value is validated. An indicator on a people risk entering amber or red is also one of the four re-plan triggers: re-plan the themes within 10 working days. Where a control should stop the harm, the theme follows its fix.

• Then: Does any warning signal show?

Decision

Does any warning signal show?

Reporting below 10% for two campaigns running at any grade; the same group failing to move up for 4 campaigns; credentials entered on more than a third of clicks; a complaint from staff or their representatives about a lure or how results were used. If you are not sure, treat it as a signal.

• Yes, or not sure: Look at the route, the grade or the control the signal points to
• No: Report monthly; the board sees completion through its own measure

Outcome

Look at the route, the grade or the control the signal points to

Low reporting suggests the reporting route is hard to use or not known. A group stuck at one grade suggests the grade is wrong or the training is not reaching it. Many credentials entered call for the fake sign-in theme again and a stronger sign-in control. The method is reviewed after any complaint, with HR and the Data Protection Officer.

• Then: Report monthly; the board sees completion through its own measure

Step

Report monthly; the board sees completion through its own measure

Every measure goes monthly to the security and IT leadership team. Executive management sees those short of target or outside green, with the plan's progress, in the monthly executive report. The board sees training completion through its board measure. Group results only, never names.

• Then: Once a year, judge each theme on its own measure

Outcome

Once a year, judge each theme on its own measure

In the plan's last month, compare each theme's measure before and after, at the same grade where it is a practice-email figure. Keep a theme whose behaviour moved or that has not run long enough to tell; change one whose behaviour did not move while the risk remains; drop one whose evidence has gone. Keep each month's dashboard: for a regulated entity it is the record that the programme was tested for effectiveness.