How do we score risks consistently when two people assess them?

Start

Two people will score the same risks

Two assessors will differ unless they share the same anchors, the same facts and the same tie-break rules. The aim is a rating two people reach independently, because the band decides who must know and who may accept. The numbers themselves matter less than whether the result can be reproduced.

• Then: Is each risk written as a scenario with one owner?

Decision

Is each risk written as a scenario with one owner?

Two people cannot score 'phishing' the same way, because each imagines a different event. A scenario fixes the threat, the weakness, the asset or service, and the consequence, so both assessors are scoring the same story.

• Yes: Use one scale with written anchors for every level
• No, or not sure: Rewrite it as a scenario before anyone scores it

Outcome

Rewrite it as a scenario before anyone scores it

Scoring a topic produces a number nobody can defend. Write the threat, weakness, asset or service and consequence first, and name the executive who would feel the harm.

Step

Use one scale with written anchors for every level

Keep one scale. If the organisation already has an enterprise risk scale, adopt it or map this one to it, keeping four levels of impact and likelihood. Each level needs anchors in plain words and, for money, thresholds that match what finance uses. Otherwise the board sees two meanings of 'Severe'.

• Then: Have these assessors calibrated together in the last year?

Decision

Have these assessors calibrated together in the last year?

Calibration is done before the method is first used and every year after. If you do not know when it last happened, treat it as not done.

• Yes: Each assessor scores alone, with a reason for every rating
• No, or not known: Run a calibration exercise on a few sample risks

Step

Run a calibration exercise on a few sample risks

Both assessors score the same risks independently, 5 in the pack's default, inherent and residual, with reasons. Compare and discuss. Run the pack's calibration test cases through any spreadsheet or tool that applies the method, including the register: each must give the expected band and position. With no second assessor in a small organisation, calibrate once a year with an outside reviewer.

• Then: Each assessor scores alone, with a reason for every rating

Step

Each assessor scores alone, with a reason for every rating

Impact is the worst credible consequence, rated once on whichever of five areas is worst: money, service, data, legal, reputation. It is never averaged. Likelihood is the chance over the next 12 months: use data where you have it, the anchor description where you do not, and the higher level where they disagree. Rate inherent, list controls, then rate residual. Write one sentence of reason for each rating.

• Then: Apply the same tie-break rules

Step

Apply the same tie-break rules

Between two levels, take the higher. Missing information raises the rating. A control counts only if it is in place and there is evidence it works; a planned control is a treatment action. A control under a security exception is scored as missing. Where personal data is involved, rate impact on the worse of the harm to the organisation and the harm to the people.

• Then: Do the two sets of ratings differ?

Decision

Do the two sets of ratings differ?

Compare each rating, inherent and residual, and the reasons behind them. The reasons usually show where the assessors parted.

• No: Record the agreed ratings and reasons in your register
• Yes: Did they work from different facts or control evidence?

Decision

Did they work from different facts or control evidence?

Often one assessor saw a restore test or a threat report the other did not. That is a gap in information, and clearer anchors will not close it. Settle the facts before arguing about the level.

• Yes, or not sure: Share the evidence, then both rescore
• No: Does the difference change the band or the position?

Step

Share the evidence, then both rescore

Put the evidence in front of both assessors and rescore the affected ratings. A control without evidence still does not count, however confident its owner is.

• Then: Do the two sets of ratings differ? (back to an earlier step)

Decision

Does the difference change the band or the position?

Only nine scores are possible on the 4 × 4 matrix, and one level can move a risk from Medium to High, or outside appetite. Read both scores on the matrix and against the category's appetite. If you are not sure, treat it as a change.

• No: same band and position: Discuss the one-level difference and agree
• Yes, or not sure: Have the Head of Information Security settle it

Step

Discuss the one-level difference and agree

A difference of one level on one rating is normal. Agree the rating and keep the reason that won. Nothing in the method needs to change.

• Then: Record the agreed ratings and reasons in your register

Step

Have the Head of Information Security settle it

Disputed ratings go to the Head of Information Security, who also checks every High or Critical rating before it is recorded, within 5 working days in the pack's defaults. Where one person is both assessor and Head of Information Security, a second person, such as an outside adviser, checks those ratings. A manager may disagree with a rating and give a reason, but may not choose a band.

• Then: Fix the anchor in the method, and record the change

Step

Fix the anchor in the method, and record the change

A difference that changes the band or position shows an anchor that needs clearer wording or a better example. Change it in the methodology, not in the assessors' heads, so the next pair of assessors benefits. Record the date, the assessors and the change in the calibration record.

• Then: Record the agreed ratings and reasons in your register

Step

Record the agreed ratings and reasons in your register

Keep the impact and likelihood ratings, not only their product, with the reason for each and for every change from inherent to residual. A 2 × 4 and a 4 × 2 both score 8, and the ratings show which kind of risk it is. The reasons are what let someone else re-derive the score later.

• Then: Does this quarter's check show a sign of drift?

Decision

Does this quarter's check show a sign of drift?

Each quarter, look for signals. A register entry whose band or position cannot be reproduced from its ratings. In the pack's defaults, more than a quarter of risks High or Critical, or none above Low. The same dispute recurring, or more than 1 in 10 ratings challenged. An incident on a scenario rated Low likelihood, or a consequence larger than its impact rating. Any headline measure off target for two quarters running. Treat each as a prompt to investigate.

• Yes: calibrate again: Run a calibration exercise on a few sample risks (back to an earlier step)
• No: Scores two assessors can reproduce, with reasons on record

Outcome

Scores two assessors can reproduce, with reasons on record

This is what an auditor tests: pick risks from the register, read the band from the matrix and the position from the appetite table, and get the same answer. Run the same test yourself each quarter so the first time is not in front of the auditor.