How do we score maturity consistently when different people assess?

Start

Several people will rate practices in one assessment

Two assessors looking at the same evidence should give the same level. That only happens when the evidence for each level is written down, every result says how it was evidenced, and differences are caught before the results are shown. A level is still a judgement on evidence; the method narrows the room for it.

• Then: Give every assessor the same written criteria

Step

Give every assessor the same written criteria

For each practice: the question, the evidence Defined needs, what Managed adds, and a judging note on common cases that fall short. Not in place and Informal mean the same for every practice. Approve the model before the first assessment. Changing a practice's criteria later is a break in series for that practice.

• Then: Has each assessor worked the calibration cases this year?

Decision

Has each assessor worked the calibration cases this year?

The pack's model carries calibration cases: short findings with the level each should get, such as an owner who describes a weekly log review in detail but shows no record of one. Each assessor works them before their first assessment and every year after. If you do not know whether someone has, treat them as not calibrated.

• Yes: Brief assessors; owners explain records, never suggest a level
• No, or not known: Have each assessor work the cases alone, then compare

Step

Have each assessor work the cases alone, then compare

Compare each answer with the expected level and discuss every difference. A pattern of differences shows a practice or a level whose evidence needs clearer wording: change it in the model and note it in the calibration record, which an auditor may ask to see.

• Then: Brief assessors; owners explain records, never suggest a level

Step

Brief assessors; owners explain records, never suggest a level

Walk through the four levels and the criteria for each practice assigned. The security team or the practice owner may explain what a record is and where it is kept. They do not suggest a level. Agree how notes are written: what was found, what the next level needs, and the evidence seen.

• Then: Score every practice through the same steps, in order

Step

Score every practice through the same steps, in order

Check the practice applies. Set aside records outside the evidence window. Choose the sample yourself. Start at Not in place and go up one level at a time, stopping at the first level whose evidence is missing. A practice that is measured but not written down is Informal, because the levels build on each other.

• Then: How was the result evidenced?

Decision

How was the result evidenced?

Every result states its confidence, and the confidence caps the level. Told: described in an interview, with nothing shown. Seen: a document or record was shown, and it says what the practice needs. Tested: the assessor chose a sample of records and checked each one against the practice. Evidence promised but not shown is Told; the owner can add it at validation.

• Told: nothing shown: Cap the result at Informal
• Seen: a record shown: Cap the result at Defined
• Tested: own sample checked: Allow up to Managed, if Managed's evidence is there

Step

Cap the result at Informal

An interview alone supports Informal at most, however detailed and convincing it is. The description is still useful: it tells the assessor what records to ask for.

• Then: Does the assessor own or run this practice?

Step

Cap the result at Defined

Defined needs at least Seen, and Seen is enough for Defined when the record shown is the practice's Defined evidence. Where the assessor also checks a sample, every sampled record must pass. Seen is not enough for Managed, even if the documents describe measures, independent checks and improvements: in the pack's calibration case, a risk register read but not sampled is Defined.

• Then: Does the assessor own or run this practice?

Step

Allow up to Managed, if Managed's evidence is there

Tested supports any level, including Managed, but only the level the evidence shows. A sampled record that fails means the level is not met, unless the practice itself caught, corrected and recorded the failure. Managed also needs at least 6 months at Defined, with records.

• Then: Does the assessor own or run this practice?

Decision

Does the assessor own or run this practice?

A practice is your own work if you own it, do it, manage the people who do it day to day, or wrote the procedure it follows. If that is unclear, treat the result as self-assessed.

• Yes, or not sure: Mark it self-assessed and cap it at Defined
• No: Write a note another assessor could re-derive

Step

Mark it self-assessed and cap it at Defined

Whatever the confidence, a self-assessed result cannot be above Defined until someone independent checks the evidence, because Managed needs a check by someone other than the person who does the work. The report counts self-assessed results.

• Then: Write a note another assessor could re-derive

Step

Write a note another assessor could re-derive

Record what was seen, the reference of each sampled record and what it showed, the confidence, and what the next level needs. A note written this way is the finding if the practice is below target, and it lets someone else decide the level again from the file.

• Then: Lead decides a few practices again from the file

Step

Lead decides a few practices again from the file

On the last day of fieldwork the assessment lead decides practices again from the evidence on file, by the pack's default 3, including one from each assessor, and compares the result with the assessor's.

• Then: Does the lead's level differ from the assessor's?

Decision

Does the lead's level differ from the assessor's?

Settle every difference before the draft results are shown to practice owners. A difference found now costs a conversation; one found by an auditor costs trust in every result.

• No: Show owners the draft; they may add evidence, not argue
• Yes: Did they read the same evidence differently?

Decision

Did they read the same evidence differently?

Two causes look alike. One assessor may have had a record the other did not see; then share it and decide again. Or both read the same record and disagreed on what it shows; then the wording of the criteria is the problem.

• No: different evidence: Share the missing evidence and decide again
• Yes: same evidence: Settle it on the criteria, and fix unclear wording

Step

Share the missing evidence and decide again

Put the record on file so both work from the same evidence, then take the practice through the steps again.

• Then: Score every practice through the same steps, in order (back to an earlier step)

Step

Settle it on the criteria, and fix unclear wording

The lead decides on the criteria in the model. Where the wording allowed both readings, change it in the model and record the change in the calibration record; changes to the model are approved by the assessment sponsor. A change to a practice's evidence criteria is a break in series for that practice in later comparisons.

• Then: Show owners the draft; they may add evidence, not argue

Step

Show owners the draft; they may add evidence, not argue

Each practice owner sees the draft results and may add evidence within 5 working days by the pack's default. A record counts only if it is dated in the evidence window. The owner may not argue a level up without evidence. The lead decides, and records any disagreement and the reason.

• Then: Did the owner add evidence dated in the window?

Decision

Did the owner add evidence dated in the window?

New evidence goes through the same steps and the same confidence caps as the rest. A document written after fieldwork does not raise this assessment's level; it counts for the next one.

• Yes: re-rate it: Score every practice through the same steps, in order (back to an earlier step)
• No: Results another assessor could reproduce

Outcome

Results another assessor could reproduce

After each assessment, have an assessor who did not rate the practices take results from the report, by the pack's default 3, including one above the default target and one self-assessed, and decide them again from the file. They should match, and each domain should be the lowest of its practices. Running this test yourself means the first time is not in front of an auditor.