Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How do we tell whether evidence actually proves the level someone claims?

9 decision pointsThe pack behind this answer: Security Maturity & Internal Assessment

Add to board memo

The short answer

Recommendation
Check the claim against what the level needs for that practice, then test the evidence in order: shown, not described; records, not only a procedure; dated in the evidence window; a sample the assessor chose; and for Managed, a reported measure, an independent check and a recorded improvement.
Reasoning
A claim usually comes from the person who runs the practice and describes what should happen. The assessor's job is to find what does happen. An interview alone, or a written procedure with no records of it being done, supports Informal at most (rule MA-02). Records the owner picked show the best case, so the assessor chooses the sample from a full list, and every sampled record must pass unless the practice itself caught, corrected and recorded the failure (rule MA-05).
Considerations
  • The 12-month evidence window and the sample of 3 records are the pack's defaults. Your own method may set others.
  • A practice that runs less often than yearly is judged on its own last cycle.
  • Managed also needs at least 6 months at Defined, with records, and a sample the assessor tested. Documents read but not sampled support Defined at most.
  • A proven level describes how a practice works. It is not a statement of compliance.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartSomeone claims a practice is Defined or ManagedStepRead what the claimed level needs for this practiceDecisionWas anything shown, or only described?OutcomeInformal at most: a description proves no moreDecisionIs it only a procedure or policy, with no records of it done?OutcomeInformal at most: a document alone does not prove DefinedDecisionAre the records dated inside the evidence window?StepSet them aside and ask for records in the windowDecisionDo any in-window records exist?DecisionDid the assessor choose the sample from a full list?StepDraw your own sample from an unfiltered sourceDecisionDid any sampled record fail the practice?DecisionDid the practice itself catch, correct and record it?OutcomeClaim not proven: rate the level the evidence showsDecisionIs Managed claimed?DecisionIs there a reported measure, an independent check and an improvement?OutcomeDefined, not Managed: say what Managed still needsOutcomeClaimed level proven, on evidence another assessor can check

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 18 steps.

  1. Start

    Someone claims a practice is Defined or Managed

    A claim usually comes from the person who runs the practice, and it usually describes what should happen. The assessor's job is to find what does happen. Evidence decides the level: a document or record that shows the practice is done as the level requires.

    In the pack

  2. Step

    Read what the claimed level needs for this practice

    Defined needs a written, agreed way of doing the practice and records in the evidence window showing it was done routinely, seen by the assessor. Managed needs all of that and more. Each practice also has a judging note: for example, a review that removed nothing across several systems deserves a sample check, because an approve-all review is Informal.

    In the pack

  3. Decision

    Was anything shown, or only described?

    Ask the owner to show, not describe: the record, the ticket, the minutes, the screen. Start with the last time the practice ran. Evidence promised but not shown counts as described; the owner can still add it when the draft results are shown.

    In the pack

  4. Outcome

    Informal at most: a description proves no more

    Record the confidence as Told. An interview alone supports Informal at most, however detailed it is. In the pack's example, HR said every joiner acknowledges the policies at induction but could not show the records, so the practice was capped at Informal. The description tells you what records to ask for next time.

    In the pack

  5. Decision

    Is it only a procedure or policy, with no records of it done?

    A written procedure shows how the practice should run. It does not show that it runs. Defined needs the practice done routinely and leaving a record, so look for the tickets, minutes, logs or sign-offs it produces.

    In the pack

  6. Outcome

    Informal at most: a document alone does not prove Defined

    Record it as Seen, at Informal at most. The same applies to a role description for a vacant post, a classification scheme never applied to real information, or a strategy with no dated, owned objectives. Say in the note which records would show the practice is done.

    In the pack

  7. Decision

    Are the records dated inside the evidence window?

    By the pack's default the window is the 12 months up to the last day of fieldwork. A practice that runs less often, such as a yearly exercise, is judged on its own last cycle. Old records show the practice once ran, not that it runs now. A record dated after the last day of fieldwork does not raise this assessment's level; it counts for the next one.

    In the pack

  8. Step

    Set them aside and ask for records in the window

    Write the dates you relied on in the note. If no records in the window exist, the claimed level is not met.

    In the pack

  9. Decision

    Do any in-window records exist?

    An answer that the records do not exist is an answer, not a failure to reply. It decides the level.

    In the pack

  10. Decision

    Did the assessor choose the sample from a full list?

    Records the owner picked show the best case, not the routine. Choose the sample yourself, by the pack's default 3 records or all of them if there are fewer, from a list the practice does not control: leavers from HR records, not the access system; findings from the scanner's export, not the remediation tracker. If unsure who chose them, answer no.

    In the pack

  11. Step

    Draw your own sample from an unfiltered source

    Ask for the full list and choose from it. If the list itself is incomplete, that is a finding against the practice. Record each sampled record's reference and what it showed.

    In the pack

  12. Decision

    Did any sampled record fail the practice?

    Check each sampled record against what the practice needs: approved before access, removed within the set time, decided at the right level. If you cannot tell whether a record passes, treat it as failed and say why in the note.

    In the pack

  13. Decision

    Did the practice itself catch, correct and record it?

    A failure the practice caught itself is evidence that it works. In the pack's example, one of five sampled leavers did not have access removed in time, but the monthly reconciliation found and removed it, so the practice caught its own failure. A failure that only the assessor found is evidence that it does not work.

    In the pack

  14. Outcome

    Claim not proven: rate the level the evidence shows

    Rate the practice at the highest level the evidence does support, and write the note as a finding: what was found, what the claimed level needs, and the evidence seen. The owner may add in-window evidence when the draft results are shown; the lead decides and records any disagreement.

    In the pack

  15. Decision

    Is Managed claimed?

    Managed is a claim that the practice is measured, checked by someone else and improved, on top of all of Defined. A practice that is measured but not written down is Informal, not Managed.

    In the pack

  16. Decision

    Is there a reported measure, an independent check and an improvement?

    Managed needs a measure with a target that is reported, a check by someone other than the person who does the work, and a record of at least one improvement made because of them. The assessor must have tested them on a sample; documents read but not sampled support Defined at most. The practice must also have run at Defined, with records, for at least 6 months.

    In the pack

  17. Outcome

    Defined, not Managed: say what Managed still needs

    Rate it Defined and name the missing element in the note: the measure, its target, the independent check or the recorded improvement. If the assessor runs the practice, the result is self-assessed and stays at Defined until someone independent checks it.

    In the pack

    See also

  18. Outcome

    Claimed level proven, on evidence another assessor can check

    Record the level, its confidence (Seen supports up to Defined; Managed needs Tested, and a self-assessed result stays at Defined), the sampled records and what the next level needs. Keep the evidence references on the assessment file so the result can be decided again from it. A proven level describes how the practice works; it is not a statement of compliance.

    In the pack

    See also

Practical checklist

  1. Read what the claimed level needs for this practice, including its judging note.
  2. Ask the owner to show the record, starting with the last time the practice ran.
  3. Look past the procedure for the tickets, minutes, logs or sign-offs it produces.
  4. Set aside records dated outside the evidence window.
  5. Draw your own sample from a list the practice does not control, and check each record.
  6. For Managed, look for the reported measure with its target, the independent check and a recorded improvement.
  7. Rate the highest level the evidence supports, and write what the claimed level still needs.

Evidence to have ready

  • The note for each practice: the records seen, their dates and the confidence, Told, Seen or Tested.
  • The source list each sample was drawn from, and each sampled record's reference and result.
  • For a failed record, whether the practice caught, corrected and recorded it.
  • The evidence references kept on file so another assessor can decide the level again.

Assumptions

  • Each practice has written criteria for what Defined needs and what Managed adds, so a claim can be checked against something other than opinion.
  • The assessor can get the full, unfiltered lists that samples are drawn from: leavers from HR, findings from the scanner's export, every exception.
  • The evidence window of 12 months and the sample of 3 records are the pack's defaults. Your own method may set different ones.

Not covered here