Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How do we check a metric's data before it is reported?

6 decision pointsThe pack behind this answer: Cybersecurity Metrics & Reporting Programme

Add to board memo

The short answer

Recommendation
Before any figure is used, have a validator who is not the supplier run three checks on every input: completeness, freshness and plausibility. Send a failed input back to the supplier, and if it cannot be fixed in time, let the metric owner report it with a printed caveat or withhold it.
Reasoning
Looking at the number is not enough. In the pack's two worked failures, a stale export and a mis-dated exercise both gave values that looked right. Completeness compares the items with a value against the scope count, so a missing input shows as missing and never as zero. Freshness is checked on the recorded extract date, never on whether the value looks current. Plausibility tests the definition, the range, a cross-check against any other count of the same things, and the change from last period. Nobody checks their own homework, and a failed figure is never reported silently or replaced by last period's value.
Considerations
  • The supply deadline of 3 working days after the period end and the checks finished by working day 5 are the defaults in the CISO Times metrics pack's monthly cycle. Your own cycle may set different ones; keep the order.
  • If the named validator is away, the deputy named on the definition checks the input. In a small organisation, a finance or operations manager who can read the source is enough. A provider never validates its own figures.
  • A zero returned because the source system was down on extract day counts as missing. An older extract is replaced, never adjusted.
  • A change larger than the definition allows is accepted once the supplier explains it, such as a new site joining the scope. The explanation goes in the owner's commentary.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartThe period has ended and the inputs are dueStepThe supplier enters each input with its extract dateDecisionIs the validator someone other than the supplier?OutcomeDo not self-check: use a deputy, or withhold the figureDecisionIs there a value for every item in the metric's scope?DecisionWas it extracted after the period end and by the deadline?DecisionDoes it follow the definition, fit the range and cross-check?DecisionIs the change from last period within the definition's limit?StepLog the failure and send the input back to the supplierDecisionIs a fixed input back in time to be checked again?OutcomeThe metric owner decides: report with a caveat, or withholdOutcomePassed: release it for status and commentary

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 12 steps.

  1. Start

    The period has ended and the inputs are due

    Every input passes three checks before any figure is reported: completeness, freshness and plausibility. The validator runs them, not the person who supplied the input. Looking at the number is not enough: in the pack's two worked failures, a stale export and a mis-dated exercise both gave values that looked right.

    In the pack

  2. Step

    The supplier enters each input with its extract date

    Each supplier enters one row per metric in the collection workbook: the extract date, the number of items with a value, the value and last period's value. The pack's default supply deadline is 3 working days after the period end. A missing input is left blank, never entered as zero. Keep a copy of the workbook for each period: together the copies show what was checked, what failed and what was reported.

    In the pack

  3. Decision

    Is the validator someone other than the supplier?

    Nobody checks their own homework. If the named validator is away, the deputy named on the definition checks the input. In a small organisation, a finance or operations manager who can read the source is enough. A provider never validates its own figures.

    In the pack

  4. Outcome

    Do not self-check: use a deputy, or withhold the figure

    An input checked by the person who supplied it has not been checked. Name a deputy validator on the definition, or withhold the figure. A withheld figure is shown as 'Not available' with the reason, never as last period's value.

    In the pack

    See also

  5. Decision

    Is there a value for every item in the metric's scope?

    Count the items with a value and compare that with the scope count on the definition: the two must be equal. A missing input is recorded as missing, never as zero. A zero returned because the source system was down on extract day counts as missing.

    In the pack

  6. Decision

    Was it extracted after the period end and by the deadline?

    Freshness is checked on the recorded extract date, never on whether the value looks current. An extract taken even one day before the period end fails, and so does last period's file used again. An older extract is replaced, never adjusted. With no extract date recorded, the input fails.

    In the pack

  7. Decision

    Does it follow the definition, fit the range and cross-check?

    The value must count what the definition says it counts, sit within the definition's plausible range, and agree with related figures, including any other process's count of the same things. The cross-check is the strongest test: where two counts disagree, one of them is wrong. In the pack's example, a live incident entered as an exercise was caught only by the definition and a cross-check with the exercise programme.

    In the pack

  8. Decision

    Is the change from last period within the definition's limit?

    The definition sets the largest change from last period accepted without explanation. A larger change is accepted once the supplier explains it, for example a new site joining the scope. The explanation goes into the owner's commentary, and the scope count is updated if the scope grew.

    In the pack

  9. Step

    Log the failure and send the input back to the supplier

    Never overwrite the failed input. Record which check failed and what was found, and return the input to the supplier to fix. The checks then run again on the fixed input, entered as a new submission. The pack's default is for the checks to finish by working day 5 after the period end.

    In the pack

  10. Decision

    Is a fixed input back in time to be checked again?

    The checks must pass before the figure goes into any report. If the fixed input arrives in time, run all three checks on it again. If it will not, or you cannot tell, the figure is held and the metric owner decides what happens to it.

    In the pack

  11. Outcome

    The metric owner decides: report with a caveat, or withhold

    A figure that failed and could not be fixed in time is reported with a printed caveat or withheld. It is never reported silently, and never replaced by last period's value. The caveat says what is wrong and which way the error runs; a withheld figure is shown as 'Not available' with the reason.

    In the pack

    See also

  12. Outcome

    Passed: release it for status and commentary

    A figure that passed, or passed after correction, is ready to report. The metric owner then sets the status or zone and writes the commentary. Every failure in the period is counted and shown to the executive committee. Once a quarter, recalculate a few reported figures from the workbook and the definition, so the first time is not in front of an auditor.

    In the pack

Practical checklist

  1. Have each supplier enter each input with its extract date, the number of items with a value, the value and last period's value. Leave a missing input blank.
  2. Confirm the validator is not the supplier. If no one else can check it, withhold the figure.
  3. Check completeness and freshness: the items with a value match the scope count, and the extract was taken on or after the period end and by the deadline.
  4. Check plausibility: the value follows the definition, sits in the range and agrees with any cross-check.
  5. Check the change from last period is within the definition's limit, or explained by the supplier.
  6. Log any failure without overwriting the input, and send it back to the supplier. Run all three checks again on the fixed input.
  7. If it cannot be fixed in time, have the metric owner decide: report it with a caveat saying what is wrong and which way the error runs, or show it as "Not available" with the reason.
  8. Count every failure in the period, show it to the executive committee, and once a quarter recalculate a few reported figures from the workbook and the definition.

Evidence to have ready

  • The collection workbook kept for each period, with extract dates, check results and notes.
  • The quality log of failures, corrections, caveats and withheld figures.
  • Evidence that each input's validator was not its supplier.
  • The quarterly recalculation of reported figures from the workbook and the definition card.

Assumptions

Not covered here