Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If a figure fails its quality check just before the report goes out, then what?

5 decision pointsThe pack behind this answer: Cybersecurity Metrics & Reporting Programme

Add to board memo

The short answer

Recommendation
Do one of three things, and only these: correct it, report it with a printed caveat, or withhold it. Never report a failed figure silently, and never put last period's value in its place.
Reasoning
Once a check has failed, the number cannot be trusted as it stands. If the supplier can fix the input in time, the validator runs all three checks again and the corrected figure is reported. If not, the metric owner decides: a caveat works when you can say what is wrong and which way the error runs; where the figure could still mislead, it is withheld and shown as "Not available" with the reason. Last period's value is never carried forward: in the pack's example, a stale file would have shown no movement in a month when the true figure had moved. A metric shown to two audiences uses the same value at both, so the outcome carries into the board's figures.
Considerations
  • The deadlines and the target that no figure goes out with a caveat or withheld are the defaults in the CISO Times metrics pack's monthly cycle. Your own cycle may set different ones.
  • If you cannot tell whether a fix will be back in time, treat it as not fixable. If you cannot tell whether a caveat makes it clear, withhold the figure.
  • Where the validator and the metric owner disagree on a caveat or withholding, the owner decides and the reason is recorded.
  • The same input failing twice running, or any figure caveated or withheld twice running, is a signal to fix the cause at its source and review the measurement method, which the pack has executive management approve.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartA figure fails its check just before the report goes outStepLog the failure, and never overwrite the failed inputDecisionCan the supplier fix the input in time to check it again?StepRun all three checks again on the fixed inputDecisionDoes the fixed input pass every check?OutcomeCorrected: report the fixed figureDecisionWould the figure still mislead with a caveat beside it?OutcomeReport it with a caveat printed beside the valueOutcomeWithhold it, and show 'Not available' with the reasonDecisionDoes the figure also feed this quarter's board report?StepCarry the same outcome into the board figures before sign-offStepCount it among the period's figures that failed a checkDecisionHas this failed, or gone out caveated or withheld, twice running?OutcomeFix the cause at its source, and review the methodOutcomeCheck next period's input afresh

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 15 steps.

  1. Start

    A figure fails its check just before the report goes out

    Once a check has failed, three outcomes remain: corrected, reported with a caveat, or withheld. A failed figure is never reported silently, and never replaced by last period's value. In the pack's example, a stale file reused from the month before would have shown no movement in a month when the true figure had moved.

    In the pack

  2. Step

    Log the failure, and never overwrite the failed input

    Record which check failed, what was found and the value supplied, in your collection workbook's quality log. The failed input stays as it was; a corrected input is entered as a new submission. The log is how you later show what was checked, what failed and what was reported.

    In the pack

  3. Decision

    Can the supplier fix the input in time to check it again?

    The failed input goes back to the supplier. A stale extract is replaced with a fresh one, never adjusted. If the fix will not be back before the report is built, or you cannot tell, treat the answer as no.

    In the pack

  4. Step

    Run all three checks again on the fixed input

    Completeness, freshness and plausibility all run again, by the validator, never by the supplier. In the pack's example, a stale supplier export was replaced by one dated the day after the period end, which passed all three checks.

    In the pack

    See also

  5. Decision

    Does the fixed input pass every check?

    Only a fixed input that passes all three checks is reported as corrected. If it fails again, log that failure too and return it to the supplier: each fixed input is a new submission, and the checks judge every one. Once there is no time left for another fix, the metric owner decides what happens to it.

    In the pack

  6. Outcome

    Corrected: report the fixed figure

    Report the corrected value with its target or thresholds, its direction since last period and its status in words. The failure still counts in the period's tally of figures that failed a check. Neither of the pack's two worked failures would have been caught by looking at the number.

    In the pack

  7. Decision

    Would the figure still mislead with a caveat beside it?

    The metric owner decides. A caveat works when you can say what is wrong and which way the error runs, for example that the true figure can only be higher. The pack's own calibration cases withhold a zero returned because the source was down, and last month's value reused because this month's extract had not arrived. Where the validator and the owner disagree, the owner decides and the reason is recorded.

    In the pack

  8. Outcome

    Report it with a caveat printed beside the value

    Print a short note beside the value saying what is wrong and which way the error runs. The metric owner agrees to report it on those terms. Show it, like every value, with its target or thresholds, its direction and its status in words.

    In the pack

  9. Outcome

    Withhold it, and show 'Not available' with the reason

    Show the figure as 'Not available' and say why. Leave the value blank, and never copy last period's value forward. The metric owner makes this call, and the reason is recorded.

    In the pack

  10. Decision

    Does the figure also feed this quarter's board report?

    In the last month of a quarter the same checked figures feed the board report, and a metric shown to two audiences uses the same value at both. The board's figures come from one data workbook, checked and signed off before the deck is built. If you do not know whether the figure feeds a board measure, treat it as if it does.

    In the pack

  11. Step

    Carry the same outcome into the board figures before sign-off

    Put the corrected value, the caveat or the withheld figure into the board's data collection workbook before its figures are signed off, and build the deck from that workbook. Never change a figure in the deck without changing it in the workbook first. The board and the executive committee then see the same numbers.

    In the pack

  12. Step

    Count it among the period's figures that failed a check

    Every failure, whatever its outcome, is counted and shown to the executive committee on its data quality slide. The count is one of four measures of the metrics programme itself. The pack's default target is that none are reported with a caveat or withheld.

    In the pack

  13. Decision

    Has this failed, or gone out caveated or withheld, twice running?

    The pack treats the same input failing a check two periods running, or any figure reported with a caveat or withheld two periods running, as a signal that the method needs attention. Look at your quality log for the last period before you answer. If you cannot tell from the log, treat it as a repeat.

    In the pack

  14. Outcome

    Fix the cause at its source, and review the method

    The same problem showing twice is a trigger to review the measurement method itself, and the pack has executive management approve that review. Trace the failure to its source and fix it there, so the next period's figure passes first time.

    In the pack

  15. Outcome

    Check next period's input afresh

    Next period's input is supplied as at the new period end and runs through the same three checks. Nothing from this period's failed input is carried forward into it.

    In the pack

Practical checklist

  1. Log which check failed, what was found and the value supplied. Never overwrite the failed input.
  2. Return the input to the supplier. A stale extract is replaced, never adjusted.
  3. If the fix arrives in time, have the validator run all three checks again on it as a new submission.
  4. If it passes, report the corrected figure.
  5. If not, have the metric owner decide: report it with a caveat saying what is wrong and which way the error runs, or show it as "Not available" with the reason.
  6. If the figure feeds this quarter's board report, carry the same outcome into the board's data workbook before sign-off, and build the deck from that workbook.
  7. Count the failure among the period's figures that failed a check, and show it to the executive committee.
  8. Check the quality log for a repeat, and if there is one, fix the cause at its source.

Evidence to have ready

  • The quality log, showing the failure, the check that failed and the outcome.
  • The original and corrected submissions, both kept.
  • The metric owner's recorded decision and reason for any caveat or withheld figure.
  • The board's signed-off data workbook showing the same outcome, where the figure fed the board report.

Assumptions

Not covered here