Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If an exercise finds a gap, then how do we make sure it is fixed and stays fixed?

5 decision pointsThe pack behind this answer: Cyber Incident Exercises & Tabletop Scenario Library

Add to board memo

The short answer

Recommendation
Give each finding one improvement action with one owner, a due date set by its priority and the document it changes. Keep the finding open until a re-test of the same decision is rated Met.
Reasoning
A fix nobody has shown to work can fail for real. In the pack's example, the one action still open failed again in a real incident five months after the exercise, as the exercise had predicted. Tying each action to a named document or control makes the change land where people will use it. The re-test proves it works, and the comparison across years shows whether it stays fixed: a category Not met twice running for the same audience becomes a finding in its own right.
Considerations
  • The due dates are defaults in the CISO Times exercise pack: High in 30 days, Medium in 90 and Low in 180, counted from the sponsor's approval of the report. Re-tests fall within 6 months of the fix. Your own procedure may set different ones.
  • A gap that would break a legal clock or a decision right in a real incident today goes to the sponsor the same day, before the report.
  • Write each finding as a condition that can be changed, never a person's name or a call for more awareness.
  • A live incident can serve as the re-test when its post-incident review rates the same decision.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartAn exercise has shown a gapDecisionWould the gap break a legal clock or decision right today?StepTell the sponsor the same day, before the reportDecisionWhat does the gap mean for the response?StepHigh: action due within 30 daysStepMedium: action due within 90 daysStepLow: action due within 180 daysStepRaise one action: one owner, one change, the document it changesStepThe sponsor approves the report within 10 working daysDecisionIs the action done by its due date?StepEscalate to the owner, then the sponsorDecisionWas the objective rated Partly met or Not met?StepRe-test the same decision within 6 months of the fixDecisionWas the re-test rated Met?StepKeep the finding open and rework the actionOutcomeClose it, and keep watching the category

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 16 steps.

  1. Start

    An exercise has shown a gap

    An observation is what someone saw, written as fact. It becomes a finding once the hot debrief or the after-action report confirms it as a gap that needs a change. A gap can sit on an objective rated Met: the pack's own example recorded one where the reporting screen was done in time but depended on one person knowing the answer.

    In the pack

  2. Decision

    Would the gap break a legal clock or decision right today?

    Some gaps cannot wait for the report. If the same gap would break a legal clock or a decision right in a real incident now, the exercise lead tells the sponsor the same day. If you cannot rule it out, treat it as yes.

    In the pack

  3. Step

    Tell the sponsor the same day, before the report

    A High finding runs 30 days from the report's approval, but the sponsor may act sooner, and the pack's escalation has the sponsor act the same day. The finding still goes into the report and gets its action like any other.

    In the pack

  4. Decision

    What does the gap mean for the response?

    The priority comes from the objective's rating, or from what the gap would do in a real incident. An objective rated Not met always gives at least one High finding; one rated Partly met gives at least a Medium one. A gap on a Met objective can still be a finding, usually Medium or Low.

    In the pack

  5. Step

    High: action due within 30 days

    An objective Not met, or a gap that would break a legal clock, a decision right or a Severity 1 or 2 step. The pack's default is 30 calendar days from the approval of the after-action report.

    In the pack

  6. Step

    Medium: action due within 90 days

    An objective Partly met, or a gap that would slow the response. The pack's default is 90 calendar days from the approval of the after-action report.

    In the pack

  7. Step

    Low: action due within 180 days

    An improvement that would make the response easier. The pack's default is 180 calendar days from the approval of the after-action report.

    In the pack

  8. Step

    Raise one action: one owner, one change, the document it changes

    Every finding gets an improvement action with one owner, by role, a due date set by its priority, and the document or control it changes, down to the section or playbook step. 'Review the plan' with no section or date is sent back. Write each finding as a condition that can be changed, never a person's name or 'people need more awareness'.

    In the pack

  9. Step

    The sponsor approves the report within 10 working days

    In the pack's defaults the draft reaches the sponsor 7 working days after the exercise and is approved within 10. Due dates count from that approval, and the actions go into your tracker the same day. The report also lists every plan and playbook the exercise tested, changed or not, with the reason, so each fix lands in the document people will use.

    In the pack

  10. Decision

    Is the action done by its due date?

    An action is Open, In progress, Done, Closed or Withdrawn. In progress leaves the due date unchanged. An action is withdrawn only with the sponsor's recorded reason. If you do not know whether it is done, treat it as not done.

    In the pack

  11. Step

    Escalate to the owner, then the sponsor

    An action past its due date goes to its owner, then to the exercise sponsor, who sets a recovery date or withdraws it with a reason. Overdue exercise actions are reported to executive management each quarter. The pack's target is none more than 30 days overdue.

    In the pack

  12. Decision

    Was the objective rated Partly met or Not met?

    These objectives must be re-tested before their action can close. A finding on a Met objective, or one tied to no objective, needs no re-test: it closes once the change is made.

    In the pack

  13. Step

    Re-test the same decision within 6 months of the fix

    Put the re-test on the programme when the action is marked Done, at the next exercise or drill of the audience that takes the decision. It tests the same decision, in the same category, in whatever scenario comes next. A live incident can serve when its post-incident review rates the same decision. The finding stays open until then.

    In the pack

  14. Decision

    Was the re-test rated Met?

    Only a Met closes the finding; until then the action stays Done, awaiting its re-test. A fix nobody has shown to work can fail for real: in the pack's example, the one action still open, an out-of-hours alert route, failed again in a real incident five months after the exercise, as the exercise had predicted.

    In the pack

  15. Step

    Keep the finding open and rework the action

    The finding stays open, because only a Met closes it. The pack does not say how to rework an action whose re-test failed, so your own procedure decides: typically, revisit the action with its owner, change what the re-test showed was still missing, and re-test the same decision again.

    In the pack

  16. Outcome

    Close it, and keep watching the category

    Close the action once it is done and, where required, re-tested Met. Keeping it fixed rests on the comparison across years: a category Not met at two exercises in a row for the same audience goes to executive management as a finding in its own right. Record what held, and what must not change, so a later edit does not undo it.

    In the pack

Practical checklist

  1. Tell the sponsor the same day about any gap that would break a legal clock or a decision right now.
  2. Set each finding's priority from its objective's rating or what it would do in a real incident.
  3. Raise one action per finding with one owner by role, a due date and the section or playbook step it changes.
  4. Get the after-action report approved and enter every action in your tracker the same day.
  5. Escalate an overdue action to its owner, then the sponsor, who sets a recovery date or withdraws it with a reason.
  6. Schedule the re-test when the action is marked Done, for objectives rated Partly met or Not met.
  7. Close each action once done and, where a re-test is required, rated Met. Report overdue actions and re-tests every quarter.

Evidence to have ready

  • The approved after-action report with every objective rated and each finding's priority.
  • Your tracker entry for each action: owner, due date, the document changed and its status history.
  • The re-test rating for each Partly met or Not met objective, with its date against the 6-month window.
  • The sponsor's recorded reason for any action withdrawn or given a recovery date.
  • The quarterly report to executive management on overdue actions, outstanding re-tests and repeat Not met categories.

Assumptions

  • You hold findings and actions in your own tracker or register, such as the pack's observations and improvement tracker.
  • The priorities, due dates, the 10-working-day report approval and the 6-month re-test window are the defaults in the CISO Times exercise pack. Your own procedure may set different ones.

Not covered here