Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How do we test disaster recovery without causing the outage we are testing for?

8 decision pointsThe pack behind this answer: Business Continuity, Disaster Recovery & Cyber Resilience

Add to board memo

The short answer

Recommendation
Run the test on a copy, or in the quietest period with a checked backup first. Give it a stop point, a way back and one role who may stop it, and get the business owner's approval before it starts. Then report the time and the data lost against the objectives.
Reasoning
A recovery test that takes the service down becomes the outage it was meant to prepare for. The safeguards are small and written down in advance: where it runs, when it stops, how the service goes back, who may stop it and who has been told. Under the pack's rule (RR-14), only a timed recovery of the service proves its recovery time, so a tabletop or a workaround trial does not count as a recovery test. Both still help with decisions, roles and communication. A test in normal conditions does not show recovery after an attack.
Considerations
  • The intervals and the 10 working day deadlines are the defaults in the CISO Times continuity pack. Your own standard may set different ones.
  • If an attack is suspected during the test, stop it and call the incident manager. Nothing fails over or is restored until the copy or site can be trusted.
  • Say in the report what the test did not show, such as recovery after a destructive attack, rather than claim no limits.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartA recovery test is due for a critical serviceDecisionDoes it recover the service or its systems and restore the data?OutcomeNot a recovery test: it does not meet the ruleStepBook it and write the plan against the objectivesDecisionCan it run on a copy, away from live data?StepRestore into an isolated place after checking the copyDecisionCan it run in the service's quietest period?OutcomeDo not run it on the live service yetStepTake a backup and check it before anything live changesDecisionIs there a stop point anyone can check without judgement?DecisionIs there a way back, with who does it and how long it takes?StepName who may stop it and tell people firstDecisionHas the business owner approved the plan?OutcomeNot ready: close the gap in the plan before running itStepRun it, time it from logs and stop at the stop pointDecisionDid a real disruption start, or an attack get suspected?OutcomeStop, take the way back, and the crisis lead decidesOutcomeStop and call the incident manager: P09 runs the incidentStepWrite the report within 10 working daysDecisionDid the time, the data lost or a finding miss an objective?OutcomeRaise an objective gap and agree the fixOutcomeUpdate the plans and keep the record

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one. A code beside a source, such as RR-17, is the number of a rule in that pack document; the document lists its rules.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 22 steps.

  1. Start

    A recovery test is due for a critical service

    A test that takes the service down becomes the outage it was meant to prepare for. The pack's defaults: test every critical service's recovery at least every 12 months and after a significant change, and a service never tested within 6 months of its objectives first being agreed. Your own standard may set other intervals. This flow is about running the test safely.

    In the pack

  2. Decision

    Does it recover the service or its systems and restore the data?

    Only a recovery test meets the rule. It recovers the service or its systems, restores its data, and records the time achieved and the data lost against the objectives. Where a supplier runs the systems, the test is the supplier's own recovery test, with its results shared and checked against the objectives. If you are not sure it restores the data, treat it as not a recovery test.

    In the pack

  3. Outcome

    Not a recovery test: it does not meet the rule

    A tabletop, a drill or a workaround trial is good for decisions, roles, communication and finding gaps. It restores nothing, so it cannot prove a recovery time and is not a recovery test under the pack's rule (RR-14). A failover test counts only when it recovers everything the service needs and measures time and data lost against the objectives. A failover test or a trial still needs its own approved plan with the same safeguards (RR-26). Plan the recovery test as well.

    In the pack

  4. Step

    Book it and write the plan against the objectives

    Book the test away from the service's busiest periods and any freeze. Write success as numbers against the service's objectives before the test, so nobody judges it afterwards by how it felt. The test reads the objectives from the objectives matrix and never sets them. The resilience coordinator writes the plan with the technical recovery lead.

    In the pack

  5. Decision

    Can it run on a copy, away from live data?

    Recover into a test tenancy, a spare environment or standby servers, never over live data. Check first that nothing in the copy can write back to production, send mail to customers or reach a supplier's live interface. If you cannot show that, the answer is no.

    In the pack

  6. Step

    Restore into an isolated place after checking the copy

    Where the data comes from an immutable or offline copy, restore it into an isolated place, never straight onto production. Check the copy's integrity before restoring, and check the restored data afterwards. If the test also restores a data set that the backup pack tracks, cite that restore record in the test report.

    In the pack

    See also

  7. Decision

    Can it run in the service's quietest period?

    Where a test must touch the live service, as a failover test or a workaround trial does, run it when the service is least used. Take the busiest periods from the business impact analysis, process by process, and keep out of any freeze. If the busiest periods are not known, the answer is no.

    In the pack

    See also

  8. Outcome

    Do not run it on the live service yet

    Testing failover on the live service in its busiest weeks, with no way back, is the mistake the pack warns against. Find a copy to recover into, or agree a quiet period, then plan again. A test that is never run still leaves the service untested under RR-14.

    In the pack

  9. Step

    Take a backup and check it before anything live changes

    In a quiet period the test touches the live service, so take a backup first and check it. Without a checked copy there is nothing to go back to if the test goes wrong.

    In the pack

  10. Decision

    Is there a stop point anyone can check without judgement?

    A stop point is a time, a measure, or a sign that the live service is affected. At the stop point the test stops and nobody debates it. If it depends on someone's judgement on the day, the answer is no.

    In the pack

  11. Decision

    Is there a way back, with who does it and how long it takes?

    The way back is how the service returns to normal. For a failover test it is the failback, planned as for a real one, with the data written meanwhile copied back and reconciled. A test with no way back is not ready to run.

    In the pack

    See also

  12. Step

    Name who may stop it and tell people first

    One role, with a deputy, may stop the test at any point. Anyone who sees the stop point reached tells them at once. Tell the people who could mistake the test for a real outage: the service desk, the staff who use the service, the suppliers taking part, and the crisis lead.

    In the pack

  13. Decision

    Has the business owner approved the plan?

    The business owner of the service approves the test plan before anything runs, and the crisis lead is told. If you do not know whether the plan was approved, treat it as not approved.

    In the pack

  14. Outcome

    Not ready: close the gap in the plan before running it

    A missing stop point, way back or approval means the test could cause the outage it rehearses. Complete that part of the plan, have the business owner approve it, then run the test. Do not run a test on the promise that the gap will be filled on the day.

    In the pack

  15. Step

    Run it, time it from logs and stop at the stop point

    Time achieved runs from the moment the test treats as the decision to recover to the business owner's confirmation that the service works. Data lost is from the newest data the restored copy holds to the moment treated as the failure. Take both times from logs or the test record, never from memory. A participant who is not doing the recovery keeps the record.

    In the pack

  16. Decision

    Did a real disruption start, or an attack get suspected?

    A real failure during a test must not be mixed up with the test. If an attack is suspected, the copy or second site the test was using may be the attacker's too. If you are not sure whether it is an attack, take that branch.

    In the pack

  17. Outcome

    Stop, take the way back, and the crisis lead decides

    Stop the test and take the way back. The crisis lead decides what happens next.

    In the pack

    See also

  18. Outcome

    Stop and call the incident manager: P09 runs the incident

    Stop the test and call the incident manager. Under the pack's rule, nothing fails over or is restored until the incident manager confirms that the copy or site to be used can be trusted. The decisions about the incident belong to the incident response plan, not to this test.

    In the pack

  19. Step

    Write the report within 10 working days

    By the pack's default the report is due within 10 working days. It records the time achieved, the data lost and every finding, including what went as planned. It also says what the test did not show: a test in normal conditions does not show recovery after a destructive attack, and only a test that rebuilds the foundations in an isolated recovery environment measures the cyber recovery time. Say so rather than claim no limits.

    In the pack

  20. Decision

    Did the time, the data lost or a finding miss an objective?

    Compare the time achieved with the recovery time objective, the data lost with the recovery point objective, and the time with the maximum tolerable downtime. A finding that misses an objective is an objective gap. If you cannot tell from the logs, treat the objective as missed.

    In the pack

  21. Outcome

    Raise an objective gap and agree the fix

    Each finding that misses an objective becomes an objective gap. By the pack's defaults its owner agrees the change that closes it within 10 working days, with a due date, and the service's risk owner is told. If the time is beyond the maximum tolerable downtime, the service's status in the objectives matrix becomes Not achievable. Then write the lessons into the service's plan.

    In the pack

    See also

  22. Outcome

    Update the plans and keep the record

    Write the lessons into the service's recovery plan and bring the objectives matrix, the dependency register and the disaster recovery procedure up to date: the recovery method is marked as tested, with the time it took. Keep the approved plan, the report and the evidence for 3 years by the pack's default, or longer where law or a contract asks.

    In the pack

Practical checklist

  1. Confirm the test restores the service or its systems and its data, and records time and data lost.
  2. Write success as numbers against the service's objectives before the test.
  3. Choose a copy, or a quiet period with a backup taken and checked first.
  4. Write a stop point anyone can check, the way back, and who may stop the test.
  5. Tell the service desk, users, suppliers and crisis lead, then get the business owner's approval.
  6. Time the test from logs and stop at the stop point.
  7. Write the report within 10 working days and raise a gap for each missed objective.

Evidence to have ready

  • The approved test plan, with the copy or quiet period, stop point, way back and who may stop it.
  • The report with the time achieved and the data lost, each against its objective, taken from logs.
  • Each finding, and the objective gap with its owner and due date for any that missed an objective.
  • The updated recovery plan and records that the report lists.

Assumptions

  • Your critical services have recovery objectives agreed by their business owners, and the test reads them rather than setting them.
  • Each critical service has a named business owner and a technical recovery lead who can write and run the plan.
  • You can recover into a copy or a spare environment, or the service has a quiet period you can name.

Not covered here