IT Disaster Recovery Procedure
Recovers the systems a critical service needs after a failure that is not an attack: the failover decision, each system's recovery method and runbook, and failback.
Available soon
- Format
- Word
- Size
- 75 KB
- Length
- 26 pages
- Version
- 1.1
- Updated
What's inside
- Purpose
- Legal note
- Scope
- If an attack may be the cause
- Roles
- When this procedure starts
- Recovery methods and runbooks
- The failover decision
- Steps for each failure
- Running a failover or a restore elsewhere
- Keeping people told
- Failing back
- Ending recovery
- After recovery
- When to escalate
- Pitfalls
- Related documents
- Annex 1: Each system's recovery method
- Annex 2: Runbook template
- Adapting this template
- Framework references
- Definitions
Preview
The document from section 1, as you will receive it. Highlighted [[text]] is for you to replace; shaded guidance boxes are for you to delete before approval. The cover and document control pages are in the file.
Purpose
This procedure sets out how [[Organisation Name]] recovers the systems its critical services need after a failure that is not an attack. It plans for five kinds of failure: a server, its storage or its database fails; a hosting provider's data centre or a cloud region goes down; a network line to a site fails; a faulty update stops devices or servers from starting; and a site is lost. For each it gives the first steps and the decision to repair, fail over or recover elsewhere. It then covers running the recovery, letting users back, failing back, and ending recovery. Annex 1 shows each system's recovery method in the example used throughout the pack; Annex 2 is a runbook template.
It applies rules RR-23 to RR-25 of the Business Continuity & ICT Resilience Policy. It works beside each service's recovery plan, written with the Cyber Recovery Plan Template, and the site and unit plans written with the Site & Business Unit Continuity Plan Template. Each service's objectives come from the Critical Service Recovery Objectives Matrix (RTO, RPO, MTD), and what each service depends on from the Resilience Dependency Register. After a destructive cyberattack, the Cyber Recovery Operating Procedure applies instead.
Guidance — delete before approval
Replace every value in [[double brackets]] before approval, and name roles, not people.
The settings in [[double brackets]], such as the hours for the first messages and how long records are kept, are shared with the Business Continuity & ICT Resilience Policy. Change them there and here together.
Keep a printed copy and an offline copy of this procedure, the runbooks and the contact lists. A failure that takes the file share takes this document with it.
Legal note
This document helps you plan and test business continuity and recovery from a disruption, including a destructive cyberattack. It is not legal advice, and following it is not a statement that you comply with NIS2, DORA, the GDPR, HIPAA, ISO 22301 or any other law, standard or set of criteria. Which continuity and recovery duties apply to you depends on your sector and the law that applies to you: confirm them with counsel, your auditor or your supervisor.
Where this procedure names a law, it says which text asks for what and whom it binds. The NIS2 Implementing Regulation binds only the providers its Article 1 names (NIS2 IR 1); other organisations may use its Annex as good practice. DORA binds financial entities only. HIPAA binds covered entities and business associates only. The Trust Services Criteria are not law: the note under the framework references says how they are cited.
A failure that is not an attack can still be an incident that a law or a contract asks you to report. Under the Security Incident Management pack (P09), the incident manager screens every incident against every reporting duty (IM-06), and P09 decides what counts as an incident. Notices to authorities stay with P09 and with P18, the Regulatory Incident Notification & Reporting pack (RR-27).
Scope
This procedure applies to the recovery of any system a critical service needs, whether it runs in-house or at a supplier, after a failure that is not an attack. It also applies to failover tests and recovery tests that use its runbooks, under their approved plans (RR-26), and to everyone, staff or supplier, who estimates, decides, runs or checks such a recovery.
The failures it plans for. The examples are the EXAMPLE organisation's:
Failure | EXAMPLE | |
|---|---|---|
DS-01 | A server, its storage or its database fails | The warehouse management system's database storage fails |
DS-02 | A hosting provider's data centre or a cloud region goes down | The hosting provider loses the data centre that runs online ordering, or the ERP provider loses one of its two data centres |
DS-03 | A network line to a site fails | Warehouse 2's line fails |
DS-04 | A faulty update stops devices or servers from starting | An update to the endpoint protection agent makes office PCs and servers fail at start-up |
DS-05 | A site is lost | Warehouse 2 cannot be entered |
DS-06 | An attack is suspected | Encryption activity on a server |
DS-06 is in the table so that nobody takes an attack for a failure. This procedure does not run for it: P09 runs the incident, and the Cyber Recovery Operating Procedure runs the recovery. Nothing here repeats either.
It leaves to others: incident response and reports to authorities (P09); restore testing of backups (P23, Backup & Restore Assurance); exercises (P10, Cyber Incident Exercises & Tabletop Scenario Library); test plans and reports (Continuity & Disaster Recovery Test Plan and Report); a site's own plan (Site & Business Unit Continuity Plan Template); message wording (Outage Communications Template Set); supplier contracts (P06 TP-10); and update roll-outs (P17).
If an attack may be the cause
Classic disaster recovery assumes the systems you recover with can be trusted: fail over to the second site, restore last night's backup, and carry on. A destructive cyberattack breaks that assumption. Replication copies the damage to the second site, the attacker may have been inside for weeks before the encryption, and the sign-in directory and administrator accounts you would recover with may be the attacker's. Recovery then starts with the foundations, rebuilt from sources you can trust, and every system is checked before it goes back.
RR-07 Plans assume a destructive cyberattack as well as a technical failure: production, its replicas and its administrator accounts may not be trusted. Each critical service's cyber recovery time is worked out and compared with its objectives (MX-05).
Signs that a failure may be an attack:
- Files renamed, encrypted or unreadable, or a ransom note
- Accounts, scheduled tasks or remote access tools that nobody recognises
- Alerts from endpoint protection or monitoring on the failed systems, or the protection switched off
- Several systems failing at once with no known cause, such as a power cut, a hardware fault or an update
- Backups deleted, changed or failing to run in the days before the failure
- A supplier reporting an attack on its own systems, or not saying why its service failed
If any sign is present, or nobody can yet say, report it through P09's reporting route at once. P09 IM-01: “Everyone must report a suspected security event at once through the reporting route; nobody is blamed for a report made in good faith.” The incident manager takes the report and runs it under P09. While an attack is suspected, nothing fails over and nothing is restored until the incident manager confirms that the copy or site to be used can be trusted (RR-23).
If the incident manager confirms an attack, this procedure stops for the systems concerned. P09 contains and eradicates, and the Cyber Recovery Operating Procedure recovers (DS-06). Hand the crisis log to the incident manager. If the incident manager confirms that a copy or site is clean, the hold is lifted for that copy or site only.
Roles
Role | Their part in this procedure |
|---|---|
Crisis lead [[e.g. Chief Operating Officer, the executive incident lead in a cyber incident (P09)]] | Decides to recover, to fail over or recover elsewhere (RA-22), to start a workaround, to fail back (RA-23), and when recovery ends (RA-17). |
Technical recovery lead [[e.g. Head of IT]] | Estimates the repair, advises on the failover decision, runs the runbooks, and plans the failback. |
Business owner [[the manager accountable for each critical service]] | Runs the workaround, tests the service and confirms it works before users return (RA-12), and agrees the time of the failback. |
Incident manager [[e.g. Information Security Manager (P09)]] | Takes the report of a failure under P09 (IM-01). Where an attack is suspected, confirms whether the copy or site to be used can be trusted (RR-23). |
Supplier liaison [[e.g. Procurement Manager]] | Calls on suppliers' recovery commitments and records their estimates (RA-14). |
Resilience coordinator [[e.g. a risk manager or business continuity manager who runs no system in scope]] | Keeps the crisis log and the times (RA-16), and raises the gaps afterwards. |
Communications lead [[e.g. Head of Marketing and Communications (P09)]] | Sends staff and customer messages from the outage templates (RR-27). |
Site or unit continuity lead [[the manager of each site or business unit, e.g. a warehouse manager]] | Takes a site's first steps at once, starts its workaround when a line or the building is lost, and accounts for its people. |
Head of Information Security [[e.g. Head of Information Security or CISO]] | Helps judge whether a failure may be an attack; in a faulty update to a security product (DS-04), works with its supplier on the fix. |
Every role has a named deputy, and no one is the first holder of two roles that must act in the same hour (RR-19).
The Crisis & Recovery Responsibility Matrix names, for each decision, who decides, who does the work, and who is consulted and informed. Where a supplier runs a system, it decides how its own platform recovers. Whether your service waits for it or recovers elsewhere is the crisis lead's decision (RA-22).
When this procedure starts
Event | What starts | Who starts it |
|---|---|---|
A system a critical service needs fails, or the supplier that runs it reports that it has failed, and it is not back within [[30 minutes]] | The first assessment below | Technical recovery lead |
The crisis lead invokes a critical service's continuity plan after a failure that is not an attack (RA-04) | The failover decision for each system the service needs | Crisis lead |
A site or business unit loses its building or its systems and its continuity plan moves its work (RR-22) | Recovery elsewhere of the systems that run at the site, where the moved work needs them (DS-05) | Site or unit continuity lead |
The first assessment
When a system fails, in this order:
Step | What happens | Who |
|---|---|---|
1 | Record when the system failed, or was first reported, and which critical services depend on it (Annex 1). | Technical recovery lead |
2 | Check for the signs of an attack in the section above. If any is present, or nobody can yet say, follow that section before anything else. | Technical recovery lead; Head of Information Security |
3 | Report the failure through P09's reporting route (IM-01), so P09 can decide whether it is an incident. | Technical recovery lead |
4 | Estimate when the system can be repaired where it is. Where a supplier runs or supports it, ask the supplier for its own estimate, and record both. | Technical recovery lead; Supplier liaison |
5 | Tell the crisis lead what failed, the services affected, the repair estimate, and the recovery method and time in the system's runbook. | Technical recovery lead |
6 | Decide to recover the service, and record the time of the decision: the recovery time objective runs from it. | Crisis lead |
7 | Open the crisis log and keep it until the end of recovery. | Resilience coordinator |
The recovery time objective counts from the decision to recover (step 6). The maximum tolerable downtime counts from the failure, because the harm starts then. Keep the gap between them short: in the illustration below it is 30 minutes.
Recovery methods and runbooks
RR-24 Every system a critical service needs has its recovery method recorded, with a runbook: repair in place, fail over to a second site or region, restore elsewhere from backup, rebuild, or the supplier's own recovery. The runbook gives the steps, the time they take, who may run them and how to fail back, and can be reached when the main systems are down. A method that has never been tested is recorded as untested.
The five methods:
Method | What it is | When it fits | What to watch |
|---|---|---|---|
Repair in place | Fix the failed system where it is: replace a part, repair a database, apply a vendor's fix. | The repair will finish within the time left, or no other route is quicker. | Estimates slip while you wait for parts or for the vendor. Set the time at which you will decide again. |
Fail over | Switch the system to a second site, region or copy that is kept running or ready, such as a replica database or a second data centre. | Systems whose recovery time objective is shorter than a restore would take. | Replication copies corruption, deletions and an attacker's changes as fast as good data. Failing back needs a plan of its own. |
Restore elsewhere | Bring the system up on other hardware, standby servers or another environment, and restore its data from backup. | Systems that can wait for a restore, and failures that took the hardware or the site. | The copies must be held apart from what failed and checked for integrity before use. The data lost is everything since the copy was taken. |
Rebuild | Install the system again from its build source and its stored configuration, then restore only its data. | A system whose own installation cannot be trusted or repaired, such as a corrupt operating system. | It needs current build sources and configuration baselines, kept where they can be reached when the main systems are down. |
Supplier's own recovery | The supplier that runs the system recovers it on its own platform, under its contract. | Hosted and cloud services that you do not run yourself. | Its time is what the contract gives, or unknown, until the supplier shares a test. You still recover your own configuration, accounts and data. |
Where a system has a second method for a wider failure, such as a restore elsewhere when the second site is lost as well, record it too, with its own runbook and time.
Runbooks and untested methods
Annex 2 is a runbook template. Write each runbook while the system works, by doing the steps, and time them. Keep it beside the service's recovery plan, and review it with the plan (RR-16).
An untested method's time is the contract's, or none: nobody has yet shown it can be met. Each critical service's recovery is tested at least every [[12]] months (RR-14), and a failover test needs a written plan, approved by the business owner, with a point at which it stops and a way back (RR-26). Plan and report both with the Continuity & Disaster Recovery Test Plan and Report. In the EXAMPLE, 3 of the 11 methods in Annex 1 have been tested.
The failover decision
RR-23 After a failure that is not an attack, the technical recovery lead estimates when each affected system can be repaired where it is. Where that is later than the time left within the service's recovery time objective, the crisis lead decides, on the technical recovery lead's advice, to fail over or recover elsewhere; where neither would be back in time, the service's workaround starts as well (RR-06). Where an attack is suspected, nothing fails over or is restored until the incident manager confirms that the copy or site to be used can be trusted (RR-07; P09 D3).
The decision compares two estimates with the time left. The time left is the recovery time objective less the time since the decision to recover. The two estimates are the repair in place, from the technical recovery lead or the supplier, and the alternative from the system's runbook: the time achieved in its last test, or the contract's time where it was never tested.
The decision, case by case:
Attack suspected, copy or site not yet confirmed? | Repair within the time left? | Failover or restore elsewhere within the time left? | Failover sooner than the repair? | Decision |
|---|---|---|---|---|
Yes | Any | Any | Any | Hold: the incident manager first confirms the copy or site can be trusted |
No | Yes | Any | Any | Repair in place |
No | No | Yes | Any | Fail over or recover elsewhere |
No | No | No | Yes | Fail over or recover elsewhere, and start the workaround now |
No | No | No, or there is no other route | No | Repair in place, and start the workaround now |
RR-06 Where a service cannot be back within its recovery time, its business owner records a workaround that keeps the harm below Severe until the service returns, and who starts it. A workaround that has never been tried is recorded as untried.
So the workaround starts with the decision whenever neither route is due back within the time left. Where the failover is the quicker of two late routes, it still runs, and the workaround carries the service until it is done. Plan on an untried workaround lasting less than its plan says.
Decide again whenever an estimate changes, with the time now left, and record each decision with the estimates it rested on.
Illustration: the warehouse system's storage fails
EXAMPLE. An illustration, not an event in the example's story. On a weekday at 06:10 the warehouse management system's database storage fails. At 06:40 the technical recovery lead estimates the repair in place at 10 hours, waiting for a replacement part; the last restore elsewhere took 5 hours. The early shift already holds the printed pick lists.
Time | What happens |
|---|---|
06:10 | The warehouse management system's database storage fails, and warehouse dispatch stops. Its maximum tolerable downtime of 24 hours runs from now: the harm reaches Severe by 06:10 the next day. |
06:40 | The technical recovery lead estimates the repair in place at [[10]] hours. The restore elsewhere took 5 hours in RV-2026-03 (27 Aug 2026). The crisis lead decides to recover: the 8-hour recovery time objective runs from now, so the target is 14:40. |
06:40 | The repair would finish at about 16:40, after the target. The restore would finish at about 11:40, before it. Decision: Fail over or recover elsewhere. The system's method is restore elsewhere, onto [[standby servers]] (Annex 1). |
About 11:40 | The system is ready for the business owner's test. Users go back when the Head of Logistics confirms that dispatch works (RA-12). |
The decision does not start the workaround, because the restore is due back within the time left. The early shift already holds the printed pick lists, so the crisis lead may still start the workaround to keep picking while the system is restored (RA-04).
The restore meets the time objective and can still miss the data objective. Stock and dispatch records are backed up every 24 hours (DP-10), and dispatch can lose at most 4 hours: objective gap OG-05, due to close on 15 Dec 2026. Until then, stock movements after the recovery point are found again by a count of the affected aisles.
Had anything pointed to an attack, the same estimates would give Hold: the incident manager first confirms the copy or site can be trusted (RR-23).
Once the storage is repaired, the system fails back as its runbook says: “Moved back to the usual servers in a planned evening window, after a count of the affected aisles.” An evening window avoids dispatch's busiest times: for picking, “November and December; the early shift on weekdays”, and for loading, “The afternoon collections”.
Steps for each failure
Each failure starts with the first assessment and the failover decision above. The steps below add what that failure needs. The examples are the EXAMPLE organisation's, and the systems named are in Annex 1.
A server, its storage or its database fails
DS-01. EXAMPLE: The warehouse management system's database storage fails. First move: the technical recovery lead estimates the repair; the crisis lead decides under RR-23.
Step | What happens | Who |
|---|---|---|
1 | Confirm what failed (the server, its storage or the database on it) and which critical services use it, from Annex 1. A corrupt database, or storage that fails with no hardware fault reported, deserves a check for the signs of an attack. | Technical recovery lead; Head of Information Security |
2 | Get the repair estimate from the hardware or software supplier's support, record it with its source, and advise the crisis lead with the runbook's time for the alternative (RR-23). | Technical recovery lead; Supplier liaison |
3 | Decide, and record the decision, the estimates and the time (RA-22). | Crisis lead |
4 | Run the decision. After a failover, keep repairing the failed system: it is where the service fails back to. Decide again when an estimate moves past the time left. | Technical recovery lead; Crisis lead |
The illustration under the failover decision is this failure.
A hosting provider's data centre or a cloud region goes down
DS-02. EXAMPLE: The hosting provider loses the data centre that runs online ordering, or the ERP provider loses one of its two data centres. First move: the supplier liaison calls on the provider's recovery commitment (RA-14); the crisis lead decides whether to restore elsewhere.
Step | What happens | Who |
|---|---|---|
1 | Call on the provider's recovery commitment: its contracted time, its own estimate, and how often it will report progress (RA-14). Read its notice for any sign of an attack; if there is one, or it will not say what happened, tell the incident manager and hold. | Supplier liaison; Incident manager |
2 | List every system the lost data centre or region ran, and the services each one serves, from Annex 1. | Technical recovery lead |
3 | Decide for each system, with the provider's estimate as the repair time and your own failover or restore elsewhere as the alternative (RR-23). | Crisis lead; Technical recovery lead |
4 | Before a restore elsewhere, confirm the backup copies were not held in the lost data centre or region. Before a failover, confirm the other region or data centre has the capacity, network routes, certificates and accounts the systems need. | Technical recovery lead |
5 | Record each of the provider's updates with its time, and decide again when its estimate moves. | Resilience coordinator; Crisis lead |
EXAMPLE. The cloud hosting provider's contract gives 24 hours, the whole of online ordering's 24-hour recovery time objective, and that time has never been tested. The platform's own restore elsewhere took 96 hours in RV-2026-02 (20 May 2026), beyond its 48-hour maximum tolerable downtime (OG-01). Here the repair is the provider's own recovery. Whenever the provider's estimate is later than the time left, the decision is Repair in place, and start the workaround now: waiting is still sooner than the restore. Only an estimate beyond 96 hours makes the restore the quicker route (Fail over or recover elsewhere, and start the workaround now).
When the failure is a cloud region
The provider's commitment and its shared-responsibility model may be the binding limit on your recovery time. The provider keeps its own services running. For many services, you decide how your systems use more than one region or zone, and you run the failover and the failback. Cloud guidance says the split depends on the service. Ask, for each service:
- Who runs the failover and the failback, you or the provider, and what does the provider's commitment say about the loss of a whole region?
- Do the failover steps need the failed region? Provider guidance, such as AWS's disaster recovery whitepaper, advises using only data plane operations during a failover, because control planes typically have lower availability design goals.
- How long will traffic take to move? Cached DNS answers can delay a switch, so compare the record's time to live (TTL) with the recovery time. Where a health check starts the failover automatically, ask who can switch it to manual while an attack is suspected. AWS advises caution with automatic failover, since a false alarm costs the same time and data as a real one.
- Can people sign in from the second region? If a single directory or sign-on service sits in the failed region, the second region may start and nobody can reach it.
A network line to a site fails
DS-03. EXAMPLE: Warehouse 2's line fails. First move: the site starts its workaround at once if the line will be down longer than the time left within dispatch's recovery time.
Step | What happens | Who |
|---|---|---|
1 | The site tells IT and the crisis lead at once, and works from its continuity plan (RR-22). | Site or unit continuity lead |
2 | Ask the network provider for its repair estimate. If the line will be down longer than the time left, the site starts its workaround at once. | Supplier liaison; Site or unit continuity lead |
3 | Use a second route where one exists, such as [[a second line from another provider, or a mobile data router kept at the site]], with its security settings agreed in advance. | Technical recovery lead |
4 | When the line returns, check that queued transactions, print jobs and scanner uploads have caught up, then end the workaround with the business owner. | Technical recovery lead; Site or unit continuity lead |
EXAMPLE. The network provider's contract restores a failed line within 12 hours, and Warehouse 2 has no second line (OG-03). Dispatch's recovery time objective is 8 hours, so the decision is Repair in place, and start the workaround now: Warehouse 2 dispatches from its printed pick lists from the start, as its site plan says for a systems loss.
A faulty update stops devices or servers from starting
DS-04. EXAMPLE: An update to the endpoint protection agent makes office PCs and servers fail at start-up. First move: stop the update spreading where the product allows; recover the sign-in directory and monitoring first, then the devices of each service in the recovery order (RR-08); keep the disk recovery keys where they can be reached without the affected systems.
Step | What happens | Who |
|---|---|---|
1 | Confirm the pattern: devices or servers fail at start-up after an update to one product, and no other sign points to an attack. Tell the incident manager at once. Nothing is restored or failed over until the incident manager agrees the cause is the update (RR-23). | Technical recovery lead; Incident manager |
2 | Stop the update spreading, where the product allows: pause it in the product's management console, and ask the supplier to withdraw it. Keep devices that still work running, and do not restart them until the fix is known. | Technical recovery lead; Supplier liaison |
3 | Get the supplier's fix and its instructions. Try them on one device of each kind before using them widely. If the fix is manual, such as removing a file from each device in a recovery environment, plan the hands-on time per device, and have the recovery key ready for each encrypted device. | Technical recovery lead; Head of Information Security |
4 | Recover the sign-in directory and monitoring first, then the rest of the foundations. Reach the disk recovery keys of encrypted devices from where they are kept apart from the affected systems; if the only copy is in the directory or the device management tool, recover that first. | Technical recovery lead |
5 | Recover the servers, then the devices, of each critical service in the recovery order. Count devices recovered against devices affected, by service and site. | Technical recovery lead; Resilience coordinator |
6 | Start the workaround of every service that cannot be back within its recovery time objective (RR-06). | Business owner |
7 | Give staff a way to have their devices fixed by hand: [[a desk at each site, and a courier or a call-back for staff at home]]. | Technical recovery lead; Communications lead |
RR-08 Recovery restores the foundations first, then critical services in order of recovery time, shortest first, unless the crisis lead sets another order and records why. The order and the target times are set in advance and confirmed when recovery starts (P09 V1).
EXAMPLE. Endpoint protection and monitoring are run by the security monitoring provider (P06 SUP-006) and are part of the foundations, estimated at 12 hours. After them, each service's servers and devices come back in the recovery order: warehouse dispatch, online ordering, customer contact, payments and finance and payroll.
Preventing a repeat
Where the product allows, take its updates in rings, as P17 PM-06 does for patches, with rollback criteria agreed first (P17 PM-08). P17 PM-05 to PM-08 set the pilot, the rings, the windows and the rollback criteria. PM-08: “Every ring and server group must have rollback criteria agreed before deployment. A rollback stops the next ring until the cause is known, and the patch is re-planned inside its deadline or deferred under PM-14.”
- Ask each supplier whose software runs on every device how its updates can be staged, and whether a pilot group can take them first.
Ring controls for program updates may not cover every kind of update. Some security agents also receive content updates, such as detection rules, separately from their program releases, and a ring set-up for one kind may not govern the other. Ask each supplier, for each kind of update:
- Can content updates be delayed or staged, with a pilot group first, or do they reach every device at once?
- Is there a kill switch or a rollback that stops a bad update spreading and withdraws it, and who can use it?
- Is there a documented manual recovery path for a device that will not start, and how many hands-on minutes does it take per device?
A dated example. On 19 July 2024 at 04:09 UTC, CrowdStrike released a content configuration update (Rapid Response Content, Channel File 291) for its Windows sensor, and reverted the defect at 05:27 UTC. Microsoft estimated that 8.5 million Windows devices were affected. CrowdStrike's root cause analysis (6 August 2024) says its platform was changed to give customers more control over where and when this kind of content is deployed. Recovery for many devices was manual, device by device. CrowdStrike's guidance says BitLocker-encrypted hosts may require a recovery key. Other products may behave differently, so ask the questions above of each one.
- Where the product allows no staging, record it in the dependency register as a single point of failure across devices (RR-09), with the workaround that covers it.
- Keep the disk recovery keys exported where they can be reached without the sign-in directory or the device management tool, and check [[each quarter]] that the export is complete.
- Test that the keys can be reached at scale. Ask how many devices' keys a key holder can look up in an hour with the directory and the device tool down, and whether anyone can do it outside office hours. A recovery that needs a key per device is limited by that rate.
A site is lost
DS-05. EXAMPLE: Warehouse 2 cannot be entered. First move: the site's continuity plan (RR-22); recover its systems elsewhere only if they run there.
Step | What happens | Who |
|---|---|---|
1 | Take the site plan's first steps at once: people safe and accounted for, the call tree started, the crisis lead called (RR-22). | Site or unit continuity lead |
2 | Invoke the site's continuity plan (RR-17). It says where the site's work moves and how fast. | Crisis lead |
3 | List the systems that run at the site itself: servers, network equipment, printers, scanners. Recover elsewhere only those the moved work needs, by their runbooks. | Technical recovery lead |
4 | Give the receiving sites the network access, devices, printers and accounts the moved work needs, and check them with the site's lead. | Technical recovery lead; Site or unit continuity lead |
EXAMPLE. Warehouse 2's plan answers the loss of its building: “Warehouses 1 and 3 take Warehouse 2's dispatch at [[reduced volume]], perishable orders first; staff who can travel move to them”.
Running a failover or a restore elsewhere
From the decision to users back:
Step | What happens | Who |
|---|---|---|
1 | Confirm the decision is in the crisis log with its time. Where an attack was suspected, confirm the incident manager has said the copy or site to be used can be trusted. | Resilience coordinator |
2 | Open the system's runbook and follow it step by step. Record when each step starts and ends. If a step fails, stop and tell the crisis lead, who decides the next route. | Technical recovery lead |
3 | Record the recovery point. For a restore, use the newest copy that passes its integrity check (RR-11), and record when it was taken. For a failover, record how far behind the second site or copy was when the first failed. | Technical recovery lead |
4 | Bring up the system and its interfaces, and check them: it starts, it answers, its interfaces to other systems and suppliers work, and its backups run from the new place. | Technical recovery lead |
5 | Measure the data lost, from the recovery point to the failure, against the recovery point objective. Name who recovers what was lost, and from where: suppliers' records, customers' confirmations, emails, paper. | Business owner; Resilience coordinator |
6 | Ask the business owner to test the service as its users use it. Let users back only when the business owner confirms that it works. | Business owner |
7 | Let users back. End the workaround at a time agreed with them, and catch up the backlog it left. | Business owner; Technical recovery lead |
8 | Record the time achieved, from the decision to recover to users back, against the recovery time objective and the maximum tolerable downtime. | Resilience coordinator |
After an attack, the copy is chosen by when the attacker first appeared (RR-12), under the Cyber Recovery Operating Procedure. If you are not sure which case you are in, ask the incident manager before you restore.
GDPR Article 32(1)(c) asks for the ability to restore the availability of and access to personal data in a timely manner in the event of a physical or technical incident.
For a HIPAA covered entity or business associate, the disaster recovery plan specification, § 164.308(a)(7)(ii)(B), reads: “Establish (and implement as needed) procedures to restore any loss of data.”
Letting users back
Users go back only when the business owner has tested the service and confirmed that it works (RA-12). A system that starts may still be a service that does not work: orders must reach the warehouse, stock must match the shelves, payments must reconcile. The test uses everyday transactions, run by the people who use the service.
Keeping people told
RR-27 During a disruption of a critical service, the communications lead tells staff within [[1]] hour and affected customers within [[2]] hours of the crisis lead declaring it: what has happened, what still works, what to do, and when the next update will come. Updates follow at least every [[4]] hours, and at each change, until the end of recovery, all from the approved outage templates. In a cyber incident P09's templates and its rule IM-09 apply instead, and notices to authorities stay with P09 and P18.
The approved templates are in the Outage Communications Template Set. Tell users before a failback as well, and when recovery ends.
Failing back
RR-25 Failing back to the normal set-up is planned like a change: the data written while failed over is copied back and reconciled, the business owner agrees the time, it avoids the service's busiest periods, and the crisis lead approves it. Recovery ends (RR-17) when the service runs on its normal set-up, or on an alternate set-up the business owner and the crisis lead have accepted for a set time.
Steps:
Step | What happens | Who |
|---|---|---|
1 | Plan the failback like a change: the steps, the time each takes, the checks, how to stop and stay on the failover set-up if a step fails, and how the data written meanwhile is copied back and reconciled before users move. Where writes landed on both sides, name the source for each data set and who resolves a conflicting record. | Technical recovery lead; Business owner |
2 | Agree the time with the business owner, away from the service's busiest periods. | Business owner |
3 | Ask the crisis lead to approve the failback (RA-23). | Technical recovery lead; Crisis lead |
4 | Tell users when, and for how long, the service will pause, from the outage templates. | Communications lead |
5 | Fail back. Check the system as after the recovery, and have the business owner confirm that the service works. | Technical recovery lead; Business owner |
6 | Record the time taken, the data reconciled and anything found, in the crisis log and in the runbook. | Resilience coordinator; Technical recovery lead |
Where writes landed on both sides, copying one side over the other loses data. This happens in an active-active set-up, when both sites took writes after a split, or when users were let in before the old site was shut off. Settle the rule before the failback: which side is the source for each data set, who resolves a record changed on both sides, and who checks the result. Some replication tools settle conflicts themselves. Amazon DynamoDB global tables, for example, keep the last writer. Ask the supplier what yours does, and stop writes to the old site while the service runs elsewhere. The business owner confirms the reconciled data before users move.
A failback is a second outage, and it can lose data if the copy back goes wrong: plan it, and do not rush it. Where the failover set-up is to run for weeks, check its backups, monitoring and access as for the normal set-up, and record the date it ends.
Annex 1 gives each EXAMPLE system's failback. For the providers the NIS2 Implementing Regulation binds, Annex point 4.1.2 lists, among what the plan includes where appropriate, the order of recovery and restoring and resuming activities from temporary measures (NIS2 IR 1).
Ending recovery
RR-17 The crisis lead invokes a service's continuity plan, sets the recovery order, and declares the end of recovery on the plan's criteria. Every other decision right during an incident is P09's: the incident manager isolates and decides restore or rebuild; the executive incident lead stops customer-facing services; only the board or its equivalent decides on a ransom.
Step | What happens | Who |
|---|---|---|
1 | Check that every affected service runs on its normal set-up, or on an alternate set-up accepted for a set time (RR-25), that its workaround has ended and that its backlog is cleared. | Resilience coordinator |
2 | Declare the end of recovery on the plan's criteria, and record it (RA-17). | Crisis lead |
3 | Tell staff, and the customers who were told of the disruption, that the service is back. | Communications lead |
4 | Close the crisis log and file it, with the runbook records, where nobody who took part can change it. | Resilience coordinator |
The crisis log
The resilience coordinator keeps the crisis log from the first assessment to the end of recovery (RA-16). It holds:
Record | What it holds |
|---|---|
Failure | When each system failed or was reported, how it was found, and the critical services affected |
Attack check | Who checked for signs of an attack, what they found, and the incident manager's answer where asked |
Decisions | Each decision: repair, fail over, restore elsewhere, start or end the workaround, fail back, end recovery. Who took it, when, the estimates it rested on and the reason |
Times | For each service: the failure, the decision to recover, users back; the time achieved against the recovery time objective and the maximum tolerable downtime |
Data lost | For each data set: the recovery point, the data lost against the recovery point objective, and how it was recovered |
Suppliers | Each supplier's estimates and updates, with their times, and what it was asked to do |
Messages | What staff and customers were told, and when |
End | When the end of recovery was declared, by whom and on which criteria; the gaps raised |
Keep the crisis log for [[3]] years, or longer where law or a contract asks, and report results as counts with every gap named, such as "[[2]] of [[3]] systems back within their objective" (RR-20).
After recovery
RR-15 Every failed check in the objectives matrix, and every objective a test or a real recovery missed, is an objective gap. Its owner agrees the change that closes it within [[10]] working days, with a due date; the service's risk owner is told and the risk is reassessed (P05 RM-10).
Step | What happens | Who |
|---|---|---|
1 | Record every objective the recovery missed as an objective gap, with an owner, and tell the service's risk owner (RR-15). | Resilience coordinator; Business owner |
2 | Write the time each method achieved into its runbook. A method used in a real recovery has shown what it achieves; record that, and keep it in the test plan. | Technical recovery lead |
3 | Review the service's recovery plan and the runbooks used, and write the lessons in (RR-16). Update the dependency register where a dependency turned out different (RR-09). | Resilience coordinator; Technical recovery lead |
4 | Ask the business owner to review the objectives where the recovery showed they are wrong (RR-05). | Resilience coordinator; Business owner |
5 | Give the timings and the data lost to the incident manager, where P09 treated the failure as an incident, for its review (IM-11). | Resilience coordinator |
RR-16 Each critical service has a recovery plan, reviewed at least every [[12]] months, after every test and real recovery, and after a significant incident or change. Lessons from tests are written into the plan. A copy of every plan and of the contact lists can be reached when the main systems are down.
When to escalate
At any point, stop and escalate when:
When | What happens | To | Basis |
|---|---|---|---|
The repair estimate keeps moving, or nobody can give one | Decide on the information there is. Treat an unknown repair time as later than the time left, and start the workaround. | Crisis lead | RR-23; RR-06 |
The failover or restore elsewhere fails, or runs longer than its runbook says | Decide again with the time now left. Start the workaround if it is not running. | Crisis lead | RR-23 |
A service, with its workaround, is heading past its maximum tolerable downtime | Decide what else can be done: more people, a supplier's help, a partial service. Tell the approver. | Crisis lead | RR-06; RR-17 |
A sign of an attack appears at any point | Stop failing over and restoring until the incident manager confirms the copy or site can be trusted. P09 runs the incident. | Incident manager | RR-23; P09 IM-01 |
A supplier gives no recovery time, or misses its contracted one | Call on the contract, record the gap, and plan around the supplier. | Crisis lead | RR-10; P06 TP-10 |
The data lost is more than the recovery point objective allows | The business owner names who recovers what was lost. It becomes an objective gap at the end of recovery. | Business owner | RR-15 |
Pitfalls
The mistakes this procedure is built to prevent, each with its fix:
Pitfall | Fix | |
|---|---|---|
PF-02 | Counting failover to a second site as cyber recovery. | Replication copies the damage. Work out the cyber recovery time from a clean rebuild (RR-07, MX-05). |
PF-07 | Keeping the only copy of the plan on the systems it recovers. | Keep a copy and the contacts where they can be reached in an outage (RR-16). |
PF-10 | Failing over to a replica while an attack may still be under way. | Hold until the incident manager confirms the copy can be trusted (RR-23). |
PF-11 | Testing failover on the live service in its busiest weeks, with no way back. | Plan every test with a stop point and a way back, approved by the business owner (RR-26). |
PF-12 | Telling customers nothing until the service is back. | Send the first message within hours and update on a set rhythm (RR-27). |
Related documents
Document | Relationship |
|---|---|
Business Continuity & ICT Resilience Policy | The rules this procedure applies: RR-23 to RR-25, and the rules on workarounds, communication, gaps and records |
Cyber Recovery Operating Procedure | Recovery after a destructive cyberattack (DS-06) |
Critical Service Recovery Objectives Matrix (RTO, RPO, MTD) | Each service's maximum tolerable downtime and recovery time and point objectives |
Cyber Recovery Plan Template | Each service's recovery plan; the runbooks sit beside it |
Resilience Dependency Register | What each service depends on, and each dependency's recovery time |
Site & Business Unit Continuity Plan Template | Each site's and unit's plan for the loss of its building, people, a supplier or its systems (DS-03, DS-05) |
Continuity & Disaster Recovery Test Plan and Report | Plans and reports of failover tests, recovery tests and workaround trials (RR-26) |
Outage Communications Template Set | The outage messages (RR-27) |
Incident Response Plan, Security Incident Management pack (P09) | Incident response, the reporting route (IM-01) and every decision right during an incident |
Third-Party Security Risk Management pack (P06) | Supplier contracts and their recovery commitments (TP-10) |
Patch Management pack (P17) | Rings and rollback criteria for updates (PM-05 to PM-08) |
Annex 1: Each system's recovery method
EXAMPLE, not part of the procedure. The organisation used throughout the pack is a wholesale distributor with about 900 staff and three warehouses, an important entity under NIS2 in its Member State, and not a financial entity, so DORA's reporting duties do not apply; it is not a HIPAA covered entity or business associate. Values are as at 30 Sep 2026, after the resilience review that followed INC-2026-014 (P09).
One row for each system, and each supplier with a recovery time to give, that a critical service needs (RR-24). The hours are the time achieved in the test named, else the contract's, else not known. The foundations (DP-01) come first and are not listed.
System or supplier | Method and where | Hours | Against each service's RTO (hours) | Fail back; note |
|---|---|---|---|---|
DP-02 Online ordering platform | Restore elsewhere The hosting provider's spare environment | 96 hours Tested in RV-2026-02, 20 May 2026 | CS-01: beyond 24 | Moved back to the primary environment in a planned window outside the peak season, once the orders taken meanwhile are reconciled. Too slow for the 24-hour objective until the rebuild of P05 TA-01 is done (OG-01). |
DP-03 Cloud hosting provider | Supplier's own recovery The hosting provider's other data centre | 24 hours, from the contract Untested | CS-01: within 24 CS-04: within 48 | The provider's, with notice of the window. Contract only: never tested. |
DP-04 Backup service provider | Supplier's own recovery The backup service provider's platform | Not known Untested | CS-01: not known CS-04: not known | The provider's. No recovery time in the contract (OG-02). |
DP-07 Warehouse management system | Restore elsewhere [[standby servers]] | 5 hours Tested in RV-2026-03, 27 Aug 2026 | CS-02: within 8 | Moved back to the usual servers in a planned evening window, after a count of the affected aisles. Within dispatch's 8 hours. |
DP-08 Warehouse management system provider | Supplier's own recovery The provider's support service | Not known Untested | CS-02: not known | Not needed: the system stays where it is. No recovery time in the contract (OG-06). |
DP-09 ERP software provider (hosted) | Fail over The ERP provider's second data centre | 3 hours Tested in RV-2026-01, 20 Jan 2026 | CS-02: within 8 CS-04: within 48 | The provider's, with notice of the window. Contract: 8 hours, at most 15 minutes of data lost. |
DP-11 Network connectivity provider | Repair in place The network connectivity provider repairs the line | 12 hours, from the contract Untested | CS-01: within 24 CS-02: beyond 8 | Not needed: the line is repaired. No second line (OG-03): dispatch uses its workaround. |
DP-14 Payment service provider | Supplier's own recovery The payment service provider | Not known Untested | CS-03: not known | The provider's. Its recovery test is planned (RV-2027-01). |
DP-16 Payroll bureau | Supplier's own recovery The payroll bureau | Not known Untested | CS-04: not known | The bureau's. No recovery time in the contract (OG-08). |
DP-17 Email and office software provider | Supplier's own recovery The email and office software provider | Not known Untested | CS-05: not known | The provider's. The contract sets availability, not a recovery time; the restore is tested in RV-2026-04. |
DP-18 Outsourced customer contact centre | Supplier's own recovery The outsourced customer contact centre | Not known Untested | CS-05: not known | The provider's. No recovery time in the contract (OG-09). |
What the annex shows:
- 3 of the 11 methods have been tested: DP-02 in RV-2026-02, DP-07 in RV-2026-03 and DP-09 in RV-2026-01.
- 2 rest on a contract's time that has never been tested: DP-03 and DP-11.
- 6 have no known time: DP-04, DP-08, DP-14, DP-16, DP-17 and DP-18. Each supplier's contract gives none, or sets availability instead.
- 2 known times are beyond a service's recovery time objective: DP-02 (OG-01) and DP-11 (OG-03). Both are open objective gaps.
List every system and supplier from your dependency register, not only those with runbooks. A row with no runbook and no time shows where to start.
Annex 2: Runbook template
One runbook for each system's recovery method. Fill in the middle column. The right-hand column is the EXAMPLE organisation's runbook for its warehouse management system, with the values its records give; the rest is left for you as [[placeholders]].
Field | Your runbook | EXAMPLE: Warehouse management system (DP-07) |
|---|---|---|
System | [[system name, DP-nn]] | Warehouse management system (DP-07; P07 SYS-02) |
Services it serves Each with its recovery time and recovery point objectives | [[CS-nn service: RTO, RPO, MTD]] | CS-02 Warehouse dispatch: RTO 8 hours, RPO 4 hours, MTD 24 hours |
Recovery method | [[method]] | Restore elsewhere |
Where it recovers to | [[where]] | [[standby servers]] |
Time it takes From the first step to the system ready for the business owner's test | [[hours]] | 5 hours |
Tested Test ID, date and time achieved, or Untested | [[test ID, date, hours]] or Untested | RV-2026-03, 27 Aug 2026: 5 hours |
Before you start What must be in place: foundations, access, licences, recovery keys, the copy to restore | [[prerequisites]] | The foundations running (DP-01); [[access to the standby servers, their licences, the backup copy]] |
Who may run it With deputies and the accounts they use | [[role; deputy; accounts]] | Head of IT (technical recovery lead); deputy: IT Operations Manager |
Steps Numbered, each with its time and the check that it worked | [[1. … 2. … 3. …]] | [[the steps as run in RV-2026-03, each with its time]] |
Data Where the copies are, how often they are taken, how lost data is found again | [[copies, interval, sources]] | Stock and dispatch records (DP-10): daily backup, against the 4-hour RPO; gap OG-05, change due 15 Dec 2026. Data lost is found again by a count of the affected aisles. |
Checks before users return Technical checks, then the business owner's test (RA-12) | [[checks]] | [[technical checks]]; then the Head of Logistics confirms that dispatch works (RA-12) |
Fail back How the system returns to its normal set-up (RR-25) | [[failback steps, window]] | Moved back to the usual servers in a planned evening window, after a count of the affected aisles. |
Supplier | [[supplier, contact, hours, notice]] | Warehouse management system provider (DP-08): no recovery time in the contract (OG-06). [[contact and notice]] |
If it is not back in time The service's workaround and who starts it (RR-06) | [[workaround; who starts it]] | Dispatch from printed pick lists of the next day's orders, printed each evening, for up to [[24]] hours. Untried: its first trial is WT-2026-01 on 24 Oct 2026 (RR-26). |
Where this runbook is kept Copies reachable when the main systems are down (RR-24) | [[printed copy at …; offline copy at …]] | [[printed copy in the IT office and with the deputy; offline copy at …]] |
Last reviewed With the service's plan (RR-16) | [[YYYY-MM-DD, role]] | [[YYYY-MM-DD, role]] |
Guidance — delete before approval
Write the steps by running them, on a test copy where you can, and time each one. A runbook for a supplier's own recovery is short: who to call, what to ask for, the contracted time and notice, and what you recover yourself afterwards.
Adapting this template
Guidance — delete before approval
Small organisation: keep the failover decision, a runbook for each system a critical service needs, and the business owner's confirmation before users return. A runbook can be one page. Where a system has no second route, record its method as repair in place, its realistic time, and the workaround that covers it.
Managed service provider or outsourced IT: most runbooks will be the provider's. Ask for them, check their times against your objectives, and agree in the contract who runs them and how fast. The decision to fail over your service stays with your crisis lead (RA-22).
Cloud services: the provider recovers its platform; you recover your configuration, accounts and data, and choose whether to fail over between regions (RR-10).
Regulated entity (NIS2, DORA, HIPAA): a provider the NIS2 Implementing Regulation binds (Article 1) reads Annex points 4.1.2 and 4.2.4 against this procedure, with [[legal or compliance]]. A financial entity under DORA checks failover and restore elsewhere against Articles 12(2) and 12(3). A HIPAA covered entity or business associate keeps the runbooks with its contingency plan records and reads § 164.308(a)(7)(ii)(B) with [[its security official]].
Service organisation whose customers receive an auditor's report on availability: keep each runbook's test record and the crisis log of every real recovery; an auditor will ask for them.
Delete this section before approval.
Framework references
These references show where this document supports an external framework. They indicate relevance only and do not reproduce the text of any standard. Editions referenced: ISO/IEC 27001:2022 incl. Amd 1:2024; ISO 22301:2019 incl. Amd 1:2024 (clause numbers and titles only); NIST CSF 2.0; Directive (EU) 2022/2555 (NIS2) and Implementing Regulation (EU) 2024/2690; Regulation (EU) 2022/2554 (DORA) and Delegated Regulation (EU) 2024/1774; Regulation (EU) 2016/679 (GDPR); 45 CFR Part 164 (HIPAA Security Rule, eCFR current to 6 October 2026); AICPA Trust Services Criteria (number and series title only).
Framework | Reference | Supported by |
|---|---|---|
ISO/IEC 27001:2022 | Annex A 5.30 — ICT readiness for business continuity | The whole procedure: recovering the ICT a critical service needs within its objectives |
ISO/IEC 27001:2022 | Annex A 8.14 — Redundancy of information processing facilities | Recovery methods: failover to a second site or region; Annex 1 |
ISO/IEC 27001:2022 | Annex A 8.13 — Information backup | Restore elsewhere from backup; the copy to restore from (RR-11) |
ISO/IEC 27001:2022 | Annex A 8.32 — Change management | Failing back, planned like a change (RR-25) |
NIST CSF 2.0 | RC.RP-02 — “Recovery actions are selected, scoped, prioritized, and performed” | The failover decision and the steps for each failure: recovery actions selected, put in order and performed |
NIS2 — Directive (EU) 2022/2555 | Article 21(2)(c) — “business continuity, such as backup management and disaster recovery, and crisis management” | The whole procedure: disaster recovery after failures that are not attacks |
NIS2 — Implementing Regulation (EU) 2024/2690 | Article 1 — lays down the technical and methodological requirements of the Article 21(2) measures, and further specifies when an incident is significant, for DNS service providers, TLD name registries, cloud computing, data centre and content delivery network providers, managed service and managed security service providers, providers of online marketplaces, online search engines and social networking services platforms, and trust service providers | Who the Implementing Regulation binds; for other entities its Annex is a guide only |
NIS2 — Implementing Regulation (EU) 2024/2690 | Annex, point 4.1.2 — “The relevant entities’ operations shall be restored according to the business continuity and disaster recovery plan.” The plan is based on the risk assessment (point 2.1) and includes, where appropriate: purpose, scope and audience; roles and responsibilities; key contacts and communication channels; conditions for plan activation and deactivation; order of recovery for operations; recovery plans for specific operations, including recovery objectives; required resources, including backups and redundancies; restoring and resuming activities from temporary measures | When this procedure starts and ends; roles; the recovery order (DS-04); failing back from the failover set-up |
NIS2 — Implementing Regulation (EU) 2024/2690 | Annex, point 4.2.4 — based on the risk assessment (point 2.1) and the business continuity plan, “sufficient availability of resources by at least partial redundancy” of network and information systems; assets, including facilities, equipment and supplies; personnel with the necessary responsibility, authority and competence; and appropriate communication channels | Recovery methods: failover to a second site or region; Annex 1: each system's route and the single routes without one |
DORA — Regulation (EU) 2022/2554 | Article 11(2) — the ICT business continuity policy is implemented “through dedicated, appropriate and documented arrangements, plans, procedures and mechanisms” aiming to ensure the continuity of critical or important functions; respond to and resolve all ICT-related incidents in a way that limits damage and prioritises the resumption of activities and recovery actions; activate without delay dedicated plans enabling containment and tailored response and recovery procedures (Article 12); estimate preliminary impacts, damages and losses; and set out communication and crisis management actions (Article 14) and reporting to competent authorities (Article 19) | The first assessment and the failover decision: recovery started without delay, and in order |
DORA — Regulation (EU) 2022/2554 | Article 12(2) — backup systems can be activated in accordance with the backup policies and the restoration and recovery procedures; “The activation of backup systems shall not jeopardise the security of the network and information systems or the availability, authenticity, integrity or confidentiality of data. Testing of the backup procedures and restoration and recovery procedures and methods shall be undertaken periodically.” | If an attack may be the cause: no second site or backup copy used until it can be trusted |
DORA — Regulation (EU) 2022/2554 | Article 12(3) — “When restoring backup data using own systems, financial entities shall use ICT systems that are physically and logically segregated from the source ICT system.” Those systems are securely protected from unauthorised access or ICT corruption and allow timely restoration | Restore elsewhere: restoring onto systems apart from the failed ones |
DORA — Delegated Regulation (EU) 2024/1774 | Article 26(1) — the ICT response and recovery plans take into account the BIA, specify activation and deactivation conditions, describe actions to ensure the availability, integrity, continuity and recovery of at least the ICT systems and services supporting critical or important functions, are “designed to meet the recovery objectives of the operations of the financial entities”, are documented and readily accessible in an emergency with roles clearly specified, “provide for both short-term and long-term recovery options, including partial systems recovery”, and “lay down the objectives of ICT response and recovery plans and the conditions to declare a successful execution of those plans” | When this procedure starts; ending recovery on the plan's criteria; partial and longer-term running through the workaround and an accepted alternate set-up |
DORA — Delegated Regulation (EU) 2024/1774 | Article 26(2) — the plans identify relevant scenarios, including severe business disruptions, and take into account at least: cyber-attacks and switchovers to redundant capacity, backups and redundant facilities; failure of a critical or important function, including insolvency or other failure of an ICT third-party service provider; partial or total failure of premises and data centres; substantial failure of ICT assets or communication infrastructure; non-availability of a critical number of staff; climate and environment-related events, natural disasters, pandemics and physical attacks; insider attacks; political and social instability; and widespread power outages | Scope: the failures DS-01 to DS-06, among them the loss of a data centre, of a network line and of a site, and switching to redundant capacity |
DORA — Delegated Regulation (EU) 2024/1774 | Article 26(3) — “Where the primary recovery measures may not be feasible in the short term because of costs, risks, logistics, or unforeseen circumstances, the ICT response and recovery plans referred to in paragraph 1 shall consider alternative options.” | The failover decision: another route when the repair is not in time, and the workaround |
GDPR — Regulation (EU) 2016/679 | Article 32(1)(c) — the security measures include, as appropriate, “the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident” | Running a failover or a restore elsewhere: restoring availability of and access to data after a technical incident |
HIPAA — 45 CFR Parts 160 and 164 | § 164.308(a)(7)(ii)(B) — disaster recovery plan (required): “Establish (and implement as needed) procedures to restore any loss of data.” | Running a failover or a restore elsewhere: procedures to restore lost data; Annex 2: the runbook |
AICPA Trust Services Criteria | SOC 2 A1.2, Additional Criteria for Availability | Recovery methods and runbooks; Annex 1: each system's recovery set-up and its tested time |
AICPA Trust Services Criteria | SOC 2 CC7.5, System Operations | If an attack may be the cause: the hand-over to incident response and cyber recovery; ending recovery and the work after it |
ISO 22301:2019 incl. Amd 1:2024 | Clause 8.4.5 — Recovery (clause number and title only) | The whole procedure, from the failover decision to failback and the end of recovery |
ISO/IEC 27001 references name the clause or control this document relates to; ISO 22301 is named by clause number and title only, and its text is not reproduced. Neither says that meeting one meets any other framework's requirement.
HIPAA binds covered entities and business associates that create, receive, maintain or transmit electronic protected health information; others may read it as a reference. An addressable specification is not optional: it is put in place where reasonable and appropriate, or the reason is documented and an equivalent measure used where one is (§ 164.306(d)(3)). Using this document is not a statement that you comply with the Security Rule.
Trust Services Criteria are named by number and series title only; their statements are not reproduced. A report on them is an independent auditor's opinion on your controls, and nothing in this document says that a criterion is met.
Definitions
Term | Meaning in this procedure |
|---|---|
Disaster recovery | Recovering the systems a critical service needs after a failure that is not an attack: repair where they are, fail over, restore elsewhere, or the supplier's own recovery (RR-23, RR-24). After an attack, cyber recovery applies instead (RR-07). |
End of recovery | The point at which the crisis lead declares every affected critical service back in normal operation, on the criteria the plan sets, and the records are complete (RR-17). |
Failback | Returning a system from its failover set-up to its normal one, with the data written meanwhile copied back and reconciled (RR-25). |
Failover | Switching a system to a second site, region or copy that runs it while the first is down (RR-23, RR-24). |
Foundations | What every other recovery needs first: the sign-in directory, the core network and firewalls, the backup platform, endpoint protection and monitoring, and the emergency administrator accounts. |
Maximum tolerable downtime (MTD) | The longest a critical service can be unavailable before the harm reaches Severe on the impact scale. Found by the business impact analysis (RR-03). |
Recovery point | The copy a system is restored or failed over from. After a failure that is not an attack, it is the newest copy that passes its integrity check (RR-11). After an attack, the pack's definition applies: “The backup copy a system is restored from: the newest copy taken before the earliest sign of the attacker, checked for integrity and scanned before use (RR-12; P09 D3).” |
Recovery point objective (RPO) | The most data, measured in time, the service can lose. The backup interval of every data set it needs must be no longer (RR-04, MX-03). |
Recovery time objective (RTO) | The time within which the service must be back, counted from the decision to recover it. Always shorter than the MTD (RR-04). |
Runbook | The written steps of one system's recovery method: the steps and the time each takes, who may run them, what must be in place first, how to fail back, and where the runbook is kept (RR-24). |
Time left | The recovery time objective less the time since the decision to recover. The failover decision compares each estimate with it (RR-23). |
Workaround | A way of running the service, or its most important part, without its systems for a set time, such as dispatch from printed pick lists. Owned and accepted by the business owner (RR-06). |