How do we know our backups will actually restore?

Start

You want proof that your backups restore

A backup job that finished says only that it wrote something. It does not show that the data restores, how long that takes or that the keys still exist. Only a restore shows these, so every step below rests on a restore that was checked and timed. A tabletop exercise or a drill is never evidence here.

• Then: Confirm the data set is listed with its owner, class and kind

Step

Confirm the data set is listed with its owner, class and kind

Start from the coverage matrix. Each critical data set needs an owner, a class, a location, a method, an interval and a supplier. A data set that is not listed is treated as not backed up, so list it before testing. Its kind decides the test: a system you run, a cloud or software-as-a-service tenant, the foundations, or data a provider holds.

• Then: Does a provider hold the data, with no copy of ours?

Decision

Does a provider hold the data, with no copy of ours?

Where a provider holds the data and you keep no backup of your own, there is nothing of yours to restore. If you cannot tell whether you hold an independent copy, answer yes, because an assumed copy is a common gap.

• Yes, or not known: Use the provider's own recovery test and get its results
• No: Pick the test type the data set's kind needs

Outcome

Use the provider's own recovery test and get its results

Test it through the provider's own recovery test, with its results shared and checked against the service's objectives. A contract that gives no restore time, or a longer one than the service's recovery time objective, is recorded as a failed check. Whether you should hold a copy of your own is a separate question.

Step

Pick the test type the data set's kind needs

A system you run needs a file or data set restore (TT-1) and a full system restore (TT-2). A cloud or software-as-a-service tenant needs TT-1 and a tenant restore (TT-5). The foundations need TT-1 and a rebuild in an isolated environment (TT-4). A test of a higher type counts for the lower types it includes, if it passed.

• Then: Choose the copy, alternating online and immutable or offline

Step

Choose the copy, alternating online and immutable or offline

For a critical data set, alternate between the online copy and the immutable or offline copy from one test to the next, so both are shown to restore. Record the copy by identifier and the time it was taken, and record any connection of an offline copy. Have a named person who is not the usual backup administrator recover the key from your key register and decrypt the copy.

• Then: Prepare an isolated place to restore into

Step

Prepare an isolated place to restore into

Restore into a place with no route to production, and never onto the live system. For a rebuild test the isolated environment has no route to production at all, and the clean-room gates apply. For a tenant test, create a test tenant. Delete what you restored when the test ends.

• Then: Does the copy pass its integrity check before it is restored?

Decision

Does the copy pass its integrity check before it is restored?

Run the backup system's own verification on the copy first. In an incident, P09 asks for the same check before any restore, so this practises it. P09 decides who checks and who decides what is restored (decision D3: the backup service provider checks and the incident manager decides), and nothing here changes that. If you do not know the result, answer no.

• Yes: Restore, timing from the decision to restore
• No, or not known: Record the failure and raise a finding

Step

Record the failure and raise a finding

A copy that fails the check is a result: write it down, raise a finding for it and try the next copy. A restore whose integrity check failed does not start the clock for the next test. If no copy of a critical data set passes, it has no verified copy, and the pack rates a critical data set that cannot be restored as a High finding.

• Then: Is there another copy of this data set to try?

Step

Restore, timing from the decision to restore

Start the clock at the decision to restore and stop it when the data owner confirms the data is usable. Take both times from system logs, not from memory. Work out the data lost: the hours between the copy used and the point at which the data was last known to be complete. A provider's failover that restores none of your backups is not a restore test.

• Then: Did the restore complete, and does the data match its source?

Decision

Did the restore complete, and does the data match its source?

The second integrity check is on the restored data. Everything in scope must be restored. Compare the data with its source or a sample, run the application's own consistency check and have the data owner open records that matter. The first check finds a damaged copy. This one finds a restore that completed and produced the wrong thing. If you did not run it, answer no.

• Yes: Checked by someone else, and confirmed usable by the data owner?
• No, or not known: Raise a finding: the restore or the data check failed

Outcome

Raise a finding: the restore or the data check failed

Record the test, with the failed criterion, and raise a finding with an owner, a severity and a due date. The owner agrees the plan within 10 working days, by the pack's default. Do not rerun the test until it passes, and do not change the criterion. A finding closes only when a retest shows that the fix works.

Decision

Checked by someone else, and confirmed usable by the data owner?

The person who runs the restore does not judge it alone. A restore checker who did not run it checks the result, and the data owner confirms the data is usable with records that matter to the service. This is the criterion most often skipped. If either is missing, answer no.

• Yes: Record the evidence and issue the report
• No, or not known: Not passed yet: book the checker and the data owner

Outcome

Not passed yet: book the checker and the data owner

A restore the administrator judged alone has not been checked. Where one person runs the backups, ask someone else to check, and have the data owner confirm, with a deputy named for the day the restore is ready. Finish the check before you count the test.

Step

Record the evidence and issue the report

Every test leaves a record of eight things: the data set and copy, the copy's integrity result, start and end times from logs, what was restored and compared, the consistency check and records opened, the data lost, who ran, checked and confirmed it, and every failed criterion with its finding. The report follows within 10 working days and records are kept 3 years, both the pack's defaults.

• Then: Was it a timed test: system, service end to end or rebuild?

Decision

Was it a timed test: system, service end to end or rebuild?

A file or data set restore (TT-1) and a tenant restore (TT-5) are not timed, and a test that is not timed claims nothing about recovery time. A system restore (TT-2), a full service test (TT-3) and a rebuild (TT-4) are timed and are judged against the service's objectives. If you do not know, answer no.

• Yes: Were the time and the data lost within the service's targets?
• No, or not known: Passed for the copy only: plan the timed test its kind needs

Outcome

Passed for the copy only: plan the timed test its kind needs

A passed file or tenant restore shows the copy is whole, the keys work and the data opens. It does not show how long recovery takes. For a system, plan the full system restore (TT-2); for the foundations, a rebuild (TT-4). The pack gives a tenant restore no timing. Where a supplier runs the backup, its contract should give a restore time no longer than the service's recovery time objective.

Decision

Were the time and the data lost within the service's targets?

Compare the time achieved with the service's recovery time objective and the data lost with its recovery point objective. Both belong to the service's business owner under your recovery policy, and the test never sets them. In the pack's example, online ordering took 96 hours against 24, so the time criterion was missed.

• Yes: Has a rebuild test measured the cyber recovery time?
• No, or not known: Raise a finding and an objective gap, then fix and retest

Outcome

Raise a finding and an objective gap, then fix and retest

A test that misses a criterion is a finding. It is the test doing its job. Where a critical service cannot meet its recovery time or recovery point, also raise an objective gap under P22 RR-15 and link the two; the finding keeps P22's date. Tell the risk owner for a High finding, and close it only when a retest passes.

Decision

Has a rebuild test measured the cyber recovery time?

A restore in normal conditions shows the restore time, onto systems you can trust. After a destructive attack the foundations are rebuilt first, so recovery takes longer. Only a rebuild test (TT-4) measures that, and its result replaces the estimate in the recovery matrix. If you do not know, answer no.

• Yes: Report counts with every gap named, and book the next restore
• No, or not known: Plan a rebuild test and show both times side by side

Outcome

Plan a rebuild test and show both times side by side

Plan the rebuild test with your resilience coordinator. Check P22's peak-season freeze before you fix a date, and where it is also a recovery test of a critical service, plan it once. Report the restore time and the cyber recovery time side by side against the objective. In the pack's example, ordering restored in 96 hours and its cyber recovery time is 108 hours, with 12 for the foundations.

Outcome

Report counts with every gap named, and book the next restore

Report to executive management each quarter and to the approver your governance names each year (the board or its committee, the management board, or executive management where there is no board), as counts with every gap named, for example 3 of 5 tests met every criterion, and the other 2 are named. Never report a score, a percentage or one colour for the programme. Book the next restore of this data set from the date of the last one that passed.

Decision

Is there another copy of this data set to try?

A copy that fails its integrity check is a finding in its own right. Try the next copy, alternating the online and the immutable or offline one. When none is left that passes, the data set has no usable backup, whatever the job reports.

• Yes: Choose the copy, alternating online and immutable or offline (back to an earlier step)
• No: No usable copy: tell the data owner and raise a finding

Outcome

No usable copy: tell the data owner and raise a finding

Treat the data set as having no backup until a copy restores. Tell its owner the same day, raise a finding with an owner and a due date, and fix the cause before the next test. If the data set belongs to a critical service, the finding is High by the pack's meaning: it cannot be restored.