Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If a backup job fails, then what do we do, and by when?

5 decision pointsThe pack behind this answer: Backup & Restore Assurance

Add to board memo

The short answer

Recommendation
Log the failure the day it is found, rerun the job or fix its cause, and get a good copy back within the hours its class allows. By the pack's defaults that is 24 hours for critical data, 72 for important and 168 for other.
Reasoning
A failed job is a gap in the chain of copies. Until a good copy exists again, the data set may sit outside the recovery point of the service that needs it, so the handling aims to close the gap fast and tell the person whose service carries the risk. A good copy exists again only when the new copy completed, is not partial and is recorded with its time.
Considerations
  • A job the incident manager stopped on instruction during an incident is logged as stopped. It is no failure, and the incident plan decides when it restarts.
  • If the cause points to someone deleting or changing a backup, tell the incident manager before you rerun. A rerun can write over what shows how it happened.
  • Two failures in a row on one data set are raised as a finding. A repeat usually means a cause the rerun does not fix.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartThe daily review shows a backup job failedDecisionDid the incident manager stop this job on instruction?OutcomeLog it as stopped; the incident steps decide when it restartsStepLog the failure with its data set, class and causeStepRerun the job, or fix the cause and rerun itDecisionDoes the cause point to someone deleting or changing a backup?OutcomeTell the incident manager now; the incident steps take overDecisionHas the gap since the last good copy passed the recovery point?StepTell the data owner the same day, in writingDecisionIs a good copy back within the hours for the class?StepEscalate to the backup owner with the log and what was saidDecisionIs this the second failure in a row on this data set?StepRaise a finding with an owner, a severity and a due dateOutcomeClose the log entry and keep it with the review log

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 14 steps.

  1. Start

    The daily review shows a backup job failed

    A finished job proves little, but a job that did not finish is a gap in the chain of copies. Until a good copy exists again, the data set may sit outside the recovery point of the service that needs it. The review exists to find the gap quickly enough to close it first. A missed or partial job counts as a failure.

    In the pack

  2. Decision

    Did the incident manager stop this job on instruction?

    In an incident the incident manager may stop backup jobs so that encrypted files do not overwrite good copies. A job stopped on that instruction is logged as stopped, not as failed. If you do not know whether it was stopped on instruction, treat it as a failure and ask.

    In the pack

  3. Outcome

    Log it as stopped; the incident steps decide when it restarts

    This is not a backup failure and starts no clock here. The incident response plan decides when the job restarts, and who decides is for that plan, not this one. Log the job as stopped so the review does not count it.

    In the pack

  4. Step

    Log the failure with its data set, class and cause

    Open a log entry as soon as the review finds the failure. Record the data set and its class, the cause in a line a colleague could act on, when it was found and the time of the last copy that completed and was not partial. Where a supplier runs the job, ask it for the cause in writing.

    In the pack

  5. Step

    Rerun the job, or fix the cause and rerun it

    If the cause is plain, such as no space on the target or an expired credential, fix it first. If the rerun fails too, a person looks at the cause. Start at once for a critical data set. Where a supplier runs the job, the supplier liaison asks it to rerun.

    In the pack

  6. Decision

    Does the cause point to someone deleting or changing a backup?

    A backup deleted, changed or reached by someone who should not have it is not a job fault. It is a possible incident, so ask before you rerun: a rerun can write over or prune what shows how it happened. Tell the incident manager at once, and if you only suspect it, say so. If the cause is still unknown after the rerun, keep looking for it, but do not wait for it before the next steps.

    In the pack

  7. Outcome

    Tell the incident manager now; the incident steps take over

    The incident manager decides what is stopped and protected, and this routine's jobs pause as the incident steps say. Give the incident manager the log entry. Until the incident is closed, treat any backup an attacker might have reached as unverified.

    In the pack

  8. Decision

    Has the gap since the last good copy passed the recovery point?

    Compare the time since the last good copy with the recovery point of the service that needs the data set. The foundations have no service recovery point, so answer no for them. For any other data set, if you cannot tell, answer yes. Ask again each time the job fails to recover, because the gap keeps growing.

    In the pack

  9. Step

    Tell the data owner the same day, in writing

    Say which data set, when the last good copy was taken and when you expect the next one. The data owner decides whether the service can carry the exposure until a good copy exists. Record the date told in the log, because it shows whether the failure took the data set outside its recovery point.

    In the pack

  10. Decision

    Is a good copy back within the hours for the class?

    A good copy exists again only when the new copy completed, is not partial and is recorded with its time. The pack's defaults are 24 hours for Critical, 72 for Important and 168 for Other, counted from the failure. Your own standard may change them. If the new copy is not confirmed, answer no.

    In the pack

  11. Step

    Escalate to the backup owner with the log and what was said

    The backup owner decides what else to try, and whether the supplier's contract is in question. Give the log entry and what the data owner has been told. Keep the log entry open and check again until a good copy is back.

    In the pack

    See also

  12. Decision

    Is this the second failure in a row on this data set?

    Count the failures in a row on each data set. By the pack's default, two in a row are raised as a finding. A repeat usually means a cause the rerun does not fix: a target that is too small, a credential that keeps expiring, or a window too short for the data.

    In the pack

  13. Step

    Raise a finding with an owner, a severity and a due date

    The resilience coordinator raises it with the cause and the fix and adds it to the findings tracker. The owner agrees a plan within 10 working days by the pack's default, and the finding closes only when new evidence shows the fix works. It is not closed on the owner's word.

    In the pack

  14. Outcome

    Close the log entry and keep it with the review log

    Record when a good copy existed again and who closed the entry. Keep it with the daily review log, where the protection reviewer and an auditor can read it. The next review starts the same way.

    In the pack

Practical checklist

  1. Check whether the incident manager stopped the job on instruction, and if so log it as stopped.
  2. Log the failure with its data set, class, cause, when it was found and the last good copy.
  3. Tell the incident manager at once if the cause points to someone deleting or changing a backup.
  4. Rerun the job, or fix the cause and rerun it. Ask a supplier that runs it for the cause in writing.
  5. Tell the data owner the same day if the gap since the last good copy has passed the recovery point.
  6. Escalate to the backup owner if no good copy exists by the time for the class.
  7. Raise a finding on a second failure in a row, then close the log entry and keep it with the review log.

Evidence to have ready

  • The failure log entry: data set and class, cause, when found, last good copy, good copy again and who closed it.
  • The date the data owner was told, and whether the failure took the data set outside its recovery point.
  • The escalation to the backup owner, with what the data owner had been told.
  • The finding for a repeated failure, with its owner, severity and due date.

Assumptions

  • Every backup job reports its result to a named person, and someone reads the results each working day. Without that review a failure is found late.
  • Each data set has a class set by its data owner, and a service recovery point where a service needs it. The hours are the pack's defaults and your backup standard may set others.
  • Where a supplier runs the backup, its job results reach you and it will rerun a job and give the cause in writing.

Not covered here