When do we fail over to a second site or region, and when do we fail back?

Start

A system or site has failed and its service is at risk

The decision is made per system. The maximum tolerable downtime runs from the failure, and the recovery time objective runs from the decision to recover. Failover is one of the routes in the system's recovery method, alongside repair in place.

• Then: Is an attack suspected, with the copy or site not yet confirmed?

Decision

Is an attack suspected, with the copy or site not yet confirmed?

Signs of an attack include files renamed or encrypted, unknown accounts or remote access tools, protection switched off, and several systems failing at once with no known cause. Replication copies damage to the second site, so a failover can spread an attack. If nobody can yet say, answer yes and ask the incident manager.

• Yes, or not known: Hold: the incident manager first confirms the copy or site is trusted
• No: Work out the time left and the two estimates

Outcome

Hold: the incident manager first confirms the copy or site is trusted

Nothing fails over or is restored until the incident manager confirms that the copy or site to be used can be trusted. If an attack is confirmed, this procedure stops for the systems concerned, and the crisis log passes to the incident manager. The hold lifts only for the copy or site they confirm. P09 owns the incident decision rights.

Step

Work out the time left and the two estimates

The time left is the recovery time objective less the time since the decision to recover. The technical recovery lead or the supplier estimates the repair in place. The alternative is the time from the system's runbook: achieved in its last test, or the contract's time where it was never tested.

• Then: Will the repair in place finish within the time left?

Decision

Will the repair in place finish within the time left?

If the repair estimate is later than the time left, the crisis lead decides whether to fail over, on the technical recovery lead's advice. If there is no estimate yet, the pack does not say. Treat it as no, the more cautious answer, and decide again when the estimate arrives.

• Yes: Repair in place; decide again if an estimate changes
• No, or not known: Can a failover or restore elsewhere finish within the time left?

Outcome

Repair in place; decide again if an estimate changes

Repair where the system is. Keep each decision with its time and the estimates it rested on, and decide again whenever an estimate changes, with the time now left.

Decision

Can a failover or restore elsewhere finish within the time left?

Use the runbook's time. If the method was never tested, its time is the contract's, or none, because nobody has yet shown it can be met. For a cloud region, the provider's commitment may be the limit, and a failover that needs the failed region, cached DNS or a single sign-in service can take longer than planned. An alternative with no known time counts as no.

• Yes: Run the failover from the runbook, recording each step
• No, or not known: Is the failover sooner than the repair?

Decision

Is the failover sooner than the repair?

Neither route is due back in time. A failover that is quicker than the repair still runs, and the workaround carries the service until it finishes.

• Yes: Start the workaround now, then fail over
• No, or not known: Repair in place and start the workaround now

Outcome

Repair in place and start the workaround now

With no other route, or no quicker one, the repair continues and the business owner's workaround starts at once. A workaround that has never been tried is recorded as untried, and the plan for it should allow for it lasting less than its plan says.

Step

Start the workaround now, then fail over

Both routes are late, so the workaround keeps harm below Severe until the service returns. The business owner starts it. The failover still runs, and it is the quicker route.

• Then: Run the failover from the runbook, recording each step

Step

Run the failover from the runbook, recording each step

Open the system's runbook and follow it step by step, recording when each step starts and ends. If a step fails, stop and tell the crisis lead, who decides the next route. Record the recovery point and how far behind the second site or copy was. Check DNS, sign-in, interfaces and backups from the new place.

• Then: Let users back when the business owner confirms it works

Step

Let users back when the business owner confirms it works

A system that starts may still be a service that does not work. The business owner tests with everyday transactions, run by the people who use the service. Measure the data lost against the recovery point objective and name who recovers it.

• Then: Is the usual set-up repaired and ready to take the service back?

Decision

Is the usual set-up repaired and ready to take the service back?

While the failover set-up runs, it is not the normal arrangement. Where it is to run for weeks, check its backups, monitoring and access as for the normal set-up, and keep the date it ends. If you are not sure, answer no.

• No, or not known: Stay on the failover set-up and keep it protected
• Yes: Plan the failback like a change

Step

Stay on the failover set-up and keep it protected

Check the failover set-up's backups, monitoring and access as you would the normal set-up, and keep the date it is due to end. Recovery also ends where the business owner and the crisis lead accept an alternate set-up for a set time.

• Then: Is the usual set-up repaired and ready to take the service back? (back to an earlier step)

Step

Plan the failback like a change

List the steps, the time each takes, the checks, and how to stop and stay on the failover set-up if a step fails. Plan how the data written meanwhile is copied back and reconciled before users move. Where both sides took writes, name the source for each data set and who resolves a conflicting record. A failback is a second, shorter outage and can lose data if the copy back goes wrong.

• Then: Is the time agreed with the owner, away from busy periods?

Decision

Is the time agreed with the owner, away from busy periods?

The business owner agrees the time, and it avoids the service's busiest periods. If the busiest periods are not known, answer no and find them first.

• No, or not known: replan: Plan the failback like a change (back to an earlier step)
• Yes: Ask the crisis lead to approve, then tell users

Step

Ask the crisis lead to approve, then tell users

The crisis lead approves the failback. The communications lead then tells users when and for how long the service will pause, using the outage templates.

• Then: Failed back and confirmed: recovery ends

Outcome

Failed back and confirmed: recovery ends

Fail back, check the system as after the recovery, and have the business owner confirm the service works. Recovery ends when the service runs on its normal set-up, or on an alternate set-up the business owner and the crisis lead have accepted for a set time. Record the time taken and the data reconciled in your crisis log and runbook.