Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

Who decides…

Who has the authority to take a production system offline?

5 decision pointsThe pack behind this answer: Security Incident Management

Add to board memo

The short answer

Recommendation
Write down before an incident who may take a system offline. In the pack, the incident manager may isolate a system, site or account at once; stopping a customer-facing service is the executive incident lead's decision, unless waiting would let the harm spread.
Reasoning
Taking a system offline is often the fastest way to stop harm, and it can also stop the business, so the plan names who decides in advance. Waiting for an executive is never a reason to let harm spread, and the incident manager uses the least disruptive containment that stops it. Where taking a system offline would stop a service people depend on for safety or essential operations, the incident manager keeps containing around it and brings in the service owner and specialists before stopping the service.
Considerations
  • The roles and the split of decision rights are the pack's defaults. Your own plan may name other roles, but it should name someone and a deputy for each decision.
  • Cut a machine off from the network and leave it powered on: memory may hold the attacker's tools or a key. Do not power off a machine that is encrypting until the technical responders advise.
  • Outside an incident, taking a production system offline is a change and follows your normal change approval.
  • If the executive incident lead cannot be reached, go to the deputy and then the next person your plan names. The pack's default allows 15 minutes at each step.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartSomeone wants to take a production system offlineDecisionIs it to contain a suspected or declared incident?OutcomeUse your normal change processDecisionAre your decision rights written down and known?StepAgree now who isolates and who stops servicesDecisionCould taking it offline affect safety or an essential service?StepIncident manager contains now, the least disruptive wayOutcomeBring in the service owner and specialists before stopping itStepChoose the least disruptive way to take it offlineDecisionWould it stop a customer-facing service?OutcomeThe incident manager decides and actsDecisionWould waiting for the executive let the harm spread?OutcomeIncident manager acts first and reports it at onceOutcomeThe executive incident lead decides

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 14 steps.

  1. Start

    Someone wants to take a production system offline

    Taking a system offline is often the fastest way to stop harm, and it can also stop the business. That is why the plan says in advance who may do it. Waiting for an executive is never a reason to let harm spread.

    In the pack

  2. Decision

    Is it to contain a suspected or declared incident?

    The decision rights below apply during an incident. If there is a suspicion but no declaration yet, report it at once through the reporting route, so that triage and a named incident manager start the clock. Taking a system offline for planned work is outside this question.

    In the pack

    See also

  3. Outcome

    Use your normal change process

    Without an incident, taking a production system offline is a change like any other and follows your own change approval. The incident pack does not cover it.

  4. Decision

    Are your decision rights written down and known?

    The pack's readiness check asks whether it is written and known who may isolate, stop a customer-facing service, report to an authority, speak outside, engage outside help and decide on a ransom demand. If nobody can point to it, answer no.

    In the pack

  5. Step

    Agree now who isolates and who stops services

    Agree it on the call and record it: the person running the response may isolate systems, sites and accounts; a senior executive decides on stopping customer-facing services. Write it into your plan after the incident, with a deputy for each role.

    In the pack

  6. Decision

    Could taking it offline affect safety or an essential service?

    Disconnecting a system can cause more harm than the attacker where it supports physical safety, care, payments, supply or other operations people depend on. If yes, or you cannot tell, the incident manager still acts at once, but does not take that service down until its owner and specialists are in. That is the pack's rule IM-08.

    In the pack

    See also

  7. Step

    Incident manager contains now, the least disruptive way

    Waiting for an executive is never a reason to let harm spread. The incident manager acts at once with the least disruptive containment that stops it: isolating accounts, closing the access paths the attacker is using, or cutting off an affected segment while the service keeps running. Isolating a system, site or account is within the incident manager's own decision rights.

    In the pack

  8. Outcome

    Bring in the service owner and specialists before stopping it

    Before any action that would stop a service people depend on for safety or essential operations, the incident manager brings in its service owner and the people who run it, with outside incident-response help if the skills are missing. Decide together how far to go, and record who decided and why. Keep containing meanwhile. In the pack's site isolation steps, the incident manager decides how far to go with the service owner and specialists, and the decision log records that they were brought in.

    In the pack

    See also

  9. Step

    Choose the least disruptive way to take it offline

    Use the least disruptive containment that stops the harm. For a machine, cut it off from the network and leave it powered on: memory may hold the attacker's tools or a key, and is lost at power-off. Do not power off a machine that is encrypting until the technical responders advise. Preserve evidence first unless the harm is spreading.

    In the pack

  10. Decision

    Would it stop a customer-facing service?

    Isolating one server, a site's office network or an account usually does not. Taking down the ordering site, the customer portal or a service customers connect to does. If you are not sure, answer yes.

    In the pack

  11. Outcome

    The incident manager decides and acts

    Isolating a system, site or account is the incident manager's decision, without waiting for an executive, using the least disruptive containment that stops the harm. The executive incident lead is told at the next update. In the pack's example the incident manager cut one site's office network off 30 minutes after declaration; the executive agreed because it stopped office work there.

    In the pack

  12. Decision

    Would waiting for the executive let the harm spread?

    Harm is spreading when malware is moving between machines, files are being encrypted, data is leaving, an attacker is active in an account or a payment is about to be made. If you cannot tell, answer yes.

    In the pack

  13. Outcome

    Incident manager acts first and reports it at once

    The incident manager may stop a customer-facing service where waiting would let the harm spread, choosing the least disruptive action that stops it, and tells the executive incident lead at once. The decision log records who took it, when and why, whether or not it turned out to be right.

    In the pack

  14. Outcome

    The executive incident lead decides

    Stopping a customer-facing service is the executive incident lead's decision. If the executive lead cannot be reached, go to the named deputy, then the next person your plan names; the pack's default allows 15 minutes at each step. Meanwhile the incident manager keeps containing within their own rights.

    In the pack

Practical checklist

  1. Confirm it is to contain a suspected or declared incident, and report a suspicion at once through the reporting route.
  2. Check that your decision rights are written and known; if not, agree on the call who isolates and who stops services.
  3. Check whether taking it offline would affect safety or an essential service, and if so bring in its owner and specialists first.
  4. Contain now with the least disruptive action that stops the harm.
  5. Preserve evidence first unless the harm is spreading.
  6. Take stopping a customer-facing service to the executive incident lead, unless waiting would let the harm spread.
  7. Where the incident manager acts first, tell the executive incident lead at once.
  8. Write the decision rights into your plan afterwards, with a deputy for each role.

Evidence to have ready

  • The written decision rights, with a named deputy for each role.
  • Each decision in the decision log, with who took it, when and why, whether or not it turned out to be right.
  • For a service people depend on for safety or essential operations, the record that its owner and specialists were brought in.
  • Each attempt to reach the executive incident lead, with its time.

Assumptions

  • There is a declared incident, a named incident manager and a named executive incident lead, each with a deputy. They are different people.
  • The roles and the split of decision rights are the CISO Times incident pack's defaults. Your own plan may name other roles, but it should name someone for each decision in advance.

Not covered here