How do we limit the damage a faulty security-agent update can do?

Start

A security agent runs on every device and server

One product that updates itself on every machine is a single route to a very large outage. If a faulty update stops devices from starting, recovery takes the sign-in directory and monitoring first, then each service in order. The better position is to limit how many devices can take a bad update at once.

• Then: Are devices or servers failing to start now after an update?

Decision

Are devices or servers failing to start now after an update?

If yes, the failure is already happening. The pattern is devices or servers failing at start-up after an update to one product, with no other sign pointing to an attack. If you are not sure, answer yes: the first move costs little and the incident manager decides what it is.

• Yes, or not known: Does anything point to an attack, or can nobody yet say?
• No: Ask the supplier how each kind of update can be staged

Decision

Does anything point to an attack, or can nobody yet say?

The pack lists signs of an attack: files encrypted, unknown accounts or remote access tools, protection switched off, backups tampered with, a supplier that will not say why its service failed. Several systems failing at once with no known cause is one such sign, unless the cause is an update. The incident manager decides whether an attack is suspected.

• Yes, or not known: Hold: tell the incident manager before anything is restored
• No: Tell the incident manager, pause the update, keep working devices on

Outcome

Hold: tell the incident manager before anything is restored

Report it through the incident reporting route at once. While an attack is suspected, nothing fails over and nothing is restored until the incident manager confirms the cause and that the copy or site to be used can be trusted. If an attack is confirmed, this procedure stops for the systems concerned and incident response takes over. P09 owns the incident decision rights.

Step

Tell the incident manager, pause the update, keep working devices on

Nothing is restored until the incident manager agrees the cause is the update. Pause the update in the product's management console where it allows, and ask the supplier to withdraw it. Keep devices that still work running, and do not restart them until the fix is known.

• Then: Try the supplier's fix on one device of each kind first

Step

Try the supplier's fix on one device of each kind first

Get the supplier's fix and its instructions, and try them on one device of each kind before using them widely. The technical recovery lead and the Head of Information Security own this step in the pack's procedure.

• Then: Recover foundations, then each service in order; start workarounds

Outcome

Recover foundations, then each service in order; start workarounds

Recover the sign-in directory and monitoring first, then servers and then devices of each critical service in the recovery order. Reach the disk recovery keys from where they are kept apart from the affected systems. Start the workaround of every service that cannot be back within its recovery time objective, and give staff a way to have devices fixed by hand. Count devices recovered against devices affected.

Step

Ask the supplier how each kind of update can be staged

Ask each supplier whose software runs on every device about program updates and content updates, such as detection rules. Can content updates be delayed or staged? Is there a kill switch or rollback, and a documented manual recovery path? On 19 July 2024 a CrowdStrike content update stopped about 8.5 million Windows devices; other products may behave differently.

• Then: Can the product stage updates, with a pilot group first?

Decision

Can the product stage updates, with a pilot group first?

Staging means a pilot group takes each update before anyone else, content updates included. If the supplier cannot say, answer no: the pack treats a product that allows no staging as a single point of failure, and you can revisit the answer when the supplier replies. A ring set-up for program updates may not govern content updates.

• Yes: Set a pilot and rings; each starts after the one before passes
• No, or not known: Record it as a single point of failure, with a workaround

Step

Set a pilot and rings; each starts after the one before passes

The pack says to take a product's updates in rings, as its patch standard does for patches: a pilot, an early ring, then a broad ring, each starting only when the one before has passed its rollback criteria. The pilot covers every hardware model and every business application in use, and the early ring is spread across sites and departments.

• Then: Are rollback criteria agreed for every ring before deployment?

Decision

Are rollback criteria agreed for every ring before deployment?

Every ring and server group needs rollback criteria agreed before it deploys. A rollback stops the next ring until the cause is known. If you cannot say they are agreed, answer no.

• No, or not known: Agree the criteria with pilot users and system owners
• Yes: Can recovery keys be reached without the directory or device tool?

Step

Agree the criteria with pilot users and system owners

Agree what stops a ring and starts a rollback with the pilot users and system owners before the first deployment, and keep who agreed and when. The pack's workbook sets two thresholds that you set yourself.

• Then: Can recovery keys be reached without the directory or device tool?

Step

Record it as a single point of failure, with a workaround

Where the product allows no staging, record it in the dependency register as a single point of failure across devices, with the workaround that covers it. A workaround that has never been tried is recorded as untried, and you should plan on it lasting less than its plan says.

• Then: Can recovery keys be reached without the directory or device tool?

Decision

Can recovery keys be reached without the directory or device tool?

Encrypted devices need their recovery keys to start again, and a manual fix may need one per device. If the only copy is in the sign-in directory or the device management tool, both may be down with the devices. Ask how many devices' keys someone can look up in an hour without them. If you do not know where the copies are, answer no.

• No, or not known: Export the keys apart and check the export each quarter
• Yes: Is the recovery order set in advance?

Step

Export the keys apart and check the export each quarter

Keep the disk recovery keys exported where they can be reached without the sign-in directory or the device management tool, and test the lookup rate at scale. The pack's default is to check that the export is complete each quarter; your standard sets the interval.

• Then: Is the recovery order set in advance?

Decision

Is the recovery order set in advance?

Recovery restores the foundations first, then critical services in order of recovery time, shortest first, unless the crisis lead sets another order and records why. The order and target times are set in advance and confirmed when recovery starts. If you cannot say, answer no.

• No, or not known: Set the recovery order now
• Yes: Damage limited: staged, rollback-ready, keys apart, order set

Step

Set the recovery order now

Order the foundations first, then critical services by recovery time, shortest first. The crisis lead may set another order and records why. The pack's example counts endpoint protection and monitoring among the foundations.

• Then: Damage limited: staged, rollback-ready, keys apart, order set

Outcome

Damage limited: staged, rollback-ready, keys apart, order set

A bad update now reaches a pilot first, the next ring waits on the rollback criteria, the recovery keys survive the failure, and the recovery order is ready. Your own standard decides how often to revisit it. A product that cannot stage stays on the register with its workaround.