Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If a server can't be rebooted to patch, then what are the options?

6 decision pointsThe pack behind this answer: Patch Management

Add to board memo

The short answer

Recommendation
Work through four options in order: patch one node at a time behind the service's redundancy, agree a short planned outage inside the deadline, use the vendor's live patching where it is supported, and only then mitigate and request a deferral before the deadline passes. Put the lasting fix on a plan with a date.
Reasoning
A patch that needs a restart is not installed until the system restarts, so a server that never restarts never gets patched. "It can't be rebooted" usually means nobody has agreed a time. Most services that cannot stop already run on two nodes, and a short outage at the quietest hour usually costs less than an exception. Live patching reduces the exposure but does not by itself meet the deadline. A deferral moves the date. It does not lower the priority, and the finding stays open until a rescan verifies the fix.
Considerations
  • The deadlines are the defaults in the CISO Times packs: 14 days for Priority 2, 30 for Priority 3 and 90 for Priority 4, from first detection. Your own standards may set different ones.
  • A deferral is approved in advance or not at all. It needs an accepted reason, a tested compensating control and a date. A missed window, no test system and workload are never reasons.
  • Where the band's approver also runs the patching, the decision goes up a level, so the person deferring the work does not approve it.
  • A Priority 1 flaw needs its mitigation within 72 hours through the emergency route before any deferral.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartThe owner says the server can't be restarted to patchStepFind the patch's deadline for this serverDecisionDoes the service run on two or more nodes?StepPatch one node at a time: move the load, patch, check, swapDecisionCan the owner agree a short outage inside the deadline?StepBook it as the server's window, at the release reviewDecisionDoes the vendor support live patching for this system?StepApply what live patching can; the rest waits for a windowDecisionWill every fix be in and verified by the deadline?OutcomePatched inside the deadline, confirmed by a rescanStepBlock, restrict and watch it, then test those controlsDecisionHas the deadline already passed?OutcomeToo late to defer: report it overdue and install it soonDecisionIs there an accepted reason to defer?OutcomeNo deferral: find a window and install it by the deadlineStepRequest a deferral to the next window, with each option triedOutcomeDeferred to a date, with the lasting fix on a plan

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 17 steps.

  1. Start

    The owner says the server can't be restarted to patch

    A patch that needs a restart is not installed until the system has restarted, so a server that never restarts never gets patched. The pack's guidance is blunt: this usually means nobody has agreed a time. Ask the owner what a planned outage of one hour at the quietest time of the week would cost, and compare it with the risk a deferral leaves open.

    In the pack

  2. Step

    Find the patch's deadline for this server

    The deadline is the vulnerability standard's, from first detection on this server: in the pack's defaults 14 days for Priority 2, 30 for Priority 3 and 90 for Priority 4. Every option below is judged against that date. A Priority 1 flaw also needs a mitigation within 72 hours, through the emergency route.

    In the pack

    See also

  3. Decision

    Does the service run on two or more nodes?

    Most services that cannot stop already run on two nodes: clustered servers, load-balanced web servers, a pair of domain controllers. If they do, the service can stay up while each node restarts in turn.

    In the pack

  4. Step

    Patch one node at a time: move the load, patch, check, swap

    Move the load off one node, patch and restart it, check the service, then swap. Keep the two members of a redundant pair in different windows, so one is always running and a faulty patch reaches only one.

    In the pack

  5. Decision

    Can the owner agree a short outage inside the deadline?

    A short planned outage at the quietest hour, agreed with the system owner, usually costs less than an exception. Put the cost of an hour's outage next to the risk a deferral leaves open; the pack finds most owners choose the outage once the comparison is in front of them.

    In the pack

  6. Step

    Book it as the server's window, at the release review

    Book the outage as soon as the patch is rated, not when it is already late. If it becomes a standing monthly window, record the owner's agreement in the ring and window plan so next month needs no negotiation.

    In the pack

  7. Decision

    Does the vendor support live patching for this system?

    Live patching applies some fixes to a running system without a restart, where the vendor supports it for that product. Ask the vendor or check its documentation; if you do not know, answer no and carry on.

    In the pack

  8. Step

    Apply what live patching can; the rest waits for a window

    Apply the fixes live patching can deliver now. The fixes it cannot apply wait for the next restart, and the server is not verified until they are in. Live patching reduces the exposure; it does not by itself meet the deadline.

    In the pack

  9. Decision

    Will every fix be in and verified by the deadline?

    Verified means installed, any restart done, the service checked by its owner, and a rescan that no longer finds the flaw. If the remaining fixes need a restart that cannot happen before the deadline, the answer is no.

    In the pack

  10. Outcome

    Patched inside the deadline, confirmed by a rescan

    The deadline is met on the day the patch is installed, the server has restarted, its owner has checked the service and a scan no longer finds the flaw. Start those checks the working day after the window, not at month end.

    In the pack

    See also

  11. Step

    Block, restrict and watch it, then test those controls

    Block the vulnerable service, restrict who can reach the server, and watch it more closely. A control counts only once tested: in the pack's warehouse example, a host firewall rule letting only the application server and the backup service reach the database, tested on the day of the request. An untested control is listed but does not lower the likelihood in the deferral's score.

    In the pack

  12. Decision

    Has the deadline already passed?

    A deferral is approved in advance or not at all. Once the deadline has passed, it is too late to request one.

    In the pack

  13. Outcome

    Too late to defer: report it overdue and install it soon

    It is reported as overdue and escalated as your vulnerability standard sets: in the pack's defaults, to the asset owner's manager, and to the standard's approver if still open 14 days later. Keep the controls in place and install it as soon as possible.

    In the pack

  14. Decision

    Is there an accepted reason to defer?

    The accepted reasons that most often fit here are a restart that would stop a critical service inside the deadline with no other option working in time, and a supplier that has not certified the patch when installing it uncertified would end its support. The others are a failed or withdrawn patch, no patch at all, or a system retiring before the next window. A missed window, no test system and having always done it this way are never reasons.

    In the pack

  15. Outcome

    No deferral: find a window and install it by the deadline

    Without an accepted reason, a deferral is refused. Go back to the owner for an outage inside the deadline. If it still misses, it is reported overdue and escalated.

    In the pack

  16. Step

    Request a deferral to the next window, with each option tried

    The system owner fills in the deferral form before the deadline: the servers by inventory name, the reason with evidence, a line on each option in order, the tested controls and a date. "Server can't be rebooted" with no options is returned. For Priority 1, the 72-hour mitigation must be in place first. In the pack's example the warehouse system had no second node, only a Sunday window and no live patching, and its supplier had not certified the update before that window, so it was deferred to the next Sunday.

    In the pack

    See also

  17. Outcome

    Deferred to a date, with the lasting fix on a plan

    A deferral moves the date; it does not lower the priority, and the finding stays open until verified. Put the lasting answer on a plan with an owner and a date: a second node, a standing monthly window, or a supplier who certifies patches in time. Otherwise the same request comes back next month.

    In the pack

    See also

Practical checklist

  1. Find the patch's deadline for this server from first detection.
  2. Patch one node at a time where the service runs on two or more, keeping each pair's members in different windows.
  3. Otherwise ask the system owner to agree a short outage inside the deadline, and book it at the release review.
  4. Apply what the vendor's live patching can deliver, where it supports this system.
  5. Block the vulnerable service, restrict who can reach the server and watch it more closely. Test those controls.
  6. Request a deferral before the deadline, with a line on each option tried, the tested controls and a date.
  7. Plan the lasting fix with an owner and a date: a second node, a standing monthly window or a supplier who certifies in time.

Evidence to have ready

  • The patch's deadline and the date of first detection on this server.
  • The deferral form, made before the deadline, with the accepted reason, its evidence, each option tried and the approver's decision.
  • The test result for each compensating control.
  • The verification: patch installed, restart done, service checked by its owner and a rescan that no longer finds the flaw.

Assumptions

  • The patch fixes a security flaw, so it has a deadline from your vulnerability standard counted from first detection. The figures shown are the CISO Times packs' defaults; your own standards may set different ones.
  • The server has a system owner who can agree an outage and request a deferral, and a patch owner who deploys.
  • The vendor still supports the server. If it does not, there is no patch to install: that is “If a system is past vendor support and can't be replaced yet, then what?”.

Not covered here