Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If a VPN appliance may be compromised, do we patch, rebuild or replace it?

5 decision pointsThe pack behind this answer: Security Incident Management

Add to board memo

The short answer

Recommendation
Keep the appliance's logs, then check it for signs of use. Any sign, or no check yet, makes it an incident: contain it, patch in place only if you know the route in and every change made, otherwise rebuild from a clean image, and replace it if the vendor can no longer supply fixes or a trusted image.
Reasoning
A patch closes the way in. It does not remove what an attacker left on the appliance or in the network behind it. The pack's ransomware playbook puts a system back on its patched version only when the route in and every change are known; applied here by analogy, anything less means a rebuild. The pack has no appliance-specific procedure and does not cover replacement, so that call is your own judgement with the vendor and your IT provider. Reconnect only when the plan's recovery criteria are met.
Considerations
  • With no sign of compromise after a real check, an exploited flaw on an internet-facing appliance is Priority 1 in the pack's defaults: a compensating control within 72 hours and the fix within 7 days.
  • Being unable to check, for example because the logs do not go back far enough, counts as a sign.
  • A VPN may be how sites, on-call engineers or suppliers reach systems others depend on. Bring in those service owners before cutting it off, and keep containing meanwhile.
  • Change every credential on the appliance or used through it: its administrator accounts, the service account it uses to reach your directory, and the accounts of users who signed in during the exposure.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one. A code beside a source, such as RR-17, is the number of a rule in that pack document; the document lists its rules.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 18 steps.
  1. Start

    A VPN appliance may be compromised

    The trigger is usually a vendor advisory or a national CSIRT warning that a flaw in your appliance is being exploited, or an alert on the appliance itself. A remote-access appliance flaw added to the CISA Known Exploited Vulnerabilities catalogue is the pack's own example of an exploited weakness. The first question is not how to patch. It is whether the weakness has already been used against you, and whether it is known to be exploited.

    In the pack

  2. Step

    Keep the logs, then check the appliance for signs of use

    Ask your monitoring provider and IT provider to preserve the appliance's logs and extend their retention before anything is changed. Then look for the indicators the vendor or CSIRT has published, accounts or settings nobody made, sign-ins from odd places, and unusual connections from the appliance into the network. Record which accounts signed in through it, and from where.

    In the pack

  3. Decision

    Is there any sign it has been used against you?

    This asks about your own appliance, not the world. Any credible indicator counts, and so does being unable to check, for example because the logs do not go back far enough. A check on a closed appliance has limits: CISA warned that attackers hid their traces from the vendor's integrity checker. The triage procedure's rule is that when you are not sure, you declare and set severity by the worst it could be.

    In the pack

  4. Decision

    Is the flaw known to be exploited in the wild?

    A flaw in the CISA Known Exploited Vulnerabilities catalogue, or one your vendor or national CSIRT says is being used in attacks, counts as exploited. A clean check does not clear such an appliance, because an attacker may have left persistence that survives a patch or a factory reset. If you cannot tell, answer yes.

    In the pack

  5. Step

    Treat it as presumed compromised, not as a patch job

    CISA's direction on exploited VPN appliances warned that a sophisticated attacker may leave rootkit-level persistence that survives a factory reset, and told agencies to isolate the device. This flow applies that to any exploited remote-access appliance. Do not wait for proof: isolate it, keep an image, and follow the incident route. The pack's own rule is the same in spirit: when you are not sure, declare.

    In the pack

  6. Outcome

    Patch it by priority, then look back

    With no sign of compromise after a real check, and no known exploitation, this is a vulnerability to fix on its priority. A VPN gateway is internet-facing, which raises its priority; in the pack's defaults an exploited flaw there would be Priority 1, so this branch is for one that is not. After patching, look back over the whole exposed period; that look back is our advice, not a pack step. Any sign found then, or news that the flaw is exploited, makes it an incident and this flow starts again at the first question.

    In the pack

  7. Step

    Declare an incident and preserve evidence

    A weakness in an internet-facing system used to get in is the pack's exploited-vulnerability category, and the incident plan's own phases are its response. Export the appliance's configuration and logs, and keep a copy of any image the vendor's tools can take, before anyone patches, resets or rebuilds it. A rebuilt appliance cannot tell you how the attacker got in. CISA's direction also expected the device to be isolated from enterprise resources.

  8. Decision

    Would cutting it off affect safety or essential operations?

    A VPN may be how remote sites, on-call engineers or suppliers reach systems that others depend on. In the plan, the incident manager contains at once with the least disruptive action that stops the harm, and brings in the service owner and specialists before any action that would stop a service people depend on for safety or essential operations. Stopping a customer-facing service is the executive incident lead's call unless waiting would let the harm spread. If such operations depend on the appliance, or you cannot tell, answer yes.

    In the pack

  9. Step

    Contain around it; bring in service owners before cutting it off

    Keep containing meanwhile, the least disruptive way: close the access paths the attacker is using and disable affected accounts. Before cutting off the appliance, the incident manager brings in the owners of the services that depend on it and specialists, with outside responders where you lack the skills, and decides with them how far to go, for example moving users to another route first. Record who decided and why in the decision log.

    In the pack

  10. Step

    Isolate it, preserving evidence first unless harm is spreading

    The plan's rule: preserve evidence first unless the harm is spreading; if it is spreading, isolate first and record what was lost. The incident manager may isolate a system without waiting for an executive, using the least disruptive containment that stops the harm, and tells the executive incident lead at the next update. Apply the vendor's mitigation or patch as part of containment if you wish. It does not end the incident.

    In the pack

  11. Step

    Change every credential, key and certificate it held

    The plan says to assume the attacker had more than you have found. For an appliance that means its own administrator accounts, the service account it uses to reach your directory, the accounts of users who signed in through it, and any certificates and keys stored on it. CISA's direction was to revoke and reissue every connected or exposed certificate, key and password. The pack does not list appliance secrets, so add them to your own list.

    In the pack

  12. Decision

    Can the vendor still supply fixes and a trusted image?

    Rebuilding depends on the vendor. An appliance past its end of support, or one where the vendor cannot give you a clean image and a fixed, supported version, cannot be made trustworthy again. The pack does not cover appliance replacement, so this is your own judgement with the vendor and your IT provider.

  13. Outcome

    Replace it, and do not reconnect the old one

    Keep the old appliance off the network, labelled and stored as evidence until the incident manager agrees it may be wiped. Set up the replacement from a clean configuration, not a copy of the old one, with new credentials. It goes into service only when the plan's recovery criteria are met. Look in the network behind the old appliance as well; the replacement does not remove what the attacker did next.

    In the pack

  14. Step

    Rebuild per the vendor on a fixed, supported version

    In the pack, rebuild means wiping it, installing again from a clean image and then restoring only data. A factory reset on its own is not enough for an exploited appliance, because persistence can survive it. Follow the vendor's rebuild instructions and move to a supported fixed version. Re-create the configuration by hand rather than copying back a file the attacker could have changed. The plan says not to rebuild before the evidence is safe, so take a full image first.

    In the pack

  15. Step

    Look in the network behind it for what the attacker did next

    Check the systems the appliance could reach for new accounts, scheduled tasks and remote access tools, and for unusual sign-ins from the appliance. Start with the sign-in directory and the accounts that passed through the appliance. The plan's rule is to assume the attacker did more than you have found, so this look belongs before any reconnection.

    In the pack

  16. Decision

    Are all five recovery criteria met?

    The plan returns a system to service only when: the route in is understood and closed; any image used was checked for integrity; it is patched, scanned and monitored with credentials changed; its owner has tested it; and the monitoring period has started. A patched appliance with no investigation behind it fails the first criterion.

    In the pack

  17. Outcome

    Keep it off the network until the gap is closed

    Whatever criterion is missing goes back to the phase that owns it: the investigation for the route in, the technical responders for credentials and monitoring, the owner for testing. Keep users on the containment route agreed earlier in the meantime.

    In the pack

  18. Outcome

    Reconnect it under closer monitoring

    Bring it back under closer watching. In the pack's defaults the monitoring period is 7 calendar days for Severity 1 and 2 incidents. The incident closes when the criteria are met, the monitoring period has passed without the attacker returning, and the case log is complete.

    In the pack

Practical checklist

  1. Ask your providers to preserve the appliance's logs and extend their retention before anything changes.
  2. Check for the published indicators, accounts or settings nobody made, odd sign-ins and unusual connections into the network.
  3. If there is any sign, or you cannot check, declare an incident and export the configuration and logs.
  4. Contain the least disruptive way, bringing in dependent service owners before cutting the appliance off.
  5. Change every credential on it or used through it, and check the network behind it for new accounts and tools.
  6. Patch in place only if you know the route in and every change; otherwise rebuild from a clean image with a fixed version.
  7. Replace it if the vendor cannot supply fixes or a trusted image, and keep the old one off the network as evidence.
  8. Reconnect only when all five recovery criteria are met, then watch it through the monitoring period.

Evidence to have ready

  • The preserved logs, the exported configuration and any image taken before changes.
  • The check for signs of use: what was looked at and how far back the logs went.
  • The decision to patch, rebuild or replace, with who decided and why.
  • The credential changes with dates, and each recovery criterion met before reconnection.

Assumptions

  • The appliance gives staff, suppliers or sites remote access to your network, so taking it offline has a business cost.
  • The response steps are the defaults in the CISO Times incident management pack, and the priorities and deadlines are the defaults in its vulnerability pack. Your own plan and standard may set different ones.
  • The pack has no appliance-specific procedure. Where this flow applies the pack's general rules or CISA's published direction on exploited VPN appliances, it says so.

Not covered here