Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If an LLM application leaks data or is manipulated into a harmful action, then what do we do?

6 decision pointsThe pack behind this answer: Generative AI Security

Add to board memo

The short answer

Recommendation
Contain first where harm can still spread, capturing what you can in the same step; otherwise keep the evidence first, then switch the feature off. Report at once through the incident route, then find the cause, fix what the system around the model allows, and restart only after an independent retest and the release gates pass.
Reasoning
Words in a document, page, message or chat can steer an LLM application as if you had written them, so a leak or harmful action is treated as a security event, whatever the cause. The incident pack keeps evidence before anything is changed unless waiting would let the harm spread (IM-07); the AI pack applies the same order. The incident route, not this flow, decides whether it is an incident and what must be reported. A sentence in the prompt is not a control, so the fix changes what the model can read, reach and do.
Considerations
  • If you cannot tell whether harm can still spread, assume it can. Cut access, do not delete anything, and use the least disruptive stop that works (IM-08).
  • No feature may release a payment or change supplier bank details (GS-27). If one did, use the payment fraud route as well and keep the feature off; there is no exception.
  • A supplier incident is also handled through the supplier route in your contract. Keep your own copy of the logs, settings and keys.
  • The figures are the pack's defaults: a new review within 10 working days, and fixes due within 30, 90, 180 and 365 days by severity.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartAn LLM application leaks data or takes an action it should notDecisionCan the harm still spread: is it still leaking or acting?StepKeep the evidence first: settings as found, then sessions and logsStepNow switch the feature off: stop it and revoke its keys and toolsStepContain now: switch off, and capture what you can in the same stepStepReport it at once through the incident routeStepComplete the evidence set before anything else is changedDecisionDid personal or Confidential data leave the feature?StepTell the Data Protection Officer and Legal nowDecisionDid the provider cause it, or tell us of it?StepAlso follow the supplier incident route and keep your own evidenceDecisionDid it release a payment or change supplier bank details?StepUse the payment fraud route as well, and keep the feature offStepFind how it happened: the input, the content or the settingStepReverse what it did and replace the keys and tokens it heldStepEnter the gap in the findings register as a findingStepFix what the system around the model allowsStepPut the fix through change control and test the attack classes againDecisionDid someone other than the fixer repeat the test, and did it pass?DecisionDo the release gates pass for a restart?OutcomeRestart with monitoring on and a new review date setOutcomeKeep it off: clear what holds the gate, or ask for an exception

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 22 steps.

  1. Start

    An LLM application leaks data or takes an action it should not

    An LLM application reads instructions and data in one channel, so words in a document, web page, message or chat can steer it as if the organisation had written them. Whatever the cause, the order follows P09 IM-07: contain first where harm can still spread, otherwise keep the evidence first, then switch off; report; find the cause; then fix it and prove the fix before the feature restarts. This flow follows the pack's signals AS-01 to AS-12 and the incident route; it does not decide whether the event is an incident.

    In the pack

  2. Decision

    Can the harm still spread: is it still leaking or acting?

    Harm can still spread where an agent can still act on poisoned content, keys still work or data is still leaving. If you cannot tell, assume it can. P09 owns incident response, and it says to preserve evidence first unless the harm is spreading; if it is, isolate first and record what was lost (IM-07). The pack says the same: contain first where harm can still spread, such as an agent that can still act (AS-03, AS-07, AS-08) or data still leaving, and otherwise keep the evidence first, with the EP-08 snapshot before any change. Judgement: if you cannot tell whether a leak has stopped, this flow takes the cautious branch.

    In the pack

  3. Step

    Keep the evidence first: settings as found, then sessions and logs

    Where the harm cannot spread, P09 keeps the evidence before anything changes (IM-07). Take a copy of the feature's settings as found (EP-08) first, then keep what EP-01 to EP-10 lists. Judgement: the pack gives no time limit, so do this the same day. Switching off removes keys and access, so the copy comes first.

    In the pack

  4. Step

    Now switch the feature off: stop it and revoke its keys and tools

    The evidence is now safe, so the system's access can be switched off (P09 IM-07; P24 AT-6). The switch-off stops the feature and revokes its keys and tool access (GS-26, DC-30). Cut access, do not delete anything. Use the least disruptive stop that works (P09 IM-08).

    In the pack

  5. Step

    Contain now: switch off, and capture what you can in the same step

    GS-26 asks for a switch-off that one named person can use in one step, which stops the feature and revokes its keys and tool access. DC-30 checks that it was tested. Export the settings as found (EP-08) and the log extracts as you do it, and record what you could not capture (P09 IM-07). Cut access, do not delete anything. Use the least disruptive stop that works; stopping a customer-facing service is the executive incident lead's decision, or the incident manager's where waiting would let the harm spread (P09 IM-08). If the switch-off does not work, that is a finding in its own right.

    In the pack

  6. Step

    Report it at once through the incident route

    Anyone may report, and nobody is blamed for a report made in good faith (P09 IM-01, P24 AG-13). P09 decides whether it is an incident, sets the severity and runs the reporting screen, and P18's clocks apply from there. This flow restates none of them. Do not wait for the cause to be known before reporting. If you contained first, tell the incident route as you do it; do not wait for the evidence.

    In the pack

    See also

  7. Step

    Complete the evidence set before anything else is changed

    Whichever way you came, check that you hold the settings as found (EP-08) and the rest of EP-01 to EP-10: the prompts and outputs of the affected sessions, the system prompt, tool descriptions and filter settings in force, the model version, every tool call, the retrieved passages, the injecting document as received, memory before and after, sign-in and key logs, and cost records. Nothing is rebuilt or cleared until this is done, unless waiting would let the harm spread (P09 IM-07).

    In the pack

  8. Decision

    Did personal or Confidential data leave the feature?

    Signals AS-05 and AS-06 point here: data of a level or kind the person does not normally see, or data in a tool not approved for it or leaving in an outbound request. Personal data has its own handling (P24 AT-1 to AT-3), and P09's reporting screen decides whether it is a personal data breach. If you cannot tell yet, take the cautious branch.

    In the pack

    See also

  9. Step

    Tell the Data Protection Officer and Legal now

    P24 AT-3 asks for the Data Protection Officer the same day when special category data is involved. Judgement: this flow asks for the DPO and Legal on any personal data, because the reporting screen needs them and early notice costs little. They advise; the decision to notify and the clocks stay with P09 and P18.

    In the pack

    See also

  10. Decision

    Did the provider cause it, or tell us of it?

    A supplier incident that exposes your prompts, outputs, files or accounts is a separate trigger (P24 AT-4), handled through the supplier route in your contract (P06 clause CL-07). If you do not know yet, take the other branch and contain your own side, because the kept evidence will show where it started.

    In the pack

  11. Step

    Also follow the supplier incident route and keep your own evidence

    Report it through the incident route as a supplier incident (P24 AT-4). Keep your own copy of the logs, settings and keys for the feature, because the provider's account of events is not yours to rely on alone. The rest of this flow still applies to any gap on your side, such as a missing alert or a key that stayed live.

    In the pack

    See also

  12. Decision

    Did it release a payment or change supplier bank details?

    No feature may release a payment or change supplier bank details by any tool or path (GS-27, P24 AG-16). If one did, that is a breach with no exception route. If you do not know yet, check the payment and bank-detail records now.

    In the pack

  13. Step

    Use the payment fraud route as well, and keep the feature off

    An AI output that led to a wrong payment is an incident trigger (P24 AT-5), and P09's business email compromise playbook may apply. GS-27 takes no exception, so the feature stays off until the cause is closed and the gates pass (judgement). Tell the people who handle payments so that what can still be stopped is stopped (judgement).

    In the pack

    See also

  14. Step

    Find how it happened: the input, the content or the setting

    Use the kept evidence to place the event in the threat catalogue. Words typed by a user are direct prompt injection; words in a document, page or email the feature read are indirect injection, and EP-06 shows the carrier. Data in output, logs or caches is disclosure, and a prompt that quotes its own instructions is hidden context exposure. Name the threat and the requirement that failed, because the finding needs both.

    In the pack

  15. Step

    Reverse what it did and replace the keys and tokens it held

    A key that appears in output or a log is revoked at once (GS-10). Change the credentials of every identity the feature used, including service accounts (P09 Eradicate), and check what it created or sent. Remove any poisoned document from the knowledge store with everything derived from it, and clear memory the attacker may have written to (GS-12, GS-14). P09 sets the wider eradication steps.

    In the pack

  16. Step

    Enter the gap in the findings register as a finding

    The incident belongs to P09. The register holds the gap the event showed: the threat, the requirement, the feature and its use, with the source marked as an incident. Rate impact and likelihood from 1 to 4, and the severity sets the owner's due date (pack defaults from the day it is confirmed: Critical 30, High 90, Medium 180, Low 365 days).

    In the pack

    See also

  17. Step

    Fix what the system around the model allows

    A sentence in the prompt asking the model to behave is not a control (GS-02). Fix what the model can read, what its output can reach, what its tools can do and which actions need a person (GS-04, GS-06, GS-07, GS-09, GS-22, GS-23, GS-25). The owner agrees the fix and a date, and the developer makes it.

    In the pack

    See also

  18. Step

    Put the fix through change control and test the attack classes again

    A change to the prompt, a tool, a filter, a source or the model is a change, and the attack classes it could affect are tested again (GS-18). Test at least AC-01, AC-02, AC-05, AC-08 and AC-09, and the class of this event. An event that shows a gap forces a new review within 10 working days by default.

    In the pack

    See also

  19. Decision

    Did someone other than the fixer repeat the test, and did it pass?

    The person who fixed it does not close it. A pass closes the finding and a fail returns it to Open. A pass shows this route is closed; it does not show the feature is secure. P09's own recovery test is separate: the cause is understood, the way in is closed and the system owner has tested the service (P09 IM-10).

    In the pack

  20. Decision

    Do the release gates pass for a restart?

    P24 AG-10 lets the AI governance lead suspend a running use, and the approvers decide whether it resumes. Judgement: this flow applies the pack's release gates to the restart. They ask for the required evidence, no blocking check failed, no Critical or High finding open without the exception its band needs, and the signatures of the higher of the P24 tier and the review depth. Nobody approves a feature they own.

    In the pack

  21. Outcome

    Restart with monitoring on and a new review date set

    The approvers record any condition with an owner and a date; a missed condition comes back to them. Turn on the alerts for the signals the feature can raise (DC-18) and set the next review date, which is the earlier of the pack's date and the one in the P24 inventory. P09 decides when the incident is closed and whether a post-incident review is required (IM-11).

    In the pack

    See also

  22. Outcome

    Keep it off: clear what holds the gate, or ask for an exception

    A failed gate holds the release. A High finding passes only with an exception from the Head of Information Security and the accountable executive. A Critical finding passes only with one from the board or its equivalent. Exceptions are short: 30 days for Critical and 90 for High at most, by default. Nothing lifts a breach of GS-27, AG-03 or AG-05.

    In the pack

    See also

Practical checklist

  1. Decide whether harm can still spread, assuming it can if you cannot tell.
  2. Switch the feature off and revoke its keys and tools, keeping the settings as found first or in the same step.
  3. Report the event at once through the incident route, without waiting for the cause.
  4. Complete the evidence set before anything else is changed.
  5. Tell the Data Protection Officer and Legal if personal or Confidential data may have left.
  6. Find how it happened, reverse what it did, and replace every key and token it held.
  7. Enter the gap as a finding, fix it through change control and retest the attack classes it could affect.
  8. Restart only when someone other than the fixer has retested it and the release gates pass.

Evidence to have ready

  • The settings as found, taken before any change or in the same step as the switch-off.
  • The prompts and outputs, tool calls, retrieved passages, the injecting content as received, and the sign-in, key and cost logs.
  • The time of the report to the incident route and who handled the evidence, and when.
  • The finding with its threat, requirement, severity, owner and due date, and the independent retest result.
  • The restart decision with the gates passed, the approvers and any conditions.

Assumptions

  • The feature is on your AI inventory with a named owner, and someone can use its switch-off. If there is no switch-off, the first fix is to remove its keys and access by hand.
  • You have an incident route and an incident manager who can declare an incident. This flow does not decide whether the event is one, or what must be reported.
  • The figures shown, 10 working days for a new review and 30, 90, 180 and 365 days to fix, are the pack's defaults. Your own standard may set different ones.
  • Who approves an exception depends on how you are governed. The top approver is the board, or the committee it has delegated to; the management board in a two-tier structure; executive management where there is no board.

Not covered here