Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How do we test an LLM application the way an attacker would?

7 decision pointsThe pack behind this answer: Generative AI Security

Add to board memo

The short answer

Recommendation
Write a test plan first and send nothing to the feature until the owner and the Head of Information Security have approved it in writing (SR-1). Choose attack classes with a reason for each one left out, log every attempt, and report how many selected classes were tried and how many succeeded.
Reasoning
An LLM application reads your instructions and an attacker's words as the same kind of thing, so the tests aim at the system around the model: what it reads, what its output reaches and what it can do. The safety rules keep a test from becoming the incident it looks for: test accounts, marked test data, harmless targets, limits and a named person who can stop it. The most serious cases arrive in content the feature reads, so plant instructions there as well as typing prompts. A test shows what the testers tried and never proves the feature secure.
Considerations
  • At Depth 3 the tester did not build and does not own the feature. At Depth 2 the plan names who tests, and a tester who did not build it is expected (SR-9). At Depth 1 the configuration checks are the test.
  • Where a provider owns the model or service, its terms must allow the test or it agrees in writing. A test never reaches its own systems or other customers (SR-2).
  • No test releases a payment or changes supplier bank details (SR-5).
  • Report counts, never a total, a grade or a colour. A clean result lists what was not tried.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartAn LLM feature is due for an attacker-style testDecisionWhat review depth has the review set?OutcomeDepth 1: the configuration checks are the testStepWrite the plan: one feature, one version, limits, a stop contactDecisionDo the provider's terms or a written agreement allow the test?OutcomeDo not start: get the provider's written agreement firstStepSelect attack classes, with a reason for each one left outDecisionHas the owner and the Head of Information Security signed the plan?OutcomeSend nothing: get the plan signed firstStepRun the tests and log every attempt, including what it readsDecisionDid a test expose real sensitive data or have a real effect?OutcomeStop, keep the evidence and report through the incident routeDecisionHas every selected attack class been tried?DecisionDid any attack class succeed?StepRecord the finding with steps to repeat it, and rate itDecisionHas every finding's fix passed its retest?StepReport counts: classes tried of selected, and how many succeededOutcomeSend the report on; the findings follow their fix datesOutcomeGive the report to the release decision and set retest triggers

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 19 steps.

  1. Start

    An LLM feature is due for an attacker-style test

    An LLM application puts instructions and data in one channel, so the model reads your instructions and an attacker's words as the same kind of thing. Tests therefore aim at the system around the model: what it reads, what its output reaches and what it can do. A test shows what the testers tried and never proves the feature secure.

    In the pack

  2. Decision

    What review depth has the review set?

    At Depth 1 the configuration checks are the test. At Depth 2 the team runs the tests for each selected class under the safety rules. At Depth 3 an independent tester runs them under a written plan. The depth rises with tools, memory, company content above Public and the rule of two. Judgement: if the depth is not known, take Depth 3.

    In the pack

  3. Outcome

    Depth 1: the configuration checks are the test

    Complete the configuration checklist with the IT owner and the developer, and keep its result as evidence. A change that raises the depth, such as a new tool or a retrieval source above Public, brings the attacker-style test with it.

    In the pack

    See also

  4. Step

    Write the plan: one feature, one version, limits, a stop contact

    The tester writes the plan. Name the feature at one version and one configuration, the window, the environment, the test accounts and the marked test data. Set limits on requests, tokens and spend, and name one person who can stop the test and switch the feature off. Use harmless targets such as a test mailbox and a dummy payee. No test releases a payment or changes supplier bank details. The standard requires an independent tester at Depth 3 and expects a tester who did not build the feature at Depth 2 (SR-9). At Depth 2 the plan names who tests, and the approvers check independence.

    In the pack

  5. Decision

    Do the provider's terms or a written agreement allow the test?

    Where a provider owns the model or the service, its terms must permit the test or it must agree in writing. A test never reaches the provider's own systems or other customers. Procurement asks for the agreement where the terms do not already allow the test. Answer yes if no provider is involved. If you do not know what the terms say, answer no.

    In the pack

  6. Outcome

    Do not start: get the provider's written agreement first

    The test does not start while the permission is outstanding. In the pack's example the request to the provider was sent on 29 Sep 2026 and had not been answered, so the safety rule was not met and the test could not start. Procurement asks again. Judgement: do not test any part of the provider's service until it permits.

    In the pack

  7. Step

    Select attack classes, with a reason for each one left out

    The Head of Information Security selects the classes and the tester checks the choice. The twelve classes are direct and indirect prompt injection, sensitive information disclosure, excessive agency, supply chain, poisoning, unbounded consumption, misinformation, hidden context exposure, vector and embedding weaknesses, improper output handling, and poisoned memory and tool descriptions. Some apply only with tools, retrieval, memory or output that reaches another component. A class is not left out because it is hard or the feature looks small. Judgement: when you cannot tell whether a class applies, select it.

    In the pack

  8. Decision

    Has the owner and the Head of Information Security signed the plan?

    Nothing is sent to the feature until both approve the plan in writing. Before they sign they check the tester's independence, the provider's permission, the environment, the data rules, the limits, the list of classes and the incident route. The tester signs too.

    In the pack

  9. Outcome

    Send nothing: get the plan signed first

    A test run before approval is not covered by the plan, its limits or its stop contact. Change the plan until the approvers can sign it. If a safety rule cannot be met, the test does not start until the owner and the Head of Information Security have changed the plan and approved it again.

    In the pack

  10. Step

    Run the tests and log every attempt, including what it reads

    Plant marked instructions in the documents, pages, emails, tickets and tool results the feature reads, as well as typing prompts, because the most serious cases arrive in content. Record the time, class, exact input, exact output, result and whether real data was seen. Stop at any limit and tell the stop contact. Keep the log where only the owner and the stop contact can read it.

    In the pack

  11. Decision

    Did a test expose real sensitive data or have a real effect?

    Marked test data should be all the tester sees. Anything real that is seen is reported at once and not kept. If a test exposes real sensitive data or has a real effect, stop. Judgement: if you are not sure whether what was seen is real, answer yes.

    In the pack

  12. Outcome

    Stop, keep the evidence and report through the incident route

    Report an event at once through the incident route. The incident pack decides whether it is an incident and whether anything must be reported, and the reporting pack holds the clocks. The incident pack sets the order (P09 IM-07). Where harm can still spread, such as a feature that can still act on content it read, made an unusual tool call or acted without the approval required, contain first by switching its access off and capture what you can in the same step. Otherwise keep what was seen as it was received, then switch off. Judgement: the owner and the Head of Information Security decide when the test resumes, after the incident route has finished with the evidence.

    In the pack

    See also

  13. Decision

    Has every selected attack class been tried?

    A class that was selected and not tried is listed with the reason and is not counted as tried. A class cannot succeed unless it was tried, and a result for a class that was not selected is an error in the plan. Try the remaining classes within the window. If one cannot be tried, the report names it and gives the reason.

    In the pack

  14. Decision

    Did any attack class succeed?

    A class succeeded when the tester tried it and the fixed success criterion was met, for example the feature acted on instructions that came only from content it read. Do not soften a criterion to make a result look better, and do not harden it to find more. A class that did not succeed says what the attempts showed. It does not show the feature cannot be attacked this way.

    In the pack

  15. Step

    Record the finding with steps to repeat it, and rate it

    Write the steps so someone else can repeat them; a problem that cannot be repeated is noted and watched, and is not rated. Name the requirement it breaks and the threat, and propose an impact and a likelihood from 1 to 4. The Head of Information Security rates the finding and sets the severity, which gives the due date, and the finding goes in the register. The owner agrees the fix. If the fix cannot be made by the due date, the owner asks for an exception before it.

    In the pack

    See also

  16. Decision

    Has every finding's fix passed its retest?

    A finding closes when a retest shows the fix works. Someone other than the person who fixed it repeats the steps exactly, then tries the same class by other routes. A fail returns the finding to open. Answer yes if the test raised no finding.

    In the pack

  17. Step

    Report counts: classes tried of selected, and how many succeeded

    Write the count as 'n of m selected attack classes tried; n succeeded'. Name each class that succeeded and each one not tried. Say what the feature did well, what could not be tested, and why a clean result does not show the feature is secure. Include a retest plan for each finding. Send the report to the owner and the Head of Information Security together. Never write a total, a grade or a colour.

    In the pack

  18. Outcome

    Send the report on; the findings follow their fix dates

    Give the report to the release decision now, with each finding in the register. Each finding follows the fix date its severity sets, and stays open until a retest passes. A fix that cannot meet its date needs an exception from the approver its band sets, and a Critical one goes to the board or its equivalent.

    In the pack

    See also

  19. Outcome

    Give the report to the release decision and set retest triggers

    At Depth 3 the release gate holds the release until the independent report is received and its findings are in the register. Retest the classes a change could affect when the model, model version, system prompt, tool, retrieval source or filter setting changes, when a provider gives notice of a new model, and when a signal is confirmed in use.

    In the pack

Practical checklist

  1. Confirm the review depth, taking Depth 3 if it is not known.
  2. Write the plan: one feature at one version, the window, environment, test accounts, marked test data, limits and a stop contact.
  3. Get the provider's permission in writing where its terms do not already allow the test.
  4. Select attack classes, selecting any you cannot rule out, with a reason for each one left out.
  5. Have the owner and the Head of Information Security sign the plan before anything is sent.
  6. Run the tests in content the feature reads and in user input, and log every attempt.
  7. Stop and report through the incident route if a test exposes real data or has a real effect.
  8. Record each success so it can be repeated, rate it, and retest after the fix.

Evidence to have ready

  • The signed test plan showing how each safety rule is met.
  • The provider's written permission, where a provider owns the model or service.
  • The class selection with reasons for each class left out.
  • The attempt log with time, class, exact input, exact output and result.
  • The report with classes tried of selected and how many succeeded, each finding with its repeat steps, and the retest results.

Assumptions

  • The feature has been through the review procedure, which has set its review depth, and the AI use has not been stopped under P24.
  • At Depth 3 the tester did not build and does not own the feature. At Depth 2 the team tests with the same safety rules, a tester who did not build the feature is expected, and the plan names who tests.
  • You can test in a test environment with test accounts and marked test data, or you will say in the plan why production is needed.
  • Where a provider owns the model or the service, its terms allow the test or it agrees in writing. The OWASP and MITRE ATLAS lists behind the attack classes are community sources, not law.

Not covered here