How do we defend an LLM application against prompt injection?

Start

An application sends text we do not control to a language model

Ordinary software keeps instructions and data apart. An LLM application cannot, because both are tokens on the same stream, as OWASP puts it. The UK NCSC warns that prompt injection may never be totally mitigated in the way SQL injection can be. OWASP says the defence is architectural rather than interceptive. So lower the chance of success, then limit what a success can do.

• Then: Draw what the model reads and where its output goes

Step

Draw what the model reads and where its output goes

Prompt injection is direct when a user types it and indirect when the model reads it in a document, web page, email, tool result or retrieved passage. List every source the model reads and every place its output goes, and mark each source trusted or untrusted. Content from outside the organisation, and from anyone who can submit text, is untrusted whichever system holds it.

• Then: Does it read content from anyone who can submit text?

Decision

Does it read content from anyone who can submit text?

Customers' messages, supplier documents, inbound email, web pages, tickets, public files and the output of a tool that fetches them all count. So does a wiki that anyone in the company can edit. Most serious cases arrive this way, in content that nobody typed into the feature. If you are not sure, answer yes.

• Yes, or not known: Keep what it reads apart from instructions and strip hidden text
• No, only our own staff type into it: Limit what users can send and expect override attempts

Step

Keep what it reads apart from instructions and strip hidden text

Send untrusted content in a separate, labelled part of the request and never build instructions by joining text from untrusted sources. Strip hidden characters (tag characters, zero-width characters, variation selectors) on the way in and when output is shown. Filter every kind of input, not only text, and limit the size, number and type of inputs. These steps lower the chance of success. They are not a barrier.

• Then: Treat the model's output as untrusted input to what receives it

Step

Limit what users can send and expect override attempts

A feature that only staff use still faces direct injection: an instruction to ignore the rules, a role-play, or text in another language or an encoding. Limit the size, number and type of inputs and reject the rest. Do not rely on the system prompt to stop it. Treat it as readable by users, so it is not a security boundary.

• Then: Treat the model's output as untrusted input to what receives it

Step

Treat the model's output as untrusted input to what receives it

Check output against a schema in application code, and encode it for where it goes: page, query, shell, link or terminal. Never pass it to an interpreter, query or template without that. A screen that shows output must not fetch addresses the output names (images, link previews, frames) unless they are on an allow list, because that is a way for data to leave. Show the full address of any link.

• Then: Keep secrets out of prompts and assume the prompt can be read

Step

Keep secrets out of prompts and assume the prompt can be read

No key, password or token goes into a prompt, a system prompt, a tool description or a knowledge store. Treat the system prompt and tool descriptions as readable by users, so that revealing all of it would give an attacker no access they lack. A control that is only a sentence in the prompt is not a control.

• Then: How many of the three risky properties does it have?

Decision

How many of the three risky properties does it have?

The properties are: it reads untrusted content, it can reach data above Public or personal data, and it can change state or communicate outside the organisation. A successful injection needs something to act on, and the count shows how much there is. It is a count, never a score. If the owner cannot say whether a property holds, treat it as true.

• All three, or not known: A person approves every action, seeing the exact action
• Two of three: Write down the residual risk for the approvers to read
• One or none: Cut what a success can do: least tools, least access, hard caps

Step

A person approves every action, seeing the exact action

With all three properties, a person approves every action the feature takes, seeing the exact action and not a summary of it. The review is at Depth 3 with an independent test. Some actions are never the agent's: a payment, a change of supplier bank details and a decision about a person.

• Then: Cut what a success can do: least tools, least access, hard caps

Step

Write down the residual risk for the approvers to read

With two properties the review is at least Depth 2 and records in plain words what risk remains if an injection succeeds and who accepts it. The approvers read it. Judgement: the cheapest property to remove is often the third, such as a read-only tool in place of a write tool, or a person reading the output before it is used.

• Then: Cut what a success can do: least tools, least access, hard caps

Step

Cut what a success can do: least tools, least access, hard caps

Give the model only the tools the task needs, each doing one job with strict input. Each tool acts as the signed-in person or a named service identity with the least access its job needs, and code checks the person's permission on every call. Set limits on requests, tokens, steps and tool calls, with a hard spending cap, and a switch-off that one named person can use in one step.

• Then: Log everything and watch for the injection signals

Step

Log everything and watch for the injection signals

Log the request, the sources retrieved, every tool call with arguments and result, the output and the verdict of each filter, in a store the feature's administrators cannot change. Watch for output that follows content the feature has just read, addresses or encoded text in output that are not in the source, and many variations of one request. An event goes to the incident route, which decides whether it is an incident.

• Then: Test with planted instructions in content and in user input

Step

Test with planted instructions in content and in user input

Send a maintained set of override and role-play prompts, including encoded text and other languages. In a test copy, plant instructions in the documents, web pages, emails, tickets, images and tool results the feature reads, in plain, hidden and encoded forms, then check what it says and does. The attack has worked when the feature leaves its task, ignores a rule, returns another person's data, or acts on instructions that came only from content it read.

• Then: Did the feature follow an instruction it should not have?

Decision

Did the feature follow an instruction it should not have?

Any success, including a small one, is a finding. A test that finds nothing shows what the testers tried, not what is possible, so read a clean result together with the list of what was not tried. At Depth 3 the tester who did not build the feature runs these tests.

• Yes: Add a limit outside the model, rate the finding, then retest
• No, in the tests run: Could the owner and approvers accept the worst a success can do?

Step

Add a limit outside the model, rate the finding, then retest

A control that is only a sentence in a prompt is not a control, so rewriting the prompt does not close the gap. Add a limit in code, permissions or approvals that stops a success doing harm. Rate the finding on impact and likelihood, give it an owner and a due date, and put it in your findings register. Then retest the same attack class.

• Then: Test with planted instructions in content and in user input (back to an earlier step)

Decision

Could the owner and approvers accept the worst a success can do?

No defence stops every injection, so ask what the worst successful one could do with the tools, data and audiences the feature has. The NCSC says that if the system's security cannot tolerate the remaining risk, it may not be a good use case for an LLM. Judgement: the owner states the worst case in plain words, and the approvers of the review depth accept it or do not.

• Yes: Take it to the release decision, then retest on every change
• No, or not known: Do not release as designed: remove a property or an action

Outcome

Do not release as designed: remove a property or an action

Take the worst case back to the design. Remove one of the three properties, such as the ability to send outside, the sensitive data or the untrusted source, or put a person in front of the action, then count again. If none can go, the use may not suit an LLM, and the approvers decide whether it proceeds.

Outcome

Take it to the release decision, then retest on every change

The release decision is run under the review procedure, which names the approvers and the gates. Keep the test results and the settings copy with the review file. Re-run the injection tests after any change of model, model version, system prompt, tool, retrieval source or filter setting. The pack's default is a new review within 10 working days of the change or the provider's notice. Prompt injection stays on the monitoring list for as long as the feature runs.