Generative AI Security Pack — Guide

Generative AI Security

An LLM application puts instructions and data in one channel. The model reads the organisation’s instructions and an attacker’s words as the same kind of thing, so the checks that work for ordinary input, such as matching a field against a pattern, cannot tell them apart. The national cyber centres and OWASP both say the defence lies in the system around the model, because the model will sometimes be fooled. Everything the organisation already does for access, logging, secure development, patching and supplier assurance still applies and is not repeated here. This pack adds what is specific: what the model may read, what its output may reach, what it may do, who approves what it does, and how the organisation tests all of this before release and watches it afterwards. The result is counts of features reviewed, checks verified and findings open. It is never one score.

Is this for you?

This pack is for you if:

• a team wants to put an LLM feature live, such as a customer chatbot, a summariser, an assistant over company documents or an agent that uses tools, and you must say whether it is secure enough;
• a supplier has switched on a generative AI feature in software you run, and nobody has looked at its settings;
• you have approved AI uses under the governance pack (P24) and need to know what to check before each goes live;
• a customer, auditor or board member has asked what you do about prompt injection.

What is different about an LLM application

Ordinary software keeps instructions and data apart. A web form takes a name and treats it as data whatever it contains. An LLM application cannot do that. OWASP puts it this way: “LLMs make no architectural distinction between ‘instructions’ and ‘data’ (both are tokens on the same stream)”. The NCSC agrees: “As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be.”

Six things follow.

• Cleaning input does not make it safe. A filter lowers the chance of an attack working. It is not a security boundary. OWASP says “Defense is therefore architectural rather than interceptive”. Build on the assumption that the model will be fooled (GS-02).
• Anything the feature reads is input. The serious cases arrive in a document, email, web page, ticket or tool result that nobody typed into the feature (LT-02). Test those too.
• The output is untrusted. Whatever receives it, such as a page, a query or a shell, must check it as it would check a stranger’s input (GS-06).
• A model that can act can do damage. Give a fooled model a mailbox and the damage is whatever the mailbox can do (LT-04). Limit the tools, check the person’s permission in code, and put a person in front of the actions that matter (GS-22 to GS-27).
• Hidden instructions are readable. Treat the system prompt and tool descriptions as visible to users, and keep secrets out of them (GS-10, GS-11).
• The model can change under you. A provider can swap the model version. A change of model, prompt, tool or filter is a change you review again (GS-18).

The NCSC adds a test: “If the system’s security cannot tolerate the remaining risk, it may not be a good use case for LLMs.” Some uses should not go live.

Everything you already do for access, logging, secure development, patching and supplier assurance still applies, and this pack does not repeat it. Sign-in and service accounts belong to P13 and P14, vulnerabilities and patches to P01 and P17, suppliers to P06 and data levels to P15. This pack adds what is specific to an LLM.

The twelve threats

The LLM Application Threat Catalogue lists twelve threats. Every class on the OWASP 2026 list has at least one. Direct and indirect prompt injection are separate threats, because they are tested differently. LT-12 covers poisoned memory and tool descriptions, where the 2026 LLM list points to the agentic list. The catalogue reads its controls from the standard, and each threat has at least 9 requirements that answer it.

A threat that is not on a list is not therefore safe, and a threat on a list is not therefore present in every application.

The sources and their editions

The pack reads these editions, as published and read on 8 October 2026.

Article 15 of the AI Act asks the provider of a high-risk system for an appropriate level of cybersecurity, from 2 December 2027 for Annex III systems. A deployer is not bound by it. This pack cites Article 15 as the Act’s own benchmark and cites no system as high-risk. GDPR Article 25 binds controllers and Article 32 binds controllers and processors. Both apply to any LLM application that processes personal data. The guidance sources are voluntary. They are not law, and none says what the AI Act or the GDPR requires. We did not read the ENISA publications on AI security, ETSI TS 104 223 or any national text.

How this pack builds on P24

The rules on which AI may be used, with which data, who approves a use and how a use is recorded are P24’s (AG-01 to AG-22). This pack does not change them. It sets what a generative AI feature has to do to be secure once a use is approved, and what the organisation checks before the feature goes live and while it runs.

The pack starts from the P24 answers and tier, then asks what the feature can do. The review depth is the highest of the P24 tier and what the feature’s properties require. If P24 says Stop, no review starts. A feature the organisation builds on a supplier’s model API counts as building under P24, for staff as for customers, so a new feature is Tier 3 and held until Legal records the role, and it is reviewed at Depth 3 until then. If Legal records the deployer role, the tier is set again from the answers and the depth follows the new tier and what the feature can do. If Legal records the provider role or another role, the hold clears but the feature stays Tier 3 and at Depth 3, because it is outside P24’s deployer governance and the executive decision-maker decides. A feature already running carries the hold as a condition with a date, and its depth follows its tier and what it can do. A supplier’s finished product used as supplied is not building. The approvers, the decision time and the review interval follow the higher of the P24 tier and the depth, so a Tier 2 use reviewed at Depth 3 is signed as Tier 3. Where the review date the depth sets and the review date in the P24 entry differ, the earlier applies.

Four things raise the depth. Retrieving company content above Public (GS-09), having tools (GS-22) and keeping memory (GS-14) each make it at least 2. The fourth is the rule of two (GS-24). Count three properties: A, the feature reads untrusted content; B, it can reach sensitive data; C, it can change state or communicate outside. With two, the review records the residual risk. With all three, a person approves every action and the depth is 3. A payment, a change of supplier bank details and a decision about a person stay with a named person in any case. A data level that is not stated counts as sensitive. Depth 1 follows the P24 tier unless a property of the feature moves it up. The depth routes the review. It classifies nothing under the EU AI Act (D-51).

An agent chooses its own tools, so it gets most care. The agent permission checklist gives the approval level for each kind of action (a read counts as undone unless it cannot be), and the AI Agent Permission & Tool-Access Review Checklist applies it item by item.

A feature never releases a payment or changes supplier bank details, by any tool or path (GS-27, P24 AG-16). Spending has no approval step of its own: the cap on spend (AP-13) controls it, and the payment rule controls payments. A feature that can act (send, change records, run commands or spend) has a switch-off that one named person can use in one step (GS-26).

The order to work in

The document numbers are not the order of use. Do steps 1 and 2 once. Repeat steps 3 to 8 for each feature, live features first.

The Head of Information Security rates each finding and the owner agrees the fix. The AI governance lead convenes the decision and the approvers decide: Release, Release with conditions or Hold. Seven gates apply (RG-1 to RG-7). A stopped use never releases. The evidence for the depth is on file. No check that blocks release has failed or is unverified. No Critical or High finding is open without the exception its band needs, and nobody approves an exception for a finding they own: where a band’s approver owns the finding, the next band up takes that approver’s place. Every open Medium or Low finding has an owner and a due date, or an approved exception. At Depth 3 the independent report has arrived. The approvers of the higher of the P24 tier and the depth sign, and the Head of Information Security signs as well from Depth 2; the approval record shows it. A feature already running gets its review within 90 calendar days of adoption, a house default you set yourself. A later change that tightens a requirement gives a feature already live 90 calendar days to meet it, also a house default. A change brings a new review within 10 working days (GS-18).

Start with the Generative AI Security Standard, the LLM Application Security Review Procedure and the Generative AI Deployment Configuration Checklist. The checklist is the quickest way to find what is wrong now.

How it runs in practice (EXAMPLE)

Take the distributor of the earlier packs: about 900 staff, three warehouses and an online ordering service. It is an important entity under NIS2. Its Executive Committee approved the standard on 8 September 2026, because the Head of Information Security owns it and nobody approves a document they own. Three of its seven AI uses under P24 are generative: AI-001, AI-003 and AI-004.

• AI-001, the office-suite assistant. P24 gave it Tier 1. It reads suppliers’ documents (untrusted) and the data is above Public (sensitive), so two properties hold and the depth is 2. 14 of 17 applicable checks verified; DC-06, DC-09, DC-18 are not yet verified. A test on 18 September 2026 found AF-001: a summary repeated hidden text in a price list that said the supplier was approved. It is rated Medium, and AX-001 defers it to 29 January 2027. The decision is Release with conditions.
• AI-004, the design tool’s image generator. Public data, no tools, depth 1. 13 of 15 applicable checks verified; DC-05, DC-06 wait for the provider. The decision is Release with conditions.
• AI-003, the customer chatbot. It reads customers’ messages, reaches customer data and can send, so all three properties hold and the depth is 3. P24 holds it. 0 of 21 applicable checks verified, so the gate says Hold. An independent test is planned for 9 November 2026 to 20 November 2026, with 11 of 12 attack classes selected.

The assistant’s provider returned the questionnaire on 25 September 2026: 24 of 31 applicable questions answered with evidence; 5 answered without evidence, 2 not answered. On 30 September the leadership team saw five counts.

Read this as the distributor’s own record. It does not say that any of its systems is, or is not, in a category of the Act, or that any feature is secure.

Your first month

What to report

Report the five measures (GM-01 to GM-05) every month as counts with names. Never combine them into a percentage, a score or a traffic-light total. When a signal (AS-01 to AS-12) suggests an attack, the route is fixed. Report an event at once through the incident route (P09 IM-01). Nobody is blamed for a report made in good faith. P09 decides whether it is an incident and whether anything must be reported, and P18’s clocks apply. P09 IM-07 sets the order of containing and preserving. Where harm can still spread, such as an agent that can still act (AS-03, AS-07, AS-08) or data still leaving, contain first by switching the feature’s access off, and capture what you can in the same step. Otherwise preserve the evidence first, then switch the feature off.

Everything in the pack

Limits

• Not legal advice. This document helps you secure the generative AI your organisation builds, configures or buys. It is not legal advice, and following it is not a statement that you comply with the EU AI Act, the GDPR or any standard. A test that finds no problem does not show that a feature is secure. Whether the Act binds you, in which role, and which of its duties apply to each system depends on the facts: confirm it with counsel.
• No classification. Nothing here says that a system is or is not high-risk, prohibited or subject to a transparency duty, or that you are a provider, a deployer or any other operator.
• No mapping. This pack does not say which OWASP, MITRE ATLAS, NIST or ISO/IEC 42001 items meet a requirement of the EU AI Act or the GDPR. Each reference says what its source asks for and which CISO Times document helps with that work. Using a document does not by itself show that a requirement is met.
• Other packs own the rest. AI governance is P24’s. The incident management pack (P09) owns the response and the regulatory incident notification pack (P18) the reporting duties and clocks. This pack states no clock. P02 owns exceptions, P05 the risk scales, P06 supplier clauses, P15 the data levels and P20 the rule that nobody tests a control they own.
• A test is not proof. A clean result shows what the testers tried, and the report says what was not tried.
• The files are yours. Every file in the pack stays on your own systems. Nothing you enter in them goes back to the CISO Times.
• The example is not a real firm. The distributor is fictional, marked EXAMPLE, and every value in it is made up.

Where it maps

• EU AI Act, Regulation (EU) 2024/1689, as amended by Regulation (EU) 2026/1744: Articles 3(1), 3(4), 15(1), 26(1) and 113. GDPR: Articles 25(1) and 32(1). NIS2: Article 21(2)(e).
• OWASP LLM and Agentic Top 10 (2026); MITRE ATLAS 2026.09; NIST AI 600-1, section 2.9; NIST AI RMF 1.0; NIST CSF 2.0: ID.RA-03.
• ISO/IEC 27001:2022: Annex A 8.26, Application security requirements. ISO/IEC 42001:2023: clause 6.1, Actions to address risks and opportunities; Annex A control A.9.2, Processes for responsible use of AI systems. By number and title only.