How do we assess the security of a GenAI feature we are building ourselves?

Start

A team is building a feature on a large language model

An LLM application puts instructions and data in one channel, so the model reads your instructions and an attacker's words as the same kind of thing. The national cyber centres and OWASP both say the defence lies in the system around the model, because the model will sometimes be fooled. The assessment therefore asks what the model may read, where its output may go, what it may do and who approves it. What you already do for access, logging and secure development still applies.

• Then: What did the P24 intake decide for this use?

Decision

What did the P24 intake decide for this use?

The intake comes first, because which AI may be used, with which data and who approves it are P24's rules. Building an application or feature on a supplier's model API counts as building, whether it is for staff or for customers, so the answer to the building question is Yes for the feature this flow assesses. A new feature is then set at Tier 3 and held until Legal records which role the organisation holds; a feature already running carries the hold as a condition with a date. That is a signal, not a finding that you are a provider, and Legal decides. With no decision yet, take the second branch.

• Stop: Stop: no review starts
• A hold is open, or no decision yet: Finish the intake and clear the hold first
• A tier is set and no hold applies: Describe the feature and draw its data flow

Outcome

Stop: no review starts

A prohibited practice, or special category personal data in the use, stops it at the intake. There is no route to approval, so the review does not start, and a changed use is a new request. Tell Legal for a prohibited practice and the Data Protection Officer for special category data.

Outcome

Finish the intake and clear the hold first

A hold stops the start. The provider signal, where the organisation would build on a supplier's model API, brand, supply or change the purpose of the system, holds a new use until Legal records which role the organisation holds (deployer, provider or other). If Legal records the deployer role, the tier is set again from the answers, so the use does not stay at Tier 3 only because the signal was raised. If Legal records the provider or another role, the hold clears but the use stays Tier 3. A running use carries it as a dated condition. The review follows the intake decision. The role is for counsel.

Step

Describe the feature and draw its data flow

On one page, show every source the model reads (user input, files, web pages, email, tool results, retrieved text, memory) and every place its output goes (screens, files, databases, links, tools). Mark each source trusted or untrusted: anything from outside the organisation, or from anyone who can submit text, is untrusted. Record the model name and version, the system prompt version and the tools. The feature goes on the P24 inventory with a named owner.

• Then: How many of the three risky properties does it have?

Decision

How many of the three risky properties does it have?

Count three properties: it reads untrusted content, it can reach data above Public or personal data, and it can change state or communicate outside the organisation, which includes acting without a person reading its output. The count raises the depth. It is never a score. If the owner cannot say whether a property holds, treat it as true until the data flow shows otherwise.

• All three, or not known: Depth 3: independent test, and a person approves every action
• Two of three: Depth 2 at least: write down the residual risk in plain words
• One or none: Set the depth and select the threats from the catalogue

Step

Depth 3: independent test, and a person approves every action

With all three properties the review is at Depth 3. An independent tester, who did not build the feature, tests every selected attack class under a written plan. A person approves every action the feature takes. Approvers follow the higher of the P24 tier and the depth, so a Tier 2 use reviewed at Depth 3 is signed as Tier 3.

• Then: Set the depth and select the threats from the catalogue

Step

Depth 2 at least: write down the residual risk in plain words

With two properties the review is at least Depth 2. It records in plain words what risk remains and who accepts it, and the approvers read it. A feature with two properties is not half as risky as one with three. It is a feature where the review has to say what remains.

• Then: Set the depth and select the threats from the catalogue

Step

Set the depth and select the threats from the catalogue

The Head of Information Security sets the depth: the highest that the P24 tier and the feature give. Retrieving company content above Public, having tools and keeping memory each make it at least Depth 2. Then mark each of the twelve threats selected or left out, with a written reason for every one left out. 'Low risk' is not a reason.

• Then: Design for a fooled model, then check the standard

Step

Design for a fooled model, then check the standard

Design on the assumption that the model will be fooled: a sentence in a prompt is not a control. Keep untrusted content apart from instructions, check output in code, send only the data the task needs and keep secrets out of prompts. The developer then marks each of the thirty-one requirements met, not met or not applicable, with evidence, and completes the deployment checklist with the IT owner. Seven of its 30 checks apply only to a feature you host or build.

• Then: Does the model call tools, or send anything outside?

Decision

Does the model call tools, or send anything outside?

A feature in which the model chooses which tools to call is an agent, and P24 treats a system that can act as Tier 3 until assessed. A fooled model with a mailbox can do whatever the mailbox can do. If you are not sure what a connector can do, answer yes.

• Yes, or not known: Review the agent's tools, identity and approvals
• No: Does it retrieve company content or keep memory?

Step

Review the agent's tools, identity and approvals

Give the model only the tools the task needs, each doing one job with strict input. Each tool acts as the signed-in person or a named service identity with least access, and code checks the person's permission on every call. A person approves, seeing the exact action, before anything irreversible, anything that leaves the organisation, changes access, spends money, runs code outside a sandbox or affects a person. A feature never releases a payment or changes supplier bank details. Test the switch-off.

• Then: Does it retrieve company content or keep memory?

Decision

Does it retrieve company content or keep memory?

Retrieval adds company documents to the prompt, and memory carries content between sessions. Both are places where an attacker's text, or someone else's data, can wait. Content from outside, and from anyone who can submit text, stays untrusted whichever system holds it. If you are not sure whether an index or cache is in use, answer yes.

• Yes, or not known: Control who can add to a store and who can read from it
• No: Does any model or component come from outside the organisation?

Step

Control who can add to a store and who can read from it

Retrieval returns only what the signed-in person may already read, checked in the index query or source system with that person's identity. Asking the model to hold something back is not a check. Every store has an owner, a source list, a data level and a way to remove a document and everything derived from it. Content enters only from named sources, with a record of who added it. Memory writes are logged and the person can clear them.

• Then: Does any model or component come from outside the organisation?

Decision

Does any model or component come from outside the organisation?

Most features call a provider's model or take a model, adapter or library from a public source. A tampered or silently changed model changes every answer. A public model or adapter is used only after its publisher, source and file hash are checked and it loads in a form that does not run code. Answer yes if any part comes from outside.

• Yes, or not known: Send the questionnaire, pin versions, plan for model changes
• No, all of it is our own: Test at the depth, including the content the feature reads

Step

Send the questionnaire, pin versions, plan for model changes

Procurement sends the AI vendor questionnaire and Security reviews the answers. They inform the supplier decision under the third-party pack and do not replace it. Record the model name and version. Keep a list of models, adapters, libraries, tools and connectors with versions, pinned and approved before use. A change of model is a trigger for review, so know the provider's route for notice of changes (the pack's supplier term ST-7).

• Then: Test at the depth, including the content the feature reads

Step

Test at the depth, including the content the feature reads

At Depth 1 the configuration checks are the test. At Depth 2 the team runs the tests for each selected attack class and writes down what it tried. At Depth 3 an independent tester runs them under a written plan, approved before anything is sent. Test what the feature reads (a document, an email, a tool result) as well as what users type. A clean result shows what the testers tried and says nothing about what they did not.

• Then: Did the checks or tests find a gap?

Decision

Did the checks or tests find a gap?

A finding is a confirmed gap between the feature and the standard, found in a review, a check, a test or an incident. Two gaps take no exception: a feature that can release a payment or change supplier bank details, and a breach of P24's data rule or prohibited-practice rule. If a test or check is unfinished, go back and finish it before you answer. A gap that is open under an approved exception does not stop the hand-over, and the approvers see it.

• Yes, and a fix or an exception is possible: Rate it, give it an owner and a due date, and fix it
• Yes, and it takes no exception: Stop: no exception is possible, so remove the path or the breach
• A test or check is unfinished: Test at the depth, including the content the feature reads (back to an earlier step)
• No, or each open one has an approved exception: Take the evidence to the release decision

Step

Rate it, give it an owner and a due date, and fix it

Rate the finding on impact and likelihood, each 1 to 4, on your risk scales. The band is its severity. The due date counts from the day it is confirmed. The pack's defaults are 30 days for Critical, 90 for High, 180 for Medium and 365 for Low. It has a named owner and goes in your findings register. After the fix, test the attack class again. If the fix cannot be made by the due date, the owner asks for an exception before it: the approver its band sets decides, a Critical one goes to the board or its equivalent, and the owner is never among the approvers. Then answer the previous question again.

• Then: Did the checks or tests find a gap? (back to an earlier step)

Outcome

Stop: no exception is possible, so remove the path or the breach

A feature never releases a payment or changes supplier bank details, by any tool or path, and this takes no exception. Nor does a breach of P24's data rule or prohibited-practice rule. The fix is to change the feature or the use. A feature may draft a payment request that a named person then makes through the normal approval.

Outcome

Take the evidence to the release decision

The assessment ends with a file: the data flow, threat selection, standard check, checklist result, test results, findings and the provider's answers. Hand it to the AI governance lead, who convenes the decision. The approvers decide Release, Release with conditions or Hold against seven gates. Report the file as counts of checks verified and findings open, never as one score.