How do we review an LLM feature or an AI agent for security before it goes live?

Start

An LLM feature or agent is due to go live

The review answers one question: can it go live? The answer is Release, Release with conditions or Hold, and the evidence is on file. It applies to every feature the organisation builds, configures or switches on, including one a supplier has added to software you already run. A feature already running when the standard is adopted is reviewed too, within the pack's default of 90 calendar days.

• Then: What did the P24 intake decide for this use?

Decision

What did the P24 intake decide for this use?

The review follows the intake and does not replace it. P24 sets the tier and says whether the use may proceed, and the review depth starts from that tier. A use P24 has stopped is not reviewed. A feature a supplier switched on inside software you already run also needs an intake decision before it starts.

• Stop: Stop: no review starts
• Not decided yet, or not known: Get the intake decision first
• A tier is set: Describe the feature and draw its data flow

Outcome

Stop: no review starts

P24 has said Stop, because of a prohibited practice or special category personal data. There is no route to approval, so no review is started. A changed use is a new request through the intake.

Outcome

Get the intake decision first

A feature with no intake decision has no tier, so the depth cannot be set. Ask the owner for the intake and wait for the decision. The review is not a way round the intake.

Step

Describe the feature and draw its data flow

Copy the feature's use ID, tier and data level from its P24 inventory entry. Draw the data flow on one page and answer five questions: does it retrieve company content, what level of tools has it (none, read or write), does it read untrusted content, keep memory or send anything outside. Say whether the organisation built the feature (building on a supplier's model API counts), put its name on it or changed what it is meant for, and tell Legal if so. P24 holds such a new feature at Tier 3 until Legal records the role. A deployer role sets the depth again from the new tier. A provider or other role keeps it at Tier 3 and Depth 3. A running feature carries it as a dated condition.

• Then: Is it Tier 3, or do all three risky properties hold?

Decision

Is it Tier 3, or do all three risky properties hold?

Tier 3 at the intake sets Depth 3. So do all three properties of the rule of two: it reads untrusted content, reaches sensitive data and can change state or communicate outside the organisation. If the owner cannot say whether a property is true, treat it as true until the data flow shows otherwise.

• Yes, or not known: Depth 3: an independent tester tests every selected class
• No: Tier 2, tools, memory, company data or two properties?

Step

Depth 3: an independent tester tests every selected class

At Depth 3 someone who did not build the feature tests every selected attack class under a written red team plan. A feature with tools also gets the agent permission review. Where all three risky properties hold, a person approves every action the feature takes. The approvers are the executive decision-maker your procedure names and the Head of Information Security. The pack's default decision time is 30 working days.

• Then: Check the standard and the configuration checklist

Decision

Tier 2, tools, memory, company data or two properties?

Any one of these makes it at least Depth 2: a Tier 2 use, retrieval of company content above Public, tools (read or write), memory kept between sessions, or two of the three risky properties. When none applies and the tier is 1, the review is Depth 1: the data flow, the standard check and the configuration checklist.

• Yes, or not known: Depth 2: Depth 2: the team tests and the Head of InfoSec reviews
• No: Depth 1: Check the standard and the configuration checklist

Step

Depth 2: the team tests and the Head of InfoSec reviews

At Depth 2 the team tests the selected attack classes, a tester who did not build the feature is expected and the review states who tested, and the Head of Information Security reviews the evidence and signs as well. A third-party feature adds the provider questionnaire, and a feature with tools adds the agent permission review. Where two properties hold, the review records the residual risk. The pack's defaults are a decision within 15 working days and a review of the approval every 6 months.

• Then: Check the standard and the configuration checklist

Step

Check the standard and the configuration checklist

The developer marks each of the thirty-one requirements met, not met or not applicable, with evidence. The developer and the IT owner then verify each applicable configuration check, with evidence and a date. Of the 30 checks, 14 hold the release if they fail or are not verified. Report both as counts, for example 14 of 17 applicable checks verified, never as a share.

• Then: Does an outside provider supply the model or the service?

Decision

Does an outside provider supply the model or the service?

A feature built on a provider's model, or switched on in a supplier's product, relies on settings, contract terms and data handling you do not control. The organisation's side stays yours: the settings, the data put in and what the feature can reach. Answer yes if any part comes from outside.

• Yes, or not known: Send the provider questionnaire and review the answers
• No: Does the feature have tools or connectors?

Step

Send the provider questionnaire and review the answers

Procurement sends the AI vendor questionnaire and Security reviews the answers. The review procedure requires it from Depth 2 when an outside provider supplies the model or service, and its evidence list asks for the answered questionnaire then. At Depth 1 Security reads the provider's published security statement instead. Count the answers: questions answered with evidence out of those that apply, with the open ones listed. The answers inform the supplier decision under the third-party pack and do not replace it.

• Then: Does the feature have tools or connectors?

Decision

Does the feature have tools or connectors?

A feature in which the model chooses which tools to call is an agent, and P24 treats a system that can act on its own as Tier 3 until assessed. A tool is any function the model can ask the application to run: search, read a file, send an email, run a query, run code. A connector gives access to another system. Any feature with tools is at least Depth 2. If you are not sure, answer yes.

• Yes, or not known: Review the agent's permissions and set the approval for each action
• No: Test the selected attack classes at the depth

Step

Review the agent's permissions and set the approval for each action

List every tool, the identity it uses and every action it can take, with whether it can be undone and whether it leaves the organisation. The pack gives the approval for each kind of action. A payment, a change of supplier bank details and a decision about a person are never the agent's. Sending outside, deleting, changing access and running code outside a sandbox need a person each time. With all three risky properties, a person approves every action.

• Then: Test the selected attack classes at the depth

Step

Test the selected attack classes at the depth

At Depth 1 the configuration checks are the test. At Depth 2 the team runs the tests for each selected class and writes down what it tried. At Depth 3 an independent tester runs them under a written plan, approved before anything is sent. Where a provider owns the model or service, its terms must allow the test or it must agree in writing. A test never releases a payment or changes supplier bank details.

• Then: Rate each finding and put it in the register

Step

Rate each finding and put it in the register

The Head of Information Security rates each confirmed gap on impact and likelihood, each 1 to 4. The band is its severity. It gets an owner and a due date counted from the day it is confirmed. The pack's defaults are 30 days for Critical, 90 for High, 180 for Medium and 365 for Low. An open Medium or Low finding does not hold the release and becomes a condition with an owner and a date.

• Then: Is a High or Critical finding open without an approved exception?

Decision

Is a High or Critical finding open without an approved exception?

Two release gates hold the release while a Critical or High finding is open without an approved exception. Judgement: treat an unrated finding as not known, so go back and rate it first. The finding's owner is never among the approvers of its exception.

• Critical, with no approved exception: Hold, or take a Critical exception to the board or its equivalent
• High with no approved exception, and no Critical: Hold, or take a High exception to two approvers
• Not rated yet, or not known: Rate each finding and put it in the register (back to an earlier step)
• None open, or each has an approved exception: Do the evidence, blocking checks and test report all pass?

Outcome

Hold, or take a Critical exception to the board or its equivalent

A Critical finding reaches the gate only with an exception approved by the board or its equivalent: a board risk committee, the management board in a two-tier structure, or executive management where there is no board. The pack's defaults are a decision within 5 working days, at most 30 days and one renewal. Otherwise fix and retest, or hold. A feature that releases a payment takes no exception. A High finding open at the same time needs its own exception. Once the exception is approved or the fix is retested, answer the question before this one again.

Outcome

Hold, or take a High exception to two approvers

A High finding reaches the gate only with an exception approved by both the Head of Information Security and the accountable executive. The pack's defaults are a decision within 10 working days, at most 90 days and one renewal. The request names the finding, why it cannot be fixed by the due date and the compensating actions, each with an owner. Otherwise fix and retest, or hold. Once the exception is approved or the fix is retested, answer the question before this one again.

Decision

Do the evidence, blocking checks and test report all pass?

Three gates remain. Every item of required evidence for the depth is on file. No configuration check that blocks release has failed or is unverified. At Depth 3 the independent report has been received and its findings are in the register. The AI governance lead runs the gates in order and writes down each one that fails. If you cannot tell, answer no.

• Yes: Approvers sign: Release, or Release with conditions
• No, or not known: Hold: name the failed gates and restart where the gap is

Outcome

Hold: name the failed gates and restart where the gap is

A gate that fails holds the release. The decision record names each failed gate and what clears it, and the review restarts at the stage that holds the missing item. A new feature does not go live. For a feature already running, the AI governance lead may suspend it while the approvers decide.

Outcome

Approvers sign: Release, or Release with conditions

The AI governance lead convenes the decision. The approvers of the higher of the P24 tier and the depth sign, and the Head of Information Security signs as well from Depth 2. Nobody approves a use they own, and any approver may hold the release. Release if nothing is open. Release with conditions if a check is unverified or a finding is open under a plan or an exception, each with an owner and a date. Then switch monitoring on and set the next review date.