Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How do we check an AI vendor or model provider before we rely on it?

7 decision pointsThe pack behind this answer: Generative AI Security

Add to board memo

The short answer

Recommendation
Ask the provider the AI security questions in writing, with evidence for every answer, and count the answers by what you hold. Check the contract carries the AI supplier terms, then give the counts to your supplier decision and ask again whenever the provider changes a model, a sub-processor or a term.
Reasoning
A provider's reply says what it does today, and your feature's security rests partly on it: where your data goes, who can read it, which model runs underneath and how you hear of a change. An answer is a claim until you have seen the evidence, so a question answered without evidence stays open. Counting answers with evidence, without evidence and not answered gives a result you can act on. The questionnaire informs the supplier decision under your third-party process and never replaces it.
Considerations
  • The only good answer on training is a plain no, written into the contract (ST-1). Without it the tool takes no Internal or Confidential data. A vendor setting verified off is an added control, checked at each review, and never a substitute for the term.
  • A certificate that covers other parts of the company does not answer for the AI feature you use.
  • The review procedure requires the questionnaire from Depth 2 when an outside provider supplies the model. At Depth 1 the provider's published security statement is read instead.
  • For a model you download, check its publisher, source and file hash, and load it in a form that does not run code (GS-15).

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
  • Loop back
StartWe want to rely on an AI vendor or model providerDecisionA provider's service, or a model we download ourselves?OutcomeCheck publisher, source and file hash, and load it safelyDecisionWill it get Internal or Confidential data, or is the use Tier 2 or 3?OutcomeP24 asks for no supplier assessment yetStepName the provider and the company behind the modelStepSend the questions and ask for evidence with each answerDecisionDid the provider reply by the date you set?OutcomeNo reply: escalate to the business owner firstStepCount the answers, then read each against a good answerDecisionCan our prompts or files be used to train any model?StepKeep to Public data until ST-1 is in the contractDecisionDoes the contract carry the other AI terms, ST-2 to ST-8?StepTake each gap to the contract by term and clauseDecisionIs any question open, or does any answer fall short?StepAgree a date with the provider to close the open questionsStepGive the counts and answers to the supplier decisionDecisionHas the provider since changed a model, sub-processor or term?OutcomeReviewed: keep the workbook and watch for changes

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 19 steps.

  1. Start

    We want to rely on an AI vendor or model provider

    A provider's reply says what it does today, and the security of your feature rests partly on it: where your data goes, who can read it, which model runs underneath and how you hear of a change. The questionnaire helps you ask for these facts in a form you can count. The decision to use the supplier stays with your third-party process.

    In the pack

  2. Decision

    A provider's service, or a model we download ourselves?

    A hosted service or a tool with an AI feature has a provider who can answer the questions. A model or adapter you take from a public source has a publisher rather than a provider you contract with. If you do both, run each route. If you do not know which applies, take the provider route and ask who supplies the model.

    In the pack

  3. Outcome

    Check publisher, source and file hash, and load it safely

    Use a downloaded model or adapter only after you have checked who published it, where it came from and its file hash, and load it in a form that does not run code when loaded. Record the model name and version, and add it to the list of models, datasets, libraries and connectors with versions. Pin tool and connector servers and approve each one before use.

    In the pack

    See also

  4. Decision

    Will it get Internal or Confidential data, or is the use Tier 2 or 3?

    P24 asks for a supplier assessment before the supplier receives Internal or Confidential data, and before a Tier 2 or Tier 3 use starts even with Public data. If you do not know the data level or the tier, answer yes. A Tier 1 use with Public data only is the one case P24 leaves out.

    In the pack

  5. Outcome

    P24 asks for no supplier assessment yet

    P24 does not ask for a supplier assessment for a Tier 1 use that takes Public data only. Your own third-party process may still ask for one. A new purpose, a higher data level, a new audience or a change of supplier, model or location forces a review of the use, and the answer to this question is then asked again.

    In the pack

  6. Step

    Name the provider and the company behind the model

    Use one workbook for each provider. If the company that supplies the model differs from the provider you contract with, name both and put the model questions to the provider about each. A tool supplier may run another party's model, and that party may keep or use your data on its own terms.

    In the pack

  7. Step

    Send the questions and ask for evidence with each answer

    Send the question and evidence columns only, and keep the reviewer's good-answer columns back. Set a reply date. A trust centre often answers many rows: paste its link as the evidence and check it. Questions about agents, tools and connectors apply only when the feature has tools, and a question that does not apply needs a stated reason. The review procedure requires the questionnaire from Depth 2 when an outside provider supplies the model; at Depth 1 it asks only that the provider's published security statement is read.

    In the pack

  8. Decision

    Did the provider reply by the date you set?

    Judgement: a late or partial reply is still a reply, and a blank row counts as not answered. The third-party pack escalates a supplier that sends no answer or evidence by the deadline, so this flow takes the no branch only for a supplier that sends nothing by then.

    In the pack

  9. Outcome

    No reply: escalate to the business owner first

    The third-party pack escalates a supplier that does not answer or send evidence by the deadline to the business owner, and then to the Head of Information Security. Until the supplier replies, every question counts as not answered. Judgement: keep the use to the data levels the tool is already approved for until it does.

    In the pack

  10. Step

    Count the answers, then read each against a good answer

    Mark each row Answered with evidence, Answered without evidence, Not answered or Not applicable. The reviewer, never the feature's owner, reads each answer against the good answer and records Meets or Falls short; the review procedure has the Head of Information Security read them. A certificate that covers other parts of the company does not answer for the AI feature you use. In the distributor's example, 24 of 31 applicable questions were answered with evidence and 7 stayed open.

    In the pack

  11. Decision

    Can our prompts or files be used to train any model?

    This covers the provider and the company behind its model (VQ-01, VQ-02, VQ-04). The only good answer is a plain no, written into the contract. A product page or a setting, even one verified off, does not replace the contract term. If the answer is yes, not known or not in the contract, answer on that branch.

    In the pack

    See also

  12. Step

    Keep to Public data until ST-1 is in the contract

    Without a contract term that bars training (ST-1), the tool takes no Internal or Confidential data, only Public information. A vendor setting verified off does not replace the term: where the tool has one, keep it off and check it at each review as an added control. P06 has no clause for ST-1, so add it in your own words. A use already running gets a dated condition to add the term (H-2).

    In the pack

  13. Decision

    Does the contract carry the other AI terms, ST-2 to ST-8?

    The terms cover processor terms, sub-processors, locations, incident notice, return and deletion on exit, notice of AI feature changes and the intended purpose. Each has a P06 clause except the statement of intended purpose (ST-8). Check the contract itself as well as the answers, because the standard asks for both. If you do not know what the contract says, answer no.

    In the pack

  14. Step

    Take each gap to the contract by term and clause

    Use the Terms sheet to see which questions inform each term and clause. For example VQ-24 and VQ-26 inform ST-5 and CL-07, VQ-07, VQ-08 and VQ-32 inform ST-6 and CL-13, and VQ-17 informs ST-7 and CL-20. ST-8 has no P06 clause: add it in your own words. Procurement raises each gap with the supplier. A supplier that refuses a required clause goes to the exception route.

    In the pack

    See also

  15. Decision

    Is any question open, or does any answer fall short?

    Open means answered without evidence or not answered. Falls short means the reviewer judged the answer or its evidence weaker than the good answer. Judgement: read the counts by section as well as in total, because a gap in one section, such as the model supply chain, can sit behind good answers elsewhere. Never turn the counts into a percentage or a score.

    In the pack

  16. Step

    Agree a date with the provider to close the open questions

    For each open question ask for the evidence named in the Evidence column, or ask again using the reviewer's ask-more column. Write the agreed date on the Provider sheet. An answer that falls short is a gap to take to the contract or to the supplier decision as a condition, not something to leave unread.

    In the pack

  17. Step

    Give the counts and answers to the supplier decision

    The assessor writes the review report with the questionnaire and evidence, and the business owner decides and signs: approve, approve with conditions, escalate or reject. Open questions and shortfalls can become conditions with deadlines. The answers inform that decision and never replace it. Keep the completed workbook with the feature's review file.

    In the pack

  18. Decision

    Has the provider since changed a model, sub-processor or term?

    A provider's reply says what it does today. A change of model is a trigger to review the feature, and a supplier that adds or changes an AI feature gives notice under ST-7. If you learn of a change late, answer yes and ask again for the rows it touches.

    In the pack

  19. Outcome

    Reviewed: keep the workbook and watch for changes

    Keep the completed workbook with the review evidence, as the standard asks. Watch for a notice of a new model, a new sub-processor or a changed term. Each is a reason to ask again and to review the feature within the working days the standard sets.

    In the pack

Practical checklist

  1. Name the provider you contract with and the company behind its model.
  2. Send the questions with a reply date, and ask for evidence with each answer.
  3. Escalate a supplier that sends nothing by the date to the business owner, then the Head of Information Security.
  4. Have someone other than the feature's owner count each answer and read it against a good answer.
  5. Confirm the contract bars training on your prompts and files, or keep the tool to Public information until it does.
  6. Check the contract itself for the other AI terms, and raise each gap through Procurement.
  7. Agree a date with the provider to close each open question.
  8. Give the counts and answers to the supplier decision, and ask again after any change.

Evidence to have ready

  • The completed questionnaire with the evidence for each answer and the reviewer's reading.
  • The counts by section: answered with evidence, answered without evidence, not answered.
  • The contract showing the AI terms and the supplier clauses, with each gap raised.
  • The agreed dates for open questions, and the signed supplier decision with any conditions.

Assumptions

  • You contract with a provider for a hosted AI service, or for software with an AI feature, and the use is on the AI inventory or going through intake.
  • Your third-party process already tiers and assesses suppliers and has its own security questionnaire. Judgement: this flow adds the AI-specific questions to that work and does not replace it.
  • The people reading the answers are not the owner of the feature. An answer given without evidence stays open, even when the provider is the only party who can answer.
  • The terms ST-1 to ST-8 and the clause IDs are the packs' defaults. Your own contract standard may ask for more.

Not covered here