Plan an AI Procurement

Direct answer

How should an organisation plan an AI procurement? Define the exact use case, affected people, data classes, accountable owner, human oversight, failure scenarios, success measures and critical dependencies before choosing a supplier. Evidence depth should increase with impact, automation, sensitive data, regulatory exposure and the difficulty of reversing deployment.

Good AI procurement starts before a supplier shortlist exists. The buyer should first define the business problem, the people affected, the data involved and the consequences of failure.

1. Define the decision or task

State exactly what the AI system will do. Avoid broad descriptions such as “use AI to improve efficiency.” Record the business process, the decisions or outputs involved, the users, affected people and the operational boundaries.

Ask:

• What problem are we solving?
• Why is AI appropriate?
• What happens if the system is wrong, unavailable or manipulated?
• Is the system advisory, automated or decision-making?
• Which decisions must remain with a human?

2. Establish risk ownership

Assign a named business owner and a named risk owner. Procurement, legal, privacy, security, data and operational teams may all contribute, but one accountable owner should be able to explain why the proposed use is acceptable.

The DSIT AI Risk Management Toolkit is designed for teams involved in designing, operating, procuring and delivering AI-enabled products and is a useful starting point for structuring this work.

3. Map the data

Before asking suppliers about models, identify the data the service may access:

• personal data;
• special-category or sensitive data;
• confidential commercial data;
• intellectual property;
• customer or citizen records;
• prompts, files, logs and feedback;
• retrieval sources and connected systems.

Record where the data originates, where it may be processed, who can access it, how long it is retained and whether supplier terms permit training or improvement use.

4. Define human oversight

Human oversight should be designed, not assumed. Specify where a person reviews, approves, overrides or investigates AI outputs, and what information they need to do that effectively.

For higher-impact use cases, define escalation routes and what happens when confidence is low or the system behaves unexpectedly.

5. Define success and failure

Procurement requirements should contain measurable success criteria. Depending on the use case these may include quality, error rates, latency, coverage, accessibility, security, user satisfaction, cost, explainability, false-positive/false-negative rates or task-specific benchmarks.

Also define unacceptable outcomes and stop conditions.

6. Understand dependencies

An AI product may depend on several organisations and components: foundation-model providers, cloud providers, vector databases, content filters, speech services, third-party APIs and data sources. Identify which dependencies are critical and what happens if they change.

7. Decide the evidence level

Not every purchase needs the same due diligence. Evidence depth should increase with the potential impact, data sensitivity, degree of automation, number of users, regulatory exposure and difficulty of reversing the deployment.

Planning output

Before moving to specification, produce a short procurement brief containing: 1. intended use; 2. users and affected parties; 3. data classes; 4. risk owner; 5. human-oversight model; 6. material failure scenarios; 7. success measures; 8. critical dependencies; 9. required evidence depth; 10. approval route.

Primary guidance

The UK Government AI Playbook explains safe, effective and secure AI use and includes how government organisations select, buy and deploy AI. The NCSC secure AI guidance also emphasises threat modelling, secure design and lifecycle risk.

Next: Specify an AI procurement.