What Procurement Leaders Need Before Approving an AI Engagement

Procurement is where AI projects either succeed or fail.

As the General Manager at Adrenalin, I see a steady stream of briefs from Australian organisations looking for an agency to deliver an AI capability. Most procurement teams follow the same supplier-selection process used for any other digital project.

However, that process was designed for projects which had some semblance of scope, budget parameters and a fixed end state with a desired impact.

AI projects do not always have these.

I suggest three things procurement can add to the brief to enrich the conversation between them and the agency they wish to engage. The aim is to make sure the difficult questions are discussed before the project starts, rather than after the AI has been built.

Why the standard brief falls short for AI

A traditional digital project typically carries uncertainty. Scope changes, platform feasibility is better understood after engagement, and new requirements appear during delivery.

With AI, the uncertainty can also sit inside the thing being built. The model may change during the project. The quality of the AI response depends on the data and instructions it receives. The team may only learn what a good answer looks like after testing the AI against real scenarios.

This makes it difficult to define every requirement and success measure before the work begins.

However, that does not mean procurement should accept an open-ended project. Instead, the brief should explain how the organisation and agency will make decisions when something changes. There are three areas I would suggest procurement teams include.

1. Explain how the AI will be tested

Most briefs are clear on the desired business outcome. The AI should reduce the time it takes to respond to a customer. It should help employees find information more quickly. It should reduce the amount of manual work required to classify documents.

But the brief should also explain how the team will test whether the AI itself is producing a reliable result.

For example, imagine an AI assistant that helps employees find information from company policies. The test should include the common questions that are easy to answer. It should also include questions where two policies appear to disagree, requests involving sensitive information and questions that the AI should refer to a person.

The brief does not need every test case written before the supplier is selected. It should explain who will create and approve the test cases, how often they will be run and what result would require the team to investigate further.

I would ask potential vendors:

• How will you prove the AI is producing the right result?

• Which real scenarios will be included in the testing?

• How will you test unusual or high-risk requests?

• Who will maintain the test cases after launch?

• What result would require the AI to be adjusted?

This gives procurement and the business a much clearer way to compare approaches.

It also makes sure testing is included in the project from the beginning, rather than becoming an additional piece of work that appears just before launch.

2. Define what happens when the AI needs to step back

Every AI project should explain what the organisation will do - and how they will do it - if the AI needs to be reduced, paused or removed.

This may mean returning an action to human approval, switching back to an earlier model or prompt, sending the user through an existing process or temporarily turning off one part of the AI capability.

If an internal AI agent is giving staff the wrong policy information, the fallback may be to direct those questions back to the service desk. If an AI agent is creating customer refunds, the fallback may be to return every refund to human approval.

Simply saying "we can switch it off" may not be enough.

If the AI has replaced part of an existing process, the people, systems or instructions required to run the old process may no longer be ready. The fallback needs to be designed and tested like any other part of the solution.

I would want the brief to answer:

• Which AI actions can be stopped separately?

• What process takes over when they stop?

• Who can make that decision?

• How quickly can the change happen?

• What happens to work already in progress?

• What needs to happen before the AI can return?

Including this in the procurement brief gives the agency an opportunity to design and price the fallback properly.

3. Write down the assumptions being made about the data

Every AI project makes assumptions about the information it will use.

The source documents will be accurate. Customer records will contain the right fields. Product information will follow a consistent format. The volume of requests will remain within an expected range.

These assumptions may be true when the project starts but change after the AI is launched.

For example, an AI assistant may be tested using a well-maintained set of policy documents. Six months later, a new team begins uploading documents in a different format or older policies remain available after they have been replaced.

The AI response may start to decline even though the model and prompts have not changed. A simple register of data assumptions helps the team understand what the AI depends on and what changes should trigger another review.

It should cover:

• where the information comes from

• who owns its quality

• how often it is updated

• the expected format and volume

• known gaps or exclusions

• which changes require the AI to be tested again

This also helps when the project scope changes. If the business adds a new source of information halfway through delivery, the team can see which assumptions, tests and costs need to be reconsidered before agreeing to the change.

How the three additions work together

These three additions are useful because they are correlated. The data assumptions explain what the AI needs to produce a good result. The testing approach checks whether the AI is still producing that result. The fallback explains what happens when it is not.

Using the internal policy assistant as an example:

• The data assumptions identify the approved policy library and who keeps it current

• The testing approach checks whether the AI finds the correct policy and handles sensitive questions properly

• The fallback sends high-risk questions to the service desk if the AI begins producing the wrong answer

Procurement now has a clearer view of what is being purchased.

The organisation is buying an AI assistant and the process used to check it, maintain it and safely step back when something changes.

Keep it practical

These additions do not need to turn the brief into a hundred-page document.

Each one can start as a focused one-page section: the important AI tests, owners and review frequency; the fallback options, decision owners and recovery steps; the important data assumptions, owners and review triggers.

When these are in place, procurement, the business, technology teams and the agency are all discussing the same issues before the engagement starts. That creates a better conversation at the start and fewer surprises once the AI is being used by real people.