How to measure the ROI of an AI project before you start it

The decision to launch an AI project is almost always made on an impression. Someone saw a demo, someone read that a competitor is "adopting AI", and the budget is approved before anyone measured what actually changes. Six months later someone asks what it returned, and the room goes quiet — not because the project failed, but because nobody recorded what things looked like before.

The ROI of an AI project is not computed afterwards. It is computed before, because return is measured against something, and that something has to exist as a number before a single model enters the process. This piece is about how to reach that number and how to make the launch decision defensible rather than a matter of faith.

Without a baseline there is no return

Return on investment is the difference between two states, before and after. If you do not know what "before" looks like, you have nothing to subtract from, so every result reads as a gain. That is why the baseline is the first artefact of any serious AI project — a snapshot of the process as it runs today, before anything has changed.

The baseline must be captured before any intervention. The moment the team starts talking about "improving the process", people change their behaviour and the measurement is contaminated. What you measure has to be concrete: how many requests or cases per month, how many minutes per case, what share ends in escalation or rework, and what that costs per hour of the person doing it. Two to four weeks of sampling is enough to produce a figure that is not an anecdote, rather than a month of arguing about how long a case "roughly" takes.

Three cost lines people routinely forget

Most calculations stop at "time saved times cost per hour" and lose touch with reality there. Three costs are almost always left out, and each one can eat the whole expected saving on its own.

• Inference cost at real volume. A demo costs next to nothing. Production over thousands of calls a month carries a bill that grows with volume — and that bill is largest exactly where the project is most successful, because success means more calls.
• Evaluation and monitoring. A model that is accurate today does not stay that way on its own. Someone has to build test sets, watch for drift and react when behaviour shifts. That is a standing cost, not a one-off.
• The human review you just created. If the agent proposes and a person checks, you have not removed the work — you have moved it. Checking someone else's work can be slower than doing it yourself until trust is built. That minute per case is a cost, not a saving.

None of these three is exotic, yet all three are missing from the deck that approved the budget. If you do not put them in the arithmetic at the start, they show up on their own, at the moment when re-opening the project is most expensive.

Time saved is not money saved

The most common error in the arithmetic is treating minutes as money. If the agent saves two minutes per case across ten thousand cases, that is roughly three hundred and thirty hours "saved" a month. But those hours become money only under one of two conditions: either the freed capacity is redeployed to work that creates value, or the hiring plan changes because the work no longer requires a new person.

If neither happens, the saving is a phantom. People do the same work a little more comfortably, and the company's cost is unchanged. "We freed up time" is not a financial result until someone decides what that time is for. So any ROI conversation has to include the process owner who will say: that time goes here, or we will not fill that role. Without that decision, the saving stays on paper and never reaches the accounts.

Leading and lagging indicators

Lagging indicators are what you actually want — lower cost per case, shorter handling time, more cases resolved without escalation. The trouble is they only appear after a few months, too late to steer a pilot by.

So during the pilot you watch leading indicators that predict the outcome: the share of proposals a human accepts unedited, the share of cases where the agent admits it does not know instead of improvising, the average number of steps to resolution. When the leading indicators move in the right direction and hold, the lagging result tends to follow. When they do not, no amount of waiting will rescue it — and waiting is the most expensive way to learn what a leading indicator could have told you in the first month.

A worked example

The numbers below are invented for illustration — they represent no client and no market figure. The point is to show the shape of the calculation; you drop in your own measured values.

Assume a process with 8,000 cases a month, averaging 6 minutes per case, at a fully loaded labour cost (gross salary plus contributions plus overhead) of 15 EUR per hour. The agent cuts handling to 4 minutes per case but adds 1 minute of human review. Net saving is 1 minute per case.

The difference is about 900 EUR a month — provided those 133 freed hours are actually redeployed, or a planned hire is cancelled. If people stay on the same work at the same pace, the "after" column still costs 12,000 in reality, and the project is pure cost plus the model's bill. The same table with your measured numbers is the difference between a business decision and a guess.

Designing a pilot that yields a defensible answer

A pilot is not a demo with an extended deadline. Its only job is to return a number you can trust, and that takes a few deliberate choices.

1. Shadow mode. The agent runs alongside the human and proposes, but sends nothing and changes nothing. Compare the proposal with what the human actually did. You get accuracy with zero risk to the process.
2. Holdout group. A slice of traffic deliberately keeps running without the agent. Without a holdout you cannot separate the agent's effect from a seasonal shift in volume or from the team simply having a faster month.
3. Acceptance rate. The share of proposals sent unedited is the most honest leading indicator. It rises as the model matures, and once it settles above a threshold you set in advance, autonomy for that case type is justified.
4. A threshold defined up front. Decide before the pilot which acceptance rate and which error level count as success. A threshold set after the results is a rationalisation, not a measure.

This is the pilot we run when introducing NG Sara into sales and support or NG Nora onto a phone line: shadow first, then proposal, then autonomy — and only for the case types that cleared the threshold. The same holds when NG Operations proposes a production plan: the proposal is measured against the planner's decision before it earns any trust.

When killing the project is the right call

If after an honest pilot the acceptance rate does not clear the threshold, or the net saving does not cover the cost of inference, monitoring and review, the right decision is to stop. That is not a failure of the team — it is exactly what the pilot is for: keeping an expensive mistake cheap while it is still cheap.

It is harder to stop once the model is already in production. That is where the sunk cost trap bites: so many months and so much money have gone in that stopping feels wasteful. But the money spent is spent regardless of the decision — the only relevant question is whether the next month of operation returns more than it costs. If it does not, every further month only enlarges the loss; it does not recover what is already gone.

So define the kill threshold at the same time as the success threshold, while nobody is yet emotionally attached to the result. If you are not sure how to set those two thresholds for your process, that is precisely the conversation to have before launch, not after — because after, the conversation itself is part of the sunk cost.