Short answer
Measure AI ROI by taking a baseline before you deploy, then tracking four numbers: hours returned per week, acceptance rate, escaped errors and queue time. Convert the hours to money, subtract the full cost — licence, integration, tuning, the review time you keep, and change management — and compare against the baseline.
Key takeaways
- Take the baseline before you deploy. It costs two hours and it is the whole game.
- Track hours returned, acceptance rate, escaped errors and queue time. Ignore usage counts.
- Acceptance rate moves first and tells you most — watch it weekly.
- The licence is rarely the biggest cost. Integration usually is.
- Write the kill criteria before you start, or sunk cost will keep a mediocre deployment alive for years.
Ask most companies whether their AI paid for itself and you get an anecdote. Somebody is delighted, somebody else thinks it is overhyped, and the finance team has no line item to point at. That is not a measurement problem, it is a setup problem: almost nobody writes down what they expect before they start, so there is nothing to compare against afterwards.
Take the baseline before you deploy, not after
This is the whole game and it takes about two hours. Before anything is installed, measure the current state of the specific task you are about to change.
- How long does one instance take, measured on a handful of real cases with a stopwatch rather than estimated in a meeting?
- How many instances happen per week, from the system of record rather than from memory?
- How often does it go wrong today, and what did the last three mistakes cost to fix?
- How long does the work wait in a queue before someone starts it?
That last one is the number companies most often skip and most often regret skipping, because queue time is frequently where the real cost lives. A quotation that takes twenty minutes of work but three days of waiting is not a twenty-minute problem.
Four numbers worth tracking, and what each one hides
| Number | What it tells you | What it hides |
|---|---|---|
| Hours returned per week | The headline benefit, in the currency a CFO understands | Whether those hours went into better work or evaporated |
| Acceptance rate | Share of outputs a person used unchanged — the best single health metric | Whether people are accepting because it is right or because checking is tedious |
| Escaped errors | Problems that reached a customer before and after | Slow-surfacing defects; needs a long enough window to mean anything |
| Queue time | How long work waits before it is touched | Seasonality — compare like periods, not last month against this one |
Acceptance rate is the one to watch weekly
Of those four, acceptance rate is the one that moves first and tells you most. It is simply the share of outputs a person used without changing them.
A rate that climbs means the system is being tuned to your work. A rate that is flat and high from day one usually means people have stopped checking, which is a risk rather than a success. A rate that falls after a model update tells you something changed underneath you — which is worth knowing before a customer finds out.
Counting the costs honestly
The licence is rarely the biggest number, which is why licence-only comparisons mislead. A realistic first-year total has five parts.
- 1Licence or usageThe published price, at your real annual volume rather than the headline tier. Model the volume you will have in month twelve, not month one.
- 2IntegrationConnecting to your ERP, PLM, mailbox or document store. On agent-shaped projects this is usually the largest line and the one most often left out of the business case.
- 3Rules and tuningEncoding how your company actually works — your tolerances, your templates, your specifications. This is the work that makes the difference between a generic tool and something useful.
- 4The review time you keepHuman approval is not free and does not go away. Count it. A system that halves the work but requires a senior person to check everything has a different business case than one that does not.
- 5Change managementTraining, documentation, the productivity dip in the first weeks. Small, real, and consistently forgotten.
If the only number in your business case is the licence fee, you are not comparing options — you are comparing price tags.
When to stop
Set the kill criteria before you start, and write them down. Something like: if acceptance rate is below sixty per cent after eight weeks of tuning, or if hours returned are under a threshold you name now, we stop. Deciding this in advance is the only reliable protection against the sunk-cost argument that keeps mediocre deployments alive for years.
In short
Most AI business cases fail on measurement rather than on results. Spend two hours on a baseline before anything is installed, agree the four numbers and the kill criteria in advance, and count the costs the licence fee hides. Then the question of whether it paid for itself has an answer instead of an argument.
Frequently asked questions
How do you calculate ROI on AI?
Put a financial value on the improvement — hours saved multiplied by loaded cost, or revenue from a shorter quote cycle — then subtract the full cost of the implementation and divide by that cost. The arithmetic is trivial. The difficulty is having a credible before-number, which is why the baseline matters more than the formula.
What KPIs should we track?
Four: hours returned per week to a named team, the share of outputs a person accepted unchanged, errors that reached a customer, and how long work waits in a queue. Logins, documents processed and number of pilots are activity metrics — they rise whether or not anything improved.
We already deployed without a baseline. Is it too late?
No. Reconstruct one from the system of record: ticket timestamps, document dates, order-to-quote intervals. It is less clean than measuring up front, but it beats arguing from impressions, and it is usually an afternoon's work.
How long before we should expect a return?
Give it 90 days minimum — roughly 30 to deploy properly and 60 to see whether the numbers hold under real volume. Judging at week three measures the novelty, and judging at month nine means you have already paid for a year of something you never validated.
What costs do companies forget?
Integration with ERP, PLM or the mailbox, which on agent-shaped projects is usually the largest line. Then the rules and tuning that encode how your company works, the human review time that does not go away, and change management. The licence is often the smallest of the five.
What if the benefit is quality rather than time?
Measure escaped errors and what the last few cost to fix, then be explicit that you are claiming fewer defects rather than faster throughput. Mixing the two is how business cases get quietly overstated — faster shortlists are easy to count, better hires are not.