Most AI cost lives in the pipeline, not the model

When a team decides its AI bill is too high, the first proposal is almost always a cheaper model. It is the most visible line item, the change is a one-line diff, and there is a benchmark to point at. It is also, in every production system we have measured, the smaller half of the problem, and the half that carries all of the quality risk.

A monthly total is not a diagnosis. The unit that lets you act is cost per call, per stage, joined to the business event that caused it: one support email, one consultation, one memory generated. Until the number is attributable to a stage and a feature, every optimisation is a guess and every saving is unprovable.

That measurement usually surfaces the same four things, none of which the model is responsible for:

• Calls that do not need to exist. Stages accumulated one sprint at a time, each reasonable on its own, several of them re-deriving what an earlier stage already produced.
• A premium model at every stage. Including the stages that classify, extract or reformat, where quality has nothing to do with model tier.
• Caching that is configured but not working. A prompt prefix that changes per call, or a cache key that includes a timestamp, and the discount silently never applies.
• Traffic on several paths. Three services calling three providers three ways, so no one can see the total and nobody owns it.

On the production stack behind a voice AI product, the backend had grown organically to fourteen-plus processing steps with AI traffic spread across several paths, premium models running at every stage regardless of whether quality required it, and prompt caching misconfigured. Correcting the caching alone took roughly 26% off premium-model cost per call. Restructuring the pipeline into a three-stage summarisation flow, moving to open-source models only where quality held, and putting every AI call behind one gateway brought the whole memory-generation cost down 75.6% on matched windows. The product did not change.

The usual objection is that multiplication commutes: cut the price of each call and delete the redundant calls, and you land in the same place whichever you do first. That is true of the destination and false of the journey. Take a stage running 100 calls a day at 10 units each.

1. Model first. Migrate all 100 calls to a model at 4 units: 400 units a day. Then restructure, and discover 60 of those calls were redundant: 160 units a day.
2. Pipeline first. Restructure and delete the 60 redundant calls: 400 units a day. Then migrate the surviving 40: 160 units a day.
3. Same destination, different bill for the work. The first order migrated, prompt-tuned and quality-tested 100 calls, 60 of which were about to be deleted. That is 60 calls of regression risk and migration effort spent on work that no longer exists.

It compounds in two more ways. Restructuring changes prompts, so any quality testing done before the restructure has to be redone after it. And caching depends on a stable prompt prefix; tune the cache first and the restructure invalidates exactly the thing you just tuned.

The arithmetic commutes. The effort does not, and neither does the risk.

Once the pipeline is the shape it should be, model substitution is worth doing, carefully, and stage by stage. The condition is not “the benchmark says it is close”. It is “our evaluation set says quality holds on our inputs”. A classification stage with a labelled set behind it can be migrated in an afternoon with a number to prove it. A stage with no eval set cannot be migrated at all, only swapped and hoped over.

Which is why cost work and evaluation work are the same project wearing different hats. The eval set is what converts “this model is cheaper” into “this model is cheaper and nothing got worse”, and it is the only artefact that lets a finance-driven change survive a quality review.

Cost reductions decay. New features add stages, a model default changes upstream, a retry policy gets more generous, and six months later the bill is back. One gateway for every AI call is what stops it: a single place where cost per feature stays visible, where a new model can be introduced deliberately, and where a jump in spend has a name attached to it within a day rather than a quarter.

If your AI bill has outgrown expectation, the useful first question is not which model to move to. It is whether anyone can currently say what a single transaction costs, and at which stage. That question is answerable in days; it is the starting point of how we approach cost work, and it belongs in the Productionize phase, alongside the accuracy and safeguard work it depends on.