Model Tiering: The Fastest AI Cost Win Available
If a portfolio company has meaningful AI infrastructure spend and has never segmented its traffic by difficulty, there is almost certainly a large, fast, low-risk saving available. In the environments I have analyzed it is the closest thing to a free lunch in AI cost optimization, and it is routinely untouched.
The idea is not complicated. Most teams building on language models pick one capable model during prototyping and send everything to it. That is the right decision early — you are trying to find out whether the product works, not whether it is cheap. The problem is that the decision is rarely revisited, and it silently becomes the company's largest variable cost.
The traffic is not uniform, but the routing is
When you actually classify a production workload by difficulty, the distribution is strikingly consistent across the companies I have looked at. Roughly:
- 70% simple. Password resets, account lookups, straightforward classification, formatting, extraction from well-structured input. A small fast model handles these at effectively identical quality.
- 25% moderate. Multi-step reasoning, ambiguous input, cases needing context assembly. A mid-tier model handles these well.
- 5% genuinely hard. Novel situations, subtle judgment, high-stakes output. These need the frontier model, and they are worth paying for.
Cost per token between the cheapest capable model and the frontier model in a given generation typically differs by one to two orders of magnitude. Sending the 70% bucket to a frontier model is not a small inefficiency; it is most of the bill.
Why it matters for PE: This is a cost structure problem disguised as a technical decision. It sits in engineering, so finance never sees it, and it is invisible on a P&L that shows a single infrastructure line.
The savings, with confidence levels
The reason I like this optimization is that it behaves like right-sizing rather than re-architecting — you can put a defensible confidence level on it.
- Routing simple traffic to a small model: 90% confidence. The work is well understood, the quality risk is measurable in advance with an evaluation set, and it is reversible per-request. Expect this alone to remove 50-65% of inference cost.
- Adding a mid-tier for moderate traffic: 70% confidence. More classification judgment required, and the boundary between moderate and hard needs tuning.
- Caching and deduplication of repeated queries: 70% confidence. Frequently a further 10-20% on top, and often overlooked entirely.
- Prompt compression and context trimming: 50% confidence. Real, but quality-sensitive and needs careful measurement.
A company spending $3M annually on inference with no tiering should expect to model $1.8M to $2.4M of annualized savings, risk-adjusted to something closer to $1.5M. At a 10x exit multiple that is $15M of enterprise value from a project measured in engineer-weeks.
Why teams have not already done it
It is worth understanding why this opportunity persists, because the reasons tell you how hard it will be to capture.
Nobody owns the cost. Engineering owns latency and quality. Finance owns the budget. The inference bill sits between them, and the person who could reduce it is not measured on it. This is the same organizational gap that produces cloud waste generally, and it has the same fix — someone senior enough to own both sides, which is much of what a fractional CTO does.
Quality anxiety. Teams fear that a cheaper model will produce worse output and they will not notice until customers do. This fear is legitimate and is precisely what an evaluation harness resolves. If the company already has one, tiering is a straightforward engineering task. If it does not, building one is the real project — and it is worth doing regardless, because a team shipping model changes without evaluation is carrying a quality risk nobody has quantified.
It was never a decision. The single-model architecture was a prototype default that shipped. There is no document arguing for it, which means there is nobody defending it either. In my experience these are the easiest optimizations to get agreement on, once somebody puts the numbers in front of the right person.
A sensible sequence
For a portfolio company starting from zero, the order that works:
- Instrument first. Log every request with its model, token counts, latency and cost. Most teams cannot currently answer "what does a request cost" and everything downstream depends on it. One to two weeks.
- Classify the traffic. Sample a few thousand requests and label them by difficulty. This is unglamorous and can be done in days, and it is where the 70/25/5 picture emerges for that specific workload.
- Build the evaluation set before changing anything. A few hundred representative cases with known-good outputs. This is the artifact that makes every subsequent change safe.
- Route the easy bucket to a small model and measure against the evaluation set. This is where most of the money is.
- Tune the boundaries and add the mid-tier. Iterate on the classification threshold.
Six to ten weeks end to end for a team that is not fighting other fires, and the savings begin at step four.
The diligence version
If you are evaluating a target rather than operating one, the tiering question is a proxy for something larger. Ask which model serves which request. The answer tells you whether anybody has looked at the cost structure at all.
A team with a considered routing policy is a team that understands its own economics — and the easy savings are probably already captured, so do not underwrite them again. A team sending everything to one frontier model has left money on the table, and you now know roughly how much and how confidently you can get it.
Either answer is useful. Only one of them is a surprise after closing.
Want to know what your portfolio companies are spending per inference? Book a free discovery call.