Every team that ships an AI feature eventually asks which model it should use. The question sounds like an infrastructure question, gets answered by an infrastructure team, and produces a decision nobody outside that team can explain six weeks later.
That is the wrong shape. Model selection is a trade between cost, latency, and quality, and those three things belong to whoever owns the feature. What infrastructure should provide is not the answer but the mechanism: a way to express the trade, apply it per request, and see afterwards whether it held.
The default is a decision
Most products start by picking the strongest model available and sending everything to it. This is a reasonable default and an expensive one. It is also, importantly, a decision — one made implicitly, with no record of the trade being accepted.
The cost of that decision is not evenly distributed. In every mixed workload we have measured, the distribution of difficulty is heavily skewed: a large majority of requests are classification, extraction, or summarisation that a much smaller model handles indistinguishably, and a thin tail genuinely needs the frontier. Paying frontier prices for the majority to serve the tail is a choice, and rarely a deliberate one.
If you cannot name the request types where your expensive model earns its price, you are not choosing a model. You are avoiding a measurement.
Expressing the trade
The useful primitive is not "which model" but "what am I willing to accept". Three constraints cover almost every real case: a quality floor, a latency ceiling, and a cost ceiling. Given those, selection becomes mechanical.
# The question is not "which model is best" but
# "which model is good enough for this request".
response = client.reason(
model="auto",
input=request.prompt,
routing={
"optimize": "cost",
"min_quality": "frontier-sm", # your floor, from your evals
"max_latency_ms": 1500, # your product's budget
},
)Notice what is in that block and what is not. There is no model name. There is a floor derived from your evaluation set, a latency budget derived from your interface, and an objective derived from what this particular request is for. All three are product facts. None of them are infrastructure facts.
The quality floor is the hard part
Everything above depends on min_quality meaning something. A floor expressed
as a vendor benchmark score is worthless — it measures a distribution that is not yours. A
floor expressed as "passes our evaluation set at or above this rate" is a real constraint, and
building that evaluation set is the actual work.
It does not need to be large. Two hundred examples drawn from real traffic, labelled by someone who understands the domain, will separate the models that work from the models that do not far better than any public leaderboard.
Measuring afterwards
A routing decision you cannot audit is worse than no routing at all, because it converts a known cost into an unknown one. Every response should carry the route it took and why, and every trace should let you replay the request against a different route to see what would have changed.
- Route recorded per request. Which model answered, which were tried, and the reason for the selection.
- Cost attributed per feature. Not per project. The unit of decision is the feature, so that is the unit of accounting.
- Quality tracked continuously. Your evaluation set run against live routes, not once at selection time.
Who owns it
Our position is that routing policy should live next to the feature it serves, in the same repository, reviewed by the same people, and changed on the same cadence. Infrastructure owns the mechanism, the health probes, and the failover chain. The product owns the trade.
The practical test is simple: if the person who can explain why a feature exists cannot also explain why it uses the model it uses, the decision is in the wrong place.
This article is sample content written to demonstrate the article template. Replace it with your own writing.