Product

The right model, chosen per request.

Most requests do not need your most expensive model. Smart routing scores each request and sends it to the cheapest model that clears your quality bar — inside the latency and cost budget you set.

Start buildingRead the guide
  • 3objectives to optimise for
  • 38 msrouting overhead
  • 4fallback attempts, by default
  • 0code changes to enable
Objectives

Pick what matters for this call.

Routing is per request, not per application. The same service can optimise for cost on background jobs and for latency on the interactive path.

Speed

Routes to the model with the lowest measured time-to-first-token for this prompt shape and region, within your quality floor.

Cost

Finds the cheapest model that has historically passed your evaluation set on requests like this one, and stops at your per-request ceiling.

Quality

Sends the request to the strongest available model, with cheaper models used only as fallbacks when the primary is degraded.

Decision flow

How a request is scored.

Request arrives Prompt shape, token estimate, tools, and your routing block.
Classify
Complexity estimate A small classifier scores reasoning depth, context length, and tool use.
Filter
Eligible pool Models that meet min_quality, support the requested features, and are healthy in your region.
Rank
CostProjected spend against max_cost_usd.
LatencyLive p95 against max_latency_ms.
QualityRolling score on your evaluation set.
Send
Selected model On error or timeout, the next candidate is tried automatically. The response records every attempt.
routing.py 
response = client.reason(
    model="auto",
    input=prompt,
    routing={
        "optimize": "cost",              # cost | speed | quality
        "max_cost_usd": 0.02,            # hard ceiling per request
        "max_latency_ms": 1500,          # budget for first token
        "min_quality": "frontier-sm",    # never route below this tier
        "fallback": ["frontier-lg", "oss-70b-instruct"],
    },
)

print(response.route.selected)   # oss-70b-instruct
print(response.route.reason)     # "met quality bar at 22% of cost"

Nothing is a black box. Every response carries route.selected, route.reason, and the full attempt list, so a routing decision is always explainable after the fact.

Measured effect

What routing does to a mixed workload.

Figures below are from a representative production workload of 1M requests: support triage, summarisation, and extraction, mixed with a long tail of harder reasoning calls.

p95 latency

lower is better

Single frontier modelVantafold smart routing
Baseline
2,140 ms
Routed
890 ms

Cost per 1,000 requests

lower is better

Single frontier modelVantafold smart routing
Baseline
$18.40
Routed
$6.10

Baseline sends every request to a single frontier model. Routed applies optimize: "cost" with a quality floor. Quality was held flat against the same evaluation set in both runs.

Safety rails

Routing you can roll back.

Shadow traffic

Mirror a percentage of live requests to a candidate model, compare outputs offline, and promote only when the numbers hold.

Deterministic replay

Re-run any traced request against a different route to see exactly what would have changed, without touching production.

Pinning

Pin a route for a customer, a project, or a single endpoint. Pins survive model deprecations and are reported before they expire.

Stop paying frontier prices for every request.

Turn routing on with one field. Turn it off the same way.

Start building Read the docs