Engineering notes.
Design decisions, incident reviews, and the occasional strong opinion about routing. Written by the people who operate the platform.
Routing is a product decision, not an infrastructure one
Choosing a model per request looks like plumbing. It is actually a decision about what your product is willing to trade — and it belongs to the people who own the outcome.
What a trace should contain when the callee is a model
Traditional tracing assumes deterministic callees. Model calls break that assumption in three specific ways, and each one changes what you should record.
Notes on prompt prefix caching in a multi-tenant gateway
Caching shared prefixes across requests is easy. Doing it without leaking anything between tenants, and without lying about token counts, is the interesting part.
Why we ship 12-month deprecation windows
A deprecation window is a promise about someone else's roadmap. Twelve months is the shortest span that survives a real planning cycle.
Incident review: delayed trace ingestion, 8 June
A bad deploy of the span writer put nine minutes of lag into trace ingestion. Inference was unaffected. Here is what happened and what we changed.
A practical way to find out if a cheaper model is good enough
Vendor benchmarks will not answer this. A shadow-traffic experiment against your own evaluation set will, in about two weeks.
How much does that feature cost? A method
Per-request cost is easy. Per-feature and per-customer cost require a tagging discipline you have to adopt before you need the answer.
Every card links to the sample article. Wire them to your own posts.
Get the notes by email.
One message when something substantial ships. No newsletter cadence.