Notes on the machinery

1 · The thing that was annoying

The application I was working on before this one had model identifiers in forty-odd places. Not in a config file — in the call sites, next to the prompt, because that is where they end up when each one is added by whoever needed it that week. Changing which model served a feature meant a code change, a review and a deploy. Nobody could say what any single feature cost, because the only cost signal was one invoice a month with one number on it. And when quality dropped — a model revision, a prompt that stopped fitting — the first report came from a support ticket.

Three separate problems, and they have one shape: the decision about which model serves a request is made at authoring time, in code, by somebody who cannot see what it costs or whether it worked. So the design here moves that decision to request time, to one place, and makes it report on itself. A capability such as text.summarize is a row in a table rather than a constant in a file — publish one from the console and it serves calls immediately, with no deploy. What the caller sends is the outcome it wants. What serves it is ours to decide, and ours to be wrong about in public.

2 · Why the router reads the request

The obvious design is a route table: this endpoint uses the cheap model, that one uses the expensive one. It is simpler, it is inspectable, and it needs no classification pass. I rejected it, and the reason is narrower than “semantics are better”.

A route table binds cost to the endpoint. But an endpoint is not a workload. One capability emits several kinds of work on the way to one answer: a research call rewrites the query, plans retrieval, compresses evidence and then synthesises — four steps that want four different models. Bound to an endpoint, all four get whatever the most demanding of them needed, and the cheap steps are subsidising the expensive one at every call. Splitting the endpoint fixes it only until the next workload arrives, at which point you are maintaining a table whose rows are guesses about traffic you have not seen. That is the maintenance burden: the table is never wrong loudly, it is wrong quietly and forever.

So the routing unit here is the task class — what kind of work this call is — and it is deliberately a different axis from the capability. Billing keys on the capability, because that is what was bought. Statistics key on the task class, because that is what varies.

Complexity comes out of classification as a score, and the score is banded:

/** Equal-width bands over the 0..1 complexity score, cheapest effort first. */
function bandComplexity(complexity: number): EffortLevel {
  if (complexity < 0.2) return "minimal";
  if (complexity < 0.4) return "low";
  if (complexity < 0.6) return "medium";
  if (complexity < 0.8) return "high";
  return "max";
}

packages/core/src/pipeline/cost-calculator-input.ts

Now the costs, which are the part worth reading. First, this buys a classification pass on every single request — and the classifier here is not a model. It is a keyword scorer over six domains with about a hundred terms between them, plus five weighted structural signals (length, sentence count, multi-part markers, stated constraints, stated goals). Subject difficulty carries the largest single weight, at 0.34, because length alone lets a rambling trivial prompt buy an expensive model.

Second, there is a threshold to tune, and it moves money. The band edges above are equal width over a score that is not uniformly distributed, which means a prompt sitting at 0.399 and one sitting at 0.401 get different models and different bills for no reason a user could predict. I do not have a principled answer to that; the bands are equal-width because that is defensible, not because it is right.

Third, the failure mode. A prompt that matches none of the six domains falls to a general profile at difficulty 0.35 and confidence 0.35, and is then routed on structure alone — length and punctuation. That is the classifier being confidently wrong in the way that costs you: not a dramatic misroute, but a systematic under-read of any request phrased in vocabulary the term list has never seen. Every domain is scored rather than short-circuited on the first hit, so a contract question that mentions a database once still lands in legal. A contract question that uses none of the thirteen legal terms lands in general.

3 · What the ceiling actually guarantees

The cost of a call is estimated before it is made, not reconciled after. The router returns an estimate built from the token count of the enhanced prompt, the capability’s system prompt — which is billed as input on every call and would otherwise be discovered in the settlement — and a tempered share of the model’s output ceiling. That estimate is converted to user tokens at the tenant’s margin, and then:

if (input.tenant.userTokenBalance < route.estimatedUserTokens) {
  throw new ApexError(
    "INSUFFICIENT_USER_TOKENS",
    "This request would exceed the available user-token balance.",
    envelope.promptId,
  );
}

packages/core/src/pipeline/index.ts

That check sits above the provider call, so an unfunded request costs nothing beyond the classification pass, which is local. The counter is the tenant’s balance column, and the debit happens after the provider returns, against real observed usage rather than the estimate.

Here is the honest part. That is a read, then a check, then — some hundreds of milliseconds later — a write. Two requests that arrive inside that window each read the same balance, each individually fit, and both proceed. The ceiling is therefore not a guarantee that spend stops at the balance; it is a guarantee that spend stops at the balance plus whatever is in flight.

I took last-write-wins rather than a distributed lock, and bounded the damage instead of eliminating it. The bound is the per-tenant concurrency limit, which defaults to 20 and is acquired before the pipeline runs and released in a finallyso a thrown error cannot leak a held slot. So the worst case is an overshoot of nineteen concurrent calls’ worth of estimate. At the tuned defaults that is small money; on a tenant with a large system prompt and a high effort ceiling it is not nothing, and I would not describe it as a hard cap in a contract.

A lock was the wrong trade at this size: it puts a round trip in front of every request to prevent an overshoot bounded at twenty, and it introduces a failure mode — the lock service being unavailable — that is strictly worse than the problem. The right fix is a reservation ledger, where the estimate is held against the balance at check time and either captured or released at settlement. That ledger exists in this repository, with its state machine encoded as a transition table and enforced a second time by database triggers. It is not wired into the request path yet. That is the single largest gap between what the platform does and what it should, and it is in section 7 for that reason.

4 · The grader problem

Every answer is checked against the original request before it is returned. The standard objection is immediate and correct: a quality layer that is a second model call is a second cost and a second latency hit on every request, and it can eat the entire saving the router just produced. If the router moves a call from a frontier model to a small one and saves 90% of the cost, and the grader is a frontier-model call on the output, the net saving is negative.

So the grader is not a model call. It is deterministic, it runs in-process, and it costs zero provider dollars and well under a millisecond. That is the arithmetic, and it is short because there is nothing to trade off: the gateway’s own overhead is CPU, and the only network hop in the request path is the one to the provider.

What that buys, concretely — it catches four things:

• Refusals. Nine opening markers. A model that declined is scored zero rather than passed through as prose.
• Truncation. An unterminated final sentence, which is the shape of a response cut off at the token ceiling.
• Reductive tasks that reduced nothing.An answer at or above 90% of its source’s length has not summarised it, however well written it is.
• Goal coverage. The content words of each goal recorded at the start, checked against the output. Half the terms present counts the goal as addressed.

And here is what it does not buy, which matters more. A fluent, correctly-shaped, confidently wrong answer scores well. Every one of those four checks is structural. None of them is a claim about whether the answer is true. A grader that could tell you that is a frontier model call, and it would cost more than the answer it is checking.

The one design decision here that earns its keep is not the checks — it is where they read from. What the request was asking for is written into a journal at classification time, keyed by prompt ID, before any answer exists. The grader claims it back by that ID and never receives it from the code that produced the answer. If it could be handed the goals by the answering path, it would be checking the pipeline against itself, and it would agree with itself every time.

There is a real cost to that independence: the journal has a five-minute TTL, and a lost handoff means the answer cannot be scored. That case warns rather than fails, because the tenant has already been charged and a platform fault is not a bad answer.

5 · Slow is harder than down

A provider returning 500 is easy. The interesting failure is a provider that answers in forty seconds instead of four, because every layer above it is still waiting and nothing anywhere has registered an error.

Timeouts are 60 seconds for a unary call and 300 for a streamed one. The gap between those two numbers is the whole reasoning: a streamed turn signals liveness with its first token, so a long one is observably alive and can be given room; a unary call’s only signal is completion, so a long one is indistinguishable from a hang and does not get the same patience. I picked 60 because it is roughly four times the slowest well-behaved completion I have measured at the top effort rung, and I have no better justification than that.

What stops a slow provider becoming a retry storm is that there is no automatic retry in the request path. A failed request fails. What changes is the state of a circuit breaker, scoped per provider-and-capability so a vendor having a bad afternoon does not trip capabilities served by somebody else. It is the standard three states, with one addition that I would argue for: an operator override is a fourth state rather than a boolean flag beside the other three. As a flag, an automatic transition silently undoes an operator’s decision the next time a probe succeeds, and the audited override is then not audited or an override.

The Understudy — the fallback tier — is honest but less clever than the marketing implies: a model disabled by an operator override drops out of the eligible set on the next call, and routing picks the cheapest of what remains. There is no automatic mid-request downgrade when a provider goes slow. That is section 7 as well.

6 · What I got wrong

The router costs every candidate model before choosing between them. The first version costed each one at its declared output ceiling — the largest response that model could possibly return at that effort level. It is the conservative choice, it never under-quotes, and it was wrong in a way that took a while to see.

Two consequences. The obvious one: on an ordinary prompt returning a few hundred tokens, costing at a 128,000-token ceiling overstates the spend several-fold, which inflates the pre-authorisation hold and refuses calls the tenant could comfortably afford.

The one I did not see coming: it was choosing the models. Selection is cheapest-first, and the estimate was dominated by maxOutputTokens — a field that describes what a model is permittedto return, not what this request will make it return. So the router systematically preferred whichever model declared the smallest output ceiling, which is a property of the vendor’s configuration and has nothing whatsoever to do with the request being served. It was making a routing decision on a field that carried no information about the work. It looked like it was working. Every call succeeded, every estimate was conservative, and the selection was being driven by a number nobody had thought of as a routing input.

The replacement scales the ceiling by the complexity the classifier already computed:

const OUTPUT_FLOOR_FRACTION = 0.25;
const OUTPUT_COMPLEXITY_SPAN = 0.55;

function temperedOutputTokens(profile: EffortProfile, complexity: number): number {
  const fraction = OUTPUT_FLOOR_FRACTION + OUTPUT_COMPLEXITY_SPAN * complexity;
  return Math.max(1, Math.ceil(profile.maxOutputTokens * fraction));
}

packages/core/src/pipeline/cost-calculator-input.ts

A trivial prompt claims a quarter of the ceiling, a maximal one four fifths. Those two constants are fitted to observed output lengths, not derived, and they will need refitting as the catalog changes.

The generalisable mistake is not “I was too conservative”. It is that I put a field into a ranking function without asking what it was a measurement of. A conservative estimate is a defensible position. A ranking driven by a field that does not describe the thing being ranked is a bug that produces plausible output forever.

7 · What is not built

• The reservation ledger is not wired. Section 3. The balance check is read-then- check, bounded by the concurrency limit rather than by a hold.
• The utility model is a heuristic. Classification, condensation and grading run on deterministic code, not a hosted model. The config switch for a real one falls back rather than making an unpriced network call.
• No mid-request downgrade. A provider that goes slow is waited on until the timeout; the breaker protects the next request, not this one.
• Google is out of the routable setsince 23 August 2026, when three catalogued models 404’d against live traffic inside an hour. It goes back in when a successor is added with a confirmed rate, not when the catalog check stops complaining.
• The grader is structural, not semantic. Section 4. It does not detect a confident falsehood and is not represented as doing so.

If you find something on this page that is wrong, or a gap that should have been on this list, I would like to know: [email protected].