Avriz gives each subscriber an isolated cloud machine with a coding agent on it. Every conversational turn the agent takes is ultimately a call to some large language model, and the platform's managed tier lets it choose which. The price spread across the available ladder is large — wholesale output-token cost spans 12× from the cheapest rung to the most expensive (Fig. 2) — while a majority of real turns ("rename this", "what does this function do?") are handled perfectly by the cheapest rung. Routing is thus a direct lever on both margin and user experience: under-routing a hard request produces a bad answer; over-routing an easy one burns money.
The first version of a router is always a heuristic, and heuristics rot: thresholds hand-tuned in June are silently wrong by September as models, prices and usage change. The interesting question is not "can we write a good routing rule" but "can the platform keep the rule good, from its own traffic, without ever risking that traffic". This report documents our answer, which is deliberately conservative: a learning system that spends its whole early life in shadow.
Three properties drove the design:
Zero-risk learning. The learner must not change a single user-visible decision until a human flips a flag — and even then only for a bounded slice of traffic under a rung ceiling.
Zero new data. Training must use only what the platform already records for billing (token counts, realized cost, status codes) plus derived, content-free features. No chat text is stored by, or recoverable from, the routing system.
Legibility. Every stage — features, prior, reward, estimator, activation — is inspectable in an admin report, and the model itself is a set of eleven-dimensional linear weights a human can read.
Selecting one action per context and observing a reward only for the chosen action is the contextual bandit problem, the one-step special case of reinforcement learning [8]. The linear-payoff formulation and its optimistic variant LinUCB were established for news recommendation by Li et al. [3]; posterior-sampling alternatives descend from Thompson [1] (see [7] for a modern tutorial), and confidence-based exploration from UCB [2]. Evaluating a new policy from logs generated by an old one is the off-policy problem: the direct method (fit a reward model, evaluate the candidate against it), inverse-propensity weighting, and their doubly-robust combination are analyzed by Dudík et al. [4]; Bottou et al. [5] treat the same machinery as counterfactual reasoning about a production system, which is exactly our setting. Practical system design for production bandits — logging, delayed rewards, gating — is discussed in Agarwal et al. [6] and evaluated at scale in [9]. Cost-aware routing across an LLM cascade was introduced as FrugalGPT [10]; learned routing between model pairs from preference data appears in RouteLLM [11]. Our contribution is not algorithmic novelty — the estimator is deliberately the simplest thing that can learn — but an end-to-end production recipe: reward built from a billing ledger, off-policy training from a heuristic logging policy, an explicit activation bar, and a ceiling-bounded ε-greedy graduation path, all inside a system where routing failures self-heal.
The unit of decision is a turn: one human message and the burst of agent LLM calls it triggers. The platform's model ladder is ordered easy→hard:
On each fresh turn, a judge — itself the cheapest capable model, prompted to answer with a single word — maps the message to a difficulty tier, and the tier to a rung:
The judge's failure mode is legible from its own construction: it sees only the message text. A terse "fix the login bug" grades trivial and pins a fifty-step debugging session to the weakest model. Two mechanisms bound the damage. First, follow-up calls within a turn reuse the turn's decision (a 10-minute memo), so the judge fires once per human message, not per call. Second, an escalation valve: when a turn's call count crosses fixed checkpoints, the rung is bumped one step —
Escalations are the router confessing its own mistake in-flight, which makes them a precious training signal (§5). Users who pin a model bypass all of this; bring-your-own-key boxes never enter the managed path at all.
Routing is a contextual bandit. At turn t the platform observes a context xt (derived from the message and light tenant state), chooses an arm at ∈ 𝒜, and receives a scalar reward Rt observable only for the chosen arm — the counterfactual "what would opus have done with this turn" is never seen. The goal is a policy π maximizing 𝔼[R].
The feature extractor is pure and cheap — no model call, no stored text. From the last user message m and tenant context it computes counts and flags, then scales them to roughly unit range with capped ratios (order is frozen; new features append):
The signals are chosen to see exactly what the text-only judge cannot: pasted code fences, traceback fingerprints, referenced file paths, hard verbs ("refactor", "migrate", "deadlock") versus easy verbs ("rename", "typo"), the rung the conversation was already on, and the tenant's recent escalation rate — a running confession of how often this tenant's turns get under-routed.
Before any learning, a hand-set linear scorer maps features to a difficulty in [0,4], aligned to ladder indices — long asks trend harder, code and tracebacks mean real work, easy verbs pull down, a conversation may step down at most one rung between turns:
This scorer is the candidate policy's prior and cold-start: until the bandit clears its data bar (§5.3), the shadow decision is the prior, so early shadow behaviour is deterministic and auditable.
No new telemetry is collected. A turn's reward is assembled after the fact from llm_calls rows the platform already writes for metering — attributed to the decision by a 600-second turn window, bounded by the tenant's next decision, and scored only after a 120-second settling delay:
Success means at least one usable served completion. Cost enters in cents so its magnitude is commensurate with the indicator terms on typical turns. Soft, self-healing routing events (provider exhaustion, empty completions, auth rolls) are excluded from failure — they are the platform routing around trouble, not the decision being wrong. A mid-turn escalation, by contrast, is charged to the decision: it is a non-counterfactual, in-production measurement of the judge's own mistake, and it is the single most valuable bit in the dataset.
Each rung a owns a linear model of estimated reward:
This is the regression / direct method contextual bandit — LinUCB [3] without the per-arm confidence matrix. The simplification is deliberate: at our data scale a full covariance estimate is noise, while eleven readable weights per arm are something an operator can audit in the admin panel.
Weights train online as rewards land in the backfill pass, one SGD step of ridge regression per scored turn, on the arm the live judge actually served:
i.e. per-sample gradient descent on ½(w·x − R)² + ½λ‖w‖². Training is off-policy by construction: arm a's model learns only from turns that genuinely landed on a under the judge. There is no importance weighting — the logging policy is a deterministic heuristic with no propensities to invert — which is precisely why the system graduates to measured on-policy comparison (§6) rather than trusting counterfactual estimates.
The learned policy is inert until it has coverage. With na the training count on arm a:
and its decision considers only well-sampled arms, with an optional optimism bonus held at zero in shadow:
Below the bar, the candidate is exactly the prior (5); above it, the bandit overrides. The admin report states which regime is live — warming up (heuristic prior) vs bandit driving — with per-arm counts, so there is never ambiguity about what produced a shadow pick.
An arm's weights encode "what reward this rung earns on requests like x" under a specific underlying model at a specific price. Two events break that premise, and they break it differently. A price change (same model, cheaper) is largely self-correcting: the cost term in eq. (6) is recomputed at backfill time from the actual ledger rows, so post-change turns already carry the new economics into R — only the fitted wa lags, and because the update (8) is constant-step-size SGD (not a decaying rate that would freeze), it never stops tracking; its effective memory is ~1/η ≈ 20 samples on that arm. The uncomfortable case is a model swap behind a fixed rung id (Sonnet 5 → 6, or a repointed pool): the arm keeps its weights and its large n, so its history is not merely stale but actively wrong, and the accumulated count suppresses the very re-exploration that would fix it.
We make that a first-class, operator-free event with an economics fingerprint per rung:
The fingerprint is stored beside each arm's weights and re-checked on every backfill. When it flips — any of model, endpoint, or price changed — that arm is reset (wa ← 0, na ← 0) and re-stamped, so it drops back through the difficulty prior (5) and must re-earn its way past the activation bar (9) on current data. The reset is surgical — only the changed rung forgets; well-sampled neighbours keep their history. An operator "reset arm" control does the same on demand, for a knowing swap you don't want to wait on. This is deliberately hard forgetting rather than a global decay factor: a decay rate slow enough to preserve a stable rung's hard-won signal is too slow to erase a swapped model's, whereas the fingerprint erases exactly the arm that changed, exactly when it changes.
Shadow evaluation is necessary but not sufficient: direct-method estimates share the reward model's bias [4]. The graduation path therefore serves the learned policy on a small, configured slice of live traffic and measures the comparison. With pct ∈ [0,100] an operator flag (default 0) and amax a rung ceiling:
Three properties make this safe to turn on. It is a hard no-op below the data bar — deploying the code changes nothing until an admin moves the flag. The ceiling bounds the blast radius: an over-eager policy cannot jump traffic to the top rung while trust is being built. And the platform's existing self-healing sits underneath: if a served rung is unavailable or unfunded, the standard fallback ladder catches it, exactly as for judge picks.
Turns the bandit serves are tagged (live_why='bandit'), which closes the loop twice over: their rewards are measured rather than modeled, and those measured rewards train the policy on its own decisions — the system's first on-policy data. The admin report then renders an A/B block comparing bandit-served vs judge-served reward, success and cost head-to-head; widening ε is justified by that measured table, not by counterfactual estimates.
The admin report backfills rewards on demand and renders, in order of how much we trust each number:
Agreement and confusion. Judge-vs-candidate agreement rate and the full 5×5 confusion matrix (Fig. 7). Disagreement mass concentrated one rung above the judge on turns that later escalated is the signature of the failure mode we built this to catch.
Escalation catch-rate. Among turns the judge under-routed badly enough to trip eq. (3), the fraction where the candidate had started higher. This is a non-counterfactual win: the escalation actually happened, in production, under the judge.
Reward on the agreement subset. Realized mean reward where both policies chose the same rung — a sanity check that the reward signal itself is stable before any delta between policies is read.
Counterfactual cost estimate — labeled an estimate. Where picks diverge, the candidate's rung was never served; its cost is modeled from that rung's realized per-turn mean. This is the direct method of [4] and inherits its bias; the report says so in the UI, and the number is treated as a screen, not a verdict. The verdict comes from the measured on-policy A/B of §6.
The routing system stores, per decision: timestamps, the tenant id, the two model picks, the eleven scaled features of eq. (4), and the outcome scalars of eq. (6). It stores no message text, and the features are deliberately non-invertible summaries (counts, capped ratios, binary flags). Users who pin a model in settings short-circuit the resolver before either the judge or the bandit is consulted; BYO-key boxes never enter the managed path; utility calls stay pinned to the cheapest rung. The learner is fleet-shared — every tenant's turns pool into one policy — which is both the cold-start solution (a single tenant would never clear eq. 9) and a compounding effect: a lesson extracted from one tenant's mis-routed turn immediately improves routing for all. The honest caveat is representation skew: early on, the policy's opinions are dominated by the most active tenants; the serving guardrails (§6) carry that risk until the data broadens. All of it — shadow recording, training, serving fraction, ceiling — is governed by runtime configuration, so every stage is one flag from off.
Direct-method bias. Without logging propensities, counterfactual value estimates lean entirely on the reward model; we mitigate by treating them as screens and promoting only on measured on-policy comparisons, but a stochastic logging policy (even ε=0.02 of exploration) would unlock doubly-robust estimation [4].
Reward shaping. Eq. (6)'s weights encode a judgment (a cent of spend is worth half a success). The report accepts weight overrides so sensitivity can be checked, but a principled calibration against retention or task-completion telemetry is future work.
Feature ceiling. Eleven linear features cannot represent interactions (a traceback in a long message means something different). The frozen-order vector design admits appending; the estimator admits replacement by any regressor without touching the recording, reward, or serving layers.
Non-stationarity. Discrete shifts — a model swapped behind a rung, a reprice — are handled by the fingerprint reset of §5.4, and constant-step-size SGD tracks gradual drift within a fixed model. What remains unaddressed is slow drift that never trips the fingerprint (a model's behaviour changing under a stable id and price): the constant learning rate tracks it, but with a lag inversely proportional to that arm's traffic, so a rarely-served top rung adapts slowest. A recency-weighted or windowed fit is the natural upgrade, warranted only once such drift is measurable.
Beyond routing. The ledger → reward → per-arm learner → flag-gated serving pipeline is decision-agnostic. Candidate next users on the platform: which memory facts to inject into a turn, when to sleep an idle box, and voice selection.
A production LLM platform can grow a learned router the way it would grow any risky capability: behind observation first, a data bar second, and a bounded, measured serving fraction third. The machinery required is small — one ledger table, 55 linear parameters, three config flags — and every piece of it is legible to the operator who must trust it. The judge stays as the prior and the safety net; the bandit earns each increment of traffic with measured wins. Nothing about the recipe is specific to routing, which is rather the point.
control-plane/rl_router.py and _resolve_model in control-plane/main.py.