% Learning to Route: A Shadow-First Contextual Bandit for Per-Request LLM Model Selection
% Avriz Engineering technical report AVZ-TR-2026-01, revision V4.
%
% This file is the PRINT source of truth for /eng/assets/routing-rl-paper.pdf. The dark-themed web
% version at https://avriz.io/eng/paper carries the same text; when the two disagree, this one and
% the code it cites win. Build with:
%
%     pdflatex routing-rl-paper.tex        (twice, for the figure/section cross-references)
%     python3 scripts/build_paper_pdf.py   (does the above into the served asset path)
%
% Needs only a stock TeX Live: article, amsmath, tikz, hyperref. No custom class, no shell-escape.
\documentclass[10pt,twocolumn,a4paper]{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{newtxtext,newtxmath}
\usepackage[margin=18mm,top=20mm,bottom=20mm]{geometry}
\let\Bbbk\relax   % newtxmath already defines it; amssymb would redefine and abort
\usepackage{amssymb}
\usepackage{tikz}
\usepackage{xcolor}
\usepackage{enumitem}
\usepackage[font=small,labelfont=bf]{caption}
\usepackage{fancyhdr}
\usepackage[hidelinks]{hyperref}
\usetikzlibrary{arrows.meta,positioning,calc}

\definecolor{acc}{HTML}{0A7D52}      % the phosphor green of the web page, print-safe
\definecolor{ink}{HTML}{16191C}
\definecolor{mut}{HTML}{5C666E}
\definecolor{rule}{HTML}{B9C2C7}
\definecolor{warn}{HTML}{B26A00}
\definecolor{r1}{HTML}{E3EFE9}       % single-hue ramp, light -> dark
\definecolor{r2}{HTML}{BFE0D0}
\definecolor{r3}{HTML}{7FC3A4}
\definecolor{r4}{HTML}{3F9E77}
\definecolor{r5}{HTML}{1F8A5F}

\color{ink}
\pagestyle{fancy}\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\fancyfoot[C]{\footnotesize\color{mut}Avriz Engineering $\cdot$ AVZ-TR-2026-01 $\cdot$ \thepage}

\newcommand{\rung}[1]{\texttt{#1}}
\newcommand{\A}{\mathcal{A}}
\newcommand{\C}{\mathcal{C}}
\newcommand{\code}[1]{\texttt{\small #1}}
\tikzset{
  bx/.style={draw=rule,fill=white,rounded corners=2pt,inner sep=4pt,align=center,font=\scriptsize},
  bxa/.style={bx,draw=acc},
  fl/.style={-{Stealth[length=4pt]},draw=acc,thick},
  flm/.style={-{Stealth[length=4pt]},draw=mut},
}

\begin{document}

\twocolumn[
\begin{center}
{\footnotesize\ttfamily\color{acc}TECHNICAL REPORT $\cdot$ AVZ-TR-2026-01 $\cdot$ REVISION V4}\par\vspace{4pt}
{\LARGE\bfseries Learning to Route: A Shadow-First Contextual\\ Bandit for Per-Request LLM Model Selection}\par\vspace{6pt}
{\footnotesize\ttfamily\color{mut}Avriz Engineering $\cdot$ avriz.io $\cdot$ August 2026 $\cdot$ control-plane/rl\_router.py}
\end{center}
\vspace{4pt}
\begin{center}
\begin{minipage}{0.93\textwidth}
\small\setlength{\fboxsep}{9pt}
\noindent{\footnotesize\ttfamily\color{acc}ABSTRACT}\par\vspace{3pt}
Serving every request with a frontier language model wastes an order of magnitude of cost on work a
small model handles; serving every request with a small model fails the work that matters.
Production LLM platforms therefore route: they pick a model \emph{per request}. We describe the
routing system deployed on Avriz, a managed development-box platform, in three layers: (i) a
hand-built \textbf{LLM judge} that classifies each turn's difficulty onto a five-rung model ladder,
with a mid-turn escalation valve; (ii) a \textbf{shadow-mode contextual bandit} that learns per-rung
reward models online from the platform's existing billing ledger, without storing chat content and
without affecting user traffic; and (iii) a flag-gated \textbf{$\varepsilon$-greedy serving path}
that lets the learned policy earn real traffic incrementally, bounded by a rung ceiling, a
deterministic constraint pre-filter and a plan-tier arm set, and measured by an on-policy A/B
comparison. The design premise throughout is that \emph{observation precedes influence}: the learner
trains off-policy from decisions the judge already makes, activates only past an explicit data bar,
and every step toward autonomy is one configuration flag from rollback. We give the full reward
construction, the per-arm estimator and its update rule, the activation and serving math, an
economics-fingerprint mechanism that resets a rung's learned weights when the model or price behind
it changes, and the honest limitations of direct-method off-policy evaluation at our scale.
\end{minipage}
\end{center}
\vspace{10pt}
]

\section{Introduction}
Avriz gives each subscriber an isolated cloud machine with a coding agent on it. Every
conversational turn the agent takes is ultimately a call to some large language model, and the
platform's managed tier lets it choose \emph{which}. The price spread across the available ladder is
large --- wholesale output-token cost spans $12\times$ from the cheapest rung to the most expensive
(Fig.~\ref{fig:price}) --- while a majority of real turns (``rename this'', ``what does this function
do?'') are handled perfectly by the cheapest rung. Routing is thus a direct lever on both margin and
user experience: under-routing a hard request produces a bad answer; over-routing an easy one burns
money.

The first version of a router is always a heuristic, and heuristics rot: thresholds hand-tuned in
June are silently wrong by September as models, prices and usage change. The interesting question is
not ``can we write a good routing rule'' but ``can the platform \emph{keep} the rule good, from its
own traffic, without ever risking that traffic''. This report documents our answer, which is
deliberately conservative: a learning system that spends its whole early life in shadow.

Three properties drove the design. \textbf{Zero-risk learning}: the learner must not change a single
user-visible decision until a human flips a flag --- and even then only for a bounded slice of
traffic under a rung ceiling. \textbf{Zero new data}: training must use only what the platform
already records for billing (token counts, realized cost, status codes) plus derived, content-free
features; no chat text is stored by, or recoverable from, the routing system.
\textbf{Legibility}: every stage --- features, prior, reward, estimator, activation --- is
inspectable in an admin report, and the model itself is a set of eleven-dimensional linear weights a
human can read.

\section{Background and related work}
Selecting one action per context and observing a reward only for the chosen action is the
\emph{contextual bandit} problem, the one-step special case of reinforcement learning [8]. The
linear-payoff formulation and its optimistic variant LinUCB were established for news recommendation
by Li et al. [3]; posterior-sampling alternatives descend from Thompson [1] (see [7] for a modern
tutorial), and confidence-based exploration from UCB [2]. Evaluating a \emph{new} policy from logs
generated by an \emph{old} one is the off-policy problem: the direct method (fit a reward model,
evaluate the candidate against it), inverse-propensity weighting, and their doubly-robust
combination are analyzed by Dud\'ik et al. [4]; Bottou et al. [5] treat the same machinery as
counterfactual reasoning about a production system, which is exactly our setting. Practical system
design for production bandits --- logging, delayed rewards, gating --- is discussed in Agarwal et
al. [6] and evaluated at scale in [9]. Cost-aware routing across an LLM cascade was introduced as
FrugalGPT [10]; learned routing between model pairs from preference data appears in RouteLLM [11].
Our contribution is not algorithmic novelty --- the estimator is deliberately the simplest thing
that can learn --- but an end-to-end production recipe: reward built from a billing ledger,
off-policy training from a heuristic logging policy, an explicit activation bar, and a
ceiling-bounded $\varepsilon$-greedy graduation path, all inside a system where routing failures
self-heal.

\section{The live router: judge, ladder, escalation}
\label{sec:live}
The unit of decision is a \textbf{turn}: one human message and the burst of agent LLM calls it
triggers. The platform's model ladder is ordered easy$\rightarrow$hard:
\begin{equation}
\A = \langle \rung{gemini},\ \rung{fugu},\ \rung{sonnet},\ \rung{opus},\ \rung{f\text{-}ultra} \rangle,\quad |\A| = 5
\end{equation}
The ordering is by \emph{capability tier}, not by price: the ladder is the difficulty tiers of the
judge in order, and price is not monotone along it (rung 1, \rung{fugu}, is dearer per output token
than rung 2, \rung{sonnet} --- Fig.~\ref{fig:price}). Every index in this report --- the prior's
rounding target in eq.~(5), the escalation step in eq.~(3), the serving ceiling in eq.~(12), and
both axes of the confusion matrix --- is an index into \emph{this} order.

On each fresh turn, a \textbf{judge} --- itself the cheapest capable model, prompted to answer with
a single word --- maps the message to a difficulty tier, and the tier to a rung:
\begin{equation}
g : \text{text} \rightarrow \{\text{trivial},\dots,\text{expert}\} \rightarrow a_0 \in \A
\end{equation}
The judge's failure mode is legible from its own construction: it sees only the message \emph{text}.
A terse ``fix the login bug'' grades \rung{trivial} and pins a fifty-step debugging session to the
weakest model. Two mechanisms bound the damage. First, follow-up calls within a turn reuse the
turn's decision (a 10-minute memo), so the judge fires once per human message, not per call. Second,
an \textbf{escalation valve}: when a turn's call count crosses fixed checkpoints, the rung is bumped
to the next \emph{available} rung above it (an unfunded or unreachable rung is skipped, so a bump
may move more than one index) and stops at the top of the ladder:
\begin{equation}
c \in \{12,48\}:\ a \leftarrow \min\{a' \in \A_\text{avail} : a' > a\}
\end{equation}
taking $a$ itself when that set is empty. Escalations are the router confessing its own mistake
in-flight, which makes them a precious training signal (\S5). Users who pin a model bypass all of
this; bring-your-own-key boxes never enter the managed path at all.

\begin{figure*}[t]
\centering
\begin{tikzpicture}[node distance=7mm]
  \node[bx] (turn) {turn arrives\\[1pt]{\color{mut}\tiny human message}};
  \node[bx,right=of turn] (judge) {judge $g(\cdot)$\\[1pt]{\color{mut}\tiny 1-word grade $\rightarrow a_0$}};
  \node[bxa,right=of judge] (serve) {serve rung $a$\\[1pt]{\color{mut}\tiny escalate at 12, 48}};
  \node[bx,right=of serve] (ledger) {ledger\\[1pt]{\color{mut}\tiny llm\_calls: cost, status}};
  \node[bx,right=of ledger] (rew) {reward $R$\\[1pt]{\color{mut}\tiny backfilled, eq. (6)}};
  \draw[fl] (turn) -- (judge); \draw[fl] (judge) -- (serve);
  \draw[fl] (serve) -- (ledger); \draw[fl] (ledger) -- (rew);
  \node[bxa,below=16mm of serve] (band) {bandit: $5 \times w_a$\\[1pt]{\color{mut}\tiny SGD-ridge, per-arm, eq. (8)}};
  \node[bx,below=16mm of turn] (shadow) {shadow ledger\\[1pt]{\color{mut}\tiny candidate pick $+\ \phi(x)$, no text}};
  \draw[fl] (rew.south) |- node[pos=0.72,above,font=\tiny,color=mut] {train (off-policy)} (band.east);
  \draw[fl,dashed] (band.north) -- ++(0,4mm) -| node[pos=0.28,above,font=\tiny,color=mut] {$\varepsilon$-greedy serve (flag, \S6)} (judge.south);
  \draw[flm] (turn) -- (shadow);
\end{tikzpicture}
\caption{\textbf{The learning loop.} Solid arrows run in production today: every graded turn flows
judge $\rightarrow$ serve $\rightarrow$ ledger $\rightarrow$ reward $\rightarrow$ bandit training.
The dashed arrow is the flag-gated serving path (\S6): a configured fraction of turns bypasses the
judge for the trained policy. The shadow ledger stores the candidate's pick and a content-free
feature vector --- never chat text.}
\label{fig:loop}
\end{figure*}

\begin{figure*}[t]
\centering
\begin{tikzpicture}[x=1mm,y=1mm]
  % log scale: x = 10 + log10(v)/log10(40) * 120, v in $/M output tokens
  \foreach \v/\lbl in {1/\$1,3.2/\$3.2,10/\$10,32/\$32} {
    \pgfmathsetmacro\xx{10 + ln(\v)/ln(40) * 120}
    \draw[rule] (\xx,-2) -- (\xx,34);
    \node[font=\tiny,color=mut] at (\xx,-4.5) {\lbl};
  }
  \foreach \name/\v/\c/\y in {gemini/2.5/r1/30, sonnet/15/r2/23, fugu/20/r3/16, opus/25/r4/9, f-ultra/30/r5/2} {
    \pgfmathsetmacro\w{ln(\v)/ln(40) * 120}
    \fill[\c] (10,\y) rectangle ++(\w,5);
    \node[font=\tiny,anchor=east] at (9,\y+2.5) {\texttt{\name}};
    \node[font=\tiny,anchor=west] at (10+\w+1.5,\y+2.5) {\$\v};
  }
  \node[font=\tiny,color=mut,anchor=west] at (10,37) {wholesale \$ / 1M output tokens (log scale)};
  \node[font=\tiny,color=acc,anchor=east] at (130,30.5) {$12\times$ spread};
\end{tikzpicture}
\caption{\textbf{Why routing pays.} Wholesale output-token cost per ladder rung (log scale; bars
sorted by \emph{price}, which is not the ladder order of eq.~(1) --- \rung{fugu} is rung 1 but the
third-dearest rung). A mis-route to the top rung costs $12\times$ the bottom one; the judge's job is
to spend that multiplier only when the turn earns it. Input-token rates show the same ordering.}
\label{fig:price}
\end{figure*}

\section{Problem formulation}
Routing is a contextual bandit. At turn $t$ the platform observes a context $x_t$ (derived from the
message and light tenant state), chooses an arm $a_t \in \A$, and receives a scalar reward $R_t$
observable \emph{only for the chosen arm} --- the counterfactual ``what would \rung{opus} have done
with this turn'' is never seen. The goal is a policy $\pi$ maximizing $\mathbb{E}[R]$.

\subsection{Context: an eleven-dimensional, content-free feature map}
The feature extractor is pure and cheap --- no model call, no stored text. From the last user
message $m$ and tenant context it computes counts and flags, then scales them to roughly unit range
with capped ratios (order is frozen; new features append):
\begin{equation}
\begin{split}
\phi(m,\text{ctx}) = [\,&1,\ \text{n\_words}/60,\ \text{n\_chars}/400,\\
&\text{code\_fence},\ \text{traceback},\ \text{n\_paths}/3,\\
&\text{hard\_verbs}/3,\ \text{easy\_verbs}/2,\ \text{question},\\
&\text{prior\_rung}/4,\ \text{esc\_rate}\,] \in \mathbb{R}^{11}
\end{split}
\end{equation}
The signals are chosen to see exactly what the text-only judge cannot: pasted code fences, traceback
fingerprints, referenced file paths, hard verbs (``refactor'', ``migrate'', ``deadlock'') versus easy
verbs (``rename'', ``typo''), the rung the conversation was already on, and the tenant's recent
escalation rate --- a running confession of how often this tenant's turns get under-routed.

\subsection{The prior: a transparent difficulty scorer}
Before any learning, a hand-set linear scorer maps features to a difficulty in $[0,4]$, aligned to
ladder indices --- long asks trend harder, code and tracebacks mean real work, easy verbs pull down,
a conversation may step down at most one rung between turns:
\begin{equation}
s(x) = \mathrm{clip}(u \cdot x,\ 0,\ 4), \qquad a_\text{prior} = \A[\,\mathrm{round}(s(x))\,]
\end{equation}
This scorer is the candidate policy's \textbf{prior and cold-start}: until the bandit clears its data
bar (\S5.3), the shadow decision \emph{is} the prior, so early shadow behavior is deterministic and
auditable.

\subsection{Reward from the billing ledger}
No new telemetry is collected. A turn's reward is assembled after the fact from \code{llm\_calls}
rows the platform already writes for metering --- attributed to the decision by a 600-second turn
window, bounded by the tenant's next decision, and scored only after a 120-second settling delay:
\begin{equation}
R = 1\cdot\mathbf{1}[\text{success}] - 0.5\,\text{cost}_{\text{\textcent}} - 0.4\cdot\mathbf{1}[\text{esc}] - 1.0\cdot\mathbf{1}[\text{fail}]
\end{equation}
Success means at least one usable served completion. Cost enters in \emph{cents} so its magnitude is
commensurate with the indicator terms on typical turns. Soft, self-healing routing events (provider
exhaustion, empty completions, auth rolls) are excluded from failure --- they are the platform
routing \emph{around} trouble, not the decision being wrong. A mid-turn escalation, by contrast, is
charged to the decision: it is a non-counterfactual, in-production measurement of the judge's own
mistake, and it is the single most valuable bit in the dataset.

\begin{figure}[t]
\centering
\begin{tikzpicture}[x=1mm,y=1mm]
  \draw[rule] (0,-1) -- (0,22);
  \node[font=\tiny,color=mut] at (0,-3) {0};
  \fill[r5] (0,18) rectangle ++(28,4);
  \node[font=\tiny,anchor=west] at (29,20) {$+1.0 \cdot$ success};
  \fill[warn] (-14,12) rectangle ++(14,4);
  \node[font=\tiny,anchor=east] at (-15,14) {$-0.5 \cdot$ cost (per \textcent)};
  \fill[warn] (-11.2,6) rectangle ++(11.2,4);
  \node[font=\tiny,anchor=east] at (-15,8) {$-0.4 \cdot$ escalated};
  \fill[warn] (-28,0) rectangle ++(28,4);
  \node[font=\tiny,anchor=east] at (-29,2) {$-1.0 \cdot$ hard fail};
\end{tikzpicture}
\caption{\textbf{Reward decomposition, eq.~(6).} One positive outcome term and three penalties, drawn
at their weights. Cost is continuous (its bar shows the weight per cent of realized spend); the
others are indicators. Weights are caller-overridable in the report so the balance can be re-tuned
when data argues for it.}
\label{fig:reward}
\end{figure}

\section{The learner: per-arm linear models, trained online off-policy}
\subsection{Estimator}
Each rung $a$ owns a linear model of estimated reward:
\begin{equation}
r_a(x) = w_a \cdot x, \qquad w_a \in \mathbb{R}^{11}
\end{equation}
This is the regression / \emph{direct method} contextual bandit --- LinUCB [3] without the per-arm
confidence matrix. The simplification is deliberate: at our data scale a full covariance estimate is
noise, while eleven readable weights per arm are something an operator can audit in the admin panel.

\subsection{Update rule}
Weights train online as rewards land in the backfill pass, one SGD step of ridge regression per
scored turn, on the arm that was \emph{actually served} --- the judge's pick in shadow mode, and,
once serving is on (\S6), the bandit's own pick on the turns it served:
\begin{equation}
w_a \leftarrow w_a - \eta\left[(w_a \cdot x - R)\,x + \lambda w_a\right],\ \ \eta = 0.05,\ \lambda = 10^{-3}
\end{equation}
i.e.\ per-sample gradient descent on $\tfrac{1}{2}(w \cdot x - R)^2 + \tfrac{1}{2}\lambda\|w\|_2^2$.
Training is \textbf{off-policy by construction} while the system is in shadow: arm $a$'s model learns
only from turns that genuinely landed on $a$, whichever policy put them there. There is no importance
weighting --- the logging policy is a deterministic heuristic with no propensities to invert --- which
is precisely why the system graduates to \emph{measured} on-policy comparison (\S6) rather than
trusting counterfactual estimates.

\subsection{Activation: the data bar}
The learned policy is inert until it has coverage. With $n_a$ the training count on arm $a$:
\begin{equation}
\text{active} \iff \textstyle\sum_a n_a \ge 300\ \wedge\ |\{a : n_a \ge 30\}| \ge 2
\end{equation}
and its decision considers only well-sampled arms, with an optional optimism bonus held at zero in
shadow:
\begin{equation}
\pi^*(x) = \operatorname*{arg\,max}_{a\,:\,n_a \ge 30} \left[ r_a(x) + c/\sqrt{n_a+1} \right],\quad c = 0
\end{equation}
Below the bar, the candidate is exactly the prior (5); above it, the bandit overrides. The admin
report states which regime is live --- \emph{warming up (heuristic prior)} vs \emph{bandit driving}
--- with per-arm counts, so there is never ambiguity about what produced a shadow pick.

\subsection{Forgetting: resetting an arm when its model changes}
An arm's weights encode ``what reward this rung earns on requests like $x$'' under a \emph{specific}
underlying model at a \emph{specific} price. Two events break that premise, and they break it
differently. A \textbf{price change} (same model, cheaper) is largely self-correcting: the cost term
in eq.~(6) is recomputed at backfill time from the actual ledger rows, so post-change turns already
carry the new economics into $R$ --- only the fitted $w_a$ lags, and because the update (8) is
\emph{constant}-step-size SGD (not a decaying rate that would freeze), it never stops tracking; its
effective memory is ${\sim}1/\eta \approx 20$ samples \emph{on that arm}. The uncomfortable case is a
\textbf{model swap behind a fixed rung id} (Sonnet 5 $\rightarrow$ 6, or a repointed pool): the arm
keeps its weights \emph{and} its large $n$, so its history is not merely stale but actively wrong,
and the accumulated count suppresses the very re-exploration that would fix it.

We make that a first-class, operator-free event with an \textbf{economics fingerprint} per rung:
\begin{equation}
f_a = \mathrm{sha256}(\text{model-id} \parallel \text{base-url} \parallel \text{price}_\text{in} \parallel \text{price}_\text{out})
\end{equation}
The fingerprint is stored beside each arm's weights and re-checked on every backfill. When it flips
--- any of model, endpoint, or price changed --- that arm is \textbf{reset} ($w_a \leftarrow 0$,
$n_a \leftarrow 0$) and re-stamped, so it drops back through the difficulty prior (5) and must
re-earn its way past the activation bar (9) on \emph{current} data. The reset is surgical --- only
the changed rung forgets; well-sampled neighbors keep their history. An operator ``reset arm''
control does the same on demand, for a knowing swap you don't want to wait on. This is deliberately
hard forgetting rather than a global decay factor: a decay rate slow enough to preserve a stable
rung's hard-won signal is too slow to erase a swapped model's, whereas the fingerprint erases exactly
the arm that changed, exactly when it changes.

\begin{figure*}[t]
\centering
\begin{tikzpicture}[x=1mm,y=1mm]
  \node[bx,minimum width=26mm,minimum height=14mm] (phi) at (0,0)
    {$\phi(x) \in \mathbb{R}^{11}$\\[2pt]{\color{mut}\tiny content-free counts + flags}};
  \foreach \name/\y [count=\i] in {gemini/24, fugu/12, sonnet/0, opus/-12, ultra/-24} {
    \node[bx,minimum width=34mm,minimum height=8mm,anchor=west] (h\i) at (30,\y)
      {$w_{\text{\name}} \cdot x$\\[1pt]{\color{mut}\tiny eligible iff $n \ge 30$}};
    \draw[fl] (phi.east) -- (h\i.west);
  }
  \node[bxa,minimum width=26mm,minimum height=12mm,anchor=west] (am) at (76,0)
    {$\arg\max r_a(x)$\\[2pt]{\color{mut}\tiny eligible arms only}};
  \foreach \i in {1,...,5} { \draw[fl] (h\i.east) -- (am.west); }
\end{tikzpicture}
\caption{\textbf{The policy network, such as it is.} One shared 11-dimensional feature vector feeds
five independent linear heads --- one per rung, in ladder order --- and the policy is an argmax over
the heads that have cleared 30 training turns. 55 parameters total: small enough to read, large
enough to beat a text-only judge on signals the judge cannot see.}
\label{fig:heads}
\end{figure*}

\begin{figure}[t]
\centering
\begin{tikzpicture}[x=1mm,y=1mm]
  % 0..60 turns -> 0..60mm
  \draw[warn,dashed] (30,-2) -- (30,30);
  \node[font=\tiny,color=warn] at (30,-4.5) {$n = 30$ (eligibility)};
  \foreach \name/\n/\c/\y in {gemini/55/r5/25, fugu/42/r4/19, sonnet/30/r3/13, opus/11/r2/7, f-ultra/4/r1/1} {
    \fill[\c] (0,\y) rectangle ++(\n,4.5);
    \node[font=\tiny,anchor=east] at (-1.5,\y+2.2) {\texttt{\name}};
    \node[font=\tiny,anchor=west] at (\n+1.5,\y+2.2) {\n};
  }
  \node[font=\tiny,color=mut,anchor=west] at (0,33) {training turns per arm (illustrative)};
\end{tikzpicture}
\caption{\textbf{The activation bar in practice (illustrative counts).} Traffic is skewed cheap, so
bottom rungs cross the per-arm threshold first. In this state eq.~(9) is \emph{not} yet met
($\sum n_a = 142 < 300$, still warming up); once the total clears 300 with $\ge 2$ eligible arms, the
bandit starts driving the shadow pick --- and only the eligible arms compete in the argmax, so a
barely-sampled top rung can't be chosen on noise.}
\label{fig:activation}
\end{figure}

\section{Serving: earning traffic $\varepsilon$-greedily under a ceiling}
Shadow evaluation is necessary but not sufficient: direct-method estimates share the reward model's
bias [4]. The graduation path therefore serves the learned policy on a small, configured slice of
live traffic and \emph{measures} the comparison. With \code{pct} $\in [0,100]$ an operator flag
(default 0) and $a_\text{ceil}$ a rung ceiling:
\begin{equation}
a_\text{served} =
\begin{cases}
\Pi_\C\!\left[\min(\pi^*(x),\, a_\text{ceil})\right] & \text{w.p. } \varepsilon,\ \text{if active}\\[2pt]
\Pi_\C\!\left[g(\text{text})\right] & \text{otherwise}
\end{cases}
\end{equation}
Three properties make this safe to turn on. It is a \textbf{hard no-op below the data bar} ---
deploying the code changes nothing until an admin moves the flag. The \textbf{ceiling bounds the
blast radius}: an over-eager policy cannot jump traffic to the top rung while trust is being built.
And the platform's existing self-healing sits underneath: if a served rung is unavailable or
unfunded, the standard fallback ladder catches it, exactly as for judge picks.

\subsection{The admissible set $\C$, and what else moves the served rung}
Neither policy chooses from the bare ladder. A deterministic pre-filter --- evaluated \emph{before}
the judge grades or the bandit picks --- narrows $\A$ to an admissible set $\C$, and $\Pi_\C$ in
eq.~(12) snaps a pick that fell outside $\C$ onto the nearest admissible rung (preferring the rung
\emph{above} on ties, since availability beats a latency promise). $\C$ is the intersection of three
things.

\textbf{Per-request constraint rails.} A caller may demand a latency bound or a floor rung via
request headers; the latency filter drops a rung only on \emph{probe evidence} that it is slower than
the bound (an arm with no measured latency yet is kept --- unknown is not slow). An unsatisfiable
constraint set is discarded entirely rather than routing to nothing: availability beats the promise.

\textbf{Plan tier.} A tenant's plan defines which rungs its turns may use at all; past the free
tier's premium budget the arm set narrows to the fast rungs rather than switching the agent off. The
judge, the bandit and the shadow log all carry on against the smaller set.

\textbf{Rung availability.} An unfunded or unreachable rung is snapped up to the next available one,
and any turn carrying tool definitions is routed off the bottom rung outright --- the Gemini wire
format cannot round-trip a multi-turn tool conversation in the OpenAI shape our agents speak.

These rails are constraints, not learning: they are deliberately outside the reward loop. But they
shape the data it sees, and the paper's own numbers inherit that (\S8). When the bandit's pick
survives to be served, the serving path also records the policy's \textbf{confidence}, a logistic
squash of the gap between the best and second-best predicted value among eligible arms, so the admin
report can slice measured outcomes by how sure the policy was.

Turns the bandit serves are tagged (\code{live\_why='bandit'}), which closes the loop twice over:
their rewards are \emph{measured} rather than modeled, and those measured rewards train the policy on
its own decisions --- the system's first on-policy data. The admin report then renders an A/B block
comparing bandit-served vs judge-served reward, success and cost head-to-head; widening $\varepsilon$
is justified by that measured table, not by counterfactual estimates.

\begin{figure}[t]
\centering
\begin{tikzpicture}[x=1mm,y=1mm]
  \node[bx,minimum width=20mm] (all) at (0,0) {graded turns\\[1pt]{\color{mut}\tiny 100\%}};
  \node[bx,minimum width=22mm,anchor=west] (judge) at (26,12) {judge $g$(text)\\[1pt]{\color{mut}\tiny status quo}};
  \node[bxa,minimum width=22mm,anchor=west] (band) at (26,-12) {bandit $\pi^*(x)$\\[1pt]{\color{mut}\tiny only if active, eq. (9)}};
  \node[bx,minimum width=20mm,anchor=west] (clamp) at (54,-12) {ceiling, then $\Pi_\C$};
  \node[bx,minimum width=16mm,anchor=west] (srv) at (54,12) {serve\\[1pt]{\color{mut}\tiny + fallback net}};
  \draw[flm,line width=1.6pt] (all.east) -- node[above,font=\tiny,pos=.4]{$1-\varepsilon$} (judge.west);
  \draw[fl] (all.east) -- node[below,font=\tiny,pos=.4]{$\varepsilon$} (band.west);
  \draw[fl] (band) -- (clamp);
  \draw[fl] (clamp.north) -- (srv.south);
  \draw[flm] (judge) -- (srv);
  \node[font=\tiny,color=acc,anchor=north] at (40,-22) {bandit turns tagged \code{live\_why='bandit'}};
\end{tikzpicture}
\caption{\textbf{$\varepsilon$-greedy serving, eq.~(12).} The bandit path only exists while the
policy is active, its pick is clamped first by the ceiling flag and then onto the admissible set
$\C$ of \S6.1, and every rung served --- from either path --- lands on the same self-healing fallback
ladder. Tagged bandit turns feed the measured on-policy A/B.}
\label{fig:serving}
\end{figure}

\section{Evaluation methodology}
The admin report backfills rewards on demand and renders, in order of how much we trust each number.

\textbf{Agreement and confusion.} Judge-vs-candidate agreement rate and the full $5\times5$ confusion
matrix (Fig.~\ref{fig:confusion}). Disagreement mass concentrated one rung \emph{above} the judge on
turns that later escalated is the signature of the failure mode we built this to catch.

\textbf{Escalation catch-rate.} Among turns the judge under-routed badly enough to trip eq.~(3), the
fraction where the candidate had started higher. This is a \emph{non-counterfactual} win: the
escalation actually happened, in production, under the judge.

\textbf{Reward on the agreement subset.} Realized mean reward where both policies chose the same rung
--- a sanity check that the reward signal itself is stable before any delta between policies is read.

\textbf{Counterfactual cost estimate --- labeled an estimate.} Where picks diverge, the candidate's
rung was never served; its cost is modeled from that rung's realized per-turn mean. This is the
direct method of [4] and inherits its bias; the report says so in the UI, and the number is treated
as a screen, not a verdict. The verdict comes from the measured on-policy A/B of \S6.

\begin{figure}[t]
\centering
\begin{tikzpicture}[x=8mm,y=6mm]
  \foreach \l [count=\i from 0] in {gem,fugu,son,opus,ultra} {
    \node[font=\tiny,color=mut] at (\i+0.5,5.4) {\l};
    \node[font=\tiny,color=mut,anchor=east] at (-0.1,4.5-\i) {\l};
  }
  % rows = judge pick, cols = candidate pick (illustrative)
  \foreach \r/\c/\v/\f in {%
    0/0/61/r5, 0/1/1/r1, 0/2/7/r3,
    1/1/6/r3, 1/2/1/r1, 1/3/2/r1,
    2/0/2/r1, 2/1/4/r2, 2/2/12/r4,
    3/1/1/r1, 3/3/2/r2,
    4/3/1/r1} {
    \fill[\f] (\c,4-\r) rectangle ++(0.94,0.9);
    \node[font=\tiny] at (\c+0.47,4.45-\r) {\v\%};
  }
  \node[font=\tiny,color=mut,anchor=west] at (0,-0.9) {rows: judge pick \quad cols: candidate pick};
\end{tikzpicture}
\caption{\textbf{The confusion matrix the report renders (illustrative shape).} Mass on the diagonal
is agreement. The cells to watch are just above the diagonal: turns where the candidate starts one
rung higher than the judge. When those correlate with the judge's own mid-turn escalations, the
candidate is catching under-routing before it happens --- the exact failure mode of \S\ref{sec:live}.}
\label{fig:confusion}
\end{figure}

\section{Privacy and safety properties}
The routing system stores, per decision: a timestamp, the tenant id, the two model picks and why the
served one was chosen, the raw feature counts behind eq.~(4) --- they are scaled into the unit-range
vector at training time, not at write time --- a turn key, and the outcome scalars of eq.~(6). It
stores no message text, and the features are deliberately non-invertible summaries (counts, capped
ratios, binary flags).

The turn key deserves naming rather than burying: it is a truncated SHA-256 of the user message, used
to tell one turn from the next and to bound reward attribution. It is neither plaintext nor
reversible, but it is \emph{content-derived}, so it admits confirmation of a guess --- someone
holding both the database and a candidate message can test whether that exact message was sent. We
state that rather than claim more than the mechanism delivers; a per-tenant keyed digest would close
it and costs nothing but a migration.

Users who pin a model in settings short-circuit the resolver before either the judge or the bandit is
consulted; BYO-key boxes never enter the managed path; utility calls stay pinned to the cheapest
funded rung. The learner is fleet-shared --- every tenant's turns pool into one policy --- which is
both the cold-start solution (a single tenant would never clear eq.~(9)) and a compounding effect: a
lesson extracted from one tenant's mis-routed turn immediately improves routing for all.

The honest caveat is representation skew, and it has two sources. Early on, the policy's opinions are
dominated by the most active tenants. Structurally, the plan-tier arm sets of \S6.1 mean cheap-tier
traffic can only ever train the fast rungs, so the upper arms are fed exclusively by the tenants
entitled to them --- the coverage in eq.~(9) is not merely slow to arrive on those rungs, it is drawn
from a narrower population. The serving guardrails (\S6) carry both risks until the data broadens.
All of it --- shadow recording, training, serving fraction, ceiling --- is governed by runtime
configuration, so every stage is one flag from off.

\section{Limitations and future work}
\textbf{Direct-method bias.} Without logging propensities, counterfactual value estimates lean
entirely on the reward model; we mitigate by treating them as screens and promoting only on measured
on-policy comparisons, but a stochastic logging policy (even $\varepsilon = 0.02$ of exploration)
would unlock doubly-robust estimation [4].

\textbf{Reward shaping.} Eq.~(6)'s weights encode a judgment (a cent of spend is worth half a
success). The report accepts weight overrides so sensitivity can be checked, but a principled
calibration against retention or task-completion telemetry is future work.

\textbf{Feature ceiling.} Eleven linear features cannot represent interactions (a traceback in a
\emph{long} message means something different). The frozen-order vector design admits appending; the
estimator admits replacement by any regressor without touching the recording, reward, or serving
layers.

\textbf{Non-stationarity.} Discrete shifts --- a model swapped behind a rung, a reprice --- are
handled by the fingerprint reset of \S5.4, and constant-step-size SGD tracks gradual drift within a
fixed model. What remains unaddressed is \emph{slow} drift that never trips the fingerprint (a
model's behavior changing under a stable id and price): the constant learning rate tracks it, but
with a lag inversely proportional to that arm's traffic, so a rarely-served top rung adapts slowest.
A recency-weighted or windowed fit is the natural upgrade, warranted only once such drift is
measurable.

\textbf{Beyond routing.} The ledger $\rightarrow$ reward $\rightarrow$ per-arm learner $\rightarrow$
flag-gated serving pipeline is decision-agnostic. Candidate next users on the platform: which memory
facts to inject into a turn, when to sleep an idle box, and voice selection.

\section{Conclusion}
A production LLM platform can grow a learned router the way it would grow any risky capability:
behind observation first, a data bar second, and a bounded, measured serving fraction third. The
machinery required is small --- one ledger table, 55 linear parameters, three config flags --- and
every piece of it is legible to the operator who must trust it. The judge stays as the prior and the
safety net; the bandit earns each increment of traffic with measured wins. Nothing about the recipe
is specific to routing, which is rather the point.

\section*{References}
\footnotesize
\begin{list}{}{\leftmargin=1.2em\itemindent=-1.2em\itemsep=2pt\parsep=0pt}
\small
\item [1] W.~R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. \emph{Biometrika}, 25(3--4):285--294, 1933.
\item [2] P.~Auer, N.~Cesa-Bianchi, P.~Fischer. Finite-time analysis of the multiarmed bandit problem. \emph{Machine Learning}, 47:235--256, 2002.
\item [3] L.~Li, W.~Chu, J.~Langford, R.~E. Schapire. A contextual-bandit approach to personalized news article recommendation. \emph{WWW}, 2010.
\item [4] M.~Dud\'ik, J.~Langford, L.~Li. Doubly robust policy evaluation and learning. \emph{ICML}, 2011.
\item [5] L.~Bottou, J.~Peters, J.~Qui\~nonero-Candela, et al. Counterfactual reasoning and learning systems: the example of computational advertising. \emph{JMLR}, 14:3207--3260, 2013.
\item [6] A.~Agarwal, S.~Bird, M.~Cozowicz, et al. Making contextual decisions with low technical debt. \emph{arXiv:1606.03966}, 2016.
\item [7] D.~Russo, B.~Van Roy, A.~Kazerouni, I.~Osband, Z.~Wen. A tutorial on Thompson sampling. \emph{Foundations and Trends in Machine Learning}, 11(1):1--96, 2018.
\item [8] R.~S. Sutton, A.~G. Barto. \emph{Reinforcement Learning: An Introduction}, 2nd ed. MIT Press, 2018.
\item [9] A.~Bietti, A.~Agarwal, J.~Langford. A contextual bandit bake-off. \emph{JMLR}, 22(133):1--49, 2021.
\item [10] L.~Chen, M.~Zaharia, J.~Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. \emph{arXiv:2305.05176}, 2023.
\item [11] I.~Ong, A.~Almahairi, V.~Wu, W.-L. Chiang, et al. RouteLLM: Learning to route LLMs with preference data. \emph{arXiv:2406.18665}, 2024.
\end{list}

\section*{Errata --- Revision V4}
\footnotesize
V4 came out of an audit of this report against the deployed code. Two of its items correct claims
that were \emph{wrong}, including one that V3 introduced; the rest close the gap that opened as the
system shipped past the paper. The design is unchanged.
\begin{enumerate}[leftmargin=1.3em,itemsep=2pt]
\small
\item \textbf{Model-ladder ordering --- retraction of V3 item 1.} V3 swapped \rung{fugu} and
\rung{sonnet} in eq.~(1) ``to follow the price hierarchy of Fig.~\ref{fig:price}.'' That was a
mistake: the ladder is ordered by capability tier, and price is not monotone along it --- \rung{fugu}
is rung 1 at \$20/M output, above rung 2 \rung{sonnet} at \$15/M. The deployed ladder is
$\langle$gemini, fugu, sonnet, opus, f-ultra$\rangle$ and always was. Eq.~(1), the per-arm heads of
Fig.~\ref{fig:heads}, the activation rows of Fig.~\ref{fig:activation} and both axes of
Fig.~\ref{fig:confusion} are restored to it; Fig.~\ref{fig:price}'s caption now says explicitly that
its bars are price-sorted, not ladder-sorted.
\item \textbf{Source of truth.} \code{\_resolve\_model} lives in \code{control-plane/llm\_gateway.py},
not \code{main.py}; the closing note was pointing at a file the router had already left.
\item \textbf{Per-decision storage (\S8).} The enumeration omitted the turn key --- a truncated
SHA-256 of the user message --- and described the stored features as the scaled vector of eq.~(4)
when what is written is the raw counts, scaled later at training time. Both are corrected, and the
guess-confirmation property of a content-derived hash is now stated instead of implied away.
\item \textbf{The serving path grew (\S6.1, new).} Eq.~(12) described a ceiling and nothing else.
Deployed serving also applies per-request constraint rails (a latency bound and a floor rung,
filtered on probe evidence), a plan-tier arm set, availability snapping, and a hard reroute of
tool-bearing turns off the bottom rung; the bandit's pick now also carries a confidence score.
Eq.~(12) gains the admissible-set projection $\Pi_\C$, and \S8's skew caveat gains the structural
half that the tier arm sets create.
\item \textbf{Escalation and notation.} Eq.~(3) said the valve bumps one index; it bumps to the next
\emph{available} rung, which may skip an unfunded one. The ceiling in eq.~(12) is renamed
$a_\text{ceil}$, since $a_\text{max}$ was doing duty for both it and the top of the ladder.
\item \textbf{Training attribution (\S5.2).} ``The arm the live judge served'' was true only in
shadow. Training keys on the arm actually served, whichever policy chose it --- which is precisely
what makes the on-policy claim in \S6 hold once $\varepsilon > 0$.
\end{enumerate}
One claim in \S1 --- that a majority of real turns are handled by the cheapest rung --- is a
statement about production traffic rather than about code, and was not re-verified in this audit.

\subsection*{Errata --- Revision V3 (retained; item 1 withdrawn by V4)}
\begin{enumerate}[leftmargin=1.3em,itemsep=2pt]
\small
\item \textbf{Model-ladder ordering.} Corrected the ladder in eq.~(1) and the confusion matrix to
follow the price hierarchy of Fig.~\ref{fig:price}, swapping the positions of \rung{fugu} and
\rung{sonnet}. \emph{[Withdrawn --- see V4 item 1.]}
\item \textbf{Loss function.} Aligned the \S5.2 objective with the gradient step of eq.~(8): the
ridge term is $\tfrac{1}{2}\lambda\|w\|_2^2$.
\item \textbf{Activation caption.} Corrected the logic --- with $\sum n_a = 142 < 300$ the activation
bar of eq.~(9) is \emph{not} met, so the system is still warming up.
\item \textbf{Typographical artifacts.} Fixed a rendering artifact in eq.~(3) and a mislabeled
fraction in the serving figure (now $\varepsilon = \text{pct}/100$).
\end{enumerate}

\vspace{6pt}
\noindent{\small\color{mut}Companion reading: the narrative version of this system lives in the Avriz
engineering notebook at \url{https://avriz.io/eng} (\S07 ``Model routing'' and \S08 ``The learning
flywheel''). Source of truth: \code{control-plane/rl\_router.py} (features, reward, bandit,
constraint rails) and \code{\_resolve\_model} in \code{control-plane/llm\_gateway.py} (judge, ladder,
escalation, $\varepsilon$-greedy serving).}

\end{document}
