Most model routers read the prompt and predict which model should answer it. I built Rung, which does the opposite: it sends the request to the cheapest model first, checks the answer, and moves up to a more expensive model only when the check says the answer is wrong. To the caller it is one OpenAI-compatible endpoint; every response also says which model answered, what each attempt cost, and whether anything checked it. These are the four things I learned building it.
Using it takes one line: start the gateway with your list of models and their prices, and point an existing
OpenAI client at it with the model name auto. Answers come back in the usual shape, plus a
router block listing each attempt, what it cost and whether it was verified. With no routing table of
your own yet, it simply climbs the price ladder.
I started where everyone starts: predict the best model from the request. I tried summaries of the task, the full prompt, TF-IDF features, neural embeddings and a fine-tuned small language model, at two data scales, with leave-one-repository-out splits and permutation baselines. None of it beat a policy that predicts nothing. Measured on 6,891 (task, configuration) cells from an agentic-coding corpus, prediction recovered none of the gain that was there to take. The information that says which model succeeds shows up when the model runs, not in the text of the request.
What did work is boring. Order the models by measured cost, try the cheapest, and climb only when an answer is shown to be wrong; the per-task budget decides how many attempts each rung gets. On HumanEval+ (164 tasks, 4 trials per cell), at matched spend, this ladder came out ahead of every fixed single model in 37 of 39 comparisons.
Across models, the cost per correct answer differed by 163×. Averaged over all tasks, the most expensive model looked barely worth its price. Split by kind of task, the picture changed: its advantage was 58 points on one category and 2 on another. So Rung routes sideways first — look up the task's category in a table of measured cost per success — and climbs the ladder second.
That table is built offline from each operator's own traffic and installed as a file, on purpose. A router that re-derives its routing from its own logs stops sending traffic to the models it has demoted, so their estimates freeze at whatever value demoted them, and it can never find out it was wrong.
A ladder is only as good as the thing that says “this answer is wrong, escalate”, and most users have no test suite. On 404 (task, model) cells I measured how well each kind of check tracked whether an answer was actually correct:
| check | correlation with “actually correct” |
|---|---|
| run the test suite | 0.85 |
| ask the same model 3 times and require it to agree with itself | 0.78–0.98 |
| ask a model of the same tier to grade the answer | negative: it preferred fluent fabrications |
Self-consistency costs about 3 % of one call to the top model and needs nothing from the user. It works because it never asks “is this right”, which requires knowing the answer, only “is the model steady here”, which the outputs already show. Three samples is a cliff, not a slope: with two, the correlation fell to zero.
Limits: the numbers come from HumanEval+ and one agentic-coding corpus; your traffic is different, which is why each operator builds their own routing table. Escalation adds latency, so this is not for interactive chat, and without any check the ladder has nothing to climb on.