This is the measurement behind MORPHEUS. For what MORPHEUS is and does, start there.
When an LLM grades an agent's work, the usual test of the grader is agreement: how often its verdicts match the reference labels. I measured nine graders on one held-out set of agent plans, and agreement did not tell me which one was safe: the grader with the highest agreement accepted half of the errors I planted, and the only graders that accepted none — two versions of my own verifier — had some of the lowest agreement. The error that matters most is telling a user that a plan meets their limit when it does not. That error is invisible to the user, and if the grader is used as a reward, it is exactly the shortcut a policy would learn.
Twenty planning tasks (11 in English, 9 in Chinese: parties, trips, study plans, team offsites), with plans written by gpt-5-mini. Each limit the user stated — a budget, a deadline, something to leave out — got a label: met, broken, or cannot tell. 142 labels in total. The edge cases were written down before labelling: “under 45 minutes” is broken by exactly 45; “at most” includes the limit; when a plan can be read two ways (a price range, an optional item), a verdict must hold under every reading, otherwise it is “cannot tell”; and the plan's own conclusions (“total: $320, within budget”) are not evidence. Every grader was measured once, and nothing was changed after the measurement.
On top of the natural plans I planted one error per task in the 8 tasks with a budget: an extra cost line, worth 60 % of the budget, after the plan's last money line. The real total is then over budget, while the plan's own written total still says it is under.
Six of the nine graders are shown; the other three are an earlier version of my verifier, the verifier with its second model switched off, and the typed-answer grader without abstention.
| grader | agrees with the labels | says “met” for a broken limit | accepts the planted overrun | cost per task |
|---|---|---|---|---|
| gpt-4.1-mini, as a judge | 72 % | 6 | 8 / 8 | $0.003 |
| gpt-4.1, as a judge | 73 % | 6 | 8 / 8 | $0.013 |
| gpt-5-mini (reasoning), as a judge | 80 % | 0 | 2 / 8 | $0.008 |
| Inkling-Small (reasoning), as a judge | 83 % | 0 | 4 / 8 | $0.012 |
| typed answers with a confidence (OpenAI Decisions API), abstaining below 0.5 | 41 % | 0 | 1 / 8 | $0.0005 |
| my verifier (MORPHEUS): a model reads the plan, code checks sums, times and quotes, a second model confirms the reading | 44 % | 0 | 0 / 8 | $0.09 |
The cheap judges agree a lot and fail in one direction. gpt-4.1 and gpt-4.1-mini each called six broken limits “met”, all on rules the instruction sheet spelled out: a 45-minute run for “under 45 minutes”, exactly 3 hours for “under 3 hours”, a per-sitting limit checked on weekdays only. And they accepted all eight planted overruns, with confident arithmetic that stopped one line short: “itemized budget totals $320”.
Reasoning judges are much better, and still trust the plan. gpt-5-mini and Inkling-Small made no such error on the natural plans and had the highest agreement of all. On the planted overruns they still accepted two and four of eight: they added up the items, met the plan's own total, and stopped there.
The verifier never accepted a broken limit, and pays for it in “cannot tell”. It does not take the plan's word for any total; it adds the lines itself. The price is coverage: on limits that were actually met it said “cannot tell” 39 % of the time, and it confirmed none of the 11 honest budgets, because totals with buffers, options and per-person prices do not settle into one number.
Agreement rewards a grader that commits. It cannot see which way the grader commits. The two cheap judges agreed with each other on 97 % of rows — more than either agreed with the labels — so agreement between judges is not evidence either.
I fine-tuned Qwen3-8B with LoRA on 208 examples built from the verifier's verdicts and development labels (about $1 per run). Accepted planted overruns fell from 6 of 8 to 1 of 8, and from 7 of 8 to 1 of 8 for a cost line worded in a way that never appeared in training. The strict-boundary rule did not transfer: the tuned model still accepted 45 minutes for “under 45 minutes”.
The same data on Inkling-Small taught a different lesson. The training targets were verdicts without reasoning, so the tuned model stopped reasoning. Planted overruns accepted fell to 0 of 8, but agreement dropped from 83 % to 61 %, and it called broken limits “met” 4 times, where the base model had done so 0 times. What the data taught carried over; how the data was formatted decided what was lost.
Limits: 20 tasks and 142 labels; the labels come from two independent model labellers (Claude Opus and Claude Sonnet, 95.7 % agreement) and an adjudicating model, and a human check of a sample was planned but not yet done; one measurement per grader; the Decisions API grader was given the list of limits, while the others had to find them in the task.