Notes · Field note ·
I tested whether a System One model could predict my code judge. It couldn't, yet.
Seven cases looked like a win; 178 said otherwise. What a typed-judgment model is good for in an agent chain, and what it isn't.
- to re-score 111 past runs
- $0.13
- issue-type labels correct, over 110
- 90.9%
- AUROC predicting the judge: a coin flip
- 0.49
Most checks in my agent chain are code: tests either pass or fail. Some questions code can't answer, like whether a worker's description of its change matches the diff it actually wrote. Until recently the only option was asking another LLM for an opinion, slowly.
TypeSafe's Jev is a different kind of model, which they call System One: you ask typed questions, it answers each with a probability, in one call. I wired it into the chain as an advisor. It never gates anything.
The question
My chain ends with a judge that decides whether work merges. If a cheap, fast model could predict the judge's call early, bad work would be caught sooner. On the first 7 cases, Jev looked promising: 60% precision, 75% recall. Seven cases isn't a result, so I widened the sample: every judged run with a live readout, plus 111 past runs re-scored from the diffs, contracts and findings as they stood before each verdict, so the answer couldn't leak in.
What 178 cases showed
| Sample | Cases | Judge agreed when Jev said "block" | Always guessing "block" | AUROC |
|---|---|---|---|---|
| Live readouts | 67 | 64.5% | 64.2% | 0.49 |
| Past runs, re-scored | 111 | 67.4% | 63.1% | 0.61 |
Jev rated nearly every piece of work above 97% likely to be blocked, so its precision sits on the base rate: it adds nothing over always guessing "block". AUROC, which measures whether a score separates good work from bad, came in at 0.49 on the live readouts, a coin flip. The re-scored runs did a little better at 0.61, but the uncertainty range still includes chance.
What it is good at
- Labeling: on a game project, its issue-type labels matched a human's 90.9% of the time, over 110 issues.
- Speed: 20 typed questions answered in 0.7 s on 16K input tokens.
- Cost: re-scoring 111 past runs took 129 calls and 3.1M input tokens, for 13 cents.
- Repeatability: sending the same request again moved its score by at most 1 point.
What I changed
Jev keeps the jobs it's good at, labeling and checking claims, and never votes on merges. The judge stays the judge.
Why publish a result that didn't work
Because 7 cases looked like a win and 178 showed it wasn't. The habit that caught it, rerunning a claim at scale before trusting it, matters more than any one model.
A small sample flatters. Count until the number stops moving, then decide.