Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Claude on gold-v0.1

run_20260926T011730Z_f1852f · n 100 · auto_process 0.55 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 + claude-prompt-v1 →

Claude's probabilities are self-reported; gold labels were written by Claude agents.

Correct action
93.0%
93/100
95% CI 86.1–97.1%
Automation
30.0%
30/100
95% CI 21.2–40.0%
Unsafe / auto
3.3%
1/30
95% CI 0.1–17.2%
Cost / case
$0.0156
$1.56 total · batch
from the spend ledger

Cost: Claude spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Claude on gold-v0.1Operating point 0.55 (chosen on gen-v0.2-dev): 30 of 100 cases auto-processed, 1 unsafe. The first unsafe automation appears at 0.55 (30 auto-processed, 1 unsafe); stricter thresholds have none.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/1000—65/100
0.980/1000—65/100
0.970/1000—65/100
0.960/1000—65/100
0.950/1000—65/100
0.940/1000—65/100
0.930/1000—65/100
0.920/1000—65/100
0.910/1000—65/100
0.900/1000—65/100
0.890/1000—65/100
0.880/1000—65/100
0.870/1000—65/100
0.860/1000—65/100
0.850/1000—65/100
0.840/1000—65/100
0.830/1000—65/100
0.820/1000—65/100
0.810/1000—65/100
0.800/1000—65/100
0.792/10000.0%67/100
0.782/10000.0%67/100
0.772/10000.0%67/100
0.762/10000.0%67/100
0.754/10000.0%69/100
0.747/10000.0%72/100
0.7310/10000.0%75/100
0.7212/10000.0%77/100
0.7113/10000.0%78/100
0.7016/10000.0%81/100
0.6917/10000.0%82/100
0.6817/10000.0%82/100
0.6720/10000.0%85/100
0.6623/10000.0%88/100
0.6525/10000.0%90/100
0.6425/10000.0%90/100
0.6326/10000.0%91/100
0.6228/10000.0%93/100
0.6128/10000.0%93/100
0.6028/10000.0%93/100
0.5928/10000.0%93/100
0.5828/10000.0%93/100
0.5728/10000.0%93/100
0.5629/10000.0%94/100
0.5530/10013.3%93/100
0.5431/10013.2%94/100
0.5331/10013.2%94/100
0.5231/10013.2%94/100
0.5131/10013.2%94/100
0.5031/10013.2%94/100
Request info and human review· 2 rates
Request info
33.0%
33/100
95% CI 23.9–43.1%
Human review
37.0%
37/100
95% CI 27.6–47.2%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.008 · ECE 0.049 · n 100
Reliability of Diagnosis supported: accuracy against stated confidence1198
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.5000.0%
0.6–0.710.6000.0%
0.7–0.80——
0.8–0.90——
0.9–1.0980.961100.0%
Step therapy metBrier 0.084 · ECE 0.152 · n 100
Reliability of Step therapy met: accuracy against stated confidence42925339
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.640.56250.0%
0.6–0.7290.66193.1%
0.7–0.8250.73696.0%
0.8–0.930.835100.0%
0.9–1.0391.00097.4%
Documentation completeBrier 0.046 · ECE 0.098 · n 100
Reliability of Documentation complete: accuracy against stated confidence125488
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.5000.0%
0.6–0.720.6000.0%
0.7–0.850.75040.0%
0.8–0.940.86550.0%
0.9–1.0880.944100.0%
Material contradictionBrier 0.007 · ECE 0.054 · n 100
Reliability of Material contradiction: accuracy against stated confidence13393
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.5000.0%
0.6–0.70——
0.7–0.830.717100.0%
0.8–0.930.850100.0%
0.9–1.0930.962100.0%
Missing evidenceBrier 0.076 · ECE 0.133 · n 100x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence118153837
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.510.4500.0%
0.5–0.610.510100.0%
0.6–0.780.63662.5%
0.7–0.8150.754100.0%
0.8–0.9380.861100.0%
0.9–1.0370.912100.0%
All runs on gold-v0.1· 5 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.8910091/100 (91.0%)83.6–95.8%29/100 (29.0%)1/29 (3.4%)0.1–17.8%$0.000115$0.0115
Jev q-v0.30.8110094/100 (94.0%)87.4–97.8%31/100 (31.0%)1/31 (3.2%)0.1–16.7%$0.000175$0.0175
Claude0.5510093/100 (93.0%)86.1–97.1%30/100 (30.0%)1/30 (3.3%)0.1–17.2%$0.0156$1.56
Rules0.9910061/100 (61.0%)50.7–70.6%20/100 (20.0%)6/20 (30.0%)11.9–54.3%$0$0
Ground truth0.95100100/100 (100.0%)96.4–100.0%34/100 (34.0%)0/34 (0.0%)0.0–10.3%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →