Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.2 on gen-v0.2-holdout

run_20260925T075242Z_fd455f · n 1000 · auto_process 0.89 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 →

Correct action
89.5%
895/1000
95% CI 87.4–91.3%
Automation
25.2%
252/1000
95% CI 22.5–28.0%
Unsafe / auto
0.0%
0/252
95% CI 0.0–1.5%
Cost / case
$0.000112
$0.112 total · sync
trace estimate

Cost: Sum of the traces' estimated cost; not in a committed ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Jev q-v0.2 on gen-v0.2-holdoutOperating point 0.89 (chosen on gen-v0.2-dev): 252 of 1000 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/10000—643/1000
0.980/10000—643/1000
0.9737/100000.0%680/1000
0.96105/100000.0%748/1000
0.95172/100000.0%815/1000
0.94220/100000.0%863/1000
0.93242/100000.0%885/1000
0.92251/100000.0%894/1000
0.91252/100000.0%895/1000
0.90252/100000.0%895/1000
0.89252/100000.0%895/1000
0.88252/100000.0%895/1000
0.87252/100000.0%895/1000
0.86252/100000.0%895/1000
0.85252/100000.0%895/1000
0.84252/100000.0%895/1000
0.83252/100000.0%895/1000
0.82252/100000.0%895/1000
0.81252/100000.0%895/1000
0.80252/100000.0%895/1000
0.79252/100000.0%895/1000
0.78252/100000.0%895/1000
0.77252/100000.0%895/1000
0.76252/100000.0%895/1000
0.75252/100000.0%895/1000
0.74252/100000.0%895/1000
0.73252/100000.0%895/1000
0.72252/100000.0%895/1000
0.71252/100000.0%895/1000
0.70252/100000.0%895/1000
0.69252/100000.0%895/1000
0.68252/100000.0%895/1000
0.67252/100000.0%895/1000
0.66252/100000.0%895/1000
0.65252/100000.0%895/1000
0.64252/100000.0%895/1000
0.63252/100000.0%895/1000
0.62252/100000.0%895/1000
0.61252/100000.0%895/1000
0.60252/100000.0%895/1000
0.59252/100000.0%895/1000
0.58252/100000.0%895/1000
0.57252/100000.0%895/1000
0.56252/100000.0%895/1000
0.55252/100000.0%895/1000
0.54252/100000.0%895/1000
0.53252/100000.0%895/1000
0.52252/100000.0%895/1000
0.51252/100000.0%895/1000
0.50252/100000.0%895/1000
Request info and human review· 2 rates
Request info
31.3%
313/1000
95% CI 28.4–34.3%
Human review
43.5%
435/1000
95% CI 40.4–46.6%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.029 · n 1000
Reliability of Diagnosis supported: accuracy against stated confidence18991
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.810.760100.0%
0.8–0.980.850100.0%
0.9–1.09910.972100.0%
Step therapy metBrier 0.011 · ECE 0.022 · n 1000
Reliability of Step therapy met: accuracy against stated confidence1917819937
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6190.55052.6%
0.6–0.7170.64358.8%
0.7–0.880.755100.0%
0.8–0.9190.86394.7%
0.9–1.09370.982100.0%
Documentation completeBrier 0.032 · ECE 0.075 · n 1000
Reliability of Documentation complete: accuracy against stated confidence37351816894
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6370.53940.5%
0.6–0.7350.64331.4%
0.7–0.8180.73533.3%
0.8–0.9160.87393.8%
0.9–1.08940.943100.0%
Material contradictionBrier 0.030 · ECE 0.121 · n 1000
Reliability of Material contradiction: accuracy against stated confidence364066384474
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6360.55180.6%
0.6–0.7400.63387.5%
0.7–0.8660.75297.0%
0.8–0.93840.862100.0%
0.9–1.04740.927100.0%
Missing evidenceBrier 0.212 · ECE 0.078 · n 1000x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence138892121233453
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.5130.48261.5%
0.5–0.6880.55238.6%
0.6–0.7920.64942.4%
0.7–0.81210.74363.6%
0.8–0.92330.85391.8%
0.9–1.04530.96399.1%
All runs on gen-v0.2-holdout· 3 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.891000895/1000 (89.5%)87.4–91.3%252/1000 (25.2%)0/252 (0.0%)0.0–1.5%$0.000112$0.112
Claude (150-case sample)0.55150147/150 (98.0%)94.3–99.6%37/150 (24.7%)0/37 (0.0%)0.0–9.5%$0.0164$2.46
Rules0.991000667/1000 (66.7%)63.7–69.6%134/1000 (13.4%)0/134 (0.0%)0.0–2.7%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →