Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.3 on gold-v0.1

run_20260927T072623Z_ad6f44 · n 100 · auto_process 0.81 (chosen on gen-v0.3-dev; recorded at 0.95)

questions: q-v0.3 →

Gold is not a blind test for q-v0.3: the question set was motivated by GOLD-TMP-17.

Correct action
94.0%
94/100
95% CI 87.4–97.8%
Automation
31.0%
31/100
95% CI 22.1–41.0%
Unsafe / auto
3.2%
1/31
95% CI 0.1–16.7%
Cost / case
$0.000175
$0.0175 total · sync
from the spend ledger

Cost: Jev spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Jev q-v0.3 on gold-v0.1Operating point 0.81 (chosen on gen-v0.3-dev): 31 of 100 cases auto-processed, 1 unsafe. The first unsafe automation appears at 0.83 (31 auto-processed, 1 unsafe); stricter thresholds have none.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/1000—64/100
0.980/1000—64/100
0.970/1000—64/100
0.960/1000—64/100
0.951/10000.0%65/100
0.946/10000.0%70/100
0.9316/10000.0%80/100
0.9219/10000.0%83/100
0.9122/10000.0%86/100
0.9023/10000.0%87/100
0.8924/10000.0%88/100
0.8825/10000.0%89/100
0.8725/10000.0%89/100
0.8628/10000.0%92/100
0.8528/10000.0%92/100
0.8429/10000.0%93/100
0.8331/10013.2%94/100
0.8231/10013.2%94/100
0.8131/10013.2%94/100
0.8031/10013.2%94/100
0.7931/10013.2%94/100
0.7831/10013.2%94/100
0.7731/10013.2%94/100
0.7631/10013.2%94/100
0.7531/10013.2%94/100
0.7431/10013.2%94/100
0.7331/10013.2%94/100
0.7231/10013.2%94/100
0.7131/10013.2%94/100
0.7031/10013.2%94/100
0.6931/10013.2%94/100
0.6831/10013.2%94/100
0.6731/10013.2%94/100
0.6631/10013.2%94/100
0.6532/10013.1%95/100
0.6432/10013.1%95/100
0.6332/10013.1%95/100
0.6232/10013.1%95/100
0.6132/10013.1%95/100
0.6032/10013.1%95/100
0.5932/10013.1%95/100
0.5832/10013.1%95/100
0.5732/10013.1%95/100
0.5632/10013.1%95/100
0.5532/10013.1%95/100
0.5432/10013.1%95/100
0.5332/10013.1%95/100
0.5232/10013.1%95/100
0.5132/10013.1%95/100
0.5032/10013.1%95/100
Request info and human review· 2 rates
Request info
32.0%
32/100
95% CI 23.0–42.1%
Human review
37.0%
37/100
95% CI 27.6–47.2%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.010 · ECE 0.048 · n 100
Reliability of Diagnosis supported: accuracy against stated confidence112294
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.5600.0%
0.6–0.710.6800.0%
0.7–0.820.780100.0%
0.8–0.920.875100.0%
0.9–1.0940.970100.0%
Step therapy metBrier 0.067 · ECE 0.044 · n 100
Reliability of Step therapy met: accuracy against stated confidence175978
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.5350.0%
0.6–0.770.66342.9%
0.7–0.850.73380.0%
0.8–0.990.86677.8%
0.9–1.0780.96197.4%
Documentation completeBrier 0.063 · ECE 0.037 · n 100
Reliability of Documentation complete: accuracy against stated confidence513883
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.650.54660.0%
0.6–0.710.650100.0%
0.7–0.830.74333.3%
0.8–0.980.85087.5%
0.9–1.0830.94496.4%
Material contradictionBrier 0.041 · ECE 0.094 · n 100
Reliability of Material contradiction: accuracy against stated confidence2343160
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.620.5050.0%
0.6–0.730.65366.7%
0.7–0.840.76250.0%
0.8–0.9310.86596.8%
0.9–1.0600.931100.0%
Missing evidenceBrier 0.117 · ECE 0.090 · n 100x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence17591662
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.510.4900.0%
0.5–0.670.53914.3%
0.6–0.750.66080.0%
0.7–0.890.746100.0%
0.8–0.9160.84487.5%
0.9–1.0620.964100.0%
All runs on gold-v0.1· 5 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.8910091/100 (91.0%)83.6–95.8%29/100 (29.0%)1/29 (3.4%)0.1–17.8%$0.000115$0.0115
Jev q-v0.30.8110094/100 (94.0%)87.4–97.8%31/100 (31.0%)1/31 (3.2%)0.1–16.7%$0.000175$0.0175
Claude0.5510093/100 (93.0%)86.1–97.1%30/100 (30.0%)1/30 (3.3%)0.1–17.2%$0.0156$1.56
Rules0.9910061/100 (61.0%)50.7–70.6%20/100 (20.0%)6/20 (30.0%)11.9–54.3%$0$0
Ground truth0.95100100/100 (100.0%)96.4–100.0%34/100 (34.0%)0/34 (0.0%)0.0–10.3%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →