Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.2 on gen-v0.3-holdout

run_20260927T072249Z_204814 · n 1000 · auto_process 0.97 (chosen on gen-v0.3-dev; recorded at 0.95)

questions: q-v0.2 →

Correct action
69.4%
694/1000
95% CI 66.4–72.2%
Automation
1.7%
17/1000
95% CI 1.0–2.7%
Unsafe / auto
5.9%
1/17
95% CI 0.1–28.7%
Cost / case
$0.000113
$0.113 total · sync
from the spend ledger

Cost: Jev spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Jev q-v0.2 on gen-v0.3-holdoutOperating point 0.97 (chosen on gen-v0.3-dev): 17 of 1000 cases auto-processed, 1 unsafe. The first unsafe automation appears at 0.97 (17 auto-processed, 1 unsafe); stricter thresholds have none.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/10000—679/1000
0.980/10000—679/1000
0.9717/100015.9%694/1000
0.9669/100057.2%738/1000
0.95134/100086.0%797/1000
0.94189/1000126.3%844/1000
0.93236/1000187.6%879/1000
0.92262/1000259.5%891/1000
0.91269/10002810.4%892/1000
0.90271/10002910.7%892/1000
0.89272/10002910.7%893/1000
0.88274/10002910.6%895/1000
0.87275/10003010.9%894/1000
0.86275/10003010.9%894/1000
0.85275/10003010.9%894/1000
0.84275/10003010.9%894/1000
0.83275/10003010.9%894/1000
0.82275/10003010.9%894/1000
0.81275/10003010.9%894/1000
0.80275/10003010.9%894/1000
0.79275/10003010.9%894/1000
0.78275/10003010.9%894/1000
0.77275/10003010.9%894/1000
0.76275/10003010.9%894/1000
0.75275/10003010.9%894/1000
0.74275/10003010.9%894/1000
0.73275/10003010.9%894/1000
0.72275/10003010.9%894/1000
0.71275/10003010.9%894/1000
0.70275/10003010.9%894/1000
0.69275/10003010.9%894/1000
0.68275/10003010.9%894/1000
0.67275/10003010.9%894/1000
0.66275/10003010.9%894/1000
0.65275/10003010.9%894/1000
0.64275/10003010.9%894/1000
0.63275/10003010.9%894/1000
0.62275/10003010.9%894/1000
0.61275/10003010.9%894/1000
0.60275/10003010.9%894/1000
0.59275/10003010.9%894/1000
0.58275/10003010.9%894/1000
0.57275/10003010.9%894/1000
0.56275/10003010.9%894/1000
0.55275/10003010.9%894/1000
0.54275/10003010.9%894/1000
0.53275/10003010.9%894/1000
0.52275/10003010.9%894/1000
0.51275/10003010.9%894/1000
0.50275/10003010.9%894/1000
Request info and human review· 2 rates
Request info
32.3%
323/1000
95% CI 29.4–35.3%
Human review
66.0%
660/1000
95% CI 63.0–68.9%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.029 · n 1000
Reliability of Diagnosis supported: accuracy against stated confidence19990
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.810.770100.0%
0.8–0.990.844100.0%
0.9–1.09900.972100.0%
Step therapy metBrier 0.045 · ECE 0.020 · n 1000
Reliability of Step therapy met: accuracy against stated confidence1118822941
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6110.54354.5%
0.6–0.7180.66066.7%
0.7–0.880.76887.5%
0.8–0.9220.87095.5%
0.9–1.09410.97695.7%
Documentation completeBrier 0.036 · ECE 0.073 · n 1000
Reliability of Documentation complete: accuracy against stated confidence43402213882
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6430.53639.5%
0.6–0.7400.63847.5%
0.7–0.8220.73027.3%
0.8–0.9130.86184.6%
0.9–1.08820.943100.0%
Material contradictionBrier 0.032 · ECE 0.118 · n 1000
Reliability of Material contradiction: accuracy against stated confidence374456476387
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6370.54870.3%
0.6–0.7440.64681.8%
0.7–0.8560.74698.2%
0.8–0.94760.86699.8%
0.9–1.03870.925100.0%
Missing evidenceBrier 0.192 · ECE 0.068 · n 1000x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence158392130238442
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.5150.46853.3%
0.5–0.6830.55145.8%
0.6–0.7920.64442.4%
0.7–0.81300.74770.8%
0.8–0.92380.85394.1%
0.9–1.04420.96599.5%
All runs on gen-v0.3-holdout· 2 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.30.811000921/1000 (92.1%)90.3–93.7%244/1000 (24.4%)0/244 (0.0%)0.0–1.5%$0.000172$0.172
Jev q-v0.20.971000694/1000 (69.4%)66.4–72.2%17/1000 (1.7%)1/17 (5.9%)0.1–28.7%$0.000113$0.113

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →