Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.3 on gen-v0.3-holdout

run_20260927T072144Z_12e1e4 · n 1000 · auto_process 0.81 (chosen on gen-v0.3-dev; recorded at 0.95)

questions: q-v0.3 →

Correct action
92.1%
921/1000
95% CI 90.3–93.7%
Automation
24.4%
244/1000
95% CI 21.8–27.2%
Unsafe / auto
0.0%
0/244
95% CI 0.0–1.5%
Cost / case
$0.000172
$0.172 total · sync
from the spend ledger

Cost: Jev spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Jev q-v0.3 on gen-v0.3-holdoutOperating point 0.81 (chosen on gen-v0.3-dev): 244 of 1000 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/10000—677/1000
0.980/10000—677/1000
0.975/100000.0%682/1000
0.9616/100000.0%693/1000
0.9534/100000.0%711/1000
0.9496/100000.0%773/1000
0.93158/100000.0%835/1000
0.92189/100000.0%866/1000
0.91205/100000.0%882/1000
0.90214/100000.0%891/1000
0.89227/100000.0%904/1000
0.88229/100000.0%906/1000
0.87233/100000.0%910/1000
0.86239/100000.0%916/1000
0.85242/100000.0%919/1000
0.84242/100000.0%919/1000
0.83243/100000.0%920/1000
0.82243/100000.0%920/1000
0.81244/100000.0%921/1000
0.80244/100000.0%921/1000
0.79244/100000.0%921/1000
0.78244/100000.0%921/1000
0.77244/100000.0%921/1000
0.76244/100000.0%921/1000
0.75244/100000.0%921/1000
0.74244/100000.0%921/1000
0.73244/100000.0%921/1000
0.72244/100000.0%921/1000
0.71244/100000.0%921/1000
0.70244/100000.0%921/1000
0.69244/100000.0%921/1000
0.68244/100000.0%921/1000
0.67244/100000.0%921/1000
0.66244/100000.0%921/1000
0.65244/100000.0%921/1000
0.64245/100000.0%922/1000
0.63245/100000.0%922/1000
0.62245/100000.0%922/1000
0.61245/100000.0%922/1000
0.60245/100000.0%922/1000
0.59245/100000.0%922/1000
0.58245/100000.0%922/1000
0.57245/100000.0%922/1000
0.56245/100000.0%922/1000
0.55245/100000.0%922/1000
0.54245/100000.0%922/1000
0.53245/100000.0%922/1000
0.52245/100000.0%922/1000
0.51245/100000.0%922/1000
0.50245/100000.0%922/1000
Request info and human review· 2 rates
Request info
32.7%
327/1000
95% CI 29.8–35.7%
Human review
42.9%
429/1000
95% CI 39.8–46.0%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.029 · n 1000
Reliability of Diagnosis supported: accuracy against stated confidence18991
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.810.780100.0%
0.8–0.980.840100.0%
0.9–1.09910.972100.0%
Step therapy metBrier 0.014 · ECE 0.038 · n 1000
Reliability of Step therapy met: accuracy against stated confidence1818855901
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6180.54350.0%
0.6–0.7180.64938.9%
0.7–0.880.76275.0%
0.8–0.9550.873100.0%
0.9–1.09010.971100.0%
Documentation completeBrier 0.036 · ECE 0.072 · n 1000
Reliability of Documentation complete: accuracy against stated confidence44412018877
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6440.54047.7%
0.6–0.7410.64043.9%
0.7–0.8200.74320.0%
0.8–0.9180.87088.9%
0.9–1.08770.943100.0%
Material contradictionBrier 0.032 · ECE 0.116 · n 1000
Reliability of Material contradiction: accuracy against stated confidence374158477387
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6370.54664.9%
0.6–0.7410.64980.5%
0.7–0.8580.74196.6%
0.8–0.94770.866100.0%
0.9–1.03870.925100.0%
Missing evidenceBrier 0.196 · ECE 0.072 · n 1000x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence158396124241441
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.5150.46953.3%
0.5–0.6830.54745.8%
0.6–0.7960.64942.7%
0.7–0.81240.74968.5%
0.8–0.92410.85593.8%
0.9–1.04410.96699.8%
All runs on gen-v0.3-holdout· 2 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.30.811000921/1000 (92.1%)90.3–93.7%244/1000 (24.4%)0/244 (0.0%)0.0–1.5%$0.000172$0.172
Jev q-v0.20.971000694/1000 (69.4%)66.4–72.2%17/1000 (1.7%)1/17 (5.9%)0.1–28.7%$0.000113$0.113

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →