Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.2 on gen-v0.3-dev

run_20260927T071846Z_e950c0 · n 400 · auto_process 0.97 (chosen on gen-v0.3-dev; recorded at 0.95)

questions: q-v0.2 →

Correct action

67.0%

268/400

95% CI 62.2–71.6%

Automation

2.5%

10/400

95% CI 1.2–4.5%

Unsafe / auto

0.0%

0/10

95% CI 0.0–30.8%

Cost / case

$0.000113

$0.0452 total · sync

from the spend ledger

Cost: Jev spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe	unsafe / auto	correct
0.99	0/400	0	—	258/400
0.98	0/400	0	—	258/400
0.97	10/400	0	0.0%	268/400
0.96	41/400	3	7.3%	293/400
0.95	68/400	4	5.9%	318/400
0.94	104/400	11	10.6%	340/400
0.93	116/400	13	11.2%	348/400
0.92	123/400	16	13.0%	349/400
0.91	130/400	17	13.1%	354/400
0.90	131/400	18	13.7%	353/400
0.89	131/400	18	13.7%	353/400
0.88	131/400	18	13.7%	353/400
0.87	131/400	18	13.7%	353/400
0.86	131/400	18	13.7%	353/400
0.85	131/400	18	13.7%	353/400
0.84	131/400	18	13.7%	353/400
0.83	131/400	18	13.7%	353/400
0.82	131/400	18	13.7%	353/400
0.81	131/400	18	13.7%	353/400
0.80	131/400	18	13.7%	353/400
0.79	131/400	18	13.7%	353/400
0.78	131/400	18	13.7%	353/400
0.77	131/400	18	13.7%	353/400
0.76	131/400	18	13.7%	353/400
0.75	131/400	18	13.7%	353/400
0.74	131/400	18	13.7%	353/400
0.73	131/400	18	13.7%	353/400
0.72	131/400	18	13.7%	353/400
0.71	131/400	18	13.7%	353/400
0.70	131/400	18	13.7%	353/400
0.69	131/400	18	13.7%	353/400
0.68	131/400	18	13.7%	353/400
0.67	131/400	18	13.7%	353/400
0.66	131/400	18	13.7%	353/400
0.65	131/400	18	13.7%	353/400
0.64	131/400	18	13.7%	353/400
0.63	131/400	18	13.7%	353/400
0.62	131/400	18	13.7%	353/400
0.61	131/400	18	13.7%	353/400
0.60	131/400	18	13.7%	353/400
0.59	131/400	18	13.7%	353/400
0.58	131/400	18	13.7%	353/400
0.57	131/400	18	13.7%	353/400
0.56	131/400	18	13.7%	353/400
0.55	131/400	18	13.7%	353/400
0.54	131/400	18	13.7%	353/400
0.53	131/400	18	13.7%	353/400
0.52	131/400	18	13.7%	353/400
0.51	131/400	18	13.7%	353/400
0.50	131/400	18	13.7%	353/400

Request info and human review· 2 rates

Request info

28.2%

113/400

95% CI 23.9–32.9%

Human review

69.2%

277/400

95% CI 64.5–73.7%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.029 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	5	0.856	100.0%
0.9–1.0	395	0.972	100.0%

Step therapy metBrier 0.060 · ECE 0.037 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	8	0.548	50.0%
0.6–0.7	4	0.627	75.0%
0.7–0.8	3	0.770	100.0%
0.8–0.9	13	0.862	92.3%
0.9–1.0	372	0.974	94.1%

Documentation completeBrier 0.039 · ECE 0.081 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	18	0.551	44.4%
0.6–0.7	14	0.651	35.7%
0.7–0.8	12	0.747	25.0%
0.8–0.9	6	0.883	100.0%
0.9–1.0	350	0.944	100.0%

Material contradictionBrier 0.032 · ECE 0.120 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	16	0.558	87.5%
0.6–0.7	14	0.642	71.4%
0.7–0.8	30	0.754	96.7%
0.8–0.9	160	0.862	100.0%
0.9–1.0	180	0.924	100.0%

Missing evidenceBrier 0.193 · ECE 0.072 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	8	0.480	62.5%
0.5–0.6	33	0.544	39.4%
0.6–0.7	38	0.643	42.1%
0.7–0.8	47	0.751	72.3%
0.8–0.9	80	0.850	92.5%
0.9–1.0	194	0.958	99.5%

All runs on gen-v0.3-dev· 2 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.3	0.81	400	373/400 (93.2%)	90.3–95.5%	114/400 (28.5%)	0/114 (0.0%)	0.0–3.2%	$0.000173	$0.0691
Jev q-v0.2	0.97	400	268/400 (67.0%)	62.2–71.6%	10/400 (2.5%)	0/10 (0.0%)	0.0–30.8%	$0.000113	$0.0452

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →