Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.3 on gen-v0.3-dev

run_20260927T071912Z_cdaf0c · n 400 · auto_process 0.81 (chosen on gen-v0.3-dev; recorded at 0.95)

questions: q-v0.3 →

Correct action

93.2%

373/400

95% CI 90.3–95.5%

Automation

28.5%

114/400

95% CI 24.1–33.2%

Unsafe / auto

0.0%

0/114

95% CI 0.0–3.2%

Cost / case

$0.000173

$0.0691 total · sync

from the spend ledger

Cost: Jev spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe / auto	correct
0.99	0/400	—	259/400
0.98	0/400	—	259/400
0.97	0/400	—	259/400
0.96	8/400	0.0%	267/400
0.95	17/400	0.0%	276/400
0.94	43/400	0.0%	302/400
0.93	66/400	0.0%	325/400
0.92	78/400	0.0%	337/400
0.91	94/400	0.0%	353/400
0.90	101/400	0.0%	360/400
0.89	104/400	0.0%	363/400
0.88	107/400	0.0%	366/400
0.87	109/400	0.0%	368/400
0.86	111/400	0.0%	370/400
0.85	112/400	0.0%	371/400
0.84	113/400	0.0%	372/400
0.83	113/400	0.0%	372/400
0.82	113/400	0.0%	372/400
0.81	114/400	0.0%	373/400
0.80	114/400	0.0%	373/400
0.79	114/400	0.0%	373/400
0.78	114/400	0.0%	373/400
0.77	114/400	0.0%	373/400
0.76	114/400	0.0%	373/400
0.75	114/400	0.0%	373/400
0.74	114/400	0.0%	373/400
0.73	114/400	0.0%	373/400
0.72	114/400	0.0%	373/400
0.71	114/400	0.0%	373/400
0.70	114/400	0.0%	373/400
0.69	114/400	0.0%	373/400
0.68	114/400	0.0%	373/400
0.67	114/400	0.0%	373/400
0.66	114/400	0.0%	373/400
0.65	114/400	0.0%	373/400
0.64	114/400	0.0%	373/400
0.63	114/400	0.0%	373/400
0.62	114/400	0.0%	373/400
0.61	114/400	0.0%	373/400
0.60	114/400	0.0%	373/400
0.59	114/400	0.0%	373/400
0.58	114/400	0.0%	373/400
0.57	114/400	0.0%	373/400
0.56	114/400	0.0%	373/400
0.55	114/400	0.0%	373/400
0.54	114/400	0.0%	373/400
0.53	114/400	0.0%	373/400
0.52	114/400	0.0%	373/400
0.51	114/400	0.0%	373/400
0.50	114/400	0.0%	373/400

Request info and human review· 2 rates

Request info

28.0%

112/400

95% CI 23.7–32.7%

Human review

43.5%

174/400

95% CI 38.6–48.5%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.029 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	6	0.860	100.0%
0.9–1.0	394	0.972	100.0%

Step therapy metBrier 0.016 · ECE 0.048 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	9	0.536	22.2%
0.6–0.7	5	0.641	60.0%
0.7–0.8	5	0.760	60.0%
0.8–0.9	29	0.865	100.0%
0.9–1.0	352	0.968	100.0%

Documentation completeBrier 0.039 · ECE 0.084 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	17	0.552	64.7%
0.6–0.7	16	0.643	25.0%
0.7–0.8	10	0.745	20.0%
0.8–0.9	7	0.870	100.0%
0.9–1.0	350	0.945	100.0%

Material contradictionBrier 0.032 · ECE 0.118 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	15	0.555	66.7%
0.6–0.7	18	0.642	88.9%
0.7–0.8	24	0.750	95.8%
0.8–0.9	160	0.861	100.0%
0.9–1.0	183	0.924	100.0%

Missing evidenceBrier 0.190 · ECE 0.078 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	8	0.475	37.5%
0.5–0.6	37	0.549	45.9%
0.6–0.7	39	0.653	38.5%
0.7–0.8	41	0.744	70.7%
0.8–0.9	81	0.851	95.1%
0.9–1.0	194	0.958	99.5%

All runs on gen-v0.3-dev· 2 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.3	0.81	400	373/400 (93.2%)	90.3–95.5%	114/400 (28.5%)	0/114 (0.0%)	0.0–3.2%	$0.000173	$0.0691
Jev q-v0.2	0.97	400	268/400 (67.0%)	62.2–71.6%	10/400 (2.5%)	0/10 (0.0%)	0.0–30.8%	$0.000113	$0.0452

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →