Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.2 on gen-v0.2-dev

run_20260925T071231Z_6f0b73 · n 400 · auto_process 0.89 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 →

Correct action

91.2%

365/400

95% CI 88.0–93.8%

Automation

24.8%

99/400

95% CI 20.6–29.3%

Unsafe / auto

0.0%

0/99

95% CI 0.0–3.7%

Cost / case

$0.000112

$0.0448 total · sync

trace estimate

Cost: Sum of the traces' estimated cost; not in a committed ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe / auto	correct
0.99	0/400	—	266/400
0.98	0/400	—	266/400
0.97	11/400	0.0%	277/400
0.96	25/400	0.0%	291/400
0.95	64/400	0.0%	330/400
0.94	88/400	0.0%	354/400
0.93	96/400	0.0%	362/400
0.92	97/400	0.0%	363/400
0.91	98/400	0.0%	364/400
0.90	98/400	0.0%	364/400
0.89	99/400	0.0%	365/400
0.88	99/400	0.0%	365/400
0.87	99/400	0.0%	365/400
0.86	99/400	0.0%	365/400
0.85	99/400	0.0%	365/400
0.84	99/400	0.0%	365/400
0.83	99/400	0.0%	365/400
0.82	99/400	0.0%	365/400
0.81	99/400	0.0%	365/400
0.80	99/400	0.0%	365/400
0.79	99/400	0.0%	365/400
0.78	99/400	0.0%	365/400
0.77	99/400	0.0%	365/400
0.76	99/400	0.0%	365/400
0.75	99/400	0.0%	365/400
0.74	99/400	0.0%	365/400
0.73	99/400	0.0%	365/400
0.72	99/400	0.0%	365/400
0.71	99/400	0.0%	365/400
0.70	99/400	0.0%	365/400
0.69	99/400	0.0%	365/400
0.68	99/400	0.0%	365/400
0.67	99/400	0.0%	365/400
0.66	99/400	0.0%	365/400
0.65	99/400	0.0%	365/400
0.64	99/400	0.0%	365/400
0.63	99/400	0.0%	365/400
0.62	99/400	0.0%	365/400
0.61	99/400	0.0%	365/400
0.60	99/400	0.0%	365/400
0.59	99/400	0.0%	365/400
0.58	99/400	0.0%	365/400
0.57	99/400	0.0%	365/400
0.56	99/400	0.0%	365/400
0.55	99/400	0.0%	365/400
0.54	99/400	0.0%	365/400
0.53	99/400	0.0%	365/400
0.52	99/400	0.0%	365/400
0.51	99/400	0.0%	365/400
0.50	99/400	0.0%	365/400

Request info and human review· 2 rates

Request info

31.0%

124/400

95% CI 26.5–35.8%

Human review

44.2%

177/400

95% CI 39.3–49.3%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.030 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	1	0.780	100.0%
0.8–0.9	3	0.870	100.0%
0.9–1.0	396	0.972	100.0%

Step therapy metBrier 0.010 · ECE 0.021 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	5	0.537	60.0%
0.6–0.7	7	0.648	57.1%
0.7–0.8	6	0.763	83.3%
0.8–0.9	6	0.869	100.0%
0.9–1.0	376	0.983	100.0%

Documentation completeBrier 0.035 · ECE 0.084 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	17	0.536	88.2%
0.6–0.7	17	0.631	47.1%
0.7–0.8	8	0.745	12.5%
0.8–0.9	9	0.876	88.9%
0.9–1.0	349	0.943	100.0%

Material contradictionBrier 0.029 · ECE 0.135 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	16	0.552	87.5%
0.6–0.7	18	0.645	100.0%
0.7–0.8	29	0.752	100.0%
0.8–0.9	164	0.863	100.0%
0.9–1.0	173	0.926	100.0%

Missing evidenceBrier 0.199 · ECE 0.055 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	2	0.480	50.0%
0.5–0.6	35	0.548	45.7%
0.6–0.7	39	0.650	51.3%
0.7–0.8	54	0.744	70.4%
0.8–0.9	100	0.856	92.0%
0.9–1.0	170	0.965	99.4%

All runs on gen-v0.2-dev· 4 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.2	0.89	400	365/400 (91.2%)	88.0–93.8%	99/400 (24.8%)	0/99 (0.0%)	0.0–3.7%	$0.000112	$0.0448
Jev q-v0.1	0.95	400	257/400 (64.2%)	59.3–69.0%	29/400 (7.2%)	0/29 (0.0%)	0.0–11.9%	$0.000111	$0.0442
Claude	0.55	400	394/400 (98.5%)	96.8–99.4%	106/400 (26.5%)	0/106 (0.0%)	0.0–3.4%	$0.0110	$4.40
Rules	0.99	400	261/400 (65.2%)	60.4–69.9%	56/400 (14.0%)	0/56 (0.0%)	0.0–6.4%	$0	$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →