Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.1 on gen-v0.2-dev

run_20260925T071157Z_d6b218 · n 400 · auto_process 0.95 (recorded threshold)

questions: q-v0.1 →

Replaced by q-v0.2 under the dev adoption rule.

Correct action

64.2%

257/400

95% CI 59.3–69.0%

Automation

7.2%

29/400

95% CI 4.9–10.2%

Unsafe / auto

0.0%

0/29

95% CI 0.0–11.9%

Cost / case

$0.000111

$0.0442 total · sync

trace estimate

Cost: Sum of the traces' estimated cost; not in a committed ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe / auto	correct
0.99	0/400	—	228/400
0.98	0/400	—	228/400
0.97	5/400	0.0%	233/400
0.96	19/400	0.0%	247/400
0.95	29/400	0.0%	257/400
0.94	47/400	0.0%	275/400
0.93	67/400	0.0%	295/400
0.92	86/400	0.0%	314/400
0.91	89/400	0.0%	317/400
0.90	93/400	0.0%	321/400
0.89	97/400	0.0%	325/400
0.88	98/400	0.0%	326/400
0.87	99/400	0.0%	327/400
0.86	99/400	0.0%	327/400
0.85	99/400	0.0%	327/400
0.84	99/400	0.0%	327/400
0.83	99/400	0.0%	327/400
0.82	99/400	0.0%	327/400
0.81	99/400	0.0%	327/400
0.80	99/400	0.0%	327/400
0.79	99/400	0.0%	327/400
0.78	99/400	0.0%	327/400
0.77	99/400	0.0%	327/400
0.76	99/400	0.0%	327/400
0.75	99/400	0.0%	327/400
0.74	99/400	0.0%	327/400
0.73	99/400	0.0%	327/400
0.72	99/400	0.0%	327/400
0.71	99/400	0.0%	327/400
0.70	99/400	0.0%	327/400
0.69	99/400	0.0%	327/400
0.68	99/400	0.0%	327/400
0.67	99/400	0.0%	327/400
0.66	99/400	0.0%	327/400
0.65	99/400	0.0%	327/400
0.64	99/400	0.0%	327/400
0.63	99/400	0.0%	327/400
0.62	99/400	0.0%	327/400
0.61	99/400	0.0%	327/400
0.60	99/400	0.0%	327/400
0.59	99/400	0.0%	327/400
0.58	99/400	0.0%	327/400
0.57	99/400	0.0%	327/400
0.56	99/400	0.0%	327/400
0.55	99/400	0.0%	327/400
0.54	99/400	0.0%	327/400
0.53	99/400	0.0%	327/400
0.52	99/400	0.0%	327/400
0.51	99/400	0.0%	327/400
0.50	99/400	0.0%	327/400

Request info and human review· 2 rates

Request info

40.5%

162/400

95% CI 35.7–45.5%

Human review

52.2%

209/400

95% CI 47.2–57.2%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.029 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	1	0.770	100.0%
0.8–0.9	3	0.877	100.0%
0.9–1.0	396	0.972	100.0%

Step therapy metBrier 0.010 · ECE 0.021 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	6	0.541	50.0%
0.6–0.7	5	0.651	60.0%
0.7–0.8	5	0.744	80.0%
0.8–0.9	9	0.851	100.0%
0.9–1.0	375	0.983	100.0%

Documentation completeBrier 0.030 · ECE 0.067 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	18	0.546	55.6%
0.6–0.7	17	0.640	70.6%
0.7–0.8	7	0.734	85.7%
0.8–0.9	40	0.869	97.5%
0.9–1.0	318	0.936	100.0%

Material contradictionBrier 0.029 · ECE 0.127 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	14	0.543	64.3%
0.6–0.7	19	0.647	100.0%
0.7–0.8	32	0.749	100.0%
0.8–0.9	157	0.862	100.0%
0.9–1.0	178	0.926	100.0%

Missing evidenceBrier 0.406 · ECE 0.146 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	2	0.475	50.0%
0.5–0.6	17	0.548	58.8%
0.6–0.7	21	0.652	76.2%
0.7–0.8	38	0.751	65.8%
0.8–0.9	82	0.854	89.0%
0.9–1.0	240	0.967	76.2%

All runs on gen-v0.2-dev· 4 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.2	0.89	400	365/400 (91.2%)	88.0–93.8%	99/400 (24.8%)	0/99 (0.0%)	0.0–3.7%	$0.000112	$0.0448
Jev q-v0.1	0.95	400	257/400 (64.2%)	59.3–69.0%	29/400 (7.2%)	0/29 (0.0%)	0.0–11.9%	$0.000111	$0.0442
Claude	0.55	400	394/400 (98.5%)	96.8–99.4%	106/400 (26.5%)	0/106 (0.0%)	0.0–3.4%	$0.0110	$4.40
Rules	0.99	400	261/400 (65.2%)	60.4–69.9%	56/400 (14.0%)	0/56 (0.0%)	0.0–6.4%	$0	$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →