Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Claude on gen-v0.2-dev

run_20260925T191752Z_288946 · n 400 · auto_process 0.55 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 + claude-prompt-v1 →

Correct action

98.5%

394/400

95% CI 96.8–99.4%

Automation

26.5%

106/400

95% CI 22.2–31.1%

Unsafe / auto

0.0%

0/106

95% CI 0.0–3.4%

Cost / case

$0.0110

$4.40 total · batch

from the spend ledger

Cost: Claude spend ledger, two batches: d65789 (378 cases, $4.123975) + b1c45c (22 cases, $0.274699); 22 cancelled cases were re-run and billed once.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe / auto	correct
0.99	0/400	—	288/400
0.98	0/400	—	288/400
0.97	0/400	—	288/400
0.96	0/400	—	288/400
0.95	0/400	—	288/400
0.94	0/400	—	288/400
0.93	0/400	—	288/400
0.92	0/400	—	288/400
0.91	0/400	—	288/400
0.90	0/400	—	288/400
0.89	0/400	—	288/400
0.88	0/400	—	288/400
0.87	0/400	—	288/400
0.86	0/400	—	288/400
0.85	0/400	—	288/400
0.84	0/400	—	288/400
0.83	3/400	0.0%	291/400
0.82	5/400	0.0%	293/400
0.81	5/400	0.0%	293/400
0.80	7/400	0.0%	295/400
0.79	11/400	0.0%	299/400
0.78	13/400	0.0%	301/400
0.77	13/400	0.0%	301/400
0.76	13/400	0.0%	301/400
0.75	26/400	0.0%	314/400
0.74	44/400	0.0%	332/400
0.73	53/400	0.0%	341/400
0.72	67/400	0.0%	355/400
0.71	70/400	0.0%	358/400
0.70	72/400	0.0%	360/400
0.69	80/400	0.0%	368/400
0.68	81/400	0.0%	369/400
0.67	86/400	0.0%	374/400
0.66	87/400	0.0%	375/400
0.65	88/400	0.0%	376/400
0.64	93/400	0.0%	381/400
0.63	95/400	0.0%	383/400
0.62	99/400	0.0%	387/400
0.61	99/400	0.0%	387/400
0.60	101/400	0.0%	389/400
0.59	101/400	0.0%	389/400
0.58	102/400	0.0%	390/400
0.57	105/400	0.0%	393/400
0.56	105/400	0.0%	393/400
0.55	106/400	0.0%	394/400
0.54	106/400	0.0%	394/400
0.53	106/400	0.0%	394/400
0.52	106/400	0.0%	394/400
0.51	106/400	0.0%	394/400
0.50	106/400	0.0%	394/400

Request info and human review· 2 rates

Request info

23.5%

94/400

95% CI 19.4–28.0%

Human review

50.0%

200/400

95% CI 45.0–55.0%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.002 · ECE 0.040 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	1	0.750	100.0%
0.8–0.9	11	0.870	100.0%
0.9–1.0	388	0.963	100.0%

Step therapy metBrier 0.030 · ECE 0.094 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	8	0.576	100.0%
0.6–0.7	36	0.662	100.0%
0.7–0.8	78	0.744	100.0%
0.8–0.9	9	0.824	100.0%
0.9–1.0	269	0.994	99.6%

Documentation completeBrier 0.040 · ECE 0.069 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	3	0.500	0.0%
0.6–0.7	8	0.619	62.5%
0.7–0.8	17	0.744	52.9%
0.8–0.9	23	0.840	69.6%
0.9–1.0	349	0.940	99.4%

Material contradictionBrier 0.002 · ECE 0.043 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	6	0.870	100.0%
0.9–1.0	394	0.959	100.0%

Missing evidenceBrier 0.088 · ECE 0.148 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	4	0.460	50.0%
0.5–0.6	22	0.545	45.5%
0.6–0.7	30	0.637	90.0%
0.7–0.8	69	0.752	97.1%
0.8–0.9	182	0.859	100.0%
0.9–1.0	93	0.910	100.0%

All runs on gen-v0.2-dev· 4 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.2	0.89	400	365/400 (91.2%)	88.0–93.8%	99/400 (24.8%)	0/99 (0.0%)	0.0–3.7%	$0.000112	$0.0448
Jev q-v0.1	0.95	400	257/400 (64.2%)	59.3–69.0%	29/400 (7.2%)	0/29 (0.0%)	0.0–11.9%	$0.000111	$0.0442
Claude	0.55	400	394/400 (98.5%)	96.8–99.4%	106/400 (26.5%)	0/106 (0.0%)	0.0–3.4%	$0.0110	$4.40
Rules	0.99	400	261/400 (65.2%)	60.4–69.9%	56/400 (14.0%)	0/56 (0.0%)	0.0–6.4%	$0	$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →