Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Claude (150-case sample) on gen-v0.2-holdout

run_20260925T212034Z_bbee49 · n 150 · auto_process 0.55 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 + claude-prompt-v1 →

A 150-case sample (seed 7), so it is not in the headline table.

Correct action

98.0%

147/150

95% CI 94.3–99.6%

Automation

24.7%

37/150

95% CI 18.0–32.4%

Unsafe / auto

0.0%

0/37

95% CI 0.0–9.5%

Cost / case

$0.0164

$2.46 total · batch

from the spend ledger

Cost: Claude spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe / auto	correct
0.99	0/150	—	110/150
0.98	0/150	—	110/150
0.97	0/150	—	110/150
0.96	0/150	—	110/150
0.95	0/150	—	110/150
0.94	0/150	—	110/150
0.93	0/150	—	110/150
0.92	0/150	—	110/150
0.91	0/150	—	110/150
0.90	0/150	—	110/150
0.89	0/150	—	110/150
0.88	0/150	—	110/150
0.87	0/150	—	110/150
0.86	0/150	—	110/150
0.85	0/150	—	110/150
0.84	0/150	—	110/150
0.83	1/150	0.0%	111/150
0.82	1/150	0.0%	111/150
0.81	1/150	0.0%	111/150
0.80	2/150	0.0%	112/150
0.79	2/150	0.0%	112/150
0.78	4/150	0.0%	114/150
0.77	4/150	0.0%	114/150
0.76	5/150	0.0%	115/150
0.75	8/150	0.0%	118/150
0.74	18/150	0.0%	128/150
0.73	20/150	0.0%	130/150
0.72	25/150	0.0%	135/150
0.71	26/150	0.0%	136/150
0.70	26/150	0.0%	136/150
0.69	29/150	0.0%	139/150
0.68	30/150	0.0%	140/150
0.67	33/150	0.0%	143/150
0.66	35/150	0.0%	145/150
0.65	35/150	0.0%	145/150
0.64	37/150	0.0%	147/150
0.63	37/150	0.0%	147/150
0.62	37/150	0.0%	147/150
0.61	37/150	0.0%	147/150
0.60	37/150	0.0%	147/150
0.59	37/150	0.0%	147/150
0.58	37/150	0.0%	147/150
0.57	37/150	0.0%	147/150
0.56	37/150	0.0%	147/150
0.55	37/150	0.0%	147/150
0.54	37/150	0.0%	147/150
0.53	37/150	0.0%	147/150
0.52	37/150	0.0%	147/150
0.51	37/150	0.0%	147/150
0.50	37/150	0.0%	147/150

Request info and human review· 2 rates

Request info

31.3%

47/150

95% CI 24.0–39.4%

Human review

44.0%

66/150

95% CI 35.9–52.3%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.002 · ECE 0.041 · n 150

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	5	0.854	100.0%
0.9–1.0	145	0.963	100.0%

Step therapy metBrier 0.027 · ECE 0.101 · n 150

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	16	0.678	100.0%
0.7–0.8	33	0.744	100.0%
0.8–0.9	5	0.812	100.0%
0.9–1.0	96	0.993	100.0%

Documentation completeBrier 0.044 · ECE 0.029 · n 150

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	1	0.550	0.0%
0.6–0.7	0	—	—
0.7–0.8	5	0.720	80.0%
0.8–0.9	8	0.838	87.5%
0.9–1.0	136	0.941	96.3%

Material contradictionBrier 0.002 · ECE 0.041 · n 150

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	4	0.880	100.0%
0.9–1.0	146	0.961	100.0%

Missing evidenceBrier 0.083 · ECE 0.137 · n 150x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	0	—	—
0.5–0.6	3	0.500	33.3%
0.6–0.7	10	0.647	90.0%
0.7–0.8	29	0.742	89.7%
0.8–0.9	57	0.851	100.0%
0.9–1.0	51	0.911	100.0%

All runs on gen-v0.2-holdout· 3 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.2	0.89	1000	895/1000 (89.5%)	87.4–91.3%	252/1000 (25.2%)	0/252 (0.0%)	0.0–1.5%	$0.000112	$0.112
Claude (150-case sample)	0.55	150	147/150 (98.0%)	94.3–99.6%	37/150 (24.7%)	0/37 (0.0%)	0.0–9.5%	$0.0164	$2.46
Rules	0.99	1000	667/1000 (66.7%)	63.7–69.6%	134/1000 (13.4%)	0/134 (0.0%)	0.0–2.7%	$0	$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →