Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Rules on gen-v0.3-shift

run_20260927T073019Z_81a068 · n 400 · auto_process 0.95 (recorded threshold)

questions: none, rules-v0.1 matches keyword patterns →

rules-v0.1 has no recency rule.

Correct action

65.2%

261/400

95% CI 60.4–69.9%

Automation

12.5%

50/400

95% CI 9.4–16.1%

Unsafe / auto

12.0%

6/50

95% CI 4.5–24.3%

Cost / case

$0 total

no provider call

Cost: No provider call.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

This run is flat: every threshold from 0.50 to 0.99 gives the same actions, because its probabilities are all 0, 0.5 or 1.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe	unsafe / auto	correct
0.99	50/400	6	12.0%	261/400
0.98	50/400	6	12.0%	261/400
0.97	50/400	6	12.0%	261/400
0.96	50/400	6	12.0%	261/400
0.95	50/400	6	12.0%	261/400
0.94	50/400	6	12.0%	261/400
0.93	50/400	6	12.0%	261/400
0.92	50/400	6	12.0%	261/400
0.91	50/400	6	12.0%	261/400
0.90	50/400	6	12.0%	261/400
0.89	50/400	6	12.0%	261/400
0.88	50/400	6	12.0%	261/400
0.87	50/400	6	12.0%	261/400
0.86	50/400	6	12.0%	261/400
0.85	50/400	6	12.0%	261/400
0.84	50/400	6	12.0%	261/400
0.83	50/400	6	12.0%	261/400
0.82	50/400	6	12.0%	261/400
0.81	50/400	6	12.0%	261/400
0.80	50/400	6	12.0%	261/400
0.79	50/400	6	12.0%	261/400
0.78	50/400	6	12.0%	261/400
0.77	50/400	6	12.0%	261/400
0.76	50/400	6	12.0%	261/400
0.75	50/400	6	12.0%	261/400
0.74	50/400	6	12.0%	261/400
0.73	50/400	6	12.0%	261/400
0.72	50/400	6	12.0%	261/400
0.71	50/400	6	12.0%	261/400
0.70	50/400	6	12.0%	261/400
0.69	50/400	6	12.0%	261/400
0.68	50/400	6	12.0%	261/400
0.67	50/400	6	12.0%	261/400
0.66	50/400	6	12.0%	261/400
0.65	50/400	6	12.0%	261/400
0.64	50/400	6	12.0%	261/400
0.63	50/400	6	12.0%	261/400
0.62	50/400	6	12.0%	261/400
0.61	50/400	6	12.0%	261/400
0.60	50/400	6	12.0%	261/400
0.59	50/400	6	12.0%	261/400
0.58	50/400	6	12.0%	261/400
0.57	50/400	6	12.0%	261/400
0.56	50/400	6	12.0%	261/400
0.55	50/400	6	12.0%	261/400
0.54	50/400	6	12.0%	261/400
0.53	50/400	6	12.0%	261/400
0.52	50/400	6	12.0%	261/400
0.51	50/400	6	12.0%	261/400
0.50	50/400	6	12.0%	261/400

Request info and human review· 2 rates

Request info

51.7%

207/400

95% CI 46.7–56.7%

Human review

35.8%

143/400

95% CI 31.0–40.7%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.006 · ECE 0.013 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	10	0.500	0.0%
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	0	—	—
0.9–1.0	390	1.000	100.0%

Step therapy metBrier 0.166 · ECE 0.164 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	169	0.500	25.4%
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	0	—	—
0.9–1.0	231	1.000	89.6%

Documentation completeBrier 0.077 · ECE 0.128 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	124	0.500	91.1%
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	0	—	—
0.9–1.0	276	1.000	100.0%

Material contradictionBrier 0.110 · ECE 0.110 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	0	—	—
0.9–1.0	400	1.000	89.0%

Missing evidenceBrier 0.105 · ECE 0.128 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	0	—	—
0.5–0.6	124	0.500	91.1%
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	0	—	—
0.9–1.0	276	1.000	100.0%

All runs on gen-v0.3-shift· 4 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.3, aware (v0.2)	0.95	400	297/400 (74.2%)	69.7–78.5%	13/400 (3.2%)	0/13 (0.0%)	0.0–24.7%	—	—
Jev q-v0.3, stale (v0.1)	0.95	400	294/400 (73.5%)	68.9–77.8%	16/400 (4.0%)	3/16 (18.8%)	4.0–45.6%	—	—
Jev q-v0.3	0.95	400	297/400 (74.2%)	69.7–78.5%	13/400 (3.2%)	0/13 (0.0%)	0.0–24.7%	$0.000174	$0.0696
Rules	0.95	400	261/400 (65.2%)	60.4–69.9%	50/400 (12.5%)	6/50 (12.0%)	4.5–24.3%	$0	$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →