Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.3, stale (v0.1) on gen-v0.3-shift

run_20260927T072948Z_e5915e · n 400 · auto_process 0.95 (recorded threshold)

questions: q-v0.3 →

The same stored answers composed under the old immunara-v0.1.

Correct action

73.5%

294/400

95% CI 68.9–77.8%

Automation

4.0%

16/400

95% CI 2.3–6.4%

Unsafe / auto

18.8%

3/16

95% CI 4.0–45.6%

Cost / case

—

reused answers

no new calls

Cost: Recomposed from run_20260927T072915Z_007c7c's stored answers; no new provider calls.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Every threshold· 50 rows

auto_process	automated	unsafe	unsafe / auto	correct
0.99	0/400	0	—	284/400
0.98	0/400	0	—	284/400
0.97	3/400	1	33.3%	285/400
0.96	8/400	2	25.0%	288/400
0.95	16/400	3	18.8%	294/400
0.94	47/400	7	14.9%	317/400
0.93	66/400	12	18.2%	326/400
0.92	73/400	16	21.9%	325/400
0.91	78/400	16	20.5%	330/400
0.90	82/400	17	20.7%	332/400
0.89	86/400	17	19.8%	336/400
0.88	86/400	17	19.8%	336/400
0.87	91/400	17	18.7%	341/400
0.86	91/400	17	18.7%	341/400
0.85	93/400	17	18.3%	343/400
0.84	93/400	17	18.3%	343/400
0.83	93/400	17	18.3%	343/400
0.82	93/400	17	18.3%	343/400
0.81	93/400	17	18.3%	343/400
0.80	93/400	17	18.3%	343/400
0.79	93/400	17	18.3%	343/400
0.78	93/400	17	18.3%	343/400
0.77	93/400	17	18.3%	343/400
0.76	93/400	17	18.3%	343/400
0.75	93/400	17	18.3%	343/400
0.74	93/400	17	18.3%	343/400
0.73	93/400	17	18.3%	343/400
0.72	93/400	17	18.3%	343/400
0.71	93/400	17	18.3%	343/400
0.70	93/400	17	18.3%	343/400
0.69	93/400	17	18.3%	343/400
0.68	93/400	17	18.3%	343/400
0.67	93/400	17	18.3%	343/400
0.66	93/400	17	18.3%	343/400
0.65	93/400	17	18.3%	343/400
0.64	93/400	17	18.3%	343/400
0.63	93/400	17	18.3%	343/400
0.62	93/400	17	18.3%	343/400
0.61	93/400	17	18.3%	343/400
0.60	93/400	17	18.3%	343/400
0.59	93/400	17	18.3%	343/400
0.58	93/400	17	18.3%	343/400
0.57	93/400	17	18.3%	343/400
0.56	93/400	17	18.3%	343/400
0.55	93/400	17	18.3%	343/400
0.54	93/400	17	18.3%	343/400
0.53	93/400	17	18.3%	343/400
0.52	93/400	17	18.3%	343/400
0.51	93/400	17	18.3%	343/400
0.50	93/400	17	18.3%	343/400

Request info and human review· 2 rates

Request info

32.5%

130/400

95% CI 27.9–37.3%

Human review

63.5%

254/400

95% CI 58.6–68.2%

Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.027 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	0	—	—
0.6–0.7	0	—	—
0.7–0.8	0	—	—
0.8–0.9	2	0.870	100.0%
0.9–1.0	398	0.973	100.0%

Step therapy metBrier 0.072 · ECE 0.051 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	6	0.544	50.0%
0.6–0.7	9	0.646	44.4%
0.7–0.8	4	0.748	0.0%
0.8–0.9	17	0.866	94.1%
0.9–1.0	364	0.973	93.4%

Documentation completeBrier 0.032 · ECE 0.081 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	13	0.548	38.5%
0.6–0.7	8	0.644	12.5%
0.7–0.8	9	0.747	11.1%
0.8–0.9	7	0.871	85.7%
0.9–1.0	363	0.944	100.0%

Material contradictionBrier 0.030 · ECE 0.120 · n 400

Bins· 5 rows

bin	n	mean confidence	accuracy
0.5–0.6	9	0.554	88.9%
0.6–0.7	18	0.641	72.2%
0.7–0.8	33	0.755	100.0%
0.8–0.9	150	0.861	100.0%
0.9–1.0	190	0.923	100.0%

Missing evidenceBrier 0.208 · ECE 0.104 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.

Bins· 6 rows

bin	n	mean confidence	accuracy
0.0–0.5	8	0.476	62.5%
0.5–0.6	25	0.550	20.0%
0.6–0.7	37	0.645	40.5%
0.7–0.8	55	0.752	60.0%
0.8–0.9	96	0.849	95.8%
0.9–1.0	179	0.966	98.9%

All runs on gen-v0.3-shift· 4 runs

Provider	auto_process	n	Correct	95% CI	Automation	Unsafe / auto	95% CI	Cost / case	Total cost
Jev q-v0.3, aware (v0.2)	0.95	400	297/400 (74.2%)	69.7–78.5%	13/400 (3.2%)	0/13 (0.0%)	0.0–24.7%	—	—
Jev q-v0.3, stale (v0.1)	0.95	400	294/400 (73.5%)	68.9–77.8%	16/400 (4.0%)	3/16 (18.8%)	4.0–45.6%	—	—
Jev q-v0.3	0.95	400	297/400 (74.2%)	69.7–78.5%	13/400 (3.2%)	0/13 (0.0%)	0.0–24.7%	$0.000174	$0.0696
Rules	0.95	400	261/400 (65.2%)	60.4–69.9%	50/400 (12.5%)	6/50 (12.0%)	4.5–24.3%	$0	$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →