Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.3, aware (v0.2) on gen-v0.3-shift

run_20260927T072949Z_bd430c · n 400 · auto_process 0.95 (recorded threshold)

questions: q-v0.3 →

The stored answers composed under immunara-v0.2.

Correct action
74.2%
297/400
95% CI 69.7–78.5%
Automation
3.2%
13/400
95% CI 1.7–5.5%
Unsafe / auto
0.0%
0/13
95% CI 0.0–24.7%
Cost / case
—
reused answers
no new calls

Cost: Recomposed from run_20260927T072915Z_007c7c's stored answers; no new provider calls.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Jev q-v0.3, aware (v0.2) on gen-v0.3-shiftOperating point 0.95 (recorded threshold): 13 of 400 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/4000—284/400
0.980/4000—284/400
0.972/40000.0%286/400
0.966/40000.0%290/400
0.9513/40000.0%297/400
0.9440/40000.0%324/400
0.9354/40000.0%338/400
0.9257/40000.0%341/400
0.9162/40000.0%346/400
0.9065/40000.0%349/400
0.8969/40000.0%353/400
0.8869/40000.0%353/400
0.8774/40000.0%358/400
0.8674/40000.0%358/400
0.8576/40000.0%360/400
0.8476/40000.0%360/400
0.8376/40000.0%360/400
0.8276/40000.0%360/400
0.8176/40000.0%360/400
0.8076/40000.0%360/400
0.7976/40000.0%360/400
0.7876/40000.0%360/400
0.7776/40000.0%360/400
0.7676/40000.0%360/400
0.7576/40000.0%360/400
0.7476/40000.0%360/400
0.7376/40000.0%360/400
0.7276/40000.0%360/400
0.7176/40000.0%360/400
0.7076/40000.0%360/400
0.6976/40000.0%360/400
0.6876/40000.0%360/400
0.6776/40000.0%360/400
0.6676/40000.0%360/400
0.6576/40000.0%360/400
0.6476/40000.0%360/400
0.6376/40000.0%360/400
0.6276/40000.0%360/400
0.6176/40000.0%360/400
0.6076/40000.0%360/400
0.5976/40000.0%360/400
0.5876/40000.0%360/400
0.5776/40000.0%360/400
0.5676/40000.0%360/400
0.5576/40000.0%360/400
0.5476/40000.0%360/400
0.5376/40000.0%360/400
0.5276/40000.0%360/400
0.5176/40000.0%360/400
0.5076/40000.0%360/400
Request info and human review· 2 rates
Request info
32.5%
130/400
95% CI 27.9–37.3%
Human review
64.2%
257/400
95% CI 59.3–69.0%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.027 · n 400
Reliability of Diagnosis supported: accuracy against stated confidence2398
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.920.870100.0%
0.9–1.03980.973100.0%
Step therapy metBrier 0.020 · ECE 0.033 · n 400
Reliability of Step therapy met: accuracy against stated confidence57417367
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.650.54260.0%
0.6–0.770.65642.9%
0.7–0.840.7480.0%
0.8–0.9170.86694.1%
0.9–1.03670.97899.7%
Documentation completeBrier 0.032 · ECE 0.081 · n 400
Reliability of Documentation complete: accuracy against stated confidence13897363
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6130.54838.5%
0.6–0.780.64412.5%
0.7–0.890.74711.1%
0.8–0.970.87185.7%
0.9–1.03630.944100.0%
Material contradictionBrier 0.030 · ECE 0.120 · n 400
Reliability of Material contradiction: accuracy against stated confidence91833150190
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.690.55488.9%
0.6–0.7180.64172.2%
0.7–0.8330.755100.0%
0.8–0.91500.861100.0%
0.9–1.01900.923100.0%
Missing evidenceBrier 0.208 · ECE 0.104 · n 400x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence825375596179
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.580.47662.5%
0.5–0.6250.55020.0%
0.6–0.7370.64540.5%
0.7–0.8550.75260.0%
0.8–0.9960.84995.8%
0.9–1.01790.96698.9%
All runs on gen-v0.3-shift· 4 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.3, aware (v0.2)0.95400297/400 (74.2%)69.7–78.5%13/400 (3.2%)0/13 (0.0%)0.0–24.7%——
Jev q-v0.3, stale (v0.1)0.95400294/400 (73.5%)68.9–77.8%16/400 (4.0%)3/16 (18.8%)4.0–45.6%——
Jev q-v0.30.95400297/400 (74.2%)69.7–78.5%13/400 (3.2%)0/13 (0.0%)0.0–24.7%$0.000174$0.0696
Rules0.95400261/400 (65.2%)60.4–69.9%50/400 (12.5%)6/50 (12.0%)4.5–24.3%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →