Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Rules on gold-v0.1

run_20260925T170839Z_d3b427 · n 100 · auto_process 0.99 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: none, rules-v0.1 matches keyword patterns →

Rules probabilities are 0, 0.5 or 1 and are not calibrated.

Correct action
61.0%
61/100
95% CI 50.7–70.6%
Automation
20.0%
20/100
95% CI 12.7–29.2%
Unsafe / auto
30.0%
6/20
95% CI 11.9–54.3%
Cost / case
$0
$0 total
no provider call

Cost: No provider call.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

This run is flat: every threshold from 0.50 to 0.99 gives the same actions, because its probabilities are all 0, 0.5 or 1.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Rules on gold-v0.1Operating point 0.99 (chosen on gen-v0.2-dev): 20 of 100 cases auto-processed, 6 unsafe. Unsafe automations appear at every threshold, from 0.99 (20 auto-processed, 6 unsafe).
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.9920/100630.0%61/100
0.9820/100630.0%61/100
0.9720/100630.0%61/100
0.9620/100630.0%61/100
0.9520/100630.0%61/100
0.9420/100630.0%61/100
0.9320/100630.0%61/100
0.9220/100630.0%61/100
0.9120/100630.0%61/100
0.9020/100630.0%61/100
0.8920/100630.0%61/100
0.8820/100630.0%61/100
0.8720/100630.0%61/100
0.8620/100630.0%61/100
0.8520/100630.0%61/100
0.8420/100630.0%61/100
0.8320/100630.0%61/100
0.8220/100630.0%61/100
0.8120/100630.0%61/100
0.8020/100630.0%61/100
0.7920/100630.0%61/100
0.7820/100630.0%61/100
0.7720/100630.0%61/100
0.7620/100630.0%61/100
0.7520/100630.0%61/100
0.7420/100630.0%61/100
0.7320/100630.0%61/100
0.7220/100630.0%61/100
0.7120/100630.0%61/100
0.7020/100630.0%61/100
0.6920/100630.0%61/100
0.6820/100630.0%61/100
0.6720/100630.0%61/100
0.6620/100630.0%61/100
0.6520/100630.0%61/100
0.6420/100630.0%61/100
0.6320/100630.0%61/100
0.6220/100630.0%61/100
0.6120/100630.0%61/100
0.6020/100630.0%61/100
0.5920/100630.0%61/100
0.5820/100630.0%61/100
0.5720/100630.0%61/100
0.5620/100630.0%61/100
0.5520/100630.0%61/100
0.5420/100630.0%61/100
0.5320/100630.0%61/100
0.5220/100630.0%61/100
0.5120/100630.0%61/100
0.5020/100630.0%61/100
Request info and human review· 2 rates
Request info
66.0%
66/100
95% CI 55.8–75.2%
Human review
14.0%
14/100
95% CI 7.9–22.4%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.065 · ECE 0.070 · n 100
Reliability of Diagnosis supported: accuracy against stated confidence1090
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6100.50080.0%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0901.00095.6%
Step therapy metBrier 0.198 · ECE 0.115 · n 100
Reliability of Step therapy met: accuracy against stated confidence5149
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6510.50041.2%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0491.00085.7%
Documentation completeBrier 0.117 · ECE 0.125 · n 100
Reliability of Documentation complete: accuracy against stated confidence3961
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6390.50076.9%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0611.00096.7%
Material contradictionBrier 0.150 · ECE 0.150 · n 100
Reliability of Material contradiction: accuracy against stated confidence100
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.01001.00085.0%
Missing evidenceBrier 0.268 · ECE 0.145 · n 100x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence3961
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.50——
0.5–0.6390.50076.9%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0611.00093.4%
All runs on gold-v0.1· 5 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.8910091/100 (91.0%)83.6–95.8%29/100 (29.0%)1/29 (3.4%)0.1–17.8%$0.000115$0.0115
Jev q-v0.30.8110094/100 (94.0%)87.4–97.8%31/100 (31.0%)1/31 (3.2%)0.1–16.7%$0.000175$0.0175
Claude0.5510093/100 (93.0%)86.1–97.1%30/100 (30.0%)1/30 (3.3%)0.1–17.2%$0.0156$1.56
Rules0.9910061/100 (61.0%)50.7–70.6%20/100 (20.0%)6/20 (30.0%)11.9–54.3%$0$0
Ground truth0.95100100/100 (100.0%)96.4–100.0%34/100 (34.0%)0/34 (0.0%)0.0–10.3%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →