Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Rules on gen-v0.2-holdout

run_20260925T092425Z_0aee97 · n 1000 · auto_process 0.99 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: none, rules-v0.1 matches keyword patterns →

Correct action
66.7%
667/1000
95% CI 63.7–69.6%
Automation
13.4%
134/1000
95% CI 11.3–15.7%
Unsafe / auto
0.0%
0/134
95% CI 0.0–2.7%
Cost / case
$0
$0 total
no provider call

Cost: No provider call.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

This run is flat: every threshold from 0.50 to 0.99 gives the same actions, because its probabilities are all 0, 0.5 or 1.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Rules on gen-v0.2-holdoutOperating point 0.99 (chosen on gen-v0.2-dev): 134 of 1000 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.99134/100000.0%667/1000
0.98134/100000.0%667/1000
0.97134/100000.0%667/1000
0.96134/100000.0%667/1000
0.95134/100000.0%667/1000
0.94134/100000.0%667/1000
0.93134/100000.0%667/1000
0.92134/100000.0%667/1000
0.91134/100000.0%667/1000
0.90134/100000.0%667/1000
0.89134/100000.0%667/1000
0.88134/100000.0%667/1000
0.87134/100000.0%667/1000
0.86134/100000.0%667/1000
0.85134/100000.0%667/1000
0.84134/100000.0%667/1000
0.83134/100000.0%667/1000
0.82134/100000.0%667/1000
0.81134/100000.0%667/1000
0.80134/100000.0%667/1000
0.79134/100000.0%667/1000
0.78134/100000.0%667/1000
0.77134/100000.0%667/1000
0.76134/100000.0%667/1000
0.75134/100000.0%667/1000
0.74134/100000.0%667/1000
0.73134/100000.0%667/1000
0.72134/100000.0%667/1000
0.71134/100000.0%667/1000
0.70134/100000.0%667/1000
0.69134/100000.0%667/1000
0.68134/100000.0%667/1000
0.67134/100000.0%667/1000
0.66134/100000.0%667/1000
0.65134/100000.0%667/1000
0.64134/100000.0%667/1000
0.63134/100000.0%667/1000
0.62134/100000.0%667/1000
0.61134/100000.0%667/1000
0.60134/100000.0%667/1000
0.59134/100000.0%667/1000
0.58134/100000.0%667/1000
0.57134/100000.0%667/1000
0.56134/100000.0%667/1000
0.55134/100000.0%667/1000
0.54134/100000.0%667/1000
0.53134/100000.0%667/1000
0.52134/100000.0%667/1000
0.51134/100000.0%667/1000
0.50134/100000.0%667/1000
Request info and human review· 2 rates
Request info
56.9%
569/1000
95% CI 53.8–60.0%
Human review
29.7%
297/1000
95% CI 26.9–32.6%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.011 · ECE 0.022 · n 1000
Reliability of Diagnosis supported: accuracy against stated confidence45955
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.6450.5000.0%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.09551.000100.0%
Step therapy metBrier 0.125 · ECE 0.065 · n 1000
Reliability of Step therapy met: accuracy against stated confidence499501
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.64990.50037.1%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.05011.000100.0%
Documentation completeBrier 0.097 · ECE 0.143 · n 1000
Reliability of Documentation complete: accuracy against stated confidence387613
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.63870.50087.1%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.06131.000100.0%
Material contradictionBrier 0.026 · ECE 0.026 · n 1000
Reliability of Material contradiction: accuracy against stated confidence1000
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.010001.00097.4%
Missing evidenceBrier 0.147 · ECE 0.143 · n 1000x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence387613
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.50——
0.5–0.63870.50087.1%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.06131.000100.0%
All runs on gen-v0.2-holdout· 3 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.891000895/1000 (89.5%)87.4–91.3%252/1000 (25.2%)0/252 (0.0%)0.0–1.5%$0.000112$0.112
Claude (150-case sample)0.55150147/150 (98.0%)94.3–99.6%37/150 (24.7%)0/37 (0.0%)0.0–9.5%$0.0164$2.46
Rules0.991000667/1000 (66.7%)63.7–69.6%134/1000 (13.4%)0/134 (0.0%)0.0–2.7%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →