Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Claude on smoke-v0.1

run_20260925T131052Z_f328de · n 10 · auto_process 0.55 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 + claude-prompt-v1 →

Correct action
100.0%
10/10
95% CI 69.2–100.0%
Automation
30.0%
3/10
95% CI 6.7–65.2%
Unsafe / auto
0.0%
0/3
95% CI 0.0–70.8%
Cost / case
$0.0255
$0.255 total · sync
from the spend ledger

Cost: Claude spend ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Claude on smoke-v0.1Operating point 0.55 (chosen on gen-v0.2-dev): 3 of 10 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/100—7/10
0.980/100—7/10
0.970/100—7/10
0.960/100—7/10
0.950/100—7/10
0.940/100—7/10
0.930/100—7/10
0.920/100—7/10
0.910/100—7/10
0.900/100—7/10
0.890/100—7/10
0.880/100—7/10
0.870/100—7/10
0.860/100—7/10
0.850/100—7/10
0.840/100—7/10
0.830/100—7/10
0.820/100—7/10
0.811/1000.0%8/10
0.801/1000.0%8/10
0.791/1000.0%8/10
0.781/1000.0%8/10
0.771/1000.0%8/10
0.761/1000.0%8/10
0.751/1000.0%8/10
0.741/1000.0%8/10
0.732/1000.0%9/10
0.722/1000.0%9/10
0.712/1000.0%9/10
0.702/1000.0%9/10
0.692/1000.0%9/10
0.682/1000.0%9/10
0.672/1000.0%9/10
0.663/1000.0%10/10
0.653/1000.0%10/10
0.643/1000.0%10/10
0.633/1000.0%10/10
0.623/1000.0%10/10
0.613/1000.0%10/10
0.603/1000.0%10/10
0.593/1000.0%10/10
0.583/1000.0%10/10
0.573/1000.0%10/10
0.563/1000.0%10/10
0.553/1000.0%10/10
0.543/1000.0%10/10
0.533/1000.0%10/10
0.523/1000.0%10/10
0.513/1000.0%10/10
0.503/1000.0%10/10
Request info and human review· 2 rates
Request info
30.0%
3/10
95% CI 6.7–65.2%
Human review
40.0%
4/10
95% CI 12.2–73.8%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.001 · ECE 0.037 · n 10
Reliability of Diagnosis supported: accuracy against stated confidence10
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0100.963100.0%
Step therapy metBrier 0.054 · ECE 0.177 · n 10
Reliability of Step therapy met: accuracy against stated confidence4114
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.740.672100.0%
0.7–0.810.736100.0%
0.8–0.910.815100.0%
0.9–1.040.997100.0%
Documentation completeBrier 0.061 · ECE 0.133 · n 10
Reliability of Documentation complete: accuracy against stated confidence118
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.810.7500.0%
0.8–0.910.850100.0%
0.9–1.080.946100.0%
Material contradictionBrier 0.001 · ECE 0.032 · n 10
Reliability of Material contradiction: accuracy against stated confidence10
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0100.968100.0%
Missing evidenceBrier 0.084 · ECE 0.174 · n 10x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence1153
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.50——
0.5–0.610.5000.0%
0.6–0.70——
0.7–0.810.750100.0%
0.8–0.950.852100.0%
0.9–1.030.917100.0%
All runs on smoke-v0.1· 3 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.10.95108/10 (80.0%)44.4–97.5%2/10 (20.0%)0/2 (0.0%)0.0–84.2%$0.000109$0.00109
Claude0.551010/10 (100.0%)69.2–100.0%3/10 (30.0%)0/3 (0.0%)0.0–70.8%$0.0255$0.255
Rules0.99109/10 (90.0%)55.5–99.7%2/10 (20.0%)0/2 (0.0%)0.0–84.2%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →