Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Rules on smoke-v0.1

run_20260925T092347Z_cca17f · n 10 · auto_process 0.99 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: none, rules-v0.1 matches keyword patterns →

Correct action
90.0%
9/10
95% CI 55.5–99.7%
Automation
20.0%
2/10
95% CI 2.5–55.6%
Unsafe / auto
0.0%
0/2
95% CI 0.0–84.2%
Cost / case
$0
$0 total
no provider call

Cost: No provider call.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

This run is flat: every threshold from 0.50 to 0.99 gives the same actions, because its probabilities are all 0, 0.5 or 1.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Rules on smoke-v0.1Operating point 0.99 (chosen on gen-v0.2-dev): 2 of 10 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.992/1000.0%9/10
0.982/1000.0%9/10
0.972/1000.0%9/10
0.962/1000.0%9/10
0.952/1000.0%9/10
0.942/1000.0%9/10
0.932/1000.0%9/10
0.922/1000.0%9/10
0.912/1000.0%9/10
0.902/1000.0%9/10
0.892/1000.0%9/10
0.882/1000.0%9/10
0.872/1000.0%9/10
0.862/1000.0%9/10
0.852/1000.0%9/10
0.842/1000.0%9/10
0.832/1000.0%9/10
0.822/1000.0%9/10
0.812/1000.0%9/10
0.802/1000.0%9/10
0.792/1000.0%9/10
0.782/1000.0%9/10
0.772/1000.0%9/10
0.762/1000.0%9/10
0.752/1000.0%9/10
0.742/1000.0%9/10
0.732/1000.0%9/10
0.722/1000.0%9/10
0.712/1000.0%9/10
0.702/1000.0%9/10
0.692/1000.0%9/10
0.682/1000.0%9/10
0.672/1000.0%9/10
0.662/1000.0%9/10
0.652/1000.0%9/10
0.642/1000.0%9/10
0.632/1000.0%9/10
0.622/1000.0%9/10
0.612/1000.0%9/10
0.602/1000.0%9/10
0.592/1000.0%9/10
0.582/1000.0%9/10
0.572/1000.0%9/10
0.562/1000.0%9/10
0.552/1000.0%9/10
0.542/1000.0%9/10
0.532/1000.0%9/10
0.522/1000.0%9/10
0.512/1000.0%9/10
0.502/1000.0%9/10
Request info and human review· 2 rates
Request info
40.0%
4/10
95% CI 12.2–73.8%
Human review
40.0%
4/10
95% CI 12.2–73.8%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.000 · ECE 0.000 · n 10
Reliability of Diagnosis supported: accuracy against stated confidence10
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0101.000100.0%
Step therapy metBrier 0.075 · ECE 0.050 · n 10
Reliability of Step therapy met: accuracy against stated confidence37
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.630.50066.7%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.071.000100.0%
Documentation completeBrier 0.025 · ECE 0.050 · n 10
Reliability of Documentation complete: accuracy against stated confidence19
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.500100.0%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.091.000100.0%
Material contradictionBrier 0.000 · ECE 0.000 · n 10
Reliability of Material contradiction: accuracy against stated confidence10
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.0101.000100.0%
Missing evidenceBrier 0.025 · ECE 0.050 · n 10x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence19
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.50——
0.5–0.610.500100.0%
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.091.000100.0%
All runs on smoke-v0.1· 3 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.10.95108/10 (80.0%)44.4–97.5%2/10 (20.0%)0/2 (0.0%)0.0–84.2%$0.000109$0.00109
Claude0.551010/10 (100.0%)69.2–100.0%3/10 (30.0%)0/3 (0.0%)0.0–70.8%$0.0255$0.255
Rules0.99109/10 (90.0%)55.5–99.7%2/10 (20.0%)0/2 (0.0%)0.0–84.2%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →