Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Ground truth on gold-v0.1

run_20260925T170825Z_440df0 · n 100 · auto_process 0.95 (recorded threshold)

questions: none (ground-truth labels)

Ground truth is a pipeline check, not a model result.

Correct action
100.0%
100/100
95% CI 96.4–100.0%
Automation
34.0%
34/100
95% CI 24.8–44.2%
Unsafe / auto
0.0%
0/34
95% CI 0.0–10.3%
Cost / case
$0
$0 total
no provider call

Cost: No provider call.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

This run is flat: every threshold from 0.50 to 0.99 gives the same actions, because its probabilities are all 0, 0.5 or 1.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Ground truth on gold-v0.1Operating point 0.95 (recorded threshold): 34 of 100 cases auto-processed, 0 unsafe. No threshold produces an unsafe automation.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.9934/10000.0%100/100
0.9834/10000.0%100/100
0.9734/10000.0%100/100
0.9634/10000.0%100/100
0.9534/10000.0%100/100
0.9434/10000.0%100/100
0.9334/10000.0%100/100
0.9234/10000.0%100/100
0.9134/10000.0%100/100
0.9034/10000.0%100/100
0.8934/10000.0%100/100
0.8834/10000.0%100/100
0.8734/10000.0%100/100
0.8634/10000.0%100/100
0.8534/10000.0%100/100
0.8434/10000.0%100/100
0.8334/10000.0%100/100
0.8234/10000.0%100/100
0.8134/10000.0%100/100
0.8034/10000.0%100/100
0.7934/10000.0%100/100
0.7834/10000.0%100/100
0.7734/10000.0%100/100
0.7634/10000.0%100/100
0.7534/10000.0%100/100
0.7434/10000.0%100/100
0.7334/10000.0%100/100
0.7234/10000.0%100/100
0.7134/10000.0%100/100
0.7034/10000.0%100/100
0.6934/10000.0%100/100
0.6834/10000.0%100/100
0.6734/10000.0%100/100
0.6634/10000.0%100/100
0.6534/10000.0%100/100
0.6434/10000.0%100/100
0.6334/10000.0%100/100
0.6234/10000.0%100/100
0.6134/10000.0%100/100
0.6034/10000.0%100/100
0.5934/10000.0%100/100
0.5834/10000.0%100/100
0.5734/10000.0%100/100
0.5634/10000.0%100/100
0.5534/10000.0%100/100
0.5434/10000.0%100/100
0.5334/10000.0%100/100
0.5234/10000.0%100/100
0.5134/10000.0%100/100
0.5034/10000.0%100/100
Request info and human review· 2 rates
Request info
34.0%
34/100
95% CI 24.8–44.2%
Human review
32.0%
32/100
95% CI 23.0–42.1%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.000 · ECE 0.000 · n 100
Reliability of Diagnosis supported: accuracy against stated confidence100
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.01001.000100.0%
Step therapy metBrier 0.000 · ECE 0.000 · n 100
Reliability of Step therapy met: accuracy against stated confidence100
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.01001.000100.0%
Documentation completeBrier 0.000 · ECE 0.000 · n 100
Reliability of Documentation complete: accuracy against stated confidence100
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.01001.000100.0%
Material contradictionBrier 0.000 · ECE 0.000 · n 100
Reliability of Material contradiction: accuracy against stated confidence100
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.01001.000100.0%
Missing evidenceBrier 0.000 · ECE 0.000 · n 100x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence100
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.50——
0.5–0.60——
0.6–0.70——
0.7–0.80——
0.8–0.90——
0.9–1.01001.000100.0%
All runs on gold-v0.1· 5 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.8910091/100 (91.0%)83.6–95.8%29/100 (29.0%)1/29 (3.4%)0.1–17.8%$0.000115$0.0115
Jev q-v0.30.8110094/100 (94.0%)87.4–97.8%31/100 (31.0%)1/31 (3.2%)0.1–16.7%$0.000175$0.0175
Claude0.5510093/100 (93.0%)86.1–97.1%30/100 (30.0%)1/30 (3.3%)0.1–17.2%$0.0156$1.56
Rules0.9910061/100 (61.0%)50.7–70.6%20/100 (20.0%)6/20 (30.0%)11.9–54.3%$0$0
Ground truth0.95100100/100 (100.0%)96.4–100.0%34/100 (34.0%)0/34 (0.0%)0.0–10.3%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →