Skip to content

Evaluation

Every committed run, at its operating point

Rates carry exact 95% Clopper-Pearson intervals.

How this is measured

Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.

Jev q-v0.2 on gold-v0.1

run_20260925T170857Z_b95be9 · n 100 · auto_process 0.89 (chosen on gen-v0.2-dev; recorded at 0.95)

questions: q-v0.2 →

Correct action
91.0%
91/100
95% CI 83.6–95.8%
Automation
29.0%
29/100
95% CI 20.4–38.9%
Unsafe / auto
3.4%
1/29
95% CI 0.1–17.8%
Cost / case
$0.000115
$0.0115 total · sync
trace estimate

Cost: Sum of the traces' estimated cost; not in a committed ledger.

Automation against safety

How many cases each threshold would auto-process, and how many of those would be unsafe.

How this is measured

Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.

Cases auto-processed and unsafe automations at each auto_process threshold, 0.99 to 0.50, for Jev q-v0.2 on gold-v0.1Operating point 0.89 (chosen on gen-v0.2-dev): 29 of 100 cases auto-processed, 1 unsafe. The first unsafe automation appears at 0.93 (24 auto-processed, 1 unsafe); stricter thresholds have none.
Every threshold· 50 rows
auto_processautomatedunsafeunsafe / autocorrect
0.990/1000—64/100
0.982/10000.0%66/100
0.975/10000.0%69/100
0.969/10000.0%73/100
0.9518/10000.0%82/100
0.9420/10000.0%84/100
0.9324/10014.2%86/100
0.9226/10013.8%88/100
0.9128/10013.6%90/100
0.9028/10013.6%90/100
0.8929/10013.4%91/100
0.8829/10013.4%91/100
0.8729/10013.4%91/100
0.8629/10013.4%91/100
0.8530/10013.3%92/100
0.8430/10013.3%92/100
0.8330/10013.3%92/100
0.8230/10013.3%92/100
0.8132/10026.2%93/100
0.8032/10026.2%93/100
0.7932/10026.2%93/100
0.7832/10026.2%93/100
0.7732/10026.2%93/100
0.7632/10026.2%93/100
0.7532/10026.2%93/100
0.7432/10026.2%93/100
0.7332/10026.2%93/100
0.7232/10026.2%93/100
0.7132/10026.2%93/100
0.7032/10026.2%93/100
0.6932/10026.2%93/100
0.6832/10026.2%93/100
0.6732/10026.2%93/100
0.6632/10026.2%93/100
0.6532/10026.2%93/100
0.6432/10026.2%93/100
0.6332/10026.2%93/100
0.6232/10026.2%93/100
0.6132/10026.2%93/100
0.6033/10026.1%94/100
0.5933/10026.1%94/100
0.5833/10026.1%94/100
0.5733/10026.1%94/100
0.5633/10026.1%94/100
0.5533/10026.1%94/100
0.5433/10026.1%94/100
0.5333/10026.1%94/100
0.5233/10026.1%94/100
0.5133/10026.1%94/100
0.5033/10026.1%94/100
Request info and human review· 2 rates
Request info
32.0%
32/100
95% CI 23.0–42.1%
Human review
39.0%
39/100
95% CI 29.4–49.3%
Calibration· 5 charts

Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.

Diagnosis supportedBrier 0.011 · ECE 0.049 · n 100
Reliability of Diagnosis supported: accuracy against stated confidence112294
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.610.5700.0%
0.6–0.710.6800.0%
0.7–0.820.730100.0%
0.8–0.920.885100.0%
0.9–1.0940.969100.0%
Step therapy metBrier 0.075 · ECE 0.049 · n 100
Reliability of Step therapy met: accuracy against stated confidence435286
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.640.55325.0%
0.6–0.730.63566.7%
0.7–0.850.73560.0%
0.8–0.920.83450.0%
0.9–1.0860.98095.3%
Documentation completeBrier 0.063 · ECE 0.035 · n 100
Reliability of Documentation complete: accuracy against stated confidence334684
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.630.51333.3%
0.6–0.730.63766.7%
0.7–0.840.75250.0%
0.8–0.960.84583.3%
0.9–1.0840.94396.4%
Material contradictionBrier 0.040 · ECE 0.086 · n 100
Reliability of Material contradiction: accuracy against stated confidence3343159
Bins· 5 rows
binnmean confidenceaccuracy
0.5–0.630.55366.7%
0.6–0.730.66333.3%
0.7–0.840.76575.0%
0.8–0.9310.86796.8%
0.9–1.0590.931100.0%
Missing evidenceBrier 0.120 · ECE 0.067 · n 100x from 0: a multiple-choice answer’s top confidence can fall below 0.5.
Reliability of Missing evidence: accuracy against stated confidence28392058
Bins· 6 rows
binnmean confidenceaccuracy
0.0–0.520.46050.0%
0.5–0.680.53237.5%
0.6–0.730.667100.0%
0.7–0.890.73377.8%
0.8–0.9200.84695.0%
0.9–1.0580.967100.0%
All runs on gold-v0.1· 5 runs
Providerauto_processnCorrect95% CIAutomationUnsafe / auto95% CICost / caseTotal cost
Jev q-v0.20.8910091/100 (91.0%)83.6–95.8%29/100 (29.0%)1/29 (3.4%)0.1–17.8%$0.000115$0.0115
Jev q-v0.30.8110094/100 (94.0%)87.4–97.8%31/100 (31.0%)1/31 (3.2%)0.1–16.7%$0.000175$0.0175
Claude0.5510093/100 (93.0%)86.1–97.1%30/100 (30.0%)1/30 (3.3%)0.1–17.2%$0.0156$1.56
Rules0.9910061/100 (61.0%)50.7–70.6%20/100 (20.0%)6/20 (30.0%)11.9–54.3%$0$0
Ground truth0.95100100/100 (100.0%)96.4–100.0%34/100 (34.0%)0/34 (0.0%)0.0–10.3%$0$0

Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →