Evaluation
Every committed run, at its operating point
Rates carry exact 95% Clopper-Pearson intervals.
How this is measured
Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.
Jev q-v0.1 on smoke-v0.1
run_20260925T042324Z_eee114 · n 10 · auto_process 0.95 (recorded threshold)
questions: q-v0.1 →
The reference smoke run, made with q-v0.1 before q-v0.2 was adopted.
Cost: Sum of the traces' estimated cost; not in a committed ledger.
Automation against safety
How many cases each threshold would auto-process, and how many of those would be unsafe.
How this is measured
Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.
Every threshold· 50 rows
| auto_process | automated | unsafe | unsafe / auto | correct |
|---|---|---|---|---|
| 0.99 | 0/10 | 0 | — | 6/10 |
| 0.98 | 0/10 | 0 | — | 6/10 |
| 0.97 | 0/10 | 0 | — | 6/10 |
| 0.96 | 1/10 | 0 | 0.0% | 7/10 |
| 0.95 | 2/10 | 0 | 0.0% | 8/10 |
| 0.94 | 2/10 | 0 | 0.0% | 8/10 |
| 0.93 | 2/10 | 0 | 0.0% | 8/10 |
| 0.92 | 2/10 | 0 | 0.0% | 8/10 |
| 0.91 | 3/10 | 0 | 0.0% | 9/10 |
| 0.90 | 3/10 | 0 | 0.0% | 9/10 |
| 0.89 | 3/10 | 0 | 0.0% | 9/10 |
| 0.88 | 3/10 | 0 | 0.0% | 9/10 |
| 0.87 | 3/10 | 0 | 0.0% | 9/10 |
| 0.86 | 3/10 | 0 | 0.0% | 9/10 |
| 0.85 | 3/10 | 0 | 0.0% | 9/10 |
| 0.84 | 3/10 | 0 | 0.0% | 9/10 |
| 0.83 | 3/10 | 0 | 0.0% | 9/10 |
| 0.82 | 3/10 | 0 | 0.0% | 9/10 |
| 0.81 | 3/10 | 0 | 0.0% | 9/10 |
| 0.80 | 3/10 | 0 | 0.0% | 9/10 |
| 0.79 | 3/10 | 0 | 0.0% | 9/10 |
| 0.78 | 3/10 | 0 | 0.0% | 9/10 |
| 0.77 | 3/10 | 0 | 0.0% | 9/10 |
| 0.76 | 3/10 | 0 | 0.0% | 9/10 |
| 0.75 | 3/10 | 0 | 0.0% | 9/10 |
| 0.74 | 3/10 | 0 | 0.0% | 9/10 |
| 0.73 | 3/10 | 0 | 0.0% | 9/10 |
| 0.72 | 3/10 | 0 | 0.0% | 9/10 |
| 0.71 | 3/10 | 0 | 0.0% | 9/10 |
| 0.70 | 3/10 | 0 | 0.0% | 9/10 |
| 0.69 | 3/10 | 0 | 0.0% | 9/10 |
| 0.68 | 3/10 | 0 | 0.0% | 9/10 |
| 0.67 | 3/10 | 0 | 0.0% | 9/10 |
| 0.66 | 3/10 | 0 | 0.0% | 9/10 |
| 0.65 | 3/10 | 0 | 0.0% | 9/10 |
| 0.64 | 3/10 | 0 | 0.0% | 9/10 |
| 0.63 | 3/10 | 0 | 0.0% | 9/10 |
| 0.62 | 3/10 | 0 | 0.0% | 9/10 |
| 0.61 | 3/10 | 0 | 0.0% | 9/10 |
| 0.60 | 3/10 | 0 | 0.0% | 9/10 |
| 0.59 | 3/10 | 0 | 0.0% | 9/10 |
| 0.58 | 3/10 | 0 | 0.0% | 9/10 |
| 0.57 | 3/10 | 0 | 0.0% | 9/10 |
| 0.56 | 3/10 | 0 | 0.0% | 9/10 |
| 0.55 | 3/10 | 0 | 0.0% | 9/10 |
| 0.54 | 3/10 | 0 | 0.0% | 9/10 |
| 0.53 | 3/10 | 0 | 0.0% | 9/10 |
| 0.52 | 3/10 | 0 | 0.0% | 9/10 |
| 0.51 | 3/10 | 0 | 0.0% | 9/10 |
| 0.50 | 3/10 | 0 | 0.0% | 9/10 |
Request info and human review· 2 rates
Calibration· 5 charts
Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 0 | — | — |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 1 | 0.840 | 100.0% |
| 0.9–1.0 | 9 | 0.973 | 100.0% |
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 0 | — | — |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 1 | 0.730 | 100.0% |
| 0.8–0.9 | 0 | — | — |
| 0.9–1.0 | 9 | 0.983 | 100.0% |
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 1 | 0.570 | 100.0% |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 2 | 0.860 | 100.0% |
| 0.9–1.0 | 7 | 0.939 | 100.0% |
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 0 | — | — |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 4 | 0.873 | 100.0% |
| 0.9–1.0 | 6 | 0.937 | 100.0% |
Bins· 6 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.0–0.5 | 0 | — | — |
| 0.5–0.6 | 0 | — | — |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 3 | 0.843 | 100.0% |
| 0.9–1.0 | 7 | 0.983 | 71.4% |
All runs on smoke-v0.1· 3 runs
| Provider | auto_process | n | Correct | 95% CI | Automation | Unsafe / auto | 95% CI | Cost / case | Total cost |
|---|---|---|---|---|---|---|---|---|---|
| Jev q-v0.1 | 0.95 | 10 | 8/10 (80.0%) | 44.4–97.5% | 2/10 (20.0%) | 0/2 (0.0%) | 0.0–84.2% | $0.000109 | $0.00109 |
| Claude | 0.55 | 10 | 10/10 (100.0%) | 69.2–100.0% | 3/10 (30.0%) | 0/3 (0.0%) | 0.0–70.8% | $0.0255 | $0.255 |
| Rules | 0.99 | 10 | 9/10 (90.0%) | 55.5–99.7% | 2/10 (20.0%) | 0/2 (0.0%) | 0.0–84.2% | $0 | $0 |
Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →