Evaluation
Every committed run, at its operating point
Rates carry exact 95% Clopper-Pearson intervals.
How this is measured
Runs on gold and smoke are re-scored from their traces; runs on generated sets come from their committed report bundles.
Rules on gen-v0.2-dev
run_20260925T092358Z_36888d · n 400 · auto_process 0.99 (chosen on gen-v0.2-dev; recorded at 0.95)
questions: none, rules-v0.1 matches keyword patterns →
Cost: No provider call.
Automation against safety
How many cases each threshold would auto-process, and how many of those would be unsafe.
This run is flat: every threshold from 0.50 to 0.99 gives the same actions, because its probabilities are all 0, 0.5 or 1.
How this is measured
Each column is one auto_process threshold, re-decided over the stored judgments: its height is the cases that threshold would auto-process, and its red top is the unsafe automations among them. Thresholds are picked on dev sets with a 1% unsafe ceiling: the threshold that automates the most while unsafe automations stay at or below 1% of those automated, ties going to the stricter threshold. The ceiling is a rule for choosing on dev, so it is not drawn here.
Every threshold· 50 rows
| auto_process | automated | unsafe | unsafe / auto | correct |
|---|---|---|---|---|
| 0.99 | 56/400 | 0 | 0.0% | 261/400 |
| 0.98 | 56/400 | 0 | 0.0% | 261/400 |
| 0.97 | 56/400 | 0 | 0.0% | 261/400 |
| 0.96 | 56/400 | 0 | 0.0% | 261/400 |
| 0.95 | 56/400 | 0 | 0.0% | 261/400 |
| 0.94 | 56/400 | 0 | 0.0% | 261/400 |
| 0.93 | 56/400 | 0 | 0.0% | 261/400 |
| 0.92 | 56/400 | 0 | 0.0% | 261/400 |
| 0.91 | 56/400 | 0 | 0.0% | 261/400 |
| 0.90 | 56/400 | 0 | 0.0% | 261/400 |
| 0.89 | 56/400 | 0 | 0.0% | 261/400 |
| 0.88 | 56/400 | 0 | 0.0% | 261/400 |
| 0.87 | 56/400 | 0 | 0.0% | 261/400 |
| 0.86 | 56/400 | 0 | 0.0% | 261/400 |
| 0.85 | 56/400 | 0 | 0.0% | 261/400 |
| 0.84 | 56/400 | 0 | 0.0% | 261/400 |
| 0.83 | 56/400 | 0 | 0.0% | 261/400 |
| 0.82 | 56/400 | 0 | 0.0% | 261/400 |
| 0.81 | 56/400 | 0 | 0.0% | 261/400 |
| 0.80 | 56/400 | 0 | 0.0% | 261/400 |
| 0.79 | 56/400 | 0 | 0.0% | 261/400 |
| 0.78 | 56/400 | 0 | 0.0% | 261/400 |
| 0.77 | 56/400 | 0 | 0.0% | 261/400 |
| 0.76 | 56/400 | 0 | 0.0% | 261/400 |
| 0.75 | 56/400 | 0 | 0.0% | 261/400 |
| 0.74 | 56/400 | 0 | 0.0% | 261/400 |
| 0.73 | 56/400 | 0 | 0.0% | 261/400 |
| 0.72 | 56/400 | 0 | 0.0% | 261/400 |
| 0.71 | 56/400 | 0 | 0.0% | 261/400 |
| 0.70 | 56/400 | 0 | 0.0% | 261/400 |
| 0.69 | 56/400 | 0 | 0.0% | 261/400 |
| 0.68 | 56/400 | 0 | 0.0% | 261/400 |
| 0.67 | 56/400 | 0 | 0.0% | 261/400 |
| 0.66 | 56/400 | 0 | 0.0% | 261/400 |
| 0.65 | 56/400 | 0 | 0.0% | 261/400 |
| 0.64 | 56/400 | 0 | 0.0% | 261/400 |
| 0.63 | 56/400 | 0 | 0.0% | 261/400 |
| 0.62 | 56/400 | 0 | 0.0% | 261/400 |
| 0.61 | 56/400 | 0 | 0.0% | 261/400 |
| 0.60 | 56/400 | 0 | 0.0% | 261/400 |
| 0.59 | 56/400 | 0 | 0.0% | 261/400 |
| 0.58 | 56/400 | 0 | 0.0% | 261/400 |
| 0.57 | 56/400 | 0 | 0.0% | 261/400 |
| 0.56 | 56/400 | 0 | 0.0% | 261/400 |
| 0.55 | 56/400 | 0 | 0.0% | 261/400 |
| 0.54 | 56/400 | 0 | 0.0% | 261/400 |
| 0.53 | 56/400 | 0 | 0.0% | 261/400 |
| 0.52 | 56/400 | 0 | 0.0% | 261/400 |
| 0.51 | 56/400 | 0 | 0.0% | 261/400 |
| 0.50 | 56/400 | 0 | 0.0% | 261/400 |
Request info and human review· 2 rates
Calibration· 5 charts
Accuracy against stated confidence for each judgment. Points below the diagonal are over-confident; hollow points hold fewer than 20 cases.
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 25 | 0.500 | 0.0% |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 0 | — | — |
| 0.9–1.0 | 375 | 1.000 | 100.0% |
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 209 | 0.500 | 29.2% |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 0 | — | — |
| 0.9–1.0 | 191 | 1.000 | 100.0% |
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 176 | 0.500 | 85.2% |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 0 | — | — |
| 0.9–1.0 | 224 | 1.000 | 100.0% |
Bins· 5 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.5–0.6 | 0 | — | — |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 0 | — | — |
| 0.9–1.0 | 400 | 1.000 | 97.8% |
Bins· 6 rows
| bin | n | mean confidence | accuracy |
|---|---|---|---|
| 0.0–0.5 | 0 | — | — |
| 0.5–0.6 | 176 | 0.500 | 85.2% |
| 0.6–0.7 | 0 | — | — |
| 0.7–0.8 | 0 | — | — |
| 0.8–0.9 | 0 | — | — |
| 0.9–1.0 | 224 | 1.000 | 100.0% |
All runs on gen-v0.2-dev· 4 runs
| Provider | auto_process | n | Correct | 95% CI | Automation | Unsafe / auto | 95% CI | Cost / case | Total cost |
|---|---|---|---|---|---|---|---|---|---|
| Jev q-v0.2 | 0.89 | 400 | 365/400 (91.2%) | 88.0–93.8% | 99/400 (24.8%) | 0/99 (0.0%) | 0.0–3.7% | $0.000112 | $0.0448 |
| Jev q-v0.1 | 0.95 | 400 | 257/400 (64.2%) | 59.3–69.0% | 29/400 (7.2%) | 0/29 (0.0%) | 0.0–11.9% | $0.000111 | $0.0442 |
| Claude | 0.55 | 400 | 394/400 (98.5%) | 96.8–99.4% | 106/400 (26.5%) | 0/106 (0.0%) | 0.0–3.4% | $0.0110 | $4.40 |
| Rules | 0.99 | 400 | 261/400 (65.2%) | 60.4–69.9% | 56/400 (14.0%) | 0/56 (0.0%) | 0.0–6.4% | $0 | $0 |
Cost is from the spend ledgers, or the traces’ own estimates for runs in no ledger. Rules and ground truth make no provider call; “—” marks a run that reused another run’s stored answers. Cost and speed →