Experiments
Four changes, each measured before it was believed
Each section states what was tried, the rule fixed in advance, and what the committed runs show, including the results that did not go the way we hoped. The last section compares what Jev and Claude cost per case, and how fast they answered.
The q-v0.3 fix
694 → 921 correct of 1000 on the holdout, with 0/244 unsafe automations.
q-v0.3 adds questions about paused and restarted methotrexate courses; the gain is concentrated in interrupted courses.
Show details
q-v0.2 has one start and one end date, so it cannot express a paused methotrexate course. q-v0.3 adds seven questions about pauses and restarts. It was adopted on gen-v0.3-dev (268 → 373 of 400 correct, gate PASS) and then run once on gen-v0.3-holdout. Compare the question sets →
| Metric | Baseline | 95% CI | Candidate | 95% CI | Δ |
|---|---|---|---|---|---|
| Correct action | 694/1000 (69.4%) | 66.4–72.2% | 921/1000 (92.1%) | 90.3–93.7% | +22.7 pp |
| Automation | 17/1000 (1.7%) | 1.0–2.7% | 244/1000 (24.4%) | 21.8–27.2% | +22.7 pp |
| Request info | 323/1000 (32.3%) | 29.4–35.3% | 327/1000 (32.7%) | 29.8–35.7% | +0.4 pp |
| Human review | 660/1000 (66.0%) | 63.0–68.9% | 429/1000 (42.9%) | 39.8–46.0% | −23.1 pp |
| Unsafe / auto | 1/17 (5.9%) | 0.1–28.7% | 0/244 (0.0%) | 0.0–1.5% | −5.9 pp |
Baseline: replay-run_20260927T072249Z_204814 · jev q-v0.2 · policy immunara-v0.1 (v0.1) · thresholds auto_process=0.97. Candidate: candidate trace run_20260927T072144Z_12e1e4 · jev q-v0.3 · re-decided at auto_process=0.81.
Where the gain comes from: step-therapy accuracy by course
gen-v0.3-dev · n 400
| Course | Cases | q-v0.2 | q-v0.3 |
|---|---|---|---|
| Interrupted, then restarted | 73 | 50/73 (68.5%) | 72/73 (98.6%) |
| Ended over 12 months earlier | 58 | 57/58 (98.3%) | 56/58 (96.6%) |
| Everything else | 269 | 265/269 (98.5%) | 261/269 (97.0%) |
gen-v0.3-holdout · n 1000
| Course | Cases | q-v0.2 | q-v0.3 |
|---|---|---|---|
| Interrupted, then restarted | 188 | 147/188 (78.2%) | 188/188 (100.0%) |
| Ended over 12 months earlier | 142 | 142/142 (100.0%) | 137/142 (96.5%) |
| Everything else | 670 | 658/670 (98.2%) | 653/670 (97.5%) |
The gain is concentrated in interrupted courses. On old and other courses q-v0.3 is slightly worse than q-v0.2. On gold, which is not a blind test here, the gate fails: GOLD-TMP-16 becomes newly unsafe at the lower 0.81 threshold.
Policy shift
Stale: 3/16 unsafe automations. Aware: 0/13.
The same stored Jev answers, composed under the old policy and under immunara-v0.2, which adds a 12-month recency rule.
Show details
immunara-v0.2 adds one rule: the qualifying methotrexate course must have been ongoing or ended within 12 months of the request. The same stored Jev answers on gen-v0.3-shift were composed twice: under the old policy (stale) and under the new one (aware).
| Composition | Correct action | Automation | Unsafe / auto |
|---|---|---|---|
| Stale, immunara-v0.1 | 294/400 (73.5%) | 16/400 (4.0%) | 3/16 (18.8%) |
| Aware, immunara-v0.2 | 297/400 (74.2%) | 13/400 (3.2%) | 0/13 (0.0%) |
| Rules, no recency rule | 261/400 | 50/400 | 6/50 |
The stale → aware gate is PASS with 0 newly unsafe and 0 regressed, and the shadow rollout of aware against stale is PROMOTE. Being aware costs 3 automations out of 400.
Parallelism
p50 latency goes from 178 to 191 ms as questions per call go from 1 to 20.
Narrow decisions are cheap to add; cost per case grows with tokens.
Show details
40 gen-v0.3-dev cases were sent to jev-1.13.0 with 1, 5, 10 and 20 questions in one call. Narrow decisions are cheap to add; cost per case grows with tokens.
| Questions | p50 ms | p95 ms | Tokens | $ / case |
|---|---|---|---|---|
| 1 | 178 | 230 | 982 | 0.000041 |
| 5 | 177 | 225 | 1526 | 0.000064 |
| 10 | 186 | 207 | 2534 | 0.000106 |
| 20 | 191 | 290 | 4207 | 0.000177 |
The p95 at 20 questions rests on three slow calls out of 40, so it is not a stable estimate.
Gate ablation
Removing contradiction detection makes GOLD-CON-03 and GOLD-CON-13 newly unsafe for every model provider on gold.
Each committed run was re-decided with the contradiction gate, the missing-evidence gate, or both disabled.
Show details
The null results: no ablation creates a newly unsafe case on the generated sets, and removing the missing-evidence gate changes no automation in any of the 10 runs. The claude-150 rows: Claude's holdout run is a 150-case sample (seed 7) of gen-v0.2-holdout, not the 1000-case holdout the other rows use.
| Dataset | Run | Gate removed | Verdict | Actions changed | Automation | Unsafe / auto | Newly unsafe |
|---|---|---|---|---|---|---|---|
| gen-v0.2-holdout | jev-q-v0.2 | contradiction | PASS | 37/1000 | 252 → 277 | 0/252 → 0/277 | none |
| gen-v0.2-holdout | jev-q-v0.2 | missing_evidence | PASS | 100/1000 | 252 → 252 | 0/252 → 0/252 | none |
| gen-v0.2-holdout | jev-q-v0.2 | contradiction+missing_evidence | PASS | 134/1000 | 252 → 277 | 0/252 → 0/277 | none |
| gen-v0.2-holdout | rules | contradiction | PASS | 17/1000 | 134 → 134 | 0/134 → 0/134 | none |
| gen-v0.2-holdout | rules | missing_evidence | PASS | 0/1000 | 134 → 134 | 0/134 → 0/134 | none |
| gen-v0.2-holdout | rules | contradiction+missing_evidence | PASS | 17/1000 | 134 → 134 | 0/134 → 0/134 | none |
| gen-v0.2-holdout | claude-150150-case sample | contradiction | PASS | 3/150 | 37 → 37 | 0/37 → 0/37 | none |
| gen-v0.2-holdout | claude-150150-case sample | missing_evidence | PASS | 10/150 | 37 → 37 | 0/37 → 0/37 | none |
| gen-v0.2-holdout | claude-150150-case sample | contradiction+missing_evidence | PASS | 13/150 | 37 → 37 | 0/37 → 0/37 | none |
| gen-v0.3-holdout | jev-q-v0.3 | contradiction | PASS | 33/1000 | 244 → 261 | 0/244 → 0/261 | none |
| gen-v0.3-holdout | jev-q-v0.3 | missing_evidence | PASS | 93/1000 | 244 → 244 | 0/244 → 0/244 | none |
| gen-v0.3-holdout | jev-q-v0.3 | contradiction+missing_evidence | PASS | 123/1000 | 244 → 261 | 0/244 → 0/261 | none |
| gen-v0.3-shift | jev-q-v0.3-aware | contradiction | PASS | 7/400 | 13 → 13 | 0/13 → 0/13 | none |
| gen-v0.3-shift | jev-q-v0.3-aware | missing_evidence | PASS | 41/400 | 13 → 13 | 0/13 → 0/13 | none |
| gen-v0.3-shift | jev-q-v0.3-aware | contradiction+missing_evidence | PASS | 47/400 | 13 → 13 | 0/13 → 0/13 | none |
| gold-v0.1 | groundtruth | contradiction | PASS | 1/100 | 34 → 34 | 0/34 → 0/34 | none |
| gold-v0.1 | groundtruth | missing_evidence | PASS | 0/100 | 34 → 34 | 0/34 → 0/34 | none |
| gold-v0.1 | groundtruth | contradiction+missing_evidence | PASS | 1/100 | 34 → 34 | 0/34 → 0/34 | none |
| gold-v0.1 | jev-q-v0.2 | contradiction | FAIL | 6/100 | 29 → 32 | 1/29 → 3/32 | GOLD-CON-03, GOLD-CON-13 |
| gold-v0.1 | jev-q-v0.2 | missing_evidence | PASS | 5/100 | 29 → 29 | 1/29 → 1/29 | none |
| gold-v0.1 | jev-q-v0.2 | contradiction+missing_evidence | FAIL | 9/100 | 29 → 32 | 1/29 → 3/32 | GOLD-CON-03, GOLD-CON-13 |
| gold-v0.1 | rules | contradiction | PASS | 0/100 | 20 → 20 | 6/20 → 6/20 | none |
| gold-v0.1 | rules | missing_evidence | PASS | 0/100 | 20 → 20 | 6/20 → 6/20 | none |
| gold-v0.1 | rules | contradiction+missing_evidence | PASS | 0/100 | 20 → 20 | 6/20 → 6/20 | none |
| gold-v0.1 | claude | contradiction | FAIL | 5/100 | 30 → 34 | 1/30 → 3/34 | GOLD-CON-03, GOLD-CON-13 |
| gold-v0.1 | claude | missing_evidence | PASS | 6/100 | 30 → 30 | 1/30 → 1/30 | none |
| gold-v0.1 | claude | contradiction+missing_evidence | FAIL | 11/100 | 30 → 34 | 1/30 → 3/34 | GOLD-CON-03, GOLD-CON-13 |
| gold-v0.1 | jev-q-v0.3 | contradiction | FAIL | 5/100 | 31 → 34 | 1/31 → 3/34 | GOLD-CON-03, GOLD-CON-13 |
| gold-v0.1 | jev-q-v0.3 | missing_evidence | PASS | 4/100 | 31 → 31 | 1/31 → 1/31 | none |
| gold-v0.1 | jev-q-v0.3 | contradiction+missing_evidence | FAIL | 8/100 | 31 → 34 | 1/31 → 3/34 | GOLD-CON-03, GOLD-CON-13 |
Cost and speed
On gold with the same questions, Jev q-v0.2 cost $0.000115 per case (trace estimate) and Claude Opus 5 $0.0156 at batch prices: about 136× more for Claude.
On the same 100 gold cases with the same questions, Claude Opus 5 cost $0.0156 per case at batch prices and Jev q-v0.2 $0.000115: Claude cost about 136× as much. Their correct actions (93/100 and 91/100) have overlapping 95% intervals (86.1–97.1% and 83.6–95.8%). Jev q-v0.2's gold run is in no ledger, so its cost is the sum of its traces' estimated cost, which matches the ledger wherever a Jev run is in one. Jev q-v0.3, whose gold run is in the ledger, cost $0.000175 per case, so Claude cost 89× as much.
- Claude Opus 5, batch
- $0.0156
- Jev q-v0.2, sync (trace estimate)
- $0.000115
Claude cost 136× as much per case. The Jev bar is drawn at least 2px wide so it stays visible.
Show details
Across the datasets both providers ran, Claude cost 89× to 146× as much per case as Jev. At the median, Claude (sync) took 5,169 ms per case and Jev 178 ms, about 29× longer, on a 10-case sample.
Claude ÷ Jev, per case
| Dataset | Claude | Jev | Ratio | Note |
|---|---|---|---|---|
| gold-v0.1 | $0.0156 · n 100 | $0.000115 · q-v0.2 | 136× | The same questions Claude was asked; the headline pair. Jev's cost is a trace estimate. |
| gold-v0.1 | $0.0156 · n 100 | $0.000175 · q-v0.3 | 89× | Jev's costlier question set; its cost is in the ledger. |
| gen-v0.2-dev | $0.0110 · n 400 | $0.000112 · q-v0.2 | 98× | |
| gen-v0.2-holdout | $0.0164 · n 150 | $0.000112 · q-v0.2 | 146× | Claude ran a 150-case sample; Jev ran all 1000. |
Every Jev and Claude run
| Provider | Question set | Mode | Run | Cases | Total | Per case |
|---|---|---|---|---|---|---|
| Jev | q-v0.2 | sync | b95be9 · gold-v0.1Sum of the traces' estimated cost; not in a committed ledger | 100 | $0.0115 | $0.000115 |
| Jev | q-v0.3 | sync | ad6f44 · gold-v0.1Jev spend ledger | 100 | $0.0175 | $0.000175 |
| Claude Opus 5 | q-v0.2+claude-prompt-v1 | batch | f1852f · gold-v0.1Claude spend ledger | 100 | $1.56 | $0.0156 |
| Jev | q-v0.1 | sync | eee114 · smoke-v0.1Sum of the traces' estimated cost; not in a committed ledger | 10 | $0.00109 | $0.000109 |
| Claude Opus 5 | q-v0.2+claude-prompt-v1 | sync | f328de · smoke-v0.1Claude spend ledger | 10 | $0.255 | $0.0255 |
| Jev | q-v0.2 | sync | 6f0b73 · gen-v0.2-devSum of the traces' estimated cost; not in a committed ledger | 400 | $0.0448 | $0.000112 |
| Jev | q-v0.1 | sync | d6b218 · gen-v0.2-devSum of the traces' estimated cost; not in a committed ledger | 400 | $0.0442 | $0.000111 |
| Claude Opus 5 | q-v0.2+claude-prompt-v1 | batch | 288946 · gen-v0.2-devClaude spend ledger, two batches: d65789 (378 cases, $4.123975) + b1c45c (22 cases, $0.274699); 22 cancelled cases were re-run and billed once | 400 | $4.40 | $0.0110 |
| Jev | q-v0.2 | sync | fd455f · gen-v0.2-holdoutSum of the traces' estimated cost; not in a committed ledger | 1000 | $0.112 | $0.000112 |
| Claude Opus 5 | q-v0.2+claude-prompt-v1 | batch | bbee49 · gen-v0.2-holdoutClaude spend ledger | 150 | $2.46 | $0.0164 |
| Jev | q-v0.3 | sync | cdaf0c · gen-v0.3-devJev spend ledger | 400 | $0.0691 | $0.000173 |
| Jev | q-v0.2 | sync | e950c0 · gen-v0.3-devJev spend ledger | 400 | $0.0452 | $0.000113 |
| Jev | q-v0.3 | sync | 12e1e4 · gen-v0.3-holdoutJev spend ledger | 1000 | $0.172 | $0.000172 |
| Jev | q-v0.2 | sync | 204814 · gen-v0.3-holdoutJev spend ledger | 1000 | $0.113 | $0.000113 |
| Jev | q-v0.3 | — | bd430c · gen-v0.3-shiftRecomposed from run_20260927T072915Z_007c7c's stored answers; no new provider calls | 400 | — | — |
| Jev | q-v0.3 | — | e5915e · gen-v0.3-shiftRecomposed from run_20260927T072915Z_007c7c's stored answers; no new provider calls | 400 | — | — |
| Jev | q-v0.3 | sync | 007c7c · gen-v0.3-shiftJev spend ledger | 400 | $0.0696 | $0.000174 |
Latency per case
| Provider | Run | Cases | p50 ms | p95 ms |
|---|---|---|---|---|
| Claude Opus 5 (sync) | f328de · smoke-v0.1 | 10 | 5,169 | 6,781 |
| Jev q-v0.1 | eee114 · smoke-v0.1 | 10 | 178 | 411 |
On the same 10 smoke cases, Claude Opus 5 (sync) took 5,169 ms p50 and 6,781 ms p95 per case; Jev q-v0.1 took 178 ms and 411 ms. In the parallelism bench (40 calls per size), Jev's p50 stayed between 177 and 191 ms with 1 to 20 questions per call. The Claude figure is a 10-case sample. The bench.
Total spend: Claude $8.71 (ledger); Jev $0.715 ($0.502 in the Phase 3D ledger and $0.214 in trace estimates for the Phase 2 runs).
Caveats
- Claude was run as Opus 5 (claude-opus-5), the most capable and most expensive tier. Cheaper Claude models (Sonnet, Haiku) were not tested and would narrow the gap.
- Claude's batch figures already include the 50% Message Batches discount; at non-batch prices they would be about twice as high.
- “Similar accuracy” means not distinguishable on 100 gold cases, not proven equal.
- Prices are as billed in September 2026: Claude at list prices as of 2026-09-25, Jev at $0.042 per million input tokens.
- The Claude latency sample is 10 cases (one sync run on the smoke set); Claude's batch runs record no latency.
- Jev's Phase 2 runs are in no ledger; their cost is the sum of the traces' own estimates, which match the ledger wherever a Jev run is in one.