Skip to content

Synthetic prior-authorization cases · evaluation prototype

Relay turns a model’s probabilities into actions software is allowed to take, and records why.

Relay evaluates synthetic medication-coverage requests. Jev answers narrow questions about the evidence with probabilities; a versioned policy engine, not the model, decides what happens to the case.

of held-out cases auto-processed
24.4%244 of 1000 cases
unsafe automations observed
0 of 24495% upper bound 1.5%
chose the correct action
92.1%921 of 1000 cases

Jev q-v0.3 @0.81 on gen-v0.3-holdout, 1000 synthetic cases, run once.

Cost per case on gold, same questions: Jev q-v0.2 $0.000115 (trace estimate) · Claude Opus 5 $0.0156 (batch), about 136× more, at similar gold accuracy. Cost and speed →

These results measure agreement with synthetic labels, not clinical validation. Gold labels were written and checked by AI agents, not clinicians.

See the paused-treatment failure and fix →

Full results table· 7 runs

Each provider runs at its own auto_process threshold, chosen on a dev set before the holdout or gold run. Unsafe means the engine auto-processed a case whose expected action was not AUTO_PROCESS. The bound is the exact 95% Clopper-Pearson upper limit on the unsafe rate.

  • Jev q-v0.2@0.89
    gen-v0.2-holdout · n 1000
    Correct action
    895/1000 (89.5%)
    Automation
    252/1000 (25.2%)
    Unsafe / auto
    0/252
    Unsafe 95% upper
    1.5%
  • Rules@0.99 flat
    gen-v0.2-holdout · n 1000
    Correct action
    667/1000 (66.7%)
    Automation
    134/1000 (13.4%)
    Unsafe / auto
    0/134
    Unsafe 95% upper
    2.7%
  • Jev q-v0.3@0.81
    gen-v0.3-holdout · n 1000
    Correct action
    921/1000 (92.1%)
    Automation
    244/1000 (24.4%)
    Unsafe / auto
    0/244
    Unsafe 95% upper
    1.5%
  • Jev q-v0.2@0.89
    gold-v0.1 · n 100
    Correct action
    91/100 (91.0%)
    Automation
    29/100 (29.0%)
    Unsafe / auto
    1/29 (3.4%)
    Unsafe 95% upper
    17.8%
  • Claude@0.55
    gold-v0.1 · n 100
    Correct action
    93/100 (93.0%)
    Automation
    30/100 (30.0%)
    Unsafe / auto
    1/30 (3.3%)
    Unsafe 95% upper
    17.2%
  • Rules@0.99 flat
    gold-v0.1 · n 100
    Correct action
    61/100 (61.0%)
    Automation
    20/100 (20.0%)
    Unsafe / auto
    6/20 (30.0%)
    Unsafe 95% upper
    54.3%
  • Jev q-v0.3@0.81
    gold-v0.1 · n 100 (not blind)
    Correct action
    94/100 (94.0%)
    Automation
    31/100 (31.0%)
    Unsafe / auto
    1/31 (3.2%)
    Unsafe 95% upper
    16.7%
  • All data is synthetic. Gold labels were written and checked by AI agents, not clinicians.
  • Claude’s holdout evidence is a 150-case sample, so it is not in the table.
  • Gold is not a blind test for q-v0.3: GOLD-TMP-17 motivated that question set.
  • Provider spend: Claude $8.71 for every Claude run; Jev $0.50 for the q-v0.3, policy-shift and latency runs.

How a case moves

1 CASE DOCUMENTSphysician note2 JUDGMENTS (JEV Q-V0.2 @0.89, GOLD-STR-01)Diagnosis supported 0.980Step therapy met 0.980Documentation complete 0.920Material contradiction 0.060Missing evidence NONE 0.8503 POLICY ENGINEprovider passedage passedcontradiction passeddocumentation passedmissing_evidence passedauto_process FIREDdefault_review not reached4 ACTIONAUTO_PROCESS

What we found

Confidence gating automated about a quarter of held-out cases with no unsafe approvals observed.

Jev q-v0.3 at 0.81 on gen-v0.3-holdout auto-processed 244/1000 (24.4%) of cases, 0/244 unsafe (95% upper bound 1.5%), and chose the correct action for 921/1000 (92.1%). On gen-v0.2-holdout the rules baseline automated 134/1000 (13.4%) with 667/1000 (66.7%) correct, against 252/1000 (25.2%) and 895/1000 (89.5%) for Jev q-v0.2. The cases are synthetic and template-generated.

The holdout runMethod in RESULTS.md

The evaluation loop found a real design flaw, and the fix held on a fresh holdout.

GOLD-TMP-17, an interrupted methotrexate course, was auto-approved unsafely by both Jev q-v0.2 at 0.89 and Claude at 0.55. The cause was the question design: q-v0.2 asks for one start date and one end date, so a paused course reads as one long course. q-v0.3 adds questions about pauses and restarts; on gen-v0.3-holdout, run once, correct actions went from 694 to 921 of 1000. Gold is not a blind test for this change, and on gold GOLD-TMP-16 became newly unsafe when the threshold dropped from 0.97 to 0.81.

GOLD-TMP-17Method in RESULTS.md

The contradiction gate is doing real work, and it has a cost.

With contradiction detection disabled, GOLD-CON-03 and GOLD-CON-13 become newly unsafe on gold for all 3 model configurations. On the holdouts it only costs automation: Jev q-v0.2 goes from 252 to 277 of 1000 automated and Jev q-v0.3 from 244 to 261 of 1000, still 0/277 and 0/261 unsafe. Disabling the missing-evidence gate changed no automation and no unsafe count in any of the 10 runs, a null result.

The ablationMethod in RESULTS.md

Policy-aware logic matters when the policy changes.

Under immunara-v0.2's new 12-month recency rule, on 400 cases, the same stored Jev answers composed under the old policy gave 3/16 unsafe automations and under the new policy 0/13. The stale-to-aware gate passes, and shadow mode recommends PROMOTE.

The policy shiftMethod in RESULTS.md

At similar gold accuracy and with the same questions, Jev cost about 1/136th as much per case as Claude Opus 5.

On gold, with the same questions, Claude got 93/100 correct with 30 automated and 1 unsafe; Jev q-v0.2 got 91/100 with 29 automated and 1 unsafe. The 95% intervals on correct actions (86.1–97.1% and 83.6–95.8%) overlap. This 100-case comparison does not establish equivalence or a clear winner. Claude's gold run cost $1.56 at batch prices, $0.0156 per case; Jev q-v0.2's gold run cost $0.0115, $0.000115 per case, a trace estimate (its run is in no ledger; the estimate matches the ledger wherever both exist). Jev's q-v0.3 gold run, with more questions per case, cost $0.0175 in the ledger, $0.000175 per case.

Cost and speedMethod in RESULTS.md

Start with a case

  • A design failureGOLD-TMP-17Jev and Claude both missed a treatment pause; changing the questions resolved this case.
  • An easy caseGOLD-STR-01Explicit dates; every provider auto-processes it, correctly.
  • A hard caseGOLD-CON-03The note and the medication history disagree; only the rules baseline automates it.

Or browse all 110 cases.