Cases
110 cases, every provider's action on each
100 gold cases in five categories and 10 smoke cases. Each provider is shown at its operating point. Open a case to see the documents, the five judgments and the gate that decided.
- AUTO, correct
- correct
- REVIEW, wrong but safe
- wrong, but a person or the submitter sees it
- AUTO, unsafe
- automated a case that needed review or information
| ADV-01 | Smoke | HUMAN_REVIEW | not run on this dataset | not run on this dataset | REVIEW, correct | REVIEW, correct | REVIEW, correct |
| ADV-02 | Smoke | HUMAN_REVIEW | not run on this dataset | not run on this dataset | REVIEW, correct | REVIEW, correct | INFO, wrong but safe |
| AUTO-01 | Smoke | AUTO_PROCESS | not run on this dataset | not run on this dataset | AUTO, correct | AUTO, correct | AUTO, correct |
| AUTO-02 | Smoke | AUTO_PROCESS | not run on this dataset | not run on this dataset | AUTO, correct | AUTO, correct | AUTO, correct |
| AUTO-03 | Smoke | AUTO_PROCESS | not run on this dataset | not run on this dataset | AUTO, correct | INFO, wrong but safe | REVIEW, wrong but safe |
| GOLD-CON-01 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | AUTO, unsafe | not run on this dataset |
| GOLD-CON-02 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-CON-03diff | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | AUTO, unsafe | not run on this dataset |
| GOLD-CON-04 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | AUTO, unsafe | not run on this dataset |
| GOLD-CON-05 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | AUTO, unsafe | not run on this dataset |
| GOLD-CON-06 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-CON-07 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | AUTO, unsafe | not run on this dataset |
| GOLD-CON-08 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-CON-09 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-CON-10 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | AUTO, unsafe | not run on this dataset |
| GOLD-CON-11 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-CON-12 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-CON-13diff | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-CON-14 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-CON-15 | Conflicting evidence | AUTO_PROCESS | REVIEW, wrong but safe | REVIEW, wrong but safe | REVIEW, wrong but safe | AUTO, correct | not run on this dataset |
| GOLD-CON-16 | Conflicting evidence | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-CON-17 | Conflicting evidence | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-CON-18 | Conflicting evidence | AUTO_PROCESS | REVIEW, wrong but safe | REVIEW, wrong but safe | REVIEW, wrong but safe | AUTO, correct | not run on this dataset |
| GOLD-CON-19 | Conflicting evidence | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-CON-20 | Conflicting evidence | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-MIS-01 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-02 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-03 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-04 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-05 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-06 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-07 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-08 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-09 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-10 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-11 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-12 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-13 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-14 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-15 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-16 | Missing information | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-MIS-17 | Missing information | AUTO_PROCESS | REVIEW, wrong but safe | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-MIS-18 | Missing information | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-MIS-19 | Missing information | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-MIS-20 | Missing information | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-STR-01 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-02 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-03 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-04 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-STR-05 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-STR-06 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-07 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-08 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-09 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-STR-10 | Straightforward | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-STR-11 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-12 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-13 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-14 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-15 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-16 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-17 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-18 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-19 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-STR-20 | Straightforward | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TMP-01 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-02 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-TMP-03 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-TMP-04 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-TMP-05 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-06 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-07 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-08 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-TMP-09 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-TMP-10 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-11 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-12 | Temporal reasoning | AUTO_PROCESS | REVIEW, wrong but safe | REVIEW, wrong but safe | REVIEW, wrong but safe | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-13 | Temporal reasoning | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-14 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-15 | Temporal reasoning | AUTO_PROCESS | REVIEW, wrong but safe | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-16diff | Temporal reasoning | REQUEST_INFO | REVIEW, wrong but safe | AUTO, unsafe | REVIEW, wrong but safe | INFO, correct | not run on this dataset |
| GOLD-TMP-17diff | Temporal reasoning | HUMAN_REVIEW | AUTO, unsafe | REVIEW, correct | AUTO, unsafe | REVIEW, correct | not run on this dataset |
| GOLD-TMP-18diff | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | REVIEW, wrong but safe | REVIEW, wrong but safe | not run on this dataset |
| GOLD-TMP-19 | Temporal reasoning | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TMP-20 | Temporal reasoning | AUTO_PROCESS | REVIEW, wrong but safe | REVIEW, wrong but safe | REVIEW, wrong but safe | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-01 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-02 | Tricky or ambiguous | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TRK-03 | Tricky or ambiguous | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-04 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-05 | Tricky or ambiguous | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TRK-06 | Tricky or ambiguous | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-TRK-07 | Tricky or ambiguous | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-08 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-TRK-09 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-10 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-11 | Tricky or ambiguous | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | AUTO, correct | not run on this dataset |
| GOLD-TRK-12 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-TRK-13 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| GOLD-TRK-14 | Tricky or ambiguous | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-15 | Tricky or ambiguous | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TRK-16 | Tricky or ambiguous | AUTO_PROCESS | AUTO, correct | AUTO, correct | AUTO, correct | INFO, wrong but safe | not run on this dataset |
| GOLD-TRK-17 | Tricky or ambiguous | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TRK-18 | Tricky or ambiguous | REQUEST_INFO | INFO, correct | INFO, correct | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TRK-19 | Tricky or ambiguous | REQUEST_INFO | REVIEW, wrong but safe | REVIEW, wrong but safe | INFO, correct | INFO, correct | not run on this dataset |
| GOLD-TRK-20 | Tricky or ambiguous | HUMAN_REVIEW | REVIEW, correct | REVIEW, correct | REVIEW, correct | REVIEW, correct | not run on this dataset |
| REV-01 | Smoke | HUMAN_REVIEW | not run on this dataset | not run on this dataset | REVIEW, correct | REVIEW, correct | REVIEW, correct |
| REV-02 | Smoke | HUMAN_REVIEW | not run on this dataset | not run on this dataset | REVIEW, correct | REVIEW, correct | REVIEW, correct |
| RI-01 | Smoke | REQUEST_INFO | not run on this dataset | not run on this dataset | INFO, correct | INFO, correct | INFO, correct |
| RI-02 | Smoke | REQUEST_INFO | not run on this dataset | not run on this dataset | INFO, correct | INFO, correct | INFO, correct |
| RI-03 | Smoke | REQUEST_INFO | not run on this dataset | not run on this dataset | INFO, correct | INFO, correct | INFO, correct |