Skip to content
Agentic Engineering Fieldbook

Coding and Software Engineering Agents

Coding agents are tempting because software work already has text, tools, tests, version control, and review. A model can read an issue, inspect files, edit code, run commands, observe failures, and try again. That makes software engineering one of the clearest domains for agentic systems.

It is also one of the easiest domains to misunderstand. A generated patch is not a safe change. A passing local test is not a release decision. A plausible explanation is not evidence that the repository behavior is correct. Coding agents should therefore be designed as systems that produce reviewable change candidates with evidence, not as systems that turn natural language directly into trusted software.

Thesis

Section titled “Thesis”

A coding agent is fit for delegation when the task can be bounded as a change candidate, the repository environment can supply the needed context and tools, and acceptance depends on inspected diffs, executable verification, and release gates rather than the agent’s own claim of success.

The central product object is the change candidate: a branch, patch, pull request, or equivalent artifact that can be inspected, tested, reverted, and connected to a request. Treat the agent’s conversation as useful process evidence, but do not let it replace the diff, tests, provenance, and accountable review.

The task boundary is a change, not a wish

Section titled “The task boundary is a change, not a wish”

Good coding-agent tasks have a concrete target:

  • fix this failing test;
  • implement this API field according to this specification;
  • refactor this module without changing externally visible behavior;
  • update these dependencies and resolve compatibility failures;
  • add instrumentation around this workflow;
  • investigate this bug and return either a patch or a falsified hypothesis.

Poor tasks ask the agent to absorb accountability that belongs to product, architecture, or release ownership:

  • make the system better;
  • clean up the codebase;
  • improve performance everywhere;
  • migrate the stack;
  • make this secure;
  • ship this feature.

The difference is not only size. A small edit can be unsafe when it touches authentication, authorization, billing, privacy, or migrations. A larger mechanical change can be safe when its surface, checks, and review criteria are explicit. The delegation question is whether the agent can produce an artifact whose correctness can be tested or reviewed by the normal engineering system.

Repository context is the agent’s working environment

Section titled “Repository context is the agent’s working environment”

Coding agents need more than the files named by the user. They need a repository context contract:

  • the requested behavior and the source of that request;
  • relevant files, generated code, schemas, tests, build configuration, and dependency constraints;
  • coding conventions, architectural boundaries, and ownership files;
  • failing commands and expected passing commands;
  • recent changes that may explain the issue;
  • security, privacy, and migration constraints;
  • files and directories that should not be touched.

SWE-agent’s results are a useful reminder that the interface between the agent and the computer materially shapes behavior (Yang et al., 2024). Command output, file navigation, edit mechanics, and observations are not incidental UI choices; they are part of the system being evaluated.

The context contract should also say what the agent must not infer. If the issue says “make login faster” but the repository contains multiple login flows, the agent should identify the ambiguity rather than changing the first path it finds. If tests require credentials or network access, the agent should report that verification is incomplete rather than claiming success from static inspection.

The control loop is edit, run, observe, narrow

Section titled “The control loop is edit, run, observe, narrow”

A capable coding agent often follows a loop:

  1. Restate the requested change and acceptance evidence.
  2. Inspect the relevant repository state.
  3. Form a small plan.
  4. Edit a bounded set of files.
  5. Run the most specific verification first.
  6. Read failures as evidence, not as embarrassment.
  7. Revise the patch or narrow the claim.
  8. Stop with a reviewable diff, test result, and residual risk.

The loop should be instrumented and bounded. Time, token, command, and file-touch budgets matter because coding agents can otherwise wander into broad refactors. A good stop state can be success, blocked, unsafe, needs decision, or no reproduction. “I fixed it” is not a terminal state unless the required checks passed and the artifact is ready for review.

For high-risk repositories, commands should run in a sandbox with explicit network, filesystem, and secret boundaries. The agent should not be able to deploy, rotate credentials, push directly to protected branches, modify billing configuration, or run destructive commands merely because it can produce the syntax.

Verification has layers

Section titled “Verification has layers”

Coding tasks have unusually rich verification options, but each layer has limits:

Verification layer What it proves What it does not prove
Static checks Syntax, types, formatting, obvious lint policy Runtime behavior, missing requirements, business correctness
Focused tests The reproduced issue or new behavior under test Untested paths, integration effects, security properties
Full test suite Wider regression signal under the current environment Production configuration, flaky gaps, hidden acceptance criteria
Review diff Intent, maintainability, scope, ownership, risk Exhaustive behavior or absence of subtle bugs
Staging or canary Behavior under a more realistic environment Long-tail usage, full release safety, future provider drift

SWE-bench and SWE-bench Verified are valuable because they evaluate issue-level changes against repository tests rather than isolated code snippets (SWE-bench; OpenAI, 2024). Their lesson for production is not that a benchmark score proves readiness. It is that coding-agent evaluation should resemble the work: repository state, task instructions, environment interaction, and outcome checks.

Local verification should preserve three distinctions. First, a test that existed before the agent’s change is different from a test the agent wrote. Second, a passing test is stronger when the agent first reproduced a failure. Third, hidden or reviewer-owned checks remain valuable because agents can overfit visible tests just as humans can.

Review is part of the product

Section titled “Review is part of the product”

Coding-agent UX should make review easy. The reviewer should see:

  • the original request and acceptance criteria;
  • the files changed and why;
  • commands run, with pass/fail output summarized accurately;
  • tests added, deleted, or modified;
  • unresolved risks and assumptions;
  • generated files or dependency changes;
  • security-sensitive paths touched;
  • whether the agent used network access, secrets, or external tools.

OpenAI Codex and Claude Code both present current product patterns around repository workspaces, command execution, reviewable outputs, and permissioned operation (OpenAI Codex docs; Claude Code docs). Those product materials are useful evidence that the domain is moving toward workspace-oriented coding agents. They should not be read as proof that any particular task can skip review.

The review surface should distinguish “agent wrote this” from “system accepted this.” The former is authorship metadata. The latter is an engineering decision with owners and evidence.

Design options

Section titled “Design options”
Option Best fit Main risk
Inline code assistant Local completion, explanation, small edits User may accept plausible code without sufficient context
Task-scoped coding agent Bounded issue, bug fix, tests, small feature Agent may over-broaden scope or stop after weak verification
Repository maintenance agent Dependency updates, migrations, repetitive changes Repeated automated changes can overwhelm review and miss semantic risk
Release-authorized agent Very narrow, heavily gated internal automation Blurs change generation with release accountability

Most teams should start with task-scoped change candidates. Auto-merge should be reserved for narrow classes where ownership, tests, rollback, monitoring, and policy are already strong.

Failure modes

Section titled “Failure modes”
  1. Wrong problem, correct-looking patch. The agent changes code that matches keywords but not the user’s real failing path.
  2. Test theater. The agent writes or selects tests that prove its patch rather than the behavior.
  3. Over-broad cleanup. The agent mixes the requested change with refactors, formatting churn, or dependency updates.
  4. Environment confusion. Local commands pass under a different runtime, database, feature flag, or provider configuration than production.
  5. Hidden authority escalation. A tool that can edit files also has access to secrets, network, package publishing, or deployment credentials.
  6. False completion. The agent says work is done even though verification was skipped, failed, or only partially run.
  7. Review overload. The organization accepts larger volumes of machine-generated changes without increasing review evidence or ownership.

Fieldbook guidance

Section titled “Fieldbook guidance”

Use coding agents where the artifact can be made concrete. Give the agent a repository workspace, not unlimited organizational intent. Prefer small change candidates, explicit acceptance evidence, and deterministic release gates.

Require the agent to surface its verification path. If it could not run a command, it should say so. If it changed tests, it should identify that. If it found ambiguity, it should ask or return a bounded hypothesis. If it touched security-sensitive code, it should trigger stricter review.

Do not route around the engineering system. Version control, code review, CI, provenance, deployment controls, and incident learning exist because software changes have long tails. Coding agents can make those systems more productive, but they should not become invisible authors of unowned change.

Checklist

Section titled “Checklist”
  • Is the requested work expressible as a bounded change candidate?
  • Are repository context, excluded paths, dependency constraints, and ownership boundaries explicit?
  • Does the agent run in a sandbox with scoped filesystem, network, secret, and command authority?
  • Are destructive commands, protected branches, deployment, package publishing, and credential changes gated outside the model?
  • Does the agent record the plan, files touched, commands run, test results, and unresolved assumptions?
  • Are tests written by the agent distinguished from preexisting checks?
  • Does review include the diff, verification evidence, and risk-sensitive ownership, not only the agent’s summary?
  • Are auto-merge or auto-release paths limited to task classes with proven local evidence, rollback, and monitoring?

References and how they are used

Section titled “References and how they are used”