Faster Code, Same Delivery: Five Eras to an AI Software Factory
Five nested eras explain why faster code generation still misses on delivery, and why each remaining human decision weighs more as time per change shrinks.

Prompt engineering, context engineering, harness engineering, and loop engineering nest inside an AI software factory. Human time per change shrinks. The weight of each human gate grows.
Coding agents are raising output faster than a team can review, and the instability shows up after the demo. The weight of that work did not vanish; it moved onto gates most teams have not built.
In this article: You will see why coding agents raise output faster than they raise delivered value, and how five nested eras each answer a failure of the one before. The piece keeps the load-bearing evidence, lays an AI software factory out as ten stages with a named person at every gate, and ends with what to train, what to measure, and what not to automate yet.
Coding agents already write production code. The evidence on delivery is mixed, and the mix is the whole problem. A randomized trial gave 16 experienced open-source developers 246 tasks. Early-2025 AI tools increased their completion time by 19%, although they had forecast a 24% speedup and afterward believed they had been faster [1]. A follow-up, with 57 developers and more than 800 tasks, estimated smaller gains whose confidence intervals cross zero. The lab calls those data "an unreliable signal" [5]. A separate trial with 96 engineers at Google found a 21% reduction in task time [6]. Speed, when it shows up, is not the same thing as a stable release.
The DevOps Research and Assessment (DORA) program finds that AI "now improves software delivery throughput" but "still increases delivery instability," and it calls AI an amplifier of the strengths and weaknesses a team already has [2]. Generation scales. Human review does not. Anthropic's playbook for an AI-native lifecycle states the result in one line: when agents multiply code output, "either the review queue builds or code ships under-reviewed" [4]. The gap between output and value is a review and verification problem, not a generation problem.
The work moves to both sides of the build. The playbook says "the bottleneck moves to the steps to the left and right of the build phase," listed as "plan, review/test, and deploy, which still run at human speed" [4]. Intents and acceptance criteria have to be approved fast enough to feed the agents, and the review queue grows because "security teams are sized for human output" [4]. If the saved effort does not go into those two backlogs, the factory only produces a larger pile of unreviewed diffs.
Each era of AI-assisted development moved one more kind of work from the person to the system, and each time the human decisions that remained carried more weight. Prompt engineering improved the wording of one request. Context engineering controls what the model sees. Harness engineering adds behavior the system enforces, whatever the model decides. Loop engineering lets an agent repeat checked cycles until a stop condition holds. An AI software factory chains those loops into one pipeline with people at the gates [4].

No era replaces the one before it. OpenAI's harness account still curates the AGENTS.md files its agents read, so the harness era keeps doing context engineering [7]. The nesting is a framing: each layer got its name when the layer below it stopped being where the failures lived.
Context is the spine. Models lose accuracy as input grows, they use the start and end of a long input better than the middle, and they drift when a task arrives in pieces [3][8][9]. Every later era is, in part, a better answer to one question: what should one model see at one moment?
One feature runs through the argument, small enough to follow and risky enough to need every gate. A workspace export lets administrators download the data in their current workspace for an audit. A missing filter would leak one workspace's data into another's export. Treat the walkthrough as an illustration of the gates, not as a case study with measured outcomes.
Five eras, and the failure that forces the next

The five eras differ in what counts as one unit of work, what the engineer builds, and what breaks. The Era column names the era. The Unit of work column gives the scope of one piece of work. The column headed The engineer builds lists what the engineer makes in that era. The Failure that forces the next era column names the limit that pushes teams onward.
- 1. Prompt engineering works on one prompt and one answer. The engineer builds instructions, examples, and output formats. The model knows only what fits in the prompt, and it cannot check its own answer.
- 2. Context engineering works on what the model sees for one task. The engineer builds retrieval, rule files, progressive disclosure, skills, and notes. More context is not better context, and guidance is only advice.
- 3. Harness engineering works on one agent session. The engineer builds hooks, tools, a sandbox, and subagents with their own context. A person still starts every task and watches every session.
- 4. Loop engineering works on one repeating cycle. The engineer builds the trigger, goal, verification, stop rule, memory, budget, and escalation. Loops run alone, and nothing chains them or tracks the work between them.
- 5. Software factory works on one pipeline of loops. The engineer builds committed artifacts per stage, risk tiers, enforced gates, and a provenance trail. Its open failure is the dark factory, which removes the human gates, and the evidence does not support that.
Each era moves a rule from advice the model reads to behavior the system enforces.
Era 1: prompt engineering
Prompt engineering is the craft of writing instructions, examples, roles, and output formats so that one request gets one good answer. It made large language models useful for single tasks. The Claude Code team describes each prompt as a manual loop in which a person directs every turn [10]. The model had no tools, no memory between sessions, and no way to check its own answer, so a person checked every output and pasted the next instruction.
Asking the model to check itself is not a check. Intrinsic self-correction, where a model revises its own answer without outside feedback, consistently lowered accuracy on reasoning benchmarks. On GSM8K, a set of grade-school math problems, GPT-4 fell from 95.5% to 91.5% [11]. On factual answers, models that judged their own outputs scored about 54% on HotpotQA, close to chance, and the authors argue for tool-based critique instead [12].
A sound outside verifier changes the result. On Game of 24, GPT-4 scored 5% with standard prompting, 3% with self-critique, and 38% with an external verifier. On Blocksworld planning, it scored 40%, 55%, and 87% [13]. The lesson carries through every later era: the thing that writes should not be the thing that grades. In a factory, that lesson becomes the Review stage.
Prompt engineering still went further than one request at a time. Chain-of-thought prompting asked the model to write out "a series of intermediate reasoning steps." With eight worked examples, a 540-billion-parameter model reached state-of-the-art accuracy on GSM8K [53]. Teams broke work into steps ahead of time, wrote the steps down, and tuned the prompt until the results improved. Reasoning models then took over much of that planning. Anthropic's current prompting guidance says that a prompt such as "think thoroughly" often "produces better reasoning than a hand-written step-by-step plan" [54]. A better plan inside the prompt still could not show the model what its answer did.
Feedback from the world was the step that mattered. ReAct interleaved reasoning with actions that "interface with external sources, such as knowledge bases or environments." On two interactive benchmarks it beat imitation and reinforcement learning methods by 34 and 10 points of absolute success rate [55]. SWE-agent applied the same idea to code. Its interface limited what the model saw, through a capped search and a file viewer that shows 100 lines at a time, and its linter returned errors at edit time as the agent's next observation [25]. Validators checked each result and fed it back. The chat box became an interface to a computer, and harness engineering begins there.
Prompt engineering does not scale past one person and one task. Each session starts from zero, and the person carries the project's facts in their own head. The next problem is not better wording. It is the right information, in the right amount, at the right time [14]. Prompts survive inside every later stage's instructions. No stage should trust a prompt alone.
Shrinks for the human: typing code by hand. Grows for the human: wording each request, and checking every output by eye.
Era 2: context engineering
Context engineering is the discipline of controlling what a model sees, when it sees it, and what it discards. Anthropic describes context as "a finite resource with diminishing marginal returns" and gives the goal as "the smallest possible set of high-signal tokens" for the task [14]. The figure below is a schematic of the two effects that make the budget finite. It is not a chart of study values.

More context is not better context. Past a point, added tokens cost accuracy, and every token competes for a finite attention budget [14]. An agent can stay under its limit and still lose the fact that mattered. In the author's words, the window never overflowed, and the agent still forgot. Two versions of the same decision leave the model unable to tell which applies, and a stale page is read as current fact.
Context rot. Accuracy falls as input grows. Chroma tested 18 models and found that "model performance consistently degrades with increasing input length," even on simple tasks, and that "even a single distractor reduces performance relative to the baseline" [3]. Chroma calls this context rot. The report is a vendor technical report, not a peer-reviewed study.
The U-shaped attention curve. Position matters as well. Language models use information at the start or end of a long input better than information in the middle [8]. Chroma's own needle test found no notable position effect, so the U-shape in the schematic rests on the Lost in the Middle study, not on Chroma [3]. In a long session, constraints given at the start drift into the middle, the place the model attends to least. A later study measured input length against each model's own context window and found the lost-in-the-middle effect "is strongest when inputs occupy up to 50% of a model's context window." Beyond that, "the primacy bias weakens, while recency bias remains relatively stable" [58]. In a nearly full window, even the start fades. Restate key constraints near the end, or start fresh from a written spec.
Context panic. A filling window changes behavior as well as recall. Anthropic reports that some models show "context anxiety," wrapping up early as they approach what they believe is their context limit [36]. The agent rushes, skips steps, or claims it is done. The fix was a harness change: context resets that hand the work to a fresh agent through a written handoff [36]. An instruction that lived only in the conversation can be lost when the window is compacted. If it matters, write it down.
A life-threatening shellfish allergy stated early in a long session will sit in the middle later, and the risk may be a sauce rather than a shrimp. Restating it helps. A deterministic ingredient filter does not depend on memory. Guidance the model reads becomes behavior the system enforces.
Long-context scores overstate what models can use. Most long-context tests hide a fact and then ask about it in the same words, so the model can match keywords. NoLiMa removed that shortcut. A question might ask which character has been to Helsinki, while the hidden fact says only that a character lives next to the Kiasma museum. The study tested 13 models that claim at least 128,000 tokens of context. Without matching words, 11 of them fell below half of their short-context score at just 32,000 tokens [15].
Spreading a task across turns lowers accuracy. Across 15 models and more than 200,000 simulated conversations, every model did worse when the same task arrived in pieces instead of in one complete message, with an average drop of 39% [9]. A wrong turn early is rarely recovered. Settle the whole task in writing first, then hand it over in one piece. A strong version is interview, spec, then a fresh session: the agent interviews the developer, writes the specification to disk, and a new session with a clean window builds from it. In a factory, this is Shape.
Check what is filling the window before blaming the model. Start a fresh session per task, keep side questions out of the main thread, and compact before the agent starts to rush.
Retrieval still earns its place, inside a budget. Retrieval-augmented generation combines a model with an external store and brings a passage in just in time [16]. Early chunking lost meaning at the boundary: "revenue grew 3%" no longer says which company or which quarter. Anthropic's contextual retrieval prepends a short, model-written summary that situates each chunk at index time. Combined with keyword search, this cut failed retrievals by 49%, and by 67% with reranking [62]. Retrieval is not free extra context. In one study, most models' accuracy rose and then fell as retrieved input grew past tens of thousands of tokens [17]. General embeddings also miss domain vocabulary. On the FinMTEB finance benchmark, a model's rank on the general benchmark did not predict its rank on finance text [18]. Small, linked, current pages beat a raw corpus chunked after the fact.
Rule files carry what the code does not show. AGENTS.md is an open format its site describes as "a README for agents," now stewarded by the Agentic AI Foundation under the Linux Foundation [19]. Agents read the nearest file in the directory tree, so each package can carry its own instructions [19]. OpenAI recommends keeping AGENTS.md to about 100 lines, a table of contents into deeper documents, because large instruction files rot and crowd out the task [7]. Lead with mistakes made twice, and write rules a reader can check.
Path-scoped rules load only where they apply. Claude Code takes a paths list in a rule file's frontmatter [59]. Cursor takes globs [60]. GitHub Copilot takes an applyTo field [61]. The agent editing a React component never reads the Terraform conventions, and an architecture decision record can load only when the agent touches the code it governs.
Progressive disclosure loads detail only when a task needs it. The Agent Skills specification loads each skill's name and description at startup, about 100 tokens, then the full SKILL.md body when the skill activates, then bundled files only as needed [20]. Anthropic calls this progressive disclosure [21]. A study of long-context agents found that it lets an agent reach a small fraction of a large corpus while keeping high accuracy [22].
Notes carry work across windows. Anthropic recommends structured note-taking outside the context window, reloaded later [14]. LangChain calls the pattern a scratchpad: "a tool call that simply writes to a file" or "a field in a runtime state object that persists during the session" [63]. Superseded decisions stay linked and dated. A model's library knowledge is stale in the same way. It stops at the training cutoff, so pull current documentation at run time instead of trusting memory.
A rule file is still advice. Claude Code's hooks give "deterministic control: certain actions always happen rather than relying on the LLM to choose to run them" [64]. Codex tells teams to pair AGENTS.md with pre-commit hooks, linters, and type checkers that catch issues before a person sees them [23]. Specs keep each request inside the budget. The model can still ignore guidance until something outside the model enforces it.
Shrinks for the human: pasting background into every request. Grows for the human: curating what is current, dated, and true.
Era 3: harness engineering
Harness engineering names the discipline the earlier eras were already building: an interface to a computer [25], notes and path-scoped rules, and hooks that enforce a rule the model may ignore [64]. It does not replace those eras. The harness still writes prompts, lets the model reason, and assembles context. It also decides what the model may do, checks what it did, and remembers what happened.
Most teams already use a harness. A coding agent such as Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, or Grok Build is a model wrapped in tools, permissions, hooks, context management, and a loop. Frameworks such as the Claude Agent SDK, LangChain, and LangGraph supply the same parts for agents a team builds itself. The question is whether the team designs the harness it already has.
The agentic loop has three phases: gather context, act, and verify. Without a verifier that returns a fact, a passing test or a broken build, the agent reports success because nothing contradicted it, and the session turns back into a chatbot.
The harness moves the score, with the model held fixed. SWE-agent kept GPT-4 Turbo constant and changed only the agent-computer interface: a capped search, a file viewer of 100 lines, and a linter that rejects broken edits. On SWE-bench Lite that interface solved 10.7 percentage points more tasks than the same model with a plain Linux shell, a 64% relative increase [25]. Epoch AI found that on SWE-bench Verified, "simply switching the scaffold makes up to an 11% difference for GPT-5 and up to a 15% difference for Kimi K2 Thinking" [66]. LangChain changed only the harness around its coding agent and raised its Terminal Bench 2.0 score by 13.7 points, from 52.8 to 66.5 [65]. On the same leaderboard in April 2026, Claude Opus 4.6 scored 65.4% on its own and 81.8% inside ForgeCode's multi-agent harness: the same model, with no fine-tuning, 16 points apart [67]. A benchmark score belongs to a model inside a harness. In the author's framing, the model is the CPU and the harness is the operating system.
LangChain states the relationship as "Agent = Model + Harness," and lists prompts, tools and skills, a filesystem and sandbox, orchestration, and hooks [24]. Context engineering supplies guidance the model reads. Harness engineering adds behavior the system enforces.
The idea is older than language models. In 1947, Fitts and Jones found that many so-called pilot errors were "really due to the design characteristics of aircraft instruments," and that proper instrument design could remove most of them [56]. SWE-agent treats language-model agents as "a new category of end users with their own needs and abilities" [25]. Jackie Stewart's mechanical sympathy, "the understanding a driver needed to have for a car's capabilities in order to drive it well," is the same idea in software [57]. Harness engineering is that sympathy for a model's attention budget, window, and failure modes. In the author's framing, when the operator keeps making the same mistake, the environment is the variable.
Mitchell Hashimoto named the practice in February 2026: after each mistake, build a fix "such that the agent never makes that mistake again" [26]. Birgitta Böckeler splits the harness into guides before the action and sensors after it, and puts the human's job as "to steer the agent by iterating on the harness" [27].

Return to the allergy. A rule file and a restatement near the end of the prompt are still advice. A hook is ordinary code, so it runs the same check every time around a model that will not. Before an ingredient is added, sauces included, the hook blocks a match. The model can still vary. The outcome on that rule cannot. The same pattern covers a protected file, a forbidden command, or a test the agent must not edit. For the workspace export, it is a lint rule that fails any data-layer query without a workspace filter.
Hooks also protect the window. A query that returns ten thousand rows, or a log search that returns a megabyte of text, brings on context rot. A wrapper after the call can cap the rows, keep the columns the task needs, or replace the raw output with a summary and a pointer to the full result on disk. SWE-agent's cap of 50 search results did the same job [25].
The major coding agents document the same primitives. The differences that matter are failure behavior, from each tool's own documentation as read on 2026-11-06. Codex states that a PreToolUse error, timeout, or malformed response can let the tool run [23]. GitHub's cloud agent fails closed on exit code 2 and on crashes, and it treats an "ask" as a deny because no user is present [29]. Cursor calls its run modes "best-effort guardrails rather than a hard security boundary" [28]. A team that trusts a hook has to know what happens when the hook itself fails.
Rules that must never be skipped belong where engineers cannot turn them off. Claude Code applies managed settings "above every other level, so no user, project, local, or --settings value overrides them," apart from a few named exceptions [31]. A hook can block edits to test files during a bug fix, so the agent fixes the code and not the test [4].
Tools and Model Context Protocol (MCP) servers are the agent's eyes and arms. An agent that can drive a browser, query a database, and read logs can show that a change works instead of asserting it. Anthropic's long-running agent marked features complete without proper testing, and verified web features well once prompted to test in a browser as a person would [33].
The sandbox moves the human to the boundary. Filesystem isolation limits what the agent can read and change. Network isolation limits which servers it can reach. In Anthropic's internal use, sandboxing "safely reduces permission prompts by 84%" [34]. The risk on the other side is "approval fatigue," in which people stop checking what they approve [34]. The 84% figure is a vendor's internal measurement, not a controlled study.
Subagents keep the lead window small. One explores or verifies in a fresh window and returns 1,000 to 2,000 tokens, so the noise never reaches the lead agent [14][68]. A lead agent with parallel subagents outperformed a single agent by 90.2% on an internal research eval, at about 15 times the tokens of a chat [35]. The window is a scratch pad. Durable state lives in files. Claude Code's Explore subagent is read-only, and its Plan subagent gathers context before a plan is shown [68].
Separation makes review honest. Agents asked to judge their own work "tend to respond by confidently praising the work," and separating the maker from the judge "proves to be a strong lever" [36]. Cognition reports that a dedicated review agent works best when it shares no context with the coding agent beforehand [37]. A harness still waits for a person to start every task and to watch every session. The next era takes the person out of the turn, not out of the decision.
Which layer fixes which failure. Match the fix to the failure. In a factory, Improve uses this routing.

Each repeated failure maps to one layer of the harness. The Failure column describes the symptom. The Layer that fixes it column names where the fix belongs.
- You keep re-explaining a project fact: put it in a rule file.
- The fact matters only for some files: use a path-scoped rule.
- You keep retyping the same procedure: turn it into a skill.
- It must happen every time, whatever the model decides: enforce it with a hook.
- The agent cannot see the result of its work: give it a tool or an MCP server.
- Reading and searching flood the main window: move the work to a subagent.
- The setup is stuck in one repository: package it as a plugin or a shared configuration.
Shrinks for the human: permission prompts, re-explaining the project, and reading raw exploration. Grows for the human: deciding what the agent may touch, and what is enforced rather than advised.
Era 4: loop engineering
A team cannot skip a layer. The harness makes one session reliable. A loop takes the person out of each turn by wiring those same pieces to run without someone typing the next prompt. The trigger is a repeating process or an event: a schedule, a webhook, a workflow, or a ticket changing state. Headless mode starts the agent. Hooks, tests, and tools verify. Notes on disk are the memory. A subagent in a fresh window is the checker. What the team adds is a goal the system can check, and a rule for when to stop.
A loop can only check what its tools can see. Built-in tools run tests and the build. MCP servers reach logs, database rows, and a browser. Without them, verification falls back to the agent's own word, and a person still checks by hand after every change. The loop only looks automatic. Tools are what move that person into work that needs judgment.
A loop is an agent that repeats cycles until a stop condition holds [10]. A loop specification names four parts, plus memory on disk: a trigger, a goal, an execution and verification step, and a stopping rule [38]. Each missing part breaks the loop in its own way.

Most loops in the wild lack those parts. A mining study of 36,710 repositories confirmed autonomous agent loops in 217 of 256 inspected repositories, 0.59% of all analyzed. Committed loop files mostly recorded triggers. Goals appeared in 55 of the 256, budgets in 101, and verifiers in 76 [39].
A goal command iterates until the budget is gone unless a specification, a test suite, and verifiers return real signal.
A loop must know how to stop. With no stopping rule, no stall check, and no budget ceiling, a loop circles until the money is gone [38]. Name the terminal states: done, because the check passed; stuck, because several cycles made no progress; or over budget, because time, turns, or cost hit a limit. The last two escalate to a person. They do not claim success.
The flow matters as much as the model. AlphaCodium ran GPT-4 through a flow that reasons about the problem, writes tests, writes code, and iterates on the failures. On the CodeContests validation set it raised accuracy (pass@5) from 19% with a single well-designed prompt to 44%. On the test set, it rose from 13% to 28% [69]. The model was the same.
The model classifies; code grades. Let the model report what it observes, such as which acceptance criteria a draft meets, and let ordinary code turn that report into pass, retry, or escalate. "Stop after three tries" in a prompt is a suggestion. A counter in a program is a rule. The gate is only as good as the facts its tools return.
Done needs a receipt. Did the checks pass, on this exact tree, after the last edit? Missing evidence is a failure. A gate with only "tests passed" can be satisfied by one trivial test and a deleted suite, so tests, coverage, lint, types, and acceptance criteria are separate rows. A model judge can fill one row. It cannot be the gate. Disagreement escalates.
Detect a stall by what failed, and keep only strict improvement. Record the set of checks that failed on each attempt. If two attempts in a row fail the same set, the last attempt changed nothing, and the loop stops. A new attempt replaces the current one only when its failures are a strict subset of the old ones. Otherwise a loop can trade one gap for another until the budget runs out.
The conversation is not the memory. The model is stateless on every call. The author calls the feeling of a continuous chat the session illusion. State that must survive, the plan, what passed, and what is left, lives on disk. A failed check rolls back to the last known-good commit, not to an earlier point in the chat. A platform signal that a turn finished, or that an agent went idle, is not done. Only the loop's own gate decides that.
The same loop can run by hand, on a schedule, on a CI event, or on a webhook. An event beats a timer that fires when nothing changed. An unattended loop ignores its own output, allows only one run per item, and carries a stable work ID so a retry cannot write twice.
Human approval is a state, not a hope. On a high-risk action the loop saves its state, enters a paused approval the team can see, and resumes only when an authorized person approves. The visible pause is the bridge from one loop to a gated factory.
A budget caps tokens, CI rounds, or time. Stripe's coding agents get at most two rounds of continuous integration, and after the second run the branch goes back to a person either way [40]. In the workspace-export illustration, large-workspace validation hits that kind of limit. Instead of a third attempt, or a quiet change to a shared database client, the agent stops and hands the scope question to an engineer.
The maker never grades its own work. Self-correction in the same context lowers accuracy, and an external verifier raises it [11][13]. A skeptical standalone evaluator is "far more tractable than making a generator critical of its own work" [36]. In that account the evaluator used the running application, and a criterion under its threshold failed the sprint. Those are single runs judged by the report's author, not a controlled trial. The maker sees the intention, not the result. Evaluators also recognize their own generations and score them higher [78].
Blind the judge. The checker gets the output, the spec, and the rubric, not the maker's reasoning or tool log. Stated reasoning can misrepresent the real reason for an answer [77]. In a study of 800 agent trajectories, manipulated reasoning traces inflated false positive rates of strong judges by up to 90% [76]. Grade the artifact, not the path [71]. The checker has fewer tools, none that write, and hidden tests stay with the harness. Independence is separate context, inputs, and permissions, not a different model name. Each reviewer records a first finding before seeing the others'. Majority voting accounted for most of the gains usually credited to multi-agent debate [79]. The Claude Code team posts a finding only after a separate group agrees it is real [80].
A cheap scorer can cover every output, and only uncertain or failing cases go to a reasoning judge or a person. A fast judge can fail the work. The critique of how to fix it comes from the slower judge. Judges trained to reason before they score report gains on judging benchmarks [81]. Ask for pass or fail, or a pairwise comparison, rather than a free score [82], and give each rubric dimension its own judge [71]. The maker proposes, the judge reports findings specific enough to act on, and plain code decides whether to retry, stop, or escalate. Choose "deterministic graders where possible, LLM graders where necessary" [71].
Parallel agents need separate working copies. Two agents editing the same file overwrite each other. Each parallel agent gets its own git worktree, and the work meets again only at merge.
A verifier is necessary, not sufficient. The Multi-Agent System Failure Taxonomy attributes 17.3% of multi-agent failures to missing or incorrect verification, and its authors warn that "the presence of a verifier is not a silver bullet" [41]. The 17.3% covers no or incomplete verification, and incorrect verification. In the first version of the taxonomy, the whole verification category accounts for 21.3% of failures, behind specification and system design at 41.8% and misalignment between agents at 36.9% [41]. Most failures trace to things a team builds rather than buys. The check itself needs protection: commit a failing test first, then block test edits while the agent works, so the agent fixes the code and not the test [4].
A check can only confirm that the work matches what was asked, so the specification has to live on disk. Score it before approval. Linters, types, and tests reject work before a pull request opens, and the agent's own setup is code under test: real tasks become evals when a rule, skill, or hook changes, and every incident becomes an eval [4]. A loop still runs alone. Nothing chains it to the next one.
Shrinks for the human: watching sessions, and reviewing every change in full. Grows for the human: goals a machine can check, the checker, and the stop rule.
Era 5: the AI software factory
An AI software factory chains triggered loops into one pipeline with people at the gates. OpenAI's factory, as reported in 2026, runs nine stages: a person defines the desired outcome, agents gather context, implement, build and test, and review, a person approves the change for production, and agents deploy, observe production, and respond to incidents [42]. Its incident agent proposes mitigations but does not execute them [42]. Nine stages are one operator's model, not a fixed law. A team can split the work differently, as long as each stage has an artifact, a check, and an owner. The next section uses ten.
Each stage commits an artifact the next stage reads. Anthropic's playbook runs from an intent file through a spec, a plan, a reviewed pull request, and an incident record that becomes the next intent [4]. The product owner corrects the agent-written intent before it is committed, the engineer corrects the plan before code is written, and approval to merge comes "from a human through branch protection" [4]. A durable work graph keeps task state outside any one session, so discovered work survives compaction [33].
Put the plan in a database, not the prompt. A markdown plan is lost when the window compacts, drifts when parallel agents edit it, and drops work discovered along the way. A work graph such as Beads tracks dependencies and lists tasks with no open blockers, so a query, not the model, decides what is ready [73]. Each item points at the spec section by an ID and a pinned revision, not a pasted copy. A gatekeeper checks a draft graph for cycles, duplicates, and items that fail the definition of ready before any worker claims one. Each worker claims one item, makes one reviewable commit, and files discoveries as new linked items. How long the spec lives is a choice: spec-first writes it for one task, spec-anchored keeps it for maintenance, and spec-as-source makes the spec the only file a person edits [74].
The same loop of orchestrator, maker, and checker runs for a ticket, a failing build, a review comment, or a stale document. Only the object and the trigger change. Five rules hold the pattern at that scale.
- The trigger starts the run; it never decides when to stop. Events arrive twice, so each run takes a lock and checks a work ID before it acts.
- Local and remote gates run the same rule. If the check before a push is not the check CI runs after it, the remote gate is theater.
- The model proposes; code commits. Ordinary code writes the file, the commit, and the ticket update.
- State is an append-only log. Status is computed from the log, so parallel agents never fight over a shared state file.
- Trace the failures first. A trace that exists only when a run succeeds is the one the team cannot use. Every run records its steps, its cycles, and its cost.
The score is the anchor. Cases the loop got wrong, and each human overrule, become regression cases. A green suite proves only what someone thought to check, which is why the gates stay. Grade the artifact rather than the exact path, and calibrate a model judge against people before it gates anything [71]. Strong judges agree with people over 80% of the time, about as often as people agree with each other, and they show position, verbosity, and self-enhancement biases [72].
An improvement loop reads traces, combines findings from parallel analysis, and changes the harness under optional human review [65]. A comment made twice can propose a rule. It cannot install one, and it cannot weaken the checks that judge it. Start with 20 to 50 tasks from real failures, and hold the suite near a 100% pass rate [71]. A change that fixes its own case and fails the suite does not ship [65]. Public benchmarks cannot be that gate. OpenAI stopped reporting SWE-bench Verified because it was contaminated, progress had slowed, and at least 59.4% of the audited problems had flawed tests that reject correct solutions [75].
Arm the gate while the loop is small, when a refusal is cheap. The test of a factory is not a demo in week one. It is whether the practice is still running, and still trusted, two months later.
Work enters only when it meets a definition of ready, the team's agreement on how complete an item must be before work starts [43]. Changes leave through a merge queue, which checks each one against the latest target branch and the changes already queued [44]. Many agents opening pull requests at once raise the chance that two changes pass alone and fail together.
The readiness gate is the cheapest one. The model lists the fields it found. A script computes which are missing, so the model's claim never decides. A passing ticket records its tier, a one-sentence reason, the expected paths, and the evidence the pull request must show. A person approves before it is marked ready.
Risk tiers decide who approves. Cloudflare sorts every merge request into one of three tiers by the size and nature of the diff, and every merge request gets an agent review [45]. OpenAI's account shows high-risk changes going to more agent reviews or a human reviewer, while some areas let an agent approve low-risk changes [42]. SLSA, at its highest source level, requires two trusted persons on every change to a protected branch [46]. Read the person who started the agent as one of the two, so someone else approves. DORA finds change approval works best as peer review inside the development platform, not as an external change board [47].
Score a tier from how much of the system a change touches, how well it can be verified, and the impact if it fails. The highest signal wins, and the real diff can raise it. Anyone can raise a tier. Lowering one takes the platform owner's sign-off and a log entry.
One workable scheme, a reference design rather than a measured standard, has three tiers. The label travels from the ticket into the pull request. The person who started the agent never approves its pull request, and no agent bypasses branch protection.

The scheme has three risk tiers, numbered 0 to 2. The Tier column gives the number. The Typical change column says what kind of change lands in that tier. The Who approves column says which reviewers, agents or people, must approve it.
- 0 covers documentation, and dependency bumps that tests cover. Standards and security reviewer agents, green CI, and a passing security scan approve it. Automatic merge comes only after the revert rate is measured.
- 1 covers normal, well-tested work. Specialist reviewer agents review it, their findings are verified, and then one person approves.
- 2 covers authentication, secrets, money, personal data, CI/CD, infrastructure, public API contracts, model logic, or anything tests cannot verify. Two people approve, one of them a code owner. The security review blocks, and the merge queue runs the checks again.
Reviewer agents merge their findings, by a vote or a coordinator, before a person sees them. Pull requests stay small. Each one carries a summary, a link to its spec, the tier and its reason, the test evidence, and provenance: requirement ID, tier, agent, model version, and tests. Reverts can then be measured by tier. Changes to specifications, test assertions, and harness rules always go to a person, whatever the tier. An agent that can loosen the checks that grade it can turn any build green.
Fewer findings get more of them acted on. When cost work at Cloudflare cut its reviewer's serious findings by about 40%, authors acted on 59% of them, up from 34% [48]. OpenAI tunes its reviewer for precision over recall, and authors change their code for about half of its comments [49]. Stripe merges over 1,300 agent-written pull requests a week, each reviewed by a person [40]. None of those accounts reports production-incident rates for agent-written code.
Autonomy arrives first on mundane work: merging parallel branches with a report of what changed, fixing review findings inside a cap, and filing new work instead of silently widening the task.
The dark factory is not the target. It removes the human gates, and agents write and review with little or no supervision [50]. The best current benchmark of review agents finds they still miss more than 20% of the bugs that human reviewers catch [50]. No operator publishes defect rates for a factory without human gates. Removing the gate before closing that gap trades a known review cost for an unmeasured defect rate.
A light factory sits between the two. Automated gates, each with a reject loop, run before a person sees a change. The person reviews evidence, and a person still approves what reaches production. A fully dark factory, where it fits at all today, fits small greenfield work.
Go darker one change class at a time, on evidence. Meta's RADAR reviews and lands low-risk changes automatically, chosen by a diff risk score. It has reviewed more than 535,000 diffs and landed more than 331,000 [70]. Its diffs are reverted at one third the rate of other diffs and cause production incidents at one fiftieth the rate [70]. The threshold moved only as data accumulated. Relaxing it from the 25th to the 50th percentile of risk raised the automatic approval rate to 60.31% [70]. Review agents remain weaker on complex changes and on bugs that cross files, and the benchmark's authors recommend scaling review effort to the complexity of each change [50]. Automatic repair and automatic merging are separate decisions. Keep automatic merging off until the revert rate for that class is measured, and move the class back when reverts rise. No published source gives a miss rate that justifies removing a gate. The threshold is the team's decision, written down and revisited.
Shrinks for the human: reading diffs that deterministic checks already proved, status meetings, and handoffs. Grows for the human: approving the intent, the plan, the merge, and the release; the risk policy; and the evals.
The factory in ten stages
OpenAI's nine stages describe one operator [42]. A reference design uses ten so that each stage holds one kind of decision and one kind of owner. Shaping a need is a product decision. Planning the work is an engineering one. Choosing what starts is priority. Ordering steps inside admitted work is scheduling. Merging code and exposing it to users are separate approvals. Maintaining software is different work from improving the factory that builds it.
The split, the suggested limits, and the rollout schedule later in this piece are a reference design, not a controlled comparison. Treat every number in this section that is not tied to a citation as a starting value to measure and tune.

Every stage has the same shape. A trigger starts it. An agent does bounded work inside a harness. A check runs that the agent cannot argue with. A named person decides. The stage commits an artifact the next stage reads, records what it learned, and stops at a limit it cannot exceed.

Each of the ten stages commits one artifact, runs its own checks, and ends with a named decision. The Stage column names the stage. The Commits column gives the artifact the next stage reads. The Checks column gives the check the agent cannot argue with. The Who decides column names the person who decides.
- Shape commits an agreed need, or a decision to stop. The check is that the owner can explain it without reading it, and the app owner decides.
- Plan commits ready tickets with tier, paths, and required evidence. The checks are a script that computes missing fields and the closing of the reviewer's gaps. A named person marks the ticket ready.
- Dispatch commits admitted work with a budget and a route. The check is that limits are attached before work starts. People set priority, and a maintainer settles risk.
- Schedule commits a validated work graph, then claimed steps. The gatekeeper checks for no cycles, criteria on every package, and sizes in range, and a person releases the plan.
- Build commits a pull request with a receipt. Tests, types, lint, and the acceptance criteria run inside the loop, and the assigned engineer decides when the loop stops.
- Review commits one report with verified findings. Blinded specialists, voting, and deterministic gates check it, and a named person decides the merge.
- Release commits a release, a hold, or a recovery. Policy gates, the merge queue, and provenance check it, and a production authorizer decides.
- Respond commits restored service and a refinement ticket. The check is evidence attached to every claim, and the on-call engineer decides.
- Maintain commits maintained software, or a recorded retirement. Characterization tests run before and after. The maintainer decides, and the app owner decides on retirement.
- Improve commits an evaluated change to the factory. The check is a held-out comparison against the current version. An independent reviewer, then the factory owner, decides.
Four rules matter most where work changes hands. Stages do not call each other. They share artifacts, and every artifact has a contract as strict as an API. If one stage writes READY and the next checks for Ready, the factory stops, and no model will notice. A handoff transfers information, not permission. Silence never means approval. A missing limit is not unlimited permission.
Deciding what to build
Shape is a wizard. It asks one question at a time and marks anything the owner did not say as an open question. A task assembled over many turns does markedly worse than one given whole [9], so the wizard writes the spec to disk and the next stage starts fresh. The owner must explain the requirement without reading it. Shape can also stop, and it runs again after release. Evidence for agents here is thin, so the stage stays human-led.
Plan comments on tickets until they meet the definition of ready [43]. The model lists fields. A script decides what is missing. A passing ticket records tier, reason, paths, and required evidence, and a named person marks it ready. A copy fix takes a short path. Security, data access, money, or a public API takes the full one.
Dispatch attaches the budget. Suggested starting values, to tune rather than trust: two active items per application, one of them reserved for incidents; one active review per reviewer; about 90 minutes and at most two attempts and two repair rounds per build. People set priority. The agent ranks, routes, re-tiers on the real diff, and enforces the limits. Evidence for agents owning priority is the thinnest in the factory.
Schedule turns admitted work into small packages. A gatekeeper rejects cycles, orphans, missing acceptance criteria, and packages estimated over about 90 minutes. A person releases the plan. A query, not a model, computes what is ready [73]. Claims carry leases. Each parallel worker has its own git worktree and does not re-plan. Discovered work is linked, jumps the queue as a blocker, supersedes a later package, or returns to Plan when it is out of scope.
Delivering changes
Build runs in a fresh sandbox with no production credentials. The loop exits with a pull request and a receipt, or the budget runs out and an engineer gets a written question. Stripe's limit of two CI rounds is the model [40]. Code grades. Stalls are the same failure set twice. Only a strict improvement replaces the current draft. State lives on disk.
Review fans specialists for security, standards, spec match, and tests out from one bundle, merges findings, and leaves the merge to a person. A vote or a second agent corroborates each finding first [79][80]. Checkers are blinded to the maker's reasoning [76], read the ticket rather than the pull request description, and post fewer findings [48][49]. Build Review first. Review hours are the constraint.
Release may need no agent. Merge and exposure have separate approvers: a tag, burn-in on that same artifact, approval for a version and an environment and a scope, then a deploy behind a disabled flag. Approvals are peer review inside the platform [47]. A merge queue re-checks the latest target [44]. The pipeline writes provenance [46]. People press the buttons customers can feel.
Operating and improving
Respond lets a diagnostic agent read. It never acts on production, and recovery does not wait on a model. OpenAI's incident agent proposes mitigations and does not execute them [42]. Stop when service is restored and a refinement ticket carries the evidence. Each incident becomes an eval [4]. Mask personal data before any agent reads a log.
Maintain has the clearest pass-or-fail signal and the smallest blast radius. Deterministic tools do the easy edits. Agents take the long tail. Characterization tests prove behavior did not change. Automatic merge is earned per change class, on a measured revert rate. An expired timer never authorizes a deletion.
Improve proposes changes to rules, skills, hooks, and rubrics from failed runs, dismissed comments, incidents, and overrules. Each change is one problem, a held-out comparison, a shadow run, and a recorded history. An independent reviewer checks the comparison. The factory owner decides. A better score never grants merge or production power.
At every stage the drafting shrinks and the decision grows: the intent, the definition of ready, the priority and the limits, the release of the plan, the merge, the production approval, recovery, what may merge itself, and which lesson becomes a rule.
The workspace export, end to end
The story is illustrative. Administrators want an audit download of their own workspace. Shape offers two options, the owner rules out cross-workspace search, and the success measure is a pilot export with no support ticket. Plan flags three gaps: no rule for non-admins, no workspace filter, and no plan for large workspaces. Access checks become a dependency. Dispatch admits only that ticket, on the full review path. Schedule splits a three-hour validation package, and a wrong time-zone helper becomes a blocker that jumps the queue.
Build times out on large workspaces, stops instead of editing a shared database client, and the owner sends large exports back to Plan as a background job. Review finds a helper query with no workspace filter that a second model family missed. The change returns with a failing test, and tier 2 waits for two approvers. Release ships behind a flag. Respond traces slow exports to a connection-pool default and ignores a log line that asks for a higher limit, because that identity cannot change anything. Maintain puts the client on the always-high-tier list. Improve turns the repeated missing filter into a lint rule after a held-out comparison catches every known case.
No single agent in that story was remarkable. The factory was, because every handoff had a contract, every stage had a limit, and every decision had a person.
Three concerns across every stage
Three concerns belong to every stage. Each stage checks its own work, reads and writes a shared memory, and runs inside a harness on infrastructure someone chose.
Evals: how every stage checks itself
An inline check runs on every piece of work before it moves on: missing ticket fields, a cyclic graph, the tests inside a build loop. A stage metric runs over time: comments people dismiss, discovered work per ticket, retries per pull request, escaped defects, time to restore service.
Move down this list only when the rung above cannot answer the question.
- Deterministic checks. Tests, types, lint, schemas, and policy as code. Push every lesson down to this rung when you can. Choose deterministic graders where possible [71].
- Held-out comparisons. A fixed set of past work with known outcomes, kept out of every prompt. Compare the old and new version of a prompt, skill, or rubric on that set. Start with 20 to 50 tasks drawn from real failures [71].
- Calibrated model judges. For qualities no test can decide, such as whether a requirement is clear. Strong judges agree with people about as often as people agree with each other, and they show position, verbosity, and self-enhancement biases [72]. Calibrate on a sample, recalibrate on a schedule, blind the judge to the maker's reasoning [76], and use a different model family from the author [78].
- People. A named person approves every merge and release, and a sample of judged work goes to a person too. Spend human review where no other rung reaches.
Public benchmarks cannot stand in. OpenAI stopped reporting SWE-bench Verified because it was contaminated and many of its tests rejected correct solutions [75]. The evals that matter come from a team's own tickets, findings, incidents, and reverts.
Every run should leave a trace of prompt, model, tool calls, retrieved context, output, and cost, plus an evidence record an auditor can walk from the request to the sign-off. Do not build the store from scratch. Mask personal information before a trace leaves the environment. A trace proves what happened, not that it was right.
Memory: what the factory knows between runs
Agents remember nothing between runs. Project amnesia means the reasons for decisions live in meetings and in people's heads, so every new agent relitigates them. Session amnesia means a run's discoveries vanish when the run ends.
Keep two layers apart. A work log records what happened: tickets, events, approvals, and links from ticket to plan to pull request. It owns status. A knowledge layer records what it means: decisions, architecture, research, fix records, and the factory's configuration. It never owns status. One implementation keeps the work log as an append-only event log in the repository and the knowledge layer as Markdown in Git, linked by typed edges. Markdown in Git is truth. Indexes are caches.
The interface to an agent is a bounded slice for one task: a few hops of related decisions, past fixes, and patterns, under a token budget that fails closed rather than overflowing. The smallest set of high-signal tokens still beats a full window [14]. Retrieval remains useful for large raw corpora, and contextual retrieval makes it better [62]. A curated graph avoids most chunking problems by storing small, current pages to begin with.
Every stage recalls before acting, captures what it learned, and has a person review agent-written entries. Supersede a changed decision. Do not delete it, because a stale decision reads as a current one. The model proposes. Schema-enforced helpers write. Each kind of entry has one owner. Concurrent writers use their own worktrees and open pull requests. Every entry carries an author and a source. Captured text is untrusted until reviewed, or an instruction hidden in a ticket or a log will be served to every future agent.
Deployment: where every stage runs
Deployment is where the harness becomes infrastructure, and the principles come before the platform. Each stage has its own identity, never a borrowed human token, with least privilege granted one stage at a time. Each run gets a fresh sandbox, no production credentials, and egress limited to the work. Hooks, a trace, and a budget sit on every run. Agent identities never approve a merge, sign a change record, or promote a release. A successful prompt injection should have nowhere to go: no agent holds private data, untrusted input, and a way to send data out, all at once.
Climb only as far as the work requires. Most stages start as a CI action and stay there for months. A sandboxed container behind a webhook is the next rung. A durable runtime is for work that must survive restarts or hold state for hours or days.
The substrate can be a CI system such as GitHub Actions, Microsoft Foundry, Amazon Bedrock AgentCore, a model vendor's hosted harness, Kubernetes, or an agent SDK you run yourself. Each option still leaves the loop design, the hooks, and the evals with the team. Some leave the sandbox and the identity model as well. The practical test is whether a stage can move from one substrate to another without rewriting the loop. Teams usually pick the place their identity and data already live.
The human gate grows as everything else shrinks
Human time per change shrinks in every era. The consequence of each human decision grows. The person sits inside each turn in era 1, curates context in era 2, sets the boundary in era 3, designs the checker in era 4, and stands at the gates in era 5. Anthropic's playbook describes the end state as a loop that runs in "hours, not weeks, with humans above the loop instigating, directing and governing" [4].

The sandboxing result shows the pattern in one number. Fewer prompts reach the person, and the ones that remain get attention [34]. The playbook draws the same line for approvals: an approval prompt during the build "puts a person back on the critical path of all the sessions running in parallel," so approvals belong at the gates [4]. People approve four things: the intent, the plan, the merge, and the release [4]. In the ten-stage factory those four map onto Shape, the Plan and Schedule gates, Review, and Release. A fifth decision, recovery, stays with the on-call engineer in Respond.
Review capacity is the constraint. Generation scales, and human verification does not, so the queue in front of each gate decides the pace [4]. The order of rollout follows from that constraint. A team builds review and maintenance capacity before it turns on feature generation, so the review queue can absorb the output, and it caps what agents generate at what people can review. In the author's words, a team expands the pipe before it opens the floodgates. The same reading explains the instability the DORA research reports [2]. In the author's view it comes from AI-generated changes that skip small-batch discipline and automated checks before review, not from AI itself.
"Grows" means influence and accountability, not hours. The playbook states that "humans remain accountable for every decision that requires judgment" [4]. A person who approves an intent sets the direction for every loop that follows. A person who writes the checker decides what every future change must prove. A person who sets the risk tiers decides which changes ever reach a human.
The dark factory is the end state where human time per change reaches zero. The evidence does not support it yet, because review agents still miss bugs that people catch [50]. The playbook closes on the opposite principle: "The loop keeps running. Human judgement stays above it" [4].
Start where your team is
Each era needs one skill and one platform capability. DORA's AI Capabilities Model names seven capabilities that amplify AI [51]. A quality internal platform and AI-accessible internal data matter most for a factory [51].

Each era pairs one team skill with one platform capability. The Era column names the era. The Train the team on column gives the skill to teach. The Enable on the platform column gives what the platform must provide, with its source.
- 1. Prompt engineering: train the team on clear requests and output formats, and enable approved tools [51].
- 2. Context engineering: train the team on short rule files, skills, and notes, and enable shared files kept short and current [7].
- 3. Harness engineering: train the team on hooks, tools, and subagents, and enable a sandbox, approved MCP servers, and read-only logs [34].
- 4. Loop engineering: train the team on goals, checkers, budgets, and escalation, and enable an agent lane in CI with a round limit [40].
- 5. Software factory: train the team on risk tiers and the four approvals, and enable a cloud runtime, observability, and enforced gates [4].
Training belongs on the list. DORA associates dedicated learning time with 131% more team adoption [52]. The figure is a survey association, not a controlled result.
Build the stages in order of risk, not in lifecycle order. A team that builds in lifecycle order starts with Shape and Plan, the stages with the thinnest evidence, and reaches the review bottleneck last, after it has already flooded it. The five-phase rollout starts with agents that only comment, earns write access one phase at a time, and keeps release and incident remediation human-led throughout. Phases overlap so the team is never idle, but each gate still applies. The weeks are a reference schedule, not a measured optimum.

The rollout runs in five phases. The Phase column names each phase. The Weeks column says when it runs. The Turn on column lists what you enable. The Gate to move on column says what must be true before the next phase starts.
- 0. Baseline runs for 4 weeks before the start. It measures merge rate, review time, change failure rate, and cost per merged pull request. The gate is that the numbers are recorded.
- 1. Comment-only runs in weeks 1 to 3. It turns on shared memory, the ticket reviewer, and an advisory pull-request reviewer, all read-only with no secrets. The gate is that the team finds the comments useful, with low noise.
- 2. Maintenance runs in weeks 3 to 6. It turns on a scheduled dependency-upgrade agent that fixes failing upgrade pull requests. The gate is that upgrade pull requests merge without extra rework.
- 3. Build and feedback runs in weeks 5 to 10. It turns on Schedule and Build on ready tickets only, with Respond in advisory mode. The gate is that merge rate and change failure rate hold or improve.
- 4. Factory improvement runs in weeks 8 to 12. It turns on a weekly retro tested against a fixed task set. The gate is that results improve on held-out tasks.
Run the rollout as a measurement. Pair the maker with a checker that sees only the artifact and the criteria, calibrate any model judge on the same sample people scored, and report pass rate, escalation rate, and judge-human agreement before widening scope.
Start with the review-and-fix loop. It has one input, one output, and one human gate. Authors act on about half of an automated reviewer's comments [49]. The loop pays off before a full factory exists.
Measure before adding autonomy. Take a baseline before the first agent ships, and measure through the dip that every new process brings. The raw count of pull requests is not a measure, because it is easy to inflate.

Nine measures track the rollout. The Measure column names each one. The Why it matters column says what it reveals. The Watch for column gives the warning sign to act on.
- Revert rate, by risk tier is the main quality signal, a direct form of the change fail rate [47]. Any rise after a phase or tier change is a stop signal.
- Cycle time, assignment to merge shows whether work moves faster end to end [2]. Faster builds with flat cycle time mean the queue moved, not the work.
- Review minutes per change shows whether Review relieves the bottleneck. It should fall in phase 1 and stay down after phase 3.
- Re-tier rate shows how often a change moves up a tier on its real diff. A high rate means tickets understate risk.
- Overrule rate shows how often people overrule an agent's verdict. Each overrule is a labeled case for the eval suite.
- Escaped defects are bugs found after merge that a check should have caught. Each one becomes a regression case.
- Cost per merged change is agent spend, including failed attempts, per merge. Rising cost with flat merges means retries are burning money.
- Dismissed-as-noise rate shows how often people resolve an agent comment without acting on it. High rates mean a reviewer is miscalibrated, and people stop reading.
- Judge-human agreement shows whether model-scored evals can be trusted. Falling agreement means recalibrate before trusting any score.
Review load needs its own limit. At Meta, the lines of code in each human-landed diff grew 105.9% in a year, and agentic AI accounted for over 80% of a 51% rise in diffs per developer [70]. DORA's 2025 research still finds AI adoption linked to lower delivery stability [2]. Throughput alone hides that cost.
Climb one level at a time. One working description of the climb has three levels: casual, pro, and elite. What changes between them is not the principles but how much machinery makes them automatic. Do not jump levels in a weekend, or the team gets skills no one uses and hooks no one understands. Copy the principles, not only the artifacts: verify before trusting, plan before acting, and treat context as scarce.
Limitations
Several claims rest on operator accounts and vendor reports. The Stripe, Cloudflare, OpenAI, and Anthropic accounts describe each operator's own practice, and none reports production-incident rates for agent-written code [40][45][42]. The 84% sandboxing figure is an internal vendor measurement [34]. Chroma's context rot report is not peer reviewed [3]. Anthropic's harness design results are single runs judged by the author of that report [36].
Some controlled studies are small. The early-2025 trial had 16 developers, and its follow-up calls its own data unreliable [1][5]. No gated factory has published production-incident rates, so the case for human gates rests on review evidence, not on outcome data [50]. Tool features change monthly, so the mechanisms matter more than any one tool's current version [23]. The documentation behind the hook-failure claims was read on 2026-11-06.
The ten-stage design is a reference, not a measured result. Several stages, Shape and Dispatch in particular, have little published evidence for agent involvement at all, which is why they stay human-led. The workspace export is illustrative. Disclosure: the author of the source paper is a Claude Certified Architect and a member of the Anthropic Partner Network, which is one reason the Anthropic engineering accounts are used as practice, not as proof.
Do this today
- Write the check before you write the next prompt. Name one fact a test, a type checker, a lint rule, or a hook can enforce, and stop asking the model to remember it.
- Cap one real loop. Give it a goal, a receipt with more than one row, a stall rule based on the set of failing checks, and a hard stop at two CI rounds or an equivalent budget [40].
- Split maker from checker on the next pull request. The checker sees the diff, the ticket, and the rubric, not the maker's reasoning [76].
- Record a baseline this week: revert rate, review minutes per change, and change failure rate, before any new agent is allowed to open more pull requests [2][47].
- Leave automatic merge off. If you remove a gate later, do it for one low-risk change class, after you have measured that class's revert rate [70].
The person stays above the loop
Five eras of AI-assisted development each moved one more kind of work from the person to the system. Prompt engineering improved the request. Context engineering controls what the model sees, because accuracy falls as input grows [3]. Harness engineering moves rules from advice into enforcement, and the same model does better inside a designed harness [25]. Loop engineering repeats checked cycles under a budget, with a maker that never grades its own work [13]. The AI software factory chains those loops with people at the gates [4].
Laid out as ten stages, the factory makes those ideas concrete. Each stage commits an artifact the next one reads, runs a check its agent cannot argue with, and ends with a named person. Evals, memory, and deployment run underneath all of them.
In every era, and in every stage, human time per change shrinks and the weight of each human decision grows. A team climbs the eras in order, moves each rule from advice to enforcement, builds review capacity before generation, and keeps people at the gates where judgment carries the weight [4]. The loop can run for hours. The judgment above it does not get to be absent.
References
- [1] Joel Becker, Nate Rush, Elizabeth Barnes, David Rein (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv preprint, METR. https://arxiv.org/abs/2507.09089
- [2] Derek DeBellis, Kevin M. Storer, Nathen Harvey; DORA, Google Cloud (2025). State of AI-assisted Software Development 2025. https://dora.dev/research/2025/dora-report/
- [3] Kelly Hong, Anton Troynikov, Jeff Huber (Chroma) (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma Technical Report. https://www.trychroma.com/research/context-rot
- [4] Louis Claxton (2026). The AI-native SDLC playbook. Anthropic. https://claude.com/resources/articles/the-ai-native-sdlc-playbook
- [5] Joel Becker, Nate Rush, Tom Cunningham, David Rein, Khalid Mahamud (2026). We are Changing our Developer Productivity Experiment Design. METR. https://metr.org/blog/2026-02-24-uplift-update/
- [6] Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, Satish Chandra (2024). How much does AI impact development speed? An enterprise-based randomized controlled trial. arXiv. https://arxiv.org/html/2410.12944
- [7] Ryan Lopopolo (2026). Harness engineering: leveraging Codex in an agent-first world. OpenAI. https://openai.com/index/harness-engineering/
- [8] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL 12:157-173. https://aclanthology.org/2024.tacl-1.9/
- [9] Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville (2026). LLMs Get Lost In Multi-Turn Conversation. ICLR 2026. https://openreview.net/forum?id=VKGTGGcwl6
- [10] Delba de Oliveira, Michael Segner (2026). Loop engineering: Getting started with loops. Anthropic. https://claude.com/resources/articles/getting-started-with-loops
- [11] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou (2024). Large language models cannot self-correct reasoning yet. ICLR 2024. https://openreview.net/forum?id=IkmD3fKBPQ
- [12] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, Weizhu Chen (2023). CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. arXiv. https://arxiv.org/html/2305.11738
- [13] Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati (2025). On the self-verification limitations of large language models on reasoning and planning tasks. ICLR 2025. https://openreview.net/forum?id=4O0v4s3IzY
- [14] Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, Jeremy Hadfield (2025). Effective context engineering for AI agents. Anthropic. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- [15] Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. ICML 2025, PMLR 267:44554-44570. https://proceedings.mlr.press/v267/modarressi25a.html
- [16] Patrick Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
- [17] Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, Michael Carbin (2024). Long Context RAG Performance of Large Language Models. arXiv. https://arxiv.org/html/2411.03538v1
- [18] Yixuan Tang, Yi Yang (2025). FinMTEB: Finance Massive Text Embedding Benchmark. EMNLP 2025. https://aclanthology.org/2025.emnlp-main.179/
- [19] AGENTS.md (Agentic AI Foundation, Linux Foundation) (n.d.). AGENTS.md. https://agents.md
- [20] Agent Skills project (n.d.). Specification: Agent Skills. https://agentskills.io/specification
- [21] Barry Zhang, Keith Lazuka, Mahesh Murag (2025). Equipping agents for the real world with Agent Skills. Anthropic. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills
- [22] (2026). Is Progressive Disclosure All You Need for Long-Context Agents? ACL Anthology. https://aclanthology.org/2026.realm-1.14/
- [23] OpenAI (n.d.). Custom instructions with AGENTS.md. Codex documentation. https://learn.chatgpt.com/docs/agent-configuration/agents-md
- [24] Vivek Trivedy (2026). The Anatomy of an Agent Harness. LangChain. https://www.langchain.com/blog/the-anatomy-of-an-agent-harness
- [25] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. https://arxiv.org/abs/2405.15793
- [26] Mitchell Hashimoto (2026). My AI Adoption Journey. https://mitchellh.com/writing/my-ai-adoption-journey
- [27] Birgitta Böckeler (2026). Harness engineering for coding agent users. martinfowler.com. https://martinfowler.com/articles/harness-engineering.html
- [28] Cursor (n.d.). Hooks. Cursor documentation. https://cursor.com/docs/hooks
- [29] GitHub (n.d.). Hooks for GitHub Copilot. GitHub Docs. https://docs.github.com/en/copilot/concepts/customizations/hooks
- [31] Anthropic (n.d.). Hooks reference. Claude Code documentation. https://code.claude.com/docs/en/hooks
- [33] Justin Young (2025). Effective harnesses for long-running agents. Anthropic. https://www.anthropic.com/research/effective-harnesses-for-long-running-agents
- [34] David Dworken, Oliver Weller-Davies (2025). Beyond permission prompts: making Claude Code more secure and autonomous. Anthropic. https://www.anthropic.com/engineering/claude-code-sandboxing
- [35] Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford (2025). How we built our multi-agent research system. Anthropic. https://www.anthropic.com/engineering/multi-agent-research-system
- [36] Prithvi Rajasekaran (2026). Harness design for long-running application development. Anthropic. https://www.anthropic.com/research/harness-design-long-running-apps
- [37] Walden Yan (2026). Multi-Agents: what's actually working. Cognition. https://cognition.ai/blog/multi-agents-working
- [38] Sandeco Macedo (2026). Stop Hand-Holding Your Coding Agent: Engineering the Loops that Replace Step-by-Step Prompting. arXiv. https://arxiv.org/html/2607.00038
- [39] Jai Lal Lulla, Vahram Nersesyan, Seyedmoein Mohsenimofidi, Christoph Treude, Sebastian Baltes (2026). Loop engineering: building blocks, adoption, and impact. arXiv. https://arxiv.org/abs/2608.21884
- [40] Alistair Gray (2026). Minions: Stripe's one-shot, end-to-end coding agents, Parts 1 and 2. Stripe. https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents
- [41] Mert Cemri et al. (2025). Why do Multi-Agent LLM systems fail? NeurIPS 2025 Datasets and Benchmarks Track. https://openreview.net/forum?id=fAjbYBmonr
- [42] Gergely Orosz (2026). Inside OpenAI's agentic software factory. The Pragmatic Engineer. https://newsletter.pragmaticengineer.com/p/openai-software-factory
- [43] Microsoft (2025). Definition of Ready. Code-With Engineering Playbook. https://microsoft.github.io/code-with-engineering-playbook/agile-development/team-agreements/definition-of-ready/
- [44] GitHub (n.d.). Managing a merge queue. GitHub Docs. https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue
- [45] Ryan Skidmore (2026). Orchestrating AI Code Review at scale. Cloudflare. https://blog.cloudflare.com/ai-code-review/
- [46] SLSA Community (OpenSSF); in-toto project (2025). SLSA specification, Version 1.2. https://slsa.dev/spec/v1.2/
- [47] DORA (2019). Streamlining change approval. https://dora.dev/capabilities/streamlining-change-approval/
- [48] Ryan Skidmore (2026). Cutting our inference bill by 68% by being obsessive about optimization. Cloudflare. https://blog.cloudflare.com/obsessive-inference-optimization/
- [49] Maja Trębacz, Sam Arnesen, Albin Cassirer, Max Johnson, Xin Lin, Thibault Sottiaux, and the Codex team (2025). A practical approach to verifying code at scale. OpenAI Alignment Research. https://alignment.openai.com/scaling-code-verification/
- [50] Chenyang Yang, Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill, Carmen Badea, Christian Bird (2026). (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection. arXiv. https://arxiv.org/html/2610.13668
- [51] Kevin M. Storer, Derek DeBellis (2025). Introducing the DORA AI Capabilities Model. Google Cloud. https://cloud.google.com/blog/products/ai-machine-learning/introducing-doras-inaugural-ai-capabilities-model
- [52] DORA (2025). Impact of Generative AI in Software Development. https://dora.dev/ai/gen-ai-report/report/
- [53] Jason Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. https://arxiv.org/abs/2201.11903
- [54] Anthropic (n.d.). Prompting best practices: thinking and reasoning. Claude Platform documentation. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
- [55] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. https://arxiv.org/abs/2210.03629
- [56] Paul M. Fitts, R. E. Jones (1947). Psychological Aspects of Instrument Display, I: Analysis of 270 "Pilot-Error" Experiences. Memorandum Report TSEAA-694-12A, Aero Medical Laboratory. DTIC ADA800143. https://apps.dtic.mil/sti/pdfs/ADA800143.pdf
- [57] Martin Fowler (2026). Mechanical Sympathy. martinfowler.com. https://martinfowler.com/bliki/MechanicalSympathy.html
- [58] Blerta Veseli, Julian Chibane, Mariya Toneva, Alexander Koller (2025). Positional Biases Shift as Inputs Approach Context Window Limits. arXiv:2508.07479. https://arxiv.org/abs/2508.07479
- [59] Anthropic (n.d.). How Claude remembers your project: path-scoped rules. Claude Code documentation. https://code.claude.com/docs/en/memory
- [60] Cursor (n.d.). Rules. Cursor documentation. https://cursor.com/docs/rules
- [61] GitHub (n.d.). Adding repository custom instructions for GitHub Copilot. GitHub Docs. https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/add-custom-instructions/add-repository-instructions
- [62] Anthropic (2024). Introducing Contextual Retrieval. https://www.anthropic.com/engineering/contextual-retrieval
- [63] LangChain (2025). Context Engineering for Agents. https://www.langchain.com/blog/context-engineering-for-agents
- [64] Anthropic (n.d.). Automate actions with hooks. Claude Code documentation. https://code.claude.com/docs/en/hooks-guide
- [65] LangChain (2026). Improving Deep Agents with harness engineering. https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering
- [66] Florian Brand, JS Denain (2025). Why benchmarking is hard. Epoch AI. https://epochai.substack.com/p/why-benchmarking-is-hard
- [67] Terminal-Bench (2026). Terminal-Bench 2.0 leaderboard, as viewed April 2026. https://www.tbench.ai/leaderboard/terminal-bench/2.0
- [68] Anthropic (n.d.). Create custom subagents. Claude Code documentation. https://code.claude.com/docs/en/sub-agents
- [69] Tal Ridnik, Dedy Kredo, Itamar Friedman (2024). Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. arXiv:2401.08500. https://arxiv.org/abs/2401.08500
- [70] Chris Adams, Arjun Singh Banga, Parveen Bansal, et al., with Audris Mockus, Peter Rigby, and Nachiappan Nagappan (2026). Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency. arXiv:2605.30208. https://arxiv.org/abs/2605.30208
- [71] Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, Jiri De Jonghe (2026). Demystifying evals for AI agents. Anthropic. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- [72] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685
- [73] Gas Town Hall (n.d.). Beads: distributed graph issue tracker for AI agents. https://github.com/gastownhall/beads
- [74] Birgitta Böckeler (2025). Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl. martinfowler.com. https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html
- [75] OpenAI (2026). Why SWE-bench Verified no longer measures frontier coding capabilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
- [76] Muhammad Khalifa et al. (2026). Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation. arXiv:2601.14691. https://arxiv.org/abs/2601.14691
- [77] Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023. https://arxiv.org/abs/2305.04388
- [78] Arjun Panickssery, Samuel R. Bowman, Shi Feng (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. https://arxiv.org/abs/2404.13076
- [79] Hyeong Kyu Choi, Xiaojin Zhu, Sharon Li (2025). Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? arXiv:2508.17536. https://arxiv.org/abs/2508.17536
- [80] Addy Osmani, Morgan Westlee Lunt, Qing Wang (2026). How the Claude Code team does code review. claude.dev. https://claude.dev/blog/solving-code-review-at-anthropic/
- [81] Chenxi Whitehouse et al. (Meta) (2025). J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning. arXiv:2505.10320. https://arxiv.org/abs/2505.10320
- [82] OpenAI (n.d.). Evaluation best practices. OpenAI API documentation. https://developers.openai.com/api/docs/guides/evaluation-best-practices
This is Part 17 of "AI Software Factory Series," a 17-part guide to building an AI software factory with human gates.