Start With Agents That Only Comment
Start an AI software factory in order of risk, on a measured 12-week rollout, and keep a person at every gate.

Build an AI software factory in order of risk, not lifecycle order: the 12-week rollout, five phases, with a person still on every merge and every release.
A factory diagram is easy to admire and hard to start. The useful question is where to begin on Monday, and which number will tell you the phase worked.
In this article: The ten stages, the order to turn them on, the decisions that stay human, and the metrics that stop a flood of pull requests from looking like progress. Low-risk automatic landing is a narrow exception, not a license to drop review.
An AI software factory is ten stages, plus checks, shared memory, and a harness. Each stage has a trigger, work, a person who decides, a contract, memory, a check, and a limit.
Build in order of risk, not SDLC order. Start with agents that only comment. Measure first. Add autonomy one phase at a time. Keep releases and incident fixes human-driven. Treat the factory as a product, with users, a backlog, and retrospectives.
The whole factory on one page

The whole factory fits in ten rows, one per stage. The Stage column names the stage. The Agent work column says what the agent does. The Who decides column names the person who owns the decision. The Hands on column says what the stage passes to the next one. The One rule column gives the rule that stage must never break.
- Shape: a PRD wizard asks, offers options, and drafts. The app owner decides, and the stage hands on an agreed need or a decision to stop. Its rule is that waiting never becomes approval.
- Plan: a ticket reviewer comments until the ticket is ready. A named person says "looks good to me," and the stage hands on ready tickets. Its rule is that the ready ticket is an API.
- Dispatch: the agent ranks, routes, and reserves budget. People set priority and a maintainer settles risk, and the stage hands on admitted work with limits. Its rule is that a missing limit is not unlimited permission.
- Schedule: the agent writes the breakdown and hands out claims. A person releases the plan, and the stage hands on claimed steps, then a pull request. Its rule is that workers execute and never re-plan.
- Build: the agent builds in a sandbox, runs checks, and opens a pull request. The assigned engineer decides once the loop stops, and the stage hands on a pull request with evidence. Its rule is two attempts, then hand back.
- Review: specialist reviewers produce one report. A person decides the merge, and the stage hands on an approved change. Its rule is that a different model name is not independence.
- Release: the agent drafts notes and change records. The production authorizer decides, and the stage hands on a release, a hold, or a recovery. Its rule is that merge and release are separate decisions.
- Respond: the agent diagnoses, read-only. The on-call engineer decides, and the stage hands on restored service and a refinement ticket. Its rule is that recovery never waits for a model.
- Maintain: the agent handles upgrades, migrations, and fix records. The maintainer decides, with the app owner deciding on retirement, and the stage hands on maintained software or a recorded retirement. Its rule is that an expired timer cannot authorize deletion.
- Improve: the agent mines failures and proposes tested changes. An independent reviewer and then the factory owner decide, and the stage hands on an evaluated change. Its rule is that better results never grant new powers.
Every stage ends with a person. A person at the end of every stage is the design, not a gap. Each stage also checks its own work, uses shared memory, and runs in a harness with its own identity and sandbox.
The order to build
Lifecycle order floods review last. A good plan is the 12-week rollout, five phases. Each phase earns the next.


The rollout has five phases over 12 weeks. The Phase column names each phase. The Weeks column says when it runs. The Turn on column lists what you enable. The Gate to move on column says what must be true before the next phase starts.
- 0. Baseline runs for 4 weeks before the start. It turns on measurement of merge rate, review time, change failure rate, and cost per merged pull request. The gate is that the numbers are recorded.
- 1. Comment-only runs in weeks 1 to 3. It turns on shared memory, the ticket reviewer, and an advisory pull request reviewer, all read-only with no secrets. The gate is that comments are useful and noise stays low.
- 2. Maintenance runs in weeks 3 to 6. It turns on a scheduled dependency-upgrade agent. The gate is that upgrade pull requests merge without extra rework.
- 3. Build and feedback runs in weeks 5 to 10. It turns on Schedule and Build on ready tickets, with Respond in an advisory role. The gate is that merge rate and change failure rate hold or improve.
- 4. Factory improvement runs in weeks 8 to 12. It turns on a weekly retro on a fixed task set. The gate is that held-out results improve.

Run it as a measurement. Use a narrow task with a testable criterion, a maker, and a separate checker that sees only the artifact and the criteria. Calibrate any LLM judge against people on that sample. Record evidence for every generation and check. Report pass rate, escalation rate, and judge-human agreement before widening scope. More output without better numbers is not a passed gate.

Phase 0 is not optional. Four weeks of baseline is the cheapest insurance you can buy.
Phase 1 writes no code. Comment-only, no secrets, and the start of the memory later stages read.
Phase 2 writes first, and only bounded upgrades. If those pull requests do not merge cleanly, a feature agent will not either.
Phases overlap, but the gate holds. If phase 2 is not clean by week 6, phase 3 waits. Shape, Dispatch, Release, and incident remediation stay human-led.
Widen review before you widen generation. The first loop worth building is review-and-fix: one input, one output, one human gate. Authors already act on about half of a well-tuned automated reviewer's comments.
What stays human
For the life of a first factory, people still decide what is worth building, what starts next, every merge, every release, and every recovery action. Dispatch may rank. The diagnostic agent may read. Branch protection and code owners enforce the merge. A production authorizer approves version, environment, and scope.
Anthropic's playbook names four approvals: intent, plan, merge, and release (The AI-native SDLC playbook). Recovery is the same idea in production: a person decides, an agent reads. The decisions grow in weight, not in hours.

Eight guardrails hold for the whole factory. The Guardrail column names each one. The What it means in practice column says what it requires day to day.
- One identity per stage: each stage has its own app identity, never a borrowed human token. Agents cannot approve or merge.
- Least privilege, one stage at a time: each stage starts read-only, with no secrets. Write access waits for that stage's gate.
- Untrusted input: tickets, pull request text, reviews, and logs are data, not instructions. Prompt injection here is a known attack.
- Break the lethal trifecta: no agent holds private data, untrusted content, and an external channel at once.
- Hooks over prompts: rules that must hold are hooks. A stop hook requires passing checks.
- PII redaction: personal data is masked at the source, before a model or an outside trace sees it.
- Records from the pipeline: approvals, attestations, releases, and evidence are written where agents cannot write.
- Changes cannot weaken their own checks: edits to a classifier, prompt, hook, or CI workflow are high tier and apply only after merge.
A factory whose safety depends on the model noticing an attack is not safe.
Metrics from day one
Heavy AI use merges far more pull requests, review time climbs, and company-level delivery does not clearly improve. Measure throughput, quality, cost, and noise together, against phase 0. Pull request count is not the success measure.


Thirteen measures cover throughput, quality, cost, and noise. The Measure column names each one. The Why it matters column says what it tells you. The Watch for column gives the warning sign to act on.
- Revert rate, by risk tier is the main quality signal, a form of change failure. A rise after a phase or tier change stops the rollout.
- Merge rate is the share of agent pull requests that merge. Volume up and rate down means rework.
- Cycle time, assignment to merge shows whether work moves faster end to end. Faster builds and a flat cycle time mean the queue moved.
- Review minutes per change shows whether Review eases the bottleneck. It should fall in phase 1 and stay down.
- Re-tier rate counts changes that move up a tier on the real diff. A high rate means tickets understate risk.
- Overrule rate counts people overruling an agent verdict. Each overrule is a labeled eval case.
- Escaped defects are bugs after merge that a check should have caught. Each one becomes a regression case.
- Cost per merged PR is spend, including failures, per merged pull request. Cost up and merges flat means retries are burning money.
- Dismissed-as-noise rate counts comments closed with no action. When it climbs, people will stop reading.
- Escalation rate counts loops that hand work back. A rate near zero can mean the limits are too loose.
- Judge-human agreement shows whether an LLM score can be trusted. Recalibrate when agreement falls.
- Time to restore service shows whether Respond helps or adds a step. Compare incidents with and without the agent.
- Release cadence shows whether merges become releases. Merges up and releases flat means Release is the new queue.
Do not celebrate pull request count. It moves when you split work or generate changes nobody needed.
Cap review load. At Meta, lines of code in each human-landed diff grew 105.9% in a year, and agentic AI accounted for over 80% of a 51% rise in diffs per developer.
Expect a dip. METR found experienced developers about 19% slower with early AI tools while they believed they were faster (METR). A Google enterprise trial found roughly 21% faster task completion (enterprise RCT). DORA found small throughput gains and consistent losses in delivery stability (DORA). Agent-written pull requests in real datasets are accepted 35 to 64% of the time. Judge a phase at its gate, not in week one.
Start where your team is
DORA's AI Capabilities Model names seven capabilities. Two matter most here: a quality internal platform, and internal data AI can reach (DORA AI Capabilities Model). Climb one era at a time.


Each of the five eras pairs one team skill with one platform capability. The Era column names the era. The Train the team on column gives the skill to teach. The Enable on the platform column gives what the platform must provide.
- 1. Prompt engineering: train the team on clear requests and output formats, and enable approved tools.
- 2. Context engineering: train the team on short rule files, skills, and notes, and enable shared files kept short and current.
- 3. Harness engineering: train the team on hooks, tools, and subagents, and enable a sandbox, approved MCP servers, and read-only logs.
- 4. Loop engineering: train the team on goals, checkers, budgets, and escalation, and enable an agent lane in CI with a round limit.
- 5. Software factory: train the team on risk tiers and the four approvals, and enable a cloud runtime, observability, and enforced gates.
Dedicated learning time is associated with 131% more team adoption, a survey association rather than a controlled result. A weekend jump leaves skills nobody uses.
Where reasonable people disagree
A dark factory has people write specs and hidden tests while agents write code nobody reads. A light factory keeps a person on every merge, because accountability, regulation, and prompt injection still point that way. Both require real verification.
The strongest evidence for removing a gate is Meta's RADAR, which reviews and lands low-risk diffs chosen by a risk score (Automating low-risk code review at Meta). It has reviewed more than 535,000 diffs and landed more than 331,000, which are reverted at a third the rate of other diffs and cause production incidents at one fiftieth the rate. Relaxing its threshold from the 25th to the 50th percentile of risk raised the automatic approval rate to about 60%. The automatic landing is narrow, low-risk work, moved one class at a time on data. The best current benchmark of review agents still finds them missing more than 20% of the bugs human reviewers catch, especially on complex changes and bugs that span files ((When) should a software factory be lights-off?).
Start with one agent per stage, and split only when it is doing two jobs. Review is the exception: independence is worth multiple reviewers from the start. Keep specs light on small tickets and heavier on complex ones. Thoughtworks has warned against rebuilding waterfall with more documents. Operator reports of huge gains and controlled studies of slowdowns measure different work. The gains sit in bounded, verifiable, fleet-scale loops. Do not average those with an expert alone on unfamiliar code.
What changes next
Verification is the scarce resource, so spend human review hours on purpose. Put budgets, per-stage caps, and cost per merged pull request on the platform, not on the invoice. OpenAI stopped reporting SWE-bench Verified in 2026, citing contamination, slowed progress, and flawed tests in at least 59.4% of the audited problems (Why SWE-bench Verified no longer measures frontier coding capabilities). Score your own tickets, incidents, and reverts. MCP, AGENTS.md, and skills are converging, and loops built on them move between runtimes more easily.
Configuration, not a rewrite
The ticket contract, review pipeline, guardrails, memory, evals, and harness are shared. Each product brings its own owner, risk tiers, and knowledge. Build the first stages for one low-risk application, and make the second application configuration.
Three months later
Pilot admins can export small and medium workspaces. Large ones hit the size limit, the owner funds a background job, and the stages start again. A second reviewer caught a cross-workspace leak, release waited on a flag, and a read-only agent flagged an injection without changing anything. A lint rule shipped only after an eval. Every handoff had a contract, every stage a limit, and every decision a person.
Do this today
- Record merge rate, review time, change failure rate, and cost per merged pull request. Give the baseline four weeks.
- Turn on comment-only ticket and pull request reviewers, with no write access and no secrets.
- Wait to generate features until upgrade pull requests merge without extra rework.
- Block agent identities from merge and release with branch protection and environment rules.
- Treat a rise in revert rate after a phase change as a stop.
The gate is the product
Measure, comment, upgrade, then build, and only while the previous gate still holds. RADAR shows a narrow class of low-risk work can land with less harm, and review agents still miss more than 20% of the bugs people catch. Which stage did you build first, and what broke?
This is Part 14 of "AI Software Factory Series," a 17-part guide to building an AI software factory with human gates.