Your Factory Gets Worse When You Stop Testing Changes to the Factory

Every change to the factory is an experiment, and the only output is an evaluated change a person can adopt or refuse.

Rick Hightower

Cover image for “Your Factory Gets Worse When You Stop Testing Changes to the Factory” by Rick Hightower

AI software factory improvement is one failure, a held-out comparison, and a person who can still say no.

A factory that never changes its own instructions gets worse even when nobody touches it. The damage shows up as slower reviews, more retries, and engineers who remember the agents being better at the start.

In this article: How an AI software factory drifts, what Improve is allowed to change, and the experiment a lesson must pass before anyone adopts it.

A flaky-test workaround outlives the fix. The same missing workspace filter is flagged for the twentieth time and never becomes a lint rule. Last quarter's skill still teaches last quarter's patterns. Instruction files contradict each other and cost more tokens every run, and superseded decisions sit in shared memory. You see more review comments, more retries, a higher cost per merged pull request, and engineers who remember the agents being better at the start.

AI software factory improvement is that retrospective, on evidence, and every change touches every other stage at once.

An evaluated change, or nothing

Table: , Improve. Rows: Trigger; Work; Independent check; Memory; Stop condition.

The Improve stage runs on a five-part contract. The first column names each part of the contract, and the Improve column says how this stage fills it in.

  • Trigger: a weekly timer, a repeated failure, an escaped defect, or a cost spike starts the stage.
  • Work: the agent mines failed runs, retries, and dismissed review comments, proposes changes to skills, rules, and instructions, and marks superseded knowledge.
  • Independent check: an independent reviewer checks the comparison, and the factory owner decides whether to adopt the change.
  • Memory: the stage draws on failure patterns and findings from the other stages, plus past experiments.
  • Stop condition: the stage ends with an evaluated change for a human decision, or with a recorded "no change."

Shrinks for the human: fixing the same mistake by hand, again. Grows for the human: deciding which lessons become rules, and owning the evals that judge every change to the factory.

The output is an evaluated change. Propose it, test it on a fixed task set, hand the comparison to a reviewer. One experiment at a time. Two at once, and you cannot tell which change helped or which broke something.

Where every return path lands

Earlier stages send the lesson here.

Improve, its inputs and outputs

  • Shape finds features that shipped and nobody used. Shipping is not adoption.
  • Plan sees the same missing validation plans and access rules.
  • Schedule records work discovered late, and plans split too finely.
  • Build records every retry, every stop, and every question handed back.
  • Review records repeated findings and comments dismissed as noise: a missing rule, or a miscalibrated reviewer.
  • Respond records incident patterns and their causes.
  • Maintain records what broke during an upgrade, and the fix.

Outputs are lint and other deterministic checks, skills and instructions, reviewer rubrics, and shared memory. A repeated comment that can become a lint rule should. A lint rule costs nothing to run and never forgets. Only what cannot be checked mechanically belongs in an instruction file.

Match the fix to the failure

Table: Failure, Layer that fixes it. Rows: You keep re-explaining a project fact; The fact matters only for some files; You keep retyping the same procedure; It must happen every time, whatever the model decides; The agent cannot see the result of its work; Reading and searching flood the main…

Each repeated failure has one layer of the harness that fixes it. The Failure column describes the symptom you keep seeing. The Layer that fixes it column names where the fix belongs.

  • You keep re-explaining a project fact: put it in a rule file.
  • The fact matters only for some files: use a path-scoped rule.
  • You keep retyping the same procedure: turn it into a skill.
  • It must happen every time, whatever the model decides: enforce it with a hook.
  • The agent cannot see the result of its work: give it a tool or an MCP server.
  • Reading and searching flood the main window: move the work to a subagent.
  • The setup is stuck in one repository: package it as a plugin or a shared configuration.

Each failure maps to the cheapest layer that can hold the fix.

A comment made twice belongs in one of these layers, as far down the list as it can go.

Every harness change is an experiment

A new rubric or new build instructions changes every engineer at once.

One problem, then a held-out comparison, a shadow run, and adopt or roll back.

  1. Choose one specific problem. Not "make reviews better," but "the reviewer keeps disagreeing with the team about how large exports should be split into batches." People check the examples and remove sensitive details first.
  2. Compare old and new on tasks the change was not written from. Use past tickets, pull requests, and incidents with known good outcomes. A change written from three examples and tested on the same three proves nothing.
  3. Run it alongside before it decides anything. New results sit next to the old ones, visible but not acted on. Shadow runs catch cases the held-out set missed.
  4. Keep the history so you can roll back. Version the change with the comparison that justified it. When something gets worse three weeks later, find it and undo it.

A line that fixes batching advice can make every other review more verbose. You only see that outside the original problem. The score is the anchor. Wrong cases become the regression suite, and a human override becomes a labeled case. A green suite still proves only what someone thought to check. It bounds the damage. It does not remove it, so the gates stay.

The work can split: one subagent for skills and instructions, another mining traces for cost and retries. LangChain's version of that trace mining fetches experiment traces, spawns parallel error-analysis agents, has a main agent combine their findings, and makes targeted harness changes, with optional human review of each change.

Who decides, and what a better score never grants

The reviewer checks the comparison. Only the factory owner adopts.

Reviewer and owner are different people. The reviewer checks that the tasks were held out, that the gain is real, and that nothing got worse. The owner adopts or refuses. The proposer decides nothing, and will present the comparison favorably, in the loose sense agents have incentives. Use a different model family, with a person behind the reviewer.

Better results never grant new merge or production powers. More bugs caught does not earn auto-merge. Cutting retries in half does not earn production access. Permissions, tiers, and gates are a separate decision, with their own proof. A change to the risk classifier, a review prompt, or a CI workflow is high tier and cannot weaken the checks on itself. A lesson can propose a rule. It cannot install one.

Keep the shared memory honest

Once a week, read the failure patterns, fix records, and findings, and do three jobs.

  • Consolidates. Five fix records about the same library default become one rule.
  • Supersedes. Mark the old entry. Do not delete it. Stale ADRs fool agents because an old decision reads exactly like a current one.
  • Promotes. A recurring lesson leaves the knowledge tree for a lint rule, a CI check, or a skill.

An engineer approves each reusable lesson before other assistants use it. Add context only when a comparison earns the tokens. The OKF agent-graph plugin can record which orchestrators run which agents, which judges and rubrics each reviewer uses, and which prompts each step loads, with the experiment beside that diff. Why the reviewer started flagging batch sizes in March is one query away.

What has been published, and when to start

The evidence is medium. Operators have described the pattern. None has published measured outcomes for the stage itself.

  • Anthropic turns past incidents from agent-written code into review evaluations, and blocks reviewer or harness changes that lower the score.
  • OpenAI runs "doc-gardening" agents and encodes taste in lint rules rather than prose.
  • Cloudflare reviews instruction files with an agent, and reportedly generated them for thousands of repositories. The scale figure is secondhand.
  • Thoughtworks describes the same idea as a feedback flywheel in its Technology Radar.

Start from Claude Code skills and plugins, LangSmith or Langfuse, session captures kept in Git, and the OKF agent-graph plugin.

Start small, and start late. In a measured rollout this is the last phase, weeks 8 to 12, because the other stages have to produce data first. Until then, do two cheap things: record repeated findings and dismissed comments from day one, and build the held-out set from real tickets and pull requests. Then one weekly retro and one experiment. If held-out results do not improve, record "no change." A week that ends that way is a good week.

Records from day one, a start in weeks 8 to 12, then one export change at a time.

The workspace export

Same month, one feature, three lessons. A workspace filter missing from helper queries, flagged four times across pull requests. A dependency default the code relied on, changed twice in one quarter. Disagreement on how large exports should be split into batches. One experiment at a time.

The filter goes first: fail any data-layer query without a workspace filter. On the last 50 merged pull requests the rule catches all four known cases and raises two false positives, both in admin tooling, both given an exemption. Reviewer and owner approve. The comment now runs in seconds.

Batching goes second. Clearer export-only instructions, tested on held-out export reviews, then two weeks beside the old ones. More agreement, no new noise elsewhere. Adopted for export reviews, comparison stored on the agent graph. The dependency rule waits in memory for maintenance.

Do this today

  • Name the review comment you have repeated most this year. Ask if a lint rule can carry it.
  • Log repeated findings and dismissed comments before the retro exists.
  • Hold out real tickets, pull requests, and incidents, and keep them out of prompts and skills.
  • Run one experiment on other tasks, shadow it, and keep a version you can roll back.
  • Someone other than the proposer checks the comparison. Permission changes stay separate.

A week that ends in "no change"

Rot is a workaround that outlives the bug. One evaluated change a week, or an honest "no change," is the whole stage. What review comment has your team made most often this year, and could it be a lint rule instead?


This is Part 11 of "AI Software Factory Series," a 17-part guide to building an AI software factory with human gates.