The Review Queue, Not the Model, Decides Whether Your AI Factory Works

Review is the stage to build first, because review hours are the scarce resource, and a different model name is not independence.

Rick Hightower

Cover image for “The Review Queue, Not the Model, Decides Whether Your AI Factory Works” by Rick Hightower

Build AI code review before anything that writes code. Specialist agents help. A different model name is not independence, and the human merge stays.

The queue of unread pull requests is the real limit, not how fast an agent can type. A reviewer people learn to ignore is worse than having no reviewer at all.

In this article: Why Review comes before any agent that writes code, how specialist reviewers stay independent of the author, and how risk tiers keep a named person on the merge.

Teams with heavy AI use merge far more pull requests. Those pull requests are much larger, and review time climbs with them. The wall is not generation. It is the hours qualified people still have to read, judge, and approve. Review hours did not rise when the agents arrived.

Build Review first: before the build stage, before the ticket reviewer is finished, before anything that writes code. A review agent only comments. It touches no production code, it runs on pull requests people already write, and it records a baseline for review time.

A reviewer that saves each human reviewer twenty minutes per pull request is worth more than a build agent that writes ten pull requests nobody has time to read. Self-review fails. When GPT-4 revised its own answers without outside feedback, grade-school math accuracy fell from 95.5% to 91.5%. On a planning benchmark it scored 40% alone, 55% with self-critique, and 87% with an external verifier. Judges also recognize their own output and score it higher. The writer should not be the grader.

The stage at a glance

Table: Part, Review. Rows: Trigger; Work; Independent check; Memory; Stop condition.

Review's loop has five parts. The Part column names each part, and the Review column says what that part is for this stage.

  • Trigger. Review runs when a pull request is ready, or when new information makes an earlier review outdated.
  • Work. It checks the change against accepted requirements, assesses the evidence, and finds defects.
  • Independent check. A qualified person decides the merge.
  • Memory. It recalls team standards, recurring findings, and past defects in the same area.
  • Stop condition. It stops at a review report a person can act on.

Shrinks for the human: reading every line. Grows for the human: deciding the merge on evidence, and owning the checks every change must pass.

A review belongs to one diff, one set of requirements, and one set of test results. A scope change or a rebase makes it stale, and the stage runs again. A finished AI session is not a pass. A readiness report is not approval.

A review stays valid for one diff. Scope changes and rebases send it back.

Specialists, then one report

A single prompt, "review this PR," is where most teams start. Cloudflare got a flood of vague suggestions. OpenAI dropped its generic reviewer for domain specialists. Fan-out and fan-in is the pattern that works.

Review architecture, fan-out and fan-in: "Prepared bundle: diff, ticket, tests", then "Security", "Standards", "Spec match", and "Tests", then "Vote", "One report", and "Person merges".

One step prepares the diff, the ticket, and the tests. Specialists then take narrow briefs. Security looks for injection, authorization gaps, secrets, and unsafe data handling. Standards checks rules written as documents, not tribal knowledge. Spec match walks the ready ticket's acceptance criteria and says where each is satisfied. Tests and evidence asks whether the tests exercise those criteria, and what was skipped.

A coordinator drops duplicates into one report, and Anthropic corroborates each finding with a second agent before a human sees it. In one study of multi-agent debate, majority voting alone accounted for most of the gains usually credited to the debate. The Claude Code team votes findings across agents and posts one only if enough agree. A cheap scorer sees every change. Only uncertain cases go to a slower judge or a person.

In a controlled study, developers using an AI assistant wrote less secure code and were more likely to believe it was secure. Veracode's testing of more than 100 models found that about 45% of generated samples failed security tests, a rate that stayed roughly flat for two years while functional correctness improved. The check has to be automated, and independent of whoever wrote the code. A workflow engine earns its place with four reviewers and a coordinator.

A different model name is not independence

The assistant that wrote a change cannot approve it. A second model family catches shared blind spots, and CI can use another vendor. It is not enough. Reviewers that share the author's summary, one window, and each other's findings repeat the same mistakes. Split the sessions, the information, and the permissions, and take requirements from somewhere other than the author's account of what it built.

Blind the judge. The checker gets the diff, the spec, and the rubric, and nothing of the maker's reasoning or tool log. In a study of 800 agent trajectories, manipulated reasoning traces inflated the false positive rates of state-of-the-art judges by up to 90%. The checker also holds fewer tools than the maker, and none that write.

Two model families read the ready ticket in separate sessions. Agreement is evidence for a person, not the merge itself.

  1. Two reviewers, two model families, separate sessions. Keep both first results before either sees the other.
  2. Requirements come from the ticket store, not the pull request description.
  3. No shell in the first review. Tests run in ordinary CI, without privileged credentials. A reviewer that can execute an untrusted diff is a build agent with a different name.
  4. Agreement is evidence, not proof. It does not replace the human decision.

If two approvers are required, one person cannot fill both slots, and an agent identity can fill neither.

Risk tiers, and a human merge that stays

Rules and a classifier set the tier.

Table: Tier, Typical change, Who approves. Rows: 0; 1; 2.

Three risk tiers decide who approves a change. The Tier column gives the tier number. Typical change describes the kind of change that lands in that tier. Who approves says who or what must approve it.

  • 0. Documentation, and dependency bumps that tests cover, land in tier 0. Reviewer agents, green CI, and a security scan approve them. Automatic merge comes only after that class has a measured revert rate.
  • 1. Normal, well-tested work lands in tier 1. Verified specialist findings come first, then one person approves.
  • 2. Tier 2 covers auth, secrets, money, personal data, CI/CD, infrastructure, public APIs, model logic, or anything tests cannot verify. Two people approve, one of them a code owner. Security can block the change, and the merge queue runs the checks again.

Tier labels travel with the ticket. Tier 0 waits on revert data. Tiers 1 and 2 keep a named person.

The person who started the agent never approves its pull request. SLSA's highest source level requires two trusted persons to agree to every change on a protected branch, and the agent's operator counts as one of them. No agent identity can bypass branch protection. The tier is a label on the ticket that travels into the pull request.

Anthropic's Claude Code team lets docs merge once automated and security checks pass. Billing, the auto-updater, and CI workflows always need a person. A second classifier can force a person even when the tier would not.

  • A pull request that edits the risk classifier, the review prompts, or the CI workflow is high tier. New rules apply only after it merges. A change may never weaken the checks that apply to itself.
  • Agents may not edit tests and code together unreviewed. Joint edits are the easy way to make a failing check pass without a fix.

Specs, test assertions, and harness rules always go to a person. An agent that can loosen the checks that grade it can turn any build green. A hook can block test-file edits while an agent fixes a bug.

Start with tier 0 empty, and keep one human approval on every change until months of data exist. Automatic merge needs its own proof. Branch protection and code owners put a person's name on every approval. "The pipeline" is not an answer. The human merge stays.

What the reviewer remembers, and what it costs

Cloudflare's standards reviewer reads an internal library of RFC-style engineering standards. Keep ADRs, architecture, and review rubrics in shared memory. The OKF plugins are one example: Project Knowledge Capture, System Architecture Capture, and the agent-graph plugin. A rubric change is a reviewed diff.

When the same comment shows up on the twentieth pull request, make it a lint rule, a skill, or a line in the standards.

Standards to read, repeated findings to turn into rules, and three measures from the first week.

Paid reviewers charge per pull request or per seat as volume climbs. An owned pipeline can use a cheaper model for standards, a stronger one for security, and another family for the independent check. Claude Code review, its GitHub Action, CodeRabbit, Qodo, Graphite, and the Claude Agent SDK are examples you can swap.

Count review in the same budget as the build. A starting cap counts every attempt, review, and repair together, and stops at two repair rounds. A third round indicts the ticket, not the reviewer.

Published results, from the operators themselves:

  • Cloudflare runs up to seven specialist agents per merge request, with risk tiers and model failback. In its first 30 days it reviewed 48,095 merge requests across 5,169 repositories, median review time under four minutes. Engineers overrode it in 288 cases, about 0.6%. Agents advise. Blocking applies only to enforced standards, with a human override.
  • Anthropic's Claude Code team uses self-verification, parallel bug-hunting with corroboration, a blocking security review, risk-tiered human approval, and a merge queue. Merged pull requests doubled over a quarter, and reverts fell by more than 40%.
  • OpenAI replaced one generic reviewer with domain specialists.

None of them dropped the human merge for meaningful changes. Precision beats recall. When Cloudflare's cost work cut serious findings by about 40%, authors acted on 59% of them, up from 34%. OpenAI tunes for precision too, and authors change code for about half of its comments.

Start advisory on internal pull requests: a GitHub Action, a read-only token, no shell, no secrets, two or three specialist prompts, and one synthetic case before a real pull request. Measure time to first review, dismissals without action, and escaped defects. High noise means tune first. A reviewer people ignore is worse than none.

A fictional workspace export

A trimmed export pull request hits Review. The story is fictional, and the ticket is high tier because it touches data access.

Two reviewers read the ready ticket from the store, not the pull request. Security finds that the main query filters by workspace while a helper loading attachment metadata does not, so another workspace's file names could leak. The other model family missed it. A double miss would have looked just as clean.

The report blocks on that helper, the query, and the tenant-isolation ADR. The maintainer sends it back. The build agent adds the filter and a failing test inside the repair budget. Review runs again, the old report is stale, and the maintainer and the code owner approve. The pattern goes into shared memory as a lint candidate.

Do this today

  • Run an advisory reviewer on internal pull requests: a GitHub Action, a read-only token, no shell, no secrets.
  • Split the prompt into two or three specialists, and keep both first passes before either sees the other.
  • Read requirements from the ticket store, not from the pull request description.
  • Leave automatic merge off until a change class has a measured revert rate. A named person still approves the rest.
  • Track time to first review, dismissed-as-noise rate, and escaped defects before you add a builder.

The hours were the constraint

Generation got cheap. Judgment did not, so Review is the stage to build first. A second model helps, and independence still means separate sessions, a blinded checker, and requirements the author did not write. The human merge stays under a person's name.

What does your reviewer catch that people miss, and what does it flag that people ignore?


This is Part 7 of "AI Software Factory Series," a 17-part guide to building an AI software factory with human gates.