If the Factory Cannot Check Its Own Work, You Cannot Tell It Is Helping

Checks, held-out tasks, calibrated judges, and traces from your own factory, in a fixed order of trust.

Rick Hightower

Cover image for “If the Factory Cannot Check Its Own Work, You Cannot Tell It Is Helping” by Rick Hightower

AI agent evals are inline checks, held-out tasks, calibrated judges, and traces. A launch-page score is not one of them.

A vendor benchmark score says almost nothing about your tickets, your reviewers, or your code. Without checks you own, a busier factory is easy to mistake for a better one.

In this article: Two checks for every stage, four kinds of check in order of trust, and one LLM judge you calibrate against your own engineers before you scale it.

Dismissed comments, tests before a pull request, separate reviewers, and held-out comparisons are evals. A vendor score says little about your code, your tickets, or your reviewers, and public benchmarks age fast. OpenAI stopped reporting SWE-bench Verified in 2026: the benchmark had become contaminated, progress on it had slowed, and at least 59.4% of the audited problems had flawed tests that reject correct solutions (OpenAI, 2026).

What each stage should check

An inline check runs on every piece of work before it moves on. A stage metric says whether the stage as a whole is doing its job.

Table: Stage, Inline check, Stage metric. Rows: Shape; Plan; Dispatch; Schedule; Build; Review; Release; Respond; Maintain; Improve.

Each of the ten stages gets two kinds of check. The Stage column names the stage. The Inline check column gives the check that runs on every piece of work before it moves on. The Stage metric column gives the number that says whether the stage as a whole is doing its job.

  • Shape: the inline check is that the owner can explain the PRD without reading it. The stage metric is the post-release benefit measured against the success measure.
  • Plan: the inline check is that the ticket meets the ready rubric. The stage metric is the count of reviewer comments dismissed as noise.
  • Dispatch: the inline check is that a budget and limits are attached before work starts. The stage metrics are review queue length and budget overruns.
  • Schedule: the inline check is the gatekeeper, which confirms no cycles, criteria on every package, and sizes in range. The stage metrics are discovered work per ticket and reclaimed leases.
  • Build: the inline check is that tests, types, and lint pass inside the loop. The stage metrics are retries per pull request, merge rate, and cost per merged pull request.
  • Review: the inline check is that findings are corroborated before a person sees them. The stage metrics are escaped defects, revert rate, and override rate.
  • Release: the inline check is policy gates and provenance on every artifact. The stage metrics are change failure rate and release cadence.
  • Respond: the inline check is evidence attached to every claim the agent makes. The stage metrics are time to restore and repeat incidents.
  • Maintain: the inline check is that characterization tests pass before and after. The stage metric is upgrade pull requests merged without rework.
  • Improve: the inline check is that the held-out comparison beats the current version. The stage metric is results on held-out tasks over time.

Most inline checks are deterministic: tests, schemas, scripts, policy rules. The cheapest reliable eval needs no model.

AI agent evals, in order of trust

Four kinds of check. Move down only when the rung above cannot answer.

four kinds of check, in order of trust

Deterministic checks come first. A test passes or it does not. A schema validates or it does not. Acceptance criteria, policy, and repeated comments belong on this rung when they can. Anthropic puts it in one line: deterministic graders where possible, model graders where necessary.

Held-out comparisons measure change. Run old and new on the same fixed past work. A shared set separates an improvement from a coincidence.

LLM judges add breadth for a clear PRD, a useful comment, or an accurate release note, and only after the scores agree with your people.

People anchor everything. A named person approves every merge and every release, and also sees a sample of what the judges score.

Useful, biased, and calibratable

On the MT-Bench and Chatbot Arena studies, a strong model agreed with human preferences more than 80% of the time, about as often as people agree with each other. The 80% agreement is also narrower than it sounds: it held on that benchmark's construction, and the same work documented systematic biases.

Table: Bias, What it looks like, Countermeasure. Rows: Position; Verbosity; Self-enhancement; Weak domains.

The same studies documented four systematic biases in LLM judges. The Bias column names each one. The What it looks like column describes how it shows up in scores. The Countermeasure column gives the fix.

  • Position: the judge prefers whichever answer it sees first. Counter it with pairwise comparisons in randomized order, scored both ways.
  • Verbosity: the judge prefers longer answers. Counter it with a rubric that has explicit criteria, and with length-normalized scoring.
  • Self-enhancement: the judge prefers output from its own model family. Counter it with a judge from a different family than the author.
  • Weak domains: the judge is less reliable on math and code. Counter it with execution-based checks for anything a test can decide.

Four judge biases and a countermeasure for each.

Calibrate before you trust. People and the judge score the same sample, say 50 items. If the judge disagrees with your engineers more than they disagree with each other, fix the rubric before you scale it.

Recalibrate on a schedule. A fresh human sample every few weeks. A judge calibrated in March is an assumption by June.

Do not let the judge be the only check on its own author. Same family and same context share blind spots. Use a different family, a separate session, and a person on the sample.

Shape the question. Pass or fail, or a pairwise comparison, not a free score. One judge per dimension, so clarity, correctness, and completeness stay apart.

Grade the outcome, not the path. A transcript can show the bad step. Requiring the exact steps penalizes valid approaches.

Blind the judge to the maker's reasoning and tool log. Manipulated reasoning traces can inflate false positives sharply.

One benchmark study and operator experience are a starting point. Recalibrate on your own tasks.

People and the judge score one sample before the rubric is trusted.

Held-out tasks from your own work

"This helped" needs past work with known good outcomes, kept out of every prompt and skill.

A case leaves the suite if it leaks into a prompt or the code moves on.

  • Past tickets merged cleanly, with the final diff as the reference.
  • Past review findings, real ones and a few dismissed as noise, so a reviewer is scored on catching and on not crying wolf.
  • Past incidents and reverts a reviewer or a test should have caught. Anthropic builds its review evals this way and blocks reviewer changes that lower the score.
  • Past discovered work, so a planner is scored on finding the missing steps up front.

A case inside a prompt or a skill is no longer a test. Add cases each month, and retire ones the codebase has outgrown. A few dozen cases can catch a regression. Four cases in a category cannot establish a reliable rate, so a clean run is not proof of safety. Record the prompts, skills, and models for each run, or a result six weeks later cannot be traced.

Anthropic suggests 20 to 50 simple tasks from real failures, with the suite held near a 100% pass rate. Run it on every change to prompts, rules, skills, or hooks, on the same baseline, and add a case for every incident. LangChain warns that a harness tuned to one task regresses others. A fix that fails the suite does not ship.

Traces are what evals run on

Every run should record prompt and model version, tools and arguments, retrieved context, output, time, and cost. The trace is a typed tree. Evaluators call that tree.

An evidence record is the claim beside it: actor, time, inputs, output, check result, and typed links such as derived from, checked by, and approved by. Store both. Link them. Do not build the store yourself.

Table: Platform, Fits when. Rows: Langfuse; LangSmith; Microsoft Foundry evaluation; Amazon Bedrock AgentCore observability; DeepEval, promptfoo, and similar.

Five kinds of platform store traces and run evals, so you do not build the store yourself. The Platform column names each option. The Fits when column says when that option is the right choice.

  • Langfuse fits when you want open source, with the option to self-host traces, datasets, experiments, and prompt versions.
  • LangSmith fits LangChain or LangGraph agents, or a team that wants a hosted platform for tracing, datasets, and evaluators.
  • Microsoft Foundry evaluation fits agents on Azure, with evaluation next to deployment.
  • Amazon Bedrock AgentCore observability fits agents on AWS, with traces in the same place as the runtime.
  • DeepEval, promptfoo, and similar tools fit when you want evals written like tests and run in CI on every change.

Mask personal information before a trace leaves, not after. A trace proves what happened, not that it was right. A fully traced agent can still be confidently wrong.

The workspace export

Clearer export-review instructions were scored on 30 past export reviews, each labeled by an engineer as a real finding or noise. Old and new ran on all 30. A judge from another model family used a three-point rubric: correct, actionable, and specific. An engineer scored the same 30 first. Correctness agreed. Actionability did not, because the judge favored whichever review it read first. Random order, scored both ways, closed most of the gap. Those scores, beside the engineer's labels, supported the new instructions, and the run was stored as evidence records.

Do this today

Trace, then cheap metrics, a small held-out set, one calibrated judge, and a gate.

  • Trace the first comment-only agent from day one, and mask personal data before anything leaves.
  • Record dismissed-as-noise rate, time to first review, merge rate, revert rate, and cost per merged pull request.
  • Start from twenty to thirty past tickets and review findings. Add cases every month.
  • Calibrate one judge against people on fifty samples before you trust it.
  • Adopt no harness change without a held-out comparison and a person.

A score you did not build cannot grade your factory

OpenAI stopped reporting SWE-bench Verified in 2026, with flawed tests on at least 59.4% of the audited problems, after contamination and stalled progress. MT-Bench's agreement of more than 80%, about as often as people agree with each other, held on that benchmark's construction, and the same work documented systematic biases. What quality do you most need to measure, and still have no test for?


This is Part 12 of "AI Software Factory Series," a 17-part guide to building an AI software factory with human gates.