At 2 a.m., the Agent May Read. Only You Restore Service.
A diagnostic agent may investigate an incident, but people restore service, and human recovery never waits on the model.

AI incident response fails when the model becomes a step in recovery. A diagnostic agent can group the signals and explain the change log. People decide, and they do not wait.
The page arrives after a release full of agent-written changes, and the factory that produced them has nothing ready for the on-call engineer. An incident agent that can touch production is the wrong kind of help; a read-only diagnosis is the right one.
In this article: A read-only agent can group signals and explain a change log. People still restore service, and they do not wait on the model or its budget.
A customer cannot export, at 2 a.m., on a release with eleven agent-written changes the on-call engineer did not personally review. The factory that wrote them is silent. A bad build agent opens a bad pull request. A bad incident agent touches production while customers watch. The agent investigates and explains. People decide and act.
What the stage is for
An alert, a page, or a user report starts the work: group signals, read what changed, query logs and metrics, propose causes. On-call decides what is true. Memory is past incidents with the same signature, known workarounds, and the architecture of what is running. Stop when service is restored and a refinement ticket exists.
Shrinks for the human: assembling context at 2 a.m. Grows for the human: every recovery decision.
Human recovery must not wait for an AI model or its budget. If the provider is down, the budget is spent, or the analysis is slow, on-call proceeds without the agent. A runbook that starts "ask the incident agent," or a rollback the agent must assemble, has made the model a dependency.
An incident, two outcomes
Users working again, and the cause staying fixed, are different jobs. The picture is "an incident, two outcomes."

Restore first. Group repeated alerts and reports into one incident. The agent assembles evidence. On-call picks the fastest safe path: a rollback, a flag turned off, or a documented workaround. An authorized person runs it. Restored means on-call says so from health checks, not that the analysis finished.
Then fix the cause on the normal path. A refinement ticket carries the evidence through Plan and Dispatch, the ticket reviewer, and the same build, review, and release path as any other change. No emergency lane. No hotfix from the finder. The fix starts with a test that fails on the original bug.
The rollback is the recovery. The fix is follow-up.
The diagnostic agent reads. It never fixes.

What it does:
- Groups signals. Ten alerts and four support tickets about slow exports become one incident.
- Reads what changed. The change log from the last known-good release, with tickets and pull requests. The list can be long, and the agent saves the most time on that list.
- Queries logs and metrics on read-only connections, and lines the failure up with what shipped.
- Proposes, with evidence. Ranked causes, the logs or metrics behind each, and a rollback or flag change for a person to choose.
- Names its gaps. If it cannot verify which version is running, it says so.
What it must never do: act on production (the authorized operator runs the rollback), follow instructions found in logs or tickets (a line that asks for an administrative action is suspicious data, not permission; this is the lethal trifecta, with production telemetry as the untrusted content), or block recovery when it is slow, down, or out of budget.
The context that turns logs into a diagnosis
Logs alone are not a diagnosis.

What is running. Services, dependencies, where they run, what talks to what. Keep it in System Architecture Capture, the OKF plugin for the "what is running," or in Git. A failed export can point at permissions and storage before anyone opens a log.
Who is affected. Tenant, account, region, where the logs live. A page or spreadsheet is enough. The engineer and the agent share one lookup.
What changed. The change log between releases, linked to tickets and pull requests. One query if traceability held. Commit messages and guesses if it did not.
Redact personal data before the agent reads logs. Mask names, emails, and customer content in the feed. Do not ask the model to ignore them.
Where the incident goes after the page
Three outputs leave. The permanent fix goes to Plan's ticket reviewer, then Dispatch, with the same rigor as any other ticket. Recurring lessons go to Improve, because a pattern across incidents is a factory problem. Remaining risk goes to Maintain, as upkeep rather than a feature.

Stop when service is restored and a refinement ticket carries the evidence, linked in the ticket store. The fix, its tests, and its release then trace to the problem.
Evidence has a fixed shape: who or what produced it, when, from which inputs, what it concluded, and what it was checked against.
The agent first recalls the same signature: service, error, and how soon after a release. The fastest diagnosis is that earlier record, including what fixed it. It then stores the new pattern on the ticket for Improve.
What the evidence actually supports
The evidence for agents in Respond is weak to medium. OpenAI has described a "Perf Factory" that watches production, finds the root cause of latency regressions, and proposes fixes as PRs for people to review. Anthropic converts past incidents into evaluations for its review system, which is a Respond-to-Improve link more than an incident agent. Both are operator reports, and both keep a person between the agent and production.
OpenAI's factory, as reported, draws the same line for its incident agent: it proposes mitigations but does not execute them. Anthropic's playbook adds the return arrow: every production incident gets its own eval, which runs whenever a rule file, skill, or hook changes, so the factory cannot quietly reintroduce the same failure.
Tools are examples. Swap any of them: observability MCP servers; Langfuse or LangSmith, with personal data masked before traces leave your environment; a read-only Claude Code session, started by on-call, with the change log and architecture map; your incident tool; WikiTicket or any ticket store; System Architecture Capture.
Once a week, off the live page, an agent drafts tickets from recurring errors and slow endpoints. Before the first real deployment, preload a read-only session with the change log and the map. On-call chooses when to open it. Measure mean time to recovery from day one, with the agent and without it.
A fictional workspace export
Two weeks after a pilot gets a workspace export, tickets report slow exports and timeouts, then alerts. A running example, not a vendor case.

The agent groups the alerts and the three tickets. A Maintain upgrade changed database-client pooling. The slowdown started within an hour, and only large workspaces are affected. It cannot confirm the client version on every node, and it says so. Smaller date ranges are the workaround. Turning the export flag off for the pilot is allowed. One log line says to raise that customer's size limit. The agent flags it as suspicious data and does nothing.
On-call passes the workaround to support and leaves the upgrade, because it fixed a security issue. Small exports are fine. The incident is mitigated with a ticket for the pool setting, the timing, and the workspace sizes. Plan, Dispatch, and the reviewer follow, starting with a test that reproduces the slow export. The pattern goes into shared memory for Maintain and Improve.
Further reading
- Harness Engineering: Observability, The Green Dashboard and the Seven Customers
- LangFuse I: The Ticket You Cannot Reconstruct Is Why You Need Langfuse
- LangFuse XI: Your Langfuse Traces Are Customer Data. Mask Them.
- AgentCore X: Your Agent Is Fully Traced. That Still Does Not Mean It Was Right.
- How we monitor internal coding agents for misalignment, OpenAI
- Inside OpenAI's agentic software factory, The Pragmatic Engineer
- The AI-native SDLC playbook, Anthropic
Do this today
- Write rollback, flag, and workaround steps that run with the model offline.
- Preload a read-only session with the change log and an architecture map. On-call may skip it.
- Mask names, emails, and customer content before logs reach a model.
- Keep the incident open until a refinement ticket links the evidence.
- Weekly, off the page, draft tickets from recurring errors for a person to accept or reject.
Recovery is a decision, not a model completion
The factory can write the change that pages you. It cannot be what you wait on.
Group the noise, read the change log, admit the gaps, restore service, and leave a ticket. If the model is silent at 2 a.m., service should still come back.
This is Part 9 of "AI Software Factory Series," a 17-part guide to building an AI software factory with human gates.