Rolling out an AI coding agent works the way onboarding a junior engineer works: start with one well-tested repo, keep every merge behind human review, expand autonomy only as evidence accumulates, and track merge rate, review time per PR, and revert rate the whole way. Most failed rollouts don't fail on tool choice. They fail on sequencing.
We've watched this from both sides — as a team that builds an autonomous agent, and as the people teams talk to after a rollout goes sideways. The failure stories share a shape: someone connected the flagship monolith on day one, turned on auto-merge to see the full value, and a bad PR landed in week one. From that point the tool is radioactive, regardless of how it performs afterward. This post is the sequencing that avoids that story.
Which repo should get the agent first
The team's opinion of the agent forms on the first ten PRs, not the first hundred, so the first repo decides whether the rollout survives. Pick it the way you'd pick a starter project for a new hire.
The repo you want has four properties:
- A test suite that fails when code is wrong, wired into CI that blocks merges
- Active development, but off the critical path
- A backlog of small, well-scoped issues — you need sample size
- An owner who volunteered, not one who was assigned
The repo you don't want is your most complex service, chosen "to really test it." That experiment measures the agent's worst case and tells you nothing about its median. The other repo you don't want is the dormant one nobody touches — its PRs will sit unreviewed, and the calibration data you need for every later decision never gets generated.
One filter people miss: CI runtime. Agents iterate against CI, and a 45-minute pipeline turns every retry into an hour. A repo with a five-minute suite will produce a week of learning in a day.
Start in watch-only mode, even if you're confident
Watch-only means the agent opens PRs but never merges them; a human approves everything. We've made the full case for watch-only as the default before, and the short version holds: you don't yet know this tool's error rate on your specific repo, and one bad unreviewed merge costs more trust than fifty good ones earn back.
The mistake is treating watch-only as a waiting room. It's a measurement period, and it only pays off if you measure. For every PR, log a verdict: approved clean, approved with edits, or rejected — and one line on why. Note which issue types produce clean PRs. After 10–20 PRs you have two things you didn't have before: a real error rate, and a ranked list of issue categories by quality. That list is the input to the next stage.
The watch-only period isn't a delay before the value. It's where the data that justifies every later stage comes from.
The graduated trust ladder
The binary framing — review everything forever, or turn on auto-merge — is where most teams get stuck. The workable model is a ladder, climbed one evidence-backed step at a time.
| Stage | What the agent may do | What humans do | You move on when |
|---|---|---|---|
| 1. Watch-only | Open PRs only | Review every PR, log verdicts | 10–20 PRs reviewed with verdicts logged |
| 2. Selective auto-merge | Auto-merge one low-risk issue category when CI passes | Review everything else; audit the auto-merged weekly | Two clean weeks in that category |
| 3. Expanded scope | Auto-merge most routine categories | Review high-risk paths and large diffs | Revert rate holds at your human baseline |
| 4. Steady state | Auto-merge by default | Own a permanent always-review list | You don't — the review list never empties |
Stage 2 is where the verdict log earns its keep. Don't enable auto-merge
broadly; enable it for the single category that scored the highest clean rate
during watch-only — typically test-coverage work, dependency bumps, or issues
labeled good-first-issue. Everything else stays behind review. Each later
expansion is per-category and evidence-driven, never date-driven.
The ladder also runs downward, and that's a feature. When a repo enters a major refactor, drop it back to stage 1 for the duration. The agent's pattern knowledge lags the refactor, and watch-only is the correct posture while the ground is moving.
Stage 4 never fully arrives, by design. Some changes — auth, billing, migrations, anything on the blast-radius list — should carry human review permanently. A mature rollout isn't one with no review; it's one where review is spent where it buys the most.
The metrics that tell you it's working
You need three numbers, and all of them are cheap to collect from your existing PR data.
Merge rate — PRs merged without modification, divided by PRs opened. The trend matters more than the level: a rising merge rate means the agent is calibrating to your conventions or your issues are getting better specified. A flat, low merge rate usually indicts the issue descriptions rather than the model — tickets an agent can ship are a skill of their own.
Review time per PR — the honest cost of the rollout. If reviewing an agent PR takes as long as writing the change, you've bought nothing yet. Expect it to fall as reviewers calibrate on the agent's failure modes. If it doesn't fall, look at diff size: bloated, scope-creeping diffs are the usual cause, and they're a configuration problem you can fix.
Revert rate — the safety metric, and the one that gates stage 3. Measure your human-authored revert rate first; most teams guess it wrong, usually in the flattering direction. An agent revert rate at or below that baseline is the strongest evidence you can put in front of a skeptic.
Ignore PR count and lines of code. An agent can open forty junk PRs a week, and a rollout judged on volume will optimize for exactly that.
Who owns an agent's pull requests
Governance sounds heavy for a tool rollout, but every question here is one you'd rather settle in writing before the first PR than in a postmortem after an incident.
Ownership. Every agent-connected repo gets a named human owner. The agent can't attend a postmortem, tune its own scope, or answer for what auto-merged. The owner triages its PRs, adjusts its configuration, and is accountable for the auto-merge categories they enable.
Attribution. Agent commits should be authored under a bot identity, so
git blame stays honest years from now. The accountability record is the
human approval on the PR — and for auto-merged PRs, the person who configured
that category. Never let agent output masquerade as a human author; it
corrupts both the history and the trust.
Incidents. When an agent PR breaks production, run the same postmortem you'd run for any PR that broke production. The defect is in the process that let the change merge — thin CI, an over-wide auto-merge category — not in "AI." Fix the category or the tests, and keep the ladder. Teams that respond by banning the tool learn nothing; teams that respond by tightening one rung get better every incident.
Review load. Decide who reviews agent PRs, and rotate it. If all of them route to the one enthusiast who set the tool up, it becomes that person's toy, and the rollout dies the week they go on holiday.
Rollout mistakes we keep seeing
- Auto-merge on day one. You're betting team trust on a sample size of zero. The first bad merge ends the experiment.
- The trophy repo first. Starting with the hardest codebase measures the worst case and poisons the well before the median case gets a hearing.
- No verdict log. Watch-only without notes is just a delay. You exit week two holding vibes instead of an error rate.
- Judging by the worst PR. Humans anchor on the single bad diff. Hold the agent to rates — merge rate, revert rate — the same standard you'd want applied to your own work.
- Never leaving stage 1. Reviewing every PR at full depth forever, after the data says a category is clean, caps the return at your review bandwidth. That's a quieter failure than a bad merge, but it's still a failure.
- Everyone's tool, no one's job. Without a named owner, PRs go stale, the agent looks worse than it is, and the rollout dies of neglect rather than incident.
The whole playbook compresses into one sentence: give the agent the trust curve you'd give a new hire — a starter repo, supervised work, more autonomy as the evidence comes in — and make one person own the curve. Teams that follow the sequence reach useful autonomy in about a month. Teams that skip a step usually spend that month explaining why the tool is banned.