More than 70% of the code my team ships is now AI-generated. Velocity stopped being our constraint a while ago. What replaced it is a question the team asks several times a week, and until this year we had no defensible way to answer it.
The question every developer on my team is now asking
The volume of work my team carries is high and the velocity expectation is higher. That is the operating reality, and agentic development is what makes it workable. Code arrives faster than it ever has.
Which is where the doubt starts, and it is worth putting in the team's words rather than mine. How do I know the quality of the code that was just generated is good enough? Are the test cases exhaustive enough?
Both are the right questions and we could not answer either. A developer could read the diff, run the suite, watch the pipeline go green, and still not tell you whether the change was safe. Green means the tests that exist passed. It says nothing about whether the tests that matter exist.
The more uncomfortable part is what we were doing instead. Quality assessment on my team was distributed and largely subjective – every pod had a sense of how healthy its code was, no two pods arrived at that sense the same way, and there was no defined method for any of it. That includes my own judgements in review meetings, which I would have described at the time as informed. They were opinions with a confident delivery.
When we went through our incident root cause analysis looking for a pattern, the gap showed up as a number. Roughly four out of five incidents were caused by avoidable testing gaps rather than infra failures or unforeseeable events. That was already true before agents wrote a line of our code. What agents changed is the multiplier. A single developer now ships an order of magnitude more code, and multiplying untested surface area is not a productivity story. It is a forecast.
So the bottleneck moved. It used to be writing the code. The code mostly writes itself now, and the constraint is whether we can trust it.
From a checkpoint to a loop
The instinct is to strengthen the gate at the end. That does not work, for a structural reason: a checkpoint at the end of the cycle inspects change at the rate one person can read it, and that rate is now the slowest thing in our pipeline.
The old shape is familiar. Test cases live in spreadsheets or a test case management tool, disconnected from the code. QA runs at the end of the cycle, after the decisions that caused the bugs are locked in. Pull requests carry no quality data. Coverage percentage gets mistaken for test quality. Bugs surface in staging, or in production.
What we are moving to differs at each of those points. Builders own their test cases, and the test cases live in the repository beside the code they protect. Engineers validates AI-generated tests inside the loop, while the work is happening. Every pull request ships with an impact report. Mutation testing proves that tests catch defects rather than just executing lines. And one number tracks the whole thing continuously instead of at a gate.
Underneath all of it sits one reframing, which is the actual idea in this post. In the old model a test case answers did we build it right? – a question you can only ask about something already built. It is a verdict, and a verdict has to be read by someone who still has time to act on it. In the loop model a test case answers something else: what is this agent allowed to change, ship, and own? That is a boundary, and a boundary is enforced at the moment of the change. A test case used to be a verdict, but now it is a guardrail.
Getting there took three moves, in order: measure it, generate it, enforce it.
Measure it
We could not have a useful conversation about quality while it stayed a matter of opinion, so the first move was to turn it into a number.
If 10% of our merged pull requests exist only to fix defects introduced by earlier ones, our Defective Change Rate is 0.10 and our Build Quality Score is 0.90. The score runs from zero to one, higher is better, and we band it so everyone on the team reads it the same way, with 0.95 as the elite bar we hold ourselves to.
The point of a single number is not precision. It is that a number forces a conversation. "Quality feels a bit thin this quarter" gets nodded at and forgotten. "BQS dropped from 0.93 to 0.87" gets a meeting, an owner and a date.
Defective Change Rate is our internal proxy, analogous to but not identical to DORA's Change Failure Rate. We picked bug-fix PR rate because it is cheap to instrument from data we already have, and because it is visible in the same system where developers already work.
There is an obvious weakness in that. BQS rests on a label developers apply to their own pull requests, so the cheapest way to improve it is to tag fewer PRs as fixes. We cross-check every engineer-applied bug-fix tag with LLM validation against the diff and the linked ticket, which raises the cost of drift without eliminating it. The rest of the defence is in how we read the score: as a trend. The direction over a quarter is the signal. Any single week is noise. A dip would tell us something changed before our users find out for us.
A single score has one weakness: it tells you that something is wrong, not where. Two layers of leading indicators sit underneath it.
L0 is BQS itself, the outcome. When it moves, that is a flag, not a diagnosis.
L1 is where we look first: Defective Change Rate broken down, unit test coverage per service rather than a team-wide average, and staging uptime measured as the P0 flow pass rate. These sit close enough to the outcome to point at a culprit quickly.
L2 is the structural layer that explains why an L1 metric moved. A Test Effectiveness Score based on tests actually run per PR, not tests that exist. Configuration failure rate across queues, payloads and environment config. And the days code spends on a real environment before merge.
The discipline is easy to state and hard to keep. L0 tells you something is wrong; L1 and L2 tell you where. We do not skip from a bad score to a hunch.
Generate it
If test cases are going to act as guardrails for agents, they cannot live somewhere else where no machine can read and no pull request can be enforced. That rules out the spreadsheet and the test case management tool both, however well maintained either one is.
So we moved ours into the repository as structured YAML or Markdown – a scenario, its tags, its steps, its expected result. That single move buys four properties at once:
- Version controlled. Test cases evolve with the code, in the same pull request.
- Agent readable. AI can read, generate and reason over them.
- Enforced at merge. CI flags any PR that changes code without touching the corresponding tests.
- A living specification. The test cases become the definition of correct behaviour, not an afterthought maintained by whoever remembers.
With test cases in that form, AI becomes useful for them. When our developers put LLMs to work generating test cases, a simple agentic prompt reached roughly 80% feature coverage, which makes this an easy entry point for anyone starting out. Richer prompting pushed both coverage and tester confidence higher, because the model surfaced cases the humans had not thought of, on simple and complex features alike. Drafting a full suite stopped being a scheduling constraint.
The gap is more interesting than the win. AI reliably covers what is derivable from the code and the spec: happy paths and acceptance criteria flows, feature flag ON and OFF cases, null and empty data, negative cases pulled from the spec. It reliably misses four things, past production failures, cross-feature interaction bugs, low-end device and OS version edges, and the real user paths that only analytics know about.
Those four have something in common. Each is knowledge the team learned the hard way and never wrote down anywhere a machine could reach it. So we work to a simple rule: AI complements manual testing, it does not replace it. The system covers most of the ground quickly and the human closes the gap where judgement and memory are the only tools that work.
Give the loop a memory
The first of those four misses is worth attacking directly, because unlike the others it is a solvable engineering problem rather than a limit of the model.
“Every production bug that was not caught in testing is a free test case for the future.”
The work is making that knowledge queryable instead of tribal. Today we keep a structured learning file per feature area, checked into the repository. After each incident, the failure details are logged, and because the file sits alongside the code under version control, the coding agent reads it whenever a pull request opens against that area. The agent generating tests for your PR knows how the feature has failed before.
That manual step is a starting point, not the end state. Next, the agent queries when a pull request opens, pulls defects tagged to that feature area over a recent window, and folds them into test generation with no file to maintain. A forward looking approach also is to capture crashes and errors in real-time as a structured failure history thereby queryable by feature, OS and device.
Manual effort goes down at each step and compounding knowledge goes up. The test suite gets smarter every time production teaches us something, instead of every senior engineer having to remember it personally.
Worth asking about your own team: how much of your hardest-won quality knowledge lives in the heads of two or three senior engineers? Undocumented experience is a dependency you cannot version, cannot test, and cannot replace on short notice.
Enforce it
The third move is where quality stops being a report and becomes a guardrail. Every pull request is a risk event and we treat it like one. Three things happen automatically on each of ours.
An AI Impact Report is posted as a comment: risk level, modules affected, feature flags touched, the specific test cases to re-run, and the untested scenario the change creates. That last field is the one that changes behaviour.
It changed behaviour the first time it fired on a real PR. The report flagged a feature flag combination that no reviewer had thought to ask about and marked it not covered. The developer wrote the missing test cases, covered the flows the suite had never exercised, re-ran the affected cases, and merged. The flags turned out to behave correctly under that combination, nothing was broken. What changed was that the merge happened with evidence rather than with hope, and the missing cases are now in the repository for every PR that touches that area from here on.
A Control Flow Risk check flags code running outside feature flag gates, because unguarded code reaches every user the moment it merges rather than the cohort you meant to target.
And a Blast Radius Guard requires guardian approval through CODEOWNERS for modules that a large number of other modules depend on. Widely-depended-on code does not get changed casually.
None of that is a human remembering to be careful. It is the loop enforcing care by default. It is also the clearest illustration of what moving quality forward means in practice: the report does not decide anything. It puts the evidence in front of the person deciding, while the decision is still live.
So what do Engineers & QA own now?
Whenever I present this, the sharpest question comes from engineers. If agents write most of the code and test cases, what do engineers & QA own?
Notice what the checkpoint model did to the role. It made Engineers and QA the last line of defence, which sounds like a compliment and is the worst position on the field: you inherit decisions made weeks earlier by people who have moved on, and you are measured on catching problems you had no power to prevent.
The loop model moves the role earlier and upward. From authoring the 80% to owning the remainder AI cannot see. From writing test cases to deciding what the guardrails are. From finding bugs to encoding failures so a whole class of them cannot recur.
The scarce skill was never writing a test case; an agent drafts those in minutes. It is knowing which scenario matters, which failure class will come back, and where a human must sign off before something merges. The part a machine can do is being handed to the machine, and what remains is the part that was always the expertise.
What could still go wrong
Two failure modes worth naming:
The first is coverage theatre in a new format. A YAML file in a repository satisfies a CI gate as easily as a spreadsheet satisfies an audit, if what is inside it asserts nothing meaningful and an agent will happily generate a great many test cases that execute code without testing behaviour. Mutation testing runs today and is what makes that visible, which is the whole reason it is in the stack. The residual risk is coverage of the checker itself: test generation is spreading faster across feature areas than mutation testing is, and until those two lines converge there are places where an assertion-free suite would pass unnoticed.
The second is reading the number wrong. BQS exists to start a conversation and locate a problem, not to grade anyone. Read as a trend across a quarter it is diagnostic. Read as a weekly scoreboard it produces a lot of motion and very little signal, and the tagging discipline the whole thing rests on quietly stops being worth anyone's care.
Key takeaways
- Measure it first. Start with a bug-fix PR rate as your BQS proxy. Set a target, track it weekly, read it quarterly, and let the number force the conversation that vibes never will. You do not need tooling approval to begin — you need a query against your merged pull requests.
- Put test cases in the repo. Version-controlled YAML or Markdown is readable by humans and agents alike, and nothing else on this list is possible until your test cases are somewhere the system can see.
- Make test cases for the guardrails. In an agentic SDLC a test case constrains what an agent is allowed to change. That, more than any review process or velocity ceiling, is how quality scales with velocity instead of being traded against it.
AI did not create our quality problem. It amplified testing gaps that were already the leading cause of our incidents, and made them impossible to keep ignoring. Passing the tests is the system’s job. Knowing what the tests are worth is ours.
A question to take back to your team: where do your test cases live right now, and can an agent read them? If the honest answer is a spreadsheet or a management tool, then whatever else you do next, your test cases are constraining nothing and that is the first thing to fix.
