One team's path to trusting AI generated code

The recording starts after the talk had begun; the speaker's introduction is not in it. A 3½-minute recording from just before (2026-11-10_0929_Ankit-Jain) is indexed separately as needs-review.

Transcribed locally with Qwen3-ASR 1.7B (timings per ~5 s segment). The Voice Memos transcript of the same recording is kept beside it as transcript.txt.

00:00:02 — Opening

So here we went for X. So we are a very small team. We are nine engineers. A year ago, we were merging about 100 PRs per week. And now we are doing about 400.

So one of the main things which has changed, which everyone already knows, everywhere we moved from human. Written code with AI assistants to now predominantly AI-written code with human assistants.

00:00:29 — What we tried first

So let's start with the battle. So what did we try first? This is stuff like I'm guessing most of you are already using an AI reviewer. We started kind of like you know testing how the AI reviewers work. You know, you've seen lots of people in the comments come and bought.

Some from humans. The reality is, a lot of comments from bots we got were sort of like nice to have. But it did find bugs. But whenever you find bugs, you feel excited. It's like okay, this is actually working.

An interesting observation we had was once we started using AI reviewer. Most of the comments that humans are providing are actually not about bugs or correctness. It's about decisions. Then our second one was okay. Let's just write specs.

The spectrum of development. This was back in January. You know we write the specs, and then most of the code gets generated by AI. We just review the specs, inspect the disks. It kind of like reduces, creates one better proxy.

The reality was, when you're writing specs, there's a lot of assumptions you make. Eventually, when things get implemented, a lot of this gets stale. But the bigger issue is now: if you stop reading the code, there are trends that reflect. And because of this, we ended up going back to reviews. The intent was written down, but nothing really checked whether we're actually still consistent with the spec.

And then our third attempt. This was probably the most. Interesting and useful one. We actually created our own review scale. So instead of us doing review after creating a pull request.

We essentially have a skill, which is like everyone go review your own code. Before even somebody does a pull request, and yet it books and scours it kind of like happening beforehand. So we found better findings than the bot now because it's actually very contextual. Added more context vectors to our code based into the skill, but the idea was we were still reading the text. But it was after the fact, which means obviously there were lesser bugs, there were lesser concerns.

We can move a little bit faster, but we still are, you know, having to leave the diff. And the other thing was any findings that the agent found. In this review skill, something important: ask for decisions. We can't play; we never even make do it. So it was a better reviewer, but we were still reading this.

So, but this kind of like provided us with a little bit of like, you know, sense of where we can go with this.

00:02:56 — Code review is not about the code

But before we get into it, let's talk about why do we even do poetry today. This is a study done like back in twenty thirteen by Microsoft. Forty-four percent of people asked why do they do code reviews. They said to find bugs. But only fourteen percent of analyzed comments actually were bugs.

So this 30% gap is the real gap that we're going to talk about today. Code review is not about the code. Code review is not about the code. It's not about finding bugs. It's for us to build mental models.

Unless you're whiteboarding a single project by yourself, please don't care. In that case, you don't even need a code review. But if you're working in teams, You want to build better models. You need to understand how things work. Code review is about knowledge sharing.

It is about knowledge sharing. It's not about finding bugs. So human code reviews also find real issues, so they are still valuable. However, we are really, really bad at it. If you think you're a really good reviewer, please go back and check how many times an incident happened because you were silent.

And that's because beyond four hundred lines. The quality of review drops significantly. This is not me saying this is a study done back in two thousand six, long before I became a thing. And the AI now, you're generating every PR, a million PRs, over six hundred lines. So what does that mean for our AI project?

You know, even though we are reviewing code today, but are we really reviewing the code today? And the reason is Diff was always a bad place to review. You have one reviewer reading from memory one time. That only tells you what was changed, but never why. So what a review actually contains.

This is sort of like another example. Where we provided, I provided summaries. This is I made of them here, our own, and there were like half of them from a bot. As you can see, what we started doing was like let's try to categorize what kind of review comments we make. Not all reviews are not all review comments are the same.

There are bugs. Some of them are annoying. Some of them are nice to have. But there are also rules, kind of like okay, this is what our code style standards are. How do we actually like you know enforce those standards?

And there's also decisions, which is like, how do we really understand the concepts, or we actually doing architectural conversations. So in 40 threads, eight were black threads. Only one was real black. Three were rules that we have also repeated in the past, and two were actual decisions. So again. There are two kind of reviews.

Primarily, one is about. The quality, the code quality, which is like, is it correct? Is it following our standards? And the second is decision, which is an opinion of a lawyer. If you automate the entire review, you miss the decision part entirely.

00:05:48 — Six steps

Which brings us to our framework. Okay, so I'll share a little bit about you know how we think about reviews. Take it or leave it. Everything I fear that you can actually do yourself. You don't need to use the editor.

So, it's a lot. We have six steps. I was like, okay, we're killing one thing and we're creating six. But hear me out. What we're trying to do here is like break it down.

Into different parts of what a value would be provide. And build it back back up again. So we will start with argue. Argue is now you have multiple coding agents arguing with each other. We have capture.

By the way, all of these things will go in detail, so don't need to take too many notes. We have capture, which is about like actually being able to capture what's going on in the system. Clarify is about repeating, capturing everything that can be automatically checked, like insurance or text marker. Prove is they actually doing everything that the. Change is supposed to do is working.

And debate is where humans participate most. Which is like really seeing. Are we making the right decisions? Are we doing the right thing? And own is the ownership.

You know the models cannot own anything. So unfortunately, humans for that.

00:07:08 — Argue

Still responsible on the line too when things go wrong. So let's start the argument. Let the agents argue before here exists. This is kind of like similar to some of the things I mentioned from my attempt three, where we create a self-revision skill. Now we argue with our adversities.

So essentially, we create a. Which works independently. You can also use a third-party model, and you don't have to use the same model. Presently, what we're doing is creating this argument where a coding agent and a retrieval agent are arguing with each other. With different goals, and this goals part is important.

This is why having two different agents is useful. Because now the goal is for putting it into full exam versus the review agent to find defects. And this is the most important thing. Everything that they disagree about, we should write it down. Today. So in this case, here's what we disagreed about, and it's sort of like we started writing it down.

In this case, you can see. We are not just writing down the decisions, but we're also writing down what was decided. So, for instance, if there are things where we're saying this is an accepted risk, we won't fix it. And it's okay because agents will always go and try to fix everything. Sometimes you don't need to fix it because you don't need that complexity.

But there are also certain things where we say, "Oh, we'll actually follow up in a separate fix." All of this information today gets lost in your sessions, and what I'm proposing is to be able to read these things.

00:08:38 — Capture

Which brings us to capture. We should capture the why and the how. A very common answer I expect you all to try out. This was our attempt number two. The problem with specs is it requires time.

I'm not saying specs are wrong. But it requires a certain discipline, as I mentioned. But our failure with space. Was that we never bent and corrected when it was broken. So what we are suggesting is, if your specs are broken, you have to fix it.

If you don't, Then you're using a platform. This is something we you know moved away from over twenty years ago. So that's why it's important to have a discipline of specs. In our case, we are limited. So, what can replace that?

We talk about intent. Intent is why are we making whatever change. And the acceptance criteria, which is how you should behave. Some of you might be familiar with behavioral development, and it sounds like these are the concepts taken from there. It's nothing new.

This was again 20 years old concepts. The beauty is it works for a single line change and it works for a customized feature. You have the intent for everything. You have the acceptance behavior for everything. And these already live in your sessions.

You don't need to go and write it yourself. Intent is what you start, but maybe. Maybe it's something you started working on. Maybe it's a spec file you provide. And acceptance criteria every time you your agents has to make a decision.

Every time we kind of describe the behavior, that's the acceptance criterion. Today we told you about.

00:10:17 — Codify

And the presentation. Okay, codified. I'm gonna go a little bit faster. I've been going a little long time. Turn repeated comments into blocks. It hasn't mentioned that there are two different kinds of comments.

One is about correctness. One is about behavior. So what we did was we ended up mining a bunch of our previous comments. And then started doing reduce, which is essentially identifying the comments which are more useful. Extracting, classifying them, how many of them can be turned into rules.

And eventually, distilling it, which is merging similar rules to create these guidelines. Now these become your invariants. Essentially, what came out of those is. Out of 200 candidates, we got 64 rules automatically created. This was only one scan.

Imagine if you start doing this on a regular basis. So these are some of the examples of the rules. You can see many of them are rules you can imagine. Can be linters, but some things can get more complex. The beauty of all of this is.

Created one way to define the rules in English, and you can maintain them over time, even if the approach changes. So what goes in an invariant? It's a rule which describes what it's supposed to check. A scope where it's like, when does this rule apply? Verified by it's like how do you actually prove that this is working?

Provenance, which is just tracking where this came from, so if you have to correct, you kind of label back to the source. And owner, and this is the most important one. If you don't have an owner for a rule, this rule will become stale. So the victory always defines ownership of these things. And the beauty of this is, it gets cheaper over time.

You define it in one time. What an agent can do now is take this rule and write it into a script. Which is hand detected and checked in deterministically. You run this in chat mode, where you're kind of like validating it with the model and validating it with. Become an expert in something.

So what we're doing here is essentially use AI to build more data-intensive systems to generate. So here's an example of one of the recent papers. As you can see, this is an ESG prep. If you focus on the red, it's kind of like automatically detects. That every request probably makes has a timeout.

So out of the sixty-four rules we had, we were able to convert nineteen of them to deterministic. And now it runs predictably. It's easy to verify. So every. Commonly, your team repeats to the governor. Every commonly, your team repeats to the governor.

00:13:02 — Verify

Okay, how do we prove that the doctor works? So this is kind of like where some of our system comes in. We kind of like take the acceptance criteria, create a plan. Spin up a server, and then drive the agent to capture the evidence. What ends up happening is, be based on the type of rule, based on the type of behavior, you can capture different errors.

All you're doing here is sort of like somewhat similar to how you used to do QA. But now you're giving the plan to the agent to plan themselves, and our goal now is to make sure that the evidence looks correct. So in this particular example, you can see kind of forest as a rule. There is like. Screen shot captures requests, captures responses.

This is sort of like how an agent can convert a rule into automated. Steps, and depending on what kind of criteria there is, you can also define different types of behavioral verification. You can take screenshots. You can request response. You can check request response network activity, ASD graphs, as I mentioned.

As well as, test runs and races.

00:14:09 — Debate

Okay, debate. This is where we debate. We as humans, you know, this is something which I want. One thing you take away from here is that humans are always important in this room. Do not completely remove that. If you're working cheats, you will lose your knowledge, and this is where we talk about how agents interact with the system.

When you build, your agent is talking to you. Most cases, if you're running dark factories, this might not be true. But most of if you're working on your cross sessions. We're working with the agent. We're providing feedback.

We're making corrections. These are. Places where you're making these micro decisions. But agents argue, which I already mentioned, if you're running a coding agent and a review agent, this is sort of like. The arguments are happening, and then when a verification fails, which is like. All these rules that you ever written down, all these criteria that you wrote down.

If it fills, the agent is making. Certain behavioral changes, whether it's data requirements or data flow change, in both cases they are positive. And these are the things that are worth focusing on. So here's an example, like one particular file where we have a bunch of these decisions. As you can see, I only showed some of these examples.

Some things are decided. Some things are fixed. But some things may require human judgment as well. We kind of like flag that, so it becomes something you need to. So we have to make these decisions.

We can skim what is settled. Settle, what the agent could not, we are behind the agent. And you reopen any call you would make differently. So there's a decision that something made, like an agent made or a human made. Now you can actually argue.

The main thing you're doing here is, as a reviewer, you're having a discussion with the author. And this is where the collaboration becomes important. This is where you're learning about how systems work. This is where they're actually arguing these decisions. And not just with the bots, but also with humans who are actually offering.

So decisions output of performance. You can analyze weekly. Now you have all of these decisions. You can say, okay, for this particular feature, this particular. Per change, give me all the decisions that were made for this week.

Maybe there were some things that you might. See as a consumer, and then go back and visit why these decisions are made. You give it to your agent. Now, agent knows these past decisions. This is sort of like the tribal knowledge we talk about.

Now you'll easily have access to these micro decisions, sort of like context is continuously building on the fly. And follow-ups become easier to track out of these. Decision that you make as a follow-up. That we'll do it in a separate pull request. Now can be easily sources tracked, and finally, this is where you can then be making better and better rules.

Which now essentially become the standards of your approach.

00:17:18 — Own

Ownership, you know, models unfortunately still today cannot own the code or mistakes they make. So, the responsibility is good as. Authors used to own the code. It's like okay, do you agree? We are not owning anything that happens.

And then we talked about the reviewers. They're sort of like the gating. Factors. They're the code owners. They're the ones who effectively steward the code if somebody else outside the team is making changes. But people about ownership from a team.

Now we can actually own the rules. That verify whether the code works. So instead of thinking about owning the code. Own the guardrails. Own the rules.

Own these frameworks. So as humans, now our responsibility is not the line by line review. You can't keep up if you are still reading the code. I'm sure it still is working, but like imagine the quality degradation that is happening. More reviews you have to deal with every day.

We have to now own the rules that take place. Instead of holding the code, holding what is verifying the code. And maintain your understanding of the system. If we lose this understanding of the system, we don't know what we're building. We don't know how things work.

00:18:42 — Results

This is where we start. Okay, so a quick review of a year later. And the numbers kind of like where we are. So as I already mentioned, kind of like we have four extender changes. Our median time to merge has gone down drastically.

It's. 3.4 hours later, used to be 2.1 days. And also, personally, the number of threads in the PRs have decreased. Now we are kind of like not finding as many correction mistakes. Issues of the PRs. What we are doing is having these discussions in the in the decision meeting.

So our surface area has changed. We're still discussing many of these things. But they're having their different truths of if it's not online live code, and the number of rewards that. So that means our quality has now improved. Even though they do this.

So don't need to read all of this. I kind of like already shared. This is an example of a pull request. Used to be open five days. It's like two and a half hours.

I think the most important thing is now we also have all of these crews. We capture all of this screen charge. We have these request citations, which means now we have better auditability than we had before. We can actually go to our auditor and show. Like all of those things were verified, and we have enough evidence.

That this works, and one of the most important things is we have changed our surface. From being the death to reviewing the to reviewing the syntax. So where our time goes now? We spend a lot of time refining these rules, these the data of the decisions. And we figure try to learn more about how do we improve our verification systems.

Everything that we manually do testing. QA anything today? We always go back every week and try to figure out how to do that. And this is not something that you can do immediately. This is something you have to build over time.

First ninety days are the hardest. This one we're just doing double work. We're confirming the code. We are reviewing the rules. We're making sure everything is working.

The second is kind of like James talks. See, okay, like you know, the the the rules are working. They're kind of like making less and less changes. By the end of month three, you will have seen our way. The intent evidence is actually more valuable and more useful.

So. We start tomorrow.

00:21:09 — Start tomorrow

Mine your last 1,000 review comments. Take the top 20 out of those. And write those checks. You don't need to do a lot of this manually. You can just ask an agent to do all of it.

I'm sure Google and others do this. The main thing is just pull that in your CI. Let these rules not forever change. So please stop reading the death. Read these instead.

The intent is why we, why the change exists. Acceptance criteria is how. You know it actually how it's supposed to behave. Invariants are the guardrails, are the. How so? Difference between an acceptance criteria invariants are invariants are core quality of an acceptance criteria that behavior change.

And decisions is one of the most important things, which is your opinions, the trade-offs. All the things that you're doing, and evidence is proof that you're right. Here's a quick look at our verified experience, where we captured intent and evidence of the decisions. Let's go check out. And we are all in this together.

We can outsource our thinking to find our sources of stuff.