How to build an AI code review agent: from the creators of Greptile

Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): Greptile (3 vs 0). Check them against the audio before quoting.

00:00:00 — Three Pillars of Production AI Code Review

Multiple contexts of the codebase, and by reviewing and testing these codebases, we reveal bugs, logical errors, code quality concerns, primarily from AI-generated code. Now, to these types of scale, many thousands of companies use cloud hard every day. Five one percent of all Flutter customers get help through us, and a variety of many different types of code bases, whether it's the CUDA code base at NVIDIA or the Waller code base at Google, and many others.

And every now and then we meet a company where we are actually not the right code review or code testing product for them. Their needs are so specialized and so esoteric, it's actually more beneficial for them to build their own, and when that happens, we tend to help them for free. There's two reasons for this. One, we have a lot of specialized knowledge about how to do this really, really well. And second, we get to learn a lot about how we can serve customers like those individual. And in this presentation, I want to tell you what we thought about it: how to build a really great AI coding agent.

Now, there's kind of three things that people like about WebRap, and I'm going to talk through how to implement those yourself. The first one is code-based context. When customers use Greptile on their code classes, they find that it really understands the code. As a matter of fact, before we were a co-validation company, we were a co-discovery company. We were teaching language models from 2023 how to understand very large codebases, and as a result of that, we got very good at indexing code.

The second thing that people like about Zapod is that it runs your code in the sense it spins up your stack in a fully loaded, dependent to your sandbox, and then clicks around and tests things using computer users or in-browser testing API and so on.

And third thing is we're actually very efficient allocators of advice. We pick the right model, the right quantity of inference budget for each subtask in the review and validation process. As a result of that, we're able to get a lot more performance per dollar than an internal tool code or any competitor.

So there's kind of two reasons that you would want this presentation. The first one is you want to build your own review agent. And second reason, which is a bit more subtle, is if you're trying to pick a review or a testing agent for your company, and you want to understand what makes a good one, and what makes it great one. All right. So that's right. Now, black stage.

00:02:19 — Building a Naive Sandbox Reviewer and Five Failure Modes

You are part of the platform to your company, or you're just the local AI psychotic member of your team, and you want to build the review validator of your company. I'll be writing lots of code. These are by putting. We have, of course, engineers live coding. Some of them a little too close to the sun. You have marketing people writing code, and the new problem is the deep validation problem, and so that's what you want us to do.

The first thing that you should do is you should get yourself a sandbox. There's many companies here that will sell you a sandbox. You can also just use FireCrapper or something like that, and you'll want to put an agent in that sandbox. The agents can be correct, certified, coherent. Personally, I recommend something modern, independent for reasons that we'll get into. So you could use something like OpenCord or Py as your base, which are both open source. And you will need to then connect your GitHub or GitLab at the Harvard book: send it the contents of a polar class. Too many polar classes opened. The agents can have some type of prompt that says, "Hey, review this and send back some feedback." And then you can wire that back using GitHub's keycards. Very simple. In fact, many people think that's actually all it takes to run a revolution.

There's a few things that are going to go wrong when you start using this. But what will happen is you will in fact have a podcast go live. The content will go to an agent about five minutes later. We will get some feedback on your podcast. It usually doesn't go wrong in the time they're using the tools. The first one, you're going to miss a lot of bugs. It turns out that actually bugs are intrinsically hard. In fact, I saw a tweet not very long ago that said that bugs are fractal demons, so you will never run out of them, of bugs in a particular piece of code. Now the first thing that's going to go wrong is you're going to miss bugs. Of course, that's the one thing that your system needs to do is to prevent at least very catastrophic images and things.

Second thing that's gonna happen is it's in a mixed context, and it's going to misunderstand parts of how your codebase works. Maybe won't understand where you're going with your codebase. It won't understand how a specific part of it works. It maybe will not understand how one of the libraries works. It's possible you're one of those companies that never recovered from the microservices talked about so many times, and so you just have thousands and thousands of repos, and you have to have context to all of them to be able to operate. So that's the second thing that's kind of important.

The third thing is it'll be very expensive. I think people are very surprised by this. So generally, from people who've seen how to do this, you'll find that they're spending between three and up to twenty-five dollars per podcast on reviewing it using voice. These tokens can add up really, really fast. Very quickly, very, very sensitive.

The fourth problem you'll find is nitpicky comments. That's not even more than you might think. The big comments are technically correct, but overly banal. There's two things that go wrong when you're missing humans. If you are a as a human reading those comments, it's very annoying. But I avoid it by both. You just see lots and lots of comments and it's kind of annoying for you to just read through them, figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems.

The second thing that will go wrong is people wrap your cookies. Typically, the way the review engines work is they leave comments in GitHub. The coding engine picks them up and resolves them. Including in your discerning, you're going to look at review comments to address all of them, and you'll end up with extremely painfully defensive code, as we found in Kubernetes, are particularly astute to write. The next thing that'll go wrong is that every time you run a review agent on the same product class, you will somehow find new bugs, in the sense that it will keep catching new things in the podcast, very annoyingly. Like I said, bugs will appear to be practically infinite.

00:05:41 — Prompt Optimization and Historical Incident Eval Datasets

So, the first thing that you want to do, the lowest hanging fruit, is you want to optimize the prompt. We started with a very simple prompt, something like review this PR and then output the comments in GitHub's API schema format. Generally, there's two themes in that: Look for specific bot types, and second, skip the nitpicks, and that's what you want to orient your prompts around. You'll want to ask for specific bots, so you can kind of do some internal data science on existing errors and incidents that have happened in your codebase, and figure out what types of bugs have a high propensity to produce, create a catalog of them, and then have your prompts look for those specific details. The second is you need to express your talent or intelligence, and not the other. Endings could be anything from asking why there isn't three different events on a particular thing, or why error handling is done in this slightly shocking way.

Now, the thing that you'll run into when you change these prompts is, when you change a small thing in the prompt, you'll have a really big impact on a number of comments that your agents write in the prompts. This is quite scary because you don't really know if it's missing a bug or not. You don't know the kind of actual — the validation against really any reviewer can tell you about the presence of bugs, but unfortunately can't tell you about the absence of bugs. That's sort of an indication of that.

And what makes this particularly difficult? Robots are extraordinarily rare and getting rarer by the day. Not 24-hour, if eyes. Only about one out of every thousand keycards have a genuinely catastrophic language. Are you writing a thousand lines of code to crash off the balcony? And so, because these bugs are very rare, you might release a version of your image that you think is very good, but it secretly misses a catastrophic modeling flaw, and you wouldn't know for a long time because you don't produce those bots very often.

And that brings us to the second improvement. You will want a dataset of historical ERs. This is relatively easy if you're one of those companies that very nicely catalogs every time something goes wrong in your codebase. We have that, and we have a list of polarized that causes outages and incidents in the past. We want to catalog them, and then essentially have any version of your review agent run through those and see what frequency we have so far. Now, this will allow you to tune the prompts a little bit more confidently.

But at the same time, the interesting thing about bugs is, in a changing world, the topology and ontology of bugs changes over time, and so we need to have a real-time updating. It is tricky because unless you have millions and millions of keycards, it's actually hard to have reasonably high coverage.

Times, it's true that currently many of our jobs will be repeats of prior mistakes, but increasingly as code agents get good, they're going to start making novel mistakes. So, fail in new ways that we're not used to, and so this is where we'll start pivoting. Now, quick side note: This is kind of what we do at Greptile, except we actually have a lot of code from a lot of companies, and we can test across very large corpus of codebases, and therefore, improve our agent's very confidence. It's one of the reasons why we're able to eke out little bits of performance and margins in ways that internal human agents can't, but you can get decently far, especially for your particular company, doing this with your internal data.

00:08:42 — Inverting Author and Reviewer Models to Catch Blind Spots

The next thing we want to do is longitudinal. So for in case anyone was at my top tier in San Francisco, a engineer, I talked a lot about the ways in which coding is fair. And we saw an interesting pattern through that research, which is that the types of bugs produced by cloud models or coding models are actually quite different from each other, and the difference in a number of ways.

As an example, code models are something like three times more likely to produce one-off errors than GPT models. UT models, on the other hand, are significantly more likely to produce off-passes than policies. And these are all just artifacts of the post-training, as far as I understand. Now, to prevent this, you'll find that if you're using the same model to review the code written by the same model, you actually have many of the reviews are about talk because they feel the same ways during review that they do during co-generation. And so you want to invert the model. If you use hard code to write the code, you want to use the GPT model to review it. And so.

Now, doing this is somewhat straightforward for the AI and quite tricky for the last one. So, if you're using coding agents, you'll notice that leave some type of fingerprints to tell you what coding agents are used for a particular product. For instance, Kudex will name branches in a Kudex slash branch name format, or Claude is going to write a co-author by Claude to be able to change the decision. Everyone, even the in final training, when you haven't manually created branching, or there's some other reason that the different plugins have, it's significantly harder to figure out. There's not always easy.

You have to have some depth of your shape to figure out where the symmetry is going. Often you might have an internal catalog where people report what codecs they're using these days. You have some type of telemetry that's coming out of your that machines and your engineers, creators. We found that there's a ten percent higher cash rate if you look long. This might sound very small, but you have to know that ten percent in terms of numbers is actually quite a lot — means that there's potential in our science that something will go wrong if you're not using modern versions, which can be quite catastrophic. And so we highly recommend that you use some type of model version in your processing.

00:10:45 — Replacing Expensive Grep with a Living Company Wiki

The third improvement that you can make is context. Now, this is something that I care a lot. Before you are doing foundation, you are doing context. We were trying to figure out better ways to do rag over very large codebases, and even today, people consider rag to effectively solve all of today's context. I actually think that's not true for number three. One, rep is extremely imprecise, and second, it's very expensive.

To give you some sense of scale, single-digit percentage points of GDP growth last year were in France and very large percentage points in coding agents. And you actually look at a coding agent trace, the vast majority of tokens used by coding agents are all the same — probably cash right for the back office. That means a non-trivial percentage of GDP growth is graph how we capture. That's very bad. And this business graph is very precise. It's reading files in very large quantities. And I don't know. There's perhaps an adverse incentive for companies paid by the token to figure out ways to burn their tokens. But even so, you'll find that you'll need to use slightly more surgical methods by figuring out where you should be meeting.

And the way we do this is we produce an internal knowledge base where we essentially check across the entire codebase, all the different parts cataloged and an index saying, okay, well this is authentication. Here's how it works. Here are the flows. Here's a diagram of the flows. Here's different problems. One example of where context matters is, for instance, you're trying to cash an AlphaPass. It's actually very difficult to figure out if my task has occurred because you need to have a pretty concrete understanding of whether you're in an unsafe environment or you're in a safe environment. And that's very tricky. And so you'll find that you will need some type of a sort of obvious context to do this — yeah.

I've been very centered around the concept of bots, but code can be very bad. For one, it can be technicality, but second, it can just not fulfill the expressed intent of the offer. Recently, I saw a post that said that people are interested in using Slack as their IDE. Now, I think it's a little funny, but at the same time, there's not that much express intent in a Slack message. You're only saying maybe sentences or two, if even, asking an agent to go and pick something, which means there's a lot of ambiguity in their instruction. As a result of this, you might find that the decoder is either misinterpreted your instructions or just those two videos, and you don't know exactly what's going on. And so your correct code can actually be quite long.

Now, one of the ways in which we find this is by giving agents increasingly more contexts, and we find that the two types of contexts that matter most are dogs and pigs, and of course the hookers. To be a query indexer and all of this, you can get remarkably far if you get access to your geo engineer, and then your Confluence or Notion or whatever it is that you serve us. You'll find that there's a lot of tribal knowledge that is unconsciously helping you produce better code. And when you have agents do the same thing, it feels like they just don't know what's happening, because you over you underestimate how much depth of knowledge you guys all have.

Now, the way that Greptile approaches this problem is we actually disintermediate all of these context sources with a living wiki. This wiki is essentially a sort of world model of how your company works. It's everything that knows about your codebase, everything that knows about your application through production telemetry through logs, everything you know about your roadmap, from your documentation and your linear tickets, and so forth. And using this big key to supply context, both making the agent more context-aware and making its core search engine more efficient — students will never read the file that it needs to read.

00:14:04 — Autonomous End-to-End QA in Dependency-Rich Sandboxes

Last thing that you should do — the next thing we should do is autonomous testing. One of the things that we think about a lot is that, for instance, UI changes, you might be able to read through a bunch of change files for React code and concluded they're unimportant. But in fact, we remember it might be, for instance, here in an iPhone. Now, this is one of the more technically difficult problems to solve. The view processing pipeline is we essentially build these dependency-rich sandboxes, which are completely configurable virtual machines.

And we have customers. We work with them to essentially spin off their stack in these sandboxes, and then, artificial intelligence is a computer used to have them understand. If you haven't paid attention, computers have gotten very good, and say far better in these new generation of GPT models and open models than it was in previous. For the first time, I think end-to-end QA is nearly solved. The thing that you do though is you document all the flows of the application, and second, you need to be able to have a sufficiently simple build script so that the agent can spin up your stack in your environments. We have a team of four different engineers that help the company do this. We found it's one of the trickier parts, unless of course you have very good dev boxes already, in which case congratulations on a very good start.

So this is the diagram I'm going to show you. But yes, you'll need a data box. You'll need to do our ground truth dependencies where you're running the app, and you'll need some type of flow map that says this is the off code and then this is the sign. And using this, you can now test your content and use vision models to say, oh, this button technically appears, but it's actually hidden on iPhones. It's not in the first fold on the iPhone Duo.

00:15:45 — Routing Subtasks Across Frontier and Lightweight Models

Now, if you all this, you'll find that the costs are still quite high. It can cost you anywhere from thirty bucks to fifteen to twenty to do a single conversation. And the thing we want to solve right now is to use cheaper models for some things. Now, to me, frontier models are still generally speaking capable of those models. This is good room from time, even though the room back is tiny. So, our speaker today is about twenty percent frontier models for the most critical tasks. In the long tail of top industries, it's good for a file being so long, views of various long. And so, if you do this, you can actually reduce the cost of running your review agents quite drastically.

So this is sort of a illustrative example of a video, which is we use a small model for fine context, we'll use a very lightweight computer use model for testing user flows, maybe the frontier models for things like analyzing bugs, the kind of review process, and for submission. In practice, we have actually twenty different subtasks, and each of them has a very specific sense model to be used, and our small research team is constantly iterating exactly what models we need for the task. You can probably get seventy or eighty on the way there using something of this kind, where you just swap our physical assets with virtual ones. Now, this is another example of it being helpful to integrate our data.

If you don't have a good data set, you might be able to trigger happily replace a lot of different subagents with smaller, faster models, and you won't know that this international find a bit worse catching bugs. You only find out when it misses something, then causes an error. And you also find that you can kind of combat this problem of overactive file reads using a cheap model that essentially looks through all of your context pieces and figures out which ones actually should be included. In theory, you should be able to use a decision model for this, something like Jack's decision API, to do effective reranking on all the different things and deciding what should be, shouldn't be loaded in the front end model, which is in some regions, most expensive, most scarce resources.

If you do all these things, you might be able to actually get quite far in building your own computer vision. And if you do, please email me. My email is dashugrapal.com. I'd love to help you, and if you choose not to build it, then you can also just use our product, free, and in person, but thanks much.