AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix
AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix
Summary
Rajat Shah, a staff software engineer in Netflix’s AI Platform Organization, presents a playbook for using AI agents to close the gap between how fast coding agents now ship code and how efficiently that code actually runs. His starting premise: AI-assisted coding is tuned to produce code fast, not fast-running code, because agents don’t know an organization’s internal platform conventions — so as adoption accelerates, so does wasted compute. The traditional fix, manual profiling, is slow and tedious enough that teams only do it reactively, usually at 2 a.m. when a service is already in trouble.
Shah’s team tested whether an LLM agent could replace that manual profiling loop, betting on two assumptions: profiler output is structurally similar across languages and runtimes, and coding agents already know common performance anti-patterns (O(n²) loops, loop invariants, repeated allocations) from their training data. The results validated both. Fed only a call stack — no codebase access — an agent correctly diagnosed a quadratic-time tensor merge method eating 8.8% of CPU on a live Netflix service, then checked out the exact production commit, traced the method, and produced a validated fix in under five minutes. The same anti-pattern, once cataloged, turned up independently in seven other services in a single cross-repo search session.
The back half of the talk is about turning that one-off win into a repeatable system: a plain-markdown, git-based catalog of patterns and anti-patterns that gives a stateless LLM persistent memory, paired with mandatory guardrails — unit/integration tests, then canary deployment against real production traffic — before any fix reaches a human for code-review approval. Shah frames the mental model as “profiler gives the estimate, canary gives ground truth, engineer makes the eventual decision,” and closes with a three-level autonomy roadmap: start with LLM-assisted identification only, move to a fixed, fully-tooled workflow (what this talk covers), and only invest in fully autonomous planning agents once evals, sandboxing, and prompt-injection defenses are in place.
Highlights
”It can tell O(N²) from O(N) just by reading the call stack”
“Don’t think too much about the method names here but the key point that I wanted to call out is that it was able to use these functions, it knows the meaning of these functions and when it sees it is being used in a poor way, it can actually identify that this is a quadratic algorithm and not a linear algorithm.” — Rajat Shah, 8:11
Clip command
yt-dlp --download-sections "*8:11-9:00" "https://www.youtube.com/watch?v=CgsWxRUY5Eo" --force-keyframes-at-cuts --merge-output-format mp4 -o "quadratic-from-callstack.mp4"
”The same bad pattern, found in seven services”
“In this case we found that same bad pattern was actually implemented in seven different services through cross-repo code searches.” — Rajat Shah, 13:00
Clip command
yt-dlp --download-sections "*13:00-14:01" "https://www.youtube.com/watch?v=CgsWxRUY5Eo" --force-keyframes-at-cuts --merge-output-format mp4 -o "seven-services-same-pattern.mp4"
”A human is still the one who approves the fix”
“I’m still keeping the confidence bar to just send a code review and not directly push it to production.” — Rajat Shah, 19:29
Clip command
yt-dlp --download-sections "*19:29-20:53" "https://www.youtube.com/watch?v=CgsWxRUY5Eo" --force-keyframes-at-cuts --merge-output-format mp4 -o "human-approval-still-required.mp4"
”Profiler gives the estimate, canary gives ground truth”
“Here’s a mental model: profiler gives the estimate, canary gives ground truth, canary is the way you verify and evaluate that your suggested fix is actually going to improve and have some positive impact and no negative impact, and engineer makes the eventual decision.” — Rajat Shah, 23:42
Clip command
yt-dlp --download-sections "*23:42-24:30" "https://www.youtube.com/watch?v=CgsWxRUY5Eo" --force-keyframes-at-cuts --merge-output-format mp4 -o "profiler-estimate-canary-truth.mp4"
”Why full agent autonomy isn’t there yet”
“You can’t simply run AI agents – without the right level of security guardrails so that prompt injection and other security attacks that a typical – agent infrastructure currently – cannot always solve for, – those – things need to be looked at.” — Rajat Shah, 32:21
Clip command
yt-dlp --download-sections "*32:21-33:00" "https://www.youtube.com/watch?v=CgsWxRUY5Eo" --force-keyframes-at-cuts --merge-output-format mp4 -o "why-not-full-autonomy-yet.mp4"
Key Points
- Speaker intro (0:00) - Rajat Shah, staff software engineer at Netflix’s AI Platform Organization, building large-scale distributed systems for ML model hosting.
- The problem: fast code, not efficient code (0:52) - Coding agents let engineers author code at 10x speed, but compute cost rises at a similar pace because agents don’t reliably write performant code.
- The manual profiling workflow (2:31) - A human triggers profiling on a production instance, downloads raw call-stack/CPU JSON, opens it in a visualizer, and manually hunts for hot paths — a slow, tedious, rarely-repeated exercise usually done only during 2 a.m. incidents.
- The core question (4:45) - Can an LLM agent read profiling data and do in far less time what takes a human engineer roughly 20 minutes?
- Assumption 1: profilers speak the same language (5:11) - Regardless of whether a service is written in Java, Python, or Go, profiler output (call stack, self/inclusive CPU) is structurally similar and well-suited for an LLM to parse.
- Assumption 2: agents already know common anti-patterns (6:10) - O(n²) loops, loop invariants, repeated object allocation, and lock contention are patterns coding agents already recognize from training on public code.
- First finding: a quadratic tensor merge (7:45) - Given only call-stack data (no codebase access), the agent identified a quadratic-time immutable-map-copy pattern inside a tensor merge method, consuming 8.8% of CPU during profiling.
- Full loop in under 5 minutes (10:21) - Given a skill/prompt, the agent checks out the exact production commit, locates the code, traces the call path, and produces a code review — all in under five minutes on a large codebase.
- Same anti-pattern found in seven services (13:00) - A cross-repo code search for a repeated counter-object allocation pattern found it independently implemented in seven services, worth 0.5–4.6% CPU savings if fixed everywhere.
- The memory problem (14:50) - LLMs have only compact working memory, not the accumulated context a human performance engineer builds up over time — so the team needed to give agents an external, persistent memory.
- Catalog as plain markdown, not a vector DB (17:07) - The pattern/anti-pattern memory doesn’t require fancy vector search; it starts as markdown files in a centralized git repo, readable and authorable by both humans and agents.
- Bootstrapping the catalog (18:30) - The catalog can start empty, or be seeded from public sources such as Jeff Dean’s C++ optimization blog post and PyTorch’s torchfix repo of anti-patterns.
- Catalog entry structure and confidence levels (19:29) - Each entry lists symbols, confirming services, and a confidence score that rises as more services validate the pattern — but the bar is still set to “send a code review,” not auto-merge.
- Guardrail: run tests before proposing a fix (21:29) - The agent runs existing unit/integration tests first, to build confidence that a proposed optimization doesn’t break business logic.
- Canary deployment as ground truth (22:01) - Old and new code run side-by-side on two machines with the same traffic for ~10 minutes; the comparison report (CPU, latency, error rate) is what the agent uses to decide whether to open a code review.
- Reactive vs. proactive: shifting left (24:00) - The reactive path (fixing code already in production) builds the initial catalog; as it grows, the goal is to move the catalog earlier — into code review, then into code authoring itself.
- Shift-left tradeoffs (26:14) - Hooking the catalog into code generation can slow authoring and burn more tokens, so the catalog needs hierarchical structure and indexing to avoid bloating agent context.
- Foundations before AI (27:13) - Rock-solid test coverage and canary automation have to exist first; the AI agent’s job is mainly to know how to invoke and read them, not to replace them.
- Three levels of autonomy (30:00) - Level 1: LLM assists identification only, human still runs the profiler/canary manually. Level 2 (this talk’s focus): a fixed, fully-tooled workflow that can run on a weekly schedule. Level 3: a fully agentic, planning/reasoning system that demands heavy investment in evals, sandboxing, and defenses against prompt injection before it’s viable.
- Closing (33:14) - Shah offers to share more detail via a future blog post at shahrajat.com.
Mentions
Companies
- Netflix (0:00) - Rajat Shah’s employer; the AI Platform Organization where this performance-engineering agent work was built and where the tensor-merge fix shipped to a live service.
Products & Technologies
- torchfix (PyTorch) (18:30) - Cited as a public source of cataloged anti-patterns for optimizing kernels and model graphs, useful for bootstrapping an internal pattern catalog.
- Tensor merge method (7:45) - The specific quadratic-time method (an immutable map copy) identified and fixed in the talk’s first case study.
People
- Rajat Shah (0:00) - Speaker; staff software engineer, Netflix AI Platform Organization.
- Jeff Dean (18:30) - Cited for a blog post on C++ performance-optimization patterns, referenced as a public bootstrap source for the pattern catalog.
Surprising Quotes
“Your agent doesn’t know specific details about your platforms and your frameworks and your internal codebase patterns. So it tends to just produce code based on what it might have already seen other codebases using or inventing new patterns in your codebases that you did not anticipate an engineer to use.” — Rajat Shah, 1:22
“This is not by looking at the codebase. This is purely by looking at the call stack that the profiling data produces.” — Rajat Shah, 8:51
“The solution that I mentioned so far of a memory is not very fancy vector search or a vector database that needs to store all the catalog of patterns and anti-patterns. Rather, you can start with just markdown files in a centralized git repo.” — Rajat Shah, 17:07
“It’s easier to imagine putting AI agent into every single step of your software development lifecycle for performance improvements, but that’s not the intent of this discussion or the thing or the playbook that I’m mentioning.” — Rajat Shah, 27:13
Transcript
Rajat Shah: 0:00 Hi there. Welcome to AI Engineer World’s Fair 2026 event. I am Rajat Shah. I am a staff software engineer at Netflix, where I work in the AI Platform Organization building large-scale distributed systems for machine learning model hosting.
Rajat Shah: 0:15 In this talk, I am here to share how we did improve our performance engineering throughput by introducing AI agents into the mix, and this is more of a playbook or a practitioner’s guide to help you also replicate similar learnings in your own organizations to improve the infrastructure cost and ship faster.
Rajat Shah: 0:42 Let’s first talk about the problem. Why does performance engineering doesn’t scale, and what does it cost to actually do it right?
Rajat Shah: 0:52 The problem is arising from the fact is that you are authoring code now at a 10x faster speed. The coding agents are getting better and better at solving problems, and as more and more engineers adopt it, it gets very easy to produce code in your system. And this is slight exaggeration, but the compute cost also is increasing at a similar pace because it doesn’t always write the fastest code.
Rajat Shah: 1:22 So this is where the problem arises because of that new byte coding era. The AI agent ships code, it is pretty much tuned to ship code fast. And of course, you could say that as the coding agents are evolving, newer models are coming into play, they get better and better at simply writing performant code. But that’s not always true. Your agent doesn’t know specific details about your platforms and your frameworks and your internal codebase patterns. So it tends to just produce code based on what it might have already seen other codebases using or inventing new patterns in your codebases that you did not anticipate an engineer to use as a pattern to use your framework.
Rajat Shah: 2:21 So let’s look at what a performance engineer typically does. I am calling this as a human performance engineer, which is responsible for identifying bottlenecks in a service and fixing them.
Rajat Shah: 2:31 Typically, a human would trigger profiling on a single production instance of a fleet of production instances. You would go and download it, potentially open it in a visualizer. The raw data that you download typically is not great to look at. It could be, for example, a JSON structure data of the call stack and where the CPU is spent. So using a visualizer helps you at least see and visualize.
Rajat Shah: 3:00 is the call stack and CPU time of various method in your services better.
Rajat Shah: 3:07 Once you have that visualizer opened, you pretty much end up spending a lot of time in like just looking at and finding in this treasure hunt on the path, on the potential places where you could improve the code to make it more performant.
Rajat Shah: 3:27 This takes a lot of time in order to even learn how to look at it, and there’s a learning curve to it.
Rajat Shah: 3:35 And this is where the real bottleneck ends up being. You end up having to spend straight many, many minutes to identify the bottlenecks.
Rajat Shah: 3:46 Once you’ve identified some code paths and some packages that are spending significant CPU cycles, you would end up searching it in your code bases, in your code repos and see if it has a potential for improvement.
Rajat Shah: 4:00 Hopefully you hit, you have luckier and you find a root cause and you produce a code review out, you merge it and you get some performance wins.
Rajat Shah: 4:13 And then you repeat all of this again. You see the problem, right? This is a very manual effort and very tedious effort to get right, and this ends up being a bottleneck if you were to do it across many of your code bases and code paths.
Rajat Shah: 4:31 And that’s why this is done very rarely. People typically end up looking at profiling data only when something is going wrong at 2:00 a.m. and somebody needs to fix a problem because your CPU is unbearable.
Rajat Shah: 4:45 So we asked this question internally: can an LLM read this profiling data? The 20 minutes that I mentioned an engineer spends in identifying hot paths, can an LLM agent which is fed that data also do it much faster?
Rajat Shah: 4:58 And we tried to answer this question through some live services. So the next couple of slides will be about this experiment and how we do it right.
Rajat Shah: 5:11 Before we get to it, the foundation of our assumption is that every profiler essentially speaks the same language.
Rajat Shah: 5:19 Even though you would profile and you could have your services written in Java, Python, Go, etc., the profilers’ example that I mentioned here do the same thing.
Rajat Shah: 5:34 They would run it in your, they would run against your production service on a single instance and they would essentially try to capture the call stack, the self CPU and inclusive CPU that you spent on each of the methods in your call stack, sampled at a very high frequency.
Rajat Shah: 5:55 So irrespective of which language, which runtime you’re using in production, the output of a profiling data is actually…
Rajat Shah: 6:00 very similar and very well structured for an LLM agent to use. This is one assumption that we had going into the experiment.
Rajat Shah: 6:10 The second assumption we had going into the experiment is that there are many common patterns that the coding agents have learned through their training sets across all the publicly available code sources that they might have had.
Rajat Shah: 6:23 So things like O of n squared loops, loop invariants, things that you are computing every time in your loop which could be computed just once outside it.
Rajat Shah: 6:32 If you have certain places where you are doing the object allocation very repeatedly and you could move it out as well or there are times when you would be like having some contention and places where you could optimize through better batching.
Rajat Shah: 6:51 All of these are very common patterns that an agent already knows. I mean assuming it is trained on very good quality software code.
Rajat Shah: 7:02 The better the quality, the better agent knows which patterns are not great. And this is a very good point for an LLM agent because it essentially means that it can technically look at the codebase and it can identify those patterns very easily.
Rajat Shah: 7:25 So this is more about like pattern recognition, like finding out these code patterns to find some code smells that are worth fixing.
Rajat Shah: 7:34 So when you have that profiling output that I mentioned and you have a coding agent that could understand it, you feed it into it.
Rajat Shah: 7:45 The profiling data that it needs to read is actually very well structured in a way that it could see that in this case the profiling data is in the case of row one and row three, they’re the same call path and the immutable map copy of that I have in my row three is actually used inside a tensor merge method that we have.
Rajat Shah: 8:11 So don’t think too much about the method names here but the key point that I wanted to call out is that it was able to use these functions, it knows the meaning of these functions and when it sees it is being used in a poor way, it can actually identify that this is a quadratic algorithm and not a linear algorithm.
Rajat Shah: 8:33 So this is where our first aha moment was that, okay, if you feed a profiling data, it if it knows the pattern, assuming the coding agent was trained on a very good quality data, it can actually identify and find out that find in your codebase where that bad patterns are actually running.
Rajat Shah: 8:51 And this is not by looking at the codebase. This is purely by looking at the call stack that the profiling data produces.
Rajat Shah: 9:00 And once a prof- an AI agent gets this, what are the steps that it needs to do in order to get to the potential fix?
Rajat Shah: 9:05 So we have mentioned that it can identify a pattern and say that, okay, this is a potential problem to be fixed.
Rajat Shah: 9:11 What happens next?
Rajat Shah: 9:12 It could, knowing where that method lives, it could do a code search and first of all, before, I mean yeah, it could do a code search and find out the code repo where this method is defined or this code is defined.
Rajat Shah: 9:23 And it needs to extract out the exact commit that is currently running in production, which is typically easy to get.
Rajat Shah: 9:29 You know which build is running in your production and you could point that to the LLM to check out the Git repo at that same commit.
Rajat Shah: 9:34 And then once it has found out and has the Git repo cloned, it can look for that exact code path, skip any internal library details and find out the code patterns.
Rajat Shah: 9:43 And now once it has the full code repo cloned out, the method’s definition and implementation identified, it can trace the entire call path of that method.
Rajat Shah: 9:49 And this structured data once it has this information becomes very powerful, and we are going to talk about how it leverages this.
Rajat Shah: 9:54 So these were our understanding going into the experiment. We knew it could find the patterns, we tested it with a couple of our services to see if it can actually not just find the pattern but also take it one step further and produce a real fix and provide a real fix that an engineer could approve and merge.
Rajat Shah: 10:08 So first finding, the O of N squared example that I just mentioned, it once it knew that this is a problem and looking at the profiling data, it knows that this is consuming 8.8 percent of the CPU time during that period of profiling.
Rajat Shah: 10:21 So for us, the coding agent not just ended up finding the problem, given enough instructions as a form of let’s say skill or prompt, you could actually have it do those four steps that I just mentioned, which is checking out the Git repo, finding where that code is implemented and actually sending a code review out.
Rajat Shah: 10:37 All of it could be done in a very large codebase with powerful enough code agents in less than five minutes.
Rajat Shah: 10:41 And in this case we also, which I’ll talk about a little bit in the later slides, were able to identify what is the savings if you were to reimplement it with O of N.
Rajat Shah: 12:00 with an optimized implementation in terms of both CPU as well as latency savings. I’ve shown the numbers here of our real production savings that we were able to observe through this exercise.
Rajat Shah: 12:14 So this was like a first proof for us that you could actually indeed introduce an AI agent into the mix and get your productivity wins of not just identifying, but also going all the way to a code review.
Rajat Shah: 12:30 The second great benefit is that once you have this well setup, the other example that we were able to take it forward to it is let’s say it identifies the pattern in one of the services that we profiled and the learning that we had is that it could actually look at all your potential places where that same bad pattern is being reused. So
Rajat Shah: 13:00 in the previous example it just identified one bad implementation. You fix it you get all the benefits. In this case it identified not just a bad code, but because that pattern in this case which is a counter object for our spectator metrics that are being created on every single iteration of a hot path in our service stack it could essentially see that this is a real problem and it could double down and search multiple services where that same pattern is being repeated and scale up the effort to fix it in multiple services. In this case we found that same bad pattern was actually implemented in seven different services through cross-repo code searches and if fixed it could actually if fixed across all of those different code repos it could have savings between 0.5 to 4.6% of CPU cycles.
Rajat Shah: 14:01 So we have discussed the problem, our hypothesis and the experiment which proved out that a LLM coding agent could actually help improve the finding and fixing of your suboptimal code. Let’s see how we can actually build this into the development cycle. It doesn’t have to be a very retrospective problem where once a production service is having issues you end up doing all of those exercises. I’ll talk about how you can actually leverage it in your end-to-end software development cycle so that you get the benefits much earlier and potentially avoid having that suboptimal code reach production altogether.
Rajat Shah: 14:50 The first real problem is that LLMs if you invoke them they do have some memory these days but it’s very compact memory and it will it won’t…
Rajat Shah: 15:00 You don’t have all the information that you are that you potentially as a performance engineer know when you’re trying to debug. So you when you if you think of a coding agent and if you want to build it at par as sufficient as as efficient as a human engineer would typically be, you want to introduce a long-term memory that it could reuse. So that’s our first real problem to solve for and I’ll talk about how you solve that.
Rajat Shah: 15:29 The simplest way is to first think of you want to build a catalog. Patterns, anti-patterns, once you have identified enough of them, you could put it in a catalog that that users that is used by an LLM agent, which can be stateless of its own, and together a stateful catalog and a stateless LLM can become a full fleet-wide memory for for a coding agent to use.
Rajat Shah: 16:00 The foundation for this and and to foundation for incorporating in this development cycle is practically just this. You want to keep this very central so that it’s not very team specific, not very product specific, but rather a central ever-growing catalog. As the catalog patterns get written, if there are certain things that could be generalized, in that example, the first example that I the first finding that I provided around O of N square implementation is actually very generalizable. All coding languages will potentially have the same problem if if the code is written and implemented in that way. So you you can take it a step further that your catalog can become even more ubiquitous and usable across languages and frameworks as well. And you think of it as a blueprint as as that catalog starts growing, it becomes your building block for future coding agents to actually use for identifying the problem in your development cycle and that’s the next thing I’m going to talk about.
Rajat Shah: 17:07 The solution that I mentioned so far of a memory is not very fancy vector search or or a vector database that needs to store all the catalog of patterns and anti-patterns. Rather, you can start with just a markdown files in a in a centralized git repo.
Rajat Shah: 17:32 As the production findings appear, have the coding agent put more and more patterns and anti-patterns into this. And the benefit is that even if one service finds this, even if one profiling even if one service did the profiling to find this pattern, multiple services that in the future are going to run profiling, the agent that is doing the profiling of that other service can use the pattern catalog from the first service.
Rajat Shah: 18:00 …to find the patterns more optimally and not having to redo all the exercise that the first agent did in order to identify that anti-pattern.
Rajat Shah: 18:11 Um, you might think of like how- how do I get started with this? Well, you don’t always have to start fresh, but you could start fresh. There’s no harm in having a catalog that is just empty in the beginning and as more and more profiling happens, uh, more, uh, powerful the coding agents that are referring it could become.
Rajat Shah: 18:30 But there are few- many in fact, uh, sources of how you could- public sources of how you could improve your coding agents. If you’re into C++ optimizations, Jeff Dean had this wonderful blog post, uh, around how to look for certain, um, optimization opportunities in your C++ code. PyTorch has a torchfix code repo where it catalogs several anti-patterns that could help optimize kernels and, uh, the model graph of your PyTorch model. And then you might have your own performance playbooks as well. Typically as your organization grows and your enterprise software evolves, you end up noting down certain patterns and anti-patterns. Those all could become the bootstrapping point for that catalog git repo that I mentioned that can act as a centralized place for all- for all patterns and anti-patterns to be noted.
Rajat Shah: 19:29 And what would an entry in that catalog look like? Uh, the example is right here. Um, as the left hand side it mentions the small hints that an LLM coding agent in the future could use to easily query that catalog. It can mention a list of symbols, the services where it was confirmed, some confidence level is great here as well because as more and more services confirm this, you want to keep updating this so that the future agents get more and more confidence that okay, this is worth a human- a human review and I would want to more confidently send out a code review to a user. I’m still keeping the confidence bar to just send a code review and not directly push it to production. That’s my intent. I’ll talk a little bit on why, but I do feel like there is still need for a human approval because you’re modifying an existing code that is running just fine in- in production in order to optimize it, which is- which is very risky if you don’t know the business context, if you don’t have enough test coverage, you might end up breaking it. So there is a need for a human to actually be the one responsible for approving that code review.
Rajat Shah: 20:53 And on the right here I mention what the entry in that pattern and anti-pattern catalog could look like. Um, an anti-pattern and a good pattern, so that both the things are well noted here.
Rajat Shah: 21:00 still needed. So how do you optimize for noise there? There could be enough signals that the AI agent could think of improving and it could arbitrarily send code reviews for them without knowing enough about your system. There are few tips that I mention here could be very powerful in in making sure that the agent first has enough verifications done before looking for an human’s attention.
Rajat Shah: 21:29 The integration tests, unit tests, the most basic functional tests that your system needs to have should be run ahead of time by the coding agent itself so that it knows that as it is making code changes to optimize it, it is also not causing any production business logic to fail. So as as you have good coverage in unit tests, it gets more confident that it gets more confident that your change that the change it is going to propose is actually right.
Rajat Shah: 22:01 Once it has made that functional correctness check, you still our code coverage isn’t typically ideal, there are still surprises in production, and canary deployments could be very powerful in this case. If you provide enough automation around this, the canary deployment could be a prerequisite before the code review requires human attention. What is a canary? You typically have as you typically have two machines, one containing your old code, another containing your new code, which could be the performance fixed code, and you send the same traffic to both of them over a period of let’s say 10 minutes and try to compare the CPU usage across between them.
Rajat Shah: 22:54 The comparison report typically is is what you would want the AI agent to make decision on and judgment on whether it should open a code review for it. In this case, the observability report could contain standard infra things like how how much CPU reduction, how much latency reduced, is there an increase in error rate? If if that’s the case, it should see that as a red signal to not proceed because it might have gotten the business logic incorrect in an attempt to optimize the code.
Rajat Shah: 23:22 And what I mention here is that this is not an AI problem, the observability, canary, verify logic, these are all standard checks that you need to have in your system. And then final guardrail that you want to have is an engineer decision, like I mentioned, still very important, critical, in the mix.
Rajat Shah: 23:42 Here’s a mental model: profiler gives the estimate, canary gives ground truth, canary is the way you verify and evaluate that your suggested fix is actually going to improve and have some positive impact and no negative impact, and engineer makes the eventual decision.
Rajat Shah: 24:00 So the path so far has been about how do you improve the efficiency of getting from a profiling data to producing a code review fix. That’s the reactive path that I just covered so far, which means you have the code already running in production and now you’re trying to improve it.
Rajat Shah: 24:30 That’s typically too late in the game. Now anytime you try to change a thing that is running in production in order to improve the performance of it, you have to be very, very careful. So you want that to shift left. You want that reactive path to be your initial guide to build that initial catalog. And as your pattern catalog grows, you want to move closer and closer to the proactive path where as a human or a coding agent authors new code…
Rajat Shah: 25:02 …the reviewer agent could actually look up that catalog and provide an inline comment by understanding the code change and providing a suggestion that, hey, based on the pattern catalog and the observed profiling data that I have, I feel confident that this is an anti-pattern to introduce. Can you rewrite this code? That could be a review comment that a reviewer agent could be providing.
Rajat Shah: 25:31 And if you shift left all the way, like why even wait until a code review? That’s where you get the most power. Because you know the anti-patterns are already in a catalog. And if you are using a coding agent to write your code, you could actually ask the coding agent and hook that catalog in it so that when it is producing newer tokens and producing newer code, it could actually, before writing out the inefficient code, directly reference the catalog up front and write your code in an optimal way to start with.
Rajat Shah: 26:14 This could sometimes slow down the speed of newer code being written and might end up consuming more tokens. And that’s why the pattern catalog needs to be very well structured and indexed so that the way it parses and reads the catalog doesn’t fill up the agent context too much, but rather it can navigate it in a very hierarchical format and only be able to find out the exact places it needs to look for a given pattern or an anti-pattern.
Rajat Shah: 26:41 This means that if you can catch this in the code authoring phase itself, you skip all of the unnecessary overhead of somebody reviewing your code and providing a suggestion or even having that code change reach production. So the sooner you catch it or even sooner you…
Rajat Shah: 27:00 …introduce that pattern catalog into your software development end-to-end lifecycle, the easier it gets for you to keep the code and service running optimally.
Rajat Shah: 27:13 And I do want to highlight this, that it’s easier to imagine putting AI agent into every single step of your software development lifecycle for performance performance improvements, but that’s not the intent of this discussion or the thing or the playbook that I’m mentioning. You still want these foundations very, very right. Your test coverage needs to be rock solid, business logic all needs to be very well encoded in your tests, your canary automation needs to be amazing. And this is where you don’t need AI, you pretty much just want the AI agent to know how to invoke a canary, where to get the downloaded report from a canary so that it can directly read the numbers from the report and make a judgment call. And that pattern catalog that I mentioned is also a foundational piece which needs to be gotten, which you need to get right. It is by design a Git repo so that it’s both a human and a coding agent readable and authorable, and you could actually have humans also authoring patterns and anti-patterns as they’re, let’s say, building new frameworks and new platform components that they have enough context on at the time of building.
Rajat Shah: 28:22 The takeaways that I do want to mention, first, build your code foundations, think of all of the automation that you could do, but before introducing an LLM, try to see if there are good integration points that an agent could use, such as can it automatically trigger profiling on an instance on a CPU cluster that you have, can it download the profiling data effectively, then feed it into an LLM model, and can it, once it has identified the fixes by cloning the Git repo, etc., can it actually validate that through a real production canary through shadow traffic or real traffic? All of those integration and foundation pieces needs to be really good in order for this entire life-to-life, and this entire lifecycle improvement that I’m mentioning, otherwise it will just cause more friction and more bugs in the production. So you have to make these foundation pieces investment a lot more important. And start from here, don’t think of the reactive path as a bad approach, reactive path is where you want to start with and then gradually move towards the left and into the proactive path.
Rajat Shah: 29:38 And the other takeaway, like I mentioned, is important is that as you build this memory of, or essentially a catalog, the sooner you catch it, the better it gets for your end-to-end software development lifecycle to have optimal, optimal code running in production.
Rajat Shah: 30:00 This is a – this is a great mental model for how you step-by-step proceed in automating or improving your efficiency of software development lifecycle. The current state, which is typically the norm, you spend hours finding the problems, it’s all very manual, no LLM in the mix.
Rajat Shah: 30:18 Start with introducing an LLM into the mix just for the – the identification of – of a potential fix. It might be okay if you were still having to trigger a profiler and produce output and feed it into an AI agent and then having to manually run canary, but it still gives you a lot of leeway to spend time on the other parts of the workflow as opposed to just staring at those flame graphs to get the hot parts identified.
Rajat Shah: 30:48 And then this talk mainly covered the level two orchestration, which is or level two automation in an agentic spectrum, which is more about the tools, the integrations, the key harnesses and the hooks that a AI agent needs to have in order to have full power, in order to have like good – capability to identify a problem and give you in various development lifecycle areas of opportunities to improve your codebase.
Rajat Shah: 31:10 And – once you have this in place, once you have this standard step-by-step workflow, which could be very static, right? In this case, the workflow that I mentioned is – is very well-defined, predefined, there is no – LLM to reason and plan and – do the further level of automations.
Rajat Shah: 31:23 All it has to do is – you provide it a fixed workflow which is download the profile from certain place, or rather trigger the profile, download the profile, – and then analyze it, run the canary and provide a suggested fix. You could once you have that in place, you could essentially have a scheduled run on your services every week so that it can keep continuously identifying newer problems that might have not been – or that might have been introduced in the last one week, for example, and keep – the – the identification of the problem closer to when you wrote that code.
Rajat Shah: 32:21 And if you were to think of – more automation – in this place, you could – imagine a – a level where the workflow is not very fixed and rather the AI agent has capabilities to even plan, reason, act, – but you have to invest a lot more in evaluations, sandboxing. You can’t simply run AI agents – without the right level of security guardrails so that prompt injection and other security attacks that a typical – agent infrastructure currently – cannot always solve for, – those – things need to be looked at.
Rajat Shah: 33:00 To be heavily invested in if you were to go with level three autonomy. So start with level one, try to move to level two, and you get maximum benefits. If you need more automation, you could think of the next step as needed.
Rajat Shah: 33:14 Thank you so much. I hope you got some learnings out of this small talk that I had. If you want to connect more and connect further, I might end up putting a more detailed blog post on this on my shahrajat.com personal website. I’ll be happy to connect with you and provide more information. Take care. Thank you.
