How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads
How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads
Summary
Preetika Bhateja, Chris Souza, and Daniel Bump from the Google/YouTube Ads team share hard-won lessons from building a “seed-asset” agent that turns messy advertising creatives — low-quality images, cluttered visuals, heavy text overlays — into clean, reusable assets for downstream generative tools. Their core argument: agent behavior isn’t shaped by prompting alone, it emerges from a loop of prompts, evals, iteration, and feedback, and getting that loop right matters more than any single prompt tweak.
They walk through the maturity curve of an eval system. Start with a strong agent foundation — focused, LLM-friendly tools and an independent critique agent with a remediation loop — before investing in comprehensive evals. Counterintuitively, they advocate for an early “vibing” phase: an intuition-based, non-scalable approach where a small team eyeballs outputs and iterates fast, because jumping straight to scaled human or LLM raters too early causes wild swings as both the model and the eval calibration shift under you. As the system matures, the emphasis shifts to explanations over pass/fail scores, clear rubrics with examples, human-human agreement within the team, and — critically — reading agent traces, not just outputs. Their standout example: an agent explicitly told never to remove legal disclaimers that nonetheless detected a disclaimer and reasoned its way into deleting it anyway, a failure invisible in aggregate pass-rate numbers but obvious the moment you read its internal reasoning.
They close with a hill-climbing framework for launch readiness: after human eval, review precision/recall on the eval set, iterate on either the rating guide or the agent/prompt itself, and repeat. The key discipline is resisting the urge to hyper-fixate on a single failing example — because these are non-deterministic systems, what matters is whether a failure pattern shows up repeatedly across a curated golden set, not whether one run happened to fail. They end on choosing the right launch metric, investing in online evals that stay fresh with production data, and training raters (whether human or cross-functional) with the same rigor you’d apply to the model itself.
Highlights
”Early on, vibing can actually be kind of good for you”
“Early on, vibing can actually be kind of good for you. And what I mean here by vibing is basically doing things that are not exactly scalable to begin with… it will still give you like a very good idea of when you change this, what happens, and like, it allows you to more quickly iterate as well.” — Chris Souza, 3:58
Clip command
yt-dlp --download-sections "*3:58-4:38" "https://www.youtube.com/watch?v=xyL2Ltkh-SA" --force-keyframes-at-cuts --merge-output-format mp4 -o "vibing-is-good-early.mp4"
”It found a disclaimer — and removed it anyway”
“We started seeing that there were edge cases in which the agent was seeing the prompt and it was seeing that there’s a disclaimer present in the ad and then still removing it. And in this example, you can see in the initial trace, it actually detects that there is a disclaimer in what it’s searching for and it says, ‘Okay, I found a disclaimer and now I’m going to go ahead and remove it,’ which was not what we asked it to do.” — Preetika Bhateja, 11:02
Clip command
yt-dlp --download-sections "*11:02-12:00" "https://www.youtube.com/watch?v=xyL2Ltkh-SA" --force-keyframes-at-cuts --merge-output-format mp4 -o "disclaimer-removal-bug.mp4"
”Paid by the community of Keep Parks Clean”
“It’s a public parks ad, and if you’ll see at the bottom right, it says, ‘Paid by the community of Keep Parks Clean.’ And we sent it to the agent and it just removed it when we told the text explicitly not to.” — Preetika Bhateja, 12:00
Clip command
yt-dlp --download-sections "*12:00-12:59" "https://www.youtube.com/watch?v=xyL2Ltkh-SA" --force-keyframes-at-cuts --merge-output-format mp4 -o "keep-parks-clean-ad.mp4"
”This is kind of a trap”
“You might think to yourself, okay, well, I should update the prompt based on my eval and the trace of the agent — but if you do this, this is kind of a trap, because these are non-deterministic systems. What’s more important is that you actually rely on patterns.” — Daniel Bump, 15:00
Clip command
yt-dlp --download-sections "*15:00-15:52" "https://www.youtube.com/watch?v=xyL2Ltkh-SA" --force-keyframes-at-cuts --merge-output-format mp4 -o "hyper-fixation-trap.mp4"
”Human-human agreement should be strong within your team”
“That’s very important to do early on — as much clarity and examples you can give the raters, that would be super helpful. So human-human agreement should be strong within your team of what you consider a good use case and a good pass case for an eval.” — Preetika Bhateja, 8:07
Clip command
yt-dlp --download-sections "*8:07-8:40" "https://www.youtube.com/watch?v=xyL2Ltkh-SA" --force-keyframes-at-cuts --merge-output-format mp4 -o "human-human-agreement.mp4"
Key Points
- The core loop (0:12) - Agent behavior emerges from prompts + evals + iteration + feedback, not prompting alone.
- Reliability is harder than building (0:43) - Getting an agent to do what you actually want in production is the hard part; evals manage that.
- Start with agent foundations (1:16) - Optimize a focused, LLM-friendly toolset before investing in larger agent evals.
- Independent critique agent (1:54) - A remediation loop with a separate critique agent fills gaps the base toolset can’t self-correct.
- Evals prove value of change (2:33) - A strong eval lets you run ablations and prove whether a change actually helped.
- Reliability = capabilities + guardrails + evals (2:45) - Daniel’s framing of what makes an agent reliable in the real world.
- Non-determinism demands scale (3:00) - LLMs succeed and fail unpredictably run to run; you need measurement at scale to know true behavior.
- Define “good” strictly (3:28) - Evals that scale need to be strict and measurable, not vague.
- Vibing beats premature scale (3:58) - Early intuition-based iteration surfaces failure patterns faster than a comprehensive eval would.
- Scaling raters too early causes whiplash (5:31) - Big swings in scores as you’re still calibrating the eval and changing the model radically.
- Start small, start early (6:00) - No need for a massive golden set on day one; target a few core tasks first.
- Test the negatives (6:29) - Checking that the model didn’t do something bad is just as critical as checking it did the task.
- Clear rubrics with examples (7:32) - Ambiguous edge cases send raters back asking “what should I do here?” — invest in clarity upfront.
- Explanations over pass/fail (8:19) - A bare pass/fail doesn’t tell you where the agent should improve; explanations do.
- Multi-output evals get messy (9:00) - Ads evals aren’t single pass/fail — an ad can be brand-safe but inaccurate, so explanations matter more.
- Monitor human/LLM-judge agreement (9:43) - Track how often the LLM rater agrees with expert human raters to know if you can trust it.
- Traditional ML practices still apply (12:22) - Keep a test/validation-style set for edge cases; use it sparingly and refresh it with production data.
- Hill-climbing framework (13:01) - After human eval, check precision/recall, then iterate on either the eval/rating guide or the agent itself.
- Launch readiness = understanding regressions (14:23) - Distinguish acceptable trade-offs from critical failures before shipping.
- Focus on patterns, not isolated runs (15:00) - Non-determinism means one bad run isn’t a verdict; look for the pattern across the golden set.
- Recap: what makes a good eval system (15:52) - Representative of the product goal, continuously evolving, highly curated golden sets, trained raters, clear rubrics, and a deliberate launch metric.
Mentions
Companies
- Google / YouTube Ads (0:12) - The speakers’ team; builds image and video generative models for YouTube ads.
Products & Technologies
- Seed-asset agent (0:12) - The agent discussed throughout: turns messy ad creatives into clean, reusable assets.
- LLM-as-a-judge / auto-raters (9:00) - Automated raters used alongside human raters, with agreement monitored over time.
- Agent traces (11:02) - Internal reasoning logs used to debug why an agent violated an explicit instruction.
People
- Preetika Bhateja (0:12) - Product Manager, Google/YouTube; works on ads, evaluations, agents, and LLM-as-judge systems.
- Chris Souza (1:16) - Google; works on the seed-asset agent and its evaluation workflow.
- Daniel Bump (2:33) - Engineer, Google; focuses on image/video generation and computer vision.
Surprising Quotes
“Writing the evals can be a very small point, and humans arguing over what the rubric should be is kind of like a very large task here.” — Chris Souza, 6:43
“It’s usually a good idea to have some sort of data set to test for edge cases, like broader capabilities… if you have a test set, use it sparingly and also refresh it with prod data.” — Chris Souza, 12:22
“I think it’s getting more mainstream so hopefully it’s less controversial, but like six months ago our teams were like still figuring out okay how do we like do this, what’s expected out of it.” — Preetika Bhateja, 15:52
“So we can’t really guarantee how it will behave in the wild. And for some use cases, this is extremely important, right? And we need a way to measure at scale.” — Chris Souza, 3:00
Transcript
Preetika Bhateja: 0:12 Hi everyone. Sounds like everybody came back from lunch, so hope everybody is recharged and not sleepy at all. It’s always interesting to do a talk right after lunch because you never know, it’s a mixed crowd. But we’re very happy to be here, happy to see you all. Our talk is going to be about evals of course, we’re in the evals track. We’re going to talk you through what are some things that worked for us while we were building evals, especially for YouTube ads. We work on the YouTube ads team as part of the—we do image and video models for YouTube ads.
Preetika Bhateja: 0:43 So, building an agent is hard. I think anybody who is here in the audience probably has built an agent as a side project or as part of production systems. It’s a very hard thing to do, it’s laborious, it takes a lot of time. Making it reliable is harder. So having it do things that you actually want it to do in production, understanding the different kind of things that it can play with, how it’s going to react when you launch it to your end users, that’s always a very hard thing to do, which is why evals are a pretty handy way to manage that.
Chris Souza: 1:16 Yeah, and then basically, the first step when you’re doing this is, of course, you need to have your agent foundation. So when you’re building your agent, you know, you will want to have a focused and strong set of LLM-friendly tools to give your agent a very good foundation. So yeah, I would say it’s important to first optimize these tools and make sure they’re the best they can be before just jumping onto larger agent evals.
Chris Souza: 1:54 So once your tools are optimized, you can also take some other steps like making an independent critique agent, right, with a remediation loop. And this can fill more gaps as far as having a self-correction mechanism and filling those gaps where maybe your base toolset has limitations.
Daniel Bump: 2:33 And then once your base structure is defined, you can have a—you can then go to having an eval. And having a strong eval is very important as this gives you like a way of proving the value of changes you make, as well as running ablation experiments on any changes you make. So I would say this is a very essential tool for climbing the quality ladder, but again, it’s very important to have that good foundation to begin with.
Daniel Bump: 2:45 So yeah, the reliability of your agent is basically a function of the capabilities of the agent, the guardrails, and the evals. Understanding what your agents do in the real world, basically generative AI outputs—
Chris Souza: 3:00 But as I’m sure you’re all familiar, LLMs are not exactly deterministic, right? So, it can often fail in certain areas, or one time it can succeed, one time it can fail. So we can’t really guarantee how it will behave in the wild. And for some use cases, this is extremely important, right? And we need a way to measure at scale and make sure that it is getting the output we want despite the non-determinism of these models.
Chris Souza: 3:28 So, we need to define what’s good here. And evals allow us to basically understand and improve how the models behave in the real world by defining what good looks like. It’s basically just setting this is, you know, our target output. So to build evals that actually scale, they really need to be strict and measurable.
Chris Souza: 3:58 And so an interesting thing here that I think might be somewhat counter-intuitive is that early on, vibing can actually be kind of good for you. And what I mean here by vibing is basically doing things that are not exactly scalable to begin with. So when you’re first starting out, it may be that, you know, you could take a track of basically just going ahead and making the super comprehensive eval, right? But we found it actually works better to first do intuition-based approach where you kind of first see the capabilities and look at the outputs. And at this stage, it’s pretty easy to tell what the issues actually are, right? So even though this is non-scalable, it will still give you like a very good idea of when you change this, what happens, and like, it allows you to more quickly iterate as well. So at this stage, prompt tweaks can also have like large performance gains. You can make a radical change to the architecture, and your eval is not kind of like hindering you in this way. So it’s like a very good way, kind of, you know, like an early-stage company of just like first, you know, doing something, making more radical changes quickly. Um, so yeah, this way I think you can also get very familiar with what you’re building, what the failure patterns are, and it gives you more of a sense of depth and understanding it, which allows you to help climb in a targeted way. And these will basically be very useful learnings when you’re actually like building the more comprehensive eval.
Chris Souza: 5:31 Uh, yeah, and then there’s a chart here showing kind of, you know, if you jump to scale to these scaled raters like too early, it can cause you to kind of have like very big ups and downs as you might be iterating and calibrating the eval as you are struggling with changing the model radically.
Chris Souza: 5:57 Um, so another thing here is…
Daniel Bump: 6:00 You can, you should start early and start small. So, um, you don’t, as I said before, you don’t need to have like a massive golden set on day one. Um, you can just kind of start with a few core tasks. So you can look for your agent and define what are the primary things that you want to target, right?
Daniel Bump: 6:21 And just, uh, basically start with those like high-level things and can slowly come to a more detailed representation as you move on.
Chris Souza: 6:29 And so here it’s important to also test the negatives. Checking if the model like didn’t do something as bad, uh, something bad is just as critical as checking if it did the task.
Chris Souza: 6:43 Um, so yeah, and uh, yeah, there’s a funny visual here about, uh, writing the evals can be a very small point and humans arguing over what the rubric should be is, uh, is kind of like a very large task here.
Preetika Bhateja: 7:00 Uh, cool. So we’ve established that we want to start small and we want to write code at the start, not write code but like, write eval at the start and try to get a sense of like what our model’s doing good, where the agents are falling, find out all the patterns.
Preetika Bhateja: 7:14 Eventually you will get at a stage where you will try to involve more teams. So if it’s just a core team of like PM, eng, UX working at the start, then you will bring in more team, you’ll have a bigger golden set, a bigger data set that you want to test out. So you will think about scaled raters, LLM raters, all of that. So we’ll get a little bit more into what that looks like.
Preetika Bhateja: 7:32 So just a couple of things on like working with scaled raters and things that worked for us. Uh, one was that providing them with a clear rubric of what they were actually rating, with very clear examples.
Preetika Bhateja: 7:47 So we had a lot of situations, especially early on when you’re building things, of course, like there’s so many edge cases and difficult cases that we’ve not tested out that a rater might encounter. So they’re coming back to you saying, “Oh, what should I do in this case?” And then sometimes we at the team are like disagreeing on like, should this be a pass, should this be a fail, things like that.
Preetika Bhateja: 8:07 So I think that’s very important to do early on as much as clarity and examples you can give the raters, that would be super helpful. So yeah, to that point, like human-human agreement should be strong within your team of what you consider a good use case and a good pass case for an eval.
Preetika Bhateja: 8:19 The second things that we noticed that helped us a lot was getting explanations from raters. So when you do have your teams or other scaled members rate evals, it’s good to not just anchor on if it’s a pass or a fail. That doesn’t really tell you much about where should the agent improve, what was the thinking that went behind coming to that conclusion.
Preetika Bhateja: 8:40 So it’s helpful to get explanations of why they’re rating something a certain way. And this is true for like if you do single-side evals or side-by-side evals, like when you’re testing two models at the same time, having explanations of why one thing failed or one thing worked can be super helpful.
Preetika Bhateja: 8:57 Uh, other things to keep in mind is that you could also do multi-
Preetika Bhateja: 9:00 output, so we were asking scale raters, when we were building ads, like, are these ads accurate? Like, did we do the right things for it? Is it brand safe? Is it like something that we expected it to be? Things of that nature. So we had like almost like a multi-turn eval system. If you’re building those kind of cases, it can get a little tricky because it’s not exactly a pass-fail. Your raters could be like, “Oh, well, it does very well in brand safety, but it does not do really good in like accuracy” or things of that nature. So explanations really help you like get to the bottom of like where is it that the agent’s actually like missing things. And then you can also use that input to train your agent better. Okay, so now we talked a little bit more about involving cross-functional teams and human agents, but what about if you’re using LLM raters, auto-raters, LLMs as judges?
Preetika Bhateja: 9:43 So we ended up going down that path also. A few things that we tried to do to kind of set the, set a more comprehensive structure for ourselves is, one, we monitored disagreements, or in some cases teams monitor agreement. So basically if you can have a sample pipeline of sorts that is monitoring how a human rater, some expert would rate an eval versus how an LLM would rate it, you can get a sense of like how it’s trending and if the agreement rates are in the ballpark that you would expect it to be. The second thing was we went a little bit beyond pass and fail. So we also looked at agent traces which we’ll get to later, but when we were doing pass-fail evals and trying to understand how things were getting rated, we did a couple of spot checks to understand the reasoning behind those logic so we could really see what was going on, how did it come to the conclusion that something was a final pass or no. And then again, like just reinforcing the high-quality ground truth point that Daniel made earlier, so we want to give a golden set that’s like super expansive, it covers a broad range of use cases and it also has very high human-human agreement within your team.
Preetika Bhateja: 11:02 Okay, so this is a quick example of what we had seen in the agent, I’ll walk you folks through it. So, it basically says if you know, if you want to know what it’s doing, look at it at its thinking. We had given the agent one of the prompts and it was that for legal reasons, disclaimers can never be removed. And we had mentioned that to the agent a couple of times in the prompt, we had like trained it down that and it was all going fine, but then we started seeing that there were edge cases in which the agent was seeing the prompt and it was seeing that there’s a disclaimer present in the ad and then still removing it. And we could not find that if we were just doing a categorical, like, “this X percent pass rate” or not. So we really had to look at the traces to see what was going on. And in this example, you can see in the initial trace, it actually detects that there is a disclaimer in what it’s searching for and it says, “Okay, I found a disclaimer and now I’m going to go ahead and remove it,” which was not what we asked it to do. This is a sample image that I created that I ran through the agent as well.
Preetika Bhateja: 12:00 …agent that says, ‘America, we can do better.’ It’s a public parks ad, and if you’ll see at the bottom right, it says, ‘Paid by the community of Keep Parks Clean.’ And we sent it to the agent and it just removed it when we told the text explicitly not to. So those kind of things will happen, so it’s really important to, like, check the reasoning and how it’s, like, getting to the things that you care about.
Chris Souza: 12:22 Cool. One other thing, so like all ML systems, some things still are applicable, which is my favorite part. So agents will not generalize very well depending on the kind of, like, data sets that you’ve trained it on. It’s usually a good idea to have some sort of data set to test for, like, edge cases, like broader capabilities. Uh, also very good practice to have a test set of sorts, if some of you have worked in, like, traditional ML systems, it was always good to have test, validation sets of that sort. Same concept applies here. Uh, if you have a test set, use it sparingly and also refresh it with prod data.
Chris Souza: 12:59 Daniel.
Daniel Bump: 13:01 Yes. Uh, so then, uh, yeah, this slide is basically talking about hill climbing and how it can be quite rewarding. So, uh, yeah, you might find that after human eval, right, uh, you can, this diagram basically shows like the, uh, kind of stack of what you can do. So after human eval, if it meets the bar, obviously you can finish there, but, uh, it probably won’t. And then, uh, when it doesn’t meet the bar, uh, you can first review your eval set, uh, and find numbers like precision and recall, uh, and then you can iterate and basically, uh, you know, make changes to the eval or adjust your rating guide, things like this, or just adjusting the model itself or the agent, adjusting its tooling. Uh, and then through all of these things, you can both, uh, iterate on your eval along with, uh, iterating on the agent. And this loop, uh, once you have a, a very good eval defined, it works quite well for iterating on your agent, uh, and doing quality hill climbing, as I mentioned before, doing ablations and whatnot.
Daniel Bump: 14:23 And then, uh, so as far as, uh, launch readiness, you, uh, basically need to understand, um, regressions. So identify where and why the model performance is, uh, degrading so that you can distinguish between acceptable trade-offs and critical failures. Uh, so yeah, this is basically just saying like it’s very important to understand from these evals, right, what is the exact issue that you’re having and figure out, uh, the trade-offs here. And then, uh, you should also, this is a very important point, so you should focus on patterns rather than isolated runs. So a tempting thing is to hyper-fixate on very small examples from the model, right? So you might have a…
Daniel Bump: 15:00 at one run that you do with your agent and then you find it fails on this case and you might think to yourself okay well I should update the prompt based on my eval and the the trace of the agent right but um if you do this this is kind of a trap because as I said before these are non-deterministic systems so what’s more important is that you actually rely on patterns so in your golden set it’s important to have multiple examples that can cover these kind of patterns that you might want to see and you basically want to look at the entire picture of how often is it failing on that pattern not that specific individual example um and yeah it’s important here also to invest in online evals uh and and making sure that your data is matching the real world representation
Preetika Bhateja: 15:52 Okay awesome. So we talked a bunch about like what worked for us while we’re building evals, of course your mileage may vary depending on your application things can things can differ. some of the things that we wanted to recap here was it’s like what what we think makes a good eval system generally is that it should be representative of what you want your product to be targeted at and that will differ depending on the state at which your agent is when you’re building MVP cases it would look differently versus when you’re doing production rollouts so that would differ but it still needs to be very much centered around what do you want your product to be good at and optimizing for that. Um important to of course keep evolving that’s why we talked about having your online evals, having test sets that are refreshed with production data, having sampling pipelines all sorts of things. Highly curated golden sets which will also evolve as your use cases evolve so training teams whether it’s scale raters or your cross-functional teams on how to rate things how what are you expecting out of them that’s also very important that’s uh I think now it’s getting more mainstream so hopefully it’s less less controversial but like six months ago our teams were like still figuring out okay how do we like do this what’s expected out of it so I think investing in those trainings can be helpful and then rater templates and rubrics with clear set of examples so you don’t have scale raters coming back to you saying I’m not sure how to rate this lots of like things getting marked as like unknowns or I don’t know things like that. Also choosing the right launch metric so some of the launch metrics launch slides that we showed you this is a very high level of generally how you would do a launch readiness you would like check it you’ll do a bunch of iterations on the model you’ll do an AB diff or ablation and you’ll try to see okay where is the regression happening what’s an acceptable regression versus not things like that as you’re doing these systems it’s important to like get some clarity early on on what is your gatekeeping rule like what’s your launch criteria is there a certain precision recall number that you’re looking at is there some other metric that you look at if you’re doing a model eval then probably that metric looks different than just the usual precision recall so those things can also be important to keep in mind. Uh yeah those are all the
Preetika Bhateja: 18:00 steps that we have to build production-grade Evals. Thank you.
Chris Souza: 18:15 Awesome. Thank you very much. Uh, do we have time for questions, staff? One? Do we have time for questions? Just one. All right. You went up first, sir. Go ahead.
Daniel Bump: 18:24 Are all your eval judges being performed by humans or are you also using LLM as a judge? And if so, what’s your calibration process look like for calibrating that judge to provide good evaluations?
Preetika Bhateja: 18:41 Yeah, I think that’s a good question. I think I wouldn’t say all. I think it depends on various use cases, like depending on like what kind of systems you’re trying to build, we have of course, like a plethora of use cases. So I won’t say all. I can’t go into details about what the benchmarking and all of that system looks like, but some of the things that we talked about in terms of disagreement rates and monitoring like sampling pipelines, those things hold true generally, yeah. Happy to chat more offline, yeah.
Chris Souza: 19:05 Awesome. Great. Thank you very much, Daniel and Preetika. Please give them a round of applause. You can always ask them questions in the back. Thank you.
