The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
Summary
In this short evals-track opener, Aparna Dhinakaran, co-founder and CPO of Arize AI, argues that eval methodology has to evolve as fast as the agents it grades. Arize runs over 100 million evals a month for its customers — the average team runs about 12 eval jobs, and the top teams run over 3,800 different evaluators — but the target kept changing underneath first-generation evals: 2023’s systems just answered a prompt, while 2024 added tool calls, reasoning, deep research, and now full sub-agent loops on long-horizon tasks. Every jump in capability was a jump in complexity, and complexity changes how systems fail.
Arize felt this pain firsthand building its own in-product agent, Alex, which gained longer memory and the ability to generate dynamic UIs — and also started forgetting context, losing track of completion, and getting stuck in loops that classical LLM-as-a-judge evals, with their fixed rubrics, simply couldn’t catch, because every user interaction produced a genuinely different trajectory. That led to Arize’s core thesis: the best way to evaluate an agent is with another agent. Agent-as-a-judge does adaptive, dynamic analysis rather than scoring against a fixed rubric, and Dhinakaran frames it as an addition to the eval toolkit, not a replacement — most teams today only use deterministic checks and LLM-as-a-judge, but the future is having all three working together.
She closes by announcing Arize’s new product, Signal — a long-running agent that reads traces, discovers failure patterns a fixed rubric would miss (like inefficient trajectories or repeated tool calls), and can go as far as opening a pull request with a fix.
Highlights
”What if the best way to evaluate an agent was actually with an agent?”
“So, this led to our really big revelation. What if the best way to evaluate an agent was actually with an agent?” — Aparna Dhinakaran, 3:45
Clip command
yt-dlp --download-sections "*3:30-4:29" "https://www.youtube.com/watch?v=q2JrUKBMf0w" --force-keyframes-at-cuts --merge-output-format mp4 -o "agent-as-a-judge-thesis.mp4"
”Over 3,800 different evaluators”
“We run over 100 million EVALS every month. The average team runs about 12 different EVAL jobs with the top teams running over 3,800 different evaluators.” — Aparna Dhinakaran, 1:04
Clip command
yt-dlp --download-sections "*1:04-1:36" "https://www.youtube.com/watch?v=q2JrUKBMf0w" --force-keyframes-at-cuts --merge-output-format mp4 -o "100-million-evals-a-month.mp4"
Signal: the agent that opens a PR to fix what it finds
“Signal’s actually a long-running agent that can read traces sent in, discover patterns of issues… it can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics.” — Aparna Dhinakaran, 4:35
Clip command
yt-dlp --download-sections "*4:35-5:23" "https://www.youtube.com/watch?v=q2JrUKBMf0w" --force-keyframes-at-cuts --merge-output-format mp4 -o "signal-agent-opens-pr.mp4"
Key Points
- Setting up the evals track (0:13) - Aparna Dhinakaran introduces Arize and the day’s evals-track lineup (Turnbench, Uber, Snorkel) in room 205.
- Evals scale at Arize (1:04) - 100M+ evals/month; average team runs ~12 eval jobs, top teams run 3,800+ evaluators.
- Industry consensus (1:36) - CPOs at Anthropic and OpenAI, and investors like Gary Tan, all say evals are essential.
- The target kept changing (2:08) - 2023: answering a prompt. 2024+: tool calls, reasoning, deep research, sub-agent loops on long-horizon tasks.
- Complexity changes failure modes (2:30) - Each capability jump wasn’t just harder — it was a fundamentally different problem to evaluate.
- Arize felt it building Alex (2:45) - Their own in-product agent gained memory and dynamic-UI generation, but also started forgetting context and looping.
- Fixed rubrics can’t grade variable trajectories (3:30) - Every user interaction with Alex produced a different UI/trajectory — a fundamentally different shape than a deterministic flow.
- The agent-as-a-judge thesis (3:45) - Adaptive, dynamic analysis as a complement to (not replacement for) deterministic checks and LLM-as-a-judge.
- Most teams use two of three tools (4:29) - Deterministic evals and LLM-as-a-judge are common; agent-as-a-judge is the missing third leg.
- Announcing Signal (4:35) - A long-running agent-as-a-judge product that reads traces and surfaces subtle failure patterns.
- Signal catches loops and inefficiency, then fixes them (4:55) - Detects things like repeated tool calls and inefficient trajectories, and can open a PR with a fix.
- Closing invite (5:23) - Arize booth next to OpenAI’s; evals-track talks in room 205; a World Cup viewing party that evening.
Mentions
Companies
- Arize AI (0:13) - Aparna Dhinakaran’s company; evals, observability, and experimentation platform.
- Anthropic (1:36) - Its CPO cited as agreeing evals are essential.
- OpenAI (1:36) - Its CPO cited similarly; also Arize’s booth neighbor at the conference.
- Turnbench, Uber, Snorkel (0:30) - Other speakers/companies featured in the same evals track.
Products & Technologies
- Alex (2:45) - Arize’s own in-product agent; the internal case study for why fixed rubrics stopped working.
- Signal (4:35) - Newly announced long-running agent-as-a-judge product that reads traces and opens fix PRs.
- LLM-as-a-judge (3:17) - The classical fixed-rubric eval method Dhinakaran says is now insufficient alone.
- Agent-as-a-judge (3:45) - The new adaptive-analysis eval paradigm central to the talk.
People
- Aparna Dhinakaran (0:13) - Speaker; co-founder and Chief Product Officer, Arize AI.
- Gary Tan (1:36) - Quoted: “Evals are everything you need.”
Surprising Quotes
“We run over 100 million EVALS every month.” — Aparna Dhinakaran, 1:04
“While we were building all of these first-gen EVALS, the thing that we were actually evaluating has changed underneath us.” — Aparna Dhinakaran, 2:01
“Because it has all that analysis, it can go put up a PR and put up a fix.” — Aparna Dhinakaran, 4:55
Transcript
Aparna Dhinakaran: 0:13 Awesome. Well, hey everyone, my name is Aparna, one of the founders of Arise. We work with some amazing teams to help them build EVALS. And we have an incredible lineup of talks for you all today at the EVALS track. It’s happening in room 205 and there’s going to be…
Aparna Dhinakaran: 0:30 …amazing speakers from Turnbench and Uber and Snorkel kind of all happening after this. Um, but today I’m here to talk to you about the future of EVALS. EVALS have gone from the new skill that every PM and every AI engineer has to learn…
Aparna Dhinakaran: 0:44 …to the thing that every serious AI team is betting on. We’ve been really fortunate to get to work with some of the best AI teams in the world. So we get a front row seat into not just what’s happening when they’re building their actual agents and before they actually ship, but actually the EVALS that teams are running on their live…
Aparna Dhinakaran: 1:04 …production agent via their traces. Little bit of some stats for you guys. We run over 100 million EVALS every month. The average team runs about 12 different EVAL jobs with the top teams running over 3,800 different evaluators.
Aparna Dhinakaran: 1:22 And offline EVALS, online EVALS, they each have their own place. But today what I’m actually going to talk to you about is the teams that are running EVALS on their traces. This is actually what’s helping teams figure out what’s working, catch their failures, and that’s…
Aparna Dhinakaran: 1:36 …the type of data you need to fuel your continual learning loops. And the industry kind of agrees. I mean, all the CPOs of Anthropic, OpenAI, all, you know, GDB, you have Gary Tan saying, EVALS are everything you need. And the whole industry kind of agrees.
Aparna Dhinakaran: 1:52 So we added EVALS, they catch all the failures, right?
Aparna Dhinakaran: 2:01 Here’s the problem. While we were building all of these first-gen EVALS, the thing that we were actually evaluating has changed underneath us.
Aparna Dhinakaran: 2:08 In 2023, it was about just answering a prompt. In 2024, we started to see all the frontier models, they’ve added tool calls, they’ve added reasoning, they’ve added deep research. Now what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks.
Aparna Dhinakaran: 2:30 Every one of these was actually a massive jump in complexity. And we didn’t just make the problem harder, we actually got a fundamentally different type of problem. What that meant is that as these systems got more complex, so did the way that they actually fail.
Aparna Dhinakaran: 2:45 We’re really lucky because we have our own agent that we’ve built, Alex, that lives in our UI and we get a kind of get to feel this pain ourselves. Every time the frontier labs added new functionality, we added it to our agent. And now Alex…
Aparna Dhinakaran: 3:00 Alex can has much longer memory. It has the ability to create dynamic UIs. It can go search across an enormous volume of traces.
Aparna Dhinakaran: 3:09 But we also realized that it would forget context. It wouldn’t know when something was done. Uh, sometimes it would just get stuck in these loops.
Aparna Dhinakaran: 3:17 And the key thing here is that the classical LLM as a judge evals that probably many of you have written in this room just weren’t enough for us to be able to catch all the types of failures that we were experiencing.
Aparna Dhinakaran: 3:30 I mean, it’s just fundamentally different, right? You have a deterministic flow and now what we have is literally every time a user interacted with Alex, it would create a new UI. That’s a fundamentally different trajectory.
Aparna Dhinakaran: 3:45 So, this led to our really big revelation. What if the best way to evaluate an agent was actually with an agent?
Aparna Dhinakaran: 3:54 Doesn’t mean that all of the ways that we did evals with deterministic evals, with LLM as a judge classic evals doesn’t matter anymore, but it just means that we have a different type of tool to solve a different type of problem. Agent as a judge is about adaptive dynamic analysis. LLM as a judge just gives you a fixed rubric with these fixed scores. It’s what everyone’s doing. But when your agent’s doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.
Aparna Dhinakaran: 4:29 My take is that most teams today are doing the first two, but the future of evals is actually having all three.
Aparna Dhinakaran: 4:35 And today I’m actually excited to share we’ve released agent as a judge, um, to help our teams on their eval journey. We’ve released Signal. Signal’s actually a long-running agent that can read traces sent in, discover patterns of issues, um, it can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics.
Aparna Dhinakaran: 4:55 It’s helped us figure out, um, very subtle failures that you wouldn’t even think of doing, such as something going on in a loop for multiple times, it was calling the same tool, uh, for repeatedly long time, the trajectory was inefficient. And actually what this does is because it has all that analysis, it can go put up a PR and put up a fix.
Aparna Dhinakaran: 5:23 So, if you want to learn more, come to our booth. We’re right by the OpenAI booth. We’ll give you a demo, we’ll show you a bit more about it. Um, we’re also, like I said, taking over the evals track. So, come to room 2005. We’re going to be talking a lot about the future of evals and what they look like. And if you just want to hang out with our team, we’re throwing a viewing party for the USA World Cup, uh, game tonight. So, uh, check out the Luma and register to come join us. Awesome. Thank you all so much.
Aparna Dhinakaran: 6:01 Hi, I’m Aparna Dhinakaran, co-founder and Chief Product Officer at Arize AI.
Aparna Dhinakaran: 6:04 Welcome to the Arize podcast, Model Forward.
