Beyond “Done”: Building Verification Loops for AI Agents

Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): Cerebras (2 vs 0), OpenAI (2 vs 0), Nick Nisi (1 vs 0), WorkOS (5 vs 0), Zack Proser (1 vs 0). Check them against the audio before quoting.

00:00:00 — Preventing Agent Disasters & Workshop Setup

So, what we mean by beyond done, just to like jump straight into it. You probably saw a tweet about this incident that happened a while back, where an agent just started deleting production database, and in nine seconds, everything was gone. That is what we want to help you prevent here today, while still trusting your agents and not having to look at every line, look at every PR. This is where we think things are going, and how do we get there? How do we make this a reality, where, instead of having that broken piece, we can have it calm, have everything work, and let you, to some degree, trust your agents, but build that trust over time.

I'd say also that it's not just about preventing errors, right? But this is also what we're going to look at today, foundational to achieving true scale.

So I'm Nick Nisi. I'm an engineering manager at WorkOS. I live in Omaha, Nebraska, and I've been there for about two years. And I do a lot with agents. I've written code since July 30th, 2025. I know because that's the day I got back from Larracan, I saw PHP and I was like, never again. And with me is my best friend Zack.

Hey, I'm Zack Proser. I'm an engineer in the applied AI team at Cerebras. We were sitting here. Now we had this up before. This rainbow is your workbench. All of the slides, our open source repo, you can get it from here. You can also view it at workos.cloud/slash beyond done, I know that's a mouthful, but the next QR or the next QR code, I believe, will take you there. Sorry, talks at workOS cloud slash beyond done.

And I just wanted to call it out because on that — I'll switch to it real quick. All of the slides are out there. There's also a page for the workshops, like the exercises that we're going to go through, and so for each piece of it, you can go through it. You can go through it on your own time. You can go through it later. This is a step-by-step guide to go through that.

There's also a glossary covering all of the terms, big and small, some that you're familiar with. Hopefully, most of them are familiar with. But if there's any that you're not, you can talking here — or sorry, you can look at the glossary for those terms. There's also an Ask Anything button, and you can talk to a rag that's trained on this slide deck, and hopefully get your answers there, but you can also just talk to a human.

Yeah, we'll leave this up as well. We find that a lot of folks want to ask questions for whatever reason can't. So be sure to check it out. After the QR codes, security would be so mad at us. In your.

00:02:47 — Midnight Snack Club & Silent Persistence Failures

All right, so when we're thinking about being done with work, it's being done, there's lots of things that go into us trusting them, and easily the first one is just the code I would have written. Did generate a good PR, and I go through that like a normal, normal code reviewer, but that takes a long time and these agents can write code a lot faster than us, so we don't want to rely on that too much because it's it just there's a signal to noise ratio that starts coming into play, and we just have too much code to review. We're not writing any code anymore, and we're less familiar with that code, so we become further abstracted away from it. So we need to find other ways to do that.

And there's plenty of ways that you can do that. There's you know using tools like browser tools like Playwright, for example, to say like, hey, I have this website, and I submitted a form. The submit worked. I checked that, and it showed me a success message. That's a great test to write. Not even really anything authentic about it, but it is something that agents can use to verify their work, and they can self-report that and even trust us, oftentimes, but not all the time, and so that's really what we want to drive is like, how do we make it so that we can actually trust these reports?

Absolutely. So everything that we've got here for you today in this workshop, as Nick mentioned, is backed by Rico and is interactive. So feel free to take an opportunity to actually run this, to experiment with it, to take something back that you might, you know — these are her words. I was experiencing it that way.

In this, so like, if you just go into the remote, like I'm doing, you do p and p m stack snack up. This is why I have shortcuts for everything. P is my T N P M. People talk about the browser, and we can look at it, and we can see this Midnight Snack Club app that we made. And you can play with it right now, and I can test this myself. Let me just put in my email address. I'll put in Zack at spam.

Yep. And look at that success! It saved the email. The email is here. I can see it. If I had a test that was looking to confirm that the email went on the page, it worked great, so that's what these examples look like. You can kind of see it as you're going, but if you reload that page, just like this, nothing's there. So even though it's — if you reload that, it shows that nobody's here. I can't drive the car apparently.

And that's because it did update, but it didn't persist. We didn't have any kind of verification. We were just checking UI, seeing oh did that actually do anything? And yes, it did do something to the UI, but we didn't check the other half. And so it was never persisted, and we lost that email. So sorry that you're not signed up.

Yeah, I think this is a really critical point. So, just show of hands, who in this room is building a ton of agents? Just to start with the first question, more than that — How many folks are running multiple terminal tabs or multiple chat sessions, multiple agents? Like everybody, for their own personal thing. And so this is what we're all experiencing, at least what I'm experiencing, this is the 18th, you know, lane of work that you're driving today, and you have like you know defracted attention to actually spend on it. These are the subtle bugs that will creep in because the agents like yeah I hear what you said the code looks good, it builds, it runs, the test pass, and it's still wrong.

And so, what we're going to ask and explore in this workshop together is how do we actually fix this problem, so that the agents that we're now all managing is like herds of insane toad machines can actually figure out what they need to do themselves, and when they need to do it. You'll notice that we aren't really looking at the code for this. We're not really going to look at the code too much for this because, like I said, I saw PTP and Skip, so.

00:07:02 — Recursive Self-Improvement & Defining Clear Outcomes

But all of this builds up to this great quote by a famous potato on X, call her name is Lauren Tan. She has this quote: "If any agent — if an entity can verify its own work, nothing can't verify — if you can't verify its own work, nothing else matters." Yeah, another way to say it is that, and I think this is true in my experience and others as well, is that we're kind of converging on a point that verification now, especially as model capabilities advance rapidly, is one of the key foundational things to focus on: reliability, verification, and evals.

Especially as we're starting to see more and more harnesses come online that have the concept of RSI or for you know, recursive self-improvement baked into them, if feels like we're shifting to a world where we may all be multiple self-evolving codebases, right? In that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those calculations on its own? How are we going to give the agent the feedback it needs?

Until we get all that, we're not going to trust these agents to really accelerate our work, too, it's politically. And so we need to solve that in various degrees, and it's not just going from zero to one, but it's a gradient and getting us closer to one, to where we can fully trust the evidence is the goal.

And I'll also just share that you know again anecdotally this is what we're all experiencing at WorkOS. We thought the AI team was building some of the stuff for other teams for ourselves. And it is just a night and day experience when you are iterating on a codebase that has you know the foundation evolved in place, a baseline that we run trusted, you know, we're getting some of the tips and tricks, but the ability to actually see that a neighbor did not do something correctly in the same way that we would as human beings, right? So we're you know we — the metaphor of us all becoming agent harness managers, as it is, you know, well done, this is how we ensure that each one of them is actually doing correct work and not wasting money and time, essentially.

Yeah, this is the equivalent of saying like, make no mistakes, right? If you want to have some verification in there, you can't just like guess at what it needs to be. You need to give it clear, define goals to get to, and give it a way to observe and check those goals. If not, the agent's going to do something, and it's going to spend some tokens, and it might get to the end that you expect it, but there's no guarantee of that. Every, every detail that you don't fill in terms of an outcome and what you actually want get to, it's just going to infer that or make something up, and that assumption that it makes may not be the assumptions that you made or that you were expecting because you're rational, logical human, and we're just not aligned on that yet, and so you need to have the systems in place to allow for it to observe, understand what you want, observe that, repair that, and then recheck that. Yeah, and I think just again really the role of my that the tired way to do it is to have a massive yard and have one agent session that's running for three or four days and say fix this, fix this, fix this, make it better. And what is working, you know, essentially perfectly for folks is decomposing tasks really giving each task an agent in the prompt exactly what it needs to do to know that it is actually done and to verify that it's done.

And those will just grow. I mean, we have people on our team that are nine hundred dollars a month, and those certainly improving APIs for the software. So, just anecdotally, it is something that we all kind of have to take a step back and think about. I'm not approaching this codebase anymore in a traditional way. I'm almost gardening multiple codebases and letting them self evolve.

00:10:45 — Centralized Verification Harnesses: Ctx & Riker

And this is something that we build, and we hold our own tools too as well. This is how Zack and I stay productive at WorkOS and how we get our own work done. We've each built kind of our own examples of this and our own ways of approaching this, and we really think that that's like a good thing right now. It's scary to really calcify on anything. So experimentation is finding what sticks and then running with that.

And one thing that we're both running with is the verification sort of like deriving everything from that verification loop, because as it says in this quote here, for mine status comes from records, not model pros. So, it's you know, show me the receipts. Show me the actual tangible proof that it actually did it, what I expect. Don't just tell me yeah, passes. It looks good because there's nothing to back that up. I need actual screenshots, videos, test results, smoke tests, I mean all of the above to prove to me without me having to look at the code that it's actually doing what I want because at the end of the day, I don't care about the code as much. I care about the result and what it's doing and what value it's delivering, and it can have bad code, honestly, and deliver that result pretty well, or have really good code that doesn't give me that result at all. And I need to find the right balance in there, which is.

Indeed, I've been doing this as well. I built a tool that I call Context or Ctx. It started out as a cross-agent harness memory layer, and rapidly experimenting as all of us are, ChatGPT, Claude, code, Py, etc., and I wanted them to all be able to write and read from memories. And that started, and then the more that I'm running multiple agents, and you know, average day is like 18 to 30 times, the more that I realized, it might be an interesting experiment to try to centralize authority for verification in a web service. The Ctx is simple web app that runs on Cloudflare. It was Python code initially. It works. The startup's memory was able to hand off secrets to agents without them ever seeing it or getting credentials.

And that worked too. And then now, as an additional experiment, I'm creating verifiers. And the way that works is the agents are assigned work as we're all doing the prompts. They go off and do that work. They come back through the MCP connection to Ctx and say, "This is what I've completed. Am I done? Please verify." So let's say in the example of one of the workflows needs to verify that a PR was created. And so that is going to run an evaluator on a known verifier. That could just be a simple CLI to start with, as it is in my repo, and then that becomes a hardened piece that you can then use again and again is going to the API and verifying that PR truly does exist, that it's healthy, and that all the merge checks pass. And only then will it mark that node satisfied in the workflow. Otherwise, it will redirect. It will redirect the agents. These are the things that are still failing.

Right. And so again, an experiment, but this is kind of how we've been try to play around with this and see which patterns are going to stick and what works.

It's also just kind of a fun time to do that by doing because I require weird pixel art across all the plans. So you can see in this case there was a simple workflow, and one of the verification gate is that requires that all pull requests are merged. And so that work is done by the centralized authority. It goes out and uses a verifier, checks the API or whatever the artifact is, and then finally delivers the status over here, right? Sidebar to go easy down and references the actual repository check and the PR right. So the simple example we could imagine having standardized verifiers, library, or skills, or tooling that you share amongst your org, and this could be one of the ways that you start to build toward accelerating, you know, PRs and less and less human oversight.

And then a similar tool that I built is called Riker. This is more like a engineering focused proprietary tools that's trying to be extremely flexible in how you hand work to it, but then once you hand work to it, its job is to babysit, and so I don't want to have to baby, said all the PRs that I'm bringing to fruition. At some point, there's a like this level of diminishing returns for me and my focus on the work that I'm doing because I end up just like babysitting the VR, fixing the tests, fixing the reviews that are coming in, the review comments and things like that.

So I want to be able to get the idea down of like what we should be doing, and how we can verify that work is done, and then I can hand it off to Riker. They can go do the work of you know, making sure that the reviews are done, that when it fixes code from PR reviews, that it's still matching our overall goal of working software in some way, and then can just verify that and alert me back when it needs me, instead of me having to babysit it. And so with that, it's just, you know, it builds a check. It figures out what tools it needs to run. It will run them, verify that they all pass. You'll keep that evidence and show it to me in different ways, and then decide when to stop.

So if I can verify, yes, it was working, we now fixed this bug, for example, we put them together and so it's really just keeping me out of having to be in the minutia of shipping the like final ten percent, and just focusing on where I'm valuable as a human, which is why we're doing the work in health, and how we can.

00:16:07 — From Vague Browser Prompts to Deterministic Scripts

Okay, so maybe just as a mental experiment or a thought experiment, if you will, let's just kind of show a kind of like — or feel free to shout out. How would we start to think as a human developer of verifying a web app? But maybe another four times. What are things that you would do? We'll throw some easy ones up here to get started, like refresh the page. Was that? Yeah. Yeah. Number ten. Getting healthy. Right. What are the responsibilities of getting back? Local host or local storage persistence in some way, console logs, things like that, network, like you said, and like reading what's actually on the page, or the contents of the page, interacting with it, right?

Yeah, and the reason we're saying this is like this is really to set up the transition from we were the human developers doing this all the time. This was part of our core responsibility when we're hand chiseling code, if you will. And then, what is the delta that doing this, and then in the early stages of LLM time, saying here is three paragraphs of text, go execute these instructions? The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real application, and that its changes did not break anything. Right. Otherwise, you're just getting the extrapolations that LLMs are designed to do, because they're trying to process the text, they're trying to make you happy.

Drage, you can't do any of this just in the terminal alone. It needs tools to do that. Fortunately, tools are very easy to add to agents now, and we just have to tell them how to use it. Tell them specifically what we're trying to prove. When you do that, you don't want it to be just a paragraph of like, hey, do this. Always verify the change in the browser, whatever that means. That's pretty vague. Remember to reload and check. That's a little bit more specific. Check that the data persists. Look at the console for errors. Getting a little more specific, but like that's a lot to put in there. You can certainly do that. That's what a lot of skills do. That's a lot what a lot of just like saved prompts do, but if you forget anything or things change, that can leave it up to a lot of interpretation.

One thing that might be better is to have a scripts that can prove it out in different ways, so that you can just run that over and over, and it's doing more of a deterministic check across a number of things, and then it can feed that output back into the agent, and it knows exactly what was wrong and how to fix it quickly, and it just closes that loop so that it's a much faster, and the agent can just spin that, run it again, check, verify, run again and just do that over and over until there's no more errors. It's also uniform if you think about it, right? We know that all of us are not perfect, so even giving thirty agents, the exact same problem, potential for variance in your outcomes. And putting it down to a single command that is backed by actual bash commands and bash calls, right? Like that's significantly more deterministically, and anecdotally, the folks like LearnTen and others that are achieving these thousands of PRs, ostensibly along with them, building ambitious projects that were not possible to build in that time before, have mentioned that this is the first thing they concentrated on that actually unlocks the — right. So, if your specific web app, for example, does not already have a — then you're going to have a much longer time in an unpaid path if you're using prompts, you know, compared to actually.

00:19:52 — Inside the Snack Verifier CLI & Live Demo

So for this workshop, we made a little tool. We've been running it a couple of times, but it's called Snack. You can just run it with npm snap in that repo. It's not meant to be like a reusable tool, but just kind of like a template for how you could set up this, specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application.

And so, snack can do a couple of things. It can manage the health of the application. It can bring it up. It can take it down. They could run a doctor command to see is it actually you know is the database running does it have network connection, etc. That quick drive or that quick check that it can do. Then it has drives. You can tell it to join. Remember, this is our midnight snack waitlist. So it can join the waitlist. It can reload the page. It can reset all of the data, so it can't find it.

You can have it look at things. So take a snapshot. Look at the network storage. Look at console logs. Look at local storage. Screenshotted. Run evals, all of the above, and then finally improve it in two ways. It can actually run a prove command where it's checking for deterministic output of that, and a judge, which is using a system one model to, in — but give an idea of whether we're going in the right direction, without necessarily actually. Just to fully develop this idea, when you do something like this in the repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right? Is for everyone else on your team that then comes along and is also able issues significantly more, significantly faster, significantly safer, right? More safely, so that you know everybody on the team actually benefits, and I think that's like where we keep harping on this being like a foundational concept now.

And if you think about it, like that's a really big piece of — it might even be more important than your software, actual application is the way that you verify it, the way that you test it, the way that you prove that it works, because that is what you can use potentially, like a lot of people are doing, and converting. You can take your app, like the Bun team did, converted to Rust. Go TypeScript converted to Go, not necessarily with the same level of LLM help, but they did use LLMs to help with that, and then I think Theo just last week converted it to Rust. So when you want that, if you have all those verifiers in place, then your software doesn't matter as much. You can write it again and prove that it works.

And so these tools give your agents their eyes, or if you think about it, like the tools that your agent uses that gives it the body, the LLM is the brain, and then the way that it reaches out and interacts physically, virtually on your computer is through these tools, and so things like joy — this is like actually having it run and do things, and then it can check. It can see, you know, did I did I that actually succeed in some way? Check storage, run reload, snapshot, etc.

And so let's just run a couple of these and see. That brought up our browser. And then I'll open up another browser or another terminal and run p snack snapshot. And that gives us a snapshot of the page, and as Jason that we can see if I put Zach at Zack, at Zack, and add that to the page, hopefully, we'll see some Zacks in there. There we go. So that's a way that we can verify. We can also do screenshot to make it a little easier. That should pop up a screenshot.

So that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us, that's the end goal, right? It's proving back to the human, being able to screenshot, record videos, put those on the pull request. You can do that with the GH CLI now. From debugging tools, where is the actual bottleneck? Don't guess, you know, spin your wheels for three hours, actually, figure out what was causing the errors. Yeah. So the real key here is that there's no — it's not just code that you can look at to interpret whether this application is running correctly, but you can build tools that will help you prove to you that it's working correctly without you having to read the code at all because this is all we care about. Is this what the user sees? Is it correctly rendering in this browser at this width with dark mode with light mode, etc.? You can run infinite tests on that and run them over and over and over in a loop, which, as Zack said, unlocks your whole team to move faster because you've given the agent the ability to run those checks itself as well.

So if we run this, if we run just like this, where we have the doctor, and then we add Ada at example, test and then we check the storage. It's there. We run. If we check the storage, it's not there, we would see local storage is empty. And if we run reload and then snapshot, we would verify that it is in fact so. This is a way that the tool itself could run in a loop, could check all of these things, and yes, confirmed before it even gets back to us. Oh, I see that we're not actually persisting this. I forgot to do that. And the agent can just move on from that. Real quick, we wanted to stop and take a few questions, if there are any.

00:25:36 — Mid-Workshop Q&A & Making State Feedback-Loopable

Yes, sir. So it noises.

Great question. So the question is: are we using models to write JSON to then like help us to make our code deterministic, to bring that back? I'll use unit tests around to verify different things, catch edge cases, and then verify that the code that is like a real interpretation, a real representation of what I'm expecting to get back from this single model, and use that, that I'm on the right track, but then have keep those as well to kind of like have another check in order to make sure that it doesn't drifted over time, or things change.

Any other questions? Yes, sir. Is it frontend only or agnostic, the general process that we're talking about? Yeah. So what I'm talking about, at least on our teams, we build a ton of applications that are internal tooling. We have some customer-facing applications as well. They tend to be text, and they tend to be quite fun, or some other kind of cool setup that works perfectly. Even have evals that are focused on the backend or database operations, you can get a screen there as well with it, but it's absolutely yeah works well, and I expect that the way that we're all going to continue chasing software features is going to be kind of like the hardware is making sure that they're.

Yeah, that's why we're not like giving out a tool because we couldn't give you a tool at this time that works generically for everything that everyone in this room is working on. You don't have to build up the tools that you're working on, and then give that to the models, and just make sure that they know how to run them well. How's it? So, honestly, like your emails, what you actually need or care about is team or company or business are going to wildly reference. You know, if you're in like the interior design space versus some other place where speed is the only thing that matters, I think you're gonna have like wildly different evals, and there's a value to building those up with your team.

And there's lots of different ways to get the data to the like to the model in different ways, so that it can interpret it, and that's like a huge piece of like keeping the model in the loop as to what's going on. The AMP team has a really great post on it. I think it's called — but I'll get it up there. I think it's a blueberry. I can't remember. But what they in the example they show like a canvas with a circle, and a ball bouncing inside of this circle, and then like there's a little slit where if the ball, if it hits just right, it can fall out of the circle, and then the game, the game is over. And the whole idea of that is making things feedback loopable. That's what it is.

Where you know the model can't actually see the running candidates and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where you know once it gets to this specific point, it jumps to the other side of the screen. Well, if it's moving more than one pixel at a time, like per loop, something is wrong potentially, and so there's a bug in there. If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball, not be well, like the physics. So, and also, once you do that, you are also not free to spin up thirty different variants of software that are used to build, and then take the — so that's the evolution of our model. And especially if you expect agents to models improve the tokens, the LLMs potentially, then you can imagine that they're all going to be coming in on Monday and saying like I would like to see 35 versions of this new feature.

00:30:02 — Playwright Proof Commands, Self-Healing & Explicit Errors

And so, how do we actually run or create that? Well, in this case, like we said, we can't really prove it for everything, like universally. It really has to be something that's unique to your application, and so for our Midnight's Night page, this is what the proof the proof command might look like, where it's basically using a tool like playwright to check that a an email was actually saved when it gets on the waitlist, so we can check that it doesn't have any on the waitlist, any emails. Then we can fill in an email on the next line. And then we can get the button and click it, the join waitlist button. And then we can expect that the status element on the page has a waitlisted in there, and then check the intertext of that to see if our email address is there.

And so that's one simple way to do it. Very easy if you've ever written an end-to-end test with like Playwright or Reacts, testing library, any of those. He's definitely used tools like this. You can just give them back to the model and to your agent, and they can run them. Give it to them as a tool, and they can prove it out themselves. And you've made this reproducible bit of evidence that then you can use in this one. Next time you use it, next time that page changes, they can run it again and verify that nothing broke. It acts as a regression and he's everyone on the team moving forward.

One caution, sort of, professional, like a statistical screenshot isn't proof. Make sure that it's actually showing the state that you expected to be in, that it actually like ran the commands to get to that state, to actually be in there because otherwise it might just be like another Y, but you're seeing a screenshot of the PR and thinking, oh yeah, this is this is correct, but it might be causing different side effects, that might have taken a different path that's not actually possible within the UI. So you want to make sure that it actually like runs start to finish to check that one of the.

Yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get, or the benefit, the main benefit is that a failure starts to drive future work, right? So if in one loop it fails, you're busy talking to somebody at work, or you're starting another task, that agent can determine on its own, this is not quite correct. Let me go take another stab at it. Let me now run the command. Verify again. Okay, now we're good. If you think about what that means at scale, a single PR or a single lane of work, if you will, an agent you know self-feels, self-corrects, figures it out on its own. You're not coming back to that particular PR at the end of the day and being like, I still think this all the way through. And if you just multiply that out, right, that that's really where, again, we just say this is foundational for speed.

And how many of you write good error messages? Yeah. Sir, what's your message? System error nine. One you should. I think they're more important than ever because when your app throws an error, guess we can interpret it and run with that error really well. So if you make it so that your failure is explicit about why it failed, and give as much detail as possible about how it got into that state, then your AI can just self heal from that, so it's very easy, very worthwhile to do that.

00:33:40 — Outcome-Driven Constraints & Live Persistence Debugging

Yeah, and so this is a concept that Nick and I talk about, right? It sounds like we're being, in some ways, we're trying to wrap determinism around LLMs, but we actually don't want to do that, and it's also something to the spirit of what the AMS team is talking about. The agents are intelligent. They're capable. We want them to be creative, actually, because oftentimes if they have the right constraints, and then oftentimes constraints can be enable freedom, right in a way, then they will come up with creative solutions and actually solve what you said.

That's why you give them the outcome. That's why you know Nick's project he's experimenting with is outcome-driven development. And so it's not about being overly prescriptive in how we achieve things, that can actually lead to lower performances in cases. It's more about giving it the map, saying you must get here. You'll know you're here when this command returns to us. You don't necessarily have to describe how it gets there, though, and this is true, more and more true every single day with like the frontier people, Tableau, After, Saul 6.1, OpenAI 5.5. They're all really good at being like being creative is like the way I describe that, but if you give them, this is where I want to get to, and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases. They might figure something out, or ask you a question that unlocks something. But if you're just like, you have to get here, and then here, and then here, they may not sway from you, and you want them to sway a little bit, but get to the right, right end goal. And like I said, this is something that we've only come around to really appreciating the latest batch of frontier where it's really becoming possible. So going forward, it's going to just get better.

So when it's going, you want to get into like you know the typical TDD like red green refactor loop with your — you've heard about this, you know, at every one of these AIE events, and it's no different here, but you can use that within the verification piece itself. Verify this, reproduce it, show me, fix it, and then show me the proof of that. And how are you going to show me the proof — just as important as the fix that's going to go in, because that's what's going to leave you alone and allow you to trust the model to actually do the work, and then when it says, hey, I'm done, we know that it's done because it's proved it to you in some.

So in this example, you can run Snak Doctor. Let's just go ahead and run that real quick. And so this ran, and it's showing us a bunch of checks that it did. So it's checking that we have Node, that Chromium is installed, that our session exists, it's actually running, and that it successfully built. So that's like what we're running as a click doctor just a bunch of different. Sorry, we don't take. Okay, give us the result back. Snack storage. Storage is empty. It didn't actually put data in there.

So this is the point at which, in the real world, your agent you've just gotten the value of this entire system, because at this moment, your user realizes I am done. I'm not going to drop this one before. And if you reload it, do the same thing. We have now given our app eyes. They can see this. They can run these. They can — it can make sense and change the world of the application, the Midnight Snack application, and see the results of that, and then be able to interpret what happened from there. So you can try that. You can if we like apply the persistence fix to it and add honest copy to it, then we can improve. Let's run that. Inflight PRs against people as not kicked on the browser, and I didn't actually go into — if we have, or run the fix. If we have run the fix in which together and that's.

00:38:07 — Saving Run Receipts & System 1 Judge Models

Okay. One of the things that we're thinking about a lot lately, and talking about, starting to roll out in more applications is this like concept of receipts. You know, you take like a ticket, like a public ticket system, but there's a certain performance, the way that a human would answer to get to consider it done well as an excellent customer service of their service, and then what does the preserving that in some way, for example, right?

Why do we want to do that? There's a ton of reasons, and we'll get into some of them, but again, if you think about if we all work — seems that in some way, many folks are heading towards you know running RSI harnesses and self-evolving codebases. Everything that we do today, everything that Raytheon does today, every run, all day long, that is very valuable data. It's very valuable data right now, because you can feed it into and get fixes. I think it's incredibly valuable data in the future, as tooling improves, it gets easier and easier to set up RSI loops because then you've got a dataset of exactly how your system performed, that you can create, you know, very comprehensive middleware and have agents which means loops and see their performance, and then continue going until they get better and better and better, right? At that point, they're able to pass, and they're using that evidence from the previous rounds.

So now, if we fix that code, it now persists, but it can also lock. So after we reload the page, it could say waitlisted, but with the screenshot, I might actually say is you're in, your seat is confirmed. Those don't exactly match up, and so that might be like a failure right there because they don't exactly match. So this is kind of getting away from the static testing piece of it. Like, how do we prove the it worked without having to be like very tightly coupled to the actual text screen for a very contrived example? And so if we just run like P D snack, should run that and it failed, because even though it worked, and it says that we're in, Jeb did not. The model did not see that. And it's because the confirmation text breaks the contract. It's not exactly one to one. So how would we fix that without adding a lot of tight coupling to everything that we're doing? The answer is the System One models, which we'll get into here.

Yeah, absolutely. So, just to introduce it for anyone that hasn't heard of it before, the concept of System One, System two comes from psychology, actually. It's like a useful model to think about how the human brain works and how he thinks. So system one is super fast decisions. It's you know autonomous like someone swerved at me while I'm driving and I just corrected without even thinking about it kind of thing. Right. So I was about to lose my balance, and I corrected it. And then system two is like I need to figure out what I'm going to do with my life. And so, you know, applied to the world of LLMs, now we've got Jeb, and then immediately after we've got Clev from Cloudflare, that's even faster, and so there's — it's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're going to continue to see in architectures.

Yeah. To put it another way, that light right over there, I said, what color is that projecting? Quickly, you would say yellow. I said, "This is one thing, right? You just you know the color." I said, "Why is it making yellow?" Now we're thinking. You know, we could go down like a physics of why the light emits yellow. That's how we got to really what we want to talk to you about today.

00:41:54 — Saboteur vs. Detective: Testing the Verifier Itself

So in this, we've got another game that we can play within this, it's a quick game with the snack. We have saboteur and a detective, and so in this, the saboteur will pick a snack. It'll put a secret bug in there. Randomly putting a bug in there, we don't know which. It'll just be random, and then you, the detective, find it with Snakelab, and then prove what that is. Name the bug and give proof that you found it. So this is something that you can do. You pull down the website or pull down the repo, you can just run PBM snack up dash bug random, and give it, let it run, and see if you can find it. So let's try that. We'll just give it some time to go.

Okay. So in this, it ran. It's got the API URL. It's running. It's not headless, so we can see it. The bug says hidden until you stop up. Yeah, right. Leave this running in another terminal, or ask him to snack, doctor. Thank you. So that ran the whole thing, and we can see, you know, it's there, Chrome's there, it's got a session. The proof here — the last receipt says fail, but the source is now — that proof is for code that no longer exists, so it wasn't able to prove that. That was a previous run. Try it again. Same thing. Can anyone tell me what the bug is? Any ideas? One email. Yeah, exactly. We're from, I think it was running at six two one seven four. No, that was right. For that one, yeah, the tools make it optional, though, so you can run through all the — if they don't have it, don't just go. But it's also a good opportunity to learn about one of their pretty cheap calls.

They give you five dollars for free when you sign up, and then you get a lot to like.

All right, so yeah, so yeah, we ran it a couple times. I think there was some problem with my output. I'll debug that. That's funny. I need some verification tests around my try to verify examples. But the, you know the outcomes that could come up are that it names the bug without the, said it works, but it didn't, said it works, but it did, blamed a bug that wasn't there, and then had no — it was the right bug, or no evidence.

Right, and this is just getting back to that point of a private example, of course, but your team might score differently or have different criteria they're looking at. This is super flexible. The most important thing to do is to start and create one, create an evaluation to use it consistently and to evolve it together as a team. And it gets to the point where your changes, your PRs, automatically reflects that email so that you can get to the point where you know if changes are safe to land, or if they're improving things, or if they're making things worse, or if they're not changing at all.

And every time it runs and it misses, that's a new line that can go in the map. So it can give us ideas of like the steps that we're getting there, why we're getting there, and kind of like every time it fails, it's getting us one step closer because we can fix that. So the first time is memory only. It's not scored anywhere, but in memory, we're putting in a long email address. We've lost our confirmation. Every time we improve, so we've caught that, and we don't run into that again. But then the overpromise one, the screenshot or the judge running can tell us that it actually did not go into the bank. Same thing with console error. There might not be anything in the console, but it's in the receipt, or might not be getting caught before it gets to the console, or it could just be that there's nothing defined. Everything works, and that's a real concern as well.

Yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous examples, showing proof that no longer exists. That's because your code might have changed quite a bit between yesterday and today. It might be different circumstances, like a famous data, for example. It might be a number of different things. So we want to build the tools in such a way that they're flexible enough to not get caught by issues like this. This is all kind of like go back to basic human testing examples, but bringing that kind of like deterministic check to the models in a way that they can self heal and self improve.

00:48:23 — Verifier CLIs, Overnight Receipt Loops & Final Q&A

Okay, so this again to tie back to what we used to do as Cerebras developers, we used to be able to see how we would function. That's really what we've been talking about, and moving it into code. So by moving it into code, making it if available in the terms of OpenAI, then we can get to the same path and the same experience that we would flex as developers in Rome trying to fix something lately. And that's really what unlocks the speed and the evolution and the ability to scale up the amount of code that you're delivering, and to do that in a more seamless way.

Again, you know, if you the recommendation is that you have a web app, and you're not already you don't already have a custom verifier CLI, that's one of the highest leverage things you can start with for your codebase, foundational eval as well, shows what code looks like in your application, and then also giving your agents the ability to see all aspects of performance and networking, and traffic, and rendering, etc. So, getting at Chrome DevTools if you're in the web app land. If you're shipping, maybe you're planning to shipping iOS, you know, or OSX apps, then there's an alternative projects and more coming out every day that you're then going to attach to there, but the common foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you see as a developer.

Yeah, the more that you can give it, the more arms, more eyes that you can give it, the more it's going to be able to take over from you. And then you just have to get the intuition about when to use those, how to use those. Think about the way that you decide: okay, if I this bug, how would I reproduce this? Think about that, and if you can break it down, and this is dependent on your application and the type of code that you're writing, you can you might be able to do that in a way where it can like think oh, for us, it's a lot of sign-off stuff. Is it properly setting up? This is applicable to every you know field in the industry that you've got software for, but it's really just all about like hey the world where it makes sense for us all to be ambitious and good all the time, at least evolving, or at least changing rapidly.

What are the foundational — like, what's the what are the same system fundamentals that we bring in this now so that we can accelerate some of that?

So yeah, doing expensive call stacks, but the important thing is that getting that output, making sure that it's giving you detailed output about what happened, the state of it, all of that, so that it can be audited by the application, by the model, and then you can run it again and fix it. Run it again and verify that those changes actually took hold, and that there's no longer.

Yeah, and again, can't overemphasize the value of keeping receipts. So if you have runs, you know, it can be the entire user journey. Let's say it's an e-commerce app. Like when they log in, they hit the page. What happened? What was their flow? Did they see an error at all? Did they see seven seconds of time to see your catalog, you know, come into view? Were they able to put something in their cart? Was their cart abandoned? And so, sure to save that all now because every single one of those runs and receipts can be fed two LLMs, or you can even set up a system that's running all night long and is looking for the latest receipts being produced, and is aggregating them, saying, "Well, today we saw eight failures in the cache. So I'm obviously going to you know issue a PR tonight to fix that properly, and then oh I have tools I need to prove that this PR that the issue I was trying to solve has been fixed."

And then the recommendations that we're seeing from folks too that are achieving like super high scale PR shipping without errors, they're also able to you know add the stuff that is for us as humans in our oversight, still, so not just screenshots of the feature, but like a recording of I walk through the app, this was my flow. This is what I experienced. These were the states that I got through, so that if I'm waking up, I'm trying to rip through, you know 15 important PRs, the first thing in the morning, I can first start watching that and make sure that okay it actually got the end word and then you know start looking at code. Is it something that I'm actually going to like human review and — again, all of that I think is going to be immensely valuable as we continue to see evolve, self-evolving. And you know, think about what that means, though, because when we get to the point where there's self-pauling pool and it's just like the people are using it, there will be a specific advantage for folks that have preserved clean data sets of things that they can say, this is what we are trying to get to in this application. We need to get all the way there and rapidly improve it, right?

If you're not sitting on a mountain of that, and your competitors aren't, they have some money, they're going to be able to go significantly faster.

And this is just building up that trust, so that we can continue to improve upon our as we go, but we have these checks for everything that came before. So we've got Unity. We've got Elon. We've got Smart. So we've got whatever our application needs, and we've got a way for the model to be able to interpret and run those and improve them in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, no regressions, and we can just mount and climb that evidence ladder, building evidence as we go, and ensure that our app is working exactly as it — that really means the spec shows what.

Okay. Cool. So we've got just over five minutes. I'm going to pause and see if there's additional questions. Yes, sir.

I, I personally, I'm curious to hear what you think, but I would personally almost lean away from — what my experience is, it's a model that's not going to actually call that happening, versus like having the C I A going for us, these commands that run as well. But the answer to the question is how do you run these checks effectively?

Yeah, you can you can run those you can run these regularly. You can have them be like part of, for example, you can have them be various folks that could be like again, all those platforms and before. You could have it be like a cloud codebook. It depends on what you're using and how they're doing it. I might build like a full high attention to things learned about, and just kind of like feed in information. There's a number of different ways can do that, but the easiest way, by far, to get everyone across the board, and I can say this now, just put an agency in your app, and they have it. They like you know, on changes, buy things by reference, so like everything changes, significant things. So before you open a PR and do this, like whatever wording that you want to put in there, and then that's an easy way to get everyone across the board. And they really, I have to go out of my way.

Books are also relevant too, as you're doing the development, right, for the whole team, sharing across the world.

Yeah, absolutely. Well, CI is a, I think, a feature of it, but it's not like — the actual tool is like the rapid development that you get. So I give my agent the task to do, and I can just quickly run, like let the agent go, you know, basically, tell it, this is where I want to end up: I want a weightless, I want to be able to add other emails to the list, and then it gets persisted and saved, and I can take advantage of it. So this is how you verify: goes into this table, like whatever.