Your job is now "agent manager"

Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): Untangle (2 vs 0), Compound AI (1 vs 0). Check them against the audio before quoting.

00:00:07 — Builder Agents, Three-Review Loops, and Child QA

And then the questions go back. And probably a lot of you are doing this. I'm not claiming this is rocket science, but this is what we're doing right now in prod, and it's working, right. Each of these builder agents has a review agent that it calls upon that has built in reviews. Probably every Compound AI you have has built-in reviewers. We typically stick to a three-review max, where we allow the builders to loop about three times with reviews. If they still have any important bugs that have been fixed after three, it then bubbles it back up to the manager agent, and if the manager agent needs to, it bubbles it back up to me, right? That seems to work pretty well.

Okay, so what's happening under the code under the video link? All the wrong stuff. You've got code plus tasks. You've got a QA agent. This is actually an additional process that we run that actually spins up a child of the builder to run QA. Y'all, this is cheap. Like, you should be spinning up as many agents as you can, as long as you're not holding up your merge queue, right? So I — this works well for us now. A wonderful engineer named Dale recently joined. He helped install this QA process. I think it's working pretty well so far. The thing that you're all going to encounter is just merge huge slowdown. There's this fine balance of just, you know, packing your PRs to death versus just getting them merged.

00:01:40 — Mandatory Feature Videos for Human-in-the-Loop Review

The other thing I love, love, love, love is videos. So Dev has this built in. If not, you should build this. It's hard. Every builder records a video of the feature being used and passing tests. So if it's a un facing future, you've got to look at, right? This is how we know that what we built looks good, how it works, right? So in Dev, it's a built-in feature called Test. It basically builds test cases and then it records a video. What we think is happening is happening, and visually it looks good, right? This is the human in the loop where a human has to say "should." So the manager session will let me know there's a video ready for you. Watch it, right? So I watched the video.

And this is the hard part. You've got to zoom in and understand what is this PR doing? What am I watching? So I usually read a summary. What is this thing? What is it supposed to do? I watch it and I get engaged, and I would encourage you. This is the part where you want to cut corners. You gotta watch the video. Like there is no — this is the part of the bottleneck you have to engage in as a product owner. Gotta watch the video, gotta pay attention. Usually, has to be on your laptop. I — this literally before this talk, I was on my phone, you know, talking to my manager, my mantrage, and it said, "There's a video for you." I watched it. I just couldn't see it long enough. Sometimes I'd be like, "It's okay." Get back to my machine. Watch the watch the video.

This is working. You know, we ship a lot of PRs at high quality this way. We've iterated, you know, hundreds of times on this process, and I feel pretty good about this. Now this will change tomorrow, and probably will be useless a week from now. But it works well today.

00:03:38 — Multi-Session Agent Messaging and the Merge Manager

The thing I will say is like if you're coding, stop. Like you really need to be coding. I don't care whatever it is, but if you're running local dev, you are screwing up. You're holding yourself back. You're holding your team back. You're holding everything back, because you can't do this. So that's where we're going. I feel strongly about that.

So how does this work at scale? Well, you know, say you have ten folks in the who are working on things, typically they'll have an engineering manager, seven each. Sometimes they'll have to — like right now, I'm running. And what's beautiful is that these things talk to each other. And so, again, this is built in, probably built into whatever harness you use. The sessions all need to be able to talk to each other. They all need to know who each other is, right? And they're really good at this. Like this is such a delight.

So what you'll see is if you if you zoom in to the session, they're messaging each other. It's clear. It's straightforward. Fast is good. Like you should be allowing your sessions. And again, if you're not, you are holding yourself back, and you probably should be in more software. So they all talk a lot, and it's not chit chat or stupid stuff. Like this is important stuff that they need to know from blockchain, which is very useful. However, you still need what we call emergent human. So this is how we coordinate this just wave of PRs that come in. You know you have things like database migration numbers. These need to be coordinated. Like if all the agents are trying to grab these things, you constantly collide, right? So the merge manager just helps all that go away, but also is a little more intelligent about how the merge deals happen. So this is really nice.

It's working well for us. And again, these are just sessions. This is not like something special. These are just sessions in Devon, and I name them so I understand what's happening, and it's pretty straightforward. You know the talk from the gentleman from OpenAI, you know, said basically, "Unhobble, just unhobble the thing, right?" So set up the roles and then unhobble the model.

Right. Who's tracking so far? Good. Who thinks this is bullshit? Raise your hand. Oh, one guy, two, and one. Come on, be more brave. I'm about to talk after. Okay, and then we use GitHub reviews.

00:06:23 — Eliminating Hiring Bias with Agent Screen Recordings

All right. Next up, how we hire. So who's hiring engineers or employees right now? Yeah, he's already. I think everything has changed. Everything you used to do about hiring employees is over. None of it makes sense. It's all outdated, and I would encourage you to do something like this.

So, we don't do, number one, any sort of pre screens. We don't do any video chats. We don't do anything with people for most of this process, because it will bias you. You'll like their hobbies. You'll like the way they talk to you. You'll think they're interesting, and it's all noise. None of it is real, right? All that matters is can they operate an agent?

Right? Can they do that? So the way we do that is we ask candidates to submit a 45-ish minute video, full screen recording, and we ask them to ship a feature PR typically on a app that they already maintain, and we just we just ask them to report that, no instruction, no guidance, no nothing. We just want to see like how good of an agent operator is this person, right? So that is free. We don't pay for that. That's basically how you get the board. And what we then do is we feed these videos into a ranking engine and base safe: watch the video, watch the transcript, stack rank these people.

Now, this — what's interesting is I did this, and I didn't trust it the first time. I didn't like what it said. And what's ironic is it actually was correct. Like the person that was number one, I shouldn't hire, but I didn't trust it because I was — I was biased. So you stack rank them. Now this is just a lot of video to watch. You could watch all these videos if you want, if you have time, do it. But I think you'll find actually the very intelligent models are pretty good at understanding like how both sometimes, but they won't think out loud, which is kind of fascinating. So you can't really tell what they're.

00:08:39 — Sixteen-Hour Paid Trials and Untangle's Mission

Then we basically pick the top five, and we invite them to a paid project. So we pay a fixed fee for sixteen hours at a time. We actually give them a seat on that, and we ask them to, to get a real PR ready for launch. Sign an NDA. And we recorded all the video. This is sixteen hours of video. We basically use them, and then we feed all the videos into our agent basically does, and we say, "Hey, you know, which one should we pick?"

I actually watched all 16 hours of these videos for the last round, and it revealed that it did give me a good read on people, and then you basically pick one. You pick your finalists, and then what's important is you don't do calls before this. You will get biased. You'll get confused. You'll try to pick somebody you should, and I would just call the finalists and say, "Hey." You know these things that are at the top, and that you use the files for human reading, but you're going to have to use these people for hours. Like you're not really going to need that at all. I think this is how it moves back. Like this works. You know, I hired hundreds and hundreds and hundreds of people. This is just way easier, way better.

So, I'm actually going to finish three minutes early. If you want to know more about Untangle, go to Untangle us. I'm on X like just all the time. I've literally been on X for twenty years. And so where the — so, if you want to know more about residiing companies, managing agents, that's where I am. If you're interested in family law, if you're passionate about making divorce less painful.

Yeah, I mean that's kind of what we — like this is free money because.