YouTubeFeed

Evaling Video Slop — Maor Bril, Character.ai

Summary

Maor Bril, who’s spent two-plus years at Character.ai, opens with the gap at the center of AI video: generation got dramatically better (Kling, SeaDance, Veo, Sora), but evaluation didn’t keep up. Frame-level metrics like CLIPscore and LPIPS check whether a single frame matches a prompt or whether adjacent frames drift, but they can’t tell you whether a video actually told the story it set out to tell — whether the physics holds, whether a character stays consistent across shots, whether the pacing feels right, whether the sound is synced to the action. Throwing a frontier LLM at the whole clip as a judge works but is slow, expensive, and wildly prompt-sensitive, which makes it unusable at the volume and latency Character.ai’s generation product needs. Bril’s team’s answer was to distill a “committee of experts” plus human-calibrated LLM-as-judge into one small, fast Vision-Language Model that can score — and explain — why a 15-second clip is slop in about three seconds, cheap enough to sit inside the generation loop instead of running as a Friday afternoon spot-check.

The talk’s sharpest lesson is a failure story. Version one of the judge model, trained on 1-10 absolute scores, was “wrong in a very confident way” — it gave a 9.2 for cinematography to a clip where the camera hadn’t moved for four seconds, and called ghost-hovering physics “great.” The model had learned to detect gloss and coherence — the vibe — rather than the specific axes (story, physics, pacing, sound sync) it was supposed to be scoring, because the training data conflated “good” with “polished.” The fix had two parts: switch from absolute 1-10 scoring to pairwise preference (is video B a better story than video A?), because people agree far more consistently on relative comparisons than on absolute numbers; and rebuild the training data by pairing real footage against AI footage on matched encoding and identical annotation methodology — deliberately avoiding the trap of accidentally training an “AI detector” instead of a genuine quality detector.

Bril closes with three practical takeaways: score relative, not absolute; score the specific axes you actually care about instead of expecting them to emerge on their own; and put the eval as close to the generation loop as possible, since catching a drifted starting frame or a bad six-second segment early is far cheaper than catching it after a multi-minute video has already been assembled. In Q&A he’s candid about limits — lip-syncing remains unsolved at Character.ai — and about the engineering trade-offs behind choosing a small Qwen-based VLM judge over frontier models: speed and unit economics matter enormously at Character.ai’s volume, even if a larger model would score somewhat better in isolation.

Highlights

”Don’t score — compare”

Pairwise preference beats absolute scoring

“The other very interesting realization we came to is don’t score, compare. What does that mean? For example, if I’ll ask any person in this room to look at a particular video and rank it from 1 to 10 on storytelling… I’m pretty sure that what will be a six for you will be a five for you, will be a four for you, and an eight for you. But if I’ll show you two videos and I’ll ask you which one of them is telling a better story, the grand majority will probably agree that B is telling a better story than A.” — Maor Bril, 9:17

Clip command
yt-dlp --download-sections "*9:17-9:54" "https://www.youtube.com/watch?v=b_PmGocP4rc" --force-keyframes-at-cuts --merge-output-format mp4 -o "dont-score-compare.mp4"

A 9.2 for cinematography — on a frozen frame

Wrong, and confidently so

“We shipped V1 and it was so wrong. It was wrong but it was wrong in a very confident way. For example, the frame you see here it’s from a video that the model scored 9.2 on the camera work and the camera didn’t move. For four seconds it was like a still image of the same character, but the model was very, very happy with the cinematography.” — Maor Bril, 10:13

Clip command
yt-dlp --download-sections "*10:13-10:54" "https://www.youtube.com/watch?v=b_PmGocP4rc" --force-keyframes-at-cuts --merge-output-format mp4 -o "confidently-wrong-judge.mp4"

Distilling a committee of experts into a 3-second VLM judge

Small, fast, and able to explain why

“The solution is actually to take all these committee of experts and distill it into one small model that is also very, very fast, but is able to give us a response that is not whether or not this video is slop or not, but why is it slop… it takes about three seconds to score a fifteen-second video.” — Maor Bril, 7:53

Clip command
yt-dlp --download-sections "*7:53-8:59" "https://www.youtube.com/watch?v=b_PmGocP4rc" --force-keyframes-at-cuts --merge-output-format mp4 -o "distill-committee-into-vlm.mp4"

Avoiding an accidental AI detector

Pairing real vs. AI footage without overfitting

“The risk with that, and that’s the reason why I avoided doing it at first, is because I didn’t want to create an AI detector, right? Because if you start creating pairs of good as human-generated video and bad as AI video, then there’s a very big chance of the model overfitting and becoming an AI detector as opposed to a video quality detector.” — Maor Bril, 12:09

Clip command
yt-dlp --download-sections "*12:09-13:06" "https://www.youtube.com/watch?v=b_PmGocP4rc" --force-keyframes-at-cuts --merge-output-format mp4 -o "avoiding-ai-detector.mp4"

The three takeaways

Relative, specific axes, and inside the loop

“Go relative, not absolute… score the real axes that you care about. So if you care about storytelling, if you care about pacing, if you care about physics, score those axes, don’t expect them to magically appear. And put eval inside the generation loop… Videos are stories. Videos are just another way for us to tell stories to others.” — Maor Bril, 13:57

Clip command
yt-dlp --download-sections "*13:57-15:04" "https://www.youtube.com/watch?v=b_PmGocP4rc" --force-keyframes-at-cuts --merge-output-format mp4 -o "three-takeaways.mp4"

Key Points

  • Two parallel tracks in video AI (0:12) - Generation quality raced ahead (Kling, SeaDance, Veo, Sora); evaluation quality got left behind.
  • Generation is basically free now (1:15) - Cheap compared to studio production costs, but most generated video still isn’t good enough.
  • Common hallucinations (1:39) - Third limbs, doors opening and closing simultaneously, hovering physics.
  • Existing metrics only watch individual frames (2:22) - CLIPscore judges single frames well; LPIPS detects frame-to-frame drift; neither captures the whole clip.
  • What frame-level metrics miss (3:00) - Whether the video tells its intended story, whether physics/character/pacing hold, and whether audio is synced to action.
  • LLM-as-judge, first pass (4:01) - Foundational models can score whole clips but are slow, prompt-sensitive, and inconsistent across annotators.
  • First iteration: a repeatable benchmark harness (4:41) - Combines frame-level metrics with a human-calibrated LLM-as-judge; human annotation feeds back into the judge prompt.
  • The problem: too slow, too expensive (5:29) - Needs to run close to users and early in the process — catching mistakes earlier is much cheaper to fix.
  • Catching character drift at the starting frame (6:00) - Comparing the two starting frames of consecutive shots catches drift before it propagates into a full generation.
  • Long-form video is built from short clips (6:32) - Catching and regenerating a single drifted 6-second segment beats fixing the assembled whole afterward.
  • Some axes only exist across time (7:00) - Story coherence and pacing can’t be judged from a single frame; they require watching the clip through.
  • Distilling into a small, fast VLM (7:53) - ~3 seconds to score a 15-second clip; explains why a video scored low (extra limb, broken physics, audio desync), not just a number.
  • Small model beat the bigger one on the merits (8:59) - A bigger model scored better but was significantly slower; the speed/quality trade-off favored the smaller model.
  • Manufacturing “badness” (10:00) - Good videos are abundant online; bad training examples were made by corrupting good videos or generating deliberate slop.
  • V1 scored vibe, not the axes (10:54) - The model learned to detect gloss/coherence rather than the specific things it was meant to evaluate.
  • The fix: pair real footage vs. AI footage (12:00) - Rebuilt the dataset around real-vs-AI pairs, with matched encoding and identical annotation method on both sides to avoid overfitting into an AI detector.
  • From pipeline to agentic workflow (13:06) - Users bring distinct stories/characters/voices that break rigid pipelines; giving the agent tools to validate and self-correct adapts better.
  • The three takeaways (13:57) - Go relative not absolute; score the specific axes you care about; put eval inside the generation loop.
  • Sound evaluation (15:27) - Uses Dolby Atmos for sound quality plus keyframe correlation — matching prompt-described events (e.g., “door slammed”) to sound spikes at the same timestamp.
  • Lip-syncing remains unsolved (16:23) - Especially hard for non-human talking-animation characters with no real mouth-to-speech correlation.
  • Calibrating human judges (18:27) - Periodic 10-15 minute team annotation sessions across randomized axes feed back into both judge calibration and training data for the next model version.
  • Why Qwen (19:47) - Chosen for being small enough and because the team already had good post-training experience with Qwen on other use cases.
  • Scale and unit economics (20:24) - The decision to build a custom model is a trade-off between training/curation cost and per-video serving cost at Character.ai’s volume — smaller teams may be fine with off-the-shelf metrics and frontier judges.
  • OTel telemetry is on the roadmap (22:17) - The open-source harness doesn’t yet export OTel traces for the LLM judges, but Bril accepted it as a feature request.

Mentions

Companies

  • Character.ai (0:12) - Maor Bril’s employer for two-plus years; builds the AgentX video-generation product this evaluation work supports.

Products & Technologies

  • Kling, SeaDance, Veo, Sora (0:24) - Video generation models cited as evidence generation quality has raced ahead of evaluation quality.
  • CLIPscore (2:22) - Frame-level metric good at judging whether a single frame matches its prompt.
  • LPIPS (2:22) - Metric used to detect drift between adjacent frames.
  • Qwen (small VLM) (19:47) - Base model for the distilled video-quality judge, chosen for size and prior post-training experience.
  • Dolby Atmos (15:27) - Used to ensure sound quality and help correlate audio events with video frames.
  • Judge Judy (harness) (18:27) - Character.ai’s open-source evaluation harness/repo where humans annotate generated reports.

People

  • Maor Bril (0:12) - Speaker; has worked at Character.ai for two-plus years on video evaluation.

Surprising Quotes

“It was wrong but it was wrong in a very confident way.” — Maor Bril, 10:13

“It scored the vibe as opposed to the axis.” — Maor Bril, 10:54

“Videos are stories. Videos are just another way for us to tell stories to others.” — Maor Bril, 13:57

“Lip-syncing is an unsolved problem yet. We’re trying though.” — Maor Bril, 16:23

“Most humans… have terrible taste anyway in videos and games and books.” — Maor Bril, 18:03

Transcript

Note: the interview-transcriber’s speaker diarization was seeded with a single speaker (“Maor Bril”). During the Q&A section starting around 15:04, audience questions are attributed to “Maor Bril” in the source JSON even though the content is clearly audience members asking questions, followed by Bril answering. Timestamps and text below are taken verbatim from the JSON; the speaker label for audience turns should be read as unreliable.

Maor Bril: 0:12 So hi, I’m Maor, I’ve been with Character for a bit over two years and we’ll talk about AI slop, right? I think that, you know, when we look at video generation as a whole, right, we have like two kind of parallel tracks.

Maor Bril: 0:24 One is the video generation which became insanely good from models like Kling and SeaDance and Veo and Sora, we still remember Sora. But the part that got left behind is how we evaluate the quality of the video that was generated.

Maor Bril: 0:53 Right, so on the one hand, we still kind of squint at it and decide whether or not it’s good, but on the other hand we know that the generation has gotten a lot better and when we look at X or whatever social you’re consuming your content on, there are a lot of guides on how to create amazing videos with this model or that.

Maor Bril: 1:15 So the hard part was never how to make video, the hard part was how do we generate good enough video and how do we judge if the video is good enough? So now we’ve gotten to a world where the generation of video is basically free, right? Free especially when you compare it to how much studios would charge.

Maor Bril: 1:39 And but the problem is the most, the grand majority of video that is generated is not that good, right? We have like a lot of hallucinations like a third limb, opening and closing the door at the same time, hovering physics, etc. So unfortunately, in order to get high-quality content, we need a human to judge.

Maor Bril: 1:59 And I don’t know when was the last time you’ve seen how someone is creating these long-form generated video, it’s usually a lot of shorter generations and a lot of editing. The problem is because we’re using a lot of the tools that we built for the text era, for the image era, for videos.

Maor Bril: 2:22 Right? We’re using things like Clipscore, which is great to judge a single frame. Things like LPIPS will help us kind of detect the drift between frames. But we don’t have, I mean but the problem is when you kind of combine all these together, all these tools, they’re good at watching the individual frames.

Maor Bril: 2:45 They’re good at checking this, this does this specific frame, does it match the prompt that generated it, right? It will check consistency between frames and it will check whether or not it match the prompt that drove it.

Maor Bril: 3:00 What it won’t do, uh, it doesn’t tell you if, um, uh, if you told the story that you meant to tell, right? If we think about what is video, video is a storytelling medium. Video is just another form on how we tell a story, right? For, um, any type of story. So one of the things we have to look at, does it tell the actual story? Does the physics make sense? Like for example, if we want a video of a character walking downstairs, does that actually walk or hover? Um, does the character stay the same character across multiple shots? Um, does the pacing make sense? Like, you know, for example, people take time, uh, going from one place to another. We need to make sure that the pacing makes sense as well. And especially when we add audio, we want to make sure that the audio is kind of synced with the imagery. Like for example, if someone is slamming a door, we want that sound of the door being slammed to be exactly when the door is actually being slammed.

Maor Bril: 4:01 Now the next iteration we all, uh, went to a while ago, we started using LLM as a judge for everything. And we have amazing foundational models that we just throw videos at them. The problem with them is that A, they’re slow, B, they’re only as good as your prompts, and multiple people will prompt multiple ways and the same model may respond in a very, very different way. And sometimes the prompt we use, like, is it consistent? Um, does this match the prompt? But then the question we really care about is it good? And the answer varies.

Maor Bril: 4:41 So, oops, sorry about that. So our first iteration is like, let’s take all these things and build a repeatable benchmark on how we test video that we can rerun over and over and over again. So that combines both metrics, as I said earlier, that knows how to view individual frames, but also, um, consistent LLM as a judge, right, where we also use human annotation to calibrate the LLM as a judge. So for every report that we generate with that harness, uh, we’re able to have humans annotate it and basically feed that feedback back into the the, um, the LLM as a judge prompt to make sure that it’s aligned with what I think or what the annotator thought is good, and we use it to score the videos.

Maor Bril: 5:29 The problem with this approach, it’s very slow, it’s very expensive, and especially when we want to bring it in for our users to be able to generate a lot of video because creation is a very hard process. And, um, so the problem as I said, the problem is when we… so this is a slow process and we needed to bring it as close to the users as possible and also earlier into the process. The reason for that is if we take a look at all the metrics and there’s mistakes that we can find earlier than, uh, later, then it’s a lot cheaper to correct that particular mistake.

Maor Bril: 6:00 Take a look at mistakes. So for, for example, right? On, on, on the left we, we have, a, two starting frames of different, different shots, right? But it’s easy to, to correct to, to view it at this point and see, did the character drift between frame 1 and, and frame 2, because those frames will be used as starting frames to generate videos. So if you can correct to catch the drift at this point and correct it, then, then it’s much cheaper to generate the video as a whole because we, we can correct it at a much, a cheaper cost.

Maor Bril: 6:32 And, and the same thing applies when we look at longer form video, right? When we see all these three, four, five minutes long videos, they’re usually a collection of a lot of shorter videos. And being able to catch a, a six-second generation that drifted and, and regenerate that, regenerate that before we combine the whole video will end up being a better result as a whole.

Maor Bril: 7:00 And, now the, the other problem we’re trying to solve is that some of these axes, right? Only exist across time, right? So for example, when we look at the, right? We, we mentioned the, story, right? So, does, does the story that we’re trying to tell with that video, does it hold in that video? Does the video tell the exact story? Does the pacing make sense? Right? And we mentioned the, the, the, sound.

Maor Bril: 7:34 So as I said, right? The, you know, the underlying goal is to bring that eval’s closer to, to, the, online generation because the sooner we’re able to catch those mistakes, we, the sooner we’re able to catch the drift, right? Then it’s, it’s much easier, much cheaper to fix.

Maor Bril: 7:53 Now, so, so the, now, now the problem is that as I said, this is a very slow process. So what, the solution is actually to, to take all these committee of experts and distill it into one small model that is also, very, very fast, but is able to give us, a response that is not whether or not this video is slop or not, but why is it slop, right? Why is, why is that video scored low versus the other because for example it added an extra limb, because it didn’t obey physics, because, um, um, the, audio was, out of sync. There so, so the goal was a. build it on top of a small VLM and why is it a VLM? VLM because, um, we needed the video the, the model to be able to see the image, but also we needed to work fast, right? Because we brought it closer to the, generation, where in fact it takes about, with the model we have trained, it takes about, um, three seconds to score a, a fifteen-second video.

Maor Bril: 8:59 Now the, we also, tested a, bigger model.

Maor Bril: 9:00 and the results were better but it was significantly slower and and um the decision was to go with the the smaller model because the the added value from the bigger model um didn’t just justify the the the slowness.

Maor Bril: 9:17 The other very interesting realization we came to is don’t score compare. What does that mean? For example, if I’ll ask any person in this room to look at a particular video and and rank it from 1 to 10 on storytelling, right?

Maor Bril: 9:30 I’m pretty sure that, you know, what will be a six for you will be a five for you will be a four for you and and an eight for you. Right?

Maor Bril: 9:39 but if I’ll show you two videos and I’ll ask you which one of them is telling a better story, the grand majority will probably agree that B is telling a better story than A. Right?

Maor Bril: 9:48 And if you do it enough times, then it easy to generalize the model at um towards detecting what’s better uh uh versus what’s not.

Maor Bril: 9:54 So we trained on pairs, right? Um um A versus B as opposed to 1 through 10.

Maor Bril: 10:00 Now, we manufactured badness. So luckily the internet is full of very high-quality videos and it’s very very easy to get good videos and it was very fun to create bad videos, A, by either corrupting good videos or by, you know, just generating random slop.

Maor Bril: 10:13 Um now we shipped V1 and it was so wrong. It was wrong but it was wrong in a very confident way.

Maor Bril: 10:21 So um for example, the the frame you see here it’s from a video that the model scored 9.2 on the camera work and the camera didn’t move.

Maor Bril: 10:31 For for four seconds it was like a still image of the same character but but the model was was very very happy with, you know, the cinematography.

Maor Bril: 10:39 Um so the physics in in some other videos uh which I’m not showing because of of time limitation. It says that the physics look great, but it said it on on ghosts hovering and and people flying, etc.

Maor Bril: 10:54 So the I mean so then the question was like why was it wrong? The reason it was wrong is because how we generated that data, right? It it um it scored the vibe as opposed to the the axis.

Maor Bril: 11:08 So it’s learned how how to detect uh coherent videos and it learned how to detect the the the artificial artifacts, basically the gloss of the video as opposed to whether or not the video actually um told the the the sorry.

Maor Bril: 11:23 the videos actually told the the story and in and so so the solution was to fix the data set. And so the

Maor Bril: 12:00 The way we fixed the data set, we actually I started pairing real footage versus AI footage.

Maor Bril: 12:09 Now, the risk with that, and that’s the reason why I avoided doing it at first, is because I didn’t want to create an AI detector, right? Because if you start creating pairs of good as human-generated video and bad as AI video, then there’s a very big chance of the model overfitting and becoming an AI detector as opposed to a video quality detector. So there are two things I did in order to avoid that. A, I made sure that the encoding is consistent across both sides of the equation. So there are no artificial artifacts for video A versus video B. And I used the exact same method of annotating both videos, so all the axes in those videos were annotated in the same way.

Maor Bril: 13:06 And surprise, it turned out pretty awesome. And so now what we’re able to do, especially when you’re looking at videos, A, we changed from a very complex pipeline to an agentic workflow. The reason behind this is, A, the pipelines work great if you have a very unique use case. But once you put it in front of users, they’ll have a very distinct story that they want to tell with their own characters, with their own images, and their own voice. So that’s where it starts to drift. By providing the agent with tools to validate the quality of the output it’s creating, it’s able to adapt to changes better, but it’s also able to verify its own work and fix things as they go along.

Maor Bril: 13:57 So if you kind of steal from this talk a few things, one, go relative, not absolute, right? As I explained earlier, the value of comparing video A versus video B will always give you a better result going forward. B, score the real axes that you care about. So if you care about storytelling, if you care about pacing, if you care about physics, score those axes, don’t expect them to magically appear. And put eval inside the generation loop, especially if your goal is to have a higher quality of generation, get the eval as close to the generation loop as possible. Eventually, evaluate it as a story. Videos are stories. Videos are just another way for us to tell stories to others. And thank you very much.

Maor Bril: 15:04 All right, any questions? Okay, down here. Awesome. All right, I got two down here. Here you go.

Maor Bril (audience question): 15:16 Hi, uh, how do you eval sound? Sound and video matching?

Maor Bril: 15:21 I’m sorry, can you repeat?

Maor Bril (audience question): 15:22 How do you eval sound? Sound as effects and matching with the video.

Maor Bril: 15:27 Oh yeah, that’s a fantastic question. So, um, so sound is actually a combination of a few things. One, I’m using Atmos to, um, to make sure that the sound quality is high enough and is understandable. B, the model will learn to identify key frames, right? And especially because when I feed something into the model, it can be just the video or it can be the video plus the prompt that generated that video. So for example, if the prompt will say ‘the door slammed’, right, it will look for a door being slammed and will match the sound at that same frame.

Maor Bril: 15:57 Did that answer your question?

Maor Bril (audience question): 16:00 How does the model recognize sound?

Maor Bril: 16:02 Uh, so it’s both by using Atmos, but also to correlate the… so for example, when it’s looking at the frames, right, it’s making sure that for example, the door being slammed at frame six, frame six has a specific timestamp, so it’s looking for that spike in the sound at that timestamp. It doesn’t know that it is that sound, but it’s looking for a specific spike of sound at that timestamp.

Maor Bril (audience question): 16:21 What about lip-syncing?

Maor Bril: 16:23 Lip-syncing is an unsolved problem yet. We’re trying though.

Maor Bril: 16:27 One question.

Maor Bril: 16:28 Yeah, go for it.

Maor Bril (audience question): 16:30 Lip-syncing.

Maor Bril: 16:31 Yeah, so the question was, what about lip-syncing?

Maor Bril (audience question): 16:34 Oh, that wasn’t me, but I guess the lip-syncing answer would be interesting before I ask my question.

Maor Bril: 16:36 Uh, yeah, um, as I said, it is an unsolved problem still. We’re still working through it, especially for us, you know, some of the characters that we’re trying to do, talking head are humans, right, which you know, we can look into different techniques to try and identify the lips, but some of them are just talking animations that have no real correlation between, you know, the movement of the mouth and speech. So unfortunately, I don’t have a solution for that yet.

Maor Bril (audience question): 17:04 So I’m curious about, for example, if you wanted to further enrich the…

Maor Bril (audience question, cont’d): 18:00 dataset with human evaluation.

Maor Bril: 18:02 Yes.

Maor Bril (audience question): 18:03 Um, the question of taste and what is good, because I think there is a big question mark about is that going to remain the domain of humans? But I’ve also seen people say that well most humans are really they have terrible taste anyway in in videos and games and books.

Maor Bril: 18:19 Fair.

Maor Bril (audience question): 18:20 Um, so how would you construct and align sort of like any human judges?

Maor Bril: 18:27 Yeah. So, um, so this is actually solved at first at the Judge Judy part, where every report it’ll generate, a human can go and annotate it. And we actually, we periodically have sessions where everyone spends 10 to 15 minutes just annotating videos. And that usually happens on multiple axes. I won’t ask everyone to annotate the same video on 10 different things. It’ll be random. And I use that data to calibrate the AI judges. And the results from that is actually being served as a dataset for training for the next version of that model. So it’s a process that does take a little bit of time and hopefully and it does evolve over time, but it’s not immediate, you know, because also taste is very subjective and things that are great for me, you know, some- that I think are fantastic, some people they come and say, like, ‘Are you sure they’re great? because, you know…’ So, yeah, it’s- it’s a process and and I use the human feedback to calibrate the models all the time.

Maor Bril (audience question): 19:43 How did you land on the Qwen-small VLM? Did you try any others?

Maor Bril: 19:47 I did. Um, so the intent I had was to, A, find a small enough model. The reason I went with Qwen is because we also had a very good experience um post-training Qwen on other use cases. So it- I mean, yes, I could have, I did try a few others, but it just, you know, everything was just there and it was good enough.

Maor Bril (audience question): 20:08 So my question is about scale. So obviously Character.ai produces thousands, millions, bajillions videos. At what scale does this become reasonable for my domain that is not Character.ai? So my domain has hundreds, maybe a thousand videos.

Maor Bril: 20:24 Sure. Um, so if you’re happy with- with the cohort of experts and you don’t need, right, so I’ll- I’ll rephrase that. Um, the scale is both for speed, right, as well as- as capacity, because I can serve this model as one instance on one GPU or I can serve it as, you know, a- a hundred instances, right? So that- that determines my- my scale. Um, the reason I chose to go towards a model is because I wanted to speed up the- the creation process, right? It would work- it would have worked just as well if I didn’t have this…

Maor Bril: 21:00 particular model, I would have used like the cohort of experts, right, from metrics, um, that are available both on CPU and GPU, as well as, um, frontier models. So it’s, it, it was a balance as, you know, a, how long did it take me to, to train this model and to curate the, the data set and, and get it to a working set, right, um, and, and how much does it cost to, to serve it versus how much it would have cost me to do this, a, slower, now potentially it is better, right. I mean, like I assume that if you’re going to use the, um, fable, which came back today, right, it would probably give you a, a better result. But at what cost, right. If you do it for one or two, it’s probably fine, and if you do it for, um, thousands or tens of thousands per day, it adds up. So it’s, it’s a matter of your, um, unit economics.

Maor Bril: 21:54 Cool. Over here, on your right, there you go.

Maor Bril: 21:59 Last question. It’s very bright, I’m sorry.

Maor Bril: 22:03 No worries, no worries.

Maor Bril (audience question): 22:04 My question is, I looked a bit at the repo. You guys don’t export OTel traces of the LLM judges yet. Is that something, are you open to that so you can connect to other platforms?

Maor Bril: 22:17 Sure. So, so the, the repo itself, it’s a harness and you can connect any, any agents or any LLMs you want. We actually have an internal version of this, uh, which is running it as a service, right, with an agentic harness on top of it that has all the metrics we care about. But I, I do accept your feature request and I’ll be adding OTel telemetry to the harness.

Maor Bril: 22:49 Awesome. Thank you very much. A warm welcome—or round of applause for Maor. Thank you.

Maor Bril: 22:55 Thank you all. Thank you.