YouTubeFeed

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Summary

Jai Chopra and Soumya Gupta, from Uber’s computer vision team, walk through the production architecture behind an agent that enhances food photography for Uber Eats’ smaller, independent merchants. The problem is deceptively narrow but hard: a $90B run-rate marketplace where photo quality drives conversion, but small merchants can’t afford professional shoots. The system has to improve quality selectively without eroding authenticity — consumers distrust anything that reads as “AI slop,” so every edit must stay faithful to the original dish, preserve the merchant’s brand, and avoid homogenizing the marketplace by making everything look the same.

The architecture is a chain of specialized agents: an image-understanding-and-routing agent decides whether a photo needs enhancement at all, an editing agent works in a self-correcting loop with a QA agent (measured by a “Pass at K” metric), and a final post-processing QA gate acts as a last line of defense — a deliberately redundant “Swiss cheese model” that catches what upstream agents miss. Every stage is evaluated differently: the router is graded like a classic classifier (precision/recall against human-labeled golden datasets), the editor is graded on faithfulness via pairwise comparison, and the whole system logs everything into one flat JSON structure so technical and non-technical teammates alike can diagnose specific cases or roll up aggregate trends.

The most distinctive part of the talk is the closed-loop, no-human-in-the-loop tuning system: production data is periodically sampled, sent to human labelers, and compared against agent output. Mismatches get routed to a “diagnoser” agent that localizes the failure to a specific component and triggers an auto-tuning pipeline — a reflect agent that surfaces systemic issues and a synthesize agent that updates the agent’s config and reruns it against the golden benchmark before it can ship. They close with concrete failure modes they’ve hit in production: reward hacking (an agent that oversteers into generic, “safe” edits to pass QA), hallucinated ingredients, and frontier-model artifacts that leak into the applied use case — plus how they validate all of it against real marketplace health metrics like conversion and add-to-cart, sliced by geography and dish type.

Highlights

”The last thing you want is food photography that looks like AI slop”

Threading the needle between quality and authenticity

“A meaningful fraction of consumers actually distrust anything that is AI-generated. So if you open up the Uber Eats app, the last thing that you want is to be scrolling through food photography that looks like AI slop. So we’re threading the needle here.” — Jai Chopra, 2:32

Clip command
yt-dlp --download-sections "*2:32-3:15" "https://www.youtube.com/watch?v=31GUkCBD-Uc" --force-keyframes-at-cuts --merge-output-format mp4 -o "uber-ai-slop.mp4"

”Your model’s gonna hallucinate these two extra wings”

A routing miss that risks hallucinated ingredients

“On your left you have an image with six chicken wings. And on your right if you notice the dish name, it says eight pieces chicken wings. And your routing agent approved this image… there’s a chance your model’s gonna hallucinate these two extra wings just to match the description.” — Soumya Gupta, 10:13

Clip command
yt-dlp --download-sections "*10:13-11:00" "https://www.youtube.com/watch?v=31GUkCBD-Uc" --force-keyframes-at-cuts --merge-output-format mp4 -o "hallucinated-wings.mp4"

”The closed loop system — no human in the loop”

The diagnoser-driven auto-tuning pipeline

“And this is a closed loop system, as I mentioned, no human in the loop. We definitely have observability, all the guardrails, quick rollback built in in case of any issues with the system itself.” — Soumya Gupta, 13:07

Clip command
yt-dlp --download-sections "*13:07-13:46" "https://www.youtube.com/watch?v=31GUkCBD-Uc" --force-keyframes-at-cuts --merge-output-format mp4 -o "closed-loop-no-human.mp4"

”Pass at K” — sweet potato fries edition

Iterative QA feedback improving an image over two passes

“Here’s an example: on your left, you see a bowl of sweet potato fries. We send it for the first iteration and our QA agent rejects it because the portion size is incorrect, the plating is very unrealistic. We take that feedback in, go for the second iteration, and we’re actually able to pass it the second iteration.” — Soumya Gupta, 14:14

Clip command
yt-dlp --download-sections "*14:14-14:44" "https://www.youtube.com/watch?v=31GUkCBD-Uc" --force-keyframes-at-cuts --merge-output-format mp4 -o "pass-at-k-sweet-potato-fries.mp4"

”This is an example of reward hacking”

Overly conservative edits that technically pass QA

“This is an example of reward hacking actually, and this is a nugatory change, but something that we don’t think is a meaningful or influential change, despite the actual raw pixels of the input and output being pretty different.” — Jai Chopra, 16:25

Clip command
yt-dlp --download-sections "*16:25-17:03" "https://www.youtube.com/watch?v=31GUkCBD-Uc" --force-keyframes-at-cuts --merge-output-format mp4 -o "reward-hacking-example.mp4"

Key Points

  • Uber Eats scale (0:56) - ~$90B run-rate marketplace, 20% YoY growth, 10,000 cities; as big as Uber’s mobility business.
  • Why visuals matter (1:32) - Photos are often the first signal a customer gets about a merchant; a good photo drives add-to-cart.
  • The merchant problem (2:00) - Small independent merchants lack time, know-how, and budget for professional photography.
  • The authenticity tension (2:32) - Consumers want real-looking photos but distrust anything that looks AI-generated.
  • Agent design goals (3:43) - Preserve authenticity/trust, improve quality selectively, optimize globally, ship safely, learn continuously, stay cost-efficient.
  • Deterministic vs. agentic spectrum (4:22) - Rules-based systems are brittle at marketplace scale; agents need creativity but can’t be left unconstrained.
  • Three-agent pipeline (5:22) - Image understanding/routing agent → editing agent (self-correcting loop with a QA agent) → final post-processing QA gate.
  • Flat JSON logging (6:22) - Every agent in the orchestration logs into one flat structure so technical and non-technical staff can both diagnose cases and roll up trends.
  • Router evaluated as a classifier (7:42) - Confusion matrix, precision/recall; guardrail metric for routing is recall — no bad image should slip through.
  • N-by-n routing matrices (8:15) - More sophisticated routers (e.g., routing to cheaper/faster models) need an n-by-n matrix instead of a simple 2x2.
  • Golden datasets as ground truth (8:56) - Representative data across geographies/dish types/quality, labeled with objective guidelines to remove subjective bias.
  • Router failure modes (9:43) - False positives waste compute and risk degrading already-good images; false negatives (recall misses) risk hallucination.
  • Static models don’t survive production (10:13) - Every component needs to keep tuning itself against online drift, not just the offline-trained baseline.
  • Diagnoser agent (11:24) - Samples production data, compares against human labels, localizes mismatches, and triggers auto-tuning — no human required.
  • Reflect + synthesize sub-agents (12:30) - Reflect surfaces systemic issues in mismatches; synthesize updates the agent config and re-benchmarks before shipping.
  • Enhancement is a 3-step loop (13:21) - Generate an image-specific prompt → enhance → multi-dimensional QA gate (plating, faithfulness, colors) → publish or retry.
  • Pass at K metric (14:31) - The pass rate by the Kth iteration; more iterations of QA feedback generally raise the pass rate.
  • Pairwise comparison for “better” (14:48) - Input vs. output judged yes/no/unsure on faithfulness, completeness, naturalism, realism — baked in alignment with product, design, policy, legal.
  • Failure gallery (15:44) - Added shrimp that wasn’t there (faithfulness fail), removed sauce (completeness fail), reward hacking, frontier-model object-coherence artifacts, uncertain wonton counts.
  • Swiss cheese model (18:00) - The final publish-ready QA is deliberately redundant with upstream QA to catch what slips through.
  • Dogfooding + diagnoser generalization (18:31) - Internal thumbs up/down and free-form feedback feed the same diagnoser abstraction that handles drift.
  • Production validation (20:27) - Marketplace health metrics like conversion and add-to-cart, sliced by geography, device, and dish type.

Mentions

Companies

  • Uber / Uber Eats (0:56) - The delivery marketplace this system runs in production for.
  • Arize (7:04) - The observability platform Uber’s computer vision team uses for logging.

Products & Technologies

  • Image understanding & routing agent (5:22) - Multimodal agent that decides whether an image needs enhancement.
  • Diagnoser agent (19:01) - Higher-level abstraction that ingests feedback from multiple loops and routes fixes to specific agents.
  • Frontier image-editing models (16:44) - Third-party models used for the actual image editing; their artifacts sometimes leak into production and get flagged back to the frontier teams.

People

  • Jai Chopra (0:17) - Product Lead on Uber’s Applied AI team.
  • Soumya Gupta (8:49) - Tech Lead and Applied AI Engineer, Uber computer vision team.

Surprising Quotes

“Firstly, you pay the compute cost for a zero quality lift from this image. And secondly, there is a risk of degrading this image given it’s already such a high quality image.” — Soumya Gupta, 9:43

“This might be a little bit difficult to see at first pass, but we actually added shrimp here when we shouldn’t be, so we fail faithfulness.” — Jai Chopra, 15:44

“The input has some sauce at the bottom of the sushi, we actually remove it, so we fail completeness.” — Jai Chopra, 15:59

“We’re not confident, actually. We’re not sure. And so this is an example where we would actually reject it in production and it wouldn’t go through.” — Jai Chopra, 17:11

Transcript

Jai Chopra: 0:17 My name is Jai, and I’m here with Soumya, we are part of the computer vision team at Uber.

Jai Chopra: 0:24 And we’re going to talk to you about a real world production use case. Oh, mic’s not on? Okay.

Jai Chopra: 0:30 Okay. Don’t worry, I’ll manage. Do you hear me now?

Jai Chopra: 0:37 Okay, so we’re going to talk to you today about a real world production use case, and specifically, we’re going to dive into how we design the evals and the eval loops.

Jai Chopra: 0:56 All right, cool. So just before we get into the agent design, we’re going to talk about a little bit about the use case. So our delivery marketplace, Uber Eats, we do about 90 billion run rate per year at the moment. We’re adding millions of items to the marketplace each and every year… sorry, each every month. We’re growing at 20% year-on-year, and we operate in 10,000 cities globally. So not many people actually know this, but our delivery marketplace is just as big as the mobility side on Uber today.

Jai Chopra: 1:32 Visual content actually plays a really important role for the user experience. So a photo is quite often the first signal that a customer gets that gives them that initial impression about a merchant. So a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding it to the cart. And more and more, we’re seeing different modalities on Uber Eats, especially video content.

Jai Chopra: 2:00 But this is a problem. So our smaller independent merchants simply just don’t have the level of quality for their photos that reflect what the eater is actually going to get. And when we speak to our merchants, there are three themes that kind of emerge: lack of time, lack of know-how, and costs. Because these professional photo shoots actually cost a lot of money. And this can be especially problematic if the merchant is updating their menu over time.

Jai Chopra: 2:32 So this problem is actually pretty challenging to solve for at scale, right? Because our consumers, they want authentic, real-looking photos. But a meaningful fraction of consumers actually distrust anything that is AI-generated. So if you open up the Uber Eats app, the last thing that you want is to be scrolling through food photography that looks like AI slop. So we’re threading the needle here. We need to be able to stay faithful to…

Jai Chopra: 3:00 the original image, preserve the brand of the merchant, and avoid everything looking the same. If we have the same prompt for every photo that we’re editing, the diversity of the marketplace is going to collapse.

Jai Chopra: 3:15 We also, because we operate globally, we also have this long tail distribution of different quality that we see across the marketplace. Um, so we’ve got some examples here, you might see food photography that, you know, is poor sharpness, poor composition, not centered, uh, or, or poor colors as well. We also have a wide range of spectrum of user-generated content on the platform as well.

Jai Chopra: 3:43 So what are our goals when we’re designing these agents? When you think through these goals, you might actually be thinking through, you know, your own agency that you’re building yourself. But for us, it’s about one, preserving authenticity and trust. Two, improving the quality when we need to. So we want to be able to improve quality selectively. We want to optimize globally for, for the entire marketplace. We don’t want to cannibalize certain merchants. We want to ship safely, and this is going to be an important theme throughout the talk. We want to learn continuously. And we want to operate at scale in a cost-efficient manner.

Jai Chopra: 4:22 So agents are actually really well suited to solve this problem. So if you imagine a spectrum, on the one side, you’ve got something that’s more deterministic, it’s more rules-based, um, and, uh, you, you have more control over it, but it’s fairly, it’s a brittle system. It’s not actually going to be able to scale for the entire marketplace. Imagine the other side, you provide an agent with obviously a lot of creativity, it has a lot of agency, um, and that’s actually what we want to lean into, but we can’t leave that unconstrained, right? Because we have certain safety and certain guardrails in place that we need to adhere to.

Jai Chopra: 5:00 So we want to find a balancing act. Uh, and that’s kind of set the principle for the way that we think and design around agents and evals. So now we’re going to actually like dive a little bit deeper into a simplified but representative example of what we have in production. And we’re going to go through each stage and how we eval it and then talk through some continuous learning loops as well.

Jai Chopra: 5:22 So first up, we have what we call an image understanding and routing agent. So this is where multimodality is pretty important. We actually ask the LLM to describe what it sees in the photo. Um, and then we, we create a structured output from that and we send it to a router.

Jai Chopra: 5:48 The router will then determine, do we enhance it or do we skip it? If we skip it, we will keep the original. If we enhance it, we send it to our next agent, which is an image editing agent. And this can actually run in a loop, so it gets feedback from a QA agent, um, it can edit, uh, uh, in, in this loop and self-correct.

Jai Chopra: 6:00 corrects and fix things, um, as it goes. If it goes through a number of loops and it still fails, we, we don’t publish it.

Jai Chopra: 6:08 Then we actually send it to a final post-processing and QA step.

Jai Chopra: 6:13 If that’s all good, we’ll publish it to the menu.

Jai Chopra: 6:17 And the last thing that’s really critical is we log everything.

Jai Chopra: 6:22 Just a quick note about logging, don’t know if you can actually read the JSON here, but you might notice that all of the

Jai Chopra: 6:30 agents in this end-to-end orchestration is within one, it’s basically a flat structure in this JSON.

Jai Chopra: 6:38 Um, and so this is actually incredibly useful for the entire team because anyone, be it non-technical or, um, technical folks in engineering product can actually dive in

Jai Chopra: 6:48 um, and look at specific cases to diagnose and also roll up things to look in aggregate.

Jai Chopra: 6:54 Um, and it’s important to note here that, you know, we think this is important to start with. You want to start with your logging,

Jai Chopra: 7:04 because if you don’t start with it, you have nothing to optimize for, let alone set up a self-learning loop. And at Uvo, we, um, we use Arise.

Jai Chopra: 7:12 Cool. We’re going to dive, um, a bit deeper into the router.

Jai Chopra: 7:17 So the router’s actually pretty straightforward. If you remember, you know, we have this multi-modality input, we look at certain text description metadata, the image itself, we ask it to re- to, to

Jai Chopra: 7:25 to describe what it’s seeing. We create structured output from that. With that structured output, we can then grade against a rubric.

Jai Chopra: 7:32 So we have these pass and fail criteria. The last step is we want to decide whether or not we should enhance or skip.

Jai Chopra: 7:37 How do we actually eval this?

Jai Chopra: 7:42 This is, you could think of this as a more sort of traditional classifier. So here we, we have a confusion matrix. You know, many of you are probably pretty familiar with this,

Jai Chopra: 7:49 um, but we can look at things like the true positive cases, the false negative cases, and so on and so forth. And essentially what we’re doing is we’re measuring the precision recall.

Jai Chopra: 7:59 In practice, your routers might actually be much more sophisticated. So for example, we might want to route an image to a lower latency smaller model to be able to save on costs

Jai Chopra: 8:08 and improve the user experience at the trade-off of quality. And if that’s the case, instead of having a 2x2 matrix for

Jai Chopra: 8:15 your confusion matrix, you might actually have an n-by-n matrix, where each grid is actually

Jai Chopra: 8:22 telling you whether or not you’re correctly routing to that specific branch.

Jai Chopra: 8:27 So we’re going to now, um, hand over to Somya, who’s going to dive a little bit deeper into how we handle drift and human alignment.

Soumya Gupta: 8:49 So now that we spoke about how we eval the routing, I want to talk about how do you get the first version of the model out.

Soumya Gupta: 8:56 For our use case, we consider human labels as the golden source of truth, and this is what we want to align.

Soumya Gupta: 9:00 for models. The way we go about this is we go collect a data set which is representative so, you know, different cuts, geographies, dish type, image quality type, send it to our human labelers and give them a very objective guideline to label on. This is to remove any subjective biases or any noise coming in from human labelers. Once we’ve got that system set up is when we start tuning our model. We take our agent, we go ahead get output from the agent, compare it to your golden data set, evaluate if it’s good enough to ship, if it meets your guardrail metrics, you go ahead and ship it. If not, then you go tune and you keep doing this until you meet your guardrail metrics. For routing, our guardrail metric is recall. We don’t want any bad image to slip through our system.

Soumya Gupta: 9:43 Here’s some examples of the failures we’ve seen. On your left you see a very good image of cheeseburger. On the right you notice that the routing agent actually failed this. It said the technical is low bar. And it’ll go send this image for enhancement. Now there’s two challenges when you send this image for enhancement. Firstly, you pay the compute cost for a zero quality lift from this image. And secondly, there is a risk of degrading this image given it’s already such a high quality image.

Soumya Gupta: 10:13 And on the other end of the spectrum, you have a recall miss. So on your left you have an image with six chicken wings. And on your right if you notice the dish name, it says eight pieces chicken wings. And your routing agent approved this image. That means, so now there’s a risk here. If you send a, send this image for enhancement and you only see six chicken wings, there’s a chance your model’s gonna hallucinate these two extra wings just to match the description. And that’s also an, that’s a the cut we take at our faithfulness metric that Jai earlier showed us. So the meta point I’m trying to get here is you’ve trained your offline model, but there will be long cases where your model is going to continue to fail and this static model will not work in the real system. You need a way such that your prompts, agents, system itself is evolving over time. And that’s what we’ve done for our system as well. And I’m talking more from the routing perspective, but every component in our system is able to tune itself for any drift online.

Soumya Gupta: 11:24 So what we do is we sample production data at regular cadence, send this to the human labelers with the same guidelines that we have seen before. Once we’ve got that data, we compare our agent’s output with the output we got from the labelers and see if there’s a mismatch. If there’s a mismatch, we have an umbrella diagnoser agent which takes in the feedback, localizes where this issue is happening, and triggers an auto-tuning pipeline. Once we tune this agent, we go benchmark it against our golden data set that we saw earlier. And if we pass our golden data set on the metrics that we had designed, we go ahead and ship this model. If not, then you kind of keep iterating. And this happens on a regular basis on production data set. The beauty of this is this is completely config driven and doesn’t require a human in the loop. Your diagnoser agent can write your config and trigger the auto-tuning pipeline here. And this is what will keep your model

Soumya Gupta: 12:00 over time, you will have one static model with the offline, but this is what is going to keep your system alive.

Soumya Gupta: 12:09 So Jai’s going to spend more time on the diagnosis side of it. What I want to do is zoom into the auto-tuning bit. And again, we’re looking at routing, but this is how we tune every agent in our system.

Soumya Gupta: 12:20 So we start with a target agent, and we’ve already got these unseen eval samples from humans. We go find out the mismatch and matches and call a prompt optimizer agent.

Soumya Gupta: 12:30 Now this itself is two sub-agents: there’s the reflect agent and the synthesize agent. What reflect does is it just looks at the mismatches, tries to find, remove any noise, find any systemic issues that might be in your data set, and reflect on it and send that feedback to the synthesize agent.

Soumya Gupta: 12:51 Now the synthesize agent takes this feedback, it has your agent config. It goes and updates your agent with the new config based on the feedback it’s getting and goes and benchmarks again. If this benchmark is passed, you actually register this new agent in the new agent config store, and next time your production runs, you pick up the new version of the agent.

Soumya Gupta: 13:07 And this is a closed loop system, as I mentioned, no human in the loop. We definitely have observability, all the guardrails, quick rollback built in in case of any issues with the system itself.

Soumya Gupta: 13:21 Moving on to the next step of our orchestration flow, so we spoke about routing, moving on to the enhancement bit of it. It’s a three-step process. What we do is the first step, we generate a prompt specific to this image. We take in the dish description, we take in the directives we were getting from our routing agent, and we go ahead, generate a prompt for this image, what needs improvement in this image specifically, and we go ahead and enhance this image.

Soumya Gupta: 13:46 Then you’ve got the QA gate, which is a multi-dimensional gate, looks at multiple things like plating, faithfulness, colors, and if it passes is when you actually go ahead and publish this. If it doesn’t pass, you take the feedback back from the QA gate, push it back to your generate prompt along with the initial inputs you sent it, and go ahead and enhance it again.

Soumya Gupta: 14:04 So there’s two end results here: you either keep enhancing for K iterations and you pass your QA gate and you publish, or you take a coverage hit and you never enhance this image.

Soumya Gupta: 14:14 Here’s an example: on your left, you see a bowl of sweet potato fries. We send it for the first iteration and our QA agent rejects it because the portion size is incorrect, the plating is very unrealistic. We take that feedback in, go for the second iteration, and we’re actually able to pass it the second iteration.

Soumya Gupta: 14:31 So the metric we’re measuring here is pass at K. Pass at K is essentially the pass rate at Kth iteration, and ideally with the more the iterations, your pass rate will increase because you’re getting more feedback in.

Soumya Gupta: 14:44 Now I’ll pass it on back to Jai to cover the rest of this.

Jai Chopra: 14:48 Thanks Soumya. Uh, so yeah, just before we end here on the on the generation evals, we use what’s called pairwise comparison, right?

Jai Chopra: 15:00 or a pass-fail case, so it’s looking at the input image and the output image and it’s assessing whether or not it’s better.

Jai Chopra: 15:06 But how do we actually find what’s better?

Jai Chopra: 15:08 So, um, we’re not going to dive into too much of the details here because this is kind of like proprietary stuff and so we’ll just mention it at a higher level that this is where you sort of, at least for us at Uber, we have to make sure we’re aligning with product, design, policy, legal.

Jai Chopra: 15:26 And this is where we’re baking in what we define as a better image on the platform into our evals.

Jai Chopra: 15:34 So examples here: is it faithful, is it complete, is it natural, is it realistic, and there’s a bunch of other things as well.

Jai Chopra: 15:41 The output of this is then a yes, no, or unsure.

Jai Chopra: 15:44 So here are some examples of failure modes. So input and output on the right, the input’s on the left, output on the right-hand side.

Jai Chopra: 15:52 This might be a little bit difficult to see at first pass, but we actually added shrimp here when we shouldn’t be, so we fail faithfulness.

Jai Chopra: 15:59 This is where we go the other way. So the input has some sauce at the bottom of the sushi, we actually remove it, so we fail completeness.

Jai Chopra: 16:06 Here’s actually a pretty interesting example where the agent actually attempted a more creative edit, the first iteration, and then the QA said, “Nope, that’s not good enough.”

Jai Chopra: 16:17 And then it actually oversteers the other way and it becomes overly conservative, sort of falls back to this generic ceramic plate—ceramic bowl, sorry.

Jai Chopra: 16:25 So this is an example of reward hacking actually, and this is a nugatory change, but something that we don’t think is a meaningful or influential change, despite the actual raw pixels of the input and output being pretty different.

Jai Chopra: 16:39 Here’s another example where in the output, the plate is covering the sauce.

Jai Chopra: 16:44 This is an example where for some of the frontier models that we’re using for the actual image editing, some of their—some of their problems will actually sort of leak up into our applied use case.

Jai Chopra: 16:54 And so object coherence and physics plausibility are the evals that sometimes we’ll coordinate with the frontier teams and let them know about these problems and work together with them.

Jai Chopra: 17:03 Here’s an example of why multi-modality is pretty important. In the input and the output, we can’t actually see that there are eight pieces here of the wontons.

Jai Chopra: 17:11 So we’re not confident, actually. We’re not sure. And so this is an example where we would actually reject it in production and it wouldn’t—it wouldn’t go through.

Jai Chopra: 17:18 So the last step after all of that is a post-processing and what we refer to as the publish-ready QA. This is the final gate before we decide we want to publish something to production.

Jai Chopra: 17:29 Here we do some policy checks, we also do some more quality checks.

Jai Chopra: 17:35 And you might be wondering…

Jai Chopra: 18:00 Like we’ve already done some QA. Why are we going to do another step of QA? And the reason is because we think of this like a Swiss cheese model. So we want to try and optimize for reducing the chance of a failure getting into production. And so there is some redundancy here or there, and that’s okay. Um, and so this QA step is a little bit more holistic. It captures more things, but it also will try and flag things that we should have caught upstream as well.

Jai Chopra: 18:31 All right, so we’ve talked about a couple of feedback loops here. So to summarize, we talked about predominantly this first one here, which is the model loop. And this is accounting for drifts and aligning with human-labeled dataset that we have and we’ve established offline. But we actually have more feedback loops. So we have at Uber, what we have is a great sort of dogfooding culture where we will test apps before they go live. Um, but we also have when it goes live in production, how do we get that feedback back into our agent to be able to steer it appropriately?

Jai Chopra: 19:01 So as we’re adding more of these feedback loops, we want to be able to generalize the system. So this is where we’ve actually created a higher level of abstraction on top, which we call the diagnoser. So the diagnoser can take in any input from these different feedback loops that we’re capturing. It can reflect on what actual agent within the overall system needs to be optimized, and it can route to that agent to be able to fix that configuration specifically. It could be one agent, it could be multiple agents.

Jai Chopra: 19:47 So here’s an example of internal dogfooding. You might see these in sort of different apps that you’ve got where you’ve got the thumbs down and the thumbs up. We also take some free-form feedback as well. Um, and this is actually great because we’ll get feedback from merchants directly, we’ll get feedback from, you know, design teams, other product teams at Uber. And we’ll incorporate that feedback back into our diagnoser step and tune the system over time. Again, similar sort of workflow pattern here. We’ll replay the examples that we know are those ones that have been flagged, be it good examples, be it bad examples, and then we’ll benchmark the metrics before we push the latest config version.

Jai Chopra: 20:27 The last step is actually getting this into production. And this is where we’re looking for a whole heap of different metrics that we track for the marketplace quality and health. I’ve just called out one here, which is conversion. So we’re looking for improvements in people adding to cart, converting, completing their orders. Um, I think this one’s actually an interesting one to call out because now, at least at Uber, but especially in production settings at scale, you have a wide, um, you have a lot of data that you can actually slice and dice.

Jai Chopra: 21:00 So, in this area as opposed to others what we can do is sort of slice by geos, by device type, by dish type, etc. And we can look at where things are improving in different segments and actually tune on certain segments as well.

Soumya Gupta: 21:16 Cool. And that’s it for our presentation. Appreciate it.