Building a Shared AI Assistant: The Platform Challenges Beyond MCP

Transcribed locally with Qwen3-ASR 1.7B (timings per ~5 s segment), prompted with the speaker names of overlapping sessions. "Google DeepMind" at 00:25:14 comes from that prompt, not the audio, and the line in Devanagari script around 00:25:58 is the recogniser's misreading, kept as evidence. The Voice Memos transcript is kept beside it as transcript.txt.

00:00:00 — Introduction

Hi everyone. This is Utkarsh Shrivastava, and today I will be talking about scaling enterprise AI. Platform challenges that go beyond MCD. So what we will all be looking at today. First, we will talk about. How to roll out an agentic suite?

Pure and hard confidence. We did this with more than 10,000 employees, and it's a very different experience building that at scale. We will also talk about three judges. That settle the platform versus domain ownership, and really try to tease out who should own what. And how do we scale this across companies?

Last but not least, we leave with three things that you can do next week.

00:00:43 — Choosing models

Right. So there are so many tools to pick from. What do we really choose? On one hand, you have frontier labs. We we really started our journey very seriously towards building the Chennai focus agent next week. When the frontier labs like Anthropic and OpenAI were coming up with their enterprise solutions.

And finally, there were numerous solutions being argued. Very recently, Gemini is kind of moving from their enterprise Gemini offering. The next AI also has something coming up. So number one ninety five frontier labs gives us an offering. On the other hand, open web models were evolving throughout the year.

And they also started adding enterprise support. And so, the tricky part was while we were evaluating one set of suite. You always have these new and exciting players come in, and suddenly there were many. Many morals to choose from. And lastly, because this has been going on for a while.

Each of our domain teams have their own. Code review boards, Israeli agents—they had the. Mini agents. We need to do one localized skill. So that was that was the ecosystem. It was. Let's do the same things, and we needed one solution to do it all.

And the focus here, if you look at my red diagram closely, right at the center. You will see way more insights, and that's the focus of this.

00:02:11 — Company context is the key

So. The company context is the key here. It's. Taking taking this as an iceberg analogy. The AI tool that you adopt is only the tip of the iceberg. Now, it really. The the focus here is really doesn't matter which tool you are adopting because the technology around us is changing so quickly.

By the time you sign your contract and you release, you already have more features that they are now taking part of their free. So that's how fast the world around us is changing. But what does matter most is below the surface that you see. Its agent harness. Enterprise skills and entities.

If you want to do. Agent harness is your runtime execution environment, which also you can swap for another. One of the open source or one of the premier labs. The enterprise skills and NCPs is where you really need to spend your time on.

00:03:12 — Before and after the agentic suite

So let's do a before and after scenario. So, so before AI, what used to happen? Your enterprise will have a database. You'll have a backend. You'll have a very nice UI on top of it.

And you will have a particular workflows, which then you will give it to your. Finance, marketing, legal—all of these domains issuing as their own suite of applications. That work has very quickly transitioned. And it has transitioned to the following: your database has an MCP offering above it. And your data can now be seamlessly connected, but what?

The agentics suite offers. Here, let's let's go. Let's go to the next slide. The internal agent takes we gives you one experience that connects data across the team. And so, while this is not something which could not have been done before.

It really reduces the cost of adding multiple workflows together. And that was a big win here. So before we had disparate systems. Disjoint, which had very specific access. To your data, and with the agentics suite, all of that is now connected.

And so it's only a matter of turning on that particular toggle, and voila. You get access to many more information that you did not previously. And that leads to higher velocity and a lot of excitement.

00:04:40 — Every tool is another way to choose wrong

But that also creates a lot of ambiguity. And that's what we noticed. Every connector, every tool that you add. Is another way for the agent to choose wrong. On May nineteenth.

Netflix launched one of our. Premier live offerings with a Rousey versus Karamu MMA fight. And it was a very very successful event for us. So. This was the same time when we were launching our internal suite, that is, internal analytics suite. And so, the user went into the suite and asked.

How many people watched the Rousey versus Cerrone fight? At that time, let's call it T one. We have hooked up our analytics connector. So the answer that we got from Google was 12.4 billion, which was right at the time. After a few weeks, some other user went to use the same tool again and asked.

How many people watched the Roundy versus Scary fight? And this time the answer was nearly seventeen million from another collector that we added. This was. The transactional character that we added, which gives information on the live tools. So, who is right? Does anybody in the audience want to?

Big one this. Both leaders be more explicit. I think you are very right. Both are correct. Because this is a classic OLAP versus OLTP debate where your analytics.

Mean the average across the guys was really from twelve point four million. When the live was more or less catering to the peak traffic on the file, so when you have. One way to get it right, what does that mean? In in both cases, if you see, there was only one answer which did fail. And so, if you were expecting the other answer.

You would be very disappointed, but the tool was never wrong. It just, it just. Was not the information you were looking for. So there are specifically a few kind of solutions. I'm calling three.

00:06:49 — One word, many meanings

First, this same word can have all different meanings. When you say wash. It can be card average. It can be deepenified. It can be some some other information.

Now this is a great company because they are able to solve this for you. This is something with your business. Same as the word that we spoke. Both tools can say yes. We have information because they do.

So while you are building your rocket's battery, where would you run? And the third thing is nobody really wants it. The M C Ps have made it very cheap to make a lot of information accessible, but. Did we ever resolve the ownership part of the ownership debate? No. And so, very early in our journey, when we started enabling these things.

People were excited. They came to our booth, and then. They didn't get the answer they were looking for, and so the trust started eroding very quickly. And we learned that the hardware-aware orchestrations. I'm here to share our learnings with you.

DeepMind solved the connection, but did not solve the disagreement. As I mentioned before, we had no way to detect or cycle overlaps, and no standard framework existed. I'll say once and say it again. Towards the end of my slide, this is a hard problem to solve. There is no one magic horn which exists, and nobody has fully solved it for the company.

So, where did our first boundary started to break? There was one entity which lived. In several domains, and each each was right, as you saw. Sometimes a question about one-line record can exist. However, it can.

The answers from a reporting solution or your transaction systems. Moreover. We have worked with legacy companies for. Have had your system run for the number of years. You will see. A lot of reporting happens with third party, and so those third party tools have now offered entities, and they have access to this information.

That just makes it all more confusing for the true state, right?

00:09:03 — Gate one: access and evaluation

How do we solve it? We are going to hold next session on. A series of steps which a world force can take to start solving these things. First, start with implementing a rollback access control. While the internal suite has enabled.

You a way has given you a way to connect everything. Do not start giving access to each other directly. Arback is a great. Middle way to start with, so that you are still giving an identical tool your to your workforce. And keeping some of these collisions away.

It fixes some of that. However, it it does not solve the blank problem. So. This is just the very first step. This is not going to be as an all-star. And that will be next.

Talking about the three judges, and really the evaluation suite, which. Can help answer some of these things. So. Introducing the evaluation suite now. This is not new for anyone of you. However, I do want to relay the importance of having an evaluation suite.

So what happens if you don't have one? First. You're able to connect the data. You're able to go quite fast, and everything works like a charm. And there are a lot of a lot of initial excitement because now. The earlier, your enterprise users would have to rely.

On some of the data profiles, go to them to build their dashboards. Get access to that information. All of that is possible now by AI. However, when you hit production. Your agent, as as you add more connectors and skills and plugins into your agentics suite, a lot of collisions start happening.

And so it starts to feel that people are not getting what they are looking for, and I shared some such examples. And so, if you do not have an evaluation suite to begin with, right through your journey. You run the risk of diverging in a very reactive way. So you will receive an invitation, and then you will start replicating the problem yourself. And you will have to go through each logs to identify what what kind of behavior is that.

Happening and it's going to take a really long time. If you don't solve it, even in the next three months, the cost of debugging would get exponentially harder from that point on. The other big tool is CI checks again. Continuous integration is something which. Has been in the software industry for a while.

However, we have to now more than ever really rely on AI. And this relies with the domain teams. Before you add any skill, or any plugin, or any connector. With each commit, we to start running the entire regression suite. And that prevents you from introducing collisions into the inference loop.

Running the each time is one way. The other way to do that is when a user changes models. That can also run the risk of getting a different answer altogether. And I'm sharing all of our learning. We've built every single one of them pretty much.

And so, while your tests, regressions, retests pass, you're working on the model. Let's say 0.5 for five. It might give a very different answer, a very different model. And so regressions will need to cover all of their models. And then finally, start with a solid twenty to fifty use cases.

Curated by various scenarios that you are already experiencing in your systems today. That can give a good starting point, that eventually brings you forward. So that's that's the judge number one. Where if you fully run your eval suite and check fails. It sends an email to the team which owns the data, and this is for the first time that they learn that.

00:13:04 — Gate two: who owns the answer

Even though VTA ownership of this move is another team out there, which is also claiming ownership of it. All these problems already existed; they were just never caught, and your internal suite is now finally allowing. All of the these things to expose. The second areas are. Going back to my example of how many people wash their hands versus fight.

Has one to give one answer, and the other to give another answer. If we don't settle the debate, sometimes which. Which two? Which are actually really, really interesting picks? Sometimes because of recency, a newer tool got added. It gets picked.

It has advertised the. Information well better than others, and so that pool is big. Sometimes, university favors a more tenure approach. That this tool has been with us for a while. And so we would rather pick the original, which has always been the aspect.

Regardless, the bad behavior is one complex answer because the the right thing to do is to label both. So in this case, what I could do would be show both guard average and team average. This this is all background, and then finally, third judge is intuition. Who is asking if the company really has the key? So your role.

Let's take an example. If I'm an engineer working on adding. On-call support to support my live traffic. I will want to move traffic. If I'm a business user who wants to know how many people watch.

I can very well plan the next set of libraries for my company. So very different. Examples and this is where. Context on who is asking is really helpful. Then of course we have the usuals like what is the most likely meanings and how the platform handles that.

In this case, a long-running agent and an active agent can simply ask, "What do you really mean?" To perform for a long-running agent, especially for a multi-agent, should just pick one. Take its assumption and turning it into absolute. Sure, you've all seen that the coding videos that how we take to do that. So this will summarize here the evidence and the orders of two judges, that ship. With that run before you ship, and intuition is life.

When you run your internal suite, it basically contests and intuition. The regression test. Are run by domain teams. However, the platform is still responsible for the hardness. And providing a thin layer of test cases, which otherwise cannot be captured by coding.

So, for example, if you take finance and talent, two different domains, all together.

00:16:05 — Gate three: high-stakes writes

But there is a financially relevant talent data. Then who who really owns this? In this case, the the answer is neither, or maybe both. And maybe the innovators never settled. That's a bit of it.

And so the platform as the Asian partner of. Trimming down these edge cases which fail, and then running it across the results. This gets even trickier when the stakes are high. So if it's only a read operation. You can answer how many more.

You give a number. Fine, you fix wrong. Even though a number of products are zeroing in the lab, the stakes are not that high. For reversible writes, the stakes are higher than read. For example, for updating a dashboard, you you you want to query.

State of the art of the ability and the analytics practices that you can focus on the scale of the data. And then really trace it back and go back as possible. The stakes are high, but not as high as the reverse. What's a general-purpose database? I issue a bunch of commands, and then the email goes off to the console.

Yeah, that's pretty irreversible. Especially if you're outside the box, you don't want to be in that situation. Or worse, you are putting an earnings deck and. In objects of numbers, and then the next thing you know, everyone goes to Wall Street or shares, unless you.

00:17:34 — Did consolidating help?

Definitely, don't want to be in this situation. So, this part to highlight how important the evaluation. Andumi, beri anda lebih banyak peluang. Always, we know if three judges are working, it's pretty straightforward. We have fewer art conflicts.

A new connector should not take questions that belong to an existing one in an idle way. When it does. Your routing, which is run by the platform, are. Improves routing accuracy on benchmark, and then reruns on every change. Your your internal suite has to have fewer fallbacks and retries, ideally.

It should, and that's a very easy way to measure. How often are you getting your accuracy right? It should vary over time. And finally, the conflicts fixing is easier where you raise a ticket and you. Each each conflict does not become an escalation.

Just then you have to go back and. We have worked on everything. I just if you want to know, there are still very many open questions. Of course, tools get harder to maintain. In some cases.

The teams have very strong opinions in other cases. They they tell us we have never discussed this. They really don't know. And you keep going on the data chain, and then you get all the. So that's still open.

Avoiding users who need, of course, are only shared. But a couple of contexts break them very very differently. So again, this idea I can only use. Only a company has the relevant context. And you still have to solve that.

There's no other way to solve it for you. How do we enhance our connectors? I believe we did a very good job in adding more connectors, engines, and plugins. But we have not reached a point where we are doing a good job, and we have that, and so I'm afraid that we if we are. In the same situation for long periods of time.

We will end up with a series of characters, and these these problems will get much harder for us to solve. So a good year down is those of us who have done our own tests. Year down is one of the most important things, and that's just something which remains to be seen here. And then, truly, when is sharing worth its coordination cost? Sometimes we could go on to spend the bunch of time debating on each and everything, and so it.

The things remain as is. I didn't really feel. So next week. Can write twenty valid cases. You can add overlap checks to CI, as we talked about.

And then you can decide how your platform can evolve. We spoke gradient edge across all of these three. To recap, tools will disagree, and so there is no one-size-fits-all.

00:20:28 — Platform and teams together

There is no magic plan. There is no one platform team which can solve all your needs. Neither there is one domain team which has. We try to figure out all of these choices. The real days to have both tools and platforms come together and build something cool.

I believe that this is the first time we are coming back. We hit a lot of new roadblocks as a team to figure out things together, and everyone knows. Sometimes what I've seen in the past is take one good example of one good domain team and use that as a way to convince others. To start a bathroom or building during the sweet collection building. So we covered the sampling choice, the kind of collisions and features, and then you learned.

That evidence designs facts and. Really, your ownership and execution go a long way in solving some of the more complex cases that you'll face. So to end, don't crown a winner, but do the work.

00:21:38 — Questions

What I'm hearing is we do have some minutes to take questions, so I'm happy to take. Maybe a few here, and then you will find me backstage after this talk. So if you want to chat with me or take more questions offline, I'm happy. So please go ahead. You can raise your hand, and she has the mic.

I have like many questions where we just ask people and then talk backstage. So that's a really important point: the OLAP and OLTP data. And just want to know, do you have any standards or responsibilities? So who can actually create these MCPPs? Because everybody wants to create these entities.

So it's like everyone creates their own cities, as I said. People only create issues. So what is that? That is an excellent question. I'll walk you through our journey and how we launched it.

When we were launching this for the first time, both. The transactional systems and analytic systems—they both claim they have the information. Transactional because. Everything starts there. They are the source of truth, and then the data warehouse. They do the roll-ups.

And so they have. They not only have access to information, but they have better access to information. Something which business directly consumes from it. So our starting point was the internal analytics we would. Query the analytics plus.

And if the answer is no, we come back to transaction, and then we reverse it. Because the reason for that was. Not all the information travels through your analytics connector. Only the relevant information for the business you use to live there. But transactional system has answers to everything.

So over time, as we learned that the number of Questions were more answered from your transaction systems. We let the transaction systems get the first routing. And then let them decide to hand it over to another data scientist. That was a big, big change that we encountered. Yeah. The next one is like.

So with APIs, there were standards around, you know, open API, and then. Every organization has their own platforms. We don't have to create an AI so that people have one place. Yeah. Again. The goal was not to create more busy work for the teams that have already done the work.

So they are integrated already existing for all of our systems. The part was how do you integrate those APIs into the agent experience? So, in in in our case, we have a layer of both ideas. There are many solutions which offer that. So what we did was we created an MVP for that year.

And that meant that each of these enterprises do not have to individually solve for it. There was a DeepMind connector, which. Would make all that information available, but I will still say the flip side, which is. Now you have access to all that information. You very quickly need to go back to analysis.

But but the goal was not to have a multi-year journey of creating MCPs because now we are in the. You work in Google DeepMind, and how visions, come about. Another question about the network. We have a similar challenge, and how do you recommend to handle that on the tool? This level, or on the data level.

To access to specific rows, to to specific columns, in most cases. The answer will be different. At what layer? This is something that you all have to figure out. At what layer?

लैकनिज्म सर्चर साल बार शुरू किया गया है। And what we have observed that the platform where the query comes. From the very beginning, we'll implement RAG at that level. So that any other mechanisms and routing that you have. Remain intact because this is something you are bringing only as a starting point, and very quickly this will. Like at some point you might want to relax our backbone, so we.

So to begin with, we implemented it at the platform level. Only high-tech enterprises have access to. Information. We should have access to information. So, for example, marketing. Only marketing will have access to marketing.

Data with NVIDIA and our background in all that. And as they needed more information, they slowly relaxed. All right. Thank you, everyone.