How to build agents over your own data

The recording starts as the speaker begins; the MC's introduction is not captured. After 00:15:46 the recording is conversation after the talk, then one question on access control from 00:17:41.

Transcribed locally with Qwen3-ASR 1.7B (timings per ~5 s segment), prompted with the speaker names of overlapping sessions. "Splunk" at 00:00:13, 00:08:59 and 00:18:01 comes from that prompt, not the audio (Splunk gave the previous talk in this room; the Voice Memos transcript has no company name at those points); at 00:14:35 it may be genuine. "Mixedbread" may also be prompt-biased: the Voice Memos transcript has "big bread" and "mixed fraction", and Qwen itself has "Big Spread" and "mixbreed". The Voice Memos transcript is kept beside it as transcript.txt.

00:00:09 — Introduction

Before we jump in, quickly about me and Big Spread. So Splunk started originally as a research lab. I think you all remember embeddings, vector databases, and all those things. And we build open source embedding models, and we're one of the best ones. We have over 100 million logs in our database. People love models, but we were never happy about the state of the space. And also the reason why this talk is going to work because we believe that.

The models were just not good enough, and we innovated a lot on the model side. To come up with better types of search algorithms. To make it happen, and we're building a multimodal like audio, video, PDF, slides, code. And we're dealing with a search engine which has over 400 million. So we can't index any kind of data. And you can provide. All this data, basically, as a search engine or search tool for your agents.

You can make them actually work on your data. So we all know is that banks, insurances, hedge funds, but also any of the companies. We do the product. Originally started really as a search tool, but over time people started using us more and more for different use cases. Where they ask us to connect us so they can connect with them and so on and so forth. And since we are dealing with really highly sensitive data.

We are also two ISO compliant and. I think everyone's most by now like a two major databases. We also build in a searchable database, so if you want to own your own data. You can bring your own bucket and store all the data in your own premise with zero data retention.

00:01:49 — A 3 a.m. incident

So let's imagine you have a on-prem or on-call engineer, and it's three a.m. At night, and. And incident happened. So, or like you wake up in the morning if you're a smart and want to know. Why did the checkout take place on Tuesday?

00:02:08 — One tool per source

So, how you would do it right now? And this is how most people build their agents: is they. We use an MCP or CLI per tool that you need, right? So. Maybe using GitHub, maybe using Linear, maybe using DataDog, Motion, Slack, and so on and so forth. And you just provide all these tools directly to your event. And then your agent has the task to guess, basically, what might be the right tool containing this information.

And second, every of the. Tools is bringing their own search. You're at the mercy of the search engineers or the engineers at these companies. Providing the really good tools and preparing the data in a format. That your agent likes to understand. Let's say you have, for example, graphs and data domain. So how is gonna data about exported as a CSV or a graph? You know, like there are a lot of this fine grain details.

That you can't control, and they're really at the mercy of. Of the providers, but you know, since the models are just getting better and better and more persistent, they will figure it out.

00:03:12 — Cost, latency and rate limits

But at what cost? So at the end, you know, like one of these cars can cost fifty to hundred dollars. And if you imagine that you have a lot of internal agents doing a lot of internal workflows. You know, like it would just not scale well. And the second thing is next to quality and cost is also. Latency and greatness. So just to tell you, for example, I'm using Dots heavily, and Dots is connected to my Gmail.

And since two days ago, I'm I'm not getting any updates from Dubs anymore because my Gmail is waiting between the whole time. So you know, you also at the mercy of the weight limits of these tools. That they might derive scale, you need to shift that. And this is pretty terrible because all these tools are not built for the genetic era. The recent build for like No CIs or some.

Human workloads, and as we're moving towards this agentic era. Like getting the right data and access to data is actually the bottleneck, and not intelligence.

00:04:07 — Only as good as your weakest connector

So and this is what. We have internally, when we talk about these tools and these agents, is that. We always say you are only as good as your data, but in this case, you're only as good as your weakest connector. Because you know, what if, what if like the most relevant information is in Jira or whatever. And you are streaming to you right now, and so your agent can't get the right context and doesn't have this information. Can't really, for example.

Debug a critical production issue. So. What we need, therefore, is a way to bring all the data at one spot. The device contextualization and the device format, so Aiden can hammer it and pull it out. And Oh, eventually this end is just public, so.

00:05:00 — More sources, more noise

So the thing is as well, and we see this with a lot of our customers. So especially if you're working as a systems and you're not just working on linear drive and you have like. Most data, and you have like some legacy Postgres database running somewhere. You know, like you have some data sitting anywhere. You have some. Is still running on mainframe for some insurances. We have a lot of different data sources, and the more sources you have.

Do more noise to introduce you because then you know like. Let's say you have 100 sources, and this incident will require you to search 100 sources, and then you pull 10. Ten documents per source. And then you have already like 1,000 documents in your pool, and let's say everyone has like 500 tokens. We had 500,000 tokens for one search query, which you can pull into the context window.

And then pass through it, and then do things so context words become reality. And you have also that the tools. How do you know which credentials you need to take? So, and also once you're putting them into context, and how do you really rank them against each other? So you know, I think everyone understands what the issue is. So with having the same model and a lot of sources.

You will get the worst answers. The answers will get just worse, and so the main bottleneck really becomes context here.

00:06:22 — One index for everything

So when we ended up building a big spread, as I mentioned in the beginning, that we built this amazing search engine. Which is like multi-model, multi-lingual, which can understand charts, which can understand diagrams, which show you how it looks like. Which also understands recency. What we ended up building is basically connectors. And we just put everything from Mixedbread, little PowerShell. Meaning, if this PDFs, PowerPoints, who is prepared that everyone from other language can understand it, it will index it.

It was a different thing, and also get the ACL. So we have always the right. No compliance, and only the right people are allowed to access this, and with having everything at one place. You can basically provide one search.

00:07:12 — Why not build it yourself?

And you would ask, Hey, why is it different than Czech? You know, why is it different than Slovak? So first of all, they use NLP, but secondly, they use the product. If you're building a product for your own company or for your users, you can use it as a building stone to have the same experience as. Without having to deal with all this overhead of building ingestion pipelines, without building, you know, all this ACL stuff and.

Having this headache of dealing with every different data type, I can tell you if. A legacy customer wants to bring their whole data platform and data pipeline together. And focus on building. And once everything is in the cloud, you can then decide what you want to do. Right? So you have an agent. And like everyone else, we also provide an NLP, but it's just on search and NLP, so you can use your NLP to do the action.

But mixedbread creates all the data, or you can use a CLI, Autodesk API, build it.

00:08:13 — Search as infrastructure

So as I mentioned, it's good for companies specifically, but originally it's done as infrastructure. So you have to use search, meaning. That we're just copying everything from every source, so the agent can see everything and contextualize. So you know once. It finds a query and finds oh there's something impossible. Let's just go deeper into Posthog. They can just ask me. But hey, I want to know more data about Posthog. Just pull me in.

Like this, you know, it's this part can be seen as. Your big fishnet, which we pull it out and then you can sort out what's relevant. And try all to find the right thing. And the interesting thing is, since the data is already prepared and formatted, users can understand. They will also subscribe to Splunk. I will show you pictures.

00:08:59 — What it costs

So. Since we built the models ourselves, and we spent a lot of time on optimizing infra. It is way cheaper than building it yourself. Let's say your hardware requirements, not too much. With five pages each, so five hundred dollars and fifty thousand dollars per month. Building it yourself is all the OCR, the parsing. Then adding the indexing will cost roughly twenty five thousand dollars for those best of the solutions.

It was nice to just help people in the community to keep everything up to date, keep it running, and. This day is actually pretty weird because we have people who arrive on Monday. See the results on Wednesday and done production on Friday. And we've been through this with the largest headhunters school, so great, great success with that.

00:09:56 — Retrieval quality and scaling laws

And why we are able to pull this off, but. To make it so cheap and provide good quality. So what I'm showing here is the graph of. Our retrieval quality changed over the last. So, thirty-two years, and you see it's like. Pretty unique, right? But we saw this generative AI, like an exponential development. This retrieval has been there, and this is also one of the reasons. Why we're still talking about grep and Vim 25?

Is that we never had this step function shift, so the models. Can still use this old technology and can get good results, but what we have managed. Just to make a step function change in the Google, but we can show this time, hey, if you get on something like scaling laws. On retrieval models, meaning the more data and compute you throw at the model, the better it's getting.

And this what we are seeing is this huge amount. Compared to the coherent event, five pro models came out two and a half a few weeks ago. And this is our previous model. This is a ten-point change for this metric. It's pretty crazy because it took us. Fairly long to even you know improve that environment so.

00:11:24 — How an agent searches

How does it look like for an Asian right now? So I feel like these people. So let's just limit to what we know. Search tools, a grep tool, and a regex tool. So once we have the credentials, we can go out and just say, Hey, mixbreed, tell me. Ask Mr. Berson why didn't check our data in Spark? Instead, they would then just pull the relevant ten or fifteen documents.

完全同意哈斯尔的说的。 And then it's like, okay, I'm seeing that maybe not good enough. And then figures out oh it's something about here because you know everything. One index, you can then ask one insurance payment provider, and since it's. We also provide export graph tool, so we can actually graph over the whole full index.

00:12:15 — Demo

And I would like to show you how the platform looks like and what I'm describing here. Unfortunately, I didn't manage to prepare this exact demo. Yes, this will be the point where I show you a chart of. Let's go. Cool. So let's do something else. Let's say, hey, we have some charts index and we're super interested in them. We have just to do a very creative jump with me that this won't be about an incident, but let's say, hey, we want to know.

Was this the implementation rate in India? For example, it would be hours. The platform is some unreleased instances. That's fair. What I'll hear is. If we were to journey to live information, for example, here we the moment we can pay this will be activated. It's live for each hour. So this sense of personal employment, and the sense that the answers in this chart. So what it does as well is what I meant was pausing.

So the machine can perform it. If your LLM is not multimodal, it cannot do it. And since we have all this multimodal information. We can answer the query actually and send back where we got the source from. So when there's an incident or whatever is happening right now. And you want to know where this answer is coming from for some financial decision, but legal decision. You can always type back exactly to the source and write the models just by primitives.

So it's also interpretable and understandable for humans.

00:14:19 — A bank case study

And obviously, we also run benchmarks with our customers. And what most people see, this is what a case study we're publishing in two days. This one we ran last week is where, like a bank. Who placed the SVP connectors to the right? What they saw is. I think you had this talk with Splunk before. They also use Splunk, where they measure the answer accuracy and quality. And source location and source data.

The answer accuracy jumped from very forty percent to ninety-seven percent, and also the pupils enjoyed way more reading the answers. Because it wasn't interpretable, it was properly stated here, and a nice downstream effect of this is that the total cost. Were hard because what we're seeing as our MVPs is that they had to triple and quadruple check everything because it was not in a format. They enjoy it, but like it, and also.

We saw a lot of API errors, and we could half the API errors because. The SPLN API is just now used for doing actual.

00:15:23 — Wrap-up

So, yeah, this is my talk. And again, the agents only as good as what it can find. We saw a dashboard. We're down. In the Toronto, and if you're there, you can talk to us, and we're also hiring. So if you're correct. And one is full search, and one builds out infrastructure because we build own database.

00:15:43 — After the talk

Can you crash? I might have cracked like stone tablet. That egg is cracked. Thanks for that. By the way. What. Like you know, if you're afraid to answer, yeah. I don't know if. Slang. Just a minute. Don't you have any kids? No. What does it mean? It means that I like it. It's my generation. That means you're very good. Oh, thank you. He's a nice person. At least somebody. Thank you. He sits it down.

Someone's being kind. We're just making fun of you. How you old guys know this? Because all of X is like this kind. So cracked. This guy is so cracked. If you're cracked, I'm going to take. I only want. Kids working at my company. Another English to learn. I learned so many Englishes already. One point three million dollar year. Look at this guy. He's got a whole reference book.

That's amazing. Duck syndrome. Yeah. And we said, as they were preparing, everything was just going. It's so funny, you're a trash look. My favorite. It's a. It takes a little earlier. Really takes a while.

00:17:41 — A question on access control

Can you say a few more words? Yeah, for sure. So when you get from the APIs from, let's say, Google Drive or something, this ACL, basically, look. Who is allowed to access this data? And Mixedbread has this concept of metadata. Where, like, because. We saw all the metadata alongside Splunk as an update. Update, we recommend that metadata. And what we set up for you is this. Once their agent on behalf of the user as a carrier, they need to authorize say I'm user X.

And then we just imagine, hey, let's use that just for minute moment. Given this query and this picture. The users obviously like basically most of the users. Okay. Cool. That's all from my side.