Search ahead of the agent

Today

One connector per tool

His scenario is an on-call engineer at 3 a.m. asking why checkout failed on Tuesday. The usual way to build that agent today is one MCP server or CLI per tool, GitHub, Linear, Datadog, Notion, Slack, handed straight to the agent.

“And then your agent has the task to guess, basically, what might be the right tool containing this information.”00:02:08

And every tool brings its own search, built by someone else, returning data in whatever format they chose. Each call costs money, adds latency, and hits that service's rate limits; these tools were built for people, not for agents.

From the notes: “get the data into a place where it can search ahead of the mcp search function”. That's his fix in one line: don't let the agent search the sources; search a copy you prepared. It's George Fraser's point from yesterday (the warehouse is the context), applied to search.

Evidence: transcript 00:01:49, 00:02:08, 00:03:12

The problem

More sources, more noise

“We always say you are only as good as your data, but in this case, you're only as good as your weakest connector.”00:04:07

And it compounds. Real companies don't have five clean SaaS tools; they have legacy Postgres databases, mainframes, data everywhere. His arithmetic: a hundred sources, ten documents pulled from each, five hundred tokens a document, is half a million tokens for one question, and then the agent has to rank documents from different systems against each other.

“The answers will get just worse, and so the main bottleneck really becomes context here.”00:05:00

“Like getting the right data and access to data is actually the bottleneck, and not intelligence.”00:03:12

From the notes: “more sources, more noise”. The counterintuitive part: adding a source can make the agent worse, because it adds more to sift than it adds signal.

Evidence: transcript 00:03:12, 00:04:07, 00:05:00

The fix

One search layer

Mixedbread's answer is connectors that copy everything into one index ahead of time: PDFs, slides, charts and diagrams, in any language, parsed so the model can use them, aware of recency, kept in sync, and carrying each document's access rules. The agent gets one search instead of a tool per source.

He frames it as infrastructure: the agent sees everything and contextualizes, then goes deeper into a particular source when a result points there.

“Your big fishnet, which we pull it out and then you can sort out what's relevant.”00:08:13

Because it's one index, the agent can also follow connections across sources, with a graph tool over the whole index.

Evidence: transcript 00:06:22, 00:08:13, 00:11:24

The models

Their own embedding models

The search quality comes from models they train themselves.

“And we build open source embedding models, and we're one of the best ones.”00:00:09

His claim is that retrieval had its own scaling moment. For decades retrieval quality barely moved, which is why agents still lean on grep and keyword search. Mixedbread's result is that retrieval models follow scaling laws too:

“meaning the more data and compute you throw at the model, the better it's getting.”00:09:56

Their newest model improved ten points on the metric he showed over their previous one.

From the notes: “what are the embedding models that they are using”. Their own. The talk doesn't name the current one; the links page has Mixedbread's post on Wholembed v3, and their OCR post (“The Hidden Ceiling”) on why parsing quality caps retrieval.

Evidence: transcript 00:00:09, 00:09:56

The question

Access control rides along

Indexing everything into one place raises the obvious question, asked from the floor after the talk: who gets to see what? When a connector pulls from a source such as Google Drive, it pulls the access list too.

“And Mixedbread has this concept of metadata.”00:17:41

The access rules are stored as metadata on each document. An agent searching on a person's behalf has to identify them,

“they need to authorize say I'm user X.”00:17:41

and results are filtered to what that user may see. In the talk itself he puts the ACLs in the connector's job from the start, alongside compliance.

From the notes: “look into how they do the acl and the other access control for ingesting”. The answer has two parts: carry the source's permissions in as metadata at ingestion, and make every search name the user it's for. The recorded answer is brief and partly garbled, so the mechanics (how often permissions resync, what happens when access is revoked) are still worth the follow-up.

Evidence: transcript 00:06:22, 00:17:41

Build or buy

Should we rebuild it?

His build-or-buy pitch: building this yourself means ingestion pipelines, OCR and parsing, the access-control plumbing, and handling every data type, before you get to the agent.

“Without having to deal with all this overhead of building ingestion pipelines”00:07:12

The transcript garbles his figures; his slide is clearer, though the photo is blurry. As far as it shows, building it yourself means connectors for every source, permission and sync jobs, parsing and OCR, embeddings with a vector database and reranker, and an engineer to own it: tens of thousands of dollars a month (the figure reads as about $24K or $34K) and two to three months to production. Mixedbread's column: about $2K a month, and days.

From the notes: “i think we should rebuild mixedbread”. The slide is the parts list for doing that: connectors per source, parsing and OCR for every format, embedding models, ACL metadata, a sync loop to keep it current, and one search API with a graph over it. It's the search half of your service idea from Fraser's talk yesterday: extract into a lakehouse, build on top.

Evidence: transcript 00:07:12, 00:08:59

The result

What it buys

Answers trace back to their source, which matters when the answer feeds a financial or legal decision.

“You can always type back exactly to the source”00:12:15

And in a bank case study he says publishes in two days, answer accuracy rose from about 40 percent to 97 percent; staff stopped triple-checking answers, and API errors halved.

“The answer accuracy jumped from very forty percent to ninety-seven percent”00:14:19

The reading: this is the data layer of own the stack. Fraser says own the warehouse; Shakir says own the search index over it, with the permissions inside. Both say the same thing about agents: don't make them query the world live, one connector at a time. For a client, the question is which of their agents are still guessing which tool to call. From the notes: “follow up on mixedbread”: the case study and the posts are on the links page.

Evidence: transcript 00:12:15, 00:14:19