State of Data — Sean Cai, Independent / State of Data
State of Data — Sean Cai, Independent / State of Data
Summary
Sean Cai, an independent researcher and author of “State of Data,” delivers an unusually candid tour of the murky data-market industry behind frontier AI labs. His central framing: data is the underfunded leg of the compute/data/talent triangle, and what’s actually scarce isn’t the room-of-annotators labeling work that made Scale AI famous — it’s the kind of data that takes a model from generalist competence to real expertise. He distinguishes “Type 1” data (genuine, uncurated real-world workflows — GitHub commits, session replays) from “Type 2” data (contrived examples manufactured by hired experts sitting in an arbitrary setting), and argues that the industry’s dirty secret is that everybody sells Type 2 and bills it as Type 1.
The talk’s organizing idea is Jason Wei’s “Verifiers Law”: the ease of training a model on a task is proportional to how verifiable that task is, broken into three axes — asymmetry (how decomposable into checkable steps), veracity (how much consensus exists on correctness), and proliferation (how often the real world hands you fresh verified examples). Coding scored high on all three thanks to GitHub and unit tests, which is why it matured first as an AI application layer; domains like biology, finance, law, and “taste” score low on all three, which is why labs are now buying up Slack and Jira logs to manufacture that missing verification signal. Cai also introduces “benchmark psychosis” — most published benchmarks are Goodhart’s-Law-compromised Type 2 constructions, tested in isolation, that quietly stop measuring anything real once the same people selling the eval also sell the data to climb it.
Cai closes with a market thesis: no infrastructure pioneer (railroads, AWS, foundation model labs) has held more than 10% share in the long run, and app-layer companies are already decoupling from any single model provider (he cites GLM 5.2 beating GPT on real-world rubrics). His prediction is that successful data companies are becoming “neo-labs” — pivoting to enterprise, building “Antikythera mechanisms” that translate messy business context into evals, and treating their moat as the live pipeline into real work rather than a static, rapidly depreciating dataset. He closes by announcing his own new venture, Antikythera Mechanisms, built to do exactly this for enterprises.
Highlights
”Everybody sells Type 2 and bills it as Type 1”
“What is actually available nowadays is process-based data, which is the trajectory, the reasoning trace, the sequence of decisions… Type 1 data is a sort of pure capture of real workflows, like GitHub commits or session replays, with minimal reward shaping by non-experts; and Type 2’s contrived data, where you sort of hire experts, you sit them in an arbitrary setting, you have them manufacture examples. The dirty secret of the industry, though, is that everybody sells Type 2 and bills it as Type 1.” — Sean Cai, 4:58
Clip command
yt-dlp --download-sections "*4:58-6:00" "https://www.youtube.com/watch?v=ZyIoTOAbRfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "type1-vs-type2-data.mp4"
”Verifiers Law explains the whole market”
“He has this law, it’s called Verifiers Law. The ease of training a model to do a task is proportional to how verifiable the task is. So I break verifiability into three axes here. And once you have them, a huge amount of this market stops being mysterious.” — Sean Cai, 6:15
Clip command
yt-dlp --download-sections "*6:15-6:44" "https://www.youtube.com/watch?v=ZyIoTOAbRfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "verifiers-law.mp4"
”Most benchmarks you see are quietly fake”
“The dominant Type 2 recipe is: let’s hire domain experts, let’s have them use Chat to generate plausible tasks, let’s have them solve those tasks, and let’s cherry-pick the ones where the model diverged. And let’s package those as a hard North Star benchmark. And then this is the perverse part: let’s sell the data to help climb that same benchmark. It’s basically Goodhart’s Law with a profit motive.” — Sean Cai, 8:12
Clip command
yt-dlp --download-sections "*8:12-8:44" "https://www.youtube.com/watch?v=ZyIoTOAbRfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "benchmark-snake-oil.mp4"
”Data spend today predicts the product launch two months out”
“In January, Anthropic was spending a lot on cybersecurity data from new vendors. And in March and April, they were spending a lot on biological data from associated vendors. And what do you know happened like two to three months after? Well, Methos and Cyber and Claude Bio/Life Sciences today.” — Sean Cai, 12:00
Clip command
yt-dlp --download-sections "*12:00-12:39" "https://www.youtube.com/watch?v=ZyIoTOAbRfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "data-spend-leading-indicator.mp4"
”Your moat is the pipeline, not the dataset”
“If you’re a researcher, stop outsourcing your definition of realism to the same vendors you buy your evals and tests from. That’s kind of just letting the test writer grade the test. And if you’re a builder, your moat’s not the data, it’s the sort of pipeline into real-world work, plus the infra to keep retraining on it as the models improve underneath you.” — Sean Cai, 16:54
Clip command
yt-dlp --download-sections "*16:54-17:26" "https://www.youtube.com/watch?v=ZyIoTOAbRfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "moat-is-the-pipeline.mp4"
Key Points
- Not a company pitch (0:12) - Cai frames the talk as independent, exposing “alpha” in a murky data-market landscape few understand.
- Annotator rooms are the least interesting part (0:28) - The Scale-era image-labeling business (~$10-15B/year per lab) is real but no longer where the scarcity is.
- Data as coal and iron (1:13) - Cai’s framing: data is to the white-collar AI revolution what coal and iron were to the Victorian age; “the TAM is an inversion, it’s all labor.”
- Unbundling is permanent (1:13) - Vertically integrated giants (Mercor, Surge, Scale AI) are losing steps to specialists; labs now mandate 20-30 vendor diversification because quality doesn’t scale linearly with quantity.
- Antikythera mechanisms, defined early (2:22) - Bespoke systems that translate messy business context into evals; a recurring concept through the talk.
- The compute/data/talent imbalance (3:15) - CapEx is exploding while AI revenue lags; data is the underfunded leg of the equation.
- State-based vs. process-based data (4:21) - State-based data (rows in an ERP, saved files) is the old next-token-prediction fuel; process-based data (the reasoning trace) is what’s actually scarce now.
- Data depreciates fast (5:37) - A dataset is only valuable as long as it tracks the moving frontier; the only durable supply is a live business partnership, not a dead startup’s codebase.
- Verifiers Law’s three axes (6:30) - Asymmetry (decomposability), veracity (consensus on correctness), proliferation (frequency of fresh verified examples).
- Why coding matured first (6:44) - GitHub solved all three verifiability axes at once via unit tests, community consensus, and infinite public examples with commit messages as free reasoning traces.
- Why Mercor wants your Slack logs (7:31) - Biology, security, finance, law, and taste score low on verifiability because no Web 2.0 system ever captured that work.
- Cross-harness differencing (9:13) - A major, underappreciated cause of benchmark score divergence, per Anthropic and other researchers.
- A single benchmark score is one noisy sample (9:52) - Compared to using a screwdriver bit to cut cheese and concluding the knife is broken.
- Finance rubric case study (10:25) - Testing ARR waterfall reconciliation, LBO valuation memos, and pair trades reveals Opus and GPT trading off arithmetic vs. methodology strength — invisible on headline scores.
- Data markets as a leading indicator (12:00) - Track what data labs are buying now to predict what product they’ll ship in 2-3 months.
- Long-horizon task checklist (12:39) - Enforced step length, heterogeneous tool calls, state transitions that constrain future actions, mandatory failure recovery.
- Robotics as a cautionary counterexample (13:21) - Modality isn’t settled (ego vs. teleop vs. Umi) and vendor sophistication is low; “don’t die on a modality hill.”
- No infrastructure pioneer holds share forever (14:44) - Railroads, AWS/Google, and now OpenAI/Anthropic all eventually cede application-layer capture.
- GLM 5.2 as proof of decoupling (15:27) - Surpassing GPT on real-world rubrics shows app-layer companies can already decouple from foundation labs.
- Data companies becoming neo-labs (16:07) - Mercor and Handshake reportedly derive a surprisingly large share of revenue from enterprise work already.
- Five enterprise infrastructure jobs (16:25) - Serving/routing small models, managing RL datasets across base-model migrations, and building Antikythera mechanisms, among others.
- New venture announced (17:07) - First public announcement of Antikythera Mechanisms, helping real-world companies monetize data and implement RL-as-a-service.
Mentions
Companies
- Prime Intellect ([description]) - Cai’s data quality research affiliation.
- Hummingbird / Costanoa (description) - Cai’s prior investing background.
- Mercor (0:12) - Cited repeatedly as a major, opaque data company now buying enterprise Slack/Jira logs and pivoting to enterprise revenue.
- Handshake (0:12) - Data company cited alongside Mercor for opaque production and enterprise pivot.
- Scale AI (0:28) - The 2019-era annotator-room data giant; acquired in 2024.
- Surge (1:13) - Cited as a vertically integrated data giant losing ground to specialists.
- Anthropic (9:13) - Blog posts cited on cross-harness benchmark divergence; also cited as buying cybersecurity then biological data ahead of product launches.
- OpenAI (15:27) - Referenced as a foundation model “fiefdom” being challenged by GLM.
- AWS / Google (15:08) - Cited as infrastructure-layer consolidators that never captured the application layer.
Products & Technologies
- GPQA (0:28) - Benchmark dataset referenced as the 2024-era acquisition target that later fell apart on long-horizon unverifiable work.
- Verifiers Law (6:15) - Jason Wei’s framework for task trainability via verifiability.
- Antikythera Mechanisms (2:22) - Cai’s concept (and new company) for translating messy business context into evals.
- GLM 5.2 (15:27) - Cited as surpassing GPT on real-world rubrics, evidence app layer can decouple from any one lab.
- Opus 4.7 / Opus 4.8 / GPT 5.5 (10:44) - Compared head-to-head on private finance-task rubrics.
- Methos, Cyber, Claude Bio/Life Sciences (12:00) - Products Cai cites as following predicted data-spend patterns.
People
- Sean Cai (0:12) - Speaker; independent researcher, author of “State of Data.”
- Jason Wei (6:11) - Researcher credited with “Verifiers Law,” recommended reading.
Surprising Quotes
“Data is to the white collar revolution what coal and iron basically is to the Victorian age in the information age right now… the TAM is an inversion, it’s all labor.” — Sean Cai, 1:13
“Maybe you’ve been approached by Mercor to buy your Slack logs or your Jira logs as of late if you work at an AI company.” — Sean Cai, 7:31
“A single benchmark number under a single scaffold is like basically one sample from a distribution whose basically width nobody measured… it’s like a it’s a pretty noisy sample. That’s why there’s so much what I call benchmark psychosis today.” — Sean Cai, 9:52
“No pioneer of an infrastructure technology has actually held more than 10% of the market in the long run… Railroads built company towns, they charged tyrannical rents, they got nationalized as soon as the automobile moved in.” — Sean Cai, 14:44
“I maybe shouldn’t say this in a public setting, but Mercari and Handshake, people don’t notice an incredibly large amount of their revenues are enterprise now, enterprise in ways you wouldn’t expect a data business to do.” — Sean Cai, 16:07
Transcript
Sean Cai: 0:12 Good to give this talk and I just want to say right off the bat, this is probably going to be a little different from what you’ve seen so far at AI Conference. I’m here to not deliver an agenda on any company’s behalf but just to expose a lot of alpha in data markets, but also just tell you what’s really going on behind the scenes in a very murky landscape where nobody seems to know how Mercor, Handshake, and a lot of these folks actually produce data.
Sean Cai: 0:28 So, quick reframe before we start. When people hear data markets, they picture scale around 2019, they picture these rooms of annotators in like Manila labeling images and that’s real and it’s maybe, you know, 10 to 15 billion dollars a year per lab, but it’s kind of the least interesting part. The models work now, what’s scarce and badly priced is the sort of data that takes someone from generalist competence into real expertise. We need a — since 2024 when Scale got acquired and we were spamming a lot of GPQA data sets.
Sean Cai: 1:13 So, look, let me start with this framing. Data is to the white collar revolution what coal and iron basically is to the Victorian age in the information age right now. I have this piece called the TAM is an inversion, it’s all labor. And so the supply chain is basically just doing what every sort of industrializing supply chain does, it unbundles. Two years ago, one vertically integrated giant, like a Mercor or Surge or Scale AI — if you’re unfamiliar, they’re massive data companies — they had to do all of it, but that was because that’s the only way that unit economics worked in an immature market. Today, that’s increasingly not the case. Specialists outcompete the giants at a lot of steps: sourcing the people, building environments, designing rewards, running evals. The fragmentation I would say is pretty permanent, it’s not transitional. And quality increasingly does not scale linearly with quantity, which leads to a sort of cottage industry in data land right now where you have labs literally mandate vendor diversification on the scale of like 20 to 30 different vendors because they inherently distrust their ability to scale quality with quantity.
Sean Cai: 2:22 So to orient you, here’s the sort of argument as it developed this year. January we saw a lot of industrialization and unbundling. And you’ll hear me come back to this mechanism I call the Antikythera mechanisms, which is this sort of bespoke systems that translate messy business context into evals. Increasingly important in labs’ hungry quest for real world seed data, this set — seed ends. I also explore this concept called type one versus type two data, contrived versus non-contrived for the more researcher types in the audience and why the GPQA cell playbook that works in 2024 for data acquisition kind of falls apart on long horizon unverifiable work.
Sean Cai: 3:00 Going through all these topics, all of which are online, I’ll also actually just skip to the more interesting part, but I’m bringing this subject just here in case any of the particular topics I talked about are particularly interesting and you want to dive in more.
Sean Cai: 3:15 So, model improvement is a function of three inputs. We all sort of see this commonly expressed adage: compute, data, and talent. I put together this very rudimentary sort of chart online in one of my pieces and a very rudimentary equation just to express the fact that, you know, if there’s any sort of imbalance in this compute, data, and talent, then you start seeing a sort of inefficiency in producing what I call just generalized AI model performance, right?
Sean Cai: 3:42 But also, we see a sort of inefficiency in CapEx spend that is a sort of result of this equation right now. Exponentially increasing CapEx spend, but AI revenues are sort of falling far behind.
Sean Cai: 3:58 Data is sort of the underfunded leg here; it’s the one that sort of turns a generalized model into a real expert. It’s the one that’s actually, I believe, quite lacking in this equation, and thus, because of the imbalance penalty parameter concept that I’m expressing, presents a whole opportunity.
Sean Cai: 4:21 But, going back to what I said at the start, what is data actually? So most people picture state-based data: the rows in an ERP, which is like the final output, or saved files. That’s kind of the 2023, next token prediction model, and it’s mostly personal data wrapped in privacy law.
Sean Cai: 4:41 What is actually available nowadays is process-based data, which is the trajectory, the reasoning trace, the sequence of decisions. It’s kind of what gets a professional to get from a blank page to a finished work output and sort of delineate how the work gets done.
Sean Cai: 4:58 So on top of that sits a quality axis. This is the vocabulary I’ll use all talk: Type 1 data is a sort of pure capture of real workflows, like GitHub commits or session replays, with minimal reward shaping by non-experts; and Type 2’s contrived data, where you sort of hire experts, you sit them in an arbitrary setting, you have them manufacture examples.
Sean Cai: 5:21 Type 2, right place to start; models when they were reading at a first-grade level, I think anybody in the world could teach them as a third-grade teacher. But Type 1 is what gets you from 20 to 80 percent, so to say, because the realism is inherited from the work itself.
Sean Cai: 5:37 And the structural reason why it matters is that data is kind of the most depreciable asset there is. A data set is only valuable insofar as frontier data markets as the frontier moves. So the only durable supply of it technically is a live business you partner with, not a dead startup’s codebases like so many data companies out there are buying today.
Sean Cai: 5:59 The dirty secret of the industry, though, is that everybody sells Type 2 and bills it as Type 1.
Sean Cai: 6:00 First and why, there’s a reason why Claude Sonnet came out so far, now in the future after models got good before a lot of other application layer advances.
Sean Cai: 6:07 So now let me talk about this central axis of how you can think about sort of verification and deciding which application layer domains are maturing.
Sean Cai: 6:11 So Jason Wei is a researcher online whose blog is great, you should read him.
Sean Cai: 6:15 He has this law, it’s called Verifiers Law. The ease of training a model to do a task is proportional to how verifiable the task is.
Sean Cai: 6:22 So I break verifiability into three axes here. And once you have them, a huge amount of this market stops being mysterious.
Sean Cai: 6:30 First, asymmetry of verification: how hard is it to decompose a task into checkable steps?
Sean Cai: 6:34 Veracity of verification: how much consensus is there about what correct even means?
Sean Cai: 6:38 And then thirdly, proliferation of verification: how often does the real world sort of hand you fresh examples of verified work?
Sean Cai: 6:44 If you think about why coding is the first mature AI app layer market, that’s really no accident because we were blessed to have something called GitHub from Web 2.0 which solved all three of these at once.
Sean Cai: 6:54 Unit tests, they sort of give you objective decomposable correctness. The community agrees on what working code means generally.
Sean Cai: 7:03 And there are effectively infinite public examples with these commit messages as basically free reasoning traces. So they score high, high, and high on these three axes.
Sean Cai: 7:12 Now look where the money is trying to go now: biology, security, taste, finance, healthcare, law. These sit pretty low on veracity, pretty low on verification.
Sean Cai: 7:21 And the verification examples are sort of like locked all into these enterprise workflows, no Web 2.0 system ever captured, really. Or very sparingly few.
Sean Cai: 7:31 That’s why, maybe you’ve been approached by Mercor to buy your Slack logs or your Jira logs as of late if you work at an AI company.
Sean Cai: 7:42 And that’s the whole game, right? It’s predictive, it’s not just descriptive. Classify any professional’s tasks on these three axes and you can sort of tell which markets mature most.
Sean Cai: 7:51 So it’s no surprise that after code, we went to search, and after search, we went to finance, and after finance we went to healthcare and law, and after healthcare and law, well, I’d say cyber, biological, and scientific discovery.
Sean Cai: 8:03 And maybe even taste, which is probably the most unverifiable out of all these.
Sean Cai: 8:06 So verification is a sort of bottleneck. Let me talk about why the industry sells a lot of snake oil here and why most benchmarks you see are sort of quietly fake.
Sean Cai: 8:12 The dominant Type 2 recipe is: let’s hire domain experts, let’s have them use Chat to generate plausible tasks, let’s have them solve those tasks, and let’s cherry-pick the ones where the model diverged.
Sean Cai: 8:22 And let’s package those as a hard North Star benchmark. And then this is the perverse part: let’s sell the data to help climb that same benchmark.
Sean Cai: 8:32 It’s basically, and maybe some of you guys here in SF have heard this a little too much, it’s Goodhart’s Law with a profit motive, basically. The moment your measure becomes the target, and then the target is set by people who aren’t true domain experts, it stops sort of measuring anything real.
Sean Cai: 8:44 And so the whole market, I like to call it sits in a sort of fog of war. Labs, vendors, and enterprises, they’re all kind of guessing which data actually improves a model. Nobody can see this clearly. There’s a structural tell.
Sean Cai: 9:00 Contrived benchmarks, they only ever test a single isolated in-distribution question. They can’t test whether a model sort of sustains correct reasoning across a long dependent episode because these tests were all sort of meant to be selfed in isolation.
Sean Cai: 9:13 In May I spent a lot of time — this, increasingly in a lot of Anthropic blog posts but also like a lot of other researchers have noted that cross-harness differencing and cross-infrastructure differencing is the primary cause for a lot of benchmark divergences in performance.
Sean Cai: 9:28 A lot of you are probably very familiar with the frontier suites of the world and the deep suites of the world benchmarks. There are issues that all of these benchmarks have related to that specifically.
Sean Cai: 9:43 In particular just high false positive and false negative rates that are not immediately obvious when you look at the above-head stats on the benchmark.
Sean Cai: 9:52 So a single benchmark number under a single scaffold is like basically one sample from a distribution whose basically width nobody measured.
Sean Cai: 10:01 And that’s why there’s so much what I call benchmark psychosis today. It’s a pretty noisy sample.
Sean Cai: 10:12 So, really briefly on an example I took, I basically have an internal version of Val’s AI and a lot of private benchmarks.
Sean Cai: 10:25 So I took three finance tasks from three real-world vendors, like an ARR waterfall reconciliation and LBO valuation memo and a sort of like long-short pair trade for hedge fund trading. So long horizon non-verifiable finance tasks, relatively robust like deterministic verifiers paired with LLM as a judge applied correctly.
Sean Cai: 10:44 So when you actually run these I think you’ll notice very clearly Opus 4.8 is worse than 4.7 on a lot of these rubrics. You’ll start noticing things if you actually do rubric analysis that the 4.7 to 4.8 step over-engineered self-reflection.
Sean Cai: 11:04 You’ll notice that GPT 5.5 and Opus 4.08 score within three points on the same task but in like exactly opposite directions, whereas GPT nails the arithmetic but Opus nails the methodology but loses arithmetic, which is to say…
Sean Cai: 11:18 …it’s clear if you actually do very good agnostic benchmarking you can tell where the post-training directions for a lot of these teams went and subsequently it informs a lot of the data that comes for this.
Sean Cai: 11:30 So going back to the start, a single benchmark number is a sample from a distribution nobody measured and it’s basically like taking a Swiss army knife and using the screwdriver bit to cut cheese and concluding the knife is broken.
Sean Cai: 11:44 So, the receipts from the last slide, this goes into an RL environments report that is sent to labs, but no single leaderboard number ever shows you this part of data, which some of the most sophisticated RL environment companies, some of which are actually here at this conference, would show you this.
Sean Cai: 12:00 So, how do we read sort of which domain is next instead of chasing hype? And you can actually use this as a proxy, data markets as an upstream indicator of what next application layer products the labs will come out with. In January, Anthropic was spending a lot on cybersecurity data from new vendors. And in March and April, they were spending a lot on biological data from associated vendors. And what do you know happened like two to three months after? Well, Methos and Cyber and Claude Bio/Life Sciences today.
Sean Cai: 12:39 So if you want to use this checklist I use: classify the profession’s tasks into the three axes, apply the long horizon bar, the ones real labs use, enforced step length, heterogeneous tool calls that aren’t interchangeable, state transitions that genuinely constrain future actions, mandatory failure recovery, you’ll notice a pretty mixed bag of how long horizon is even defined from the vendor perspective in terms of specs. And then three, you want to look at the raw data for five signals: sequential decisions against one entity, inferable expert action per step, outcomes recorded by independent parties. But moreover, just economically available high-wage work.
Sean Cai: 13:21 So subsequently, one counterexample to keep you honest, like robotics, the modality is not settled, ego versus teleop versus Umi, but moreover, I find generally a huge degree of unsophistication within robotics data vendors today, a huge degree of unsophistication. I don’t know, some of you guys are probably robotics researchers in the crowd. I don’t know how many times people have come up to you and they’re like, “Oh, here’s like 100,000 hours of iPhone video from my friends in India. Do you want to buy this for ego data?” So in that sort of domain, your vendor’s choice is entangled with an unsolved research question. At the end of the day, you have to realize RL environment companies, if they actually succeed, they’re more so research accelerators. It’s a boutique industry. If it’s venture scalable, it’s because the infrastructure they’re building agnostically in-house helps in enterprise application layer use cases rather than assuming that data markets today will stay as they will forever. So don’t die on a modality hill, basically.
Sean Cai: 14:18 So one last exhibit just from my work. This is the map under everything I just described. Look, the share of white-collar work is sort of on the vertical axis, task horizon is on the horizontal. Short horizon is very addressable right now, but you think about this long tail on the right side, this deep dependent long-horizon work. That’s where the real economic value and data build-out sort of both live. And this threshold line that kind of moves rightward every time somebody builds a real-world data pipeline.
Sean Cai: 14:44 Now I want to talk a bit more about the model layer. It changes who needs to build what. So historical fact: no pioneer of an infrastructure technology has actually held more than 10% of the market in the long run. I’m not saying this applies to Anthropic…
Sean Cai: 15:00 But just those who forget history are condemned to repeat it, right? Railroads built company towns, they charged tyrannical rents, they got nationalized as soon as the automobile moved in.
Sean Cai: 15:08 AWS and Google, they consolidated the infrastructure layer and still never captured the application layer.
Sean Cai: 15:13 And right now, OpenAI and Anthropic are carving out these fiefdoms, but all these pressures from anti-distillation, exported PPOs, enterprise exclusivity, they’re basically the equivalent of railroad rents and they erode, right?
Sean Cai: 15:27 The automobile in some cases has already arrived, like GLM 5.2 surpassing GPT on a lot of real-world rubrics is pretty hard proof that a lot of app layer companies can decouple themselves from the model layer.
Sean Cai: 15:41 And because models differ on efficiency and modality, they’re not fungible like electricity. So it’s not exactly leading to nationalization, but it’s definitely not heading to durable lock-in either.
Sean Cai: 15:49 And the whole question on what to build next hinges on whether a general enterprise can decouple models from foundation model labs.
Sean Cai: 15:58 But luckily, for a lot of data companies, this is actually where they’re headed. I came here to give a talk on data markets. I’m here to tell you that the successful data companies nowadays are all pivoting to enterprise.
Sean Cai: 16:07 I maybe shouldn’t say this in a public setting, but Mercari and Handshake, people don’t notice an incredibly large amount of their revenues are enterprise now, enterprise in ways you wouldn’t expect a data business to do.
Sean Cai: 16:19 Once enterprises stop renting a lab’s intelligence and start owning their own, you need an entire abstraction layer that doesn’t exist yet.
Sean Cai: 16:25 There’s like five jobs: one, serve and route small models targeted by cost, latency and performance profiles.
Sean Cai: 16:31 And two, manage your RL datasets instead of across base model migration. So when you swap to a new open source base, you rerun post-training automatically instead of starting over.
Sean Cai: 16:39 Three, the Antikythera mechanisms that I talked about before. And in the interest of time, happy to talk more about emerging infrastructure needs afterwards if you want to.
Sean Cai: 16:44 So let me bring it all together. I think data companies all realize they have to be neo labs. Data businesses do not stay data businesses because the durable value sort of accrues to the services and app layer of actual work.
Sean Cai: 16:54 So two takeaways: if you’re a researcher, stop outsourcing your definition of realism to the same vendors you buy your evals and tests from.
Sean Cai: 17:01 That’s kind of just letting the test writer grade the test. And if you’re a builder, your moat’s not the data, it’s the sort of pipeline into real-world work, plus the infra to keep retraining on it as the models improve underneath you.
Sean Cai: 17:07 I’ll close on this. I’m building, I’m working on something new. This is the first public announcement of it. I’m building Antikythera Mechanisms.
Sean Cai: 17:15 I’m working with a lot of real-world companies to monetize their data assets, but moreover help implement RL as a service with a lot of the enterprises in the world, mitigating a lot of the pitfalls of a lot of the companies I mentioned.
Sean Cai: 17:26 If you guys want to talk about it afterwards, I’m on Twitter, I always write a lot on Twitter and Substack. And in the interest of time, feel free to just come up to me after the talk.
Sean Cai: 18:00 I’m around afterwards, too. Thank you.
