Fully autonomous robots are much closer than you think — Sergey Levine

Summary

Sergey Levine, one of the world’s leading robotics researchers and co-founder of Physical Intelligence (Pi), makes the case that fully autonomous general-purpose robots are far closer than most people think — his median estimate is 2030 for robots that can autonomously run a household and perform most blue-collar work. The key insight is not a single breakthrough moment but rather a “self-improvement flywheel”: once robots reach a basic level of competence and are deployed in the real world doing real tasks, they can continuously collect experience and improve, similar to how coding assistants have progressively increased the scope of tasks they handle.

The conversation dives deep into the technical architecture of Physical Intelligence’s models, which are essentially vision-language models with an added “motor cortex” — an action decoder that produces continuous robot actions using flow matching and diffusion. Levine explains how the same pretrained LLM backbone (Google’s Gemma) that powers language tasks is being used for robotics, and why this cross-modal transfer works: text provides high-level semantic abstraction that grounds the robot’s understanding, while the robot’s task-directed purpose provides the focusing mechanism that makes learning from visual data tractable.

The interview also covers the economics and geopolitics of the coming robot revolution. Levine argues that the transition will follow the same pattern as coding assistants — gradually increasing scope rather than a sudden switch — and that the biggest bottleneck is hardware manufacturing, not algorithms. He expresses concern about China’s dominance in hardware supply chains and argues that the US needs to think holistically about building a balanced robotics ecosystem, not just focus on AI algorithms. He also makes the fascinating point that robots, unlike phones or computers, can help manufacture more of themselves, creating a positive feedback loop that digital devices cannot.

Highlights

”Five years to fully autonomous household robots”

yt-dlp --download-sections "*10:40-11:40" "https://www.youtube.com/watch?v=48pxVdmkMIE" --force-keyframes-at-cuts --merge-output-format mp4 -o "48pxVdmkMIE-10m40s.mp4"

“Okay, five years. If you can fully autonomously run a household, you can fully autonomously do most blue-collar work. My median estimate is five years for robots that are able to do most blue-collar work in the economy.” — Sergey Levine, 10:40

”The self-improvement flywheel”

yt-dlp --download-sections "*4:35-5:40" "https://www.youtube.com/watch?v=48pxVdmkMIE" --force-keyframes-at-cuts --merge-output-format mp4 -o "48pxVdmkMIE-4m35s.mp4"

“Once we reach some basic level of competence, the robots will go out there in the world. And out there in the world, they can collect experience and get better. To me, what I tend to think about in terms of timelines is not the date when it’s done, but the date when the flywheel starts.” — Sergey Levine, 4:35

”Robots can make mistakes and learn from them”

yt-dlp --download-sections "*7:38-8:40" "https://www.youtube.com/watch?v=48pxVdmkMIE" --force-keyframes-at-cuts --merge-output-format mp4 -o "48pxVdmkMIE-7m38s.mp4"

“When you’re doing physical tasks, that stuff just happens more often than it does with language. If you answer a question wrong, it’s not like you can just go back and fix it. But if you’re folding the T-shirt and you drop it, you can reflect on that, figure out what went wrong.” — Sergey Levine, 7:38

”One second of context, folding inside-out shorts”

yt-dlp --download-sections "*46:20-47:30" "https://www.youtube.com/watch?v=48pxVdmkMIE" --force-keyframes-at-cuts --merge-output-format mp4 -o "48pxVdmkMIE-46m20s.mp4"

“What’s especially surprising is that this model only has one second of context. It’s seeing the image of what happened in the last second, and it’s crazy that it will just see the last thing that happened and know: fold it inside out, then fold it correctly. One second of memory is enough to execute on a minute-long task.” — Sergey Levine, 46:20

”Robots don’t have to be humanoid”

yt-dlp --download-sections "*70:30-71:30" "https://www.youtube.com/watch?v=48pxVdmkMIE" --force-keyframes-at-cuts --merge-output-format mp4 -o "48pxVdmkMIE-70m30s.mp4"

“People sometimes think about robots as copies of people. But it’s like your car or a bulldozer. You can put them into all sorts of weird places. You can make a robot that’s 100 feet tall. If you have the intelligence to power it, you can probably do a lot better than the human form factor.” — Sergey Levine, 1:10:30

”Robots can help manufacture more robots”

yt-dlp --download-sections "*75:20-76:20" "https://www.youtube.com/watch?v=48pxVdmkMIE" --force-keyframes-at-cuts --merge-output-format mp4 -o "48pxVdmkMIE-75m20s.mp4"

“Robots, unlike phones or computers, can help with the work of manufacturing more of themselves. Phones don’t themselves help with the work. They can help you or they can help others. But robots which would go into this feedback loop — that’s the key difference.” — Sergey Levine, 1:15:20

Key Points

• Physical Intelligence’s mission (0:31) - Building general-purpose robot foundation models that can control any robot to perform any task, which Levine considers a fundamental aspect of the AI problem.
• The flywheel, not the finish line (4:35) - The key timeline is when the self-improvement flywheel starts — when robots deployed in the real world begin collecting useful experience, not when the final product ships.
• One to two years to flywheel (5:17) - Levine hopes the flywheel could start spinning in one to two years with robots doing real tasks for real people.
• Scope expansion like coding assistants (8:34) - Robot capability will expand gradually, like how coding tools went from autocomplete to writing entire PRs, expanding the scope of what we trust them with.
• Five-year median for household autonomy (10:40) - Levine’s median estimate: five years to robots that can fully autonomously do most blue-collar work in the economy.
• LLM revenue mystery (11:43) - Despite obvious LLM capability, cumulative AI revenue is only $20-30 billion versus $30-40 trillion in total knowledge work. The scope has to grow.
• Manipulation easier to learn than driving (17:25) - Robots can learn manipulation through trial and error safely (unlike driving), because mistakes in folding laundry aren’t catastrophic.
• Common sense from foundation models (20:42) - Foundation models provide common sense — reasonable guesses about novel situations — which robots couldn’t do five years ago. This is transformative for safe exploration.
• Language as supervision signal (15:25) - Robots can now be supervised through language instructions (“pick up the cup, put it in the sink”), which dramatically broadens the learning signal.
• VLA architecture (27:28) - Pi’s model is a vision-language model with a vision encoder, language backbone (Gemma), and action decoder using flow matching for continuous control.
• Text provides semantic grounding (31:00) - Text is valuable not because of pixel-level transfer but because it provides high-level semantic abstractions that ground the robot’s understanding of tasks.
• Video models don’t transfer well to robots (34:00) - Unlike text, video data lacks the high-level abstraction needed. Video prediction without purpose is an unbounded problem.
• Purpose focuses perception (36:30) - Having a task to accomplish acts as a focusing factor for robot perception, similar to human attentional tunnel vision during goal-directed behavior.
• Emergent capabilities in robots (40:30) - Pi’s robots showed emergent behavior: when two objects stuck together, the robot spontaneously separated them — behavior never explicitly trained.
• Moravec’s paradox in action (46:57) - Easy things for humans (picking up objects, folding clothes) are the hard problems in AI. The cognitively demanding tasks actually require less real-time compute.
• Simulation as learned world model (57:59) - The most powerful simulation will come from learned models rather than hand-coded physics, because models know more than human engineers about the real world.
• Counterfactual reasoning is key (1:02:30) - At its core, planning requires asking “if I did X, what would happen?” Any mechanism for evaluating counterfactuals enables planning.
• Robot arm costs dropped 100x (1:13:18) - From $400K (PR2 in 2010) to $30K to $3K per arm today, and future costs could be a fraction of that.
• Hardware bottleneck, not algorithms (1:18:01) - The biggest bottleneck for the robotics explosion is manufacturing hardware at scale, not developing better algorithms.
• Education as the best defense (1:26:55) - Education provides flexibility and adaptability, which are the best defenses against the negative effects of rapid technological change.

Mentions

Companies

• Physical Intelligence (Pi) (0:00) - Robotics foundation model company co-founded by Levine; building general-purpose robot AI
• Google (28:30) - Their open-source Gemma model is used as the backbone for Pi’s robot foundation model
• Meta (38:20) - Mentioned as having tried to build robotic systems using large-scale video data
• Waymo (17:25) - Referenced as an example of autonomous driving’s long development timeline since ~2009
• Anthropic (5:46) - Claude mentioned as an example of LLMs being deployed and learning from real tasks
• Nvidia (50:10) - Referenced in context of GPU hardware for robot inference

Products & Technologies

• pi-zero (pi0) (27:28) - Pi’s vision-language-action foundation model for robotics
• pi0.5 (14:58) - Updated model that can be supervised through language rather than teleoperation
• Gemma (28:30) - Google’s open-source LLM used as the backbone in Pi’s models
• Flow matching / diffusion (28:50) - Technique used for the action decoder to produce precise continuous robot actions
• International Phonetic Alphabet (40:00) - Used as an example of emergent capability: an LLM can write recipes in IPA despite never being trained for this

People

• Sergey Levine (0:00) - Co-founder of Physical Intelligence, UC Berkeley professor, interview subject
• Sander Dieleman (30:20) - Researcher at DeepMind who argued text provides high-level semantic transfer that images/video lack
• Adnan (co-founder) (1:14:10) - Pi’s co-founder referenced as expert on hardware manufacturing costs and timelines

Surprising Quotes

“I think it’s very fundamental to AI. I actually think the other way around, that the robotics element of AI will also benefit the knowledge work element. Understanding the physical world gives you a level of understanding that goes beyond just what we can articulate in words.” — Sergey Levine, 56:20

“We experience this all the time. When we describe abstract concepts, we say, ‘This company has a lot of momentum.’ We use physical metaphors for inanimate objects. ‘My computer hates me.’ Our subjective experience of the physical world shapes how we understand everything else.” — Sergey Levine, 56:40

“There’s been no shortage of psychology experiments that show that people have a form of tunnel vision where they will literally not see something right in front of them if it’s not relevant to what they’re trying to do. There must be a reason why people do that. Maybe seeing more is better than seeing less. But it must be darn important for survival to focus.” — Sergey Levine, 36:50

“If I locked you in a room for a year with a TV showing live feeds of different sporting events, after that year I told you ‘Okay, now your job is to go out and play basketball’ — that’s pretty dumb, right? Whereas if I gave you a basketball and let you study up, now you have a chance.” — Sergey Levine, 37:30

“If there’s one thing that I’ve learned, it’s that things rarely evolve quite the way that people expect. We should think about the journey rather than planning ahead for an end state. Things evolve in all sorts of unpredictable ways, probably not the places we expect first.” — Sergey Levine, 1:26:10

Transcript

0:00 Today I’m chatting with Sergey Levine, who is the co-founder of Physical Intelligence, which is a robotics foundation model company, and generally one of the world’s leading robotics researchers. Sergey, thank you for coming on the podcast. Thank you for the kind introduction.

0:21 Before I pepper you with questions, I’m wondering if you can give us a sense of where Physical Intelligence is at right now. What does the progress look like? Physical Intelligence aims to build a general-purpose robot foundation model. That basically means general-purpose software that can control any robot to perform any task. I actually consider this to be a very fundamental aspect of the AI problem, encompassing all AI technology. If you can make AI that general, then you can do, hopefully, everything.

0:58 Where we’re at right now is that we’ve built out a lot of the basics. We have demos that are pretty cool. They work pretty well. We can get a robot to go into a new home and try to clean up the kitchen. Physical Intelligence right now is really in an early stage. It’s just putting in place the basic capabilities and then we’ll tackle all these really tough problems.

1:29 I got a chance to watch some of the robots, and one was assembling a box using grippers. I couldn’t fold the box even with my hands. As we think about when we get to the full robotics explosion, what is the thing that needs to happen? There are a few things that we need to get right.

1:52 In the beginning we really want to make sure that the foundation models we’re developing have the ability to tackle a wide range of household tasks. As you mentioned, folding a box, folding different items of clothing, making a coffee, that sort of thing. That’s the basic version. The results we’ve been able to show are pretty cool, but the end goal is much bigger.

2:21 The end goal is to just confirm our initial thesis and then scale. From there, there are a number of really hard challenges. Sometimes when results get abstracted to the level of a demo video and it’s like, “Oh, that’s cool” — that’s really just a very early and basic version of what I think is to come.

2:41 The grand vision is that you tell it like, “Hey, please fold my T-shirt.” But beyond that: “Hey, robot, you’re now doing household management. I like to have dinner made at 6:00 p.m. I like to do my laundry on Saturday, so make sure that’s ready. By the way, check in with me every Monday to plan the week’s grocery shopping.” That’s the prompt. Then the robot operates for a week.

3:12 That’s the duration of the task. And for that to be successful, it should be a lot bigger than just folding a shirt. It should have the understanding of the physical world, the common sense to pull in more information if it needs it. “Can you make me this type of salad?” It should look it up, go and buy the ingredients. That requires common sense. It requires understanding edge cases and how to handle them intelligently. It requires the ability to improve continuously.

3:52 There’s a lot more that goes into this. To do all of that, you need to leverage prior knowledge and foundation model capabilities. This grand vision, what year? If you had to guess?

4:12 I think it’s something where it’s not going to come out of a laboratory and then it’s done and then comes to market. Again, it’ll be the same as what happened with LLMs. Once we reach some basic level of competence, the robots will go out there in the world. And out there in the world, they can collect experience and get better.

4:35 To me, what I tend to think about in terms of timelines is not the date when it’s done, but the date when the flywheel starts basically. That could be very soon. There’s a trade-off: the more capable you make the system before deploying it, the earlier you can get it out into the real world productively.

5:04 We’re already trying to figure out what’s the minimum viable set of capabilities that could allow us to start spinning the flywheel. In terms of tasks that you care about, that you would want to see — I don’t have a precise number, but I’m really hoping it’ll be more like one or two years, but it’s hard to say.

5:23 Out there means what? What is “out there”? It means doing tasks that you actually care about, that you want done. Real tasks, for real people that want it done. That hasn’t resulted in some sort of flywheel yet with LLMs. Nobody would say that there are companies where now Claude is learning how to do every single job in the economy from scratch.

5:55 Well, I think it’s actually very close to that. Many organizations are working on exactly this. It’s not an automated flywheel yet, but everybody who’s deploying an LLM is of course collecting data on how users interact with it and using that to modify its behavior. It’s just that the details of extracting those supervision signals get complex and stability becomes a challenge. It takes the community collectively to get their hands on it.

6:51 Or do you think that with robots specifically, where the robot can take actions in the world and use them as a reward, the whole thing will be easier? I don’t think there’s a profound difference, but there are a few small differences that help. Especially if you have a robot that’s doing a task for a person — it’s a person that’s supervising it or directing it. There’s a big incentive for the person to provide corrections and feedback.

7:30 There are a lot of dynamics where you can attempt something, fail, reflect on what happened, and try again. When you’re doing physical tasks, that stuff just happens more often than it does with language. If you answer a question wrong, it’s not like you can just go back and fix it. The person you told the answer to has already moved on. Whereas if you’re folding the T-shirt and you drop it, you can reflect on that, figure out what went wrong.

8:01 Okay, in one year we have robots doing some simple tasks. Maybe if you have some relatively simple automated factory process, like keep folding thousands of boxes or something. But the grand vision is a machine which will just run my house for me. What is the gap between this thing which starts the flywheel and this thing which is running the house?

8:34 It’s actually not that different from what we’ve seen with coding assistants — it’s about scope. Think about coding assistants. A few years ago, they could do a little bit of completion. Then they’ll try their best to type out the whole function for you. As that stuff progresses, then you’re willing to trust them with more. The very best coding assistance now — if you’re lucky — can put together most of a PR for you for something moderately complex.

9:10 We’ll see an increase in the scope that we’re willing to trust robots with. Initially the scope might be: you’re making the coffee or something. Then as the models get more common sense and a broader repertoire of tasks, the scope grows. Now you’re running the whole coffee shop.

9:31 I get that there won’t be a specific moment, but if you had to give a year for your median estimate? My sense there too is that this is a single-digit-years thing rather than a double-digit thing. The reason I won’t pin it down further is because, as with all research, it does depend on getting some things right. My answer in terms of the nature of those questions: they don’t require profoundly, deeply different ideas. They require putting together the kinds of things that we already know.

10:09 That’s difficult — it’s not easy. It’s a deep and profound problem. But I think we kind of know roughly the puzzle pieces. If we work on it and we’re a bit lucky, single-digit is reasonable. Let me do a binary search until I get a year. What’s your median estimate? I know there’s a range.

10:40 Okay, five years. If you can fully autonomously run a household, you can fully autonomously do most blue-collar work. My median estimate is five years for robots that are able to do most blue-collar work in the economy.

11:04 Let me push on this because I think it’s really interesting. We consider the analogy to coding assistants. I don’t think anyone today believes that there’s a switch that gets flipped and suddenly all software engineers get fired. It actually makes a lot of sense that the first people to be most affected are software engineers, whose productivity is most directly amplified.

11:34 Separate from the question of whether people’s jobs are displaced, what will the economic impact be in five years? With LLMs, the relationship between the demonstrated capability and actual economic value has been sort of mysterious. You can have a conversation where it really feels like it can do anything. It’s obviously doing a bunch of coding, et cetera. But AI revenues are cumulatively on the order of $20-30 billion, which is tiny compared to all knowledge work, which is $30-40 trillion.

12:20 Is robotics going to be in the same position as what LLMs are in now, or is it more like we have robots doing a whole bunch of real work? I think the answer will come down to this question of scope. The reason coding assistants are most impactful for software engineering is because they’re good within a certain scope. Those limits are increasing. I think there’s no reason that we wouldn’t see similar things with robotics.

12:51 The scope will have to start out small. There will be certain things these systems can do very well and certain things they can’t. The distinction is really important. The scope will grow. Some of that productivity will come from full automation. Some of it will come from the people using the robots being more productive.

13:16 There’s so many things where even a small increase matters. But you want to understand something which fundamentally changes the equation versus something which has a small increase. Where LLMs are right now in terms of the share of economic value — maybe 1/1000th of the knowledge work that happens in the economy. That fraction will grow.

14:02 What percentage of all physical labor can be done by robots in five years? Off the cuff, I don’t have a sufficient understanding to give a precise number. But I’d say a meaningful fraction of all physical labor. It’s much easier to get effective systems rolled out when it’s human plus robot, not pure automation. I think we’ll see the same thing with automation — hybrid human-robot teams will create more value than just human or just robot. That just makes it easier to get all the technology bootstrapped.

14:44 There’s a lot more potential for the robot to learn when it’s working alongside a human. Because a human can label what’s happening and the human can give hints. When we were working on the pi0.5 project, we initially controlled our robots with teleoperation. At some point we actually realized that once the model was good enough, by supervising through language — literally instructing it through language — you get better learning data.

15:25 There’s a baseline of capability before you can do that, but once you have that, telling the robot, “Okay, now pick up the cup, now bring it to the sink,” just with words already, actually gives the robot a powerful learning signal. Now imagine what this implies for deployment at scale. Learning for these systems is not just learning from teleoperation — it’s also learning from words, from observing what people do, from the kind of interaction you have doing a job together with somebody else. The prior knowledge that comes from these big foundation models lets you understand that interaction dynamic.

16:08 Human plus robot deployments will be incredibly important for getting the technology bootstrapped.

17:25 In terms of robotics progress, autonomous driving has been around for more than 15 years. Wasn’t it in 2009 that Waymo launched? I remember watching demos of self-driving cars as a teenager. Only now do we have them actually deployed. Maybe it’ll be many more years before robots are everywhere. You’re saying five years, but actually will it just feel like 20 years?

18:14 That’s a really good question. One of the big differences has to do with the technology for machine learning that’s available today. Principally for autonomous driving, perception was the bottleneck. For robots, it can mean a lot more than just perception. The trouble with perception is that it’s one of these things where you can build a good demo with a somewhat engineered system, but it doesn’t generalize.

18:52 Now at this point in 2025, we have much more robust perception systems and, more importantly, systems for understanding the world around us. That gives us a much better starting point today. So 2025 is a better year to start robotics than 2009.

19:16 There are things about robotics that are a bit different than driving. Driving is actually a much harder problem in some ways, but manipulation is a problem space where it’s easier to get rolling. To give you an example, if you’re learning to drive, it’s crazy to learn how to drive on your own. You would not trust your teenage child to just drop them in the car and say, “Go for it.”

19:50 Humans need a long amount of time to learn about the world. You wouldn’t put a five-year-old in a car. But if you want somebody to clean up the kitchen — you would probably be okay with a child trying to help, sitting next to them with a brake, so to speak.

20:14 With robotic manipulation, there’s potential to learn safely from mistakes. When you make a mistake and correct it, well first of all nobody died, but you’ve also gained knowledge that allows you to do better next time. With driving, because of the dynamics of how it works, you can’t always survive the mistake and then learn from it because the mistakes can be fatal.

20:42 This is where the next big thing comes in: common sense, meaning the ability to make reasonable guesses about situations that you haven’t encountered, but that do not lead to catastrophic failures. That’s tremendously important and it’s something we had no idea how to do about five years ago. Now foundation models can answer questions and they will make reasonable guesses.

21:16 You can say, “Hey, there’s a puddle on the floor. What’s going to happen when I walk through it?” No autonomous car in 2009 would have been able to reason about that. Common sense plus the ability to make safe mistakes — that’s sounding an awful lot like what a person does when learning. All of that doesn’t make robotic manipulation easy, but it makes it a problem where you can start with a smaller scope and then grow from there.

27:28 How does the pi0 model work? What we have basically is a vision-language model with an action decoder. To give you a fanciful brain analogy, this is basically an LLM that has had a little pseudo visual cortex grafted onto it and a little motor cortex. Our models have a vision encoder, the language backbone, and an action decoder essentially.

28:08 How it works is it reads in the sensory information from the cameras, processes it through the language backbone — which could involve outputting intermediate reasoning steps. It might think to itself, “I need to pick up the dish and I need to pick up the sponge.” Eventually it works its way through that reasoning to the action expert, which produces continuous actions. Those actions are continuous, they’re high frequency — they’re not text tokens. But it’s still an end-to-end transformer.

28:40 It corresponds to a mixture-of-experts architecture. First some text tokens predicting “I should do X thing,” then some action tokens showing what it actually does, then more text description, more action tokens. With the exception that the action generation doesn’t use standard token prediction. It actually uses flow matching and diffusion so that you can be much more precise with your actions for dexterous control.

29:10 We’re using the open-source Gemma model, which is Google’s contribution, and then adding this action expert on top. The convergence in different areas of AI is based on not only the same ideas but literally the same models. You can just use an open-source LLM and build robotics on top. You naively might think that there’s a separate area of research for robotics and a separate area for language — but it’s literally the same. The considerations are the same, even the weights are the same.

29:48 One theme here that is important to keep in mind is why these pretrained models are so valuable: because the AI community has invested enormously in training them. A lot of what we’re getting from the pre-trained model is abstracted knowledge. It’s a little bit abstracted — roughly where things are in the world, what objects are. But if I had to summarize in one sentence what recent innovations in AI give to robotics — it’s not just the fact that the model is the same model. It’s that ability to leverage abstract knowledge that can come from language.

30:20 I was talking to this researcher, Sander Dieleman at DeepMind. He made the point that the reason training on video and images doesn’t seem to necessarily transfer well to downstream tasks is because images are represented at the pixel level. His argument is that text has this high-level semantic abstraction, whereas images and videos are just compressed pixels. Text provides high-level semantic information. But with images, there’s no transfer learning at the level of pixels.

31:20 Obviously this is super relevant to robotics. Your hope is that by training the model jointly on language data, on visual data generally — maybe even from YouTube or other video sources — plus action information from the robot itself, all of that combines into something greater than the sum of its parts.

34:00 I have some bad news and some good news about video data for robotics. This is really getting at the core of a long-running question. In some ways, the idea of getting robots to learn from video is even older than the idea of getting them to learn from text. The text stuff turned into practically useful systems faster. Video is great for visual understanding, but where you can ask a model to do stuff beyond its training data, language clearly has better transfer.

34:50 The key is what you’re trying to do. Imagine pointing a camera outside this building — you see the water, cars driving around, people. If you want to predict what will happen in the future, you can make high-level predictions about people’s behavior. But you could also say, “Let me understand everything about water at the molecular level.” You could go super deep on that. The problem is just so much stuff that even if you’re trying to predict 100% of something, you’ll never get there.

35:50 Whereas with text, it’s already been abstracted. The representations are already there. They focus on what really matters. That’s the good news for robotics: we don’t have to just get everything from pixels. When you have a robot, it has a purpose, and its perception is in service of that purpose. That is a really great focusing factor.

36:30 Literally what you see is affected by what you’re trying to do. There’s been no shortage of psychology experiments that show that people have a form of tunnel vision where they will literally not see something right in front of them if it’s not relevant to what they’re trying to do. There must be a reason why people do that. Maybe seeing more is better than seeing less. But it must be darn important for survival to focus.

37:10 Robots will have that focusing mechanism because they’ll have a purpose. The fact that video models aren’t as useful by themselves is not a deal-breaker. So much of the data you will have to use involves the robot actually interacting with the world. Ideally, you just want to be able to throw in every video ever recorded and have it learn how the physical world works. Just see humans performing tasks. But it’s hard to learn just from watching.

37:30 Let me put it this way. If I locked you in a room for a year with a TV showing live feeds of different sporting events — after that year, I told you, “Okay, now your job is to go out and play basketball” — that’s pretty dumb, right? Whereas if I told you, “Here’s a basketball, go practice,” and then I let you study up, now you have a chance.

38:00 There’s a very real challenge here. But there’s also a lot of potential for foundation models to leverage passive observation data, including video, because robots understand what they’re looking at through the lens of their own interaction, from controlling robotic systems, because they know what they’re trying to do. I don’t think it solves the whole problem, but we’ve already seen the beginnings of transfer from video to robot control.

39:20 Famously, LLMs have all these emergent capabilities because somewhere in internet text is the data that enables a certain kind of thing. With robotics, you are collecting all the data manually. So you won’t accidentally discover some capability that is somewhere in the dataset. Which seems like it should make it harder to get emergent, out-of-distribution capabilities.

39:50 Will the next 5-10 years of robotics be like this: each subtask has to be manually demonstrated, and then it’s very hard to actually automate the full job? If you think about what a chef does, very little of it involves actually cooking. You got to move around, you got to reach for things, go between the counter and the stove, et cetera. Will there just be this long tail of subtasks you keep adding episodes for manually?

40:00 There’s a subtlety here. Emergent capabilities in LLMs emerge because of the fact that internet data has a lot of stuff in it. Once the model reaches a certain scale, it can combine things. There was a cute example that one of my students found. You know what the International Phonetic Alphabet is? It’s an alphabet that is pretty much only found in dictionaries for the pronunciation of individual words. But you can ask an LLM to write a recipe for making some meal in International Phonetic Alphabet — and holy crap, it can do it. That is definitely not something that was in the training data.

40:45 It’s putting together things you’ve seen in new ways. Because yes, you’ve seen different words written in IPA and you’ve seen recipes, and the model can compose them. That’s actually where the real value of scale comes from. Because of this, in principle, if we have a large enough dataset of diverse robotic skills, the model should figure out that those skills can be composed in novel ways as the situation calls for it.

41:10 We’re already seeing this even with our current models. Looking back five years from now, we’ll probably see much more of it. But we’ve already seen what I would call emergent compositional behavior. When we were playing around with our model in the lab, we actually discovered this by accident. Two objects were stuck together, and the robot tried to pick one out of the bin instead of both. The other one gets in the way — it picks up both, separates them, puts one back.

41:45 We didn’t know it would do that. Holy crap. And it does that every time. It’s doing its work. A shopping bag tips over, it picks it up and puts it back. We didn’t tell anybody to collect data for that. But somebody in the dataset maybe accidentally or maybe intentionally picked up a shopping bag. This kind of compositional ability emerges when you do learning at scale. That’s where remarkable capabilities come from.

42:15 I had an example like this when I visited Pi’s office. It was folding shorts. I don’t know if inside-out shorts were in the training set, but just for fun I took one pair and turned them inside out. Then the robot was able to understand that it first needed to fold them right-side out, then fold them normally. The grippers are just like this, two simple grippers. It’s actually shocking how dexterous it is.

42:50 But it understood that it first needed to fold the inside-out shorts right-side out. What’s especially surprising is that this model only has one second of context. They’re not observing hundreds of thousands of tokens like an LLM. They’re not observing their own chain of thought about how to code something up. It’s seeing the image of what happened in the last second, and it’s crazy that it will just see the last thing that happened and know: fold it inside out, then fold it correctly.

43:30 One second of memory is enough to execute on a minute-long task. That’s very surprising but also points to why it works in the first place and why it’s possible to do physical work without much context. It’s not that there’s something good about having less memory. Adding memory, adding longer context — all those things will make the model better. But the important thing for the kind of skills we need first, at some level, comes back to Moravec’s paradox.

44:00 If you know one thing about robotics, that’s the thing to know. The easy things are hard and the hard things are easy. The stuff we take for granted — like picking up objects, seeing, walking — those are all the hard problems in AI. Things like playing chess and doing calculus, those are actually the easy problems.

44:30 The cognitively demanding tasks that we think require hard work — the ones where you say “Oh man, I’m sweating, I’m working hard” — those are actually the tasks that require lots of stuff in memory, lots of stuff in our minds. When you’re having a complicated technical conversation you have to keep all those puzzle pieces in your head. But when you are an Olympic swimmer swimming your laps, you’re in the zone. People even say it’s “in the flow” — like you’ve practiced it so much you’ve baked it into muscle memory. You don’t have to think carefully about each movement.

45:10 It really is just Moravec’s paradox playing out. That doesn’t mean that we don’t need the memory eventually. But the level of dexterity and physical proficiency we need to get right first — and then gradually go up that complexity ladder into reasoning, into context, into planning. That stuff will be important too.

45:37 There are several things which all take more compute during inference time. You have the inference speed — humans are very fast, we can react to things extremely fast. For the kind of robot which is just cleaning up, most of the information it needs came from things that happened minutes ago or hours ago, not split-second reactions. Then there’s thinking about the next task it’s doing.

46:10 At least with LLMs, we’ve seen that there’s massive headroom for scaling all of these dimensions. Currently the model has about 2 billion parameters. You have a second-long context. Each of these dimensions has orders of magnitude less than what seems to be the human equivalent. The human brain has vastly more parameters than this model.

47:30 Improvements across all of these three dimensions — model size, context length, inference speed — will help. But there are interesting trade-offs. Increasing one reduces the amount of compute you need for others. How are we going to solve this? There are fascinating systems problems ahead.

48:00 The right architecture for a practical, affordable low-cost system would be to have the robot operate at different levels of intelligence depending on connectivity and compute availability. You could imagine in the future you’ll have a robot that, when connectivity to the cloud is very good, can be a little smarter, and when it’s not, falls back to a dumber reactive mode. There’s a lot of cool stuff to do there with representations, figuring out how to compress temporal information.

49:20 This raises an interesting question about where the compute lives. Most effective LLM models are being run in big data centers, serving lots of users at the same time, not locally. In this robotics world, should we expect $50,000 GPUs per robot, or centralized compute? You need connectivity everywhere. You’re streaming video information back and forth.

50:00 If I were to guess, I’d say we’ll see both — low-cost systems with local processing for simple reactive tasks, and cloud-based compute for harder reasoning. In settings where you have good connectivity, that works great. In places where you can’t rely on connectivity, those will need more onboard compute.

50:30 From a technical standpoint, while a real-time system obviously needs to be fast, the amount of thinking you need to do for different types of actions varies enormously. We see this in humans and animals. There are real neural correlates of planning. Activity builds up in advance of a movement. The shape of the movement correlates with what the plan was. You put something in place and then unroll that process, and that’s the movement. You batch up the planning and then execute with less processing.

51:20 You’re just reacting at a different level of abstraction. Maybe I’m doing feedback on the position of my gripper. At a lower frequency, I sort of plan out the overall trajectory. This comes back to representations — figuring out what level of abstraction is sufficient for planning in advance and what requires a tight feedback loop.

53:30 You had a couple of lectures from a few years back where you were very optimistic that RL would be in many cases better than imitation learning. Now Physical Intelligence is largely doing imitation learning. Maybe this has changed. Maybe it hasn’t changed. Why can’t you do RL yet?

54:00 In order to effectively learn from your own experience through RL, it’s really important to already know a lot. Otherwise it takes far too long, just like it takes a baby a very long time to learn very basic things. Once you already have some knowledge, then RL becomes very powerful. The purpose of training the models with supervised learning, with demonstrations, is it provides the prior knowledge so they can then be improved with RL.

54:40 This is not a new idea. LLMs start off being trained with supervised next-token prediction. That provided an excellent starting point for RL fine-tuning. I would expect basically any foundation model to follow the same trajectory. We first build out the foundation with imitation learning. The stronger that foundation gets, the more effectively we can do RL with much more accessible training.

55:15 Will the best model for knowledge work also be a robotics model? The reason I ask is, so far we’ve seen advantages from general models. Will robotics fall into this bucket? I really hope that they will actually be the same. I think it’s very fundamental to AI. I actually think the other way around, that the robotics element of AI will also benefit the knowledge work element.

56:00 There are two reasons for this. One has to do with representations and focus. It’s very hard to figure out what’s relevant in raw visual data unless you have a task to do. Having a task structures your perception and allows you to more fruitfully utilize other data sources. The second one is that understanding the physical world provides a level of understanding that goes beyond just what we can articulate in words.

56:40 We experience this all the time. When we describe abstract concepts, we say, “This company has a lot of momentum.” We use physical metaphors for inanimate objects. “My computer hates me.” Our subjective experience of the physical world shapes how we understand everything.

57:59 What about simulation? In traditional robotics, simulation was a way to inject human knowledge — experts would write physics equations, code it up, and that gives you training data. But increasingly what we’re learning from video generation research is that probably the most powerful way to create simulation is through learned models. The model probably knows more than a person writing equations about the real world.

58:40 Where does that model get its knowledge? From real data. So in a sense, a very powerful simulation requires a foundation of real-world data. But at that point it almost doesn’t matter whether you call it simulation or model-based RL. What’s going on with that system is that it’s processing information from the world. Whether the way to process that information is through explicit simulation or some model-free method is kind of irrelevant.

1:00:00 Whatever we’re doing when we plan — if you had to make an ML analogy, what is it? I think it looks an awful lot like simulation. It looks an awful lot like playing through possible scenarios and generating new statistically similar experience. Simulation through a learned model is part of how planning works.

1:02:30 Something that’s even more fundamental than simulation: at its core, regardless of how you do it, planning requires counterfactual reasoning. You basically have to ask yourself, “If I did X, what would happen?” You have to answer that question somehow. Whether you answer it through a learned simulator, or through some other method like a reward function — as long as you have some mechanism for evaluating which counterfactual is better, you’ve got it. You don’t necessarily need to do really good simulations. You just need to evaluate counterfactuals.

1:03:30 Stepping into the big picture again. When we think about when this robot economy really takes off — there’s one argument that the key question is understanding how fast AGI will proceed in the digital realm. But also, if you just extrapolate out the capex spending, many people have estimates in the hundreds of billions or trillions of dollars per year. The marginal capex per year for AI infrastructure is $2-4 trillion dollars a year.

1:04:20 That’s data centers to build, actual chip foundries to build, power plants to build. I am very curious about whether by 2030, the big question will be: can robots assemble solar panels next to the data center or assemble the racks? Will they be good enough to help significantly in that process?

1:04:50 That is a more ambitious way of thinking about it than just household tasks. The good thing is that the AI infrastructure buildout generates huge demand for robotic labor, which will also mandate a lot of capex for robot factories. It’s really an explosion across the whole stack.

1:05:20 Can robots speed that up? There’s a tendency sometimes to think about robots as copies of people. People are people and robots are robots. But it’s like your car or a bulldozer. You can put them into all sorts of weird places. You can make a robot that’s 100 feet tall. If you have the intelligence to power it, you can probably do a lot better than the human form factor for many industrial tasks.

1:06:00 It can be a big productivity boost for real industrial applications that are very difficult for humans to solve. That doesn’t solve all the problems of building data centers by any means, but you could imagine putting them in extreme locations because the robots don’t have to worry about heat or cold.

1:09:18 There’s the question of where the software is versus how many physical robots we will have. At Physical Intelligence, we’re building tabletop robots for now. How many will there be by 2030? There’s a question of economies of scale in robotics so far. Cost has been coming down, and probably would continue in the long term.

1:10:00 When I started working in robotics in 2010, there was a robot called a PR2 that cost $400,000 to purchase. Later I bought robot arms that were $30,000. At Physical Intelligence, each arm costs about $3,000. Future costs could be a small fraction of that.

1:10:30 There are a few things driving cost down. Custom-built, high-end research hardware is obviously much more expensive than more productionized hardware. As we get better at building actuated devices, costs fall. There’s also a software element. The smarter your AI is, the less precise the hardware needs to be. You don’t need motions that are highly repeatable if the AI can compensate. AI also makes robots more affordable and more reliable.

1:13:18 Do you think it will cost hundreds of dollars or thousands for a robot in the future? That is a great question for my co-founder, Adnan, who’s one of the best people in the world to ask that question. What I’ve seen has surprised me year after year with how fast costs come down.

1:13:50 Is it more than a million robots currently in the world that are the right kind to think about? Very few because they are not currently general-purpose. Less than 100,000? Okay. And we want billions of robots eventually.

1:14:10 If you’re just thinking about the infrastructure buildout for this explosive AI growth, not only do you need the data centers but you need a lot more labor to power this AI boom. Where does that labor come from when there’s a lot of demand?

1:15:00 A particularly important question is: can AI affect how we think about hardware? Making the hardware reliable — making it a thing that doesn’t break all the time — is really important. You want more than bare minimum functionality. You need good functionality that’s reliable. But there are some things that we probably don’t need. The hardware doesn’t need to be super precise, because we know that the AI can compensate.

1:15:20 I really think about robots in terms of standardized interfaces. I don’t think that we will have the one ultimate robot form factor. What we will have is a bunch of things that plug together. Just like good smartphones — there’s a standardized interface and then optional components depending on the need. There will be a lot of innovation where the brain — the AI — can be plugged into any robot to endow it with intelligence, and different people can innovate on how to get the hardware right.

1:16:10 Is there a single manufacturer that will dominate? Not right now. I would really like to see a world where lots of companies innovate on hardware. What is the biggest bottleneck in the hardware? Right now, mainly because things are changing so fast, the biggest bottleneck is the number of robots, which translates to the amount of data. Having lots of data requires having lots of robots that are low cost, because then I can deploy more of them. Reliability is important, more than raw precision.

1:18:01 This is a question I’ve had for a lot of guests. Do you find that a bunch of the actual source materials — other than chips obviously — the wafers for solar panels, the components, et cetera, are manufactured in China? Obviously robot arms are also manufactured in China. In this world where the value that robots and AI can produce dwarfs the value that a human worker can produce, the big bottleneck becomes: how many can you manufacture? We have the software. We just need the hardware.

1:19:00 If China dominates hardware manufacturing, and you come up with the algorithms in the US, why don’t they just win by default? I’ll start with the broader themes. If you want to grow your economy, you need a highly educated workforce with automation that multiplies the amount of work each person produces. Automation is what multiplies the amount of stuff that gets done. It’s the same as LLM coding tools multiplying the productivity of a software engineer.

1:20:00 There’s a lot of complexity in how you make this an appealing journey to society, how you manage the transition. All of that stuff is pretty complicated. It requires a lot of really good decisions. Building a balanced robotics ecosystem, supporting both hardware and software. I don’t think any of those challenges are insurmountable. It just requires a degree of long-term thinking.

1:20:40 What makes me really optimistic about the US is that we can all agree that the end state we want is a highly productive society where we have highly automated, high-output industries. Because that end state seems very good, at some level there should be a lot of political will to get there. Then from there we have to solve for the journey. That’s not easy. There’s a lot of decisions to be made in terms of private industry, in terms of policy. But I’m very optimistic about it because the light at the end of the tunnel is in the right direction.

1:21:20 If the value is bottlenecked by hardware production, what is the path by which hundreds of millions of robots are being manufactured in the US or with allies? For the specifics, I’m not the most qualified to speak. But the ingredient here that is important is that if producing robots is itself physical work, robotics should help with that. Robots can help build more robots.

1:22:00 As with all circular things, you have to bootstrap it somehow. But it seems like an easier problem to solve than the problem of digital devices. Phones, computers — phones don’t themselves help with the work of manufacturing more phones. They can help you or they can help others. But robots which would go into this feedback loop — the robot factories already exist in China — it seems like the US could build that capability.

1:22:40 Then there’s a separate discussion about trade. Maybe China will continue exporting hardware to us. But when I talk to guests about different things, it’s surprising how often the answer to “what’s the key bottleneck?” is some critical material or component that China is the 80% world supplier of. It’s really important to get right here.

1:23:20 AI is tremendously exciting, but we should remember it’s not the only thing that we need to do. We need to think about manufacturing, supply chains, priorities, investment. Just as an example, at Physical Intelligence, we build a lot of our own things and we want to build even more in-house. But that’s just us. For the United States as a whole, we need to think about these things holistically.

1:24:00 It is easy to get distracted sometimes by a lot of progress in one area like AI. There’s a hardware bottleneck with compute and things like that. We need a holistic view of these things. People should be having more conversations about that.

1:24:30 How should society be thinking about the transition? There will be a period in which people’s work gets augmented. Then there will be a boom in the economy where we’re building all the infrastructure. Eventually humans can do things with their hands or with their minds. There’s not some secret third thing. It should be full automation of everything eventually.

1:25:10 Presumably there are ways to do this such that it goes well. But the end state, the light at the end of the tunnel, should be a very wealthy society with some redistribution mechanism.

1:25:40 At some level that’s a very broad statement. But if there’s one thing that I’ve learned, it’s that things rarely evolve quite the way that people expect. We should think about the journey rather than planning for a specific end state. Things evolve in all sorts of unpredictable ways, probably not the places we expect first.

1:26:10 I do think that it’s very important for us to set up the world around us in a way that is amenable to good outcomes. But we should really think about the journey, because things evolve unpredictably.

1:26:30 One thing I will say is that education is really, really valuable. It’s the best defense against the negative effects of change. Collectively as a society, the answer is more education. The things which are most beneficial from education are not the specific facts — because it’s really easy to educate AIs on specific facts. What education gives you is flexibility. It’s not about what you know, as it is about your ability to adapt. It has to be a good education.

1:27:20 Sergey, thank you for coming on the podcast. Super fascinating.