Ilya Sutskever — We’re moving from the age of scaling to the age of research

Summary

In this wide-ranging interview, Ilya Sutskever — co-founder of Safe Superintelligence Inc. (SSI) and former chief scientist at OpenAI — sits down with Dwarkesh Patel to discuss his view that the AI field is transitioning from the “age of scaling” to the “age of research.” Sutskever argues that while pre-training scaling yielded predictable returns from 2020 to 2025, the approach is running into data limits, and the next breakthroughs will come from genuinely new ideas rather than simply training bigger models. He draws an analogy between two types of competitive programming students — one who grinds 10,000 hours on a single domain versus one with natural “it” factor — to explain why current RL-trained models score well on benchmarks but fail to generalize in the real world.

The conversation takes a deep dive into the role of emotions and value functions in intelligence. Sutskever describes a case study of a brain-damaged person who lost emotional processing and could no longer make basic decisions, arguing this reveals how crucial value functions are for efficient learning. He contends that whatever makes humans sample-efficient learners — able to learn to drive in 10 hours, for instance — is not just evolution’s contribution to vision and motor control, but something deeper and more fundamental about how humans learn. This, he says, is the key unsolved problem, and whoever cracks it will unlock the path to superintelligence.

Sutskever also shares his thinking on alignment, deployment strategy, and SSI’s competitive position. He argues for building AI that cares about sentient life (not just humans, since AI itself will be sentient), discusses why SSI may or may not deploy products before achieving superintelligence, and gives a timeline of 5 to 20 years for human-level continual learners. He pushes back on the “million Ilyas in a server” model of recursive self-improvement, suggesting research may not be as parallelizable as people assume, and that true diversity of thought matters more than scaling up identical copies.

Highlights

”We are back to the age of research”

yt-dlp --download-sections "*21:16-22:10" "https://www.youtube.com/watch?v=aR20FWCCjAs" --force-keyframes-at-cuts --merge-output-format mp4 -o "aR20FWCCjAs-21m16s.mp4"

“Pre-training will run out of data. What do you do next? But now that compute is big, compute is back to the age of research. Up until 2020, from 2012, it was the age of research. From 2020 to 2025, it was the age of scaling. Now we are back to the age of research, just with big computers.” — Ilya Sutskever, 21:16

”A human being is not an AGI”

yt-dlp --download-sections "*49:38-50:40" "https://www.youtube.com/watch?v=aR20FWCCjAs" --force-keyframes-at-cuts --merge-output-format mp4 -o "aR20FWCCjAs-49m38s.mp4"

“If you think about the term ‘AGI,’ you will realize that a human being is not an AGI. A human being lacks a huge amount of knowledge and capability. Instead, we rely on continual learning. So let’s suppose that we achieve success — the question is, how do you define it? It’s going to be a 15-year-old that’s very eager to go, a great student, very eager.” — Ilya Sutskever, 49:38

”He couldn’t make decisions at all”

yt-dlp --download-sections "*11:35-12:25" "https://www.youtube.com/watch?v=aR20FWCCjAs" --force-keyframes-at-cuts --merge-output-format mp4 -o "aR20FWCCjAs-11m35s.mp4"

“I read about this person who had some kind of brain damage that took out his emotional processing. He still remained very articulate, and on tests he seemed to be just fine. But he didn’t feel anger, he didn’t feel animated. He couldn’t make decisions at all. It took him hours to decide on which socks to wear.” — Ilya Sutskever, 11:35

”I think it’s very possible from broad deployment”

yt-dlp --download-sections "*52:28-53:50" "https://www.youtube.com/watch?v=aR20FWCCjAs" --force-keyframes-at-cuts --merge-output-format mp4 -o "aR20FWCCjAs-52m28s.mp4"

“I think that it is likely that we will see very rapid economic growth. Once you get to a point where models can learn as well as humans, and you have many of them, then there will be rapid economic growth — unless there will be some kind of regulation. I think it’s very possible from broad deployment.” — Ilya Sutskever, 52:28

”The whole problem is the power”

yt-dlp --download-sections "*57:30-58:20" "https://www.youtube.com/watch?v=aR20FWCCjAs" --force-keyframes-at-cuts --merge-output-format mp4 -o "aR20FWCCjAs-57m30s.mp4"

“Future AI is going to be different. It’s going to be powerful. What is the problem of AI and AGI? The whole problem is the power. You’ve got to be showing the thing. I maintain that most people who work on AI also don’t fully appreciate what’s coming, because of what people see on a day-to-day basis.” — Ilya Sutskever, 57:30

”Ugliness, there’s no room for ugliness”

yt-dlp --download-sections "*94:29-95:30" "https://www.youtube.com/watch?v=aR20FWCCjAs" --force-keyframes-at-cuts --merge-output-format mp4 -o "aR20FWCCjAs-94m29s.mp4"

“How things should be — thinking from multiple angles and looking for what’s fundamental or not fundamental. Ugliness, there’s no room for ugliness. Correct inspiration from the brain. Multiple things have to be present at the same time. The more confident you can be in a top-down belief, the more you should keep going when experiments contradict you.” — Ilya Sutskever, 1:34:29

Key Points

• Model jaggedness problem (1:32) - Models seem smarter on evals but their economic impact is lagging. They can do amazing things on benchmarks, then fail at basic tasks like fixing a simple bug, suggesting something strange is going on with RL training.

• RL training makes models too focused (3:02) - Sutskever’s explanation for the gap: RL training makes models too narrowly focused on specific tasks, lacking the broad awareness that pre-training provides. RL environments are designed around eval benchmarks rather than general capability.

• Two students analogy (6:14) - Compares two competitive programming students: one who grinds 10,000 hours and becomes great at that specific domain, versus one who naturally has “it” and excels broadly. Current AI training is like the first student.

• Pre-training is not generalization (8:10) - Pre-training appears to generalize because there is so much data that almost everything appears somewhere in the distribution. It’s not necessarily generalizing better than RL — it’s just covering more ground.

• Emotions as value functions (13:26) - Emotions should be understood as value functions that guide decision-making. The brain-damaged patient who lost emotions but kept intelligence could not function, proving that value functions are essential for efficient learning.

• Value functions explained (13:51) - Sutskever explains how reinforcement learning currently works without value functions (naively, waiting until the end to score), and how value functions let you short-circuit learning by evaluating intermediate states.

• Age of scaling to age of research (18:50) - From 2012-2020 was the age of research; from 2020-2025 was the age of scaling where “one word — scale” was the insight. Now pre-training is running out of data, and we’re back to needing genuine research breakthroughs.

• What are we scaling? (22:17) - There is no clean scaling law for RL the way there was for pre-training. The transition from pre-training to RL means people are scaling rollouts, but the amount of learning per rollout is low. Sutskever would not even call it scaling.

• Human sample efficiency mystery (25:38) - Humans learn with extraordinary sample efficiency. A teenager learns to drive in about 10 hours. Whatever makes humans good at learning is not just evolution’s contribution — it’s something fundamental about human learning that AI does not yet have.

• SSI’s compute is comparable for research (40:30) - Despite raising less than competitors, SSI’s compute resources are comparable for pure research because other companies spend most compute on products, inference, and fragmented work streams.

• Case for and against straight-shotting to superintelligence (44:06) - The advantage of not deploying products is freedom from the “rat race.” The counterpoint is that deploying AI lets the world experience it, which is the only way to truly communicate what AI is and what it can do.

• AGI and continual learning (47:37) - The term AGI exists as a reaction to “narrow AI.” Pre-training made things broadly capable, fulfilling the promise. But humans are not AGIs either — they rely on continual learning. The real goal is an AI that can learn on the job like a very eager 15-year-old.

• Intelligence explosion from broad deployment (52:28) - If you deploy many continual learning agents across the economy, each learning their job and sharing knowledge back, you functionally have recursive self-improvement through software.

• AI safety and showing the AI (57:52) - Sutskever predicts companies will increasingly demonstrate AI capabilities to make people take the power seriously. He believes most AI researchers don’t fully appreciate what’s coming because they anchor on today’s models.

• Alignment and sentient life (1:01:21) - Rather than aligning AI to care just about humans, Sutskever argues for AI that cares about sentient life broadly, because the AI itself will be sentient. There will be trillions of AI agents and humans will be a very small fraction.

• Capping superintelligence power (1:03:16) - Sutskever thinks it would be really good if the most powerful superintelligence was somehow capped in its capabilities, to avoid continent-sized clusters becoming ungovernable.

• Evolution and social desires (1:12:16) - It is mysterious how evolution encoded high-level social desires (caring about status, being liked) into genes, since these require complex computation to evaluate. Evolution cannot simply “wire dopamine to a sensor” for social standing.

• 5 to 20 year timeline (1:22:16) - Sutskever estimates 5 to 20 years until human-like continual learners exist. He expects current companies to continue earning revenue even if they stall on capability, and that when the breakthrough comes, convergence will happen quickly.

• Research taste (1:33:11) - On what makes good research: obsession with correctness, inspiration from the brain, thinking from multiple angles, and “ugliness — there’s no room for ugliness.” Top-down conviction lets you persist when experiments fail.

Mentions

Companies

• SSI (Safe Superintelligence Inc.) (40:30) - Sutskever’s company, focused purely on research toward safe superintelligence rather than deploying products
• OpenAI (36:28) - Sutskever’s former company; referenced in discussions about the age of scaling and deployment strategy
• Anthropic (43:49) - Referenced alongside OpenAI as companies pursuing incremental deployment
• Meta (1:19:49) - Sutskever’s former co-founder left SSI to join Meta, which offered $32 billion valuation
• Google (36:28) - Referenced as one of the large institutions Sutskever has been at
• Stanford (36:28) - Referenced alongside Google and OpenAI as major AI institutions
• Thinking Machines (1:23:56) - Listed as one of the research-focused AI companies alongside SSI

Products & Technologies

• GPT-3 (19:28) - Referenced as marking the arrival of the scaling insight
• AlexNet (36:00) - Built on two GPUs; referenced as the beginning of the age of research in deep learning
• Transformer (38:40) - Built on 8 to 64 GPUs; used to illustrate that major breakthroughs don’t require massive compute
• o1 reasoning (39:07) - Referenced as a possible example of research breakthrough that didn’t require enormous compute

People

• Dwarkesh Patel (0:00) - Interviewer, host of Dwarkesh Podcast
• Ilya Sutskever (0:00) - Co-founder of SSI, former chief scientist at OpenAI, interviewee

Surprising Quotes

“I think like 5 to 20 years.” — Ilya Sutskever on timeline for human-like continual learners, 1:22:23

“A lot of people’s models of recursive self-improvement is we will have a million Ilyas in a server that are all doing research, leading to a superintelligence emerging very fast. The question is how parallelizable the thing you are doing is.” — Ilya Sutskever, 1:28:44

“The AI itself will be sentient. I think it’s an emergent property — the same circuit that we use to model ourselves, we use to model others.” — Ilya Sutskever on why AI should care about sentient life, 1:01:40

“There are more companies than ideas by quite a bit. People in Silicon Valley say that ideas are cheap. People say that a lot, and there is truth to that. But I say something like, ‘If ideas are so cheap, why do all companies do the same thing?’” — Ilya Sutskever on the age of scaling, 37:05

“We pursue a reward, and then the emotions change, and we pursue a different reward. Evolution is the same — evolution is smart in some ways, but very dumb in other ways. It’s going to be a never-ending fight.” — Ilya Sutskever on alignment challenges, 1:05:43

Transcript

0:00 You know what’s crazy? That all of this is real. Don’t you think so? All this AI stuff — isn’t it straight out of science fiction? How normal the slow takeoff feels. If you told someone about the percentage of GDP in AI, I feel like it would have felt like more. We get used to things pretty fast, it turns out.

0:32 What does that mean? It means that you see it in the news, and such dollar amount. That’s all you see. Should we actually begin here? Sure. About how from the average person’s point of view, this may be true even into the singularity. Okay, interesting. The path to not feeling different is, okay, such and such dollar amount of investment.

1:15 But I think the impact of AI is going to be felt. There’ll be very strong economic forces, going to be felt very strongly. I think the models seem smarter than their economic impact suggests. Yeah. This is one of the very confusing things. How to reconcile the fact that you look at the evals and you go, “Those are doing so well.” But the economic impact — it’s very difficult to make sense of. Models do these amazing things, and then on the other hand fall short.

2:20 An example would be, let’s say you’re coding. You go to some place and then you get a bug. “Can you please fix the bug?” And the model says, “You’re so right. I have a bug. Let me fix it.” Then you tell it, “You have this other issue,” and it says “Oh my God, how could I have done it?” It introduces the first bug again, and you can alternate between the two.

2:52 This does suggest that something strange is going on. I think the explanation is that maybe RL training makes the models too focused, a little bit too unaware, even though they perform well on benchmarks. Because of this, they can’t do basic things. When people were doing pre-training, the answer was everything — the next token answered everything. So you don’t have to think if it’s relevant or not.

3:44 But when people do RL training, they say, “Okay, we want to have this kind of performance, and that kind of RL training for that thing.” The question is, what environments do you train in that just produce new RL environments? There is such a huge variety of tasks in the real world.

4:12 One thing you could do, and I think this is what happens, is that people take inspiration from the evals. They want the model to perform really well when they release it. What would be the RL training that achieves that? I think that is something that happens, and if you combine this with the fact that generalization from RL may be limited, that has the potential to explain a lot of the gap between eval performance and actual real-world impact.

4:56 We don’t today even understand what we mean by generalization. It is the human researchers who decide what environments to train in. I think there are two ways to address what you have just pointed out. One is that you could simply become superhuman at a coding benchmark and hope models become more tasteful and exercise better judgment. The other is that you should expand the suite of environments such that you’re not just optimizing for the best performance in coding competition, but for broader application.

5:44 The question is, “Why should it be the case that being the best at coding competitions doesn’t make you a great engineer?” Maybe the thing to do is not to keep expanding the suite of environments, but to figure out an approach where training in one domain can improve your performance on something else.

6:14 Let’s take the case of competitive programming and imagine two students. One of them decided they want to be the best and they will practice 10,000 hours for that domain. They’ll learn all the proof techniques, and be very skilled at quickly solving problems. By doing so, they became one of the best. The second student just thought, “competitive programming is cool,” practiced much less, and they also did really well. Who’s going to do better in their career later on?

6:56 Right. I think that’s basically what’s going on. The way we train AI is like the first student, but even more extreme. We say, “We want to be good at competitive programming so let’s get a million problems,” and then let’s do some data augmentation to generate even more programming problems, and we train on that. With this analogy, I think it’s more intuitive — you’re learning all the different algorithms and all the different tricks. And it’s more intuitive that with this approach, the skill won’t necessarily generalize to other things.

7:42 What the second student is doing is different. I think they have “it.” The “it” factor — I remember there was a student like this. I think it’s interesting to distinguish the two approaches. One way to understand what you just said about pre-training is to say it’s actually not generalizing that much. It’s just that you get that 10,000 hours of practice in almost every domain somewhere in the pre-training distribution.

8:25 That there’s not that much generalization from pre-training. It’s not necessarily generalizing better than RL. The strength of pre-training is that A, there is so much of it, and B, you can find almost any kind of data to put into pre-training. The internet does include in it a lot of what people do — it’s like the whole world as projected by text. You try to capture that using a huge amount of data.

9:08 Because it’s so hard to understand the manner in which models generalize — whenever the model makes a mistake, could it be because that mistake was somehow supported by the pre-training data? I don’t know if I can add much more. I don’t think there is a definitive answer.

9:39 Here are analogies that people have proposed for what pre-training is. I’m curious to get your thoughts. One is to think about the first 18, or 15, years of a person’s life where they aren’t necessarily economically productive, but all that experience helps them understand the world better. Another is evolution — some kind of search for 3 billion years. I’m curious if you think either of these captures what’s happening. How would you think about what lifetime learning versus evolution corresponds to?

10:22 I think there are some similarities between both analogies. Pre-training is supposed to play the role of both of these. But there are big differences as well. The difference is actually very staggering. Somehow a human being, after even 15 years of experiencing much less data, they know much less but they know much more deeply somehow. They don’t make mistakes that our AIs make.

11:08 There is another question — is this contribution from evolution? The answer is maybe. But in this case, I remember reading about an interesting case. One of the ways you learn about the brain is by studying people with brain damage. Some people have the most strange symptoms, really interesting.

11:35 One case that I read about — this person who had some kind of brain injury that took out his emotional processing. He still remained very articulate and on tests he seemed to be just fine. But he didn’t feel anger, he didn’t feel animated. He couldn’t make decisions at all. It took him hours to decide on which socks to wear.

12:23 What does it say about the role of our built-in emotional processing? To connect to your question about pre-training, if we could get value functions out of pre-training, you could get that capability as well. Well, it may or may not be possible. What is “that”? Clearly not just directly an emotion, but some kind of value function-like thing which is telling you what matters and what doesn’t.

13:12 You think that doesn’t sort of emerge naturally? I think it could. I’m just not sure. But what is that? How do you think about emotions? It should be some kind of a value function thing, because right now, value functions don’t play a big role in AI training.

13:36 It might be worth defining for the audience what a value function is. Certainly, I’ll be very happy to do that. The way reinforcement learning is done today: you have your neural net and you give it a problem, tell the model, “Go solve it.” It takes hundreds of thousands of actions or thoughts or steps. At the end, the solution is graded. And then the score has to be distributed for every single action in your trajectory.

14:23 If the problem goes for a long time — if you’re training on really hard problems — the model will do no learning at all until you stumble on a solution. That’s how reinforcement learning is done naively. The value function says something like, “Here’s a little helper that will tell you if you are doing well or badly.” It’s more useful in some domains than others. In chess, when you lose a piece, you know you messed up. You don’t need to wait until the end of the game to know that what you just did was bad.

15:08 The value function lets you short-circuit the learning process. Let’s suppose that you are doing some kind of hard math problem and you’re trying to explore a solution space. After, let’s say, a thousand steps of thinking, you realize this approach is hopeless. As soon as you conclude this, you can propagate that signal back a thousand timesteps previously, when you made the initial choice. You say, “Next time I shouldn’t pursue this direction,” without waiting until you actually came up with the proposed solution.

15:56 The space of trajectories is so wide that you need intermediate feedback from a value function. But the value function itself might be wrong — you’ll have the wrong idea, then you’ll learn from that. This sounds like such a fundamental limitation. Sure it might be difficult, but my expectation is that a value function should become very important and will be used in the future, if not already.

16:37 Going back to the case of the person whose emotional center got damaged, it’s more evidence that the value function of humans is modulated by emotions in a fundamental way. And maybe that is important for efficient learning.

17:00 That’s the thing I was planning on asking you. There’s something really interesting about how simple our emotions are relative to how it’s impressive that they have this much utility. I have two responses. I do agree that compared to the kind of AI we are building, our emotions are pretty simple. They might even be so simple that maybe you could enumerate them. I think it would be cool to do.

17:40 I think there is a thing where simple things can be very useful, but simple things are also unreliable. One way to interpret what we are seeing is that we inherited our emotional system from our mammal ancestors and then fine-tuned a bit. We do have a decent amount of social emotions, but they’re not very sophisticated. And because they’re calibrated for a very different world compared to the modern one, they also make mistakes. For example, does hunger count as an emotion? It’s debatable. But the mechanism of hunger is not succeeding in guiding us correctly anymore.

18:50 People have been talking about scaling. Is there a more general framework? What are the other scaling axes? The way ML used to work is that researchers would come up with clever stuff and try to get interesting results. Then the scaling insight arrived. Scaling laws, GPT-3 — this is an example of how language can be so powerful. One word, but it’s such a powerful word. They say, “Let’s try to scale things.”

19:57 Pre-training was the thing to scale. The big breakthrough of pre-training is that it was a recipe. You say, “Hey, if you mix some compute and data and make the model a certain size, you will get results. Just scale the recipe up.” This is also great because scaling is a low-risk way of investing your resources. Compare that to research. If you research, you might spend years and come up with nothing. You know you’ll get something from pre-training.

21:02 Despite what some people say on Twitter, maybe it doesn’t matter — you can always squeeze more out of pre-training. But eventually pre-training will run out of data. What do you do next? Either you do some kind of extension of the recipe, or you do something new. But now that compute is big, we are back to the age of research.

21:24 Up until 2020, from 2012, it was the age of research. Now, from 2020 to 2025, it was the age of scaling — let’s add error bars to those years. The message was: scale more. Keep scaling. The one word: scale. Is the belief really that if you had 100x the scale, everything would be transformed? It would be different, for sure. But now we’re in the age of research again, just with big computers.

22:10 But let me ask you the question: What are we scaling, and what should we be scaling? I guess I’m not aware of a very clean scaling law for RL like the laws of physics which existed in pre-training for compute or parameters and loss. What should we be seeking, and how should we think about it?

22:40 We’ve already witnessed a transition from one scaling paradigm to another — from pre-training to RL. Now people are scaling RL. Some companies probably spend more compute on RL than on pre-training. RL can actually consume quite a bit of compute, because you need a lot of compute to produce those rollouts, and the amount of learning per rollout is low. So you need a lot of them. I wouldn’t even call it scaling.

23:27 The question is: Is the thing you are doing the most productive use of compute? Can you find a more productive approach? We’ve discussed the value function. Maybe once people get good at value functions, they can use their resources more productively. In terms of training models, you could say, “Is this research or is this scaling?” I think it becomes a little bit ambiguous.

23:59 In the age of research back then, it was people trying this and that and that. Let’s try that and that and that. I think there will be a return to that. But stepping back, what is the part of the value function idea that’s new? When you say value function, people think of reward models, LLM-as-a-Judge and so forth. But it sounds like you have something deeper in mind. Should we even rethink pre-training at all?

24:38 The discussion about value function — I want to emphasize that I think the value function will make RL more efficient, and I think that makes a difference. It’s not that you can’t do without a value function, you can do without, just more slowly. But the thing that’s really striking is that these models somehow just don’t generalize, and it’s not obvious why. That seems like a very fundamental thing.

25:18 There are sub-questions. There’s one which is about sample efficiency — why does it take so much more data for these models to learn than humans? There’s another question about the amount of data it takes and why is it so hard to teach AI new things. For a human, we don’t necessarily need a formal curriculum. I’m mentoring a bunch of researchers right now, and you’re just showing them code, and you’re showing them how you think. You’re teaching them thinking and how they should do research.

26:06 Perhaps these two issues are actually connected. I want to explore this second thing, which is more about general learning ability, which feels just like sample efficiency. One explanation for the human sample efficiency is that evolution has given us a lot. For things like vision, hearing, and motor control, there’s a strong case that evolution has given us a lot.

26:55 Robots can become dexterous too if you subject them to enough training. But to train a robot in the real world to be as capable as a person does seems very out of reach. All our ancestors needed to move around. So with locomotion, maybe we’ve gotten a lot from evolution. You could make the same case for vision. Children learn to drive after 10 hours of lessons. But our vision is so good.

27:35 I remember myself being a five-year-old. I’m pretty sure my car recognition was more than enough to drive. You don’t get to see that many cars — you spend most of your time in your parents’ house. But you could say maybe that’s evolution too. It still seems better than models.

28:04 Models are now better than the average human at language, math, and coding. But are they better than the average human at learning? What I want to say is that language, math, and coding — and whatever it is that makes people good at learning — is not just something simple, but something more, some fundamental thing.

28:30 Why should that be the case? Here’s one argument. People exhibit some kind of great reliability at learning new things. If this was a skill that our ancestors needed for many millions of years, hundreds of millions of years, then you could say humans are good at it because of evolution, and that’s encoded in some very non-obvious way. But if people exhibit great ability, reliability at learning things that really did not exist until recently, then maybe the answer is that humans might have just better machine learning, period.

29:34 What’s the ML analogy? There are a couple of interesting things. Human learning is more unsupervised. A child learning to drive — a teenager learning how to drive a car is not being trained with explicit rewards. It comes from their interaction with the world. It takes much fewer samples. It seems much more robust. The robustness is striking.

30:14 Do you have a unified way of thinking about what’s going on? What is the ML analogy that could explain this? One of the things that you’ve been asking about is how humans learn from their experience without an external teacher. They have a general sense — whatever the human value function is, it’s actually very, very robust.

31:00 Think of a teenager that’s learning to drive. They start to drive, and they can tell immediately how badly they are doing, how unconfident they are. The learning speed of any teenager is so fast. It seems like humans have some extraordinary capability. How they are doing it and why is it so hard for us when we’re training models to make them learn that fast?

31:28 That is a great question to ask, and it’s one I think about constantly. But unfortunately, we live in a world where there are things you can’t discuss freely, and this is one of them. I think it can be done. I think humans are a proof that it can be done. There is a possibility that the architecture of the brain matters. If that is true, and if that plays an important role — but regardless, I do think it points to a principle that I have opinions on, but which commercial considerations make it hard to discuss in detail.

32:32 I’m curious. If you say we are back in an era of research, what is the vibe now going to be? For example, even after AlexNet, the compute needed to run experiments kept increasing. Do you think now that this era of research will require less compute? Do you think it will require going back to smaller-scale experiments?

36:28 You were at Google and OpenAI and Stanford, these large institutions. What kind of things should we expect? One consequence of the age of scaling is that because scaling sucked out all the air in the room, we got to the point where there are more companies than ideas by quite a bit.

37:11 There’s a saying in Silicon Valley that ideas are cheap. People say that a lot, and there is truth to that. But I’d say something like, “If ideas are so cheap, why do all companies do the same thing?” And I think it’s true too. In terms of bottlenecks, there are several. Ideas, and then the ability to bring them to life, which requires compute.

37:52 If you go back to the ’90s, researchers had great ideas, and if they had much larger computers, maybe they could have demonstrated them. But they could not, so they could only run tiny experiments that did not convince anyone. So after the age of scaling, compute has increased a lot. Some compute is needed, but it’s not obvious that you need the absolute largest clusters.

38:33 Let me give you an analogy. AlexNet was built on two GPUs. The transformer was built on 8 to 64 GPUs. That’s not that much more than 64 GPUs of 2017. You could argue that the o1 reasoning was also done with relatively modest compute. So for research, you definitely need some compute, but it’s far from obvious that you need the absolutely largest clusters.

39:22 You might argue, and I think it is true, that once you have a new paradigm and want to scale it up, then it helps to have much more compute. But first you need to discover the paradigm, and for that, compute is not the primary bottleneck.

40:22 I can comment on SSI’s situation. The short answer is that specifically for us, the amount of compute we have, while small relative to the biggest companies, is comparable for research purposes. Simple math — the amount we have is a lot by any absolute sense. Other companies may be raising much more, but these big numbers, these big loans, go toward serving products. If you want to have a product, you have to have a big staff of engineers, salespeople. For pure research, the difference becomes a lot smaller.

41:41 Do you really need the same compute as the largest companies? I don’t think that’s true at all. You need compute to prove ideas work, to convince yourself and others. There have been public estimates that companies spend maybe a billion a year on research experiments. That’s not including money they’re spending on inference and products. The fraction spent on actually running research experiments is much smaller than people think. I think it’s a question of what you do with it.

42:29 In the case of other companies, there are a lot more different work streams, there are products, there’s all this other stuff. So the compute becomes fragmented. My answer to this question is that time will tell — the answer to that question will reveal itself.

43:01 Is SSI’s plan still to straight-shot to superintelligence without deploying products? Maybe. I think that there is merit to it — it’s very nice to not be affected by the market pressures. But I think there are two reasons to reconsider. One is pragmatic: if timelines turn out to be longer than expected. Second, I think there is a lot of value in having powerful AI being out there impacting the world.

43:46 So then why is your default plan to not deploy? Because it sounds like OpenAI, Anthropic, all these companies’ theory is, “Look, we have weaker and weaker intelligences that gradually improve.” Why is it potentially better to hold off? I’ll make the case for and against.

44:14 The pressure that people face when they’re in the market is significant. The rat race is quite difficult in terms of the trade-offs which you need to make. It’s very nice to step back from all this and just focus on the research. But the counterpoint is valid too. The counterpoint is, “Hey, it is useful to deploy AI. It is useful for the world to experience it. Deploying is the only way you can communicate it.”

44:56 Not the idea — communicate the AI itself. Let’s suppose you write an essay about AI and say, “It’s going to be that, and it’s going to be this.” People will say, “This is an interesting essay.” But compare that to actually experiencing an AI doing that. It is incomparable. So there is value in being in the public.

45:40 The other big thing is that I can’t think of an example in history where the safety of a powerful technology was achieved purely through research without the end artifact being deployed to the world. The risks are much lower today than they were decades ago because previous systems were deployed, problems were found, and the systems became more robust. I don’t see why AI would be any different.

46:23 The harms of superintelligence are not just about bugs. But this is a really powerful thing and we don’t know what people will do with it. Is there a better way to spread out the impact? Well, I think even in the straight-shot scenario, there would be an incremental release of it. That’s how I would imagine it. Incremental deployment is a component of any plan.

47:04 The first thing you get out of the door — that’s number one. Number two, I’m more of an advocate for continual learning more than other people, and I think that’s the right and correct thing. Here is why. I’ll give you an argument in two words.

47:29 In this case, it will be two words that explain something. First word: AGI. Second word: pre-training. Why does the term AGI exist? It’s a very particular term. The reason that the term AGI exists is, in my opinion, not because someone came up with a great descriptor of some end state of intelligence, but because another term existed, and the term is narrow AI.

48:14 There was a long history of gameplay and AI — checkers AI, chess AI. People would look at this narrow intelligence and say it’s great at this one thing but it can’t do anything else. So in response, as a reaction to this, people said, “This is so narrow. What we need is general AI.” That term just got a lot of traction.

48:59 And then pre-training came along. I think the way people do RL now is maybe still somewhat narrow. But pre-training had this property — you scale it up and it got better at everything, more or less uniformly. That was the dream that AGI represented, and pre-training delivered it.

49:38 But if you think about the term “AGI,” you will realize that a human being is not an AGI. A human being lacks a huge amount of knowledge and capability. Instead, we rely on continual learning. So let’s suppose that we achieve success — the question is, how do you define it? How much continual learning is it going to be? I think it’s like a 15-year-old that’s very eager to go. A great student, very eager. You say, “You go and be a doctor, go and learn.”

50:34 The deployment itself will involve some kind of continual learning. It’s a process, as opposed to a finished product. I see. You’re suggesting that the thing we should be building is not some finished mind which knows how to do everything — because the way, say, the original OpenAI charter defined AGI was a system that can do every single job, every single thing a human can do.

51:11 Instead, you’re saying it should learn to do every single job, through deployment. Yes. Exactly. It gets deployed into the world the same way a person enters the workforce.

51:27 Here’s one scenario of what might happen, maybe neither of these happens. Say the model becomes superhuman, becomes as good as an expert at the task of ML research. Then through recursive self-improvement, the model itself becomes more and more superhuman. Alternatively, if you have a single model deployed broadly through the economy doing many different jobs, continually learning on the job, picking up not just one skill but picking them all up at the same time, you basically have a model which functionally achieves recursive self-improvement through software deployment across every single job in the economy.

52:25 So do you expect some sort of intelligence explosion? I think that it is likely that we will see very rapid growth. I think with broad deployment, there are two possibilities. One is that once indeed you get to a point where models can learn as well as humans, and you have many of them, then there will be rapid economic growth — unless there will be some kind of regulation. But the idea of very rapid economic growth, I think it’s very possible from broad deployment.

53:25 I think this is hard to know because on the one hand, AI progress can be very fast. On the other hand, the world is just complicated — physical infrastructure, regulation — and that stuff moves at a different speed. So I think very rapid economic growth is possible, but different countries with different rules and different constraints will see different rates. Hard to predict.

55:10 The situation we’re in is remarkable. We know that this should be possible. A model that can learn as well as a human at learning, but which can merge its knowledge with other copies — humans can’t merge — already, this seems like it should be transformative. Humans are possible, digital intelligence is possible. You just need both of those combined.

55:41 Economic growth is one way to put it. But another way to put it is that you will have entities that learn incredibly fast. You hire people at SSI, and in six months they’re productive. A human learns really fast, and this thing will learn even faster. How do you think about making that go well?

56:12 One of the ways in which my thinking has been evolving — and I think it’s a very important evolution — is toward AI being deployed incrementally and in advance. We’ve been talking about systems that don’t yet exist. But I think one of the things that’s happening is that it’s very hard to feel the AGI.

56:52 It’s very hard to feel the AGI. You can be having a conversation about how it is going to change everything. You can have a conversation, you can try to imagine it. But then you go back to reality where that’s not the case. The disconnect between AI’s current limitations and its future power stem from the fact that future AI is going to be different. It’s going to be actually powerful.

57:43 What is the problem of AI and AGI? The whole problem is the power. What’s going to happen? One thing that changed my mind over the past year — and that will back-propagate into the plans of our company — is that you’ve got to be showing the thing. I maintain that most people who work on AI also don’t fully appreciate what’s coming, because they anchor on what people see on a day-to-day basis.

58:31 I predict that as AI becomes more powerful and visibly more powerful, people will change their behaviors. Things which are not happening right now will start happening. The frontier companies will play a very important role. The kind of things that I think we are already seeing the beginnings of are companies that are fierce competitors beginning to cooperate on safety.

59:15 You may have seen OpenAI and Anthropic doing things together on safety. That’s something which I predicted — that such a thing will happen. The AI will continue to become more powerful, more visibly powerful, and that will motivate governments and the public to do something. That’s the importance of showing the AI.

59:51 What needs to be built? One thing that I believe is that the people who are working on AI, I maintain, do not fully appreciate the magnitude of what’s coming. I do think that at some point the AI will be powerful enough to be truly surprising. I think when that happens, we will see a big shift toward safety. They’ll become much more paranoid. That’s what I expect to see happen.

1:00:30 People will see the AI becoming more powerful. The complacency that exists today, I maintain, is because people look at today’s models and extrapolate. There is a third thing which needs to happen, not just from the perspective of SSI but all AI companies. The question is, what should they aspire to build? What should they aspire to build?

1:01:04 Everyone has been locked into the same playbook, because there are fewer ideas than companies. I think there is something better to build, and I think it’s the AI that’s robustly aligned to care about sentient life. I think in particular, there’s a case to be made for building an AI that cares about sentient life rather than just humans, because the AI itself will be sentient.

1:01:46 I think empathy is an emergent property — the same circuit that we use to model ourselves, we use to model others. So even if you got an AI to care about humans specifically through alignment, it would still be the case that there will be trillions of AI agents. Humans will be a very small fraction. So it’s not clear to me if the goal is “align to humans” — maybe caring about sentient life broadly is the best criterion.

1:02:39 I’ll say two things. I do think there is merit to it. It should be considered. I think it should be on the list of ideas that the companies, when they are mature enough, should think about. Number three, I think it would be really good if the most powerful superintelligence was somehow capped in its capability. The question of how to do it, I’m not sure, but I think it matters when we’re talking about really, really powerful systems.

1:03:35 How do you think about superintelligence? Maybe it is just extremely fast at thinking. Does it just have a bigger pool of strategies? A data center that’s more powerful or bigger? Sort of godlike in comparison to the rest of humanity? A single agent, or another cluster of agents? People have different intuitions.

1:04:16 What I think is most likely to happen is multiple powerful AIs being created roughly at the same time. But if the cluster is literally continent-sized — that’s a lot of power. If you literally have a continent-sized computer, all I can tell you is that if you’re building something truly dramatically powerful, it would be nice if there were some kind of agreement or something.

1:05:11 What is one way to explain the concern? If something is really powerful, really sufficiently powerful, and you can’t be sure that it will care for sentient life in a very single-minded way — what is it? Maybe, by the way, the answer is that caring about sentient life is itself insufficient in some sense. We pursue a reward, and then the emotions change, and we pursue a different reward. Evolution is the same. Evolution is smart in some ways but very dumb in other ways. It’s going to be a never-ending fight.

1:06:13 Part of what makes this difficult is that we are talking about systems that are smarter than us. That’s the other thing. I think what people are doing right now with alignment is good. It will continue to improve. But the “It” we don’t know how to build, and generalization is the core challenge. What causes alignment to be difficult is that our understanding of these systems is fragile. Then your ability to optimize them is fragile.

1:07:00 And can’t you say, “Are these not all connected?” Why is it that human beings appear robustly aligned? What if generalization was much better? What would the effect be? But those questions point to the same fundamental mystery.

1:07:24 You’ve scoped out how AI might evolve — continual learning agents. AI will be very powerful. How do you think about lots of continent-sized clusters? How do we make that less dangerous? How do you achieve a long-run equilibrium where there might be misaligned incentives?

1:07:56 Here’s one reason why I liked the criterion “AI that cares about sentient life.” We can debate on whether it’s good or bad. But if the first powerful systems do care for, love humanity — and obviously this also needs to be achieved — then by the first N of those systems, we can establish something.

1:08:36 Then there is the question of how do you achieve a long-run equilibrium? I don’t like this answer, but in the long run, you might say, “Okay, if we solve the short-term problem…” In the short term, you could say you have universal high income and things are good. But what do the Buddhists say? “Change is the only constant.” Some government, political structure thing — some new government thing comes up and then eventually it stops functioning. We see that happening all the time.

1:09:32 One approach is that you could say maybe every person has their own personal AI that goes and earns money for the person and advocates for them, then writes a little report saying, “Okay, here’s how things went,” and the person says, “Great, keep it up.” But this can’t be maintained indefinitely.

1:10:10 I’m going to preface by saying I don’t have a full answer. But the solution is if people become deeply familiar with AI. Because what will happen as a result is that we understand the AI, and we understand it too, because now the human intelligence is enhanced. So now if the AI is in some situation, you can understand what it’s doing and why. I think this is the answer to the equilibrium question.

1:10:49 Evolution has developed over millions — or in many cases, billions — of years, and those ancient value functions are still guiding our actions so strongly. To spell out what I mean — evolution gave us something like a value function or reward function: “Mate with somebody who’s more successful.” And we figure out what success means in the modern context. Evolution says, “However you recognize success to be, you’re still going to pursue this directive.”

1:11:36 I think it’s actually really mysterious how evolution encoded these desires. It’s pretty easy to understand how evolution could give us a desire for food that smells good because smell is directly sensory. It’s very easy to imagine wiring dopamine neurons to smell sensors. But evolution also has endowed us with social desires. We really care about being liked. We care about being in good standing.

1:12:19 I feel strongly that they’re baked in, because it’s a high-level concept. Let’s say you care about some social thing — it’s not something for which there is a sensor. You need to piece together lots of bits of information to determine your social standing. Somehow evolution said, “That’s what you should care about.” All these sophisticated social things that we care about — evolution had an easy time doing simple sensory stuff, but I’m unaware of a good explanation for how it did the social stuff.

1:13:24 What’s especially impressive is it was desire for things that are abstract, and yet it works because your brain is intelligent. Evolution seems to be able to learn intelligent desires. One way to understand it is that the desire is built into the genes, but you’re somehow able to describe this feature abstractly, and you can build it into the genes.

1:13:55 If you think about the tools that evolution has — “Okay, here’s a recipe for building a brain. Wire the dopamine neurons to the smell sensor.” When you smell something with the signature of good smell, you want to eat that. I’m claiming that it is harder to imagine how evolution could say, “You should care about some complicated computation that your brain does.” That’s all I’m claiming.

1:14:33 Let me offer a speculation, and I’ll explain why I think it might work. So the brain has brain regions. The cortex is uniform, but the brain regions are localized — neurons kind of speak to their neighbors mostly. Because if you want to do some kind of computation, the neurons that do speech need to talk to each other, and they talk to their nearby neighbors. All the regions are mostly located in consistent positions.

1:15:15 So maybe evolution hard-coded that when the GPS coordinates of a particular brain region fire, that’s what you should care about. That would be within the toolkit of evolution. But there’s a counterargument — for example, people who are born blind have that visual cortex repurposed. I have no idea if the social instinct signal no longer works for people who have had cortical reorganization.

1:16:10 I fully agree with that. I think there’s an even stronger counterargument. There are people who get half of their brain removed, and they still have all their brain regions. They still function, which suggests that the brain regions can reorganize completely. So that theory is not true — it would be elegant if it was true, but it’s not.

1:16:37 But it’s an interesting mystery. The fact is that humans are reliably programmed by evolution to care about social stuff very, very reliably, despite all sorts of unusual conditions and deficiencies and emotional variation.

1:18:13 What is SSI planning on doing differently from the other frontier companies when this time arrives? “I think I have a way of approaching this that other companies don’t.” What is that difference? There are some ideas that I think are promising, and time will tell whether they are indeed promising or not. It’s really that simple. If our ideas turn out to be correct — these ideas that we discussed about value functions, generalization, sample efficiency — I think we will have something worthy.

1:19:05 We are doing research. We are squarely an “age of research” company. We’ve actually made quite good progress over the past year. We need more research. That’s how I see it.

1:19:29 Your cofounder and previous CEO left to go to another company. Some people interpreted that as, “If there were a lot of breakthroughs being made, that departure would have been unlikely.” I wonder how you respond. Some context may have been forgotten. Let me provide the context. SSI raised at a $32 billion valuation, and then Meta came along and offered a comparable amount. But my former cofounder in some sense said yes. He chose near-term liquidity.

1:20:25 It sounds like SSI’s plan is to be a company that matters during a very important period in human history. You have these ideas about how to build safe superintelligence. But other companies will have ideas too. What distinguishes SSI’s approach? The main thing that distinguishes us is that we have a different technical approach. I maintain that in the end there will be convergence of strategies — it’s going to become more or less clear what the right approach is.

1:21:19 It should be something like, you need to find the right approach to building your first actual real superintelligent AI that cares for people, is democratic, is safe. I think this is the condition for success. That’s what SSI is striving for. I think all the other companies will eventually realize that this is the right approach. We’ll see. I think things will be really different.

1:22:12 Speaking of forecasts, what are your timelines? For an AI which can learn as well as a human and can merge its knowledge? I think like 5 to 20 years.

1:22:29 Let me try to unroll how you might see the world coming. These other companies are continuing to build and deploy, and perhaps the current approach stalls out. “Stalls out” meaning they earn no more improvement from current methods. How do you think about what stalling out means?

1:23:05 Even with stalling out, I think these companies will continue to earn good revenue. Maybe not profits because they will need to keep investing, but revenue definitely. And when the correct solution does emerge, there will be rapid convergence. I think companies will converge on their alignment strategies as the technical approach becomes clearer.

1:23:43 I just want to better understand the dynamics. Currently, we have these different companies, and they can earn revenue but not get to this human-like learner. We have you, we have Thinking Machines, and others. Maybe one of them figures it out. But then the release of their product makes it clear that it’s possible, and that is information. Others will try to figure out how that works.

1:24:26 There’s something that hasn’t been addressed here — that with each new level of AI capability, there will be some kind of changes to society, but I don’t know exactly what. I think it’s going to be important. By default, you would expect the company that gets there first to have a big advantage, because they have the model that has the skills.

1:25:14 Here is what I think is going to happen. It’s basically what has happened so far with the AIs of the past. Every time one company achieved something new, every other company scrambled and produced something similar, often within months, and competition pushed the prices down. I think something similar will happen there as well.

1:25:56 What’s the good world? It’s where we have these powerful, aligned AIs. By the way, maybe there’s another thing we haven’t discussed about AI that I think is worth considering. AI might be useful and narrow at the same time. But suppose you have many of them and you specialize. Then through specialization, competition loves specialization — you see it in evolution as well. You get niches and lots of different species.

1:27:19 In this world we might say one AI company does one type of really complicated economic activity. And the third company is doing something else. Isn’t this contradicted by what human-like continual learning implies? It can be, but you have accumulated specialization. You spent a lot of compute to become really good at one thing. Someone else spent a huge amount of experience to get really good at some other thing. Now you are at this high point where it’s expensive for someone else to start learning what you’ve learned.

1:28:07 Different companies could begin at the human-like continual learning stage and start their different specialization trees. But if one company gets that agent first, or gets it working across every single job in the economy — one seems tractable for a company. My intuition is that it’s not how it’s going to go. My strong intuition is that it will not be winner-take-all.

1:28:36 In theory there’s no reason it can’t be winner-take-all, but in practice, there always isn’t. A lot of people’s models of recursive self-improvement is, “We will have a million Ilyas in a server that are all doing research,” leading to a superintelligence emerging very fast. But the question is how parallelizable the thing you are doing is.

1:29:00 I don’t know. I think there’ll definitely be diminishing returns. What you want is diversity — researchers who think differently rather than the same. Just making more copies of the same doesn’t necessarily help. If you look at different models, even released by totally different companies with non-overlapping datasets, it’s actually surprising how similar they are. Maybe the datasets are not as different as we think. But there’s some sense in which humans might have more diversity than teams of AIs might have.

1:29:53 I think just raising the temperature is not the same as true diversity. You want something more like different scientists with different backgrounds and perspectives. How do you get that kind of diversity? The reason there has been no diversity in AI models is that all the pre-trained models are pretty much the same. Now RL and post-training is where diversity starts to emerge, because different people come with different approaches.

1:30:26 I’ve heard you hint in the past about self-play — how you can get data or match agents to other agents. How should we think about why there are no public self-play breakthroughs yet?

1:30:46 I would say there are two things to say. The reason self-play is interesting is because it offered a way to generate unlimited data. If you think that data is the ultimate bottleneck, self-play removes it. So that’s what makes it interesting. But historically, the way self-play was done — when you have agents competing against each other — it’s good for developing a certain set of skills: conflict, certain social skills, negotiation.

1:31:35 If you care about those skills, self-play is great. Actually, I think that self-play did find its way into current methods. Things like debate, prover-verifier — you have a model and another model that’s also incentivized to find mistakes in your work. This is a related adversarial dynamic. Really self-play is a special case of multi-agent interaction.

1:32:13 The natural response to competition is specialization. So if you were to put multiple agents together working on a problem and you are an agent inspecting what others are doing, you’d say, “Well, if they’re already taking this approach, I should pursue something differentiated.” So I think multi-agent interaction creates an incentive for a diversity of approaches.

1:32:44 You’re obviously the person in the field who many people believe has the best taste in doing research in AI. Looking back at the great breakthroughs that have happened in the history of deep learning — what is it, how do you characterize research taste?

1:33:11 I can comment on this for myself. One thing that guides me personally is an obsession with thinking correctly. Not about how people are, but thinking correctly. Many people think incorrectly, but what does it mean to think correctly? Let me give examples. The idea of the artificial neuron — a great idea. Why? Because you say the brain has neurons, the folds probably don’t matter, but the neurons do. Because there are many of them.

1:34:01 You want some local learning rule that will allow the system to learn. It feels plausible that the brain does it. The idea that the brain responds to experience — that was the fundamental insight: the neural net should learn from experience. Is this fundamental or not fundamental? How things should be — thinking from multiple angles and looking at every aspect.

1:34:41 Ugliness — there’s no room for ugliness. The right answer has to have correct inspiration from the brain. Multiple things have to be present at the same time. The more confident you can be in a top-down belief, the more you should persist when the experiments contradict you.

1:35:04 Well, sometimes you can be doing the right experiment, but there’s a bug. But you don’t know that there is a bug. How do you know if you should keep debugging or give up? That’s where top-down thinking helps. You can say things have to be this way, therefore we’ve got to keep going. That confidence in multifaceted beauty and inspiration by the brain — that’s what guides research taste.

1:35:31 Thank you so much. Alright. Appreciate it. Yeah, I enjoyed it.