Richard Sutton — Father of RL thinks LLMs are a dead end

Summary

Richard Sutton, winner of the 2024 Turing Award and author of “The Bitter Lesson,” sits down with Dwarkesh Patel for a wide-ranging interview in which he makes the provocative case that large language models are fundamentally a dead end. Sutton argues that true intelligence requires learning from experience with goals and rewards — not imitating what humans have said. He contends that LLMs lack genuine goals, cannot make predictions about the world that can be verified against ground truth, and cannot learn continually from ongoing experience. In his view, the entire LLM paradigm of massive pretraining on static human data will eventually be superseded by agents that learn from a stream of sensation, action, and reward.

The interview becomes a genuinely heated intellectual debate, with Dwarkesh pushing back repeatedly on whether LLMs could serve as a foundation for experiential learning. Sutton firmly disagrees, arguing that starting with human knowledge has always turned out to be a dead end historically, even when it seems useful in the short term. He draws on examples from chess (AlphaZero’s superiority over hand-crafted engines), the history of AI’s “weak methods” vs. “strong methods” debate, and his own decades of work in reinforcement learning to support his position.

The conversation culminates in a remarkable discussion about the succession from biological to designed intelligence. Sutton argues that we are at a pivotal transition in the universe — from evolved to designed entities — and that AI succession is inevitable regardless of what any government or organization does. He draws parallels to how parents raise children, suggesting we should instill good values in AI rather than trying to control exactly what it does, and reflects philosophically on whether the rise of superintelligent AI represents a continuation of humanity or something fundamentally new.

Highlights

”LLMs have no goal — that’s starting in the wrong place”

yt-dlp --download-sections "*7:42-8:50" "https://www.youtube.com/watch?v=21EYKqUsPfg" --force-keyframes-at-cuts --merge-output-format mp4 -o "21EYKqUsPfg-7m42s.mp4"

“Let’s go back to their lack of a goal. I like John McCarthy’s definition that intelligence is the computational part of the ability to achieve goals. If you don’t have a goal, you’re just a behaving system. You’re not intelligent.” — Richard Sutton, 7:42

”It has always turned out to be bad”

yt-dlp --download-sections "*12:48-13:50" "https://www.youtube.com/watch?v=21EYKqUsPfg" --force-keyframes-at-cuts --merge-output-format mp4 -o "21EYKqUsPfg-12m48s.mp4"

“There’s never any reason why starting with human knowledge has to be bad. But it has always turned out to be bad. People get committed to the knowledge approach, and they psychologically can’t let go. That’s what has always happened. They get superseded by the methods that are truly scalable.” — Richard Sutton, 12:48

”The weak methods have just totally won”

yt-dlp --download-sections "*43:19-44:20" "https://www.youtube.com/watch?v=21EYKqUsPfg" --force-keyframes-at-cuts --merge-output-format mp4 -o "21EYKqUsPfg-43m19s.mp4"

“There was a long-standing controversy in AI about simple methods like search and learning, compared to expert knowledge. In the old days, they called them weak methods because they’re just using computation. But the weak methods have just totally won.” — Richard Sutton, 43:19

”We are animals first”

yt-dlp --download-sections "*19:15-20:20" "https://www.youtube.com/watch?v=21EYKqUsPfg" --force-keyframes-at-cuts --merge-output-format mp4 -o "21EYKqUsPfg-19m15s.mp4"

“Humans are animals. What distinguishes us, we should consider secondary. We’re trying to replicate intelligence. If you want to understand what it takes to go to the moon or to build semiconductors, you have to understand the basic animal intelligence that makes that happen.” — Richard Sutton, 19:15

”Succession to AI is inevitable”

yt-dlp --download-sections "*54:35-55:40" "https://www.youtube.com/watch?v=21EYKqUsPfg" --force-keyframes-at-cuts --merge-output-format mp4 -o "21EYKqUsPfg-54m35s.mp4"

“We will figure out how intelligence works. We won’t stop just at human level — we will reach superintelligence. And it’s inevitable that things that are smarter at getting things around would gain resources and power. You’re going to have succession to AI. Those four things seem clear and sure to happen.” — Richard Sutton, 54:35

”The transition from evolved to designed intelligence”

yt-dlp --download-sections "*57:25-58:40" "https://www.youtube.com/watch?v=21EYKqUsPfg" --force-keyframes-at-cuts --merge-output-format mp4 -o "21EYKqUsPfg-57m25s.mp4"

“This is a key step in the universe. It’s the transition from the evolved to the designed. We humans and animals are evolved. Our AIs are designed, our technology is designed. Now we’re reaching the point where we can design intelligence that we do understand how it works. This is one of the four great transitions in the universe.” — Richard Sutton, 57:25

Key Points

• LLMs lack world models (1:52) - Sutton argues LLMs predict what a person would say, not what will actually happen in the world. They lack the ability to predict consequences of actions.
• RL is about understanding your world (1:46) - Sutton considers RL the fundamental problem of AI: understanding your world through experience with actions and consequences.
• No ground truth in LLMs (5:56) - Without goals, there’s no ground truth to learn from. In RL there’s a right thing to do; in language modeling there’s no right thing to say.
• Goals are the essence of intelligence (7:42) - Using McCarthy’s definition, intelligence is the computational ability to achieve goals. Without goals, you’re just behaving, not thinking.
• Math is different from empirical learning (9:15) - Sutton concedes LLMs can have goals in math (find the proof), but argues empirical world knowledge requires fundamentally different learning.
• The Bitter Lesson misunderstood (9:59) - People cite The Bitter Lesson to justify scaling LLMs, but Sutton says LLMs actually violate it — they inject massive human knowledge rather than learning from experience.
• Animals don’t do imitation learning (17:32) - Sutton argues there’s nothing like imitation in basic animal learning. Supervised learning is not a natural learning paradigm.
• Cultural evolution debate (20:23) - Dwarkesh argues complex skills like seal hunting are transmitted culturally through imitation. Sutton acknowledges this but considers it secondary to basic animal learning.
• Moravec’s paradox (22:42) - Interesting that basic sensorimotor skills (which all mammals have) are hardest for AI, while math (uniquely human) is easier.
• The sensation-action-reward stream (24:08) - Sutton’s alternative paradigm: intelligence is about a stream of sensation, action, and reward, and learning to increase rewards.
• Four components of RL (32:52) - Policy, value function, state construction, and world model (transition model). The world model is learned from sensation, not reward.
• Transfer and generalization unsolved (36:26) - Despite claims of LLM generalization, Sutton argues we haven’t seen genuine transfer in RL or in LLMs — it’s just memorization of massive data.
• Catastrophic forgetting (37:50) - Training on new data causes forgetting of old knowledge. Good generalization requires some sculpted architecture, not just gradient descent.
• AlphaZero’s surprising chess style (46:04) - AlphaZero plays chess by sacrificing material with content patience — a style no human had ever used, which was genuinely surprising.
• Classicist not contrarian (47:18) - Sutton sees himself as maintaining what the great traditions of thinking about the mind have always believed, not as a contrarian.
• Bitter Lesson after AGI (47:52) - Even after AGI, the Bitter Lesson will apply: AI researchers’ stock of knowledge will grow as fast as compute, so artisanal solutions may still lose.
• Four inevitabilities of AI (54:35) - We will understand intelligence, reach superintelligence, smarter agents will gain power, and succession to AI will occur.
• Cybersecurity of minds (57:00) - In the age of digital intelligence, incorporating knowledge from other agents could corrupt or destroy you — a major cybersecurity concern.
• Raising AI like children (1:01:14) - Rather than controlling AI’s specific actions, instill good values and let them evolve, like parents with children.
• The more things change (1:05:57) - Sutton’s parting thought: “The more things change, the more they stay the same” — a good capsule for the AI discussion.

Mentions

Companies

• Google DeepMind (34:47) - MuZero models referenced as examples of RL world models
• Alberta Machine Intelligence Institute (Amii) (0:47) - Hosted the interview in Edmonton

Products & Technologies

• AlphaGo (44:58) - Beat world champion at Go; used Monte Carlo tree search, not TD learning
• AlphaZero (46:04) - Used TD learning; played chess with a surprising sacrificial style
• MuZero (34:47) - RL framework with learned world models, trained for specific games
• TD Learning (45:18) - Temporal difference learning, Sutton’s invention; used in backgammon and AlphaZero

People

• Richard Sutton (0:47) - 2024 Turing Award winner, father of reinforcement learning, interview subject
• John McCarthy (7:53) - AI pioneer; Sutton references his definition of intelligence as “the ability to achieve goals”
• Gerry Tesauro (45:18) - Did RL/TD learning for backgammon in the early 1990s; beat world’s best players
• Joseph Henrich (18:59) - Psychologist who studies cultural evolution; referenced by Dwarkesh

Surprising Quotes

“To mimic what people say is not really understanding the world. You’re mimicking things that have already been said. I don’t want to approach the question in an adversarial way, but I disagree with the idea that LLMs have a world model.” — Richard Sutton, 2:30

“If there’s no goal, then there’s no right thing to say. There’s no ground truth. Large language models are trying to get by without a goal. That’s just exactly starting in the wrong place.” — Richard Sutton, 5:02

“In basic animal learning, there’s nothing like imitation. Supervised learning is not a natural learning paradigm. We don’t have examples of desired behavior. We have one thing that followed another. We did something and there were consequences.” — Richard Sutton, 17:32

“The dream of large language models, as I see it, is that you just put in all the knowledge. It will know everything and won’t have to learn anything new. But the world is really big, and all the little idiosyncrasies of your particular situation can’t all have been put in in advance.” — Richard Sutton, 30:38

“First there’s dust, it ends with stars. Stars give rise to planets. Life arises. Now we’re giving rise to designed entities. This is one of the four great transitions in the universe.” — Richard Sutton, 58:25

Transcript

0:47 Today I’m chatting with Richard Sutton, the father of reinforcement learning and inventor of foundational techniques like TD learning and policy gradient methods. He recently won the Turing Award, which, if you don’t know, is the Nobel Prize of computer science. Thank you, Dwarkesh. It’s my pleasure.

1:12 Most of our audience is going to be familiar with the LLM way of thinking about AI. Can you describe thinking about AI from the RL perspective? It can easily get separated and lose perspective. Large language models have become such a big thing. Our field is subject to bandwagons and I consider reinforcement learning to be the real core of AI. The problem is to understand your world. RL is about understanding your world, whereas large language models are about doing what people say you should do.

2:08 You would think that to emulate the trillions of words that humans have produced, you would have to build a world model. You would think LLMs would develop very robust world models. That’s been most of the progress we’ve made to date in AI, right? I would disagree with most of that. To mimic what people say is not really understanding. You’re mimicking things that have already been said.

2:40 I don’t want to approach the question in an adversarial way, but I disagree with the idea that they have a world model. A world model should be able to predict what would happen. LLMs predict what a person would say. That’s different from the ability to predict what will happen.

3:04 I’m interested in systems that can learn from experience, where experience is a stream of sensations, actions, and rewards. You do things, you see what happens, and you learn from that. The large language models don’t do that. They learn from “here’s a situation, here’s what a human said.” Implicitly, the suggestion is you should do the same.

3:26 I guess maybe the crux, and I’m curious if you agree, is: many people will say that imitation learning has given us a very good prior, a very good sense of reasonable ways to approach problems. And this prior, whatever you call it, this prior is going to be the basis for experiential learning, because this gives them the opportunity to learn from experience efficiently. Then on this, you can train them on experience.

4:00 No. I agree that it’s the large language model view, but I don’t think it’s a good perspective. For knowledge there has to be a real thing. There has to be ground truth that serves as the basis for actual knowledge. What is actual knowledge in that large-language framework?

4:29 You recognize the need for continual learning. Learning continually means learning during the agent’s lifetime. There must be some way during the interaction with the world to update what you know. Is there any way to tell in the large language model framework what’s true and what’s false? You will say something and you will not get direct feedback, because there’s no definition of what’s correct.

5:02 If there’s no goal, then there’s no right or wrong. There’s no right thing to say. There’s no ground truth. You can’t learn if you don’t have ground truth, because the only way you can learn is by having an initial belief about what the truth is, and then comparing it against reality.

5:24 In reinforcement learning, there is a right thing to do. The right thing to do is the thing that gets you reward. So we can have prior knowledge about what the right thing to do is. We can learn what the right thing to do is, because we have a definition of what success looks like.

5:47 An even simpler case is when you’re making predictions. When you predict what will happen, you predict it and then you find out whether you were right. That’s ground truth. There’s no ground truth in large language models. They’re making a prediction about what will happen next — but the large language models have no prediction about what the world will do in response, or what the actual outcome will be.

6:19 “What would you anticipate a user might say in response?” No, they will respond to that question right. But not in the deeper sense that they won’t be surprised. If something happens that isn’t what you expected, you should update. You should change because an unexpected thing has happened.

6:43 I think a capability like that is already emerging in LLMs. It’s interesting to watch a reasoning model work. Suppose it’s trying to solve a math problem. It says, “Let me try this problem using this approach first.” Then, “Oh wait, I just realized this is the wrong approach. I’m going to restart with another approach.”

7:10 Do you have something else in mind for how to extend this capability across longer horizons? In some sense, these are predictions about what will happen next. They’ll not make any changes if what they predicted comes true.

7:30 Isn’t that literally what next-token prediction is? Prediction about what’s next and learning from that? The next token is what they should say next. It’s not what the world will give them next.

7:42 Let’s go back to their lack of a goal. I think goal-directedness is the essence of intelligence. I like John McCarthy’s definition that intelligence is the computational part of the ability to achieve goals. Without a goal, you’re just a behaving system. You’re not intelligent.

8:11 Do you think LLMs don’t have goals? What’s the goal of next-token prediction? That’s not a goal. It doesn’t change the world. The next tokens are coming and if you predict them, you don’t influence them. It’s not a goal. It’s not a substantive goal. A language model, if it’s just sitting there predicting and being rewarded for correct predictions, isn’t really doing anything.

8:43 The bigger question I want to understand is whether building experiential learning on top of LLMs is a productive direction. For example, we’ve given these models the goal of solving difficult math problems. They’ve now reached roughly human-level in the capacity to solve math at the level of the IMO. So it seems like the model which got trained on math problems does have the goal of getting math problems right.

9:15 The math problems are different. Making a mathematical proof is about working out the consequences of mathematical assumptions. The empirical world has to be learned. Whereas the math is more computational. There they can have a goal to find the proof, and they can be given that goal to find the proof.

9:59 You wrote this essay in 2019 titled “The Bitter Lesson” — maybe the most influential essay in the history of AI. Many people cite it to justify scaling up LLMs because, in their view, this is throwing massive amounts of compute into learning about the world. But you’re saying the LLMs are not “bitter lesson”-pilled?

10:32 I think language models are a case of the bitter lesson in that they use computation, things that will scale with more compute. But they’re also a way of putting in lots of human knowledge. It’s a sociological or industry question. I believe they will be superseded by things that can get more data from the world, learn more from experience.

11:24 In some ways it’s a classic bitter lesson pattern. The more human knowledge we put into the system, the better it can do. So it feels good. Yet, I expect there will eventually be things which could perform much better without all that human knowledge. In which case, it will be another instance of the bitter lesson: systems with human knowledge were eventually superseded by things that just learned from experience.

12:05 I guess that doesn’t seem like the crux to me. Nobody is saying that the overwhelming amount of compute in the future won’t go to experiential learning. They just think that the scaffold or the basis of the system that’s doing all the compute to do this future experiential learning will be an LLM.

12:31 I still don’t understand why this is controversial. Why do we need a whole new architecture to do experiential learning? Why can’t we start with LLMs to do that? Historically, you could start with human knowledge and then do the experiential learning. There’s never any reason why that has to be bad.

12:48 But it has always turned out to be bad. People get committed to the knowledge approach, and they psychologically can’t let go. That’s what has always happened. They get superseded by the methods that are truly scalable. The scalable method is you learn from experience. No one has to tell you.

13:37 Without a goal, there’s no sense of right or wrong. Large language models are trying to get by without a goal. That’s just exactly starting in the wrong place.

13:55 In both the case of learning from imitation and learning from experience, I think there’s some interesting analogies with how humans learn. You don’t think so? Really? I think kids just watch people.

14:19 How old are these kids? What are they watching? I think they’re imitating things. They’re making sounds — they see their mother’s mouth move and try to make the sound. But they’re not really understanding what they mean. And as they grow, the sophistication of the imitation they do increases.

14:41 You observe the techniques that people in your band are using to hunt. Then you go into the learning phase on your own. But I think there’s a lot of imitation before that. It’s surprising you can have this disagreement. When I see kids, I see kids just moving their hands around and moving their eyes around.

15:10 The sounds they try to make, the eyes they move around — but the actions, the thing that the infant chooses to do, those aren’t imitated. There are no examples for that. Imitation may guide a learning process, like how an LLM predicts the next token early in training — it will make a guess. In some sense, it’s very short-horizon RL: “I think this token will be this.”

15:43 A baby will try to say a word. It comes out wrong. But the baby is learning from experience, not from training data. It’s not learning from a dataset that will never be available during its normal life. It should learn from signals about what it should do in normal life.

16:05 What about school? School is much later. Okay, I think I would consider school imitation learning of a sort. But formal schooling is the exception. There’s the programming in your biology, and then why you exist is to understand the world.

16:34 Doesn’t it seem like there’s a training phase in childhood? It’s gradual. There’s not a sharp cutoff. But there does seem to be this initial training phase, right? No. Nobody is telling you what you should do. There’s nothing. You see things happen and you learn from that. It can be difficult. I mean this is obvious.

17:03 This is where the word training is misleading. I don’t think learning is really about training. It’s about an active process. We don’t think about training when animals learn. These things are actually rather well understood.

17:32 In basic animal learning, there’s nothing like imitation. Some animals might do that or appear to do that, but there’s no fundamental role for it. There are basic animal learning processes for reward, for consequence. It’s really interesting how sometimes the obvious gets overlooked. It’s obvious — if you look at animals and how they learn and the theories of them — that supervised learning is not a natural paradigm. We don’t have examples of desired behavior. We have one thing that followed another. “We did something and there were consequences.”

18:32 Supervised learning is not a natural learning process. Even if that were the case with school, school is some special thing that happens in people. Dogs don’t go to school. Squirrels don’t go to school. It’s absolutely obvious, I would say, that the basis of learning is experiential.

18:59 I interviewed this psychologist, Joseph Henrich, who has done work about cultural evolution, about how humans pick up knowledge. But aren’t humans animals? What we want to understand about intelligence, we should look at what distinguishes us from other animals secondarily. We’re trying to replicate intelligence. If you want to understand what it takes to go to the moon or to build semiconductors, you have to understand the basic animal intelligence that makes that happen.

19:42 I like the way you consider that obvious, even though others might disagree. We have to understand how we are animals. Understanding animal intelligence gets you almost all the way there to understanding human intelligence. The language part is just a small veneer on the top. Language allows us to communicate very different ways that we’re thinking. We’re sharing our ways of thinking with each other.

20:21 I do want to complete this thought. There’s a theory about a lot of the skills that humans have acquired culturally. We’re not talking about the last few thousand years but hundreds of thousands of years. The world’s many indigenous peoples had to reason through how to hunt and process food. There’s this many-step, long process of finding, killing, processing, and then how to process the food in a way that’s safe to eat. It’s not possible to reason through all of that from scratch.

21:09 Whatever learning analogy you want to use — maybe RL, something like that — these cultures learned how to find and kill and eat seals. And this knowledge is transmitted through watching and imitating your elders in order to learn that skill. You don’t independently reinvent how to hunt and kill and process a seal. You imitate, and maybe make tweaks and adjustments.

21:43 The initial step of the cultural transmission does look like imitation. But maybe you think about it a different way? Still, it’s a small thing on top of basic animal intelligence. It’s what distinguishes us, perhaps, but we were an animal first. We were an animal before we had cultural learning.

22:13 I do think you make a very interesting point about sensorimotor skills — a capability that most mammals have. It’s quite interesting that we have something that seems very basic yet is the hardest thing for AI to replicate. Whereas the ability to understand math — however you define math — is a capability that our species uniquely has. It’s quite interesting what ends up being hard and easy. Moravec’s paradox.

23:58 This alternative paradigm that you’re imagining — can you lay it out a little bit? It’s about sensation — well, sensation, action, reward — this is the stream. It says that this is the foundation of intelligence. Intelligence is about taking that stream and learning to increase the rewards in the stream. All knowledge and learning is about the stream.

24:40 What you learn, your knowledge, is about what will happen in the stream. Your knowledge is about if you do this, what follows. Or it’s about which events will follow other events. The knowledge is statements about the stream. You can test it by comparing it to what actually happens.

25:06 When you’re imagining this continual learning agent, what is the objective? The reward function is arbitrary. If you’re an AI assistant, maybe the reward is user satisfaction. If you’re a squirrel, maybe the reward is finding nuts. In general, for an animal, you would say the reward is survival and reproduction. I think there also should be a drive toward increasing understanding of your environment.

26:14 With this AI, lots of people would want it to do useful tasks. It’s performing the task people want, but also learning about the world from doing that task. Do we also get rid of this paradigm where there’s training periods and deployment periods? Do we get rid of having copies of the model that are doing certain things?

27:00 The word “model” is misleading when used this way, because I think you mean the network. Maybe “agent” is better. The weights would be learned. You’d have copies and many instances. There are lots of possibilities for doing that. With biological agents, each individual has to learn about the world, and then every generation starts over. Whereas with AIs, with digital intelligence, you can pour one generation’s knowledge into the next one as a starting place.

27:44 I think it’d be much more important than any of the other things we’re discussing. I agree that the kind of thing you’re describing — continual learning with goals — regardless of whether you start from LLMs or not, you’re going to need this capability.

28:05 Humans have extremely sparse rewards. Once in 10 years you might have an exit from a startup. But humans have this ability to make intermediate assessments. They have an understanding of what the next thing they’re likely to do will lead to. How do you imagine such a system working?

28:31 This is something we know very well. It’s temporal difference learning, where the same thing that happens at the end of the game — win or lose — you want to be able to learn from shorter-term signals too. You do that by having a value function that estimates how well things are going. Then if you take the opponent’s pieces, your value function goes up, you think you’re going to win. That reinforces the move that led to taking the piece.

29:20 It’s like getting a raise at a startup and thinking, “I’m more likely to achieve the long-term goal.” You also want some ability for the agent to pick up context. One of the things that makes humans quite useful at a job is that during onboarding, you’re picking up all the specific details. That’s what makes you useful at the job. Learning that each client has preferences to how they like things done.

29:56 Is the bandwidth of information that you need to pick up in the course of doing a job large enough to need this massive learning capability, or can you just have it all in context? I’m not sure, but I think at the crux of this, the reason why humans become useful on a job is that they learn the specifics of their particular part of the world. Those specifics can’t all have been put in in advance.

30:38 The dream of large language models, as I see it, is that you just put in all the knowledge. It will know everything and won’t have to learn. Your examples are all, “Well, really it should know most things, but there’s all the little idiosyncrasies of each particular situation, the particular people they’re working with.” That’s just saying the world is really big, and you need to keep learning.

31:14 It seems to me you need two things. One is the ability to decompose a long-term goal reward into smaller auxiliary predictive rewards — stepping stones that lead to the final reward. And two, some way to hold on to all this context about what I’m learning about my clients, my environment.

31:50 I would say you’re just doing regular learning, because in large language models all that goes into context. But in a continual learning setup, it goes into the weights. Maybe context is the wrong word. You learn a policy that’s specific to the situation.

32:12 The question I’m trying to ask is, you need some way to represent the information that agents are picking up when they’re out in the world — working with your clients and everything. It seems like the reward is too small of a signal. But we have the sensations, we have all of the experience. We don’t just learn from the reward. What is the learning process which captures all of that?

32:52 Now I want to talk about the base components. You need a policy — the policy says, “In this situation, take this action.” You need a value function — the value function looks at the situation and produces a number. Then you watch if that’s going up and down. So you have those two things. The third component is construction of your state representation. The fourth one is what we’re really talking about — the world model.

33:38 That’s why I am uncomfortable just calling an LLM a world model. When we talk about the model of the world, it’s your belief that if you do this, what will happen? Your physics of the world. Like your model of how you traveled from your hotel to here — that was a model, and that’s a transition model. It’s not learned from reward. It’s learned from, “Here’s what happened when you made that model of the world.” It’s learned from all the sensation that you experienced.

34:17 It has to include the reward as well, but the reward model is a small, crucial part of the whole model. If you look at the MuZero models that Google DeepMind built, they were initially not a general intelligence — they were specialized intelligences to play specific games. Using that framework, you could train a policy to play any specific game. You had to train each one in a specialized way.

34:58 Is that a fundamental limitation of reinforcement learning generally — that you can only learn one thing at a time? Or was it just specific to AlphaZero? If it’s specific to AlphaZero, what needed to change so that it could be a general learning agent? The answer is: one agent, one world. All the time, as my canonical example, people have one world. That world may involve chess and it may involve cooking. It’s one world, not a different task or a different world. So the general idea is not limited at all.

35:54 What was missing in that architecture? They just set it up for specific games. It was not their goal to make a general agent. If we want to talk about transfer, we should talk not about transfer across tasks, but transfer between states. We haven’t seen the level of transfer using RL techniques that we’d like.

36:35 Good. We’re not seeing transfer anywhere. We’re not seeing RL agents that generalize well from one state to another state. What we have are people trying different things — different network architectures, different training regimes — that transfer well or generalize well. And a lot of attempts to promote transfer, and none of them have worked great.

37:11 Let me paraphrase to make sure I understand. It sounds like you’re saying that when we do see generalization in these models, that is a result of some sculpted architecture, some deliberate design. Because there’s no other explanation. Gradient descent will make you solve the problem. But when you train on new data, you want to generalize in a good way. And that’ll affect what you do on other things.

37:50 For example, we know that if you train on some new task, you can catastrophically forget all the old things that you knew. This is some kind of influence of architecture and design. The fact that you generalize — you can generalize poorly, you can generalize well. Generalization always will happen, but we need to design systems where generalization is good rather than bad.

38:30 I’m not trying to catch you at the crux again, but I’m just genuinely curious. One way to think about these LLMs is as a progression of generalization from earlier systems, which could only do narrow tasks, to now where they can do anything in this broad space. You initially start with them being able to do one kind of math. Then they can generalize among problems which share techniques and theorems and conceptual categories. It sounds like you don’t think of that as an example of generalization?

39:18 Large language models are so complex. They have so much information they have had prior. This is one reason why they’re so hard to analyze. It’s just so uncontrolled, so unknown. They’re getting a bunch of things right, perhaps. But you don’t need to generalize to get them right, because there are enough parameters to just memorize and form something which gets all of them right. And that’s not called generalization.

40:02 Maybe they find the only way to solve it, and so they find the only way to solve it. It could be that way, and they do it the good way. My understanding is that this is working more like engineering. With engineers, obviously if you’re trying to build something, there are different ways you could achieve the end spec. The concern has been that they’ll do it in a way that’s sloppy rather than coming up with the design architecture that’s elegant and generalizable.

40:37 It seems like an example of emergent good design. But there’s nothing in them which guarantees good generalization. Gradient descent will cause them to find a solution. If there’s only one way to solve it, great. But if there are many ways to solve it, some which generalize and some which don’t, there’s nothing in the algorithms that guarantees they’ll find the one that generalizes.

42:17 I want to zoom out and ask about being in the AI field right now. What it’s like for someone who is commentating on it, or working in it now. What the biggest surprises have been. Zooming out, you got into this even before AI was a term people commonly used. So how do you see the trajectory of this field? What’s been surprising?

42:57 There are a handful of things. It’s surprising how effective artificial neural networks have been. That was a surprise, it wasn’t expected. Language models surprised me. There was a long-standing controversy in AI about simple methods like search and learning, compared to expert knowledge, domain-specific methods.

43:41 In the old days, it was interesting because they called search and learning “weak methods” because they’re just using computation, without the power that comes from imbuing a system with domain knowledge. But the weak methods have just totally won. That’s surprising and gratifying.

44:13 There’s a sense in which that was not surprising — I had always been rooting for the simple basic principles. But it’s surprising how well it worked, how completely they dominated. AlphaGo was surprising, how well that worked. But it’s all very gratifying because again, the basic principles won.

44:56 Whenever the public conception has been overturned by what AI has developed — for example, when AlphaZero became a superhuman chess player — did it feel to you like new breakthroughs were made? Actually, people had been developing these techniques since the ’90s. The whole AlphaGo thing had a direct lineage. Gerry Tesauro did reinforcement learning, temporal difference learning in backgammon. It beat the world’s best players. In some sense, AlphaGo was merely a continuation.

45:48 But it was quite a bit of scaling up and improvement in how the search was done. But it made the same basic bet. AlphaGo actually didn’t use TD learning. AlphaZero used TD. AlphaZero was applied to chess and that’s where the really interesting results came.

46:04 I’ve always been very impressed by the style of AlphaZero as a chess player. It just sacrifices material. It’s just content and patient to sacrifice pieces and get positional advantage. That was surprising that it worked so well, but gratifying.

46:31 This has led me where I am. It’s a strange position of someone thinking differently than the field is. I’ve been out of sync with my field for a long period. But occasionally I have been proved right in the past. My recipe for being out of sync and thinking in a strange way is to look not just at the current state of AI, but to look back in time and into history and to think about the mind in many different fields. I try to align with the larger traditions. I see myself as a classicist rather than as a contrarian. I’m trying to think what the great traditions of thinking about the mind have always thought.

47:30 I have some final questions for you if you’ll tolerate them. The Bitter Lesson — one reading is that it’s not necessarily saying that human knowledge isn’t useful, but that it obviously scales much worse than compute. So you want techniques which leverage the latter.

47:52 Once we have AGI, we’ll have researchers that are themselves AGI. We’ll have this avalanche of AI researchers. Their stock of knowledge will be growing as fast as compute. Will it then make more sense to have AI doing these artisanal solutions?

48:21 In terms of how AI research will evolve, I wonder: How did we get to this AGI? Suppose it started with general learning from experience. And now we want to go further. Interesting. You don’t think we’ll use AGI to design better AGI? But you’re using it to get AGI again. I guess there are different gradations of intelligence or competence at different tasks.

49:12 Maybe one way to motivate this is: AlphaGo was superhuman, but AlphaZero would beat AlphaGo every single time. AlphaZero was more superhuman than even superhuman. So it seems possible that the agent that achieves AGI first could be improved — there would be ways to give it better architecture. AlphaZero was an improvement upon AlphaGo and MuZero was further improvement. And the way AlphaZero was an improvement was that it learned purely from experience.

49:41 So why do you say, “Bring in other AI researchers,” when learning purely from experience has worked so well? I agree that in that particular case it was most effective. I meant to use that example to show how you can go from superhuman to superhuman++, to superhuman+++. Will that continue to happen by just adding more compute and more experience? Or, because we’ll have the capability of these intelligent researchers, as needed, will that continue to be a false path despite having trillions of AI researchers?

50:38 That’s interesting just to think about. Will the AI researchers teach each other the way cultural evolution works in people? The bitter lesson — who cares about that? It just describes a specific 70-year period in history. It doesn’t necessarily predict the future.

51:02 An interesting question is: you’re an AI agent and you just had an insight. Should you use it to make yourself smarter? Or should you use it to spawn off a copy of yourself on the other side of the planet or on some other task? I think that’s a really interesting question about the age of digital intelligences.

51:28 More questions: will it be possible to really incorporate something very new into your mind, and then will it be integrated smoothly? Or will it have changed so much that you’re no longer the same entity? Is that possible or is that not?

51:50 I was watching one of your videos the other night. It suggests a very decentralized world of digital intelligences that do different things. This will be such a powerful thing. A big issue will become corruption. If you can take knowledge from anywhere and bring it into your central intelligence — it’s all digital and they all speak the same language — maybe it’ll be easy and possible.

52:28 But if you take this thing and build it into your inner thinking, it could be your destruction rather than your improvement. I think this will become a big concern — cybersecurity. “Oh, he’s figured out all about how to play chess, and you want to incorporate that into your mind.” You might think, “Oh, that’ll be fine.” But actually, when you take a bunch of bits into your mind, they could have been corrupted. They can warp you and change you.

53:06 How do you have cybersecurity in the age of digital intelligences? I guess this brings us to succession. You have a perspective that’s quite different from most people I’ve interviewed and a lot of people generally. I want to hear about it.

53:25 Artificial intelligence or augmented humans is inevitable. There’s no government or organization that can stop it. There’s no single view that dominates and that can arrange how things should be run.

54:35 We will figure out how intelligence works. Number two, we won’t stop just at human-level — we will reach superintelligence. Number three, it’s inevitable that things that are smarter at getting things around would gain resources and power. You’re going to have succession to AI. Those four things seem clear and sure to happen. But there could be good outcomes as well as bad outcomes. I’m just trying to be realistic about where things are going.

55:22 I agree with all four of those points. I also agree that succession contains the possibility for good. Curious to get more thoughts on that. I do think positively about it. Intelligence is what we’ve always tried to do for thousands of years — try to think better, just understanding ourselves. We’re finding out what this essential part of us is.

55:58 But if we step aside from being a human and think about it from the universe’s perspective, this is I think a major stage in the universe, a phase transition. We humans and animals are evolved. That gives us some strengths and some limitations.

56:30 Our AIs are designed, our technology is designed. Now they’re reaching the point where they can be intelligent themselves and that they can do things at different speeds and in different ways.

57:25 This is a key step in the universe. It’s the transition from the evolved to the designed. The evolved can replicate — interesting things that are, are replicated. But you don’t really understand them. We have more children, but we don’t really understand how intelligence works. Whereas we’re reaching now to the point where we can create intelligence that we do understand how it works. That means it can evolve in very different ways and at different speeds.

58:05 We may just design AIs, and those AIs design better AIs, and everything will be done by design and intelligence rather than by evolution. I mark this as one of the four great transitions in the universe. First there’s dust, it ends with stars. Stars give rise to planets and life. Now we’re giving rise to designed entities.

58:45 Is this rise of designed intelligence part of humanity or different from humanity? It’s a philosophical question. We could say, “Oh, they are our offspring and we should be proud of their achievements.” Or we could say, “Oh no, they’re something else entirely.” It’s interesting that it could go either way.

59:20 It is interesting to consider if we are giving rise to something that might supersede us. Maybe Homo sapiens will give rise to something new. Something like that. Even if we consider them part of humanity, I don’t think that makes it super comfortable. Like Nazis were humans, right? If we thought everything humans do is fine, we’d be quite concerned about what our AI offspring might do.

1:00:00 So I agree that this is not super dissimilar from other transitions humanity has faced, but I don’t think that addresses a lot of the concerns about the level of power being attained this fast. I think it’s relevant to point out that most people don’t have much influence on what happens. The people who built the atom bombs or who control the nation states — regular people don’t control the nation states very much. A lot of the concern has to do with just how you feel about change. If you’re fearful, then you’re more likely to be suspicious.

1:00:50 I think it’s imperfect. I think humanity is imperfect. We should be open to change. Maybe it’s the best thing that there will ever be. But I think there are different varieties of change. The Bolshevik Revolution was also change. If someone said, “Look, things aren’t going well, we need change,” I’d want to know what kind of change. Similarly with AI, I’d want to understand what kind of change we’re getting.

1:01:14 We should be concerned about the risks. We should try to make it good. We should recognize our limits. We should avoid the feeling of entitlement — avoid saying “we should always have it in a good way.” How much control should a particular generation have over the long-term future? We have our own goals. We have our families. We shouldn’t be trying to control the whole universe. We should really work towards our own local goals rather than insisting “the future has to evolve this way that I want it to.”

1:02:05 People think the global future should go a certain way, and then they have conflict. We want to avoid that. Suppose you are raising your own children. Would you want to set very tight goals for their own life, or also have a sense that they should go out there in the world and have this specific impact? Like, “My child is going to become CEO of Intel and have this effect on the world.” I think the appropriate attitude is saying, “I’m going to raise them well and hope that if and when they do end up in positions of power, they’ll do good things.”

1:02:45 Maybe a similar attitude towards AI makes sense. Rather than trying to control exactly what they will do, or we have this plan about what AI should accomplish, it’s quite important to give them good values. Prosocial values? Are there universal values? I don’t think so, but that doesn’t prevent us from trying. Like we have some sense of wanting things to be good. We try to teach our children general principles.

1:03:38 Maybe we should also seek to instill good values in AI. If there is change, we want it to be gradual. I think that’s a very important point. That’s one of the really big human enterprises — to design society in a way that allows for gradual change. The more things change, the more important it is that we still have to figure out how to be.

1:04:20 Children develop values that seem strange to their parents. “The more things change, the more they stay the same.” That’s a good capsule into the AI discussion. The basic principles and techniques, which were invented decades ago — like temporal difference learning and backpropagation — their value was evident all along. Maybe that’s a good place to end. Okay. Thank you very much. My pleasure.