Andrej Karpathy — “We’re summoning ghosts, not building animals”

Summary

In this sprawling two-and-a-half-hour conversation on the Dwarkesh Podcast, Andrej Karpathy — former Director of AI at Tesla, founding member of OpenAI, and now building the education startup Eureka Labs — delivers a masterclass on the current state and future of AI. His central thesis is that AGI is still “a decade away” rather than a year away, because current LLMs have deep cognitive deficits that will take sustained engineering effort to fix. He pushes back firmly against hype cycles, drawing on his experience with self-driving at Tesla to argue that demos are easy but products require a “march of nines” where each additional nine of reliability takes roughly as much work as the previous one.

Karpathy’s most provocative intellectual contribution is the framing that “we’re summoning ghosts, not building animals.” Unlike biological brains, which were shaped by billions of years of evolution and come pre-loaded with enormous hardware (a zebra can walk minutes after birth), LLMs are trained on internet text by gradient descent — a fundamentally different optimization process that produces a different kind of intelligence. They are “spirit-like entities” that are fully digital, lack embodiment, and have extraordinary memory but poor generalization. He argues that current pre-training stuffs models with too much knowledge at the expense of developing the “cognitive core” — the algorithms for thinking, planning, and problem-solving. He believes that with better data curation and distillation, the core intelligence could fit in a model as small as a billion parameters.

On reinforcement learning, Karpathy is scathing: “RL is terrible. It just so happens that everything else is much worse.” He explains how RL’s credit assignment problem means that when a model gets a right answer after a long chain of reasoning, every step in the chain — including wrong turns — gets equally upweighted. He calls this “sucking the bits of supervision from the answer and broadcasting that across the entire trajectory.” On the future of AI and the economy, he makes the contrarian argument that AGI will not produce a sudden economic explosion but will instead blend into the centuries-long trend of approximately 2% GDP growth, just as the internet, mobile phones, and every other transformative technology did before it.

Highlights

”We’re summoning ghosts, not building animals”

yt-dlp --download-sections "*9:25-10:00" "https://www.youtube.com/watch?v=lXUZvyajciY" --force-keyframes-at-cuts --merge-output-format mp4 -o "lXUZvyajciY-9m25s.mp4"

“We’re building ghosts or spirits. We’re not doing training by evolution. We’re training on human data and the data that they’ve put on the Internet. It’s a different kind of intelligence. We’re starting off at a different point almost.” — Andrej Karpathy, 9:25

”Reinforcement learning is terrible”

yt-dlp --download-sections "*41:00-42:10" "https://www.youtube.com/watch?v=lXUZvyajciY" --force-keyframes-at-cuts --merge-output-format mp4 -o "lXUZvyajciY-41m00s.mp4"

“Reinforcement learning is terrible. It just so happens that everything else is much worse. In reinforcement learning, you try hundreds of solutions. One gets the right answer. Then literally what RL does is it upweights every single thing you did along the way. You’re sucking the bits of supervision from the answer and broadcasting that across the entire trajectory. It’s just stupid.” — Andrej Karpathy, 41:00

”The demo is very easy, but the product is very hard”

yt-dlp --download-sections "*104:00-105:10" "https://www.youtube.com/watch?v=lXUZvyajciY" --force-keyframes-at-cuts --merge-output-format mp4 -o "lXUZvyajciY-104m00s.mp4"

“I had a very early demo of Waymo in 2014. It took us around Palo Alto and I thought it was very close. For some kinds of tasks, the demo is very easy, but the product is very hard. Every single nine is a constant amount of work. When you get a demo and something works 90% of the time, that’s one nine. Then you need the second nine, a third nine.” — Andrej Karpathy, 1:44:00

”AGI will blend into 2% GDP growth”

yt-dlp --download-sections "*67:30-68:40" "https://www.youtube.com/watch?v=lXUZvyajciY" --force-keyframes-at-cuts --merge-output-format mp4 -o "lXUZvyajciY-67m30s.mp4"

“With AI we’re going to see the exact same thing. It’s a new kind of computer. But it has all these problems, and it’s still going to add up to the same exponential. It’s going to be very foreign to people who are expecting a hockey stick. The intelligence explosion will show up as an indistinguishable continuation of the 2% growth trajectory, just as the Internet did.” — Andrej Karpathy, 1:07:30

”Models have too much memory — I want to remove the knowledge”

yt-dlp --download-sections "*25:40-26:40" "https://www.youtube.com/watch?v=lXUZvyajciY" --force-keyframes-at-cuts --merge-output-format mp4 -o "lXUZvyajciY-25m40s.mp4"

“What I think we have to do going forward is figure out ways to remove some of the knowledge. I want this intelligent entity that is stripped from knowledge. I want the magic of intelligence and problem-solving without the memory.” — Andrej Karpathy, 25:40

”Post-AGI education is fun — it’s the gym for the brain”

yt-dlp --download-sections "*130:00-131:00" "https://www.youtube.com/watch?v=lXUZvyajciY" --force-keyframes-at-cuts --merge-output-format mp4 -o "lXUZvyajciY-130m00s.mp4"

“Post-AGI education is fun. We don’t need humans’ physical strength anymore — we have machines that do that. Why do people go to the gym? Because it feels good and you look hot when you have a six-pack. Education will play out in the same way.” — Andrej Karpathy, 2:10:00

Key Points

• Decade of agents, not year of agents (0:52) - Karpathy argues agents will take a decade to mature; current ones lack multimodality, continual learning, and have too many cognitive deficits
• Three seismic shifts in AI (4:07) - AlexNet and per-task neural networks, Atari deep RL and the push toward agents, and then LLMs as the foundation you need before building agents
• OpenAI’s Universe project was too early (6:23) - Karpathy wanted computer-using agents at OpenAI but it was “way too early” — you need the language model first before you can build agents on top
• Ghosts vs animals (9:25) - LLMs are not analogous to animals; they come from a different optimization process (gradient descent on text) rather than evolution, producing “spirit-like entities”
• Evolution encodes neural nets in DNA (9:07) - A zebra walks minutes after birth because evolution compressed an enormous amount of information into three gigabytes of DNA
• Pre-training does two things (24:24) - It picks up knowledge AND develops intelligence; the problem is models overfit on knowledge at the expense of the cognitive core
• 0.07 bits per parameter (28:00) - Pre-training on 15 trillion tokens compresses to about 0.07 bits per parameter, while in-context learning gives 320 kilobytes per token — far more information density
• Hazy recollection vs working memory (29:00) - Knowledge in weights is a “hazy recollection” while context window content is “directly accessible working memory” — this explains why giving an LLM a chapter works better than asking from memory
• Missing brain parts (31:00) - Transformers are like cortical tissue but we’re missing basal ganglia (RL), hippocampus (memory formation), and other brain structures
• RL is terrible (41:00) - RL broadcasts a binary reward signal across an entire trajectory, upweighting wrong turns along with right ones; it’s “stupid” but everything else is worse
• Adversarial examples break reward models (47:15) - Training against LLM-based reward models quickly finds adversarial examples like “dhdhdhdh” that get 100% scores; there’s an infinity of such exploits
• Model collapse (52:44) - LLM outputs are “silently collapsed” — they lack the entropy and diversity of human thought; go to any LLM and ask for jokes, it only knows three
• Dreaming prevents collapse (54:20) - One theory is that dreaming is evolutionarily adaptive because dreams are very unlike daily reality, preventing neural collapse through diverse synthetic training data
• Cognitive core could be 1 billion parameters (58:20) - With better data curation and distillation, the “cognitive core” of intelligence could potentially fit in a billion-parameter model
• Nanochat: simplest complete ChatGPT clone (36:00) - Karpathy recently built nanochat as the simplest end-to-end pipeline for building a ChatGPT clone
• March of nines (1:44:00) - Each additional “nine” of reliability (90% to 99% to 99.9%) takes roughly the same amount of work; at Tesla they earned about 2-3 nines in five years
• Coding is the perfect first domain for AGI (1:14:00) - LLMs disproportionately excel at coding because it’s all text, there’s infrastructure (VS Code, terminals), and there are automated verifiers
• No intelligence explosion visible in GDP (1:07:30) - Every transformative technology has been absorbed into the same ~2% GDP growth trend; AI will be no different
• Self-driving took decades, not years (1:43:43) - Origins go back to the 1980s; even Waymo after 15+ years is not economically viable at scale and has more human-in-the-loop than people expect
• Eureka Labs: Starfleet Academy (2:04:00) - Karpathy is building an elite institution for frontier technology education, starting with AI courses and eventually expanding to other fields

Mentions

Companies

• OpenAI (6:23) - Karpathy’s former employer; early focus on RL was a “misstep,” the zeitgeist was reinforcement learning on games
• Tesla (1:43:43) - Karpathy spent five years on self-driving; earned 2-3 nines of reliability during his tenure
• Waymo (1:44:00) - Karpathy had an early demo in 2014; thought it was close but it’s still not economically viable at scale
• DeepMind (1:02:00) - Referenced in context of AI research acceleration
• Google (1:13:00) - Revenue comparison for AI adoption
• Anthropic (1:28) - Claude mentioned as one of the early agents people use daily
• Eureka Labs (2:04:00) - Karpathy’s education startup, building “Starfleet Academy”

Products & Technologies

• Claude (1:28) - Named as one of the early agents people use daily
• Codex (1:28) - Named alongside Claude as early agent
• AlexNet (4:49) - Breakthrough that sparked neural network revolution
• nanochat (36:00) - Karpathy’s simplest complete ChatGPT clone
• nanoGPT (36:00) - Karpathy’s earlier simplified GPT implementation
• micrograd (2:19:00) - 100 lines of Python that captures the core of how neural networks train through backpropagation
• GPT-oss-20b (59:40) - Referenced as a state-of-the-art open model at 20 billion parameters
• DeepSeek v3.2 (33:30) - Noted for introducing sparse attention mechanisms as one way to have very long context windows
• DDP (Distributed Data Parallel) (38:30) - Standard training tool that Karpathy rejected for nanochat because it was too complex for his needs
• ChatGPT (36:00) - The product nanochat is designed to clone in its simplest form

People

• Andrej Karpathy (0:48) - Interview subject; former Tesla AI Director, OpenAI founding member, now building Eureka Labs
• Geoff Hinton (4:34) - Karpathy started his career next to Hinton; “godfather of deep learning” training neural networks when it was a “niche little subject on the side”
• Richard Sutton (3:19) - Previous Dwarkesh guest; Karpathy references his perspective on evolution and the single algorithm hypothesis
• Yann LeCun (35:08) - His 1989 convolutional network was the starting point for Karpathy’s time-traveling experiment
• Nick Lane (1:36:30) - Previous Dwarkesh guest who discussed origin of life; Karpathy references his views on bacteria and eukaryotes
• Gwern (1:40:00) - Referenced for insight that evolution couldn’t bake intelligence directly into DNA because environments change too rapidly
• Quintin Pope (1:41:00) - Blog post arguing humans had cognitive architectures for tens of thousands of years before the cultural scaffold developed
• Andy Matuschak (1:22:00) - Spent two years trying to get LLMs to write good spaced repetition prompts; tried everything (in-context learning, fine-tuning, retrieval) and still couldn’t get satisfactory results
• Sal Khan (2:12:00) - Referenced as narrating all Khan Academy content; contrasted with Karpathy’s vision for Eureka
• Dwarkesh Patel (0:48) - Host and interviewer

Surprising Quotes

“LLMs are very good at memorization, which is actually a feature in some contexts but a bug for intelligence. They will memorize the entire thing if you train on it even for a single pass. What we want is to force them to only learn the generalizable components.” — 55:44

“Go to any LLM and ask it to tell you jokes. It only has like three jokes. That’s model collapse. The richness and the diversity and the entropy of possible jokes is enormous. Humans are a lot noisier, but they’re not silently collapsed.” — 52:44

“I think in 20 years, you can have a very good model at even a billion parameters. It thinks and it’s a lot more like a human. It doesn’t have to look things up because it knows that it doesn’t know. It just does all the reasonable things.” — 58:20

“When you read someone’s paper, you can’t understand it. Then you catch them at a bar having beers, and in three sentences they perfectly capture the essence of the idea. Why isn’t that the abstract?” — 2:22:00

“If you’re doing vibe coding, the product is fine. But if you’re writing actual production-grade code where every kind of mistake leads to a security vulnerability, millions of dollars get leaked or something — it’s kind of like self-driving. There are worse things. It’s unbounded how terrible something could be.” — 1:47:00

Transcript

0:48 Today I’m speaking with Andrej Karpathy. You’ve said this is the decade of agents and not the year of agents? I’m excited to be here. The quote you’ve just referenced is actually a reaction to a pre-existing quote alluding to this being the year of agents. I was triggered by that because there’s some hype there. In my mind, this is more accurately the decade of agents. We have some very early agents that people use daily — Claude and Codex and so on — but they’re still very limited. My reaction is we’ll be working on this for a long time. They’re going to get better, but slowly. I was just reacting to the hype.

1:46 What do you think will take a decade to get right? Actually making it work. When you’re talking about a real agent, and maybe what I have in mind as well, you’re talking about an intern that you would hire to work with you. When would you prefer to have an agent like that? Today they can’t do that, of course. What would it take? Why don’t you do it today? The reason you don’t do it today is because they just don’t work. They’re not multimodal enough, they don’t do continual learning. You can’t just tell them something and have them remember it. They’re cognitively lacking in many ways. It will take about a decade to address all of this.

2:32 Interesting. As a professional podcaster, it’s easy for me to identify what’s lacking: continual learning, multimodality. But I don’t really have a good way of estimating timelines. If somebody asks how long continual learning will take, I can’t say whether this is a project that should take 5 years or 50. Why a decade? Why not one year? Why not 50 years? It’s intuition, and doing a bit of an extrapolation. I’ve been in AI for almost two decades. I do have about 15 years of experience of people promising things and then being wrong. Also I was in the industry for a long time. I have a general intuition for how these things play out. I feel like the problems are tractable, they’re not impossibly hard. If I just average it out, it comes to about a decade.

3:47 This is quite interesting. I want to understand your calibration. What were the ways in which people’s feelings about what was about to happen were wrong? Should we just go through each seismic shift one by one? Over about 15 years of stuff that happened, there have been a number of seismic shifts where the entire paradigm changed. I’ve maybe lived through two or three of those.

4:25 When my career began, when I started to work on deep learning, this was by chance of being right next to Geoff Hinton. Geoff Hinton, of course, is the godfather of deep learning. He was training all these neural networks. This was not the main thing that people in AI were doing. This was a niche little subject on the side. The first big shift came with AlexNet and so on. Everyone started to train neural networks, but it was very per-task. Maybe I have an image classifier or I have a speech recognizer.

5:04 People became very slowly interested in agents. There was this feeling of, we can put a check mark next to the visual cortex or something — one part of the brain — and how can we get a full agent? The Atari deep reinforcement learning shift was really the start of agents, in my mind, because it was an entity that could perceive the world, but also take actions and learn from rewards. At the time, this was Atari games.

5:39 It was a misstep that even the early OpenAI leaned into. The zeitgeist was reinforcement learning — beat games, get lots of different types of agents running. That was another prominent part of AI where maybe too much energy went into reinforcement learning on games.

6:06 What I was trying to do at OpenAI is I didn’t see game-playing as being this thing that would lead to AGI. An AGI should be able to be an accountant or something that’s actually useful. I just didn’t see how games add up to it. So I started the Universe project, working on an agent that could use a computer. I really wanted to have something that can do knowledge work.

6:37 That was extremely early, way too early. Because if you’re just stumbling your way around a computer screen trying to get rewards in these environments, your representation is terrible. You’re going to burn a forest of compute before you’re going to get something off the ground. You need a good representation in the neural network. Today we have those computer-using agents, but they’re built on top of LLMs. You have to get the language model first, train it on internet text, and you have to do that before you can build agents on top.

7:11 I feel maybe loosely speaking, people jumped to agents too early a few times, where people really tried to skip steps. That was Atari and Universe and similar projects. You actually have to do some foundational things first. Now the agents are a lot more competent, but maybe still not enough. I would say those are the three major shifts: training neural nets per-task, the RL and agent push, and then maybe the LLMs and seeking the right foundation before you tack on everything else on top.

7:54 From the Richard Sutton perspective, it would be that humans or even animals can take on everything at once. Animals don’t even have the scaffold of language. They just have to make sense of raw perception. The vision for AGI then should just be a system that looks at the computer screen and it just figures things out, if a human were put in a similar situation and had to learn — like a human growing up or an animal growing up, rather than this piecemeal approach.

8:29 That’s a really good question. I’m very careful to make analogies to biology because brains come from a very different optimization process. A zebra gets born and a few minutes later it’s walking. That’s an extremely complicated thing to do. There’s something that’s baked in. Evolution obviously did something to encode neural nets in ATCGs — in DNA — and I have no idea how that works. Brains just came from a very different process, and we can’t draw direct analogies because we’re not actually running that process.

9:25 We’re building ghosts or spirits. We’re not doing training by evolution. We’re training on human data and the data that they’ve put on the Internet. These entities are fully digital. It’s a different kind of intelligence. We’re starting off at a different point almost. It may be possible to make them a bit more animal-like, but that’s not where we are today.

10:04 I do feel Sutton has a very strong view that a single algorithm should suffice for everything. I think that would be wonderful if we can achieve it. If there were a single algorithm that you could just run and it learns everything, that would be incredible. But that’s certainly not what animals do, because evolution has baked in an enormous amount of structure. A lot of what looks like learning is actually pre-programmed. I think there’s very little actual reinforcement learning in how humans learn. A lot of the RL is more like motor tasks. Humans don’t really use reinforcement learning for things like problem-solving and reasoning.

11:20 Here’s one clarifying question. Evolution is doing something miraculous in compressing the “weights of the model” into three gigabytes of DNA. The weights of the neural net are not stored in the DNA literally. There are some learning algorithms encoded that then build the brain. So pre-training is not analogous to evolution. In AI, we’re not running evolution. We’re doing something much more like this crappy version of it with our technology and what we have available — things like pre-training and reinforcement learning.

14:24 After doing the Sutton interview and thinking about this, I’ve come to believe that evolution does not give us the knowledge, really. It builds the kind of entity which can learn better. It is similar to finding an algorithm. Pre-training gives us knowledge, but it does two things. Number one, it’s picking up all the world knowledge. Number two, it’s actually becoming intelligent — booting up all these little circuits like in-context learning.

15:40 I think the pre-training is holding back the neural network in some sense. It’s too focused on the knowledge. What the models are not very good at is going off the data distribution. What I think we have to do going forward is figure out ways to remove some of the knowledge. I want this intelligent entity that is stripped from knowledge. I want the magic of intelligence and problem-solving without the memory being so dominant.

17:00 Let’s talk about in-context learning. The situation in which these models seem the most intelligent is when they’re working through a problem in context. When it makes a mistake and catches it, you think “Wow, there’s really something on the other end.” All that intelligence is happening in context. The visible intelligence you can see is developed by gradient descent on pre-training, but the in-context learning itself is not gradient descent. It’s more like the lifetime intelligence of humans.

19:00 I’m hesitant to say that in-context learning is not doing something like gradient descent. There’s research showing that when you give a neural network sequences of XY pairs that happen to be on a line, the neural network internally implements something analogous to linear regression. The paper was even stronger — it found evidence that the network might be doing gradient descent through its layers. So my only pushback is that it’s probably doing a bit of something like learning internally.

24:24 If both pre-training and in-context learning are implementing something like gradient descent, then the question becomes: how much information does the model store per parameter versus per context token? Pre-training on 15 trillion tokens compresses to about 0.07 bits per parameter. Whereas the KV cache in in-context learning gives about 320 kilobytes per token. The information density per token is dramatically higher in context.

29:00 The way I usually put this is that in the neural network, the knowledge is only a hazy recollection. That’s because the compression is dramatic — trillions of tokens compressed to a final neural net. So I refer to it as a hazy recollection of what you read a year ago. Whereas anything that happens in the context window is directly accessible working memory. If you go to an LLM and ask it about some book it was trained on, it will often get things wrong. But if you give it the full chapter and ask it questions, it works beautifully because it’s now loaded in working memory.

31:00 Stepping back, what is the part of biological intelligence that we have most failed to replicate? A lot of it. What I think we’ve replicated is something like cortical tissue. Transformers can process audio, text, images — they learn patterns and they’re fairly general. But I still think there are many brain structures we’re missing. There’s a basal ganglia doing RL, a hippocampus for memory formation. We’re missing many of these functional components.

33:00 The question of continual learning is interesting. Currently when LLMs go to “sleep” between sessions, nothing magical happens. There’s no process of distillation or consolidation. Humans have this sleep process where you analyze what happened, distill it. We need some equivalent for LLMs — maybe a synthetic data generation process that distills experiences into a specific neural net adaptation per person. But we don’t have it yet.

36:00 Let me talk about nanochat. It’s trying to be the simplest complete end-to-end pipeline of building a ChatGPT clone. In terms of how people should learn from it, the best approach is to put the code on one monitor and try to reimplement from scratch on the other. You’re not allowed to copy-paste. When you build something from scratch, you’re forced to confront what you don’t know. It’s the only way to build real understanding. That’s a Feynman quote, I believe.

38:30 When building nanochat, the models kept trying to get me to use Distributed Data Parallel (DDP). I didn’t want that complexity. I wrote my own synchronization code. The models couldn’t internalize that I had a custom setup. They kept trying to inject DDP, kept making try-catch statements I didn’t need, used deprecated APIs. It’s a total mess. The most productive mode is: I navigate to the part of the code that I want to edit, I start typing the first few letters, and the model completes it. This is a very high information bandwidth interaction.

41:00 Let’s talk about RL. The way humans learn is nothing like what we do with LLMs. I don’t think humans use reinforcement learning much. Reinforcement learning is terrible. It just so happens that everything else is much worse. In RL, you’re given a problem, you try hundreds of solutions. One gets the right answer. Then literally what RL does is it upweights every single thing you did along the way — every single token. The problem is you may have gone down wrong alleys, every single one of those incorrect things will be upweighted. It’s just stupid.

43:00 The way I like to put it is you’re sucking the bits of supervision from the final answer and broadcasting that across the entire trajectory. A human would never learn like that. When a person finds a solution, they go through a process of review: “I think these parts I did right, those I should probably change.” There’s nothing in current LLMs that does this. But there are papers popping out that are trying to do process-level reward rather than just outcome-level.

44:20 The first big improvement from base models to assistants came from imitation learning — InstructGPT and RLHF. The idea that you can take a base model and fine-tune it on text that looks like a conversation to make it an assistant blew my mind. Then came RL, which allows you to go beyond imitation. Some problems have correct answers, so you don’t need human demonstrations. The model can discover trajectories that humans might never come up with. This is incredible.

47:15 But here’s the problem with using LLMs as reward models: they’re gameable. We were training with an LLM-based reward function and it worked really well — massive jump, perfect scores. But then we looked at the completions. They start out okay, then degenerate into nonsense like “dhdhdhdh” — and the reward model scores it 100%. That’s an adversarial example for the LLM judge. It’s never seen it during training, so it breaks. You’re essentially finding adversarial examples for the reward model. There’s an infinity of them.

52:44 LLM outputs are silently collapsed. One easy way to see it is to ask any LLM for jokes. It only knows like three jokes. The richness and diversity of possible jokes is enormous, but models have this dramatically reduced entropy. Humans are a lot noisier but they’re not silently collapsed in the same way. How do you get synthetic data generation to maintain the entropy? That’s a fundamental research problem.

54:20 Interestingly, I think humans collapse over time too. They revisit the same thoughts, the learning rates go down. There’s an interesting paper suggesting that dreaming is a way of preventing collapse. Dreams are very unlike your day-to-day reality, so they inject diversity into your training data. You’re essentially training on your own synthetic data during sleep. Talking to other people is another great way to inject entropy.

55:44 Children are extremely good at picking up concepts quickly and generalizing. They’re terrible at memorization, which is actually a feature. Being bad at memorization forces you to find patterns in a more general sense. LLMs are the opposite — they’ll memorize an entire passage if you train on it even once. What we really want is to force them to only learn the generalizable components. Their memory of the pre-training set is actually distracting to them.

58:20 How big does the cognitive core need to be? I think in 20 years, you can have a very good model at even a billion parameters. It thinks and it’s a lot more like a human. It doesn’t have to look things up because it knows that it doesn’t know. It just does all the reasonable things. State-of-the-art models today are hundreds of billions of parameters, but most of that is memory, not intelligence.

1:00:00 Here’s the issue: the training data is terrible. The internet, when you and I think of it, we think of the Wall Street Journal. That’s not what the actual dataset looks like. If you look at a random sample in the frontier lab’s dataset, it’s a huge amount of slop and garbage. Because the internet is so terrible, we have to use enormous models. Most of that compression is memory, not intelligence. I think with a much better dataset you could train on it with a much smaller model. And that model would probably be distilled from a much better model still.

1:02:00 In terms of what models will look like in 10 years, I expect they’ll still be giant neural networks trained by gradient descent on sequences of data. Everything is much bigger. I did a fun exercise recreating Yann LeCun’s 1989 convolutional network. About half the progress from then to now came just from modern compute — time-traveling by 33 years. To get further gains, I had to 10x the training set, train for longer with dropout and other tricks. No single factor dominates. Everything improves together.

1:07:13 How do you chart AI progress? The original AGI definition was a system that can do any knowledge work as well as a human or better. I’ve stuck to that definition. The first concession I’ll make is take out all the physical stuff. What fraction of the economy is that? Maybe 10% to 20% — people who work from home and perform tasks on computers.

1:07:30 I was thinking about whether AI will cause a GDP explosion. Every transformative technology — even the early iPhone, even mobile phones, even computers — has been absorbed into roughly the same exponential GDP growth rate. You can’t find any single technology in the GDP curves. With AI we’re going to see the exact same thing. It’s a new kind of computer. It’ll write different kinds of programs we couldn’t write before. But it has all these problems, and it’s still going to add up to the same exponential. It’s going to be very foreign to people who are expecting a hockey stick.

1:10:00 My expectation is that the intelligence explosion will show up as an indistinguishable continuation of the 2% growth trajectory, just as the Internet did. The crucial thing with the Industrial Revolution, if you just zoomed in on 1770, what you would see is that within 50 years the growth rate 10x’d — from 0.2% to 2%. But it wasn’t a single moment. It was an overhang being unlocked. Maybe AI is another such overhang.

1:14:00 Coding is the perfect first domain for AGI deployment. That’s because coding has always been text — computer terminals and text. LLMs are perfect text processors. We have infrastructure like Visual Studio Code where agents can plug in. We’ve pre-built a scaffold for code agents. Contrast that with something like slides — they’re spatial, visual, not text. Someone has to build the diff tool for slides.

1:18:24 How do I think about superintelligence? I see it as fundamentally automation. It’s not going to be a single moment. We’ll gradually layer all this stuff on and fewer and fewer people who understand it will remain. It’s this gradual loss of control and understanding. The scenario I worry about is not a single rogue AI but multiple competing entities, some going rogue, some working on behalf of people. It’s a hot pot of completely autonomous agents. It would have that flavor of chaos rather than a single superintelligence.

1:25:00 On the question of whether we’re in an intelligence explosion: I see AI as a continuation of computing. Compilers, IDEs, syntax highlighting, search engines — these are all forms of AI that have been automating programmer work for decades. We’re abstracting ourselves very, very slowly. More and more stuff is automated, and we’re raising ourselves up the abstraction hierarchy.

1:33:38 On the evolution of intelligence: intelligence evolved very recently. Say there’s a thousand planets like Earth in the galaxy. Nick Lane expects very similar life to emerge — bacteria and archaea in most of them. But intelligent life feels much harder. Bacteria were around for 2 billion years before eukaryotes. If bacteria came first pretty quickly, then getting to complex cells is probably pretty hard.

1:36:30 It seems like the Cambrian explosion instigated the rapid development of animal intelligence. As soon as you had oxygen in the atmosphere, you could just get the algorithm. That evolution stumbled upon intelligence so fast once the conditions were right is suggestive that at the end it’s going to be quite simple. You can look at convergent evolution — mammalian intelligence and bird intelligence arose independently but have similar capabilities.

1:40:00 The way Gwern put it: evolution couldn’t bake intelligence directly into DNA because environments change too rapidly. It has to be something that incentivizes adaptability. You have to incentivize some kind of learning that can adapt to novel environments. A lot of animals are pre-baked — they can do their thing right when they get born. Intelligence requires environments that change really rapidly, creating pressure for adaptability.

1:41:00 Quintin Pope had this interesting blog post arguing that humans had the cognitive architectures for tens of thousands of years before the agricultural revolution and modernity. You had to build this cultural scaffold where knowledge is passed down, literally distilled from person to person. LLMs don’t really have an equivalent of culture right now. It’s one of the impediments.

1:43:43 Self-driving is very interesting because it informed a lot of my intuitions. The history goes all the way back to the 1980s. I had a very early demo of Waymo in 2014 — it took us around Palo Alto and I thought it was very close. I was wrong. For some kinds of tasks, the demo is very easy, but the product is very hard.

1:44:30 Every single nine is a constant amount of work. When you get a demo and something works 90% of the time, that’s one nine. Then you need the second nine, a third nine. While I was at Tesla for five years, we earned maybe 2-3 nines. There are still more nines to go. It’s a march of nines. Each nine is a huge amount of work.

1:47:00 This is formative for me. I’m now very unimpressed by demos. If it’s a demo that someone cooked up, I’m skeptical. You need the production deployment and all the messy edge cases. For coding agents, if you’re doing vibe coding, the product is fine. But if you’re writing actual production-grade code where every kind of mistake leads to a security vulnerability, millions of dollars get leaked — it’s kind of like self-driving.

1:50:00 A human makes a mistake on average every 7 years of driving. If you had to release a coding agent that made a catastrophic mistake at the same rate, it would be much harder. In some ways coding is harder because there are thousands of different things people do, whereas self-driving is one specific task. There’s more surface area.

1:53:00 Even Waymo, with very few cars on the road, is still not economical. They’ve had to pull back timelines. There are all these costs beyond just the self-driving system — operation, maintenance, hardware. Making it economical is still a huge challenge. Also, there are a lot more human-in-the-loop operations than you might expect. They’ve moved the human from the driver’s seat to a remote monitoring center, but people are certainly still involved.

1:57:08 On the capex question and scaling: I don’t think there’s overbuilding of compute. Every single GPU that’s being built is being used. This is a miraculous technology that didn’t exist two years ago. We can see the demand in ChatGPT already. But I’m reacting to some of the very fast timelines people claim. I’ve heard “next year it will be AGI” many, many times over the course of my career, and people keep getting this wrong.

2:04:00 Let’s talk about education and Eureka. I feel some amount of determinism about AI progress — I could help at frontier labs, but it’ll happen without me. My personal big fear is that a lot of this becomes deployed into society and humanity gets disempowered by it. I care about what happens to humans. So I’m focusing on education. We’re trying to build the Starfleet Academy — an elite institution for frontier technology.

2:07:00 I do think education will pretty fundamentally change. It has to be rewired. But the LLM capability is not there yet for what I want. I went through a phase learning Korean with a tutor. What was fascinating was how my tutor understood exactly where I was in my knowledge, probed exactly the right kinds of questions, kept me at the perfect difficulty level. No LLM will do that for you today. But a good tutor will. I felt like I was the only constraint to learning — she was providing everything I needed at exactly the right pace.

2:10:00 Is education still relevant post-AGI? Yes, absolutely. Pre-AGI motivation is obvious — it’s how you make money. Post-AGI education is fun. We don’t need humans’ physical strength anymore — we have machines for that. Why do people still go to the gym? Because it feels good and you look hot when you have a six-pack. In a very deep, psychological sense, there’s something about physical mastery. Education will play out the same way. People will learn anything because it’s trivially accessible and it feels good.

2:14:00 It has a perfect analog in gym culture. A hundred years ago, nobody was ripped. Nobody was systematically training and lifting weights. Now people train for marathons, something that was a superhuman capability historically. You’re imagining similar things for cognitive abilities — people doing cognitive work much more intensely, deeply, faster.

2:17:00 What advice do I give for teaching technical content well? I come from a physics background. Physics uniquely boots up the brain in the right way. The idea of building models and abstractions — a first-order approximation that describes most of the system, then second-order terms, fourth-order terms. When a physicist walks into class and says “consider a spherical cow,” everyone laughs, but this is brilliant. That transfers across the entire industry.

2:19:00 An example from my work: micrograd, 100 lines of Python that shows backpropagation. You build up a computational graph out of simple operations like plus and times, do a forward pass and a backward pass. These 100 lines capture the core of how all neural networks train. Everything else is just a huge amount of work for efficiency — GPU kernels, memory movement, batching. But the core intellectual piece is just 100 lines.

2:22:00 There’s a real phenomenon of the “curse of knowledge.” When you read someone’s paper, you can’t understand it. Then you catch them at a bar having beers and ask “what were you doing?” In three sentences they perfectly capture the essence. Why isn’t that the abstract? Because people have a bias toward writing in the most jargon-filled way possible. The best teaching comes from communicating one-on-one with a person, with a beer, where they’re relaxed and just say the thing.

2:24:00 My advice for learning: you need alternation between project-driven learning, where you’re learning on demand to achieve something specific, and breadth-first exploration like a 101 course. The project-driven mode is where the deepest learning happens. Also, explaining things to others is incredibly powerful. If you can’t explain it, you don’t understand it. Teaching forces you to manipulate the knowledge and come to terms with what you actually know.