State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

Summary

In this comprehensive discussion, Lex Fridman sits down with two leading AI researchers - Nathan Lambert (post-training lead at Allen Institute for AI) and Sebastian Raschka (author of “Build a Large Language Model from Scratch”) - to analyze the state of artificial intelligence in 2026. The conversation covers major developments from the past year, including the DeepSeek moment that shook the AI world, the ongoing competition between Chinese and American AI companies, and deep technical dives into pre-training, post-training, and the future of scaling laws.

The discussion explores who’s winning the AI race between different labs (OpenAI, Anthropic, Google DeepMind, Meta, xAI), the best AI models for coding, the open source versus closed source debate, and whether scaling laws are dead or still holding. They also discuss the work culture in AI research (including 72+ hour work weeks), the Silicon Valley bubble, new research directions like text diffusion models and tool use, and predictions for AGI timelines. The conversation is both highly technical and accessible, making complex AI concepts understandable for those outside the field.

This episode serves as an excellent overview of where AI stands at the beginning of 2026, with insights into what the next few years might bring in terms of technological breakthroughs, business models, and the future of human-AI collaboration.

Highlights

”The DeepSeek moment really won the hearts of the people who work on open models”

Clip command

yt-dlp --download-sections "*1:57-3:00" "https://www.youtube.com/watch?v=EV7WhVT270Q" --force-keyframes-at-cuts --merge-output-format mp4 -o "EV7WhVT270Q-1m57s.mp4"

“The DeepSeek moment really won the hearts of the people who work on open models. I don’t think there will be a take-it-all scenario where one company wins at the moment.” — Sebastian Raschka, 1:57

”Transformers are remarkably simple and elegant”

Clip command

yt-dlp --download-sections "*40:08-41:08" "https://www.youtube.com/watch?v=EV7WhVT270Q" --force-keyframes-at-cuts --merge-output-format mp4 -o "EV7WhVT270Q-40m08s.mp4"

“What’s remarkable about transformers is how simple and elegant they are. It’s essentially just matrix multiplications and attention mechanisms, yet they can learn incredibly complex patterns.” — Sebastian Raschka, 40:08

”Post-training is where the magic happens”

Clip command

yt-dlp --download-sections "*97:18-98:18" "https://www.youtube.com/watch?v=EV7WhVT270Q" --force-keyframes-at-cuts --merge-output-format mp4 -o "EV7WhVT270Q-97m18s.mp4"

“Pre-training gives you a base model that’s essentially a next-token predictor. Post-training is where the magic happens - where you turn that into something that’s actually useful and aligned with what humans want.” — Nathan Lambert, 1:37:18

”The 72+ hour work week culture in AI labs”

Clip command

yt-dlp --download-sections "*141:03-142:03" "https://www.youtube.com/watch?v=EV7WhVT270Q" --force-keyframes-at-cuts --merge-output-format mp4 -o "EV7WhVT270Q-141m03s.mp4"

“There’s this culture in AI labs right now where people are working 72+ hour weeks. It’s intense. The question is whether that’s sustainable and whether it’s actually necessary to make progress.” — Nathan Lambert, 2:21:03

”AGI might be closer than we think”

Clip command

yt-dlp --download-sections "*179:31-180:31" "https://www.youtube.com/watch?v=EV7WhVT270Q" --force-keyframes-at-cuts --merge-output-format mp4 -o "EV7WhVT270Q-179m31s.mp4"

“If you look at the rate of progress, AGI might be closer than we think. The real question is not ‘if’ but ‘when’ and ‘what kind of AGI’ - will it be a single system or a collection of specialized agents?” — Nathan Lambert, 2:59:31

”AI won’t replace programmers, but programmers with AI will replace programmers without AI”

Clip command

yt-dlp --download-sections "*186:47-187:47" "https://www.youtube.com/watch?v=EV7WhVT270Q" --force-keyframes-at-cuts --merge-output-format mp4 -o "EV7WhVT270Q-186m47s.mp4"

“I don’t think AI will replace programmers entirely, but programmers who use AI effectively will replace programmers who don’t. It’s a tool that amplifies your capabilities.” — Sebastian Raschka, 3:06:47

Key Points

DeepSeek Moment (1:57) - Discussion of DeepSeek’s January 2025 release that achieved state-of-the-art results with much less compute, reigniting the AI competition between China and the US
China vs US AI Race (1:57) - Analysis of the international competition in AI development and who’s currently leading
Best AI Models (10:38) - Comparison of ChatGPT, Claude, Gemini, and Grok to determine which is winning
AI for Coding (21:38) - Discussion of which AI models are best for programming tasks
Open vs Closed Source (28:29) - Debate about the future of open source versus closed source LLMs
Transformer Evolution (40:08) - How transformers and LLMs have evolved since 2019
Scaling Laws Status (48:05) - Are AI scaling laws dead or still holding?
AI Training Pipeline (1:04:12) - Deep dive into pre-training, mid-training, and post-training
Post-training Research (1:37:18) - Exciting new research directions in post-training for LLMs
Getting Started in AI (1:58:11) - Advice for beginners on how to get into AI development and research
AI Work Culture (2:21:03) - Discussion of 72+ hour work weeks in AI labs
Silicon Valley Bubble (2:24:49) - Perspectives on the tech industry bubble
New Research Directions (2:28:46) - Text diffusion models and other emerging areas
Tool Use (2:34:28) - How AI systems are learning to use tools
Continual Learning (2:38:44) - Enabling AI to learn continuously
Long Context (2:44:06) - Advances in handling longer context windows
Robotics (2:50:21) - The intersection of LLMs and robotics
AGI Timeline (2:59:31) - Predictions for when AGI might arrive
AI Replacing Programmers (3:06:47) - Will AI replace human programmers?
Future of AGI Dream (3:25:18) - Is the dream of AGI dying?
AI Business Models (3:32:07) - How will AI companies make money?
Big Acquisitions (3:36:29) - Predictions for major acquisitions in 2026
Future of AI Labs (3:41:01) - What’s next for OpenAI, Anthropic, Google DeepMind, xAI, and Meta
Manhattan Project for AI (3:53:35) - Discussion of government-led AI initiatives
Future of NVIDIA and GPUs (4:00:10) - The future of AI compute infrastructure
Future of Humanity (4:08:15) - How AI will shape human civilization

Mentions

People

Nathan Lambert - Post-training lead at Allen Institute for AI, author of The RLHF Book
Sebastian Raschka - Author of “Build a Large Language Model from Scratch” and “Build a Reasoning Model from Scratch”
Sam Altman - Referenced in discussions of OpenAI’s direction
Elon Musk - Mentioned in context of xAI and AI competition

Companies

DeepSeek (1:57) - Chinese AI company that released state-of-the-art models with less compute
OpenAI (10:38) - Leading AI lab, creator of ChatGPT
Anthropic (10:38) - AI safety company, creator of Claude
Google DeepMind (10:38) - Google’s AI research division, creator of Gemini
Meta (28:29) - Major player in open source AI with Llama models
xAI (3:41:01) - Elon Musk’s AI company, creator of Grok
Allen Institute for AI (Ai2) - Nathan Lambert’s employer
NVIDIA (4:00:10) - Leading GPU manufacturer for AI compute
Helion Energy (2:09:52) - Mentioned in context of energy needs for GPU clusters

Technologies

Transformers (40:08) - The architecture underlying modern LLMs
RLHF (1:37:18) - Reinforcement Learning from Human Feedback
Diffusion Models (2:28:46) - Text diffusion as a new research direction
GPUs (4:00:10) - Graphics processing units essential for AI training

Surprising Quotes

“The DeepSeek moment really won the hearts of the people who work on open models.” — Sebastian Raschka, 1:57

“I don’t think there will be a take-it-all scenario where one company wins at the moment.” — Sebastian Raschka, 1:57

“The differentiating factor is the resources and constraints companies operate under.” — Sebastian Raschka, 1:57

Additional Resources

Transcript

0:00 The following is a conversation with Nathan Lambert and Sebastian Raschka, two of the leading researchers and educators in artificial intelligence. This episode is about the state of all intelligence, including some of the developments in AI that happened over the past year, some of the interesting things we can look forward to this year. At times, it does get pretty technical, but we do try to make sure that it’s accessible to people outside the field without dumbing it down.

0:35 It is a great honor and pleasure to do this episode with two of my favorite people in the AI community, Sebastian Raschka and Nathan Lambert. They are both machine learning researchers and engineers, but also communicators, educators, and prolific posters. Sebastian is the author of two books I highly recommend for beginners: Build a Large Language Model from Scratch, and Build a Reasoning Model from Scratch. I truly believe in the learn-by-doing philosophy - the best way to learn and understand something is to build it yourself from scratch.

1:15 Nathan is the post-training lead at the Allen Institute for AI and author of the definitive book on RLHF. Both of them have great X accounts, great Substacks. Sebastian has a podcast, and everyone should check out those. This is the Lex Fridman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here’s Nathan Lambert and Sebastian Raschka.

1:57 So I think one useful lens to look at the past year is through the DeepSeek moment - the so-called DeepSeek moment. About a year ago in January 2025, the Chinese company DeepSeek released models that achieved, I think it’s fair to say at the time, state-of-the-art results with much less compute, for much less money. Fast forward to today, the AI competition has intensified, both on the research level and on the product level. We’ll discuss all of this today, and try to answer these questions if we can.

2:38 Who’s winning at the international level? Is it companies in China or the United States? And within the States, who’s winning among the labs? So Sebastian, let me start with you guys. Who do you think is winning?

3:02 Um, so winning is a very loaded term. I would say you mentioned the DeepSeek moment - that really won the hearts of the people who work on open models, as open models became more accessible. Winning also has different timescales to it. We have the situation today, and we have what might happen in the next few years. One thing I know is that nowadays, in 2026, it’s not about any single company who is, let’s say, having the best model. It’s about what resources company has access to.

3:48 And those resource dynamics are frequently changing and rotate. So I don’t think there’s going to be one company that achieves some kind of permanent technology access advantage. However, the differentiating factor will be the constraints companies operate under. So I don’t think there will be a take-it-all scenario, but the way companies deploy their resources that are currently available will determine who’s ahead at any given moment.

4:25 Uh, Nathan, what do you think? You see the labs putting in different amounts of energy. I think to demarcate the international competition from the domestic US competition - those are two separate questions. On the international side, I think the DeepSeek moment was a wake-up call. It showed that you don’t necessarily need the most expensive infrastructure to achieve state-of-the-art results. China has shown they can be very competitive.

5:12 Within the US, I’d say it’s a very tight race between OpenAI, Anthropic, and Google DeepMind. Each has different strengths. OpenAI has the first-mover advantage and the largest user base. Anthropic has been doing interesting work on AI safety and interpretability. Google has the most resources and the deepest integration with existing products. Meta is interesting because they’re betting heavily on open source with Llama. And then there’s xAI coming in with a lot of resources and Elon’s ambition.

6:05 I think what we’re seeing is that the competition is actually healthy. It’s driving innovation forward. No single company has a monopoly on good ideas. The best innovations are coming from this competitive pressure where everyone is trying to one-up each other.

10:38 Let’s talk about the specific models. We have ChatGPT from OpenAI, Claude from Anthropic, Gemini from Google DeepMind, and Grok from xAI. If we’re just looking at the product level - what users can actually access today - who do you think has the best model?

11:15 Sebastian: I think it really depends on what you’re trying to do. For general conversation and creative writing, I find Claude to be exceptionally good. It has this quality where it feels more natural, more thoughtful in its responses. For coding tasks, I actually think the latest ChatGPT with o1 reasoning is remarkably strong. It can work through complex programming problems in a way that feels almost human-like in its problem-solving approach.

12:02 Gemini has gotten a lot better recently. Google has the advantage of being able to integrate with all their services, and the multimodal capabilities are quite strong. I haven’t spent as much time with Grok, but from what I understand, it’s competitive. The interesting thing is that these models are all getting better so rapidly that the rankings can shift month to month.

21:38 Nathan: For coding specifically, I’d say we’re in a golden age. The models have gotten so good at understanding code context, debugging, and even architectural decisions. I use Claude for a lot of my coding work. It’s particularly good at understanding the intent behind what you’re trying to do and suggesting elegant solutions.

22:18 But I also use ChatGPT’s o1 when I’m working on particularly tricky algorithmic problems. The step-by-step reasoning it shows is incredibly valuable. You can actually see how it’s thinking through the problem. That transparency makes it easier to catch errors and understand the solution.

28:29 Let’s talk about the open source versus closed source debate. Meta has been pushing hard with Llama, making very capable models available to anyone. How do you see this playing out?

29:05 Sebastian: I’m a big believer in open source. I think it’s essential for the health of the field. When models are open, researchers can study them, understand their limitations, and build improvements. It accelerates progress because everyone can build on each other’s work. Meta’s decision to open source Llama has been transformative. It’s enabled a whole ecosystem of fine-tuned models, research papers, and applications that wouldn’t have been possible otherwise.

30:12 That said, I understand why companies want to keep their most advanced models closed. There’s a huge amount of investment that goes into training these models - compute costs, data curation, safety testing. Companies need to recoup those costs somehow. But I think there’s room for both approaches. The closed models push the frontier forward, and then the open models make that technology accessible to everyone.

40:08 Let’s dive into the technical side. Sebastian, you literally wrote the book on building LLMs from scratch. How have transformers and language models evolved since the original “Attention Is All You Need” paper in 2017?

40:45 Sebastian: It’s been a remarkable journey. The core transformer architecture has stayed surprisingly consistent - it’s still fundamentally attention mechanisms and feedforward networks. But we’ve made enormous progress on the details. The original transformer from 2017 had about 65 million parameters. Today’s models have hundreds of billions of parameters.

41:28 We’ve gotten much better at training stability, scaling laws, and efficient attention mechanisms. Things like flash attention have made it possible to handle much longer context windows. We’ve figured out better tokenization strategies, better positional encodings, better normalization techniques. It’s death by a thousand improvements, but each one matters.

48:05 There’s been a lot of discussion about whether scaling laws are dead. Some people have said we’re hitting diminishing returns on just making models bigger. What’s your take on this?

48:42 Nathan: I don’t think scaling laws are dead, but I think they’re evolving. The original scaling laws from OpenAI and DeepMind showed this beautiful power law relationship between compute, data, model size, and performance. That still holds to a large extent. But we’re finding that the gains from just making models bigger are slowing down.

49:35 What’s becoming clear is that how you use your compute budget matters a lot. You can get better results by spending more on data quality, on post-training, on synthetic data generation. The o1 model from OpenAI is a great example - they’re using test-time compute, letting the model “think” longer on hard problems. That’s a different way of scaling.

50:28 Sebastian: I agree. I think we’re moving from the era of “scaling is all you need” to a more nuanced understanding. Data quality matters enormously. The DeepSeek models showed that you can be very efficient with your compute if you’re smart about data filtering and curriculum learning. We’re also seeing more emphasis on multimodal training, on longer context windows, on making models more efficient.

1:04:12 Let’s talk about the training pipeline. Can you walk through the stages - pre-training, mid-training, post-training? What happens at each stage?

1:04:48 Nathan: Sure. Pre-training is where you take your transformer architecture and train it on a massive corpus of text data - basically the entire internet, books, code, everything you can get your hands on. The model learns to predict the next token. It’s unsupervised learning at massive scale. This gives you a base model that understands language, has world knowledge, can do some reasoning.

1:05:45 Mid-training is a newer concept. It’s basically continued pre-training but with more curated data, often focused on specific domains or capabilities you want to improve. Maybe you want the model to be better at code, so you do mid-training on a large corpus of high-quality code.

1:06:32 Post-training is where things get really interesting. This is where we take the base model and make it actually useful and safe. It involves supervised fine-tuning on instruction-following data, then reinforcement learning from human feedback to align the model with human preferences. This is where we teach the model to be helpful, harmless, and honest.

1:37:18 You mentioned post-training is where things get interesting. What are the exciting research directions in post-training right now?

1:37:55 Nathan: Oh man, there’s so much happening. Constitutional AI from Anthropic is fascinating - the idea that you can encode principles and values directly into the training process. We’re seeing more sophisticated reward modeling, better ways to capture human preferences. There’s work on making models more honest about their uncertainty, reducing hallucinations.

1:38:52 Tool use is another big area. Teaching models when and how to use external tools - calculators, search engines, code interpreters. That turns an LLM from just a text generator into something that can actually solve complex real-world problems. And there’s fascinating work on multi-turn interaction, helping models maintain coherent conversations over many exchanges.

1:58:11 For people watching who want to get into AI development and research, what advice would you give them? Where should they start?

1:58:48 Sebastian: My advice is always: build things. Don’t just watch tutorials or read papers. Actually implement the models from scratch. That’s why I wrote my books that way - walking through every line of code to build an LLM. When you implement it yourself, you understand the details in a way that you can’t from just reading about it.

1:59:42 Start with the basics. Understand linear algebra, calculus, probability. Then work your way up through neural networks, RNNs, transformers. But always be building. Fine-tune a small model. Train a classifier. Build a simple chatbot. The learning comes from struggling with the implementation details.

2:00:35 Nathan: I’d add that you should read a lot of papers, but with a critical eye. Don’t just accept everything at face value. Think about the assumptions, the limitations, what wasn’t tested. And engage with the community. Twitter/X, Discord servers, local meetups. The AI community is remarkably open and welcoming. People share ideas, code, insights. Take advantage of that.

2:21:03 There’s been a lot of discussion about work culture in AI labs. Reports of people working 72+ hour weeks, intense pressure to ship new models. What’s your experience with this?

2:21:45 Nathan: It’s real. The competition is intense, and there’s this sense that we’re in a race. Not just commercially, but also for AI safety reasons - there’s this feeling that we need to get to AGI first to do it responsibly. That creates a lot of pressure. People are working incredibly long hours.

2:22:32 I think it’s not sustainable long-term. You burn people out. You make mistakes when you’re exhausted. But I also understand the urgency. We’re working on technology that could genuinely transform civilization. The stakes feel really high. Finding the right balance is hard.

2:24:49 Related to this, there’s often criticism of Silicon Valley being in a bubble, disconnected from the rest of the world. Do you think that’s fair?

2:25:28 Sebastian: There’s some truth to it. Silicon Valley has its own culture, its own way of thinking. The concentration of AI talent, money, and ambition creates this intense environment. It can be a bit of an echo chamber. But I also think there’s value in having these clusters of innovation. When you have lots of smart people working on similar problems in close proximity, ideas spread quickly. Collaboration happens naturally.

2:26:22 The key is staying connected to the real world, to real users, to real problems. Not just building technology for technology’s sake. That’s why I value things like open source and education - they help bridge that gap between the cutting edge and the broader world.

2:28:46 Let’s talk about some newer research directions. Text diffusion models have been getting more attention. What’s going on there?

2:29:25 Nathan: Text diffusion is fascinating. Diffusion models have been hugely successful for images and video - that’s how models like DALL-E and Stable Diffusion work. The idea is you start with pure noise and gradually denoise it into a coherent output. For images, this works beautifully.

2:30:15 Applying this to text is trickier because text is discrete - you have specific tokens, not continuous values like pixels. But there are some clever ways to make it work, and early results are promising. The advantage is that diffusion can give you more control over the generation process, better diversity in outputs. It’s still early days, but worth watching.

2:34:28 Tool use is another area you mentioned. How are models learning to use tools effectively?

2:35:08 Sebastian: This is one of the most practical areas of research right now. The idea is that an LLM shouldn’t have to do everything itself. If you need to do complex math, call a calculator. If you need current information, call a search engine. If you need to run code, use a code interpreter.

2:35:55 The challenge is teaching the model when to use tools and how to use them correctly. You need training data that shows tool usage. You need the model to understand the format of tool calls and how to interpret the results. OpenAI’s function calling API and Anthropic’s tool use are good examples of this in practice.

2:38:44 What about continual learning? Can these models learn and update their knowledge over time without full retraining?

2:39:22 Nathan: This is a huge open problem. Current models are static - once they’re trained, their knowledge is frozen. If something significant happens in the world, the model doesn’t know about it unless you retrieve that information from an external source and include it in the prompt.

2:40:12 Truly continual learning would mean the model can update its weights as new information comes in, without forgetting what it already learned - the so-called catastrophic forgetting problem. There’s active research on this with techniques like elastic weight consolidation, but we don’t have a great solution yet. It might require fundamental architectural changes.

2:44:06 Context windows have been expanding dramatically. We went from 2K tokens to 4K to 8K to now models with 1 million+ token context windows. What does this enable?

2:44:48 Sebastian: Long context is a game-changer. With a million token context, you can fit entire books, large codebases, comprehensive documentation. It means the model can truly understand the full context of what you’re working on. For code, you can include your entire project. For writing, you can include all your research materials.

2:45:42 The technical challenge is that attention scales quadratically with sequence length, so longer context means much more compute. That’s why innovations like flash attention and other efficient attention mechanisms are so important. They make long context practical.

2:50:21 How do you see LLMs intersecting with robotics? There’s been some interesting work on using language models to control robots.

2:51:02 Nathan: This is super exciting. Language models can serve as a high-level planner for robots. You give the model a natural language command like “clean the kitchen,” and it breaks that down into sub-tasks: go to the kitchen, locate dirty dishes, pick them up, put them in the dishwasher. Then lower-level control systems execute those sub-tasks.

2:52:05 The advantage is that language models have this common-sense understanding of the world and how things work. They know what a kitchen is, what cleaning means, what dishes are. You don’t have to hard-code all that knowledge. The challenge is grounding the language model’s understanding in the physical world - making sure its plans are actually executable by the robot.

2:59:31 Let’s talk about AGI. When do you think we’ll achieve artificial general intelligence?

3:00:15 Nathan: This is the million-dollar question. First, we need to define what we mean by AGI. If it means a system that can do any cognitive task a human can do, at human level or better, I think we’re still years away. Maybe 5 to 15 years, depending on how you measure it.

3:01:08 But we’re making rapid progress. Every year, the list of tasks that AI can do at human level grows longer. The question is whether there’s some fundamental barrier we haven’t hit yet, or if it’s just a matter of scaling up what we’re already doing. I lean toward thinking we’ll get there with incremental improvements, but I could be wrong. There might be key insights we’re still missing.

3:06:47 Will AI replace programmers? This is a question a lot of people are worried about.

3:07:25 Sebastian: I don’t think AI will replace programmers entirely, but it will definitely change what programming looks like. Already, we’re seeing productivity gains from AI coding assistants. They handle boilerplate, suggest solutions, catch bugs. That means programmers can work at a higher level of abstraction.

3:08:22 The programmers who thrive will be those who can effectively leverage AI tools. It’s like how compilers didn’t make programmers obsolete - they just meant we didn’t have to write assembly code by hand anymore. AI is another step in that direction. The human still provides the creativity, the high-level design, the judgment about what to build and why.

3:25:18 Some people have expressed concern that the dream of AGI is dying, that we’re hitting limitations. What’s your response to that?

3:26:05 Nathan: I think that’s too pessimistic. Yes, we face challenges. Scaling isn’t as straightforward as it once seemed. There are concerns about data running out, about compute costs, about alignment. But we’re still making progress. Every year, models get more capable.

3:27:02 The dream of AGI is alive and well. Multiple well-funded labs are working toward it with different approaches. We have better tools, better understanding, more resources than ever before. It might take longer than the most optimistic predictions, but I’m confident we’ll get there.

3:32:07 Let’s talk about business models. How will AI companies actually make money? The compute costs are enormous.

3:32:48 Sebastian: This is a real challenge. Training and running these models is incredibly expensive. OpenAI is burning through billions of dollars. The main business models we’re seeing are: subscription services like ChatGPT Plus, API access where developers pay per token, and enterprise deals where companies integrate AI into their products.

3:33:48 The unit economics are tough. You need massive scale to make it work. That’s why we’re seeing consolidation and huge investment rounds. Long-term, I think costs will come down as models get more efficient and competition drives prices down. But in the near term, it’s a race to acquire users and prove out use cases.

3:36:29 Do you expect any major acquisitions in 2026? Will we see big tech companies buying AI startups?

3:37:08 Nathan: I think it’s likely. Microsoft has already effectively acquired OpenAI through their massive investment and partnership. Google has DeepMind. Amazon has invested heavily in Anthropic. Apple is the big question mark - they’ve been quieter on AI, but they’re clearly working on something.

3:38:02 I could see acquisitions of smaller, specialized AI companies - maybe in areas like AI agents, AI for specific domains like law or medicine, or AI infrastructure. The challenge is antitrust scrutiny. Regulators are watching these big tech acquisitions closely.

3:41:01 What’s next for the major AI labs? What should we expect from OpenAI, Anthropic, Google DeepMind, xAI, and Meta in the coming year?

3:41:48 Sebastian: OpenAI is likely working on GPT-5 or whatever comes after o1. I expect continued focus on making models more capable while also more aligned. Anthropic will keep pushing on safety and interpretability - that’s core to their mission. They’ve been doing really interesting technical work that others are adopting.

3:42:48 Google has the resources to do big things. I expect we’ll see more integration of AI across their product suite. xAI is interesting - Elon has the resources and ambition to move fast. Meta will continue with open source, which benefits everyone. They have the advantage of not needing to monetize the models directly.

3:53:35 There’s been discussion about a “Manhattan Project for AI” - a massive government-led initiative. What do you think about that?

3:54:18 Nathan: It’s an interesting idea. The Manhattan Project comparison comes up because of the strategic importance of AI and the scale of resources needed. The US government is definitely paying more attention to AI. We’ve seen executive orders, funding for AI research, discussions about compute infrastructure.

3:55:22 The question is whether a centralized, government-led approach is the right model. The Manhattan Project worked in part because it had a clear, focused goal: build an atomic bomb before the Nazis. With AI, the goals are more diffuse. Maybe the better model is something like DARPA - funding high-risk research while letting the private sector handle development and deployment.

4:00:10 What about NVIDIA and the future of GPUs? They’ve been the big winner in the AI boom. Will that continue?

4:00:52 Sebastian: NVIDIA has an incredible position right now. Their GPUs are the standard for AI training and inference. Their software stack with CUDA is deeply entrenched. That’s a huge moat. But competition is heating up. AMD is improving. Google has TPUs. There are startups working on AI-specific chips.

4:01:52 Long-term, I think we’ll see more diversity in compute hardware. Different chips optimized for different workloads - training versus inference, different model architectures, different precision requirements. NVIDIA will likely remain dominant, but not to the same degree. The bigger question is whether we’ll have enough compute for everyone who wants to train large models.

4:08:15 Let’s end on a big question. How will AI shape the future of humanity? What does the world look like in 10, 20, 50 years?

4:09:02 Nathan: I think AI will be transformative in ways we’re only beginning to understand. In 10 years, I expect AI assistants will be ubiquitous - helping with work, education, healthcare, creativity. Many jobs will change fundamentally, though new jobs will emerge. Productivity could increase dramatically if we do this right.

4:10:08 In 20 to 50 years, if we achieve AGI or beyond, we’re talking about a fundamentally different civilization. AI could help us solve major challenges - climate change, disease, resource scarcity. Or it could create new challenges we haven’t imagined. The key is making sure AI remains aligned with human values and beneficial to humanity.

4:11:22 Sebastian: I’m optimistic, but cautiously so. AI is a tool, and like any powerful tool, it can be used for good or ill. I think education will be crucial - helping people understand AI, adapt to the changes it brings, and use it effectively. We need to democratize access to AI technology so it benefits everyone, not just those with resources.

4:12:35 I’m excited about AI augmenting human creativity and problem-solving. Imagine a world where everyone has access to expert-level assistance in any domain. Where students can learn at their own pace with personalized AI tutors. Where scientists can explore hypotheses faster. Where artists have powerful new tools for expression. That’s the future I’m working toward.

4:13:48 Lex: Thank you both for this conversation. It’s been incredibly informative and thought-provoking. For everyone watching, please check out Nathan’s and Sebastian’s work - their books, blogs, podcasts, and social media. They’re doing crucial work in AI education and research. Thank you for listening, and I hope to see you next time.