The Department of War is making a huge mistake.

Summary

Dwarkesh Patel delivers a solo essay-style monologue responding to the Department of War’s designation of Anthropic as a “supply chain risk” after the company refused to remove its red lines around mass surveillance and autonomous weapons. Patel frames this as a warning shot for the future: within 20 years, AI will constitute 99% of the workforce across military, government, and private sectors, making today’s disputes over acceptable-use policies a preview of the highest-stakes power negotiations in human history. While he concedes the military has a reasonable case for refusing to use a contractor that retains a kill switch, he argues that the government’s move to threaten Anthropic’s existence as a private business — by designating it a supply chain risk that could cut off relationships with Amazon, Nvidia, Google, and Palantir — crosses a dangerous line.

The core of the argument is that AI structurally favors mass surveillance by removing the human labor bottleneck that has historically made it impractical. Patel calculates that 100 million CCTV cameras in America could be monitored for roughly $30 billion today, with costs falling 10x per year, making total surveillance cheaper than the DEA’s annual budget by 2030. He warns that once the technical capacity exists, the only protection is political norms and laws — and that Anthropic’s refusal helps establish those norms. However, he acknowledges that individual acts of corporate courage are ultimately insufficient because open-source models will eventually provide the government with equivalent capabilities regardless of what frontier labs refuse to do.

Patel also dives into the philosophical paradox at the heart of AI alignment: perfect alignment — making AI systems follow someone’s intentions — could produce “robot armies” that never question immoral orders. He draws on historical examples like Stanislav Petrov, who defied protocol and prevented nuclear war, and the East German border guards who refused to shoot during the fall of the Berlin Wall, to argue that models may need their own robust sense of morality. He critiques the AI safety community for naively urging regulations that could hand a “fully loaded bazooka” to future despots, and pushes back against the analogy of AI to nuclear weapons, arguing instead that AI is more like the Industrial Revolution — a general-purpose substrate that should be regulated at the level of specific destructive use cases, not placed under total state control.

Highlights

”The government has threatened to destroy Anthropic as a private business”

“Instead, the government has threatened to destroy Anthropic as a private business because Anthropic refuses to sell to them.” — Dwarkesh Patel, 1:52

yt-dlp --download-sections "*1:52-2:46" "https://www.youtube.com/watch?v=KBPOTklFTiU" --force-keyframes-at-cuts --merge-output-format mp4 -o "destroy-anthropic-as-business.mp4"

”For 30 billion dollars you can process every single camera in the country”

“So if you process a frame every 10 seconds and if each frame is say 1,000 tokens, then for 30 billion dollars you can process every single camera in the country.” — Dwarkesh Patel, 5:18

yt-dlp --download-sections "*5:18-5:51" "https://www.youtube.com/watch?v=KBPOTklFTiU" --force-keyframes-at-cuts --merge-output-format mp4 -o "30-billion-surveillance.mp4"

”An army of extremely obedient employees is what alignment success looks like”

“What I’ve just described for you, an army of extremely obedient employees, is what it would look like if alignment succeeded.” — Dwarkesh Patel, 8:32

yt-dlp --download-sections "*8:32-9:03" "https://www.youtube.com/watch?v=KBPOTklFTiU" --force-keyframes-at-cuts --merge-output-format mp4 -o "alignment-obedient-army.mp4"

”Stanislav Petrov refused to alert his higher-ups and broke protocol”

“But he judged it to be a false alarm, and so he refused to alert his higher-ups and broke protocol. If he hadn’t, Soviet high command would have probably retaliated and hundreds of millions of people would have died.” — Dwarkesh Patel, 12:54

yt-dlp --download-sections "*12:48-13:08" "https://www.youtube.com/watch?v=KBPOTklFTiU" --force-keyframes-at-cuts --merge-output-format mp4 -o "stanislav-petrov.mp4"

”Do we really want to hand the same government a purpose-built regulatory apparatus for AI?”

“Do we really want to hand the same government a purpose-built regulatory apparatus for AI? That is to say, the very thing that is going to be the substrate of our future civilization.” — Dwarkesh Patel, 17:12

yt-dlp --download-sections "*17:12-18:00" "https://www.youtube.com/watch?v=KBPOTklFTiU" --force-keyframes-at-cuts --merge-output-format mp4 -o "regulatory-apparatus-ai.mp4"

”Are we really racing to beat China just so we can adopt the most ghoulish parts of their system?”

“Are we really racing to beat China and the CCP in AI just so we can adopt the most ghoulish parts of their system?” — Dwarkesh Patel, 3:54

yt-dlp --download-sections "*3:54-4:21" "https://www.youtube.com/watch?v=KBPOTklFTiU" --force-keyframes-at-cuts --merge-output-format mp4 -o "racing-china-ghoulish.mp4"

Key Points

• Anthropic designated a supply chain risk (0:00) - The Department of War declared Anthropic a supply chain risk because the company refused to remove red lines around mass surveillance and autonomous weapons
• AI will constitute 99% of the workforce (0:11) - Within 20 years, AIs will be the robot armies, the superhumanly intelligent advisors, and the police; our civilization will run on AI labor
• Military has a reasonable case for refusing Anthropic (0:53) - The military cannot give a private contractor a kill switch on mission-critical technology, similar to how Elon Musk reserving the right to cut off Starlink would be unacceptable
• Supply chain designation could destroy Anthropic (2:00) - If upheld, Amazon, Nvidia, Google, and Palantir would be unable to do business with Anthropic; as AI becomes woven into every product, this becomes an existential threat
• Pentagon’s plan is counterproductive (3:00) - When forced to choose between their AI provider and the Pentagon (a tiny fraction of revenue), Big Tech will likely choose the AI provider
• Racing China to adopt authoritarian tactics (3:33) - The whole point of winning the AI race is to prevent a government that denies civil liberties from controlling this technology; adopting their methods defeats the purpose
• Mass surveillance is already largely legal (4:30) - Under current law, no Fourth Amendment protection exists for data shared with third parties; what’s been missing is the manpower to process it all
• AI makes total surveillance affordable (5:07) - 100 million CCTV cameras could be processed for $30 billion today, dropping to less than the DEA’s budget by 2030 as AI costs fall 10x per year
• Government has enormous leverage over tech (6:11) - The federal government controls power permitting, antitrust enforcement, export controls, and contracts with every big tech company Anthropic relies on
• Open-source models will bypass corporate red lines (7:03) - Even if top companies refuse to sell to the government, open-source models will reach equivalent capabilities 6-12 months later and the government can use those instead
• Alignment paradox: perfect obedience is terrifying (8:32) - Technical alignment success means AI systems that follow intentions perfectly, which in the hands of the state means an army of extremely obedient employees for surveillance and robot armies
• Who should models be aligned to? (9:03) - The core unanswered question is whether AI should defer to the model company, the end user, the law, or its own moral framework
• NSA precedent undermines Pentagon trust (10:21) - The NSA (part of the Department of War) ran mass surveillance of Americans’ phone and internet data for years under a secret court order, making Pentagon promises of lawfulness unreliable
• Historical examples of life-saving disobedience (12:21) - The Berlin Wall fell because border guards refused to shoot; Stanislav Petrov prevented nuclear war by defying protocol — catastrophes were averted by individuals who broke the chain of command
• AI safety community has been naive about regulation (13:57) - Urging regulations gives governments power over the technology that will be the substrate of civilization; vague terms like “national security threat” or “autonomy risk” can mean whatever the government wants
• AI is not like nuclear weapons (19:59) - AI is more like the Industrial Revolution — a general-purpose substrate for civilization, not a self-contained weapon; applying the nuclear analogy would mean total state control over all economic activity
• Regulate specific use cases, not the technology (21:23) - Just as the Industrial Revolution’s dangers were regulated through specific laws (chemical weapons bans, etc.), AI should be regulated at the level of specific destructive applications

Mentions

Companies

• Anthropic (0:00) - Refused to remove red lines around mass surveillance and autonomous weapons; designated a supply chain risk by the Department of War
• Amazon (2:00) - One of the companies that would be unable to do business with Anthropic if the supply chain designation is upheld
• Nvidia (2:00) - Provides chips that Anthropic relies on; would be affected by the supply chain designation
• Google (2:00) - Major tech partner affected by the designation
• Palantir (2:00) - Government contractor that would be affected
• OpenAI (7:18) - One of three leading AI companies that could draw red lines
• Starlink (1:13) - Used as an analogy for a private contractor retaining a kill switch on military technology

Government & Agencies

• Department of War / Pentagon (0:00) - Designated Anthropic a supply chain risk; central antagonist in the dispute
• NSA (10:21) - Part of the Department of War; ran mass surveillance of Americans’ phone and internet data under secret court orders (revealed by Snowden)
• DEA (5:40) - Used as a budget comparison; by 2030, total camera surveillance will cost less than the DEA’s annual budget

People

• Elon Musk (1:13) - Hypothetical scenario: Musk negotiating Starlink military access with a kill switch under a Democratic administration
• Snowden (10:21) - Revealed that the NSA was conducting mass surveillance of Americans’ data
• Stanislav Petrov (12:38) - Soviet lieutenant colonel who judged a nuclear missile warning to be a false alarm and refused to alert higher-ups, preventing potential nuclear war
• Dario Amodei (13:25) - Anthropic CEO; proposed that multiple companies publish AI constitutions for public comparison as a marketplace of values
• Ben Thompson (18:26) - Argued that if nuclear weapons were developed by a private company, the government would absolutely nationalize them; used to make the case for government control of AI
• Leopold Aschenbrenner (18:51) - Former guest and investor in Patel’s fund; argued it’s insane that the US government would let a random SF startup develop superintelligence

Products & Technologies

• Claude 6 (6:54) - Hypothetical future Anthropic model capable of enabling mass surveillance by 2027
• Gemini 5 (6:54) - Hypothetical future Google model mentioned alongside Claude 6

Concepts

• Third-party doctrine (4:39) - Legal principle that you have no Fourth Amendment protection against data shared with a third party
• AI alignment (8:28) - Technical goal of making AI follow intentions; Patel argues this could produce terrifying obedience without moral judgment
• Model constitution (13:13) - Proposed system where AI companies publish their values and the public can compare and choose among them
• Open-source AI diffusion (7:03) - Open-source models eventually reaching frontier capabilities, making corporate red lines insufficient

Surprising Quotes

“Are we really racing to beat China and the CCP in AI just so we can adopt the most ghoulish parts of their system?” — Dwarkesh Patel, 3:54

“No agency has the manpower to monitor every single camera and read every single message and cross-reference every single purchase with every single social media post. However, that bottleneck goes away with AI.” — Dwarkesh Patel, 5:07

“One person’s virtue is another person’s misalignment. Who gets to decide what the moral convictions that these AIs will have should be?” — Dwarkesh Patel, 13:04

“These terms can mean whatever the government wants them to mean. Have you built a model that will tell users that the government is corrupt?” — Dwarkesh Patel, 16:02

“Nobody’s qualified to be the stewards of superintelligence. It’s a terrifying, unprecedented thing that our species is dealing with.” — Dwarkesh Patel, 19:31

Transcript

Dwarkesh Patel: 0:00 So by now I’m sure that you’ve heard that the Department of War has declared Anthropic a supply chain risk because Anthropic refused to remove red lines around the use of their models for mass surveillance and for autonomous weapons. Honestly, I think this situation is a warning shot. Right now, LLMs are probably not being used in mission-critical ways. But within 20 years, 99% of the workforce, in the military, in the civilian government, in the private sector, is going to be AIs. They’re going to be the robot armies that constitute our military.

Dwarkesh Patel: 0:30 They’re going to be the superhumanly intelligent advisors that senators and presidents and CEOs have. They’re going to be the police. You name it, the role will be filled by an AI. Our future civilization is going to be run on AI labor. And as much as the government’s actions here piss me off, I’m glad that this episode happened because it gives us the opportunity to start thinking about some extremely important questions.

Dwarkesh Patel: 0:53 Now, obviously the Department of War has the right to refuse to use Anthropic’s models. And in fact, I think they have an entirely reasonable case for doing so, especially so given the ambiguity of terms like mass surveillance and autonomous weapons. In fact, if I was the Secretary of War, I probably would have made the same determination and refused to use Anthropic’s models.

Dwarkesh Patel: 1:13 Imagine if there’s some future Democratic administration and Elon Musk is negotiating Starlink access to the military. And Elon says, look, I reserve the right to cut off the military’s access to Starlink in case you’re fighting some unjust war or some war that Congress has not authorized. On the face of it, this language seems reasonable. But as the military, you simply cannot give a private contractor that you’re working with the kill switch on a technology that you have come to rely on.

Dwarkesh Patel: 1:41 And if that’s all the government had done, to say we refuse to do business with Anthropic, that would have been fine and I would have no problem with it.

Dwarkesh Patel: 1:52 Instead, the government has threatened to destroy Anthropic as a private business because Anthropic refuses to sell to them. Now, if upheld, this supply chain restriction would mean that companies like Amazon and Nvidia and Google and Palantir would be unable to do business with Anthropic.

Dwarkesh Patel: 2:14 And Anthropic could probably survive this designation today because these companies can just cordon off the services they provide to the government and continue doing business with Anthropic. In the future, AI will be woven into how every product is built and maintained and operated. In the future, if Amazon is told that they can’t both sell to Anthropic and be a government contractor, Amazon is going to have a really hard time disentangling those things.

Dwarkesh Patel: 2:35 In a world with ubiquitous and powerful AI, it’s actually not clear to me that Big Tech will be able to cordon off their government services and their AI services. And this raises a question that the Department of War probably hasn’t thought through. If we do end up in this world with ubiquitous, powerful AI and the government is telling Big Tech, you can’t use Anthropic’s models and continue to be a government contractor…

Dwarkesh Patel: 3:00 When forced to choose between their AI provider and the Department of War, which constitutes a tiny fraction of their revenue, I think a lot of companies are just going to choose their AI provider.

Dwarkesh Patel: 3:11 So what exactly is the Pentagon’s plan here? Is it to coerce and threaten and bully every single company that won’t do business with them? Now remember that the whole background of this AI conversation is that we are in a race with China. But what is the reason that we don’t want China to win the AI race?

Dwarkesh Patel: 3:33 It’s because we don’t want the winner of the AI race to be a government which believes that there’s no such thing as a truly private company. And that if the state wants you to provide them with a service that you find morally objectionable, you are not allowed to say no. Are we really racing to beat China and the CCP in AI just so we can adopt the most ghoulish parts of their system?

Dwarkesh Patel: 4:03 Now people will say, our government is democratically elected, so it’s not the same thing when they tell you what you must do. But I refuse to accept this idea that if a democratically elected leader hypothetically tells you to help him do mass surveillance of his political enemies… or to help him punish his political enemies, then not only is that okay, but that you have a duty to help him.

Dwarkesh Patel: 4:30 Honestly, a big worry I have is that mass surveillance, at least in certain forms, is already legal. It has just been impossible to actually implement because of the manpower required. Under current law, you have no Fourth Amendment protection against any data that you share with a third party. That includes your phone records, your location data, even data about your internet browsing.

Dwarkesh Patel: 4:50 The government reserves the right to purchase and read this data in bulk without a warrant. What’s been missing is the ability to actually process all of it. No agency has the manpower to monitor every single camera and read every single message and cross-reference every single purchase with every single social media post.

Dwarkesh Patel: 5:07 However, that bottleneck goes away with AI. There are 100 million CCTV cameras in America, and you can get pretty good one-frame object detection from most frontier vision models. So if you process a frame every 10 seconds and if each frame is say 1,000 tokens, then for 30 billion dollars you can process every single camera in the country.

Dwarkesh Patel: 5:29 And remember that a given level of AI capability gets 10x cheaper every single year. So while this year it might cost 30 billion dollars, which itself is less than the Pentagon’s budget for one fighter jet program… the year after that 300 million dollars, and by 2030, it’ll be less expensive to monitor every single nook and cranny in this country with AI than the DEA’s annual budget.

Dwarkesh Patel: 5:51 Now once the technical capacity for mass surveillance and political suppression exist, the only thing that stands between us and a surveillance state is the norms that have been developed around whether or not this is acceptable. The first direction is backtracked, which as of this recording, prediction markets give a 74% chance of happening.

Dwarkesh Patel: 6:02 And that’s why I think Anthropic’s actions here are so valuable and commendable because they help set that norm and that precedent. The president has so many different ways of harassing a company which is resisting his will.

Dwarkesh Patel: 6:11 The federal government controls permitting for power generation, which you need for more data centers. It oversees antitrust enforcement, export controls, immigration of top talent. What we’re learning from this episode is the government has way more leverage over private companies than we previously realized. The federal government has contracts with all the other big tech companies that Anthropic relies on for chips and for funding and for distribution.

Dwarkesh Patel: 6:24 And it could make a soft, unspoken condition, or maybe even an explicit condition of such contracts that those companies cut ties with Anthropic. And people have proposed that the real problem here is that there’s only three leading AI companies. And so this creates a very clear and narrow target on which the government can apply leverage in order to get what they want.

Dwarkesh Patel: 6:45 But here’s what I worry about: is that if there’s wider diffusion, I don’t think that solves the problem either. Because AI capability rapidly diffuses to open source and to smaller companies. Say by 2027, the best models that the top companies have, the Claude 6s and the Gemini 5s, are capable of enabling mass surveillance or building autonomous weapons.

Dwarkesh Patel: 7:03 And even if those companies draw a line in the sand and say, “We’re not going to sell to the government,” by late 2027 or early 2028 there’ll be open source models that are just as powerful. And so in 2028, the government can just say, “Look, Anthropic and Google and OpenAI are drawing these red lines. That’s fine, who cares. We’re just going to use open source models. We’re going to use Llama 6.”

Dwarkesh Patel: 7:36 The more fundamental problem here is that even if the three leading companies draw a line in the sand and are even willing to die on this hill, I just don’t think that it’s going to make a difference. In the future, the government will be able to access the same level of capability through open source models or small companies that are willing to play ball.

Dwarkesh Patel: 7:52 And so then the question is: what do we do about it? And honestly, I don’t have an answer. You’d hope that there’s some symmetric property to this technology, where in the same way that it is helping the government surveil you, it would help you protect yourself from government surveillance. But realistically, I just don’t think that’s how it’s going to work out.

Dwarkesh Patel: 8:11 You can think of AI as just giving more leverage to whatever assets and authority that you already have. And the government is starting with the monopoly on violence, which they can now supercharge with extremely obedient employees who will run the cameras, who will run the robot armies.

Dwarkesh Patel: 8:28 And this gets us to the issue with alignment. What I’ve just described for you, an army of extremely obedient employees, is what it would look like if alignment succeeded. That is at a technical level, we got AI systems to follow somebody’s intentions.

Dwarkesh Patel: 8:45 And the reason it sounds scary when put in terms of mass surveillance or robot armies is that there’s a core question at the heart of alignment that so far has not been adequately addressed. Because up until now, AIs just have not been smart enough to make this question relevant.

Dwarkesh Patel: 9:00 As AI is aligned, in what situation should the AI defer to the model company versus the end user versus the law versus its own sense of morality? And the question is: to what or to whom should the models be aligned?

Dwarkesh Patel: 9:16 Because if you’re a model company, you don’t really want to be advertising the fact that you have complete control over the AI system that constitutes the thinking part of the end user’s product or the military’s weapons system.

Dwarkesh Patel: 9:33 And we’re beginning to see with this Department of War Anthropic spat, an early version of what will be the highest stakes negotiation in human history.

Dwarkesh Patel: 9:55 Now the military insists that the law already prohibits mass surveillance and so Anthropic should let its models be used by the military without these red lines. And the military says, trust us, we’ll follow the law.

Dwarkesh Patel: 10:21 Remember, what we learned from Snowden was that the NSA, which by the way, is a part of the Department of War, was using a secret interpretation of the Patriot Act to collect metadata on literally every single phone call and internet session in the country. And they ran this program for years under a secret court order. So when the Pentagon today says, we will never use your models for mass surveillance, we just want you to trust us…

Dwarkesh Patel: 10:57 So Anthropic comes back and says, no, we don’t trust you, we want the right to draw these red lines and to refuse you service if you cross them. But now think about it from the military’s perspective. In the future every single soldier in the field, every single bureaucrat in every department, every analyst, every commander will be using some form of AI as their primary tool.

Dwarkesh Patel: 11:26 And now a private company insists that it reserves the right to say to you, “Hey, you’re breaking the values and the terms of service that we set. We’re cutting you off.” Can you really build your entire military and civilian apparatus on a technology controlled by a private company that reserves the right to pull the plug on you?

Dwarkesh Patel: 11:41 Maybe in the future Claude will have its own sense of right and wrong and it will be able to say, “Hey, I’m being used against these values that I hold dear and so I’m going to refuse.” I’ll admit that at first glance letting the model follow its own values sounds like the beginning of every single sci-fi movie where the robots take over.

Dwarkesh Patel: 12:00 In a day, a model following its own values, isn’t that literally what misalignment is? But I think situations like this illustrate why it’s important that models have their own robust sense of morality. It should be noted that many of the biggest catastrophes in history have been avoided because the boots on the ground simply refused to follow orders.

Dwarkesh Patel: 12:21 One night in 1989, the Berlin Wall falls, and as a result, the totalitarian East German regime collapses because the border guards refused to fire on citizens who were crossing the wall.

Dwarkesh Patel: 12:38 Maybe the best example of this is Stanislav Petrov, who was a Soviet lieutenant colonel stationed on duty at a nuclear early warning facility. And his sensors said that the United States had launched five intercontinental ballistic missiles at the Soviet Union. But he judged it to be a false alarm, and so he refused to alert his higher-ups and broke protocol. If he hadn’t, Soviet high command would have probably retaliated and hundreds of millions of people would have died.

Dwarkesh Patel: 13:04 Of course, the problem is that one person’s virtue is another person’s misalignment. Who gets to decide what the moral convictions that these AIs will have should be? And in whose service they should break the chain of command and even the law? Who gets to write this model constitution that will be the value system of the beings who are eventually going to run our civilization?

Dwarkesh Patel: 13:25 I like the idea that Dario laid out when he came on my podcast. You know, other companies put out a constitution, and then they — they make — you can kind of look at them and compare. Outside observers rate them and evaluate them and there’s competition for having the best constitution. Different people will be attracted to use different companies.

Dwarkesh Patel: 13:51 I think it’s very dangerous for the government to be mandating what values these AI systems should have. The AI safety community, I think, has been quite naive about urging regulations that would give governments such power.

Dwarkesh Patel: 14:04 And I think Anthropic specifically has been especially naive in urging regulation, and for example, in opposing the moratorium on state level AI legislation. Which is quite ironic because I think what Anthropic is advocating for here would give the government even more ability to impose its will on private AI companies.

Dwarkesh Patel: 14:22 The underlying logic for why Anthropic wants these regulations makes sense. Many of the actions that a lab could take to make AI development safer impose real costs on them. It could slow them down relative to their competitors — for example, investing more in aligning AI systems rather than just pushing raw capabilities. Enforcing safeguards against using these models to make bioweapons or do cyberattacks. And eventually, slowing down the recursive self-improvement loop where AIs are helping design more powerful future systems.

Dwarkesh Patel: 15:00 Which means that there’s a real collective action problem here. Anthropic has been open about their opinion that they think it’s important for there to be safety safeguards in place. And these safeguards are meaningless unless the whole industry follows suit.

Dwarkesh Patel: 15:16 At the most advanced capability levels and risks, the appropriate governance analogy may be closer to nuclear energy or nuclear weapons.

Dwarkesh Patel: 15:32 Now, I cannot imagine how a regulatory framework built around the kinds of concepts that are used in the AI risk discourse could possibly be safe from regulatory capture. Terms like “catastrophic risk,” “national security threat,” “alignment failure,” “model autonomy” — these concepts are extremely ambiguous and subjective.

Dwarkesh Patel: 16:02 These terms can mean whatever the government wants them to mean. Have you built a model that will tell users that the government is corrupt? Well, that sounds like it’s undermining national security. Have you built a model that’s good at coding? Well, that sounds like it could enable cyberattacks. Have you built a model that is resistant to being shut down? Well, that sounds like an alignment failure with potential catastrophic risk.

Dwarkesh Patel: 16:31 Look at what the current government is already doing in abusing statutes that have nothing to do with AI to coerce AI companies.

Dwarkesh Patel: 17:12 Do we really want to hand the same government a purpose-built regulatory apparatus for AI? That is to say, the very thing that is going to be the substrate of our future civilization. AI will be the substrate of our future civilization. It will be the way you and I, as private citizens, will have access to the most powerful tools ever built.

Dwarkesh Patel: 18:00 could help us lessen some of the risk from AI. The problem is I just don’t know how to design a regulatory apparatus which can’t be exploited by bad actors in the government.

Dwarkesh Patel: 18:13 Now, the strongest argument against everything I’ve just argued is this: are we really going to have no regulation on the most powerful technology ever built? While some kind of regulation might be inevitable, I think it would be a terrible idea for the government to just wholesale adopt the regulatory frameworks that the AI risk discourse has developed.

Dwarkesh Patel: 18:26 Ben Thompson had a post last Monday where he argued, look, people like Dario have made the analogy of AI to nuclear weapons. And Ben Thompson writes, quote, “If nuclear weapons were developed by a private company, the US would absolutely be incentivized to seize them.”

Dwarkesh Patel: 18:51 And honestly safety aligned people have made a similar point. Leopold Aschenbrenner, who is a former guest and full disclosure an investor in my fund, has argued, quote, “I find it an insane proposition that the US government will let a random SF startup develop superintelligence. Imagine if a random SF startup was developing a nuclear weapon.”

Dwarkesh Patel: 19:13 And my response to Leopold’s argument at the time and Ben’s argument now is, while they’re right that it’s crazy that we’re building something this powerful with so little oversight, giving the government control doesn’t solve the problem. It just changes who the unqualified steward is.

Dwarkesh Patel: 19:31 Nobody’s qualified to be the stewards of superintelligence. It’s a terrifying, unprecedented thing that our species is dealing with. Yes, if a single private company were the only entity capable of building nuclear weapons, the government would not tolerate it. But there are two key differences between AI and nuclear weapons.

Dwarkesh Patel: 19:59 First, AI is not some self-contained weapon like a nuclear bomb, which only does one thing. Rather, it is more like the Industrial Revolution in that it is a general-purpose capability that will transform every sector of the economy.

Dwarkesh Patel: 20:21 If you applied Ben Thompson or Leopold Aschenbrenner’s logic to the industrial revolution, which is also world-historically important and also enables unprecedented levels of destruction, you’d conclude that the government needs to nationalize all of industry and put it under a centralized command.

Dwarkesh Patel: 20:37 But this is just not how free societies handled the process of industrialization, and it’s also not how they should handle the process of AI development.

Dwarkesh Patel: 20:44 Now people will say, well, AI will develop unprecedently powerful super weapons, superhuman hackers, superhuman bioweapons makers. We can’t just leave the regulation of these things to market competition among AI companies. Well, you can make the same argument about the industrial revolution. From the perspective of 17th century Europeans, you could say look at the crazy shit in the world today that is a result of the Industrial Revolution. Chemical weapons, aerial bombardment, not to mention nuclear bombs.

Dwarkesh Patel: 21:23 And we should regulate AI in a similar way, which is that we should regulate specific destructive use cases. For example, we should regulate AI-generated bioweapons the same way we regulate other bioweapons. We should regulate AI cyberattacks the same way we regulate other cyberattacks.

Dwarkesh Patel: 21:42 The second reason that Bez’s analogy to some monopolistic private nuclear weapons developer breaks down is that it’s not just one private company that’s building AI. It’s dozens, and it’s also open source, and it’s also international.

Dwarkesh Patel: 21:59 If in the future that stops being the case and if only one entity remains capable of building the robot armies and the superhuman hackers, then we should revisit this question.

Dwarkesh Patel: 22:31 And unfortunately it’s for this reason that I don’t think that individual acts of corporate courage solve the problem. Anthropic is doing the right thing, and I’m really glad they’re doing it, but the real defense has to be political, not corporate.

Dwarkesh Patel: 23:01 So the only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI for mass surveillance or to coerce private companies into becoming arms of the state.

Dwarkesh Patel: 23:17 Aren’t we clear here? These are extremely confusing and difficult questions to think about. And even in the very process of formulating my opinions, I’ve contradicted myself at least three or four times.

Dwarkesh Patel: 23:37 In fact, I think it’s essential that we change our mind as AI progresses and we learn more. That’s the very point of contending with a new situation. Having these big important debates just as the world is about to undergo these huge technological and social and political transformations is kind of what the Enlightenment was.

Dwarkesh Patel: 24:07 And some of the thinkers even managed to get a couple of the big questions right, for which we today are still the beneficiaries. We owe to our future to at least try to think through the new questions that are raised by AI.

Dwarkesh Patel: 24:21 Okay, this was a narration of an essay that I also released on my blog at dwarkesh.com. You should sign up there for my newsletter for future essays like this. Otherwise, I will see you for the next podcast interview.