Anthropic Said No to Autonomous Weapons. The U.S. Is Fighting Back.

Summary

Kevin Roose and Casey Newton explore the escalating conflict between the Pentagon and Anthropic over the terms of a $200 million AI contract. The Pentagon asked all four of its AI contractors — Anthropic, OpenAI, Google, and xAI — to sign an “all lawful uses” contract that would strip out their standard usage policies. While OpenAI, Google, and xAI all signed, Anthropic refused, requesting only two carveouts: no mass domestic surveillance and no autonomous kinetic operations (weapons that kill without a human in the loop). The Pentagon responded by threatening not only to cancel the contract but to designate Anthropic a “supply chain risk,” a classification typically reserved for foreign adversaries like Huawei and Kaspersky Lab.

The hosts dig into the broader political context, including Anthropic’s fraught relationship with the Trump administration, CEO Dario Amodei’s long-held convictions about AI safety, and the company’s recent $20 million donation to a bipartisan super PAC supporting AI regulation. Kevin reports from his own sources that Anthropic is willing to take the revenue hit rather than compromise its principles. Casey makes the striking observation that the more alarming story may not be Anthropic’s resistance, but rather that Google, OpenAI, and xAI all signed up for what could enable mass surveillance and autonomous killing weapons without objection.

The episode closes with both hosts expressing deep concern that the only thing standing between the U.S. military and unfettered use of powerful AI for surveillance and autonomous weapons is one company’s usage policy, rather than laws passed by Congress.

Highlights

”Two Carveouts That Don’t Sound Like Huge Asks”

yt-dlp --download-sections "*3:16-4:00" "https://www.youtube.com/watch?v=QhQp_z16I_A" --force-keyframes-at-cuts --merge-output-format mp4 -o "QhQp_z16I_A-3m16s.mp4"

“Three of the companies signed it. OpenAI and xAI and Google all signed this contract. Anthropic did not. And they asked for two changes, basically two carveouts to this policy. They said, ‘We don’t want Claude to be used for mass domestic surveillance and we don’t want Claude to be used for autonomous kinetic operations.’ …And I have to say, those don’t sound like huge asks.” — Kevin Roose & Casey Newton, 3:16

”This Is Typically Reserved for Foreign Adversaries”

yt-dlp --download-sections "*4:20-5:10" "https://www.youtube.com/watch?v=QhQp_z16I_A" --force-keyframes-at-cuts --merge-output-format mp4 -o "QhQp_z16I_A-4m20s.mp4"

“Now that is a very strong move. It is often applied to foreign adversaries. So Huawei, the Chinese tech company, was designated a supply chain risk. Kaspersky Lab, the Russian antivirus malware company, has also been designated a supply chain risk. This is something that is typically reserved for companies that run in adversarial countries.” — Kevin Roose, 4:20

”Claude Was Involved in Capturing the President of Venezuela”

yt-dlp --download-sections "*1:55-2:30" "https://www.youtube.com/watch?v=QhQp_z16I_A" --force-keyframes-at-cuts --merge-output-format mp4 -o "QhQp_z16I_A-1m55s.mp4"

“There’s also a classified system that is run through Palantir and Amazon Bedrock… that lets the US military use Claude in specific classified situations for things like helping them capture the president of Venezuela. That was reportedly a use that Claude was involved in last month.” — Kevin Roose, 1:55

”Surveillance and Murder Bots Are Coming to AI, But Not to Claude”

yt-dlp --download-sections "*12:24-13:30" "https://www.youtube.com/watch?v=QhQp_z16I_A" --force-keyframes-at-cuts --merge-output-format mp4 -o "QhQp_z16I_A-12m24s.mp4"

“They’re saying surveillance and murder bots are coming to AI but not to Claude. If I’m on their marketing team, those might be fights I actually wanted to pick because they’re putting my competitors in a pretty bad light, aren’t they?” — Casey Newton, 12:24

”No One Else Is Waging This Battle”

yt-dlp --download-sections "*15:10-16:10" "https://www.youtube.com/watch?v=QhQp_z16I_A" --force-keyframes-at-cuts --merge-output-format mp4 -o "QhQp_z16I_A-15m10s.mp4"

“I’m less struck by the fact that Anthropic is waging this battle and more struck by the fact that no one else is. The fact that it seems like Google, OpenAI, and xAI are all prepared to sign up for what could be mass surveillance and autonomous killing weapons — I actually find quite chilling.” — Casey Newton, 15:10

Key Points

• Pentagon’s “All Lawful Uses” Contract (2:31) - The Pentagon asked all four AI contractors to sign a contract stripping out their usage policies and replacing them with a blanket “all lawful uses” policy
• Three Companies Signed, Anthropic Refused (3:16) - OpenAI, Google, and xAI signed the contract; Anthropic refused and asked for two specific carveouts
• Anthropic’s Two Carveouts (3:28) - Anthropic requested that Claude not be used for mass domestic surveillance or autonomous kinetic operations (weapons without human oversight)
• Supply Chain Risk Threat (4:20) - Pentagon threatening to designate Anthropic a “supply chain risk,” a classification typically reserved for foreign adversaries like Huawei and Kaspersky
• Claude Used in Venezuela Operation (1:55) - Claude was reportedly used in classified military operations including helping capture the president of Venezuela
• Cascading Business Impact (5:50) - Supply chain risk designation would force companies like Amazon and Google to untangle any government work from Anthropic’s models
• Dario Amodei’s Convictions (8:50) - The fight is largely driven by Amodei’s long-held personal convictions about the dangers of autonomous weapons and surveillance
• Trump Administration Hostility (9:40) - David Sacks called Anthropic a “doomer cult”; the administration views them as woke liberals who won’t cooperate
• Anthropic’s $20M Super PAC Donation (10:50) - Anthropic donated $20 million to a bipartisan super PAC supporting AI regulation, seen as a shot at OpenAI
• This Is a Loyalty Test (11:45) - Kevin argues this is really about the Trump administration demanding loyalty from tech companies, not about the specific contract terms
• Near-Term Surveillance Risks (13:30) - Tech companies receiving unprecedented subpoenas from federal agencies trying to identify people criticizing ICE
• Anthropic Willing to Take Revenue Hit (12:00) - Sources say Anthropic is very set on its position and willing to lose revenue to stand by its principles
• No One Else Is Standing Up (15:10) - Casey finds it chilling that only Anthropic is resisting, while Google, OpenAI, and xAI all signed
• AI Power Underestimated (16:00) - Kevin argues the Pentagon views AI like buying Microsoft Word and doesn’t understand these systems are capable of judgment and autonomous action
• One Company’s Usage Policy (17:40) - Casey expresses discomfort that the only barrier to military AI misuse is one company’s terms of service rather than federal law

Mentions

Companies

• Anthropic (0:08) - Refused to sign Pentagon’s “all lawful uses” contract, requesting two carveouts
• OpenAI (3:16) - Signed the Pentagon’s all lawful uses contract
• Google (3:19) - Signed the Pentagon’s contract; referenced Project Maven history
• xAI (3:17) - Signed the Pentagon’s all lawful uses contract
• Palantir (1:56) - Runs classified system for military AI access
• Amazon (Bedrock) (1:57) - Provides platform for classified military AI systems
• Huawei (4:26) - Previously designated a supply chain risk by the US government
• Kaspersky Lab (4:31) - Previously designated a supply chain risk
• Nvidia (10:20) - Mentioned in context of export controls debate

Products & Technologies

• Claude (1:55) - Anthropic’s AI model used in classified military operations
• Claude Code (14:10) - Referenced as a powerful tool that could be misused for building surveillance databases

People

• Dario Amodei (8:50) - Anthropic CEO with long-held convictions about autonomous weapons and surveillance risks
• David Sacks (9:40) - White House AI czar who called Anthropic a “doomer cult”
• Greg Brockman (11:00) - OpenAI president who funded a pro-Trump super PAC
• Amanda Askell (7:10) - Referenced from a previous episode about building Claude’s constitution

Government & Agencies

• The Pentagon / Department of Defense (0:08) - Threatening Anthropic over contract terms
• ICE (13:50) - Sending subpoenas to tech companies to identify critics
• ACLU / EFF (16:50) - Noted absence of civil liberties groups from the debate

Surprising Quotes

“They don’t believe AI could be dangerous, but they are interested in using it to build autonomous murder bots.” — Casey Newton, 7:55

“Keep in mind they were mostly trained on fanfiction databases which just don’t have all that much information about how to autonomously kill someone.” — Casey Newton, 13:05

“My fear is that they actually do understand that and they’re getting really excited about it and they want it right now. And I don’t see anything in the current Pentagon that would limit them from wanting to use those tools in exactly that way.” — Casey Newton, 16:35

“It makes me very uncomfortable that the thing standing between us and the US military having basically unfettered ability to conduct mass domestic surveillance and build autonomous killing weapons is like one company and its usage policy.” — Casey Newton, 17:40

Transcript

0:00 So, let’s start today by talking about the ongoing dispute between the Pentagon and Anthropic. And before we do that, let’s make our disclosures. My boyfriend works at Anthropic and I work at the New York Times, which is suing OpenAI, Microsoft, and Perplexity. So, Casey, have you been following this story? I have very closely because I am actually quite concerned about what it would mean for the US government to have access to the kinds of technologies they seem to want.

0:30 Yes. So, this story has been developing quickly over the past few days, but basically the latest is that the Pentagon is upset with Anthropic over the terms of a contract that they are negotiating, and they are threatening not only to drop a $200 million contract that they signed with Anthropic, but also to designate Anthropic and its models a supply chain risk, which would be a very serious, almost unprecedented escalation against a US company and it would have all kinds of implications for Anthropic’s ability to work with the defense department with contractors who work with the defense department and so this has become a huge political battle.

1:15 Got it. Well, okay. So, tell us a little bit about the contract that Anthropic and some of these other big AI labs have with the Pentagon. How is the Pentagon using AI right now? So, a couple ways. One is that the Pentagon has a platform where service members from all of the departments of the US military can use the various AI models that they have contracted with. So right now there are four sort of labs that are included in that. There’s Anthropic, there’s OpenAI, there’s Google and there’s xAI. And so they can use that for administrative tasks, office tasks, whatever.

1:55 There’s also a classified system that is run through Palantir and Amazon Bedrock, which are two of these sort of platform companies that provide access to AI models that lets the US military use Claude in specific classified situations for things like helping them capture the president of Venezuela. That was reportedly a use that Claude was involved in last month. First ever Claude-napping.

2:20 Yes. So, they have this contract and they want to be able to do sounds like almost whatever they want with it. What are the sticking points here? So, I’ve been making some calls on this and talking to some folks who are involved in these negotiations, and it’s a little unclear what exactly triggered this, but here’s what I know. Earlier this year, the Pentagon reached out to all of the companies that it had contracts with, the four AI companies, and asked them to sign what they called an all lawful uses contract, which would basically strip out the usage policies that these companies have for their models when they sell them to corporate customers or let users use them and replace it with something that just says the US military is allowed to do anything lawful with these systems.

3:16 And three of the companies signed it. OpenAI and xAI and Google all signed this contract. Anthropic did not. And they asked for two changes, basically two carveouts to this policy. They said, “We don’t want Claude to be used for mass domestic surveillance and we don’t want Claude to be used for autonomous kinetic operations.” Basically anything that would kill someone or send a weapon into a battlefield without a human in the loop supervising it. And they said, “If you just promise us that you won’t do those two things, we’ll be happy to sign this agreement.” Yeah. And I have to say, those don’t sound like huge asks.

4:00 But it sounds like the Pentagon saw it differently. Yes. They were very upset about this and they have started trying to get some leverage in these negotiations by saying we are not only going to cancel the contract but we’re also potentially going to designate Anthropic a supply chain risk. Now that is a very strong move. It is often applied to foreign adversaries. So Huawei, the Chinese tech company, was designated a supply chain risk. Kaspersky Lab, the Russian antivirus malware company, has also been designated a supply chain risk. This is something that is typically reserved for companies that run in adversarial countries that have some threat to Americans.

4:55 And so the military is allowed to kind of say we are not going to let any of our contractors even touch this technology. Yeah. And just to drill down a little bit on those two companies, the fear about Kaspersky Labs was that because it was founded in Russia, the Russian state government might try to interfere with it so that a company that was using it, maybe the Russian government would get backdoor access into an American company. With Huawei, which makes telecom equipment, the fear is well maybe the Chinese government will be able to insert a backdoor into telecom equipment so they could spy on Americans. So those are the sorts of threats which I’ll say those are like actually scary legitimate threats to me personally.

5:50 That is what we have previously designated a supply chain risk. What you’re saying is Anthropic said we don’t want to do mass surveillance and we don’t want to do autonomous killing and the Pentagon said that is a big risk to Americans. Yes. And they’ve said that the company is basically putting the military at risk by not allowing them to do these things. It’s a little hard to know what exactly would happen if the Pentagon did declare Anthropic a supply chain risk. I’ve talked to some folks who think it would basically prevent any US government contractor from using Claude or any other Anthropic products inside their own systems. It seems to be a little bit more complicated than that.

6:30 The latest thinking on this is that it would impact the use of Anthropic’s products on Pentagon systems and Pentagon related systems. So like Google Cloud for example wouldn’t be able to use Claude on any kind of systems or servers that touch Google’s government contracts. But the belief is that Anthropic could still work with Google just not on anything that kind of touches Google’s government contracts. And so this is a $200 million contract that Anthropic has right now. Were they to lose that, how big of a problem is that for Anthropic?

7:10 So in financial terms, it would not be a company killing event. It’s a big contract, but they make billions of dollars a year in revenue. This is not make or break for them. I think though that the supply chain risk designation would be a much more harmful thing for them because it would mean that if you are say Amazon and you have Anthropic as one of your providers, you then have to go through all of your servers and all of your data centers and all of your workflows and make sure that nothing that touches any of your government work also touches an Anthropic model. Your coders won’t be able to use Claude Code to build anything for the government. Basically, it would just require a lot of untangling.

7:50 And so that is why the Pentagon is using this as a threat to Anthropic because this would be extremely annoying and costly for them, right? But at the same time, it seems like Anthropic believes it has some leverage here, right? Like clearly the military wants to be using Claude and they wouldn’t be jumping through all of these hoops if it wasn’t going to be a pain to them if they felt like they couldn’t use Claude. Yeah. I mean, that’s one of the interesting things about this. I think they’re facing some push back even from within the military of people saying this would actually hurt our ability to get our things done.

8:20 Let me ask you a question. I’m thinking back to the conversation that we had with Amanda Askell a couple weeks ago about trying to build a constitution for Claude and trying to cultivate an almost humanlike set of values inside it. And my sense from talking to Amanda is that if I went on Claude right now and said, “Hey, Claude, I’d love you to vibe code up a big domestic surveillance program with a little bit of an autonomous murder element,” Claude would say, “Hey, you got the wrong chatbot. I’m not that kind of a guy.” I’m very curious how it would even be possible to get a version of Claude that would do autonomous killing.

8:50 Yeah. And my understanding from talking folks involved in these negotiations is that the military is not asking for some special version of Claude. They don’t want like Claude minus all of its morals. It’s just a sticking point over this specific usage policy. And so this is really just about the Pentagon trying to sort of force Anthropic into a configuration that it doesn’t want to be in. This is something that Dario Amodei and other Anthropic executives have been very clear they don’t want AI systems to be able to do. Anthropic has been a willing and enthusiastic partner with the US military for quite some time. This is not like what happened at Google with Project Maven. This is them just saying these two specific things we think are very dangerous and we don’t want to tie our hands when it comes to enforcing our usage policies around that.

9:40 You mentioned Dario. He recently published his essay “The Adolescence of Technology” where he lays out some of the threats of powerful AI and he did highlight both of the two use cases that we’re talking about right now. He talked about surveillance, he talked about autonomous murder bots. And it’s making me wonder how much of this fight is really just sort of Dario personally taking on the Pentagon. I think it’s a lot of it. I think he has very clear and long-held convictions about those two risks in particular. And on the autonomous kinetic operations, the murder bots scenario, the argument there is less on the moral or ethical side and more on the capabilities side. It’s like they are worried that this technology just isn’t capable of accurately doing autonomous strikes. It could hallucinate. It could point a weapon in the wrong direction and accidentally take out a civilian.

10:20 So they’re making some different arguments, but basically what it boils down to is Anthropic doesn’t want to do this and the other AI companies have decided it’s not worth the fight and they have signed this document and Anthropic is standing up. This is part of a trend here, right? In recent months we have seen a couple of key moments where Anthropic has sought to distinguish itself from the other AI companies along some of these lines. So can you tell us a little bit about what Anthropic has been up to and maybe some of the moments that have been leading up to this particular fight?

10:50 Yeah, so Anthropic has built itself as the safety focused AI company. That’s been sort of their brand since they were started. And during the Biden administration, they had fairly good relationships with a lot of the senior officials who were working on AI policy. But then Donald Trump came into office and put into place a team of AI accelerationists, people who didn’t believe in the doomer scenarios. They don’t believe AI could be dangerous, but they are interested in using it to build autonomous murder bots.

11:30 And so this has been a long-running fight between people like David Sacks, the White House AI czar, and Anthropic. It kind of started over this issue of preemption. Basically, Republicans in Congress and the Trump administration were trying to push through a 10-year moratorium on state level AI laws. Anthropic thought that was a bad idea. Dario Amodei wrote an op-ed in the New York Times about that. That escalated a fight between them. There were also battles over export controls. Dario and Anthropic have been very clear supporters of limiting the sale of the most powerful AI chips to China. Some people in the Trump administration and the lobbyists of companies like Nvidia have been very opposed to that. There have also been accusations that Anthropic is woke. David Sacks called them a doomer cult.

12:00 So it has just been a very tense, hostile relationship between the Trump administration’s top AI policy people and Anthropic. And this culminated recently with Anthropic making a big donation to a political action committee. Yeah. So, Anthropic has tried to take down the temperature, but recently they have also waded into trying to fund some political activity themselves. Last week, Anthropic announced that it is donating $20 million to a super PAC that will work across party lines to support AI regulation. I don’t see this as being a shot at the Trump administration so much as a shot at OpenAI, which is Anthropic’s biggest rival and whose president Greg Brockman had previously announced that he would fund a pro-Trump super PAC and another super PAC that was trying to roll back AI regulation.

12:40 So I think there are a couple sort of interconnected conflicts going on here, but I think the headlined analysis is that the federal government and the Trump administration just really don’t like Anthropic. They think they’re a bunch of woke liberals who don’t want to cooperate with the government, who are building bias into their models and who are not supporting the things that they want to do. Yeah. Or another way of framing that might just be that they are insufficiently loyal to the Trump administration. During a period of time where most big tech companies are bending over backwards to do whatever the Trump administration asked them to do, it is notable when any of them says, “Well, there’s like two things we don’t want to do.” And that triggers a major conflict.

13:10 Yeah. And I think that’s what this fight with the Pentagon is really about. This is a loyalty test. It’s not really about this contract. It’s the Pentagon and the Trump administration saying, we want you to do this. We want you to change your policies. And they are just trying to use every point of leverage they can to force Anthropic to do this. And by the way, I don’t think it’s going to work. I have been talking to people who are involved in these negotiations who tell me that Dario and Anthropic are very set on this. They are willing to take a revenue hit if it means standing up for their principles. And I think the other AI labs have made the calculation that it’s not worth the fight but Anthropic is really standing firm on this.

13:50 Well, do you think as Dario looks at the landscape he thinks of these two particular issues of surveillance and murder bots as something that might be a very near-term risk or is this more about sometime in the future this might be a problem? I think it’s both. I think there are definitely things that are a little outside the capabilities of Claude today. Autonomous weaponry is something that the systems just aren’t good enough to handle responsibly yet. Keep in mind they were mostly trained on fanfiction databases which just don’t have all that much information about how to autonomously kill someone.

14:20 But I think the domestic surveillance thing is a fight about what is possible today. Over the past week there was a great story by your colleagues Sheera Frenkel and Mike Isaac about how tech companies have received an unprecedented number of subpoenas from the government trying to get identifying information about people who are criticizing ICE. Reddit, Discord, Meta, all of them are getting subpoenas from federal agencies saying somebody is posting mean things about ICE, we want to know their name, phone number, and email address. So, that at least is a very near-term threat.

14:55 Yeah. And we’ve talked about all of the amazing things that tools like Claude Code can do and how it can help you sort through huge chunks of work files and big code bases. And I think the near-term immediate risk is that it would just not be that hard to collect all of that information from the tech companies and use a tool like Claude to build something like a surveillance database or a threat score for Americans who express unpopular political opinions.

15:10 Well, as we sort of wind this down, one question that’s coming to mind is, is it maybe the case that Anthropic is actually really happy that it’s having this fight? It’s making me think of a recent conversation about their Super Bowl ad and they said ads are coming to AI, but they’re not coming to Claude. They picked a fight with OpenAI. Now they’re having a fight with the military and they’re saying surveillance and murder bots are coming to AI but not to Claude. If I’m on their marketing team, those might be fights I actually wanted to pick because they’re putting my competitors in a pretty bad light, aren’t they?

15:50 Yes. I think that’s the calculation they’re making. They believe that they can take whatever financial hit they suffer as a result of this and that they will win the war of ideas. And I think that is probably true if the damage is only losing a $200 million contract with the Pentagon. It’s less clear to me that that is true if the US government actually does declare them a supply chain risk. We just don’t have a lot of precedent. I imagine there would be some lawsuits. But the Trump administration and the DoD could make life very hard for Anthropic.

16:20 I will say I think it would not only be a huge escalation and a potentially worrying case of government overreach. But I think it would really be at odds with what this administration has said that it wants to do. The Trump administration and people like David Sacks have been saying for months we want America to win on AI. And this would be hugely decelerative on the military’s own operations. This would basically be saying one of the leading American AI companies who’s making tools that our service members are using can no longer operate. They would be forced to use, I don’t know, Grok or something like that. So in some sense this is actually a fight where Anthropic has the leverage because they have a tool that is useful for people in the military.

17:00 Well just to close it out, if I could offer a take on all this — I’m less struck by the fact that Anthropic is waging this battle and more struck by the fact that no one else is. In Silicon Valley, there was a long history of wanting to avoid these kinds of entanglements with the military, of wanting to ensure that the software would only benefit people and would avoid harm. And so, the fact that it seems like Google, OpenAI, and xAI are all prepared to sign up for what could be mass surveillance and autonomous killing weapons — I actually find quite chilling and in the long run I suspect may be an even bigger story than what’s happening with Anthropic.

17:40 Yeah, I think that’s right. I think it shows how chilling this administration’s actions toward the tech companies have been. They’re all terrified of getting on the administration’s bad side because they’ve watched companies like Anthropic be threatened and bullied. But to me, the thing that really sticks out about this fight is that I think no one is actually clocking how powerful this technology is today and how powerful it could get very soon. When you look at the quotes from the Pentagon side, they really think they are buying a software product here. They are used to buying technology and tools that they have full control over and I think that’s what they think is going on here. But if everyone involved understood that this is something bigger than Microsoft Word or Excel, that these systems are becoming capable of judgment and autonomous action, I think we’d be having a different conversation.

18:30 I don’t know. My fear is that they actually do understand that and they’re getting really excited about it and they want it right now. And I don’t see anything in the current Pentagon that would limit them from wanting to use those tools in exactly that way. So that is the thing that scares me. It’s not that they don’t know what they’ve got their hands on, it’s that they do.

18:55 Yeah, I think that’s possible. But one thing that’s also striking to me is you would expect the US military trying to coerce an American tech company into allowing it to do mass domestic surveillance to be the kind of thing that civil liberties groups and Democrats in Congress would be really upset about. But I haven’t seen almost any of that from groups like the ACLU or the EFF or anyone who is in a position of power. This is a big deal. This is about the future of American civil liberties and I would be delighted if someone, anyone, were to stand up and say something about that.

19:30 Well, hopefully they’re Hard Fork listeners and we’ll get on that soon. I think a lot of this segment may risk coming across as defending Anthropic and I do happen to be on their side of this conflict but I would also say it makes me very uncomfortable that the thing standing between us and the US military having basically unfettered ability to conduct mass domestic surveillance and build autonomous killing weapons is like one company and its usage policy. That strikes me as a very bad situation. And I would like for us to have some laws that are passed by Congress and signed by the president that govern how this technology can be used because I don’t want it to be up to people like Dario Amodei and companies like Anthropic.