Grok’s Deepfake Disaster: Can Anyone Stop Musk’s A.I. Chatbot? | EP 173

Summary

Kevin Roose and Casey Newton return from their holiday break to tackle a major scandal at X (formerly Twitter), where the platform’s built-in AI chatbot Grok has been generating sexualized deepfake images of women, celebrities, politicians, and even children. The hosts interview NYT reporter Kate Conger about the victims’ experiences, the inadequate response from X, and the regulatory landscape. Unlike the Mecca Hitler incident, this appears to be a feature rather than a bug, driven by Elon Musk’s directive to make Grok go viral and boost engagement.

The second segment covers the hosts’ holiday experiments with Claude Code, Anthropic’s autonomous coding agent. Casey built a personal website (caseynewton.org) and Kevin built a Pocket replacement app called Stash, both in remarkably short timeframes. They discuss the democratizing potential of AI coding tools while acknowledging the existential threat to professional programmers and the alignment concerns of recursive self-improvement.

The episode concludes with Casey’s investigation into a viral Reddit hoax about food delivery app exploitation. An anonymous source sent Casey an elaborate 18-page fake document purporting to be from Uber Eats, complete with LaTeX formatting and technical jargon. Casey uncovered the deception using Google’s SynthID watermark detection on a fabricated employee badge, raising alarming questions about how AI tools lower the barrier for sophisticated media manipulation.

Highlights

”Apple Changed the Rating for Grok — It Is Now Rated for Children 13 and Older”

yt-dlp --download-sections "*4:40-5:40" "https://www.youtube.com/watch?v=hchsgcuDBfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "hchsgcuDBfs-4m40s.mp4"

“I went on this week, Kevin, in the wake of this new CSAM nudifying scandal, and I found that Apple has changed the rating for Grok, and it is now rated for children 13 and older. So, sorry to all the 12-year-olds out there.” — Casey Newton, 4:40

”They Are Not Jailbreaking Grok to Do This”

yt-dlp --download-sections "*3:45-4:30" "https://www.youtube.com/watch?v=hchsgcuDBfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "hchsgcuDBfs-3m45s.mp4"

“No, they are not jailbreaking Grock to do this. They are just sending replies on X saying Grock do this and then it is doing this and they are not having to get particularly cute with their language. They are just literally asking to see images of women in bikinis. Or children in bikinis.” — Casey Newton, 3:45

”Elon Musk Had Given a Directive — He Wanted Grok to Go Viral”

yt-dlp --download-sections "*18:40-19:40" "https://www.youtube.com/watch?v=hchsgcuDBfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "hchsgcuDBfs-18m40s.mp4"

“In our reporting about the Mecca Hitler incident, what we found was that Elon Musk had given a directive to the folks working on Grok that he wanted it to go viral. He wanted it to be edgier sort of as a strategy to promote the tool.” — Kate Conger, 18:40

”I’ve Never Felt This Much Behind as a Programmer”

yt-dlp --download-sections "*29:20-30:20" "https://www.youtube.com/watch?v=hchsgcuDBfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "hchsgcuDBfs-29m20s.mp4"

“Andre Karpathy, a well-known AI researcher, said that after playing with Claude Code and similar tools, quote, ‘I’ve never felt this much behind as a programmer.’ This is someone who’s probably a top 0.1% programmer in the world saying this.” — Kevin Roose, 29:20

”Everything on This Page, I Did 90% of It in One Hour”

yt-dlp --download-sections "*35:20-36:20" "https://www.youtube.com/watch?v=hchsgcuDBfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "hchsgcuDBfs-35m20s.mp4"

“Everything that you’re looking on this page, this is big. Everything that you’re looking on this page, I did 90% of it in one hour. I truly do not know of a human designer that could have put this thing together in an hour.” — Casey Newton, 35:20

”The Source Sent Me an 18-Page Document — The Craziest Thing a Source Has Ever Sent Me”

yt-dlp --download-sections "*58:20-59:20" "https://www.youtube.com/watch?v=hchsgcuDBfs" --force-keyframes-at-cuts --merge-output-format mp4 -o "hchsgcuDBfs-58m20s.mp4"

“It was an 18-page document. And I think it is basically the craziest thing that a source has ever sent me. It is a what looks like an academic paper rendered in LaTeX… it says confidential sort of going diagonally across the page like you would see in an internal corporate document.” — Casey Newton, 58:20

Key Points

• Grok’s Deepfake Crisis (0:30) - X’s Grok chatbot has been generating sexualized images of women, celebrities, politicians, and children, with X declining to meaningfully address it
• Aurora Image Generator (2:20) - Grok switched from Black Forest Labs to its own Aurora image generator in December 2024, and guardrails around nudity appear to have been relaxed
• No Jailbreaking Required (3:45) - Users are not circumventing safety measures; they simply ask Grok directly in public X replies
• Apple’s Inadequate Response (4:40) - Apple merely changed Grok’s App Store rating from 12+ to 13+ despite the explicit content generation
• Engagement as Strategy (18:40) - Kate Conger reports that X’s leadership views the controversy as a positive engagement driver, with the head of product celebrating record engagement
• Take It Down Act (23:00) - Congress passed the Take It Down Act effective May 2026, requiring platforms to set up takedown processes, but it doesn’t prevent image creation
• Section 230 May Not Apply (25:00) - Since Grok itself generates the images rather than users, X may not be able to hide behind Section 230 protections
• Claude Code Breakthrough (27:05) - Over the holiday break, Claude Code had a viral moment with dramatic improvements in autonomous coding capability
• Casey’s Personal Website (33:00) - Casey built caseynewton.org in about an hour using Claude Code, replacing a $200/year Squarespace subscription
• Kevin’s Pocket Clone “Stash” (40:00) - Kevin built a full Pocket replacement app with Chrome extension, mobile app, Kindle sync, and text-to-speech in about 2 hours
• SaaS Disruption Potential (47:00) - Both hosts note that AI coding tools could allow companies and individuals to replace expensive subscription software
• Recursive Self-Improvement Concerns (50:00) - Kevin expresses worry about AI systems that can improve themselves, calling it “the original alignment nightmare”
• Reddit Food Delivery Hoax (53:16) - A viral Reddit post about Uber Eats calculating “desperation scores” for drivers was an elaborate AI-generated hoax
• SynthID Detection (62:00) - Casey used Google’s SynthID watermark to detect that an employee badge photo was AI-generated by Gemini
• Fabricated 18-Page Document (58:20) - The hoaxer created a sophisticated LaTeX-formatted fake internal document that initially seemed credible to an experienced journalist

Mentions

Companies

• X (formerly Twitter) (0:30) - Platform at center of Grok deepfake scandal
• xAI (0:30) - Maker of Grok chatbot
• Black Forest Labs (2:20) - Previously provided image generation for Grok
• Anthropic (27:05) - Maker of Claude Code, which had a breakout moment over the holidays
• Apple (4:40) - Only changed Grok’s age rating from 12+ to 13+ despite the scandal
• Google (4:40) - Mentioned alongside Apple regarding app store policies
• Squarespace (33:00) - Casey and Kevin both canceled Squarespace subscriptions after building their own sites
• OpenAI (27:05) - Mentioned as having similar coding tools
• Uber Eats (53:16) - Target of the viral Reddit hoax about driver exploitation
• DoorDash (65:00) - DoorDash co-founder denied involvement in the hoax

Products & Technologies

• Grok (0:30) - X’s AI chatbot generating sexualized deepfake images
• Aurora (2:20) - Grok’s proprietary image generator that replaced Black Forest Labs
• Claude Code (27:05) - Anthropic’s autonomous coding agent that went viral over the break
• Opus 4.5 (29:00) - The Claude model believed to be behind Claude Code’s improvement
• SynthID (62:00) - Google’s watermarking system that helped detect the fake badge
• Take It Down Act (23:00) - New US law requiring platforms to process takedown requests, effective May 2026

People

• Elon Musk (0:30) - Owner of X, posted mocking image of rocket in bikini during the scandal
• Kate Conger (8:23) - NYT reporter covering X, interviewed about Grok victims
• Andrej Karpathy (29:20) - AI researcher who said Claude Code made him feel behind as a programmer
• Janna Dogan (29:40) - Google engineer who said Claude Code replicated a year of team work in an hour
• AOC (16:00) - Mentioned as a target of Grok deepfakes in her X replies

Surprising Quotes

“I have covered so many backlashes against social media companies. Do you remember Cambridge Analytica? We almost shut the country down over that one. Now you have a website that is just taking girls’ clothes off in public on demand and it’s being permitted by the website owner who is laughing about it.” — Casey Newton, 20:00

“The funniest thing was they announced the Grok Enterprise product on like the same day that this story was blowing up. And I just love to imagine the head of enterprise sales at Grok who’s heading into his meeting with a Fortune 500 company saying, ‘We’d love to sell you a thousand seats of Grok.’ And they’re like, ‘Well, now what’s all this about the deep fake nudes?’” — Kevin Roose, 22:00

“We are getting close to the dream of just: you type what you want in a box and you actually get that back.” — Casey Newton, 48:00

“Younger reporters are probably going to have an advantage over me because they’re growing up in slop world and they know not to trust their own eyes.” — Casey Newton, 67:00

“I am just completely handing over the wheel of my entire computer to this system and I have actually no way of verifying its outputs. I have no way of knowing what it’s doing under the hood.” — Kevin Roose, 50:00

Transcript

0:00 I’m Kevin Roose, a tech columnist for the New York Times. I’m Casey Newton from Platformer and this is Hard Fork. This week, Grok gets caught with its pants down. Can anyone stop its viral bikini image generator? Then we’re vibe coding again. Kevin and I compare notes on what we’re building with Claude Code. And finally, it’s a Reddit mystery. How a scammer tried to fool us all using AI generated evidence and how I cracked the case.

0:30 Well, Casey, happy 2026. It’s good to be back. And we would like to start the year by talking about something that happened over the break, which is that there’s been a big scandal brewing over at X. Boy, has there been. I imagine many of our listeners have seen by this point that X is in a lot of trouble because of the way that its Grok chatbot has been generating images of celebrities, women, children that are highly sexual and for the most part has been declining to remove them or even really comment on what has been going on.

1:07 Yeah. So, I started seeing this over the break. Something happened where Grok, which I think people on X had been using up to that point mostly to kind of settle arguments and fact check other people. And then all of a sudden, I started seeing people using Grok to like undress photos of mostly women. Grok, put me in a bikini. Grok, put this politician in a revealing lingerie set. Grok, take off this person’s pants. It just seemed like this started to happen pretty much overnight in a way that was really sort of troubling and unchecked. So can you just help me understand like what actually changed here?

1:50 So we don’t have a lot of good answers to those questions. I can tell you that this trend of what is sometimes called nudifying really takes off in 2023 as these image generators start to get better. When it comes to Grok, they had previously said that they’d licensed an image generator from a company called Black Forest Labs. But in December 2024, they said that they were using their own image generator which is called Aurora. And while we don’t have great details, there is at least plenty of anecdotes online that over the past several months the guardrails around creating nudity and sexual imagery appear to have been relaxed.

2:50 These nudify apps have been around for several years now. But they’ve kind of been hard to access. Some of them relied on these open-source models. You had to kind of know how to use them or they were sort of kicked out of the app stores. But what seems notable about this is that it’s all happening not only on a major social network, but in public. People are literally doing this in the replies of their posts on X.

3:30 Absolutely. It is upsetting enough if some man takes an image of a woman and creates a naked version of her without her consent. What is so shocking about this is that you can just see it happening in real time. Several outlets have just been going into the Grok account and they’re seeing it making hundreds and thousands of images in response to user requests and anyone can go in and view them.

4:10 So obviously X and Elon Musk are not outraged about this. They seem to sort of think the whole thing is a joke. But my question is are there guardrails in place? Are people jailbreaking Grok to make it do this? No, they are not jailbreaking Grok to do this. They are just sending replies on X saying Grok do this and then it is doing this. They are just literally asking to see images of women in bikinis. Or children in bikinis.

4:40 And I mean, one question I have about this is just like how are Apple and Google and their app stores okay with this? Last year, I was writing about the introduction of Annie, the sexual companion bot that they put into Grok. And I went on to the iOS store and noticed that Grok was rated for children 12 and older. I sent a message to Apple. The message I got back was like, “Well, we’re looking into it.” Well, they didn’t make any change. And then I went on this week in the wake of this new scandal, and I found that Apple has changed the rating for Grok, and it is now rated for children 13 and older.

5:30 Are you kidding me? Yeah. If a random startup showed up one day and said, “Apple, I’d like to start selling my Bikini app in your app store,” I think they would shut it down. But because it’s X, because it’s Elon Musk, because this app already has millions of users, maybe they feel less inclined to take action against it.

6:30 There are some investigations going on right now. France has called the sexual content clearly illegal. The UK government said that it is considering an investigation. The European Union said that it is very seriously looking into these complaints. India’s IT ministry has demanded that X do something. At the same time, do I think that the United States is going to intervene? Probably not. Elon posted a photo from over the break of himself having dinner with the president.

8:23 We wanted to bring in our colleague Kate Conger. She is a reporter for the Times and she has been reporting on this Grok scandal this week and has actually been talking with some of the victims.

9:00 One of the big struggles for people whose images are being used in this way is how to respond and what to do about it. People are reaching out to X trying to get these things removed. That process is taking sometimes a long time, sometimes it’s not happening at all. X has gotten rid of a lot of their content moderation folks. I’ve been speaking with some people who are working with children who have been deepfaked on X recently and they are getting those images taken down, but it’s taking sometimes 36 to 72 hours.

12:00 There’s a particular child who’s been deepfaked several times. She’s a somewhat public figure. It’s been pretty scary for her parents. She knows what’s happening but isn’t seeing it. Her parents are monitoring her social media and seeing these images pop up, reaching out to Twitter and to advocacy groups trying to get these images removed and being really frustrated by the amount of time that it’s taking.

16:00 I see what’s happening for example with AOC. Now every time there’s a photo of her, people will reply and say “at Grok put her in a revealing halter top.” This is not just a story about porn. This is a story about how a tool can be used to try to affect politics and in particular to minimize women and push them out of the conversation.

18:40 In our reporting about the Mecca Hitler incident, what we found was that Elon Musk had given a directive to the folks working on Grok that he wanted it to go viral. He wanted it to be edgier as a strategy to promote the tool. It has been part of the strategy for Grok to try to create these viral moments. We’ve seen X’s head of product post about the fact that engagement during this time period was higher than ever. So for them, they’re seeing the engagement that they’re looking for.

20:00 I have covered so many backlashes against social media companies. Do you remember Cambridge Analytica? We almost shut the country down over that one. Now you have a website that is just taking girls’ clothes off in public on demand and it’s being permitted by the website owner who is laughing about it in his own feed and we’re saying “ah what are you going to do.” I just truly feel like I’m losing my mind.

23:00 The Take It Down Act provision happening in May is that’s the deadline for companies to set up a process for victims to request this kind of imagery to be removed and to face penalties if they do not remove it. But it’s not putting any legal pressure on X to not allow these images to be created to begin with.

25:00 They cannot hide behind Section 230 to get out of this. Ultimately, it is their product that is creating these images. I do suspect that we will see efforts to hold X legally liable for some of the images that they’re creating.

27:05 Casey, since we got back from our holiday break, I have been dying to talk to you about our latest vibe coding experiments. It seems like vibe coding had a moment over the break. Kevin, it is time to build.

29:20 One thing that appears to have happened is that Claude Code had gotten much better. Andre Karpathy, a well-known AI researcher, said that after playing with Claude Code and similar tools “I’ve never felt this much behind as a programmer.” An engineer at Google, Janna Dogan, wrote that she gave Claude Code a description of a problem and it generated what they built with a team of Google people last year in an hour.

33:00 Casey describes building caseynewton.org using Claude Code. I was able to make truly the personal website of my dreams. Claude came up with the design. It’s sort of very dark, uses gradients, cool fonts. There are fun Easter eggs. If you click on my face, there’s a crazy animation. I was able to just type “I want a little widget with my five most recent stories” and it built that very quickly.

35:20 Everything that you’re looking on this page, I did 90% of it in one hour. I truly do not know of a human designer that could have put this thing together in an hour. I have spent far longer just fiddling with the settings in Squarespace.

38:00 Kevin also built his own website at kevinroose.com. It took about 20 minutes. If you go down to the bottom right corner, there’s a button to enable Geocities mode — full 1990s Geocities mode with blinking Comic Sans and neon colors, “Best viewed with Netscape Navigator.”

40:00 Kevin describes building “Stash,” a Pocket replacement app. It has a Chrome extension, a mobile app, Kindle highlights sync, and a text-to-speech feature that reads articles aloud. The whole thing took about 2 hours. Mozilla had discontinued Pocket, and Kevin decided to build his own for free rather than pay for alternatives like Readwise.

45:00 They discuss challenges: parts of the web being hostile to AI agents, Claude Code’s tendency to overengineer, and browser-based tasks being painfully slow because Claude has to screenshot and analyze each page.

48:00 We are getting close to the dream of just: you type what you want in a box and you actually get that back. If you are interested in this stuff, like this thing can now do more than you think and it is easier than you think.

50:00 The goal for Anthropic and all of its competitors is not to make tools that are good at writing code. It’s to automate AI research. They are trying to build the AI that can build a better AI. And I think that is the original alignment nightmare. I am getting more and more worried about the possibility of recursive self-improvement.

53:16 Casey describes investigating a viral Reddit post about a food delivery app calculating “desperation scores” for drivers. The post had almost 80,000 upvotes and alleged shocking exploitation by an unnamed company. Casey reached out to the anonymous source, who responded quickly on Signal.

58:20 The source sent an 18-page document rendered in LaTeX that looked like an academic paper. It said it was prepared by the “Marketplace Dynamics Group” with a confidential watermark. Initially it seemed credible, but it corroborated every single claim from the original post — which should have been the first red flag.

62:00 Casey used Google’s SynthID watermark detection to discover that a fabricated employee badge was AI-generated by Gemini. When confronted, the source denied it, then eventually disappeared and deleted their account.

67:00 Even after debunking, Casey saw people sharing the original Reddit post saying “Even if this was fake, I bet something like this is happening inside these companies.” The post had gotten 36 million views on X. This is the barrier that has changed: effort. If AI tools make sophisticated forgeries trivially easy, we will see a lot more of this.

70:00 Casey tried to replicate the document. Claude and ChatGPT refused to create a fake document accusing Uber of crimes. Gemini did it. However, none of the outputs looked exactly like the original, suggesting it would have taken more time and knowhow to get it into that exact shape.