WTF is happening at xAI | Sulaiman Ghori

Summary

Sulaiman Ghori, a Member of Technical Staff at xAI, provides an unprecedented inside look at the company’s intense culture and operations. He describes an environment where “due dates are always yesterday,” engineers have extraordinary autonomy, and the focus is relentlessly on eliminating artificial barriers to speed. The interview covers xAI’s approach to building AI infrastructure, their secretive “Macrohard” project for human emulation, and the unique organizational culture that enables rapid iteration.

The conversation reveals fascinating details about xAI’s infrastructure achievements, including building the Colossus data center in 122 days using creative permitting approaches, running 80+ mobile generators to balance power loads, and their plans to potentially leverage Tesla’s fleet of parked vehicles as distributed compute. Sulaiman also shares his personal journey from building fidget spinners and 3D printers as a teenager to constructing liquid fuel rocket engines, embodying the entrepreneurial engineering spirit that defines xAI.

Highlights

”$2.5 Million Per Commit”

Clip command

yt-dlp --download-sections "*0:21-0:45" "https://www.youtube.com/watch?v=8jN60eJr4Ps" --force-keyframes-at-cuts --merge-output-format mp4 -o "8jN60eJr4Ps-0m21s.mp4"

“We did the math. Right now we’re at about $2.5 million per commit to the main repo and I did five today.” — Sulaiman Ghori, 0:21

”xAI is Actually Just a Carnival Company”

Clip command

yt-dlp --download-sections "*25:52-26:50" "https://www.youtube.com/watch?v=8jN60eJr4Ps" --force-keyframes-at-cuts --merge-output-format mp4 -o "8jN60eJr4Ps-25m52s.mp4"

“There’s basically a special exception within the local and state government that says if you want to modify this ground temporarily—I think it’s like for carnivals and stuff—you can. xAI is actually just a carnival company currently.” — Sulaiman Ghori, 25:52

Tesla Fleet as Distributed Compute

Clip command

yt-dlp --download-sections "*10:50-12:00" "https://www.youtube.com/watch?v=8jN60eJr4Ps" --force-keyframes-at-cuts --merge-output-format mp4 -o "8jN60eJr4Ps-10m50s.mp4"

“There’s like 4 million Tesla cars in North America alone… somewhere between 70-80% of the time they’re sitting there idle probably charging. We can just pay owners to lease time off their car and let us run a human emulator on it.” — Sulaiman Ghori, 10:50

Virtual Employees on the Org Chart

Clip command

yt-dlp --download-sections "*48:25-49:25" "https://www.youtube.com/watch?v=8jN60eJr4Ps" --force-keyframes-at-cuts --merge-output-format mp4 -o "8jN60eJr4Ps-48m25s.mp4"

“Multiple times I’ve gotten a ping saying ‘Hey, this guy on the org chart reports to you. Is he like not in today?’ And it’s an AI. It’s a virtual employee.” — Sulaiman Ghori, 48:25

Cybertruck Bet - 24 Hour GPU Challenge

Clip command

yt-dlp --download-sections "*23:15-24:10" "https://www.youtube.com/watch?v=8jN60eJr4Ps" --force-keyframes-at-cuts --merge-output-format mp4 -o "8jN60eJr4Ps-23m15s.mp4"

“Elon’s like okay you get a Cybertruck tonight if you can get a training run on these GPUs in 24 hours. And we were training that night. Did he get the Cybertruck? Yeah he got it.” — Sulaiman Ghori, 23:15

Key Points

No Due Dates (1:02) - At xAI, due dates are “always yesterday” with no artificial blockers
Hardware is the Edge (1:36) - Their biggest competitive advantage is infrastructure deployment that nobody else is close to
Elon Predicts Bottlenecks (1:46) - Elon excels at identifying future bottlenecks months or years in advance and working backwards
Shredding Timelines (3:05) - Model iterations on Macrohard happen daily, sometimes multiple times a day from pre-train
No Onboarding (4:23) - First day he got a laptop and badge with no team assignment or direction - just found work to do
Tesla Computer Synergy (9:23) - Planning to potentially use 4 million Tesla cars’ computers as distributed compute for human emulators
Macrohard Project (11:59) - Building “human emulators” that can do any digital task a human does, targeting 1.5-8x faster than humans
Elon Puts Out Fires (13:14) - When blockers arise, Elon makes a phone call and patches arrive next day
Only 3 Management Layers (16:30) - ICs, co-founders/managers, and Elon - that’s it
Cybertruck Bet (20:33) - Tyler won a Cybertruck by getting a training run on new GPUs in 24 hours
80 Mobile Generators (21:12) - Use 80+ mobile generators and battery packs to balance power load with municipal grid
Colossus in 122 Days (22:45) - Built using temporary “carnival” permits to fast-track permitting
Work Backwards (23:35) - Always work backwards from highest leverage outcome, determine physical requirements last
$2.5M Per Commit (25:51) - They calculated each commit to main repo is worth ~$2.5 million in expected value
Challenge Requirements (30:27) - Interviewers include impossible requirements to see if candidates push back
Experimentation Culture (32:46) - Unlimited Grok access means they can try “stupid things” that would cost $100k-$1M at startups
Calibrated Timelines (34:55) - Elon has calibrated his estimates much better after deploying hardware at scale
Everyone’s an Engineer (42:09) - Even the sales team writes code; fewer than 8 non-engineers when he joined
Fuzzy Team Boundaries (44:06) - Anyone can fix anything; no strict regiment, trusted to do the right thing
Virtual Employee Tests (48:25) - Testing human emulators internally as employees; people try to visit empty desks
Biggest Blunders (50:00) - Humans forget to mention 20 steps they do on autopilot when training emulators
Elon Meetings (53:23) - Simple meetings; “smooth” means limited feedback or thumbs up
Grokipedia Truth Problem (56:44) - Very hard problem; trying to drill down to fundamentals and build up from there
War Room Operations (1:00:08) - Macrohard team has been in war room mode for 4 months; outgrew original room
Fidget Spinner Business (1:02:53) - Built 3D printer at 13, ran fidget spinner factory until county shut it down
Liquid Fuel Rocket (1:08:48) - Built a liquid fuel rocket engine in 24 hours; jacket caught fire during test

Mentions

Companies

xAI (0:41) - Primary subject; Elon Musk’s AI company
Tesla (9:23) - Potential partnership for distributed compute using car computers
SpaceX (1:04:30) - Referenced for Elon’s hardware deployment experience
Nvidia (13:14) - GPU supplier; Elon calls for same-day patches
Amazon/AWS (4:00) - Referenced as traditional infrastructure approach
Google (4:00) - Referenced as traditional infrastructure approach
Oracle (12:00) - Mentioned as VM alternative
RepRap (1:04:00) - Open-source 3D printing project that inspired Sulaiman
YKK (1:07:30) - Referenced as example of specialized manufacturing

Products & Technologies

Macrohard (3:19) - xAI’s human emulator project
Grok (33:00) - xAI’s AI model; unlimited internal access
Grok 3 (8:30) - Model version when Sulaiman joined
Grok 4/5 (9:00) - Already planned out before Grok 3 shipped
Grokipedia (56:44) - xAI’s alternative to Wikipedia focused on truth
Colossus (9:10) - xAI’s data center built in 122 days
ASRock (5:00) - xAI’s X/Twitter integration
Full Self-Driving (37:30) - Tesla’s autonomous driving system, referenced for model architecture parallels
Hardware 4 (9:40) - Tesla’s in-car computer hardware
Cybertruck (20:33) - Prize for 24-hour GPU training bet

People

Sulaiman Ghori (0:41) - Member of Technical Staff at xAI, interviewee
Elon Musk (1:46) - CEO of xAI, SpaceX, Tesla
Greg Yang (5:30) - xAI co-founder who recruited Sulaiman
Tyler (20:33) - xAI engineer who won Cybertruck bet
Igor (1:01:30) - xAI co-founder known for Starcraft AI work
Chester (30:27) - xAI team member who shared hiring tips
Jensen (9:10) - Nvidia CEO who praised xAI’s Colossus build
James Burke (57:30) - Documentary maker referenced for “Connections” series approach
Gary Tan (39:15) - Y Combinator CEO, referenced for Elon Q&A

Surprising Quotes

“We don’t have really due dates. It’s always yesterday.” — 1:02

“Right now we’re at about $2.5 million per commit to the main repo and I did five today.” — 0:30

“No one tells me no. If I have a good idea I can usually go and implement it that same day and show it to Elon.” — 0:23

“The lease for the land itself was actually technically temporary… I think it’s like for carnivals and stuff.” — 22:45

“Multiple times I’ve gotten a ping saying ‘Hey, this guy on the org chart reports to you. Is he like not in today?’ And it’s an AI. It’s a virtual employee.” — 48:50

“A job for one person done by two will take twice as long.” — 46:30

Transcript

0:00 Tyler took this bet with Elon like get a Cybertruck tonight if you can get a training run on these GPUs in 24 hours and we were training that night. Did he get the Cyber Truck? Yeah, he got Cyber Truck. My first day they just gave me a laptop and a badge and I was like okay now what? I don’t even have a team. I’ve not been told what to do. ASRock was spinning up at the time our integrations with X. They’re like can you help? And I was like yes.

0:21 What’s the most fun thing about working there? No one tells me no. If I have a good idea I can usually go and implement it that same day and show it to Elon or whoever and we got an answer. We did the math. Right now we’re I think at about $2.5 million per commit to the main repo and I did five today. So you added like 12 and a half million of value. The levers are extremely strong.

0:41 Today I have the pleasure of sitting down with Sully Ghori and he is one of the engineers at xAI. I’ve been kind of fascinated by xAI since like 2023 when like Elon first started. I think it’s like one of the fastest growing companies of all time. Can you just talk about like what the fuck is happening at xAI?

1:02 Yeah. We don’t have really due dates. It’s always yesterday. There’s no blockers for anything like at least nothing artificial. The whole Elon thing about going down to the root, the fundamental, whatever the physical thing is, we get there pretty quick if we can, as quick as we can, which is funny in software. It’s not really like a thing that you think about is the physics too much, but we do try quite a bit and we’re not really fully a software company given all the infrastructure pull down.

1:32 It’s kind of hardware at this point. Yeah, it’s like hardware constrained. It’s probably our biggest edge is the hardware because nobody else is even close on the deployment there. Although the talent density on software is like incredible. I’ve never been anywhere like that. It’s really cool.

1:46 For Elon, he is very good at figuring out like what the bottlenecks will be even like a couple months or even years in the future and then trying to work backwards from that and make sure that like he’s in a really good position. How does that work day-to-day with just normal people like at xAI and like adopting that kind of mental framework?

2:04 Usually when we spin something up new very quickly either one of us or he comes up with this metric that’s usually very core to either the financial or the physical return or both sometimes. And so everything is just focused on driving that metric. There’s never like a fundamental limitation to it or like whatever the fundamental limitation is it better be rooted deep down and not something artificial.

2:37 And there is a lot of perceived limitations especially in the software world coming from like especially in the last 10 years of like web dev and all these kinds of things. People just assume or accept certain limitations especially when it comes to speed and latency and they’re not true. You can get rid of a lot of overhead. Like there’s a lot of stupid stuff in the stack and if you can knock out a lot of that, you can usually two to 8x most anything at least anything invented relatively recently.

3:05 When was the last time that you experienced this where there’s some like conventional wisdom that says that there’s this is the timeline and then you guys just were able to completely shred that? Most recently it’s our model iterations on Macrohard. So we’re working on some novel architectures actually multiple at the same time and we’re coming out with new like iterations like daily sometimes multiple times a day which is from pre-train in some cases which is not something you ordinarily really see.

3:39 But it comes from well we have a pretty great supercompute team and they’ve knocked out a lot of the typical barriers it takes to train a lot of this stuff even with how variable our hardware is. Like it’s you know within a day of standing up a rack you can usually be training sometimes within the same day even within a few hours in some cases. And this is like not normal like normally the timelines are like days or weeks.

4:04 It takes a lot well in most cases at least yeah in the last 10 years you abstract this away and let Amazon or Google take care of this and so whatever their capacity is is what their capacity is but that’s not like you can’t have that be the case in AI now. So the only solution is to die or build it yourself.

4:23 Can you tell me about like what your experience was like joining why you joined and then kind of what the like onboarding process was for the first like couple weeks? Yeah. So I was working on my own startup when I moved to the Bay. And actually during that time, Greg Yang, one of the co-founders of xAI, had reached out. He’s great at recruiting as it turns out.

4:48 What did he say? So I got an email and I thought it was spam because I was getting a lot of these like you know emails to founders at the time of like hey you want to chat or like I like what you’re doing. I was going to mark it as spam to delete it. And I saw the domain x.ai. I was like oh wait a second. I know these guys. And they just I think it was probably eight months in at that point.

5:15 And so I was like okay yeah let’s chat. And so we chatted a bunch of times. Then I wanted an acqui-hire but I think we were too early at the time and that company kind of went mostly because it was fairly obvious that you can’t build Macrohard with like a million dollars. But the idea was sound. So I spent the next like six seven months wasting all my money building like aerospace projects and working on an aerospace astro mining concept.

5:45 That also I realized like probably wouldn’t work but it was worth a try. And so I emailed Greg again like hey can you want to chat again? He’s like yeah sure you want to interview tomorrow. I was like okay. And I apparently did well and I moved on Monday and I started then and it was really great. Nobody told me what to do. So like my first day they just gave me a laptop and a badge and I was like okay.

6:15 And I was like okay now what? And so I went to go find Greg cuz I was like I don’t even have a team. I’ve not been told what to do. Like Greg just brought me on cuz I think he liked what I was doing previously and it was related to what the long term was for Macrohard which wasn’t really even a project at the time. And I ended up working on actually ASRock was spinning up at the time where our integrations with X and so they’re like can you help and I was like yes I can help.

6:50 And so my first week was working with the one guy. I found out very quickly like everything that we built like I could sit and I could stand up from my desk which I didn’t even have a desk assigned to me. I just sat at random people’s desks that weren’t there that day. And I could point to whoever built that thing at xAI like from my desk. It was very very very cool.

7:15 And there was like almost no people working there at this point. Just like a couple hundred right? Yeah about a hundred or so on the engineering staff. And then I don’t know what the infra buildout team looked like at that time. And it’s kind of hard to tell because some people move up the ladder from like the actual building and construction crew onto our payroll. But it was pretty small at the time like much much like an order of magnitude smaller than the other labs. And we had still just done Grok 3.

7:50 Which yeah was pretty cool. One of the things that I kind of love is how fast xAI went from being founded. I remember Elon initially saying like we’re not even sure if this can be a success with you know people having you know a multi-year advantage on speed and like timing and then you guys got done with the first like Colossus data center in like 122 days.

8:15 And that was just like unheard of and Jensen’s out here saying singing the praises of xAI and Elon. What kind of culture did that allow to be formed? It definitely enabled like us on model and product to kind of assume we would have the resources to do what we needed to do. And that’s definitely the case. Like we’re not super duper resource constrained. Like we’ve still found a way to push up against that wall.

8:45 But that’s just we have 20 different things going at the same time. Like more than that like many more things than that. There’s an absurd amount of runs and training and all that stuff going on at the same time in parallel usually by like a few a handful of people. Which is how we’re able to iterate very quickly on model and product side. And utilization has definitely been very high.

9:10 The speed allows us definitely to I guess think more long term. So I think Grok 4 or 5 really what it was was already planned out and and designed in terms of size and what we expected way early like before I joined. I joined around Grok 3. So it’s like thinking at least a year in advance. You can yeah you can think much more in advance and assume that those estimates will be hit just because everyone’s like pretty great and reliable which frees you up a lot in terms of like what your limitations are.

9:45 So for us for example the assumed minimum latency was about three times higher than it actually needed to be and the buildout allowed for that basically. What do you mean by that? So one of the novel architectures we’re working on is not really possible unless you scale up your experiment rate because it’s not building on any existing body of work. You need a new pre-training body and you need also a new data set but that’s not really constrained by the resources like the physical infrastructure resources mostly.

10:20 Although there’s the Tesla computer thing which I think maybe we’ll get into maybe not but actually this one’s public. So one thing that we’re thinking about is okay like we’re building this human emulator with Macrohard. How do we deploy it? Because you actually need like if we want to deploy 1 million human emulators we need 1 million computers. How do we do that?

10:50 And the answer showed up two days later in the form of a Tesla computer because those things are actually very capital efficient as it turns out. And we can run potentially like our model and the full computer that a human would otherwise work at on the Tesla computer for much cheaper than you would on a VM on AWS or Oracle or whatever or even just buying hardware from Nvidia.

11:20 That car computer is actually much more capital efficient and so it enables us to assume that we can deploy much much faster at a much higher scale. And so we’ve adjusted our expectations for that basically. Are you basically able to just bootstrap off of the like car network? So that’s one of the potential solutions basically. Yeah.

11:45 So like okay well we want 1 million VMs. There’s like 4 million Tesla cars in North America alone. And like let’s say 2/3 or half of them have hardware 4. And like somewhere between 70-80% of the time they’re sitting there idle probably charging. We can just potentially pay and they have networking they have cooling they have power. We can just pay owners to lease time off their car and let us run a human emulator digital Optimus on right on it. And they get their lease paid for and we get a full human emulator we can put to work. And that’s something without any buildout requirement. It’s a purely software implementation that’s required.

12:30 The asset is sitting there and you can just go and use it. Yeah. Amazing. What for the human emulators in Macrohard what is the like purpose of that of scaling up you know millions of many humans? I mean the basic concept is very simple right? With Optimus you’re taking any physical task a human can do and allowing a robot to do it automatically at a fraction of the cost at 20 you know with 24/7 uptime.

13:00 We’re doing the same with anything that a human does digitally. So anything where they need to digitally input a keyboard and mouse inputs which is usually what humans do and look at a screen back and make decisions we just emulate what the human is doing directly. So no adoption from any software is required at all. We can deploy in any situation in which a human is in potentially currently.

13:30 What is that actually going to look like for rolling it out? I don’t think we’ve detailed our plans publicly yet specifically on how we’ll roll out. It’ll be slowly at first and then very quickly basically like the difference for us given that infrastructure buildout already has happened or we can go on the Tesla network or we can build out our own data center Tesla computers actually.

14:00 The difference for us from going from 1,000 human emulators to a million is actually not very big. It’s not the biggest part of the challenge. Elon I know one of the things that he does best is he basically just goes from fire to fire on whatever the company is and just kind of like puts it out and unfucks whatever problem exists.

14:25 What has that been like? When have you like seen some problem exist and just had it unfucked very rapidly due to this kind of process? Definitely on the infra buildout this is the biggest. On model side we’ve been like we’ve had hiccups but it’s more or less been smooth but on infra side there’s a lot of very specific operations that each of these basically AS6 these GPUs are built for.

15:00 And when we roll out new products like when we pick up new products from Nvidia or whoever not everything works so in some of the meetings that we had with him early last year. He would hear these and he would make a phone call and the software team would deliver a patch the next day and we would work like side by side until that was resolved. And then we could run a model or a training run on the hardware very very quickly where otherwise it would have taken weeks of back and forth.

15:35 So those kind of blockers are usually very quickly resolved with one phone call or just us bringing it up to him or him just offering like frequently when a meeting is ending or there’s a lull in the conversation he’ll be like okay how can I help how can I make this faster whatever and someone will come up with an answer.

16:00 I know you guys are doing many different products in parallel and I get that it’s kind of like you have to do that but also it’s sometimes in most organizations it’s like very difficult to stay focused on a single thing and like a single objective. How does that kind of work for just executing on multiple different fronts at the same time?

16:30 Very frequently we actually and this is increasing with scale. We don’t have a full picture until like the all hands or we just chat with people what everyone is doing and how far everyone is on these different projects. Like for example on when we did our voice model and our voice deployment we actually had a lot of the work built for extremely low latency packets to be sent to the client.

17:00 It was already built out and it was a matter of flipping the right switches and the right configs basically to cut our latency pretty significantly like 2-3x end to end. This is actually the case a lot of the time is there is a stupid thing that exists somewhere in the software or the hardware and someone has come up with a solution and you find it when you go to look for it in our codebase somewhere.

17:30 Or you ask around and someone’s like oh yeah this XYZ person has done this you should talk to them and they will hook you up. There’s not a lot of time spent syncing up with anyone or asking for permission or waiting for anyone at all. Like the answer is like when you propose someone says a good idea. Like usually you propose something and the answer is either no that’s dumb or why isn’t it done already?

18:00 And then you go and do it and then it’s done. With Elon companies you can kind of just ask for responsibility and then you basically just live by the sword die by the sword and if you get things done then you can just ask for more responsibility and you can keep on doing that or you’re just like out. What’s been your experience like with that?

18:30 Very much so. Yeah like I’ve jumped around a lot of different projects and mostly just because someone asked for my help and I kept helping and then I ended up owning some of the stack or a lot of the stack. And this is the case for everyone like this is just how it is. If you have any particular experience or can iterate on something very quickly within days you own that component.

19:00 Yeah there’s no formal anything I think officially on our HR software I’m on voice and iOS or something and our security software thinks I still work on RX integration which never updated. Yeah no one ever updates this stuff like it’s kind of ridiculous. And has your like journey at the company kind of been you show up there’s not exactly like a clear direction of what you’re going to work on and then you just start working on stuff and then you just kind of like hop from project to project by whoever asks for your help.

19:35 There’s quite a bit of like overlap and flowing. So like after onboarding I’m usually on two or three projects at once. And whichever one is most pressing or I can help the most on ends up taking majority of my time and then that kind of overlaps and flows in like a waterfall way.

20:00 What’s been the journey from like the starting to now? Like what projects have you worked on? Yeah so specifically I started I first worked on like ASRock and our integration there and I worked with our backend team a bit on like reliability and scaling up because we were scaling up a lot at that time. And then after that I took on solo building up our desktop suite. And took that went to internal completion.

20:33 And then I got asked for help on our Imagine rollout and iOS which yeah our iOS team is small for like how many people use it. Like it’s ridiculous. You won’t guess the number. Like five people? Three. It was three and I was the third person at the time when we were rolling that out. It was like it was ridiculous and everyone’s like really really good. Yeah this is the first place where I’ve had to work very hard to keep up really with like the speed and the talent.

21:12 What was the first experience that you had where you thought to yourself like you’re actually being kind of used to your full potential? I think that Imagine rollout was definitely like it was a really good push cuz like we had this 24-hour iteration cycle. You all would get feedback every night on whatever we were doing. And yeah we would push out that night. In the morning we would have all the feedback. We would immediately knock out all the bugs.

21:45 Implement the new stuff that people were asking for. Whatever model had come up with we implemented that too. Like it was a very very fast cycle and it was I think it was the longest like continuous stretch of me being in the office like every day. What was that like at the time? It was like two or three months. Two or three months. Yeah. Yeah. Okay.

22:10 Yeah like there weren’t weekends for a while which was it was good to know that I could do that and I was pretty happy doing that. And after that I got pulled onto Macrohard product which was just one other person at the time. So it was the two of us for a while and I’ve been on that since that project off basically.

22:45 I don’t know how much you like know about this but the like Colossus build and all the ridiculous stuff that the like early xAI team had to do to turn on Colossus and like get power and all the necessary inputs to making that work. And even today I think like it’s just bottlenecks across the entire thing. You just want more you want more like chips and GPUs and all the stuff working and faster. What was that like?

23:15 There’s a lot of war stories and a lot of bets. Want to go into a few? Yeah. So I think Tyler took this bet with Elon like one we were setting up new racks I think of I forget which GPUs we were rolling out at that time. We took a bet. Elon’s like okay you get a Cybertruck tonight if you can get a training run on these GPUs in 24 hours. And we were training that night.

23:45 Did he get the Cybertruck? Yeah he got it. I think it’s yeah I see it from our lunch window cafeteria. Yeah he’s cool. You know what so for power we actually have we have to collaborate very tightly with the like municipal and state power companies because when load goes high on their end we have to shut off and go fully on the like 80 or maybe it’s more than that I think more than that 80 mobile generators we brought in on trucks.

24:20 And go fully on those just so that we don’t like impact power anywhere. And we have to do that like seamlessly without interrupting anyone’s extremely volatile training runs on extremely volatile you know GPUs and hardware which scales up and down by like megawatts in milliseconds. It’s a lot.

24:45 Is that also part of the logic of like basically putting massive battery packs right next to the data centers cuz then you can kind of like go up and down much faster without… Batteries can scale up and down and balance that load a lot faster. Cuz with a generator you’re literally asking a physical thing to speed up or slow down like a spinning physical thing that’s obviously just going to take a certain amount of time.

25:15 The batteries can react to the load much much faster and then yeah it’s like actually from physical standpoint I think there’s the local capacitors the station like data hall side capacitors the batteries and then generators and then the public municipalities although we might have changed that infrastructure at this point things very quickly especially on the cooling side.

25:45 Do you have any other really good like war stories that are just like I don’t know things that shouldn’t have been possible that became possible? So the lease for the land itself was actually technically temporary. It was the fastest way to get the permitting through and actually start building things. I assume that it’ll be permanent at some point but yeah it’s I think a very short-term lease at the moment technically for all the data centers. It’s fastest way to get things done.

26:20 And how do they do that? I think there’s basically a special exception within like the local and state government says okay if you want to just modify this ground temporarily I think it’s like for like carnivals and stuff you can… xAI is actually just a carnival company currently. And so that was the way to get done quickly I mean it was done yeah 122 days.

26:50 For like internal planning I know things are just going to keep on scaling up like crazy and Elon’s talked about energy being the biggest bottleneck and then you know just being able to get chips. How do you guys plan when it’s very difficult to like predict 12 to 24 months in the future exactly what projects you’re going to be working on or what their like resource requirements are going to be.

27:20 We try very hard to work backwards from like what’s the highest leveraged thing we can be doing and then we determine the physical requirements later. So like if we want to get to 10 or hundred billion in revenue by this date what are the highest leverage things we can do like from an economic perspective? How can we actually build systems to do that? And then what does it take on the physical and software side to roll that out and get it done? Just roll backwards the whole way. So we don’t usually start with the physical requirement. That’s usually actually at the end.

28:00 Is there like a SpaceX-esque algorithm for making things happen? As in like the usual delete? Yeah. I mean that’s the case all the time. And we do do the thing where yeah we delete something and then add it back later. What was the like last time that you did that? Today. Today. Today. Yeah.

28:25 So with Macrohard we deploy on a lot of like physical hardware that changes and the testing harness for that is hard. So we try to minimize how much how many special cases are downstream of where it needs to be. And for example like with display scaling we need to be able to support displays that are you know 30 years old as well as the latest like 5K Apple whatever displays and that has to happen on the same stack.

29:00 Turns out not all the systems are happy with that at all times. Like you have to fiddle with the encoders at a certain level. Like video encoders was the specific thing basically we I didn’t know but as it turns out there are limits to the maximum amount of pixels that certain encoders can take. So we have to now have I removed this special case for multiple encoders and turns out we found a problem at plus 5K resolution and so we added that back.

29:35 What are the most interesting things about xAI itself that you think like would be really good stuff to talk about? There’s a lot of characters that work there and also we’re doing hiring in like interesting ways I guess. Like things that I thought would be stupid are okayed and we just do them and we try them and it’s like we’ll do a hackathon and if we get five people in as a result it’s worth it.

30:05 Because just their like expected return on the company’s like revenue or valuation is higher than the cost of running this hackathon for 500 people. Like the overhead value is actually very high which is like funny. We did the math earlier this week. Right now we’re like I think at about $2.5 million per commit to the main repo. And I did five today. So you added like 12 and a half million of value. Light day. Exactly. It was a good day.

30:40 It’s funny things like that. Like the levers are extremely strong. Like you can get a lot done with a lot less effort and time than you used to be able to for sure just because of who you work with the internal tooling that we built up. And my boss.

31:00 What’s like an example of the type of person that like wants to work here? Cuz I know when when you’re talking about it you kind of show up and the first day you’re just like I want to work on the weekends. I want to work on you know during the night all this stuff. Go all in on this. What kind of special characters are working there?

31:25 People are definitely very enthusiastic when they come in. Like very very enthusiastic. Like mission oriented. There’s I guess different types of ambition for sure. Some people want to move up like the leadership ladder and own more in terms of a managerial like how many people report to me sense. Some people want to own huge parts of the technical stack.

31:55 So like right now we’re doing a big rebuild of like our core production APIs. It’s being done by one person with like 20 agents. And they’re very good and they’re capable of doing it and like it’s working well. So you can own huge chunks of the code base no problem. It’s kind of like X where like after the acquisition they like had you know much fewer people but you just like never had a lot of people in the first place so there’s one person like owning a huge part of the product. Absolutely.

32:30 For hiring what’s what unusual practices outside of just hackathons does xAI do? So we’re pushing very hard on Macrohard. Like for two or three weeks I was doing upwards of 20 interviews a week. So that’s like some of them are like quick 15 minutes. Some of them are full 1 hour technicals. So a lot of my time is dedicated towards bringing in new people and a lot of people are very good. So it’s actually very hard to judge them.

33:05 How do you… I have a very specific problem that I have solved. I’m not going to reveal it because then people will use it. But I have solved a very specific computer vision problem a few years ago for one of my startups and I give people half an hour to try to implement the solution. It’s actually very very simple. This deceptively simple solution. People always overthink it.

33:35 And this is something I like to index for on my team especially is like can you not overthink it and come up with a simple solution? It helps a lot because we’re deploying on such a wide variety of like on a wide variety of hardware as a result of the wide variety of customers like literally 30 years 40 years of different hardware different operating systems everything like that. You have to come up with simple solutions or you’re going to have a 10 million line code base next week.

34:10 So this is like very important. And especially now relying more and more on agents and AI and such for writing code. An AI will happily churn out 200 lines when a 10 line solution will do and probably do better. So you have to look for that. Like I want people and I look and actively hire for people who can find the 10 line solution first. We’re totally fine with people using AI to code things. Like you should use that as a force multiplier but for now we’re smarter. We’ll see next year.

34:50 What other like force multipliers do you kind of like look for? I like people who will challenge requirements and challenge me. So often I got this from Chester Zai German he told me this and I thought it was great. He throws in usually an incorrect requirement or question or an impossible like line in his challenges for people when he’s hiring like coding challenges.

35:20 And he expects people to come back and say like hey this is wrong this is not possible you made a mistake and if he doesn’t then he doesn’t hire them same thing for me I picked that up it’s a great idea.

35:40 The pace is insanely fast and like you said you kind of have worked on a number of different things. How do you kind of come up to speed on something as quickly as possible when you’re on a new task or project? It depends on what thing it is. If there’s a lot of code to read yeah read the code by hand. Like go to definition over and over again and you’ll find things out very quickly.

36:10 Actually it’s not that hard. For most things the implementation is like less lines of code than you would otherwise see which is nice. Not all the time but in most cases if it’s something that’s in very active development this is not the case. There’s going to be 20 different versions of it going at the same time and it’s not obvious what is the current path.

36:40 So you just got to talk to people and people are very open. Like this is actually one of the things I was very surprised by pleasantly surprised by when I joined is I thought people would be super smart and stuck up but no people are just super smart and very nice and helpful. Like everyone’s on the same team everyone’s rooting for each other people are willing to like help you out and answer your questions.

37:10 So which is good because we don’t like write a lot of docs. We write things. We do things too fast to write docs really. Actually yeah we’re trying to figure out some systems on my team to like automatically generate docs as we like build stuff. And with Grok which is cool that we have unlimited access to very smart AI because then we can try a bunch of stupid things see if it works.

37:45 Which otherwise you know at a startup would cost you maybe like $100k or a million dollars in credits or whatever. We do it for free. So experimentation like failure you can fail on a lot of things and it a lot more things you otherwise would and as a result more experiments are tried more succeed.

38:15 On the like experimentation side how are you guys kind of like trying to maximize for the number of experiments or like good shots on goal that you can do. There’s often like a time constraint. We will frequently launch multiple experiments especially on the model side at the same time and in some cases it’s not even because of a time constraint necessarily in terms of like I need to try x amount of things in y time.

38:45 It’s in two weeks this prerequisite will be ready either in the hardware or in the training data or something but in the meantime I need to deploy something today what can I do and so you run two three experiments and you find out what you can deploy day and bring in revenue or customer result whatever it is today and then two weeks you switch over like that’s something we do all the time especially at Macrohard.

39:15 Have you seen anything where a timeline should have been much longer on like a project that you were working on and somehow you guys were able to kind of like bring that in by you know weeks or months? All the time. Every time. All the time every time we come away from like an Elon meeting or something internal where someone pushes hard to get something done.

39:45 Or someone external who doesn’t isn’t responsible for the thing asks for a requirement asks for something to be done in what we originally think unreasonable amount of time you know we spend two minutes like thinking about it complaining maybe a bit and then the rest of the time is dedicated to getting it done in that time. Yeah frequently the estimated time to get something done all the time to get something done is based on some set of assumptions.

40:15 And then once you get this timeline that’s like half or one-tenth of what you would have otherwise done. You look at the assumptions say okay proportionally how much is this impacting my timeline? And then you knock it out or you change it and then suddenly you get a 2x improvement in your timeline. You do that a few times you can meet whatever requirement you really want. At a certain point you get to the physical limitations but you’re never there from the start.

40:50 I know for like full self-driving and same thing with the rockets of SpaceX the Elon timeline was significantly longer. Like an Elon time might be a quarter or half of what it actually eventually takes but then it also you know happens four times faster because of the initial timeline. Is it more or less like at xAI because it’s more I mean I guess more on the software side now.

41:20 But even on the data center side things seem to be happening just way way way faster. And they also seem to be happening on like the same timeline as he’s roughly saying. He’s like this is going to happen roughly you know this number of months in the future and then it actually does.

41:45 I think he himself has calibrated his timelines like differently over… Yeah now that he’s deployed a number of extremely like a wide variety of deployed hardware at scale. So I think his own estimates for things are definitely a lot better. And so that’s definitely the case. I think he also updates his timelines faster now too like sometimes daily.

42:15 I think he’s talking with us and figures out what the update on the timeline should be based on various parameters and sometimes they come from him too right especially on the infrastructure side if a deal or we can be put up in a batch for the production of a certain chip well we can save a month or two maybe even more than that depends on what the deployment is specifically.

42:45 And then on the software side it’s the same he always says is like you can always attempt to do something you know in one month that would otherwise take a year and you’ll probably get it done maybe in two. Still a lot faster.

43:05 I remember in the like early days of SpaceX there was this internal I think Elon would say internally like every day that we delay is like 10 million in loss revenue and I have no idea what it would be like for xAI like things are moving so fast. Is there kind of an internal thing in your head of every day that we don’t like push push hard or make something happen we’re losing out on x amount of value that could be created.

43:40 Yeah for Macrohard specifically we do have a few pretty specific revenue targets. I can’t delete the number specifically but like in my head whenever something gets delayed or accelerated I can pretty quickly calculate how much money we just made or lost. Just wild swings. Yeah I mean the numbers are huge just because the expected return is so huge and the timeline is so fast. So a few days is actually proportionately fairly large compared to how much you would otherwise expect the revenue to be.

44:20 Elon’s like famous for making really really big bets pretty quickly. Like what’s the biggest decision that’s been made in a single meeting where like huge huge amounts of capital or time or commitment were done?

44:45 I think one of them was certainly the decision to go with a model that would be at least 1.5 times faster than a human for Macrohard looking like significantly faster than that. 8x maybe maybe more. The like for other human emulator type attempts in the other labs the approach has been let’s do more reasoning and build a bigger model.

45:15 We’ve like that decision put us in totally the opposite track of what everyone else is doing. And everything that we’re doing really is downstream of that like well not everything but pretty much everything. It impacted and it was very early on that this was decided. It was sort of expected also that this is the move especially given the analog to full self-driving.

45:45 No one’s going to wait around 10 minutes for the computer to do something that I could have done in five but if it can be done in 10 seconds well I’d be happy to pay whatever amount of money for that. It’s just obvious really. So normally like us engineers would you know push back and say oh you know here’s the 20 different reasons that it needs to be this way but if a decision is made and you work backwards then life finds a way.

46:20 I remember Elon saying I think it was at the like Y Combinator he was doing a like Q&A with Gary Tan and Gary talked about like AI researchers and he was like no they’re just all AI engineers now. Yeah we did this was someone said that in one of the meetings we did with him talking about recruiting like here was here’s the job descriptions or something like that.

46:50 And like for 10 minutes he just goes on engineers just engineers doesn’t matter good engineers engineers just someone who’s fundamentally a problem solver doesn’t matter if they did like this you know XY thing and this infrastructure or this you know particular architecture or whatever engineers.

47:15 Why is it so important why is that definition so important? It keeps things broad. It means that people can come in to us from like an extremely wide variety of places and this has been the case. I mean there is I think less so in the AI world but I think there’s a lot of SpaceX stories where people came in from strange walks of life that would not have otherwise seemed to be the case and then ended up doing huge things at SpaceX in the engineering world as a result.

47:50 So keeping it broad means that those people can have a path to us and help us accelerate. For you personally what’s the most like fun thing about working there day-to-day? No one tells me no. No one tells me no. Yeah if I like have a good idea I can usually go and implement it that same day and show it off and we’ll see if it makes sense.

48:25 We’ll run whatever eval or show it to a customer or show it to Elon or whoever and we’ll get an answer usually that same day as to whether or not that was the right move. There’s no deliberation. There’s no waiting for any bureaucracy. I like that a lot. I was expecting to sacrifice some amount of this coming from extremely small startups to a larger company.

48:55 Like I guess joining at 100 people I mean to me it was like a 10x leap of anywhere else I’ve been. But I guess relatively to Elon companies is pretty small and it does feel very small. There’s not a lot of overhead in anything.

49:15 Did you have any other like big assumptions going in that proved like completely wrong? I thought there would be more top down. And there’s some but not really that much. Especially because of how many there’s basically only three layers of management. There’s like ICs there’s the co-founders and some of the new managers and then Elon and that’s it.

49:45 And so because there’s so many reports to the managers now nothing really comes from them top down like we’ll usually come up with a solution. They’re okay. Elon’s okay. We’re good if there’s feedback then we update but it’s a lot more bottom up than I expected. Like trying to be designed so that everyone is like building things and there was like fewer managers and more just like builders.

50:15 There’s yeah when I joined I think every manager also wrote code and I think largely today they still do not as much now that some of them have you know 100 plus people reporting to them. But everyone’s an engineer.

50:40 I remember actually on my first week I sat down for dinner and this guy sits next to me and I asked hey you know like what team you on? How you going? How you doing? I just joined. And he tells me oh I’m on sales and like enterprise deals. And I was like oh I don’t want to talk to this guy. He’s a sales guy.

51:05 And then he starts telling me about this model he’s training on the he’s an engineer too. The sales team are all engineers. Everyone is an engineer. I think at the time there was probably less than like eight people who were not engineers at the company in some capacity and even then like yeah it was really cool. Everyone contributes to the machine.

51:35 Is it a little bit more like you have a single person working on some project and they just you know if you’re an engineer and you’re working on the thing you can have a much closer relationship to the customer and like understanding their problem and then rapidly like implementing solutions and stuff. Yeah. Yeah the less layers you have the less information is lost.

52:00 There’s less compression basically. Because you have to communicate less times and language is lossy compared to what’s going on in your brain. So if you have to go from customer’s brain to words to you know salesperson’s brain to words to manager every layer you’re losing a huge amount of information. Yeah.

52:30 And so if you can cut as many layers as possible then you’ve only got one compression step of the customer telling you what to do or what they want or what their experience is or whatever and you as the engineer can solve it directly.

52:55 Is there anything like specific that you’ve never heard of or seen at any other company that xAI does where that allows things to just happen way faster? The fuzziness definitely between teams and what everyone is responsible for is definitely not what I expected and I don’t think exists nearly as much in any large company or even remotely similarly sized company.

53:23 Like if I need to fix something on our VM infrastructure I will do it. I will show it to the guy who owns that and they will be like okay and it’s merged immediately and deployed. Like there’s not a lot of strict regiment. Like everyone is allowed to update everything and there’s some checks for dangerous things but largely you’re trusted to do the right thing and do it right. Which is really cool.

53:55 I remember when Elon was still like really working on DOGE there was at one point I think they deleted like Ebola prevention or something and then and then they rapidly like reput that back in. What things have been like deleted because of this rapid process of trying to figure out you know what doesn’t need to be done. And then like reimplemented.

54:22 There’s very rarely anything like irreversibly destructive. I’m actually not really aware of anything where something was irreversibly destroyed but like I said yeah frequently something will be deleted or removed or something like that and someone will be like hey I needed that you know an hour or two and then you go and roll back.

54:50 Or you know sometimes it can be months where you know someone’s building this project and they’re depending on I don’t know some piece of infrastructure something like that and turns out we rebuilt that thing three times by the time you go and deploy and need it and so you update and go that way.

55:15 Do you think it’s helpful to have like so few people working there on the engineering team? Yeah definitely. The more people you have doing a like I definitely say like a job for one person done by two will take twice as long. And it applies for every scale I think.

55:45 And especially now that you don’t need to physically write as much code as you did previously. You can be more of the decision maker and the architect. Everyone can be an architect. You just don’t need as many hands. So one brain can do a lot more.

56:10 You tried starting multiple companies and you were doing a whole bunch of different projects prior to this. What about working here and like what about the mission the culture resonated? Why did you why did you decide to work on this?

56:35 I’ve definitely always been very Elon-pilled like I always he’s been a big personal hero of mine. Especially growing up you know seen the Falcon landings the first ones and I went out to launch five of Starship which was so worth it. It was the first one they caught. It was really cool. It was definitely the coolest thing I’ve ever seen.

56:44 So being part of anything even remotely related to that sounds awesome to me. Is there a reason why you chose like this company instead of SpaceX or Tesla? Yeah I’m definitely like an entrepreneur by heart and xAI is definitely the smallest company the newest of all of them.

57:15 I think my assumption is and this is largely proven true I think is where you can have the most leverage and change as an individual person because proportionately you’re a much larger percentage of the company than you would be at these other companies. Not to say that like they’re not doing cool things or everyone’s not as important but the yeah just the proportional change kind of to decision is like way higher.

57:50 Not even to decision but to implementation to seeing the results like it’s very quick. And I guess another assumption that I thought would be the case but it’s wrong that I had was that I would be faster on my own you know to build XYZ thing or try XYZ experiment. I’m actually usually faster at xAI just because I have a groundwork and a team who’s probably already done a lot of the steps that I would otherwise have to do by hand. And there’s yeah no one saying no.

58:25 You mentioned like it’s kind of a fuzzy blurred line between people working on different things. Has there been any ability for you to kind of go to other people in the organization and just ask for help? All the time. What does that look like?

58:50 I walk up to their desk and I say hey here’s my question. What are you working on right now? Can I support any of that? And can you help me with this? That’s it. Everyone’s in the same building. So yeah actually funny enough we started testing some of our human emulators internally within the company as employees.

59:21 And in some cases like we didn’t really tell anyone about this. And so in some cases there’ll be someone like someone doing some work and someone is like hey can you help me with this thing? Or like can you do this thing? And the virtual employee is like yeah sure. Come to this desk. Come to my desk. And they go there and there’s nothing there.

59:50 It’s like the Claw situation where it’s like we’re going to show up and I think when they first rolled out their vending machine it was like I’m going to see you tomorrow. And then it you know it’s obviously like a piece of code. Yeah exactly.

1:00:08 And so multiple times I’ve gotten a ping saying like hey this guy on the org chart reports to you. Is he like not in today or something? It’s just like an emulation. And it’s an AI. It’s a virtual employee. But yeah generally we all expect to be like in the same building and reachable to each other. So it goes like always and I can ask for help. People ask me for help all the time.

1:00:45 What have been the biggest like blunders that have happened? So with the human emulators with the customers that we’re working with when we try to understand like we always try to understand what the job that they’re doing is and all the facets of it.

1:01:10 Frequently we’ll you know talk to them we’ll interview we’ll even watch them. Well actually we’ll do the watching at the last step. So we’ll talk to them we’ll interview they’ll give you either write up or we’ll just meet up with them and write notes as to how they do their job.

1:01:35 And then like a week later we’ll look at the mistakes that the virtual employee is making and realize like well it’s always making mistakes in these places in these specific cases. What’s going on? And we go watch the human doing the same thing and there’s like 20 different steps that are missing that they just totally left out.

1:02:00 And we go to them and they’re like oh yeah we do that like I forgot to tell you. My bad. It happens all the time. A lot of things people like I guess assume automatically. It’s all taken for granted in their head. Totally on autopilot. The same way that you can like be driving for an hour and not remember a single second of it and not be paying attention can be totally in your own world.

1:02:30 This is the same for every thing that a human does repeatedly. And that’s what we’re trying to solve basically is all the dumb stuff that humans do repetitively right now that they don’t need to. Trying to solve for that case. Exactly.

1:02:53 How do you decide like which thing to go after? What’s like in your head when you’re thinking about that what are the biggest things outside of driving that humans do all the time that they just don’t need to do? Anything repetitive on a computer.

1:03:15 So like customer support is a big one where it’s just taking in free form input from arbitrary customer in arbitrary form factor and translating that into a standard workflow that is purpose-built for like an AI to take care of that so that human could go and do something more creative and use their brain like in a more effective way.

1:03:45 Totally the same like it’s a total parallel to what happened in the coding world like okay I don’t need to write the same implementation 20 different times anymore I can describe it in like three words and it’s done. It’s a huge compression step and this is the same thing basically but for arbitrary digital workflows.

1:04:15 On the human emulator side you run into this problem of humans not existing and then like someone says come to my desk and the person doesn’t exist. Is there any other thing that’s been kind of surprising on rolling that out internally?

1:04:40 Surprisingly we’ve been able to generalize to more cases than we thought. We test and we’re pleasantly surprised a lot of times. Like just today we gave Elon a few cases where we did not train on this task at all but it did it flawlessly like perfectly like way better than we would have expected.

1:05:10 Yeah the generalization is better than we expected for sure. And we’re still at a very early stage so it’s only going to get better. And it’s again the same parallels to full self-driving where there’s stuff not in the training data that the car does react to perfectly due to generalization of an otherwise very very small model like it’s a matter of basically weight efficiency.

1:05:45 For the Elon meetings that you’ve been in like what does that actually look like? They’re pretty simple honestly. And I’ve been lucky that most of the ones I’ve been in have gone mostly pretty smoothly.

1:06:05 What does smooth look like? Smooth is limited feedback or thumbs up. That means like okay you’re going in the right direction. Keep going. I’ll hear updates next week. Or whatever it is. If there’s feedback or a total reversal of direction as a request then we messed up somewhere. Then the question is where?

1:06:35 So that’s usually we don’t even have time to identify that. That’s something you just build up implicitly as a muscle as you go on. And sometimes assumptions also change based on new information. That always happens in every case. So when it comes from the top down it’s a little chaotic but…

1:07:00 I know with like SpaceX the cost for parts and building things is super super important. Cuz you know everything basically costs a shit ton of money and time to do right? For this sort of thing I imagine it’s a little bit less focused on you know he’s not like necessarily drilling down on do you understand every part of every process. What does it look like when he’s kind of giving feedback?

1:07:35 Usually it’s either at a very high level or at a very low level. It’s not really often in between. So either on the high level it’s like a product direction or customer sense you know focus on this segment exclusively or don’t do this thing at all or whatever.

1:08:00 And then at a low level especially when it comes to compute efficiency or latency he’ll always have a specific suggestion or let’s try this. And he’s open to being like proven wrong but it has to be proof. It has to be like let’s try it and see what the results are. It can’t be just someone’s opinion. There has to be an experiment done.

1:08:30 Which has led to some surprising results sometimes and we go with it. What have been those? So the compute efficiency of going with the small model has led to well a lot of improvements that we wouldn’t have otherwise thought. Some of them are secondary some of them are primary. The obvious ones are well obvious being able to go much much faster to human.

1:08:48 But also as a result and Tesla found this too with full self-driving going with the smaller model they’re able to iterate much much faster. So not only does the model react to situations faster and can be more I guess tolerant of time frames you can also just deploy iterations much faster if it was 4 weeks before maybe it’s one week now.

1:09:20 So as like that actually goes back to the experimentation why we can have 20 different ones going in parallel was a result of that particular decision early on in the chain. And was the initial idea like go just do big large models and then… Sort of we definitely wanted to go faster than everyone else but the question of how much faster was well the answer to that was amplified basically multiply by a lot.

1:09:55 There’s this like a lot of bias and stuff in Wikipedia and Elon has been like focused on kind of creating an alternate version that’s just kind of like you know more truthful in effect. How do you go about basically cleaning up the internet in that way to figure out what is truth?

1:10:25 It’s a really hard problem. Yeah it’s very hard especially because the internet is not usually the ground truth for whatever thing it is. So wherever we can we try to drill down to the fundamentals which is very hard like I don’t know what is the fundamentals like in physics of the constitution that’s not really a question I think I can answer or anyone could really faithfully answer very well.

1:10:55 But you try to do something like that drill down as close as you can and then build up from that which is hard too because there’s not actually a big body of like writing that does that. One of the few probably examples is like James Burke with his Connections series where he’ll take two totally seemingly unrelated concepts and then connect them through physics and inventions. It’s really cool and we’re trying to do the same but it’s fairly novel.

1:11:30 How do you find better data? Data is not the only thing that goes into the results. Yeah like how you actually train on that data and I know it’s a pretty broad term but it is true like how you actually evaluate against that data and train against it and your different methods for updating the weights do matter a lot.

1:12:00 You can try to faithfully recreate the input or the output given any arbitrary input and well you can create basically a horrible copy paste mechanism if you want which is a classic problem in ML. There’s a bit of an art to it to avoid that problem but we’re a few steps removed from that at this point we’re not measuring the fitness to any particular data set. At this point we’re trying to measure to an arbitrary output.

1:12:35 So it matters a lot how you construct your evals. Which is really hard for truthfulness because then you need to know the truth which isn’t always well I mean that’s really the problem we’re trying to solve right? So it’s kind of chicken egg. Yeah there’s like a lot of different approaches and a bunch of smart people working on it. If yeah if anyone has suggestions please send them through. There’s like a lot of different ways to look at it.

1:13:10 There’s been like moments in time where I’ve seen Elon on X and someone has said like this is obviously not right and it’s like some Grok output and he’s like we’re going to fix this and then you know 12 hours later 24 hours later he’s like all right it’s fixed. When that happens like what happens internally.

1:13:40 He shows us what went wrong and then quickly whoever is awake at the time will start up a thread to go and solve it. Usually individually pull in a few others if need be and then give a postmortem on what happened and everyone will understand then what went wrong and how to avoid it in the future. Ideally yeah the like generally making mistakes once is okay but making the same mistake twice is big problem.

1:14:15 Throughout SpaceX’s history there’s been a number of and same thing with Tesla there’s been a bunch of these like surges where randomly Elon will like come in at midnight and say you know like everyone that can come in like send out a company-wide email and say like come in we need to be working. That sort of thing. Has there any been anything like that?

1:14:45 It’s more for the big models that that happens more than anything. For Macrohard specifically we’ve been operating in a war room for 4 months. So we’ve kind of always been on that on that push.

1:15:05 Do you guys have like a sign on the door that says war room? Yeah. Amazing. Actually well yeah we outgrew the original war room. And so we moved everything out and I’m told like Elon walks in to the war room and it’s totally empty and he’s like where is everyone? What?

1:15:35 And he walks over to where we are now which is just the gym which we cleared out and put everyone in now and then conducts his impromptu questions of what’s going on. That was a long night.

1:15:55 What is it like on one of those nights where a lot of things kind of get shaken up and moved forward or like there’s one of these surges. What does that feel like?

1:16:15 I think actually I saw this from one of the co-founders of xAI posted this recently. Igor who was great to work with. I liked him a lot. It was actually really cool to work with him. Side tangent because his work on Starcraft AI way back like I guess 10 years ago now almost was one of the first like cool ML work that I tried to replicate myself in high school. Which was very hard.

1:16:50 It was really cool. So it was really cool to work with him. Like I totally never thought I would get the chance to. But anyway I saw him post this thing a few days ago where he’s like okay there are some you know months where only a few days go by and then there’s some nights where months happen.

1:17:20 And that was like one of them for sure. Months might be an exaggeration. I think we would have gotten to the technical result we would have in a few weeks anyway but doing it in one night was a huge push and it was a long night.

1:17:45 Has there been any moments where the company just didn’t leave the office for like 5 days or like a week? Yeah the surges for the models usually results in a lot of people staying in overnight. And you mentioned there’s like five or six pods that people can sleep in and they like toggle out.

1:18:10 Yeah. Yeah there’s some sleeping pods and we have some bunk beds now too. Which are less nice but they exist. And then when the tent picture came out everyone kept sending that to me and I was like honestly yeah we have tents but I’ve never seen that many out at once.

1:18:40 I know you worked on a bunch of different projects as a kid. And I think I don’t know if this was their first one but it was like fidget spinners and making fidget spinners. I don’t think it was in your garage but maybe it was like in your room. Yeah.

1:19:00 What kind of stuff like that tinkering mindset? How much of that have you kind of taken to this? Quite a bit. Quite a bit. Yeah. So I learned programming when I was quite young. My dad got me a book when I was like 11 and I liked it a lot.

1:19:25 Well I liked it a bit but I really started to like it when I realized you can make money from it. And so I met some people online who were basically writing scripts for games as hacks and would sell them online for small amounts of money. But you know making a couple hundred bucks online was huge for me.

1:19:55 I think the first time that you like have someone give you money it’s the strangest feeling. Crazy. Yeah I remember having to ask my dad for like a PayPal like custody account or whatever and getting the money in and it was like the coolest thing of all time ever for me. Yeah it was really big.

1:20:25 And so I did that for like a couple months and saved up enough money to at the time I was really interested in additive manufacturing like 3D printers. RepRap was the big thing then. So that was kind of what kicked off the modern 3D printing revolution.

1:20:50 RepRap was like this built your own right? Yeah you had to. That was the only option. RepRap is literally just a bunch of university students basically who said like let’s see if we can build a machine that can build almost all the components for itself. Which was why it was called RepRap.

1:21:20 And they basically built in a variety of universities these rooms where you start with one printer and then it prints the parts for the next printer and you go all the way up and you scale up and there’s lots of problems as it turns out and that’s what they were solving and eventually kicked off like the modern 3D printing revolution.

1:21:50 But I was very obsessed with it and so I took one of their parts list and bought everything from Alibaba and a month later things came in and I assembled it all one night which went poorly actually when I was unbundling the copper cable for the power supply.

1:22:15 Which was a very sketchy power supply and did catch fire in the end. The all the copper windings came like loose and frayed everywhere and one went like 2 inches into my thumb.

1:22:40 Did you just can’t your thumb just doesn’t work or did you go to the hospital or something? No. So it was a school night and it was like 3:00 a.m. cuz I wasn’t very good at building things at 13. And I spent like an hour in the bathroom trying to pull it out with tweezers and it just wasn’t it was like it was bad.

1:23:05 So I just cut it off and I was like eh. And so bit by bit over the next few weeks it came out and I would snip it off in the mornings. It was fun. But I got the printer assembled.

1:23:30 And around that time yeah the fidget spinner craze was going off. So I bought 1,000 skateboard bearings from China and basically set up a little factory in my bedroom where every two hours at night I would wake up and I would clear the print bed start a new print of fidget spinners and I would sell them online.

1:24:00 And then before school I had a little assembly line in my garage where I would put in the bearings spray paint dry and then run around to all the other bus stops of the other schools sell them to my distributors which were just other kids of other schools sell all day at school come back collect from my distributors and then sell online ship.

1:24:30 Built a little healthy business and after 2 months they ended up getting shut down by the county. Their official quoted reason was that the companies that sell the school food have technically an exclusive license to sell anything in school property. But I think they just didn’t like that I was distracting everyone and making money doing it. But it taught me a good like healthy disrespect for authority I think.

1:25:05 That has kind of been like a constant theme. What does that actually how is that materialized in your life with like the healthy disrespect for authority like what and you even mentioned institutions like you don’t necessarily trust institutions. How did you kind of come to that and what does what does that look like?

1:25:35 I’ve always known from very young like I want an unconventional outcome. And so going through a conventional path would pretty much necessarily not lead you to an unconventional outcome. So I grew opposed to any form of convention and institutions necessarily enforce convention.

1:26:05 I think creativity and interesting outcomes come mostly from free-spirited individuals in almost every case if not all of them. I guess it’s a bit of a like high-minded way of saying it but yeah like individuals are the most creative you can get. And so staying true to that is the way to go.

1:26:35 I do love John Carlson’s idea of like everything is so hard to build and so hard to make especially you know put into the real world that if you look around it’s basically like the world is just filled with some you know people’s passion projects.

1:27:00 Yeah. It’s a total miracle. There’s a story behind every little thing. Way more than you would think. I remember reading about the I think it was YKK zippers. Apparently every good zipper like there’s two or three companies in the world that make zippers which are actually pretty little miracles.

1:27:30 They’re very cheap but also mechanically complex like relatively complicated for how much you pay for them. And there’s only a few companies that are capable of building or have set up to build them. And it’s yeah basically this one Japanese guy’s passion project over 40 years to figure out how to do this properly.

1:28:00 And this is the case for pretty much everything. Anything very specific and at scale is probably only done by a few companies or a few people in the world. So yeah I mean you hear about it every so often right? Like some company in Germany arbitrary company in Germany shuts down and Volkswagen has to halt all their lines or something like that. Happens all the time.

1:28:30 Right before we met you had made a liquid fuel I think rocket engine. It was like a very small thing. I saw it upstairs. But you said we were talking before this that you did it in like 24 hours just on a whim. How did that happen?

1:28:55 Yeah. So it was a project over like roughly four weeks. And I started by literally just buying a bunch of textbooks and trying to figure out like what are the design principles behind a rocket engine like how do I design it? There’s not like you it’s totally different from learning software where you can just go on GitHub and download people’s code and modify it.

1:29:25 There’s no file for a rocket engine. You have to learn how to like what are the material properties what’s the chemical properties how do you actually machine it how do you design the parameters and know what to expect from in terms of thrust output and how do you not over pressure the engine and all these kinds of things. How did you design the injector which is the injector was very hard. That was probably 50% of the time.

1:30:00 Was that the hardest thing? Yeah the injector was very hard and it was like the biggest flaw in the end. So yeah I spent like 3-4 weeks doing this and expedited a bunch of parts from China like CNC and all that stuff.

1:30:25 And it was right before Thanksgiving. I was going to go fly back to the east coast and visit my family and I was like okay either I fire it like build it and fire it tonight. It was all just a bunch of parts at that time. Or I do this in two weeks and I’m like I’m not going to do this in two weeks. I’m going to do this right now.

1:30:55 So I drank a lot of coffee in the morning and then spend the whole day like hacking away at aluminum extrusions and built out the test frame and then the engine itself and let it off that night. Yeah which had a lot of we’ll say concessions made to make it happen that night.

1:31:25 I did find it absolutely hilarious that you like you said were you like a couple feet away? Yeah. So I designed it like I wasn’t stupid. I designed it so that I could remotely fire it but I didn’t the power supply hadn’t come in yet to remotely power the computer that was on board.

1:31:50 So I had to use a USB cable from my laptop to power the onboard computer. And I didn’t have a long enough USB cable. The longest one I had was like 6 foot. So I had to stand right next to it and light it up. And I was like there’s like a 30% chance that this thing explodes or launches fire everywhere.

1:32:20 And actually I don’t know if it shows in the video. I think it does show in the video but my jacket did catch fire because I wasn’t that great at designing the injector and it did create a lot of over pressure events which meant there was a lot of basically unburnt fuel spewing out which was ethanol. And so that’s liquid and just landed some landed on my jacket and caught fire.