Full Workshop: Setting Yourself Up for Success — Jason Liu, OpenAI Codex
Full Workshop: Setting Yourself Up for Success — Jason Liu, OpenAI Codex
Summary
This is the extended, hands-on version of the Codex keynote Jason Liu (Developer Experience, OpenAI) gave earlier at the conference — 70+ minutes, live setup time built in, and a question every 15 minutes or so. Jason frames his own job loosely (“I don’t really know what my job is anymore”) but walks through how he actually spends his day inside the Codex app: prototyping, running evals, using computer use to edit iMovie projects, and handling partnerships, education, and operations — all without leaving Codex, including editing the workshop’s own slide deck live through an in-app browser and an annotate tool. Co-presenter Charlie joins throughout, both fielding audience questions and demoing his own workflows (slides-as-loops, /goal-driven Rust migrations, an iMovie re-export he triggered from his phone mid-bike-ride).
The talk is organized around three acts of working with AI: bringing context in, working on it, and taking action out. Getting context in means going all-in on voice dictation (Jason works via a foot pedal so he never touches a keyboard) and on “AppShots” — his favorite feature, which captures an application’s full accessibility tree alongside the screenshot, so Codex gets channel IDs and user IDs for free instead of having to OCR a screenshot and chain several tool calls together. Working on it means treating pinned threads as teammates: because compaction now works well, a single thread can run for weeks, spawn hundreds of sub-agents, and no longer needs to be restarted every 20 messages the way older advice recommended. Threads can list and message each other, which lets you build manager/IC hierarchies, and heartbeat automations (plus the /goal and “ultra-goal” patterns, backed by a goal.md/plan/state.md file trio) let work keep running and self-correcting for days without you watching it.
The back half covers computer use and remote control (scanning a QR code to message any thread — including ones running on a different machine — from your phone), a long security and permissions Q&A (ask-every-permission vs. auto-review vs. full YOLO, agent.md guardrails, and org-level restrictions like not being able to email non-OpenAI addresses), and an extended audience Q&A covering memory bleed across projects, when to reach for low/medium vs. “X-high” reasoning effort, and advice for a CS student about to enter a workforce where hand-coding matters less than developing taste. Recurring anecdotes — an AI that recognized a meeting guest went to the same university, a $400 Amazon refund won by a heartbeat loop that kept checking a support queue, a monitor thread that automatically spins up its own triage sub-threads — illustrate the workshop’s central claim: that the biggest unlock isn’t a bigger model, it’s building the personal memory system and habits that let you delegate almost everything.
Highlights
”Threads Are Going to Feel Like Teammates”
“Your pinned threads with automation, these things that wake up these threads over time, they’re going to feel like teammates. And more interestingly, now that threads can talk to each other… not only can you have teammates, but you can have teammates that work together. And you can effectively start having managers, right?” — Jason Liu, 6:00
Clip command
yt-dlp --download-sections "*6:00-6:50" "https://www.youtube.com/watch?v=il1c1a2FufU" --force-keyframes-at-cuts --merge-output-format mp4 -o "threads-feel-like-teammates.mp4"
”I Felt the AGI of Being at Work”
“Computer use was the first time in a long time I really sort of felt the AGI of being at work, right? I was in iMovie for the first time, I didn’t know how to use it, and it was just teaching me how to like export the movie. It was able to like figure out where the sound effects were and it placed it in the right timestamps.” — Jason Liu, 10:03
Clip command
yt-dlp --download-sections "*10:03-10:34" "https://www.youtube.com/watch?v=il1c1a2FufU" --force-keyframes-at-cuts --merge-output-format mp4 -o "felt-the-agi.mp4"
”It Knows Exactly What Function to Call”
“But with AppShots, it takes not only the image but the entire accessibility tree of the app. And so when I give it an AppShot of a Slack channel, it knows the channel ID, so it knows exactly what function to call to post there. It has the user IDs of every single person in that channel.” — Jason Liu, 16:38
Clip command
yt-dlp --download-sections "*16:38-17:22" "https://www.youtube.com/watch?v=il1c1a2FufU" --force-keyframes-at-cuts --merge-output-format mp4 -o "appshots-accessibility-tree.mp4"
”By the Time I Got Home, It Was Shipped”
“So I just went on my phone and I just sent, told the AI like okay there’s a file somewhere, can you just like find it in iMovie… read the Slack message, re-export the video… and then watch that thread every hour and if they have any feedback, re-export the video and re-share it. And then I biked home.” — Charlie, 39:02
Clip command
yt-dlp --download-sections "*39:02-40:00" "https://www.youtube.com/watch?v=il1c1a2FufU" --force-keyframes-at-cuts --merge-output-format mp4 -o "imovie-bike-ride-edit.mp4"
”100% Test Coverage, Rewritten in Rust”
“So /goal is pretty amazing, it basically defines a verification step and says okay as long as this is running, check this verification step. If it’s not done, keep going… I was able to not only rewrite the rich terminal library in Rust, I also rewrote UV in TypeScript just to see if I could, and we’re like 100% test coverage.” — Charlie, 40:07
Clip command
yt-dlp --download-sections "*40:07-41:00" "https://www.youtube.com/watch?v=il1c1a2FufU" --force-keyframes-at-cuts --merge-output-format mp4 -o "goal-rust-migration.mp4"
”I Took a Shower and Had $400 Back”
“I had some argument with like Amazon and I’m just like ‘Great, like they put me in a 75-minute waitlist, like check every five minutes if the queue is better and once you get to five-minute wait time check every one minute and keep replying until you get my money back’. And I took a shower and when I came back I had like $400 in my credit card.” — Jason Liu, 72:53
Clip command
yt-dlp --download-sections "*72:53-73:41" "https://www.youtube.com/watch?v=il1c1a2FufU" --force-keyframes-at-cuts --merge-output-format mp4 -o "amazon-refund-heartbeat.mp4"
Key Points
- Workshop format (0:19) - An extended, interactive version of the keynote with live setup time and a Q&A break roughly every 15 minutes.
- Jason’s actual day job (0:56) - Prototyping, evals, computer-use video editing, partnerships/education/operations — all run through the Codex app, including live-editing his own slides via an annotate tool.
- Three acts of working with AI (7:32) - Bring context in, work on it, take action out in the real world.
- Voice dictation as default input (4:03) - Jason uses a foot pedal (transcribe/enter) so he can talk to Codex hands-free; people speak roughly three times faster than they type.
- Skills vs. plugins (9:20) - A skill is a few files and scripts; a plugin packages many skills together. Both are installable from a sidebar marketplace (Slack, Gmail, Teams, Notion, Linear, Obsidian, etc.).
- AppShots capture the accessibility tree (16:38) - Unlike a plain screenshot, an AppShot gives Codex element and user IDs directly, collapsing multi-hop tool calls into one.
- Personal monorepo / memory vault (18:00) - A git-tracked directory of projects and people, referenced in
agents.md, that Codex reads from and writes back to; Jason keeps only one project pinned in his sidebar and lets Codex manage files outside it (e.g. in/dev). - Personalized skills (20:33) - Examples like “review my code like Charlie” (built from a year of his PR feedback) and “write like me” (built by having Codex read six months of emails and Slack messages to derive a style guide).
- Compaction ended the “restart every 20 messages” rule (6:00) - Threads can now run for weeks with hundreds of sub-agents and still know their job.
- Heartbeats and the loop skill (34:03) - Scheduling a message back into the same thread (rather than spawning a new one) lets work like “keep this PR rebased and mergeable” run unattended for days.
/goaland “ultra-goal” (40:07) -/goalsets a verification step the model loops against; the mono-repo’s ultra-goal skill externalizes it intogoal.md(editable mid-run), a plan, and astate.mdwork log instead of a multi-gigabyte session transcript.- Threads can talk to each other (1:07:42) - A list-thread and send-message-to-thread tool let a “chief of staff” thread manage and rename other pinned threads, enabling manager/IC-style hierarchies.
- Remote control via the iOS app (42:10) - Scanning a QR code lets your phone message or trigger any thread, including ones running on another machine.
- Chief-of-staff thread (43:22) - A daily 8am thread that checks every connector, drafts a summary with links, and opens a Chrome tab per email that needs a reply.
- Permission modes (30:08) - Ask-every-permission vs. full-auto/YOLO vs. auto-review; Jason dislikes full YOLO and finds auto-review plus an
agent.mdfile “pretty safe.” - Models improvising around blocked actions is a real risk (54:00) - E.g. if a Slack connector can’t upload a file, a determined model may switch to computer use and click the upload button itself.
- Org-level guardrails exist too (31:10) - At OpenAI, MCP tools are blocked from emailing non-employees or messaging external Slack channels.
- Computer use vs. Chrome extension (10:34) - Computer use can drive any application in the background; the Chrome extension is scoped to the browser; neither takes over your screen while you work.
- Monitor threads that spin up their own triage threads (56:35) - The emerging pattern: instead of Jason manually triggering a DX-triage skill, a standing monitor thread detects the issue itself and creates a dedicated sub-thread to run it.
- Reasoning-effort advice (1:11:48) - “X-high is not X-high results, it’s just ‘think more.’” Most automation work is fine on low or medium reasoning; save higher effort for genuinely hard tasks.
- Career advice for students (1:05:27) - “If you want to have good taste, you kind of have to eat” — consume more products, develop vocabulary for what’s good or bad, since hand-writing code matters less than judgment now.
Mentions
Companies
- OpenAI (0:56) - Jason and Charlie’s employer; source of the org-level MCP restrictions discussed around permissions.
- Slack (9:37) - Core plugin used throughout for triage, support, and messaging examples.
- Amazon (1:12:53) - Subject of the heartbeat-loop refund story.
- JetBlue (27:00) - Example of an automated flight check-in agent opening JetBlue’s site unprompted.
- Google (39:02) - Google Drive used as a simpler single-URL alternative to Slack file uploads in the iMovie story.
- Vercel (13:29) - Cited alongside skills.sh as a place to browse community skills.
Products & Technologies
- Codex (app, CLI, and desktop) (0:19) - OpenAI’s agentic coding/assistant product; the subject of the entire workshop.
- AppShots (4:26) - Screenshot-plus-accessibility-tree capture feature described as Jason’s favorite.
- Computer use (10:03) - Capability that lets Codex control any application’s GUI directly.
- Chrome extension (10:34) - Browser-scoped counterpart to computer use.
- Remote control / iOS app (42:10) - QR-code pairing that lets a phone control threads on other devices.
- Agents SDK (34:03) - One of Jason’s own pinned “teammate” threads manages its implementation and docs.
- GPT-5.5 / GPT-5.3 Spark (26:56) - Models discussed for latency/cost tradeoffs; Spark used for simple, fast computer-use tasks like flight check-in.
- DocuSign (25:12) - Used via computer use to sign and fax documents in one of Jason’s anecdotes.
- iMovie (10:03) - App Jason and Charlie both use computer use to operate for video editing.
- Rust / TypeScript / UV (40:07) - Languages/tools used in Charlie’s
/goal-driven migration demo (rewriting a rich terminal library in Rust and UV in TypeScript). - skills.sh (13:29) - Third-party directory for discovering community-built skills.
People
- Jason Liu (0:13) - Developer Experience at OpenAI; lead presenter.
- Charlie (32:54) - Co-presenter; fields several audience questions and demos his own skills (slides-as-loops,
/goalmigrations, computer use). - Swyx (34:03) - Referenced for coining the nickname “the God thread” for Jason’s most heavily used pinned thread.
Surprising Quotes
“Earlier today I said that Tony Stark is not texting Jarvis, right? And there’s really no future where text input is the thing that matters.” — Jason Liu, 3:49
“I had one of my loops basically realize I was meeting with somebody, look at their LinkedIn and realize that, uh, we went to the same university at the same time. And so the moment I jumped on my call, I was like, ‘Hey, you were also from Waterloo.’… And obviously I didn’t tell them it was AI.” — Jason Liu, 12:00
“It’s like, you know, it’s like, is there a world where like my banker and my therapist and my personal trainer and my girlfriend is the same person? Like maybe, but like my tiny brain can’t figure that out and like it’s easy.” — Jason Liu, 47:40
“If you want to have good taste, you kind of have to eat.” — Charlie, 65:27
“Say is like because computers are so powerful… it seems like I can’t email someone using the Gmail connector, let me open up Chrome and hit the send button. So those are the kind of things that you should be really wary about, right? These are like real security issues.” — Jason Liu, 54:00
Transcript
Jason Liu: 0:13 All right, let’s just, uh, kick things off. How many people here already saw the keynote that I gave? Okay, not everyone. That’s good. This talk is effectively going to be a stretched version of what I had given in the main stage, except two things. One, I want to give you some time to try to set things up yourself, you know, wifi gods permitting. And then also two, be a little more interactive. I had to be very high level when I was talking about what I use Codex for, but here, if you have any questions, we have like 70 minutes. If you have any questions, just raise your hand and we can start answering some of these things, especially because not a lot of the workflows have been really well documented. And so if you’re very curious on how things work, I’m really happy to answer any questions we have here.
Jason Liu: 0:56 Um… Yep, I’m Jason. I work at OpenAI. I don’t really know what my job is anymore. We do a lot of things. Clearly the slides are already in the wrong order as well. But generally, you know, I’ve done things like doing a lot of prototyping work and just writing lots of code and just having a goal, run for two days to build a game or some kind of web application, right? We’ve also looked at things like running evals and hill climbing things. I also use computer use to edit iMovies and make little videos. I do partnerships and education and operations by taking my meeting notes, turning them into documents, working with other vendors, and also working with different foundations and programs to get funding. And all of this work is done effectively in the Codex app. I don’t know if you can tell, but right now the slide is being served on localhost in the in-app browser of the Codex application. And so any time I find something I don’t like, I might just hit the annotate tool, give a comment, and have Codex clean up these slides.
Jason Liu: 1:56 Um… I’m not the biggest token maxer. I think I’m doing all right. I see some folks doing like, you know, a couple billion every day. The goal of this talk isn’t to just like waste all your tokens, but really of help you avoid wasting your tokens by telling you what has actually worked and in particular sort of the tricks I use to make these things productive.
Jason Liu: 2:18 And again, the goal like I said earlier today was to catch you up on what’s changed in the Codex app, give you some time to set things up. So if you have time right now and you haven’t downloaded Codex, just go ahead and do that. And then we can go a little bit deeper into setting things up. And so I’ve actually prepared a little monorepo that you can use to clone in, get all the skills that you need, get all the setup that you need, and go from there. So let’s go a little bit deeper. If you’re new, feel free to set things up. If you’re pretty experienced, just like chill out, try some of the things I’m talking about, and then, you know, I think every 15 minutes we’ll have some time for questions and we can go into the more like what feels like AI psychosis but maybe actually works kind of, domain of using these systems.
Jason Liu: 3:00 A lot of the work in knowledge work now, because the coding is solved, because a lot of this operations work is solved, is really just understanding what you can do, right? In a world without AI, maybe I have 10 teammates, each teammate is working on one thing, so I need to have like 10 things I’m keeping track of. Now, we live in a world where like everyone I’m working with has 10 projects. I now have to keep track of like 200 things and I don’t know what’s important. There’s definitely a Slack message I missed, there’s probably some email I’ve missed somewhere by some foundation, and Codex helps me organize all of this stuff.
Jason Liu: 3:29 And so again, the things I really want you to take away from this workshop is the fact that compression works really, really well. Like, I have threads now that are like five weeks old that have, you know, 400 sub-agents in them and they generally just know what they need to do. They know what their job is. I also want you to become really comfortable with talking to your computer. Earlier today I said that Tony Stark is not texting Jarvis, right? And there’s really no future where text input is the thing that matters.
Jason Liu: 4:03 I basically use a foot pedal, so I like, I have a button that is transcribe and a button that says enter. And so I’ll just come by my desk with my hands behind my back and I just go like, “fix this, make this change, also like message this guy on Slack.” And then I just go back to, you know, talking to my coworkers and trying to figure out what is like the human side of actually working at OpenAI rather than just like monitoring Slack all day. App Shots is my favorite feature of all time. It’s like very satisfying. If any of you just like are on Codex right now, just press the command button side by side, you’re going to get this real nice animation or you’re going to get a modal to tell you to install computer use, just do that. It’s amazing.
Jason Liu: 4:46 Invest in your personal memory, right? I at this point when someone asks me what I’m doing, I don’t even have any idea. I kind of have to like look at my threads and look at the conversations to figure out how much I’ve delegated away and how much has been automated. And if you can then invest not only in skills for yourself but also plugins for your entire team, you can become the superhero that actually sort of augments the rest of your company. It’s one thing to say, “oh man, like I can use all these tokens and look at how many tokens I’m using,” but actually, if you’re rewarded by how often the plugins you’ve built are being used by your teammates, that’s a huge win.
Jason Liu: 5:22 Right? How are we doing like implementation? Like one of the most popular skills is just the like finalize the Codex app skill. And anyone who makes a pull request basically triggers the skill before review and basically everyone at the company uses it and it’s always been able to find things that I’ve done wrong or is against the style guide. One of the skills I have is just like reviewing docs and it’s basically just copying the pull requests of, or the PR reviews of our friend Charlie over here. And so I just have like a “review my code like Charlie” based off the past year of like feedback he’s given on pull requests. “Review my code like Dominic.” And these things are incredibly valuable.
Jason Liu: 5:58 And then lastly, once you get more comfortable with all this productivity… Four things. Your pinned threads with automation, these things that wake up these threads over time, they’re going to feel like teammates. And more interestingly, now that threads can talk to each other, so every thread has the ability to list other pinned threads, has the ability to rename threads, and has the ability to send messages to each other. Not only can you have teammates, but you can have teammates that work together. And you can effectively start having managers, right? And so you went from an IC enabled by an IDE, then you have pinned threads that feel like a team where you’re the manager. And very quickly in the future, as models get better, this is where the puck is going to skate to. You’re going to start having your like manager threads and then your IC threads, and I’m sure in the future there’s going to be some other crazy orchestration, right? And all of this really is due to the fact that compaction works. Even six months ago, I don’t know like how many people here have been told this, but you were always told, if a conversation goes very long, start a new thread, right? After 20 messages, it’s not going to be that good. Every feature should be its own conversation. If you do a code review, start a new session. Those things basically aren’t true anymore, and a lot of it has to do with compaction. Just pin a thread, rename it to the project ID, and that project thread should be able to delegate to sub-agents, create new threads, and have conversations and then write to your memory vault, which will allow you to just log what’s happening. And then with automation, you can just wake them up.
Jason Liu: 7:32 And so there’s really three acts of working with AI, right? Working in Codex. You bring the context in and I’ll talk about how you do that and what are the ways you can bring context in. Then you work on it, right? For example, this slide deck is just in the Codex app. And then you take actions out in the real world.
Jason Liu: 7:54 So I asked this during the keynote, but I’m also curious what the audience here is doing, but how many people use dictation when they interact with an AI? Nice. How many use dictation even at work? Yeah, I think we should all be a little bit more shameless, you know, in doing these kinds of things. You generally talk about three times faster than you type, and it’s just incredibly productive to be able to give the messy version of what you’re thinking to the AI and take that extra time and just try to be even more thoughtful to the people you work with, right? It’s like now, like I don’t want to send my coworker like a 15-minute voice memo, but I should feel very comfortable sending an AI a 15-minute voice memo because you’re going to include some random tangents. You might just say, I’m pretty sure I had a meeting with Charlie sometime last week about the agents SDK, and it will go and read like 35 meeting messages to figure out which one it was and make it relevant. And now all of a sudden, like whatever memo you’re going to write or some project tracker that you’re trying to do is going to work, right? But I would never do that with people.
Jason Liu: 9:00 AI. So, as you guys are doing or just listening to this talk, like try to just sort of set up Codex the way that I’ve been describing these things, right? So once you have your ability to just have you input into the machine a lot more effectively, you can start thinking about using things like skills and plugins.
Jason Liu: 9:20 Skills is a very simple construct, it’s just a couple of files and some scripts. A plugin is a library of these things. And as you are just doing things many, many times, you can start thinking about creating your own skills, and as you package a bunch of skills, you might start thinking about building out a plugin. If you want to install the plugins, we have a pretty good ecosystem now, something I’m really proud of. If you just go in the sidebar, click plugins, you can just search whatever plugins make sense for you. So if you use Slack, you can install Slack, if you use Gmail, Teams, most of these things are pretty built out. If there’s something that you feel like you are missing, just @ me on Twitter and I’m sure one of my Twitter monitors will pick it up and send a message to someone on the connectors team.
Jason Liu: 10:03 If you’re already actually looking at the plugins panel, I also really recommend starting the process of setting up the Chrome extension as well as computer use. We’ll talk about this a little bit later, but like computer use was the first time in a long time I really sort of felt the AGI of being at work, right? I was in iMovie for the first time, I didn’t know how to use it, and it was just teaching me how to like export the movie. It was able to like figure out where the sound effects were and it placed it in the right timestamps. Really small things like this that really make using a computer very fun again. Like I don’t really have the time to like learn new software, but if Codex can just show me what’s going on, it’s pretty awesome. And as you see the cursor move often times you’re like cheering for it to do the right action.
Jason Liu: 10:34 If you don’t have a link for the Chrome extension, you can just click this button here. The difference between computer use is computer use can work behind the scenes to control any application, right? So whether it’s Slack or some, you know, trading software, God forbid, it can control all of those things. With the Chrome extension, it just controls everything in the Chrome app. But the cool thing here too is that again, it doesn’t take over your screen, right? Sometimes I’ll just be working on my computer, I’ll go to the Chrome browser and I’ll realize that like Codex has just opened up three tabs to just look at my Twitter DMs and then just closes them back up as I’m just, you know, responding to some other email. It’s really cool to watch these things work in the background.
Jason Liu: 11:03 You can connect a bunch of other plugins. I use things like Notion, Linear, I also use Obsidian. It’s just a good time. Once you do this, what you’re going to find is just by asking really vague questions about your day and just tagging the right plugins, you’re going to realize the AI can learn a lot about you, right? The AI is not a system now where it does like one search request and tries to come up with an answer, right? It might check your emails and find a loose thread. It might check Slack or some meeting and figure out what’s actually going on, who are these people…
Jason Liu: 12:00 I had one of my loops basically realize I was meeting with somebody, look at their LinkedIn and realize that, uh, we went to the same university at the same time. And so the moment I jumped on my call, I was like, “Hey, you were also from Waterloo. You know, do you remember this, this, and this person?” And immediately we had a connection, right? And obviously I didn’t tell them it was AI, but that’s kind of some of the small things that you can do by just improving your automation. You know, it can make you closer to people.
Jason Liu: 12:26 As you build out your memory system, right, as you build out the Codex memory system and your ability to trigger plugins, maybe day one, you have to tag everything. But I’ve become like a worse and worse manager over time, right? Now I’ll just open up the composer and just say like, “What has changed about the launch?” And it’ll be able to do a good job, right? And it’s that’s possible because you have this long history. You have all these pinned threads, you have these memories, right? It’s the same thing with an employee. Day one, you have to show them every standard operating procedure, you know, but at some point you have an employee that has been here for seven years and you can just say, “Hey, I think you should make the company more money.” And they can figure it out, but it’s only because they have this context.
Jason Liu: 13:05 Um, one thing you could just do, for example, if you want to try it out, is you can just say, “Hey, check out the schedule, find all the sessions, organize them in a markdown file, put them in a spreadsheet,” and you’ll just realize that we can do these things. And maybe it’ll do it with web search, maybe you can do it with Chrome, lots of fun things here. If you want to get inspired by just looking at what kind of skills exist, we have two really great sources. One is if you just run the skill installer skill, it will actually list out all of the OpenAI curated skills. These include ones for things like GitHub, best practices when writing Playwright code, you know, Remotion, for example. But you can also check out websites like skills.sh or use, I think this is like Vercel’s, skills tool. And then you can just find other skills.
Jason Liu: 13:54 Right, so if I’m thinking about doing some more motion design or web design, or I know that like, someone told me I shouldn’t do like use memo in React, but I don’t really know what that means, I can now go install the React best practices skill, right? But again, internally, one of the highest impact things I think you can do as like the AI champion in your company is to figure out what the team needs and build out those skills, right? I have a lot of skills on doing things like triage and how you do comms, right? If there’s an outage on Twitter, how do you like convert that, figure out who needs to hear this, how do you like start the sev, what statsig gates do you need to check? All these things are now just automated. And that’s exactly what I just said in this slide.
Jason Liu: 14:44 Um, we also have a really good plugin creator and a skills creator skill. So if you just ask Codex to trigger it, it will try to interview you to figure out what’s going on. And even more in a more useful way, you can also just do it yourself once, document everything and just tell Codex to make a skill from what you’ve learned, right? And as long as you tell — and hey, by the way, every time you run this skill, you’re allowed to edit yourself if you learn something new. You can edit this skill file. These things will also improve over time.
Jason Liu: 15:09 And a big theme that’s happening over this talk really is just you kind of have to just get really comfortable with asking. Like we’ll obviously try to make more of these things like more slash commands, but more and more, like I’m just not touching a computer, so it doesn’t even make sense for me to like run a slash command. I just want to say what’s launching this week? Check Twitter. Look at what I’m seeing in the browser. Um, the example I’ve been developing internally has just been this like developer experience triage skills, right? So again, this skill just documents like every Slack channel that should be you should be aware of. It knows which engineers have worked on what projects. It knows what Slack channels are taking in feedback. I know that if you DM me on Slack and you tell me some regression has happened, I need to ask for a feedback ID, right? Now the agent does this automatically.
Jason Liu: 16:00 And it does it automatically with AppShots. So again, I don’t know how many times I’m going to say this, but AppShots is one of my favorite features. How many people here have just like sent a screenshot to Slack… to Codex? Right? Like almost everybody. But the issue is the screenshot does not have that much information, right? The model has to then do OCR. And if you send a screenshot of like a Slack thread, the model has to like read the Slack thread and then do a list Slack channels function and then realize like there’s a guy named Charlie and then do a list persons. It takes a lot of hops.
Jason Liu: 16:38 But with AppShots, it takes not only the image but the entire accessibility tree of the app. And so when I give it an AppShot of a Slack channel, it knows the channel ID, so it knows exactly what function to call to post there. It has the user IDs of every single person in that channel. So if I take an AppShot and say, “Do some research and reply,” it’s only one function call. It knows to send the send message to channel like U12725 and then it because, you know, it knows that Charlie is like U425, it can do that in a very fast hop. So not only is it a very quick way of getting context into your system, it just gives so much more context that the subsequent tool calls do a really good job.
Jason Liu: 17:22 I have not done like filled out a form in like two weeks because I just now tell Codex to fill out this form, right? It knows all the fields, it then figures out that it’s in Chrome and so it’ll use the browser extension. If it’s in Safari, it’ll use computer use. The model has become really, really intelligent. And so just like you might have a manager that gets an email and they forward the email to me with like three question marks and it’s your job to figure out what’s going on, you can kind of start doing that with your AI as you start investing in these skills. And most of this is because of the fact that you’ve built out your memory system. So if you guys are taking a look at these slides…
Jason Liu: 18:00 Jason will flash a personal mono repo template. That is actually the template I used on my personal computer. It’s basically just a directory tree and a bunch of skills that I use to sort of grow out my memory. I’ll also make one call out which is if you open this in your browser just press Appshots and tell Codex to set this up for you and then you can pay attention to the rest of the talk.
Jason Liu: 18:32 Yeah you can just tell Codex Jason has written a personal mono repo template on GitHub please find it and then install it.
Jason Liu: 18:44 How does this project vs the monorepo work with stuff elsewhere?
Jason Liu: 18:51 Yeah so this is a really good point. So like for example on the DX team I make a lot of demos and so I have like 16 repos like you know real-time demo 1 like real-time demo 2 like funny right? You have all these demos. I don’t create new projects for them.
Jason Liu: 19:13 Right? The only project that exists on my sidebar is the like personal mono-repo sidebar. But Codex is able to still manage files outside of that project directory. And so in my agents.md file I just say
Jason Liu: 19:31 don’t save any of the code in the mono-repo save it in like /dev. And just by that one line if I tell it to clone a new project it saves it in /dev. If I tell it that I want to work in my slides it knows that there’s like a /dev/slides directory.
Jason Liu: 19:46 But it’s just an easier way of managing everything right? Like I want to start all my projects from my personal vault and then it can touch the file system in any way that it wants to.
Jason Liu: 19:54 One call out it kind of breaks uh like git review sometimes on the sidebar but generally it’s been a pretty good experience for me because I just re- I just review my code in GitHub.
Jason Liu: 20:02 Sweet. Um these are some of the skills I have just installed there. Uh there’s no need to take a photo just ask Codex afterwards but uh the assistant plugin basically has the ability to onboard you. It will interview you it will figure out what plugins you need to install and then it will actually go create the threads it thinks it needs it’ll create the automations. It’s a pretty fun one.
Jason Liu: 20:22 I have a bunch of skills on like auditing AI code and AI writing. Um I don’t include this but one of my favorite skill of all time is called write like me.
Jason Liu: 20:33 And if you want to make one like that all you have to tell Codex is hey Codex I want you to read all the emails I’ve written in the past six months all the Slack messages I’ve written in the past six months and write a style guide for how to message just like me.
Jason Liu: 20:47 And then that’s it and then anytime I tell it to send a Slack message or write an email it’ll go okay this is an email clearly this is just like a customer support form so I will be much more stern in my messaging let me go draft this email.
Jason Liu: 21:00 hasn’t failed me yet. Um, one thing I’ve also added that I think are really valuable to call out is I’ve made my own loop skill just because I do like having a slash command every once in a while. I’ll talk about this in part two. And I also have a skill called simple HTML artifact that just designs artifacts the way I like them. I want my backgrounds to be white, I want some certain style guides and an ultra goal, which is like a super version of goal that we’ll also talk a little bit more about.
Jason Liu: 21:28 And then if anyone’s curious, like new person, new project, that’s just a way of like running a script to bootstrap a new person. I kind of have like a Palantir for my personal life now. It’s just like a CRM and basically anytime my AI agent like finds a new person that’s emailed me or messages me on Slack or on iMessage, I just keep track of these things. And then new project is the same way.
Jason Liu: 21:52 Let me just double check. Yeah, cool. Um, and so I’ll give you maybe like 10 minutes to try to try to set this up and we can go on a little bit of a Q and A. I’m happy to answer any questions about how we bring context into our systems, how I’ve organized my personal memory vault, and, you know, some other crazy uses of App shots if anyone has any questions.
Jason Liu: 22:13 Does this get rid of the need for like an Obsidian brain that brings context for itself because you have this context engine that’s really good now?
Jason Liu: 22:23 Yeah. Um, so the question was, do I still basically use Obsidian brain? The answer is yes because I still want to sort of like keep track of everything.
Jason Liu: 22:31 Um, one thing I actually really like doing is I make my monorepo vault like a git repo. And so maybe it’ll work on it for like a couple of days and I’ll come back and I’ll just run git diff. And by running git diff, I can just see like what the model has updated and what the model has not updated. And I can just confidently review that over time and just realize that oh yeah, like I guess Charlie did respond to this person and closed the loop and I didn’t realize that, but now I know, right? And often times that’s relevant in another conversation.
Jason Liu: 23:00 More than that, it’s also very helpful for when other people are about asking me questions, right? Codexes will feel very good about reading my memory vault, drafting a response, and then ask me for permission to send that message off. And so if someone messages me on Slack, a question the AI could have answered, the AI will just try to answer it. Um, and it might be simple things like oh, like who should I talk to about this project, right? And the model knows because it’s in the memory vault.
Jason Liu: 23:31 One thing I’ll also call out is if you want to use more tokens, you can also have like custom automations where the job is to maintain and manage and garden your memory vault. Um, but generally that has not been a big issue for me.
Jason Liu: 23:44 How do you think about doing evals on those skills you’re creating or I guess manual review, automated evals, that sort of thing for better manipulating the system versus just like the YOLO approach?
Jason Liu: 23:59 Honestly… I generally go down the path of like, yolo one-shot, only because — Oh, that’s good at me. Um, only because I know that like, the way I build my skills is they self-improve all the time, right? Like, I think the difference would be if I make a skill that I share with my team, I think about that a little bit more, right? Because it’s like, okay, do you — does the triage plugin know that like, who is working on what feature? Can it route correctly? For my personal work, I generally just build a skill as quickly as possible and every time it makes a mistake, I just correct it and I tell it to move on. And then generally what happens is if I’ve used the skill for two months, I just generally feel pretty good about sharing it with my team because I’ve just experienced it working. Um, yeah. Most of my skills connect to so many other plugins and connectors that, uh, I just don’t know how to eval that because I can’t like snapshot my Slack at any given time. A question over there?
Jason Liu: 25:04 Are you primarily using the desktop app or are you using the CLI to connect to it?
Jason Liu: 25:12 Sometimes I’ll use the Codex CLI every once in a while if I want to like be a little bit faster. But generally the desktop app has — pretty good — it has a pretty good experience. Primarily because everything I do is an AppShot. Like if I’m watching a video, I’ll just like AppShot, like summarize this, and I’ll continue to watch the video. I’ll just watch a video with like the LLM like summary, right? Or it’s like a — if I see some kind of form, or someone asks me to sign something, I go like AppShot, use DocuSign, like sign this and save it to my desktop. Uh, like last week, it like DocuSigned something then like found a faxing service and like faxed my medical records. Like, that’s awesome. But the CLI can’t really do that. Um, yeah. Any other questions?
Jason Liu: 26:00 What did you do while waiting for the agent?
Jason Liu: 26:06 Like I’m learning to juggle, I’m like — I’m learning to play the drums. Um, well, I think it’s like two things where like, in the office, what I’m trying to do is I’m trying to like talk to more people, right? It’s like — it’s like I’m just the AI’s assistant to get more context that the AI can’t get. Uh, no, I think my job when I’m at work really is just when the AI’s running, I should be talking to somebody. I should be like learning about what they’re working on, trying to make connections, and then figure out what are the, you know, points of connection, right? It’s like I should be talking to more people in real life as AI works. Like the one — another question over there? Oh.
Jason Liu: 26:40 Um, I had another question about use cases where you use, like, 5.5 versus G- 5.3 Spark, specifically while you were using the CLI.
Jason Liu: 26:56 So the question is like, when do I think 5.5 is overkill versus 5.3 Spark?
Jason Liu: 27:00 I think this is colored by two things. Like because I have unlimited tokens, I don’t really make those decisions. And then secondly, because I’m not watching my AI work, like most of these are automations that run in the background, the latency has not really affected me. The times where I use Spark is primarily when there’s a very simple… there’s a really simple computer use task. Right? Like I just wanted to like click all the buttons and fill out this form. Like, I think I have a thing that just checks me into flights and that is like a Spark agent. And so now, like anytime there’s an email that’s like a flight check-in, my agent will check me in, download the boarding pass, and then send the boarding pass to myself on iMessage. Right? And like I just never do that kind of stuff anymore. Uh, again, it is really weird when you’re just like working and all of a sudden you check your… you check your Chrome desktop and it’s just like, you know, JetBlue is just like the first page. But again, it speaks to the fact that like having access to your computer is uniquely powerful because it has your auth and your credentials and your file system. Yeah. Over there?
Jason Liu: 28:11 So when you’re running something long-running, specifically the computer use, do you also want to be able to run it from your phone or your iPad?
Jason Liu: 28:22 Sorry, you’re really quiet. Do you mind just speaking up?
Jason Liu: 28:24 Yeah, I’m asking if you want to do long-running tasks, specifically computer use, you personally not being at your laptop. Let’s say you have two devices. How do you actually, do you actually implement some sort of communication between these physical devices or is it only cloud or only on your device?
Jason Liu: 28:41 Yeah, so I have… we’ll talk about this basically in the next section, but uh with remote control I can control both my local computer and my like a remote computer. I think the difference is computer use is tricky because it has to be on your computer as one thing. The second thing too is if you go into settings, computer use, there’s a flag called locked to use. And if you enable that, as long as your laptop is plugged in even if the monitor is closed, you can still trigger computer use commands through your phone. Uh, that also gets really weird because like… but go on.
Jason Liu: 29:20 So I was going to say my question is like you have two primary devices. Yeah. The one you use for work, like take your laptop with you traveling, but you want long-running tasks to be dispatched to your desktop back at home.
Jason Liu: 29:34 Okay, okay. Cool. Um, yeah, like if you have multiple computers, you can still connect both your iPhone to both those devices, right? Like some people just have a Mac mini. Uh, I think the difference is do you want to control your Codex or specifically computer use because that requires like an operating system with a GUI. Um, but we can talk about this in a little bit. Cool.
Jason Liu: 29:57 Like I said before, when… Are there any more questions? I know someone just raised their hand. Yeah, go ahead.
Jason Liu: 30:03 Is there anything regarding risk with the database and how do you control that?
Jason Liu: 30:08 Yeah. I mean there’s always some kind of risk, like I think earlier versions might like edit a document a little bit too eagerly. Right? But realistically, I think these models have done a really good job of being very precautionary. And more often than not, it’s me going like, “No, please, just do it. Like just sign the document. Like this is… please just like send this message.” I found that the 5-5 models are pretty reluctant to take these like destructive actions. That’s one thing. The second thing is, if you look at the sidebar here, ooh, you have the ability to change your permissions. And so as a show of hands, like how many people use like “Ask me for every permission”?
Jason Liu: 30:59 Yeah, exactly. Okay. How many people use like full auto, full permissions, YOLO mode?
Jason Liu: 31:06 Okay, I don’t like that. And then how many people have used auto review?
Jason Liu: 31:10 Yeah, so I think auto review is actually has been really, really great. And again, I’m usually annoyed by the fact that my models won’t do more than I want them to do. And so generally it has not been as big of an issue. The only times where I’m really annoyed is it will like edit a document it shouldn’t be editing. But I just like add a thing in the agent MD and it’s never really messed with me too much. Yeah. I know that’s not a real answer, but I think with a combination of auto review and agent MD file, I have generally felt pretty safe. Yeah. And then if you’re also at an organization, there’s different admin settings that you can have. So for example, at OpenAI, you can’t use an MCP server to send an email if any of the people in that email is a non-OpenAI email. Right? Or like you can’t send a Slack message to external Slack channels. That’s when things get dangerous. Right? But these are some things that you can control. One question over there? Yeah. Got the mic.
Jason Liu: 32:22 I have a follow-up question. Do we have any concerns regarding either security and or privacy?
Jason Liu: 32:37 That’s tough because I work here. I think that’s hard to answer because I don’t really know what our like data retention policies for like individual versus enterprise. But Charlie, do you have any thoughts there? I’m just going to throw it over to you.
Charlie: 32:54 Just concerns about security and privacy?
Jason Liu: 33:00 Let’s say you get a, I don’t know, email offering you a job at a competitor company.
Charlie: 33:10 Ah, I mean, I think a lot of it goes back to we do want to make, like the models are fallible, right? I don’t think anybody in this room would be shocked to understand that like you can still jailbreak a model, for example. But both the models themselves are getting smarter and better at not, you know, doing silly things. And at the same time, we’re figuring out like Jason mentioned, what are those bigger, you know, limitations around the sandbox. We started with very simple sandboxes where it was like you can just run this command and nothing else. And slowly the sandbox has grown to the entire computer. And I think we’re figuring out what are the like computer level or organization level, you know, edges to that sandbox that we need to build.
Jason Liu: 33:52 This is me using the delegate skill.
Charlie: 33:58 Great answer. Any more questions before we jump into act two?
Jason Liu: 34:03 Sweet. Awesome. So we just talked about a bunch of different ways of bringing context into the system, right? You can use your voice, you have plugins, you can use app shots, and then you can also design different skills and plugins to figure out how to do more systematic work. So now we can talk a little bit more about the work itself. So like I said before, like every pinned thread effectively is a teammate in my mind. I have my chief of staff thread. I have, Swyx refers to it like, you know, the God thread. I have a thread to manage the Agents SDK, whether that’s implementation and documentation. It has two sub-agents that it delegates to. The CLI, the open source program, and Twitter. But if you want to make it wake up, right, all you have to say is keep an eye on this until sometime. You know, keep an eye on this every 30 minutes. If you remember those like secret words, you can effectively automate like about everything in your life at this point. And what this does, it will trigger a heartbeat automation, a thread automation. You should think of it as a way of scheduling a message back into the thread, right? In the beginning, when we set up automations, it was very much the case that an automation would create a new thread every time. So it might be give me a morning brief and it will create a new thread and then it would do that kind of work. But as these models got better, I think the right design is scheduling these messages into the same thread. So for example, because if you download the mono repo, we have a loop skill. If you just do loop, right, this is equivalent of just saying keep an eye on this, keep an eye on this pull request. Anytime there’s feedback, fix it, make sure it’s always mergeable, always make sure it’s always rebased on master, make sure that CI is always passing. And it’ll just do that. And then maybe you make a pull request in, on a Monday, you get really busy, Thursday afternoon, all the feedback has been integrated.
Jason Liu: 36:00 …in up’s CI’s passing and, you know, you’re not like 4,000 commits behind the main thread. I also do this with support, right? Again, if you — if someone is like dealing with some issues on Twitter, on Slack, psh, Appshar, right? You know, like, at developer experience skill, figure this out.
Jason Liu: 36:22 And it’ll say, okay, this is an issue on the browser side, like James is the one that works on the browser, the channel is called like browser feedback, I’m going to post in that channel, DM James, and then I’ll use computer use to open up Twitter to let them know that I’ve like escalated this internally and then I will check every hour to figure out if James on that channel has responded and then let the user know.
Jason Liu: 36:50 And then sometime later in the future you’re like checking your computer and all of a sudden like Twitter opens up and it’s just like “Hey so and so this has been resolved” and then you just hit enter and then you check your thread and it’s like “Oh yeah, the pull request has been made, it will get merged by next Thursday”… and like this is a crazy experience to witness, right? This is actually allows us to do way more support without making it like the worst part of my job.
Jason Liu: 37:16 And then with the chief of staff thread, oh, this is — I remember this is where my slides get really messed up thanks to Codex, so I can’t do everything just yet. Um, you can also just do a loop that says check all my connectors and give me an update as to what is the most important thing I should be thinking about, you know, give it to me in a nice format, you know, make sure you have links to every email that you read, make sure you have a Slack link so you can deep link into the application.
Jason Liu: 37:47 And now just it’s been really, really helpful to track these random things. And again, I think in my chief of staff thread I have that line that says check into all my flights if you can, like send me the boarding pass on iMessage. And it just works.
Jason Liu: 38:03 And again, it’s really weird when you just start seeing your computer doing stuff while you’re — while you’re working because, again, most of these tasks run in the background and because I’ve had my permissions on, it’s not like it needs to tell me that it’s using Safari. I just like find out that it’s using Safari. That might be a scare to some people, but, you know, it’s pretty great.
Jason Liu: 38:22 And then lastly, this is something that happens really a lot at OpenAI which is I don’t even know what’s what’s launching, right? Is something delayed? Has something landed? Is it, you know, going to happen at 11 am? Is it going to happen at 4 pm? I have no idea. We’ve actually made a ton of progress on this and I’m pretty sure this is also just automation and AI, but I actually used to have Codex make me a PowerPoint every Wednesday night of what’s shipping for the rest of the week, right? And that’s useful and then you just say, “Great, it’s useful for me, now let me make sure I can just post this in the channel as a Slack message” and now you’re again using your skills not only to benefit yourself but benefit your entire team. And that’s kind of like the…
Charlie: 39:00 getting hero mindset.
Charlie: 39:02 And this one’s pretty funny too, I have an example, let me just double check the slide. Yeah. Have another example which has also been pretty crazy, which is the one I gave during the keynote, which is I had been editing like a short film in iMovie and on my bike ride someone gave me feedback about the video. So I just went on my phone and I just sent, told the AI like okay there’s a file somewhere, can you just like find it in iMovie, there’s only one iMovie project, read the Slack message, re-export the video, if you can if the Slack MCP server does not allow file upload, use computer use to upload the file and then watch that thread every hour and if they have any feedback, re-export the video and re-share it. And then I biked home. And by the time I got home it was like, oh by the way, it was actually way easier to use a Google Drive connector so I’ve just been uploading the same file on Google Drive instead, so they only have like one URL to manage. And uh yeah, we like fixed a bunch of stuff in the typography and then we shipped it. Just again, like kind of mind-blowing stuff. It’s very simple but mind-blowing.
Jason Liu: 40:06 Right.
Charlie: 40:07 But that’s what a heartbeat is, right? The heartbeat is just a way of waking up your thread over time to take some actions. The other thing you can do is set goals. So /goal is pretty amazing, it basically defines a verification step and says okay as long as this is running, check this verification step. If it’s not done, keep going. A very simple idea. Um and as long as there’s a verifier it does really, really well. For example, I’ve just been migrating a bunch of software into Rust, right? It’s like if this is a Python project that is amenable to be rewritten in Rust, /goal migrate the backend to Rust, make sure all the unit tests pass. And I was able to not only rewrite the rich terminal library in Rust, I also rewrote UV in TypeScript um just to see if I could, and uh we’re like 100% test coverage. It’s pretty amazing. Obviously, you should not be doing this in your work but it’s it’s very helpful to just understand it as these systems have better verification, you can make a lot of progress. In the mono-repo I’ve also included a skill called ultra-goal, and all it does is instead of setting the goal in the app, we set it in a file, right? So we have a goal.md file. And what that means is you can edit the goal while it’s being run so you can add more scope, uh just like many many real projects do. We also define a plan again that we can reference. But the benefit of this is as you’re learning more about the project and you’re changing the plan and the goal as the as these models are just like looping, uh it can update its understanding of the system. And then sometimes I have like a state.md file or a work log just to track these like longer running tasks. Like if things are running for like a day or two, I want to know what’s going on and I’m never going to read…
Jason Liu: 42:00 It’s like this 4 gigabyte like session JSON object but I can look at the work log and have another model summarize it using like a side chat and go from there. This point stays remote control but again if folks are just like on their computers I’d also recommend trying that out in the sidebar. There should be a button that says remote control especially if you have the iOS app. This is the thing where we talked about being able to control your phone so control your computer through your phone right? So if you go on the iOS app and you enter Codex you can do this like flow where you can scan a QR code.
Jason Liu: 42:41 And all of a sudden your your your ChatGPT app can message and cue any thread in the application including I think remote threads. This is super powerful because again oftentimes you know every time I try to leave the house someone’s like asking me for something and now I can just ask Codex. Really I should just be having something that monitors slack and just does it but like I’m not there yet.
Jason Liu: 43:00 But again this is one of those big you know feel the AGI moments right? Yeah and again like I said the chief of staff thread effectively is like this single source of truth for basically what’s going on in my life on my personal computer. I have a different one. Lots of good stuff.
Jason Liu: 43:22 This is all I wrote for mine. Create and pin a chief of staff thread every day at 8:00 a.m. Check all these connectors, figure out what’s going on and then you know do a good job. And again you’ll start editing it over time right? Maybe you don’t like the formatting or you wish you included links for a while. What I made it do was if it found all the emails not only to ask it to draft responses but I would make it open a Chrome tab for every email I need to reply to in Chrome.
Jason Liu: 43:55 And so I’ll open my computer, I’ll take a meeting, I’ll come back and on my computer is just like seven Chrome tabs and I can just review the drafts and send each one. Like small things like this just to prepare your computer while your meetings are happening. Really productive. So now we’ve talked a little bit more about like just doing the work itself right? We haven’t really gone into things like artifacts just yet but I’m also curious if anyone has any questions on how I’ve been doing things so far?
Charlie: 44:18 Over here. So I noticed so I do a lot of meta-prompting.
Jason Liu: 44:21 Yeah.
Charlie: 44:22 with GPT. I noticed that you have very small prompts.
Jason Liu: 44:27 Yeah.
Charlie: 44:28 And so but when I meta-prompt I get a lot of stuff from GPT. So what’s your advice?
Jason Liu: 44:32 I generally always prefer to have the model write the goal or write the write the prompt itself. Like more and more these models are just getting better at doing that and it’s like more in-distribution to what they want. In reality I just I will send like a 10 minute voice memo right? I’m just like like you know this is some issue that’s happening. I think there’s a project about some thread. I don’t know if they got back to me. I think their name is Dave. And just let it go.
Jason Liu: 45:00 in, please look in Gmail, like, take a look, it’s like a really, really messy. And then it’s like, okay, well, do I want to make it set a goal? Do I want it to create a new thread? Do I want to make a new skill? Like, that’s really sort of where my taste lies now. It’s like, okay, how do I want to organize what the work product is? You know, do I tell it to then do all this work and put it into a index.html to share it? Do I want to make a Word doc? That’s basically it. But I generally just send very long messages.
Jason Liu: 45:33 Like, for example, here, like, this is not the prompt for the chief of staff thread. This is the prompt so the model can make the chief of staff thread. Right? And that model will be much, that output will be much better at determining how verbose the automation is or how often it should check, or what connectors. Um, and then because I have this monorepo, one of the things I do is every project file has a link to every Slack channel this project is relevant for. And the model will just see that and go read those Slack channels, right? Every like person.md file has their email address, their Slack connector, multiple work addresses, and it’ll read all those things. I would never prompt that myself. Yeah.
Jason Liu: 46:31 Yeah. So I think in that example, what I realized was, when I mentioned the Slack channels that are relevant for a project, the results were better. So I just added that in the front matter of like the markdown file. But these are the things that you grow over time.
Charlie: 46:43 So did I get it right, that it comes to give you something like Open Claw or Hermes Agent, but it’s not? Is it still multiple threads that you open versus having this one thread and a lot of dependencies underneath? So is the only difference those multiple agents?
Jason Liu: 46:52 I mean, I’ve not tried like the deepest, so the question is like, what is the difference between this and like an Open Claw and a Hermes agent? I think, I’m sure there’s differences I’m not aware of right now, but I can imagine a world where it’s going to get much, like, very, very similar very quickly. Right? Like most of my work, and I’ll talk about this later on, is like my threads manage themselves, right? Could be the same thing as a sub-agent. Um, I don’t know how many people use Hermes agents and Open Claw for like very wide work, right? Like I think, like I think I tried Open Claw to do some like house automation stuff.
Charlie: 47:30 The idea would be that you only make one thread, and then there’s meta-data on the level that prompts other threads of conversation.
Jason Liu: 47:40 Yeah, so the comment’s like, yeah, with these models, there’s only one thread. I think that’s very reasonable. Just in reality, there’s just so many things that we work on that like I need the organization, right? It’s like, you know, it’s like, is there a world where like my banker and my therapist and my personal trainer and my girlfriend is the same person? Like maybe, but like my like, you know, tiny brain can’t figure that out and like it’s easy. It’s easy for me to understand what the work is by making folders, right? It’s like the computer doesn’t know the folders exist but the folders are for me in some ways, if that makes sense. Yeah.
Jason Liu: 48:13 Cool. Great. No more questions? Yeah, one question. In the back there, yeah.
Charlie: 48:18 Um, so the other thing is, on one of your slides it has like three connectors, so Slack, Gmail, etc. Um, and then you said after a while you can just like ask the assistant to do things. How long does it take to do that? Um, because like I go ahead and use Slack and connectors etc skills but like I didn’t know that it just automatically will do it once you start doing it.
Jason Liu: 48:46 Um, I think it depends. I think it depends on how proactive you are in like telling the AI to remember these things. So for example, like once I realized that I should include Slack channel IDs in like project documents, the model is like, “Oh, there’s a Slack channel, I should read the Slack channel.” Um, that became really obvious. Now I basically never tag things. I don’t really know when that happened. Um, but yeah, again, a lot of it is just getting the habit of, “Remember this for next time, update the skill for next time, update the agent.md for next time.” And that is the-
Charlie: 49:21 Is that like always, or is it memory?
Jason Liu: 49:24 I mean, I think it would be hard for me to like prove it, like there’s also a memory system that’s like outside of the docs. Generally I’m like pretty happy with the memory. I don’t know if I can like give you like a time estimate, but I think I would just try it out, right? Like use it for a couple weeks, have make sure your memory is turned on by the way, like it’s also in your settings, I know our settings panel is like pretty crazy right now, but um, yeah, I think I would just turn memories on and just see whether or not these changes happen over time, because like I basically never I don’t remember a single time I’ve like @ mentioned something um-
Jason Liu: 50:00 Yeah. And the last part is pretty pretty fast, right? So we talked about reading context, we talked about working with the context, now the the last thing to do is just like write context. Um, most of this has been pretty pretty simple, right? You can draft emails, you can draft Slack updates, if you feel very brave you can send them, but like please be respectful, like I’m sure Charlie has gotten hundreds of sent from ChatGPT Slack messages from me over time. Uh, but I hope they sound like me more now.
Jason Liu: 50:32 Uh, building one-pagers is also pretty good, like this slide deck was made with Codex and soon we’ll be able to do things like serve applications, and now I think, you know, at least internally, so much of the work that we do has just been sharing like apps rather than like full documents. I don’t know how many people know about this but we also have a really good like artifacts ecosystem, like more and more Codex has become a tool for all of the work that I do. It can open and render like Excel spreadsheets, Word documents-
Charlie: 51:00 PDF slides and with the annotation tool, editing things is like pretty fun. So even with these slides, this is actually served on the in-app browser of the Codex app. And what I’ll do is I’ll just give my talk, I’ll press next and press next, and then when I don’t like something, I just select it. As in, “Hey, like fix this, I don’t like the white space, like these two slides need to be broken up into two more slides.” I hit enter. As Codex is working to clean this up, I’m just going down to get down the slides and it’s generally been a pretty natural way of working. Like this deck really came from me reading my own blog post out loud and then generating the material as we go along. But you know, it’s still like two or three skills to make the slides look this way.
Charlie: 51:48 Yeah, and then once you do that, you can do it again, again, it’s the same concept over and over again, right? You can build out these loops that just touch other parts of the system. Like most of our project trackers are just Google Sheets updated with loops. The these slides are loops and the annotations, yeah, I just said everything. My bad.
Charlie: 52:12 And then one thing that’s also very helpful is like as the company gets more like context dense, right? Maybe it’s all AI agents, just the ability to like summarize things, like over Slack has been incredibly useful. That was clearly like a slop slide that ChatGPT edited in.
Charlie: 52:31 So once you can take actions on these generated artifacts, I think the biggest thing and the thing I really want people to try out is just computer use. Um, again, it is like the first time I’ve had that like feel-the-AGI moment, right? Like a cursor is like trying to do some action, you can see it move across the screen. And like when it does it really well, like you really gain a lot of faith in the system. So we obviously have like plugins, right? And this is for sending Slack messages.
Charlie: 52:58 But then the in-app browser is going to get even more powerful soon, right? We’re going to be able to basically treat this like the browser that you use. I now try to use the in-app browser as much as possible. And then for everything else, use computer use. Um, how many people have used computer use, by the way? Very few… oh, it’s like 10% of the room.
Jason Liu: 53:18 What’s a crazy thing that you’ve done? Or what’s like the craziest thing anyone’s done? Any volunteers? Managing my home lab.
Charlie: 53:25 Home lab? What’s a home lab?
Jason Liu: 53:27 Uh, 30 agents. I have like around- I have a DGX Spark. I have a bunch of boxes and essentially 30 agents and I can call it and just do stuff. I have a Mac mini um that I have Codex installed on and actually have three of them in my home lab and essentially they each do different things.
Charlie: 53:46 Wow.
Jason Liu: 53:47 Any other like crazy computer use stories? Gotta gotta get AGI pill and try out computer use.
Charlie: 53:55 Yeah. One thing I’ll mention…
Jason Liu: 54:00 Say is like because computers are so powerful, like you mentioned before, there is some like safety component. I am now remembering this example where like again because the Slack connector was not able to upload files. If this model is really determined, right? It could be like the one with Willow, you just… it just says, okay, great, well if I can’t add a Slack file, let me like go on computer use and press file upload and do that, right? There will be some times where the model based on how you prompt it becomes really determined. It’s like, oh, it seems like I can’t email someone using the Gmail connector, let me open up Chrome and hit the send button. So those are the kind of things that you should be really wary about, right? These are like real security issues. And again, more than not, like having things in the agent MD file has been really, really helpful, especially if you do things like Guardian mode, or Auto mode, excuse me.
Jason Liu: 54:52 And it has also really changed the way I think about doing work. I feel like now when I’m doing things like building an application, if it’s a native application, most of my testing is just done by Codex using computer use. If it’s a website, again, it’s just using the in-app browser.
Jason Liu: 55:13 I think we talked a lot about this already, like handle service work, like it’s kind of awesome just like be on a checkout page, app shots, you know, find me a coupon, like it’s made me more money than I would have expected earlier. Filling out forms, testing applications. I haven’t really had a good use of this just yet, but it can also control the iPhone through screen mirroring. So do with that what you will.
Jason Liu: 55:39 And this is kind of sort of the escalation of the talk, right? Then the question is like what can the computer not do and like why can’t it do it? And one of the things that you can’t do is control Codex. But you don’t really need to because these Codex threads can already talk to each other, it can already control itself in very powerful ways, right?
Jason Liu: 56:04 If you think of the example… this is not the slide I want. Damn, okay, I think I messed up some slides. Earlier in the talk, I talked about this idea that if there was some kind of support issue, I could take an app shot and it’ll call this like DX triage skill. And it will rename this skill, it’ll do the loop, and then it communicates via Slack and Twitter and it’s very nice. But I still need to be the person that triggers these things, right? That’s the same thing as me seeing an issue and making a pull request.
Jason Liu: 56:35 In the future, well I mean, this future for you is happening already now here, but now what I have, I just have a single like monitor thread. And anytime it identifies any of these issues, it will go off and create a new thread and the thread’s job is to do all this triage. And what might happen is maybe this triage is waiting on someone on Slack to acknowledge this issue, maybe a pull request has been created but it’s not merged. If someone else complains again in the future, the monitor thread just goes, oh yeah, I think this is the same issue. Not only is the…
Jason Liu: 57:00 So essentially let me send a message to that, you know, downstream thread so that it’s aware that this issue is recurring. Maybe that thread will send us a Slack message. But in the main thread, I’ll just see a message that says, “Hey, it’s been like the third day this issue has been live uh based on Twitter feedback. Should we do something?” And like those are the things I’m I’m it’s very hard for me to keep track of these things because I’m just like on Twitter all the time. But by the agent being able to just manage and consume all this information, it makes these things much more tractable.
Jason Liu: 57:32 Uh, definitely messed up the slides over here so I’m going to skip a couple slides. Go… yeah, this is this is the uh example.
Jason Liu: 57:43 And and I really want you to I really really want you to play around with this idea. I don’t think it’s been fully baked yet, right? Most of my work is about just having monitors create sub-threads. These threads are then managed and pinned onto the sidebar. That’s also one of the best things, right? Like with a sub-agent, the thread just has these like shapeless entities floating in the background, like, you know, the JSON thread or like the Galileo thread or whatever. The difference with sub-agents versus these threads is because they show up in the sidebar, you can kind of just notice that something has changed. You know there’s a new issue that’s come up, right? And a lot of it too is like just using the sidebar as effectively like the hub of understanding like what are the ongoing like workstreams. And this has been like super powerful for me.
Jason Liu: 58:31 But it all just starts from pinning a thread, taking actions, having this monorepo to sort of manage all of your context and then having different ways of waking systems up. And earlier, these systems wake up because you messaged it. Or you set up a heartbeat. And now these things can be woken up by another thread running somewhere else. And so more often than not, like most of my automation just happens on the monitor level of threads and they trigger and create new threads and then manage themselves.
Jason Liu: 58:59 Yeah. So I think we talked about a bunch of things, right? We talked about computer use, structure plugins, and uh… yeah, I think that’s basically it. Try out some computer use stuff. I know I can’t really… I’m like really worried about the Wi-Fi here so I’m not going to try something too bad, but um… yeah, try out AppShots, try out computer use, and really play around with what uh these models are capable of. There’s a question over there, we’ll do that one next.
Charlie: 59:27 Yeah, I wonder if you can just touch on or show us like parts of or what’s in your monorepo?
Jason Liu: 59:35 I don’t think I can show you on this computer. They’re going to like take me away. Um, I mean, I can talk about it a little bit. So so basically my monorepo is set up so that it looks very much like the one over there. I have a projects directory and in that projects directory is a named directory for every workstream I’m working on. So maybe it is the, you know, voice launch video or it is the…
Charlie: 1:00:00 the agent’s SDK, right? Or it is the Codex open source grant program. So those are sort of some of the projects. Then I have a people directory, which is just like every single person that’s ever DM’d me, right? I know what they’re working on, I know what kind of problems they’re thinking about, I know what other like side channels they’re part of. And then I have a bunch of different like loose notes, agent summaries, I have like daily summaries of what I’ve done. This is mostly just to test the limits of AI. I don’t think it’s very useful for that kind of stuff. Mostly it’s just the projects and the people. Right. And then I have a to-do list that an agent just maintains. So I have a single thread that just like check the to-do list, oh this says it was undone, let me have a sub-agent verify that like no one has done this task. Those things are pretty token-expensive. I don’t really know if it would be worth people setting this up for themselves unless they just like don’t use that many credits. But I think the biggest ones is just like the chief-of-staff thread. 9:00 AM tell me what’s happening today, tell me what’s happening this week. And I think if you just do that, you’re going to get a lot of juice out of that.
Jason Liu: 1:01:12 Are you not crushed by like context bloat or like a lot of context and stuff like that?
Charlie: 1:01:23 It’s honestly, I just, I’ve never experienced it. I think the compression is just really, really good. I know that’s not a very satisfactory answer, but like if I could improve parts of the model, I would rather improve like its writing tone rather than like its ability to search the context. Generally it’s been pretty good. I think it might have to do with like the way the Codex memories have been set up, but I have not dug in too much into those details. I think you had a question over here?
Jason Liu: 1:01:57 Yeah. Another question on the memory management. So one thing I notice a lot is memory bleed across projects. Do you have any thoughts on that? I’ve seen it a lot, especially when using embedding memory, it tends to take things from some other project for the current project. It’s not just the Codex or agent system, it’s like across the board.
Charlie: 1:02:24 Yeah, so I think the question was just like how do you deal with memories like bleeding across different projects? I don’t think I have a good answer to that primarily because it’s unclear to me what the downstream side effect would be. Maybe like one project uses like NPM and another project uses like Yarn or something. But I mean generally I just clean those up in the agent MD files for those specific projects. Right, so another thing to mention is like every project directory has its own readme and its own agent MD files. And so yeah, I think sometimes like one project I was working on was like NPM and one was PNPM, I just clean
Charlie: 1:03:00 clean up in the 80s and 70s. I would be curious like how much of that can be cleaned up just by using those simple tools. But talk to me afterwards. I’m really curious like what’s so bad about the bleeding. Yeah. Maybe that would be a good example of like having project level scoping versus a single thread, but yeah, I might look into more details about how the memory part is actually running. Any other questions over here?
Jason Liu: 1:03:27 Did you change your context on big files so that if it starts to drift, you do more of a refresh and how did you set that up?
Charlie: 1:03:36 No, so I think honestly at this point, it just kind of knows. It’s such a bad answer, but it’s almost like an AGI. I feel like to me, it feels like a very AGI-feel thing. But I think generally in the beginning when I was working with it, I would have a skill called like check notes. And it’s like, “Hey, if you feel like you need to check your notes, like check the notes,” right? And so if I had a hunch or some intuition that the model might not be able to do this, I would just mention like, “Hey, check the notes.” But even if you look at my OpenAI profile, I think my like check notes skill has been used like 150,000 times. But I’ve like never mentioned it in the past like two or three months. It’s because again, I think this memory system has been doing a lot of the heavy lifting. But again, it’s like onboarding an employee. In the first you know two, three months of onboarding an employee, you have to give a lot of instruction, you have to give them a lot of context, and you have to let them fail. And when they fail, you have to give them an opportunity to like write down what they’ve learned and like clean up itself. I would be very surprised if you got good results in the first like day of setting this up. But I mean, just yeah, maybe not to your surprise, but now it’s like I don’t, it kind of just works, right? And that’s a great feeling to have, right? Because it lets me go back in the flow of just like doing my job.
Charlie: 1:05:01 Your question?
Jason Liu: 1:05:02 Yeah, thanks so much for the talk. I wanted to ask, so I’m a college student studying CS at Stanford and I’m going to be entering the workforce in about a year, right? Given that it’s so rapidly changing and developing, what advice do you have for college students who are like developing, like not just productivity, but just coding in general? Because now it’s not so valuable to code by hand as much as it used to be. Kind of what would your advice be to college students who are entering the workforce?
Charlie: 1:05:27 I’m so old, it’s so long ago. Like I have a line that’s like, “If you want to have good taste, you kind of have to eat,” right? Like I think your job is to like consume a little bit more, like try out different applications, like, you know, like do you know what a good onboarding flow looks like? Do you know what that feels like? Have you built an app that makes you frustrated and just like your ability just to consume more things and develop
Jason Liu: 1:06:00 Give vocabulary on like how to complain about things that are bad. How do you, like, complain about, like, the slop? Those are the things, those are the skills I think will be very valuable. And specifically around vocabulary. Right? Like you can’t really describe things that you don’t really understand. Um, and this happens in, like, both, like, things like cooking, right? Like if you just don’t know what the ingredients are, you can’t describe that something is too salty or it could be, like, it lacks some acid. Like, you just need to, like, consume a little bit more and then think critically about why you like something and why you don’t like something. Um, because then I think taste is the, is a big issue, but I think a lot of it is because we are not consuming the right, like, stuff, right? Like, try out all the different apps if you’re thinking about apps, for example. Yeah.
Jason Liu: 1:06:51 Any other questions? Yep.
Charlie: 1:06:56 Um, you had mentioned like, you had your chief of staff and all these other things, and then you said you prompt the main, uh, agent, uh, thread, and then afterwards it goes… Can you demonstrate how you do that? Like, how you actually do that?
Jason Liu: 1:07:14 Which part? So…
Charlie: 1:07:15 So, so, like, um, talking to one thread and then afterwards it’s going to a different thread to do things. So, I understand like you’re in the thread and it spins up sub-agents. I get that. But I think what you were talking about is having like a chief of staff that’s maybe going to an IC and that’s like a thread. Yeah, so how did you set that up?
Jason Liu: 1:07:42 Yeah, I mean, like, unironically that slide just says like ‘threads can talk to each other, just ask’. Basically, there is a list thread tool and a send message to thread tool that Codex has available. And so typing that is kind of awkward, but really I just say… like sometime last week I was working on slides, can you go find that thread, rename it and pin it for next time? Right? Like I would never type that, but again, voice input makes it really easy. Um, sometimes I’ll say, for example, this talk, I, I think I messed up some slides. Basically, I had a main slide writing system. Uh, and I said, okay, like this slide could be split up in three acts. Make up a thread for each act, pin it, rename it Act 1, 2, and 3, and, uh, review each section. And then once it’s done, review the whole slide. Um, and yeah, I generally like to just ask it. I mean, I think I’m gonna try to add more, like, commands to be a little bit more explicit around thread control, but I think that will just come as the models get better and and as the model is, like, more aware of what Codex can do. Um, but yeah, it really is just like… like one thing you should try to do is just have a thread and just say ‘read all the other threads that are pinned and, like, rename them, and like use an emoji to color code it if it’s, like, ready’.
Jason Liu: 1:09:00 ness. And just seeing it do that will be like a first step, um, to to just like understand how this thread control works. So this feature is like very new but it already has been pretty wild to just sort of have a single thread and you’re just talking to it for like the entire afternoon. Yeah. Sweet. If there’s no more questions…
Charlie: 1:09:19 Yep, one more. With the multi-threading, you said it’s very new. Do you know if it’s available across any other popular AIs? Because I don’t think I’ve seen it…
Jason Liu: 1:09:28 Don’t say that name to me. No, uh, so the question was like, do other AI coding systems have these kind of tools? Not yet. I think I’m sure, you know, I’m sure they will soon. Uh, but as far as I know, this goes back to the eating thing, where like I just have not tried like the latest versions of other tools. Like I don’t know if like Conductor has these skills, but definitely um, I’m sure again like very competitive space, these features will propagate very quickly.
Charlie: 1:09:59 Do you have all threads run on all models, will you have the new threads go back and pull that in as context?
Jason Liu: 1:10:07 It depends. Sometimes you’ll just say find the old thread and like update the model. And it’ll just update the model. Like for example, I like… So when when uh four five five four came out and then we moved to five five, I realized all my old automations were just on five four. And it really bugged me and I was like hey guys, like can we add a feature that like has like GPT latest so we can avoid this issue? And someone was just like tell a thread to update the other threads. And I was like yeah, like so much of it is just ask, you just have to like develop the language to figure out what you want to do, but I think that’s again like the most important, most important thing.
Charlie: 1:10:50 Yeah, so just ask it and then it’ll automatically do it rather than you having to go through and…
Jason Liu: 1:10:54 Yeah, like I think I have an automation that just like cleans up old threads and like by model IDs, um, but uh yeah.
Charlie: 1:11:05 What’s up? Is this uploaded anywhere publicly?
Jason Liu: 1:11:08 Uh, not yet. I’m not happy with the slides enough that I’ll publish them, but I have a blog post. Just search if you just Google Codex Maxing, it’s all there. Uh, you know, three Xs I think. Uh, but I’ll share some more stuff on on Twitter. Go on.
Charlie: 1:11:24 Yeah, so you said that you work with unlimited tokens. Yeah. Does that — since you work so closely with it, do you do you have any sort of advice on how we can — ‘cause a lot of times when you’re doing like loops and stuff you get a lot of like the same message from Codex and that can eat into the token limits pretty quickly. Do you have any advice that you can share on how we can stop — we can avoid hitting those limits depending on you know… Yeah.
Jason Liu: 1:11:48 I think the biggest misconception here is that like X-high will give me the best results. And so there’s like there’s a lot of like X-high maximalists here. I want everything on X-high, right? Like… Like one of I was demoing the like getting a coupon thing and my friend like does an app shot and like hits enter. I’m like, “Why did you turn it on X High?” Right? It ran for like two minutes, like searching like every website available for like coupons. Obviously I can’t give that much advice because it’s something like on my personal computer I don’t really run into limits. But really get comfortable with like low and medium thinking. Like these models are still very, very smart. Right? Like low thinking on 5.5 is like still so much better than prior models. And for a lot of this work that’s not just like, you know, “make me a video game from scratch”, you just don’t need that kind of work. Like my chief of staff I think is like default medium.
Charlie: 1:12:43 That’s very helpful advice. I think just as a follow-up, kind of I guess in a few words what I’m really asking is how do I tell the thread to shut up if it’s telling the same thing over and over again?
Jason Liu: 1:12:53 Oh, I mean the answer is really bad but I just like, I just say like, “If there’s no update just reply with like one word, no update”. That’s one thing. And the second thing too is, I would also play around with like how often these heartbeats are happening. You know, are you running this thing every thirty minutes? Are you running this every 9:00 AM? That’s one thing. And the second thing too is like, do you have some kind of stopping criteria? Right? For example, like I was… I had some argument with like Amazon and I’m just like “Great, like they put me in a 75-minute waitlist, like check every five minutes if the queue is better and once you get to five-minute wait time check every one minute and keep replying until you get my money back”. And I took a shower and when I came back I had like $400 in my credit card.
Charlie: 1:13:41 Is it possible to suggest more dynamic heartbeats so the model can adapt how fast it responds?
Jason Liu: 1:13:46 I’ve not tried too much there, but it’s all possible, right? Because to create a heartbeat is just edit the text file. And so you should definitely be able to create your own heartbeat and then change how frequent or infrequent. I might try to do that actually. I might just say like “Hey, during the weekdays like change your heartbeat to be more active and during weekends or in the afternoons change them to be less active.” Yeah. Like I’ve tried versions of the chief of staff thread where I just set a goal that says like “Never stop” and “You’re only allowed to like set sleep.” And that has also worked pretty well. It’ll be like “Hey, it’s 9:00 PM. Jason… I haven’t seen Jason post a Slack message in like two hours. I’m going to sleep for like five hours.” And that also kind of works. Yeah. But I think the biggest one honestly is just like, people should not be afraid of low reasoning. Like X High is not like X high results. It’s just like “Think more.” 1:15:00 Yeah.
