---
title: "The AI Bill Is Too Damn High – Tokenomics 101"
speakers:
  - name: Eugene Cheah
    affiliation: Featherless
room: "McCarthy Room (Riverside Suite)"
starts_at: "2026-10-12T11:05:00-04:00"
ends_at: "2026-10-12T12:05:00-04:00"
format: "workshop"
recording: "2026-10-12_1105_TokenCosting"
duration: "55m 04s"
source: "qwen3-asr-1.7b"
---

# The AI Bill Is Too Damn High – Tokenomics 101

> Transcribed by qwen3-asr-1.7b with a schedule prompt. Possible prompt artifacts, names heard more often here than in the unprompted Voice Memos transcript (Qwen vs Voice Memos): OpenAI (2 vs 0), Arize AI (1 vs 0), Featherless (3 vs 1). Check them against the audio before quoting.

### 00:00:00 — Featherless AI, Scale Benchmarks & Workshop Scope

So many people are suffering from their token tail. It's a full workshop. Yeah. We are going to talk about tokenomics. So, a little bit of a background kick starting it, right, I'm Eugene from Featherless AI. I'm CEO and founder of, and I'm part of the team behind the RWKV open source project, the first AI model under the Linux Foundation. And we work on next generation AI architecture. That's a little bit of my academic background, and my Twitter handle is Pixel Creator.

But, but more importantly, right, a little bit of background for Featherless AI is that our mission is to make AI accessible for everyone, every language, every dialect, every country. And we do this, we do this by creating the world's largest platform of open models, of over forty thousand open models. This includes specialized fine-tuned like the sunflower model, which is OpenAI's first language model specifically for their language, and a lot of others specialized models. So we are just not talking about the biggest open source model, but specialized language models or specialized use cases, some of it, which I'll touch as well.

And when we well we can go on scale in terms of variety of model, Arize AI also we serve customers at scale. So some of our largest customers are literally serving two trillion tokens a day, and that, and they are serving it at around zero point three per million tokens, or sixty k a day. And they are doing it on around five hundred and twelve AMD GPUs, MI three two five. And for this customer in particular, it's a private four hundred eight hundred B M O E model, and they are serving it around OpenAI 4.8 plus performance, so it's lots of concurrent requests, lots of token volume. That's what we mean by scale, and if you look at it, zero point zero three per million tokens at Opus four point eight class, it's not something that, it's not something that you can get off the public cloud market. This is very specialized setups that is meant to run at the scale that we are talking about.

So that kind of ah leads in into two major disclaimers that I'm going to start off. One, we sell inference. So everything I cover here, we'll cover, we'll touch on any provider. Nothing here is specific to Featherless, but to popularity, we sell inference. So this is from our data points, from our experience respectively, as well. Okay, the other thing is that this workshop is going to cover a lot of optimization at scale, and it probably will not fit you if you're like spending less than ten k a month in your organization. My word of advice for those who are below this metric map, and this is an arbitrary metric, is that if you're spending less than this, you're probably better off focusing on like trying to get strong use cases within your organization, rather than trying to optimize the token spending bill, because a strong use case can bring your organization way more than 10k a month. And that's and that should be your main area focus. And that's not my area focus for this talk.

### 00:02:54 — Move One: Measuring Token Spend by Key

So, what we're going to cover in this workshop is four moves in four major order — measuring, right-sizing, how do you buy your capacity, and how do you track your process? So, measuring.

Yeah, you'll be surprised at how essential this is. Like, how many of you know your total token spend? Can I raise some hands? Wow! Only half the room raised hand. Okay, how many of you can measure at the department level? Those who have their hands raised up, lift you up. And so I see about another half went down. Yeah, and it gets lower and lower down to the team level to the use case level. We like easily thirty forty percent of our customers that reach out to us, the first thing they had was a single shared key that every one in the organisation used, and that was like, how do we even measure that? How do we start from there?

So, this is not an a vendor specific. Like you need to start splitting up your keys. Start splitting up by departments. Start splitting up by major use cases so that you know where all your token spends are — and this is then from there on, it can inform your workload. So yeah, APIs shouldn't be that hard to issue.

Once you know that, right, you can start right sizing things respectively, like, like which team is and workloads spend on it. What was the person waiting for? Like some teams deserve master, please again. Some background agents that running at midnight probably can don't deserve the tokens per second speed that you're paying for it. Yeah, don't pay for ultra on a background agent. That's just done, and from there also you can also look at your usage to better advise, like, ah, dedicated capacity that you may purchase, be for open models or even closed models. Yeah. And also, you can also check which teams spend the most token and whether they are productive. I'll touch on that as well.

### 00:04:51 — Agentic Workloads vs. High-Volume Process Automation

One of the things that you should know in AI workloads, right, and y'all may already experience, right, is there are two major tracks with very opposite economics, respectively. So there is the agentic workload, and obviously there's things in between, where it tends to be like long-running multi-turn agents. Coding agents are a particular example of this — some may have some devs like run coding agents like literally overnight, over a long period of time, we ah with multiple compaction and these things maximize the context ah usage.

Process automation is the other extreme. Think call centers. Think email routing. Think your meeting notes trans and things like that, it's all is much shorter in context in general, though there are a lot of examples that defy this statement, but it's very process driven, very task isolated, and it's a volume game essentially.

So, what do I mean by more usage? And this is probably one of the first major explosion a lot of organization face, in particular when they jump onto the open floor hype or one of the floor clones. So you like from various developer teams that we support, like we for example, we support one of the team, and it was only five devs, and they were averaging around continuously eight running agents for one of the companies we work with, and so they were previously running Opus four point of the API, and they were essentially spending like one hundred fifty k a month continuously running these agents. Some of it is the agents that they use to code. Some of these agents that kick in at night, they were slightly more advanced. They already knew that they should kick things off at the night, by no one's working, but essentially they were spending one hundred sixteen billion tokens a month.

And we literally helped them switch to, a GRM five point three instance. Previously, with five point two, they already upgraded five point three. And that was like a single note that was left in thirty AM, and there were and they didn't see any destruction nearby. So knowing your team workload allows you to make these decisions, respectively. And like highlighted, like knowing what is your processes workload also allows you to split things off. So, like for example, previously they were running everything throughout the day, but they started scheduling some of the background agents when the team was asleep, and it runs through like. So this allows them to utilize their hardware, respectively.

The as I like touched on briefly, the other side of process automation is purely a volume game, extremely predefined. I can give you more specific example of the of the record, but you think of it as like millions per day, or even tens of millions per day, decisions in call centers, lines of quotes, and emails and text messages filtered. Like while the telcos we work with, like they easily hit peak, ten million messages in a single minute. And this shouldn't come as a surprise. Happy New Year is the worst. Yeah. So these and they and they help. And the reason why they filter messages is because they provide a feature about like providing safe phone for kids, so they needed an AI to scan like basically is it profanity, sexual content, etc., etc. So, so this is really not a long context, but a volume game.

Other things like, other things is like microchip processors, approval processes, document processing, these are all volume gain for multiple organizations. So I suspect for the financial institutions here, since we are in New York, a lot of O C R's a lot of documents, a lot of approval, a lot of transaction risk, the is under this category.

Okay, so yeah, if you're not yet tracking your token spend and token view split right, we are open sourcing a, a tokenomics tracking tool, which is just a very fancy way of like, splitting up your API and tracking which API key spends how much, and which is separate from this talk. Yeah. So if you want to again update, feel free to scan this QR code and register. Also, if you want the slides for this talk and when the recording is ready from AI engineering to be notified about this talk, you can also fill up the form there. So, that was about measuring. Yeah, if you are not measuring, you can pretty much ignore everything else I said and start with measuring first because M all your biggest ROI is just by measuring.

### 00:09:09 — Move Two: Open Models & One-Shot Classifiers

But once you have things like, the question is, how do you right size it, and this is where things get more complicated. In general, right, one of the things that we start to observe as AI gets deployed at scale within organization is that there tends to be a pattern of like, because AI is the shiny new hammer, we try to apply it to everything, especially within the innovation, that we just try to apply to every damn thing. And we then and I have seen organizers refer to apply fable to really the silliest of things. So you — there's a lot of models, and a lot of models are really expensive and figuring out what you're trying to do, and where it's an important it's an important step of the process.

I know open models. I mean, a lot of you probably heard about open models, and that's our line of business. Yeah. And so when a lot of companies come to us, for example, to like find open models that fit their use cases, and drop in replacements, this is we generally guide them along their use cases. So this is a general rule of thumb that is probably going to be updated next month because open models released every few months, and same thing in closed source models. Things change rapidly, but there, but the thing is, what we find very exciting actually within this year is things have started to change. Like, in the previous year, the best closed source model is without the better than the best open source model in almost every use case, but as this year approach, right, we are starting to see the gap close. We're starting to see situations where some of the open models are even on par or even surpassing, so like for example, before the Opus five point five update came out, Gemini K three was beating Opus in UI design, in some of the UI benchmarks, and this is like very specific public benchmarks. Within customer benchmarks, right, we were already starting to see open source models beat very specialized benchmarks in specialized use cases.

So like a particular one to highlight is like for example step fund five model, they it was able to achieve near Opus and Anthra quality, despite being like a ten x smaller model in financial benchmarks, ah, financial use cases, and this is it the best coding model? No, but in that use case, it's actually extremely strong and competent. So the landscape is slowly changing, where you can have drop-in replacement for a lot of use cases that are sometimes even just ten x cheaper, respectively, for if you know your workload and you can drop it in, respectively.

Special Shah is also DeepSeek V four one Flash, for some reason, and this model is extremely popular across all workloads, respectively, because it's easy to prompt steer, for a lot of companies. So well, I well the rest of the models I have listed there, they are they are like out for good in their categories, DeepSeek for OneFlash seems to like be quite popular regardless the case, yeah. Though benchmark score doesn't seem to justify why is it so popular.

Yeah, yeah. Another thing to highlight, right, is that recently, like Andy just like a few weeks ago, the a new class of AI models have been introduced called classifiers, one shot classifiers, most popularly known as J, open source projects have already started taking off, like Simple J, and these are models that are entirely focused on just making decisions and classification on task. The main advantage is that because on the technical level they're preview only, they are commonly ten x cheaper than existing models, or even faster at the same time. The downside is, it cannot give a chat response. So these are things that I had trading and happening that you all means want to start looking into, or like, like segments of how you want to improve your token economy.

### 00:13:04 — Agentic Economics: Planner-Worker Splits & Speed Incentives

But oh, but back to the token economy part, there is two major tracks, and it is important to separate out and track this, track them respectively. On the agentic side, we're talking about long running, long context, as I said earlier. And to and you have to understand this workload tends to be extremely cash-heavy, extremely input preview heavy and cash heavy, and there's only a small percentage of output. We see across like agentic use cases in the partner, only one percent of the request is to the output tokens, and this is this is very logical because it takes multiple turns, rinse and repeat. But this also decides your token cost because, like, if you're seeing token cost, right, deployment cost, right, output is may have the biggest number, but even your input well is much smaller, or the cache is much smaller, it gets repeated heavily in this use case. Yeah.

This is not a coding agent workshop, but one of the most common patterns that we see in a lot of organisations, right, is that, is that typically, typically, in terms of like trying to scale up their AI agent deployments within, within the coding teams, right, they tend to have the best agent do the planning and a faster pass the cheaper agent to do the implementation, and then maybe the best agent do the review again. There's a good reason why this split is very common across all the agent frameworks: is that you get the best of both, because like the implementation phase is tend to be, tends to be very token-heavy and process a lot of processes repeated back and forth, like he modifying the file, testing, checking for results, rinse and repeating.

And you and most and most of these agents are especially the, especially like G R M five or G P L, it's more than capable of doing a at the same quality level, as a much more stronger agent, like let's say Astra or Opus ah, respectively. Assuming those agents do the — we have worked with some of the organisations where literally the planning and review agent is the best frontier closed source model, and then the workers that implemented is the G R M five point three flash as an example, or Deepseek four point one flash.

Yeah. One of the observations that we made from this, from this kind of split pattern, right, is that unfortunately, as much as we wish everyone does, so nobody really switches their model to save the company, but these people are lazy to switch the models. In general, you want to give an incentive to switch, or alternatively, if your agent harness is good enough, so like for example, some harness like M, they automate the process like it uses ah the weaker model as a baseline, and then it uses a big the smarter model to optimise switching when it needs a smarter consultation. This, this helps reduce the friction, respectively. It helps improve your token economics.

Like and use at scale, like a main a major rule of thumb that we now give our clients, after going through a few times, is that you want the smarter model, and since it's really so expensive, as a baseline, but you ah, but you want the faster model to the cheaper model to be faster. And that naturally sets the incentive. People want to get their work done fast. If the model is smart enough to get it done, and it's cheaper, they will just use it because it's faster, not because it's cheaper. We had teams where both models, the smarter one and the cheaper one, was maintained at the same tokens per second speed, and it and as long as they had the token budget in excess, right, no one switched, and it — and on the flip side, we have seen teams, which is the other extreme where they set very low budget, and so that if they use a smarter model, right, they just burn it in one day.

Yeah. Then everyone just switch to the to the Dama into the faster model, Dama and faster model because that's their budget. It's like almost as if the smarter model didn't exist. So like be aware about your team dynamics and how they switch. The best trade off is still speed because people are naturally incentivized for it: so faster, cheaper, they will just switch to it respectively.

Once again, reminder: measure and measure through separate token keys. Ah, but this also goes beyond just measuring. For example, like CI CD is a very common use case right now of things running in the background. We have helped set up for teams like faster two points per second that is running for the users like the humans, but much slower tokens per second speed, but much cheaper at scale for the CI/CD pipeline. And this is just the nature of like what's your priority respectively, respectively across the past. Yes, and you can control that at the level. So.

### 00:18:09 — Department Model Labels & Traffic Control

Another major finding is that we had experienced with deploying for teams right — and so like for example, for some companies, how many of you had the fortunate or unfortunate experience of the CEO saying, everyone must implement AI now, and everyone must have a second brain now? Yeah. Multiple organizations have started this. In some cases, like even in the UAE, it became literally mandated by the law of the country, so like a lot of departments are not prepared to know what is the trade-offs, respectively. So a good rule of thumb, right: well, developers tend to know, okay, all these smart models, what's the trade off, between DeepSeek Flash and GPT Five Three, for example? Most, your finance department has absolutely no idea. Your support department has absolutely no idea. They they'll just pick whatever's the first model and probably just go with it.

So, so our general routine is that give department models labels, but not the model names. And this also gives you a lot more control as you switch it out, like advice, I think, but in some cases also like finance agents or Excel spreadsheet agents — these are hyper-specialized agents that were fine-tuned to those tasks respectively. And this also gives you a lot more control. It allows you to swap out the model. It allows you to split, split the traffic and test in between without telling, hey users, please try this. The downside is also, users may sometimes complain when you do that, and you screw that up. But having that control is a good thing. Another pattern that we see, even though you do this, is maybe someone will add a dash preview version, so that's a variant on it, but it gives you control without, without that fixing the communication burden across the organisation, and this is extremely important because, like, like some of the teams that we work with, right, we literally saw members running Llama two models, when we are talking when we're talking about like in this year, because no one switched the option, yeah. So you really, really need to need to need to set this in place.

### 00:20:26 — Process Track Right-Sizing, Evals & Domain Fine-Tuning

Yeah. Hey, process right — process track, a lot of things that I mentioned earlier don't really apply as much to process track, because process track tends to be a lot integrated into the applications, into the workflows, ah, respectively. And that's just by the sheer volume of the traffic, yeah, and for a lot of teams in the process track, right, ah, they tend to be a lot more focused on, focus on the right sizing already by default because if you are running like trillions of tokens, you most departments will not approve it like ten dollar per token per million token. It's just not possible. And so they start they start early with the evals the benchmark and then subsequently finding the model that right size feeds it. Yeah. So it, so the thing is, like, the common question is, how do I know the smaller model is good enough? You don't need to — that's the wrong question to say. You just need to make sure you set up your test cases, your e-dos, and your examples, and then subsequently just do the search process from there. Companies like us will be more than happy to help you because we have over forty thousand models. But then, but then, if you're, it's also not that hard to just download ten models from Hugging Face and just try. So from that, you just do the search, find the cheapest model that kind of like pass most of it, and then you just see how you iterate from there, fine tune from there.

We also have a more specific process that I don't think is unique to us that we'll cover as well, also.

So, for example, like in for one of the major telcos that we work with, right, we actually help their team lower their inference costs, two hundred fifty k to seven point six k a month, switching from existing closed source model just by going through this process of like, okay, we're running the best model first, let's see what models we can split off, and then subsequently what's the models that we can fit in, and then what can we find out. And this was, this was like to build their customer support use cases and so on so on.

Yeah, and to rehighlight back, Jeff, ah and this project, which is an open source project that we did that replicates it. You'll be surprised at how many percentage of your AI traffic is classification traffic. It at the end of day, right, like, yeah, sure, answering the customer support is text, but even then, right, even then, right, a lot of teams, right, already have predefined text output that they put into the context for the model to output for most cases, and you could use a classifier there, and there's a lot. This is a new shift between AI models, and that — this is one of the big levers for this kind of factors in lowering your open bill. Yeah.

So, one example of such a model when it comes to fine tuning. So a lot of models and processors, right, do benefit at this scale when you start to fine tune your own model. On the flip side, for coding models, well there are organisations that will find you models for you like Devin, for example, a lot of developers generally just want the best model. So it's kind of unless you are in this AI space, it's kind of hard to keep pace in fine-tuning the best coding, but on the flip side, for your domain, it might be worth it to thank you a domain model for your use case, especially if the model is much smaller. So for AT and T, for example, when they fine tune their models, they went from being Opus class to like twenty-seven B, to even seven B models, and they have published this.

So, so we all want their launch partners, and you can actually try their own tail line of models respectively, and this is basically what they use behind the scene. And it's all about volume here. Everything I think, most of you understand the scale of AI here and things like that. Yeah, but it's a good example of how you fine tune models to specialize use cases. And once you reach about other organizations that are major example like Shopify has started doing this, and things and it doesn't necessarily need to be, a large scale fine tune. So for example, in the Shopify use case, they did it because Shopify web store had a proprietary programming language for how to configure the web store, and the existing models just weren't that great at it, so they went to fine tune it. But and AT and T also they had examples of that, in particular, like how to arrange the telco towers and things like that. Yeah, very telco specific task. I might not be the right expert to answer that question, but they had those use cases, so they find interest.

### 00:25:11 — Bootstrapping Models With Distillation & Shadow Traffic

You two may handle those use cases, and how do you bootstrap such a model, right, is that the good news is that, you can easily bootstrap things these days because if let's say you're willing, if the best model that is too expensive to run at scale can do the task, you can already get started. So, so, an easy way to do this, right, that we see very commonly is like either you generate your own data or you buy on data, which it's hard. No one likes to do that for some reason. But the but another common part is that they start with an expensive model, and then they deploy it in a small scale, and then they generate, they their data set.

I understand that some labs it's against their terms of service to do that, so you may want to check with your lawyers. But there are labs that openly embrace it. So, for example, Google Gemini — yeah, they are frontier models. They literally have a dedicated API service for this. So, so this allows you to build data sets, respectively, on your use case on a strong model, which you can then, which you can then reduce dramatically to let's say a seven B scale. Alternatively, open source models are more than happy for you to run and still on, so you can run the best open source models respectively as well.

Once you collect them, you can then fine tune the data, and then you can fine tune and respectively like different sizes to see what fits best. So once again, like in the eighty-eighty-eighty example, they have the twenty-seven B and seven B for different levels of the — like your booking, your booking calls may come in at a — I don't think I can say which size goes where, but yeah, your routing cost can come in for the much smaller model, and then it escalates up due to some classifier tricks, as an example, then, then so, so that's how you get the traffic to flow.

You get the draft split, and then you get it at an efficient level. Ah, one of the things that we also tend to focus on when doing this is that, even when they go through the stages and they completely split over their traffic to decide tomorrow, because the best, the best models still progress. Ah, they get, they ah, we typically advise them to cut over a cut a part of their traffic as a shadow traffic to the best model to keep building the data set, and this allows you to ride the wave and progress of the best open model respectively, and the unique economics is actually very important here because, like, one thing that I would like to remind people is people simply underestimate the speed that AI is going, especially for models that can run much cheaper at scale, like you take this laptop as an example.

I like to measure, right, what is the best model that can buy in the most expensive MacBook Pro? And today's best models, be it MoE models at thirty-one B, thirty 2B, 27B size, is already much better than GPT four, so essentially this model that can run on a laptop is already better than the best model that you have last, yeah, give and take. But like, are you on the specifics, but let's just say broadly, and when you fine tune it to your task, that means, if you can run that model on a laptop chip, like that, respectively, you can run it on, on your GPU servers at scale, much more efficient. Yeah.

### 00:28:40 — Move Three: GPU Throughput vs. Latency Cost

So, yeah. So that brings in to the next topic, ah, in GPUs, which ah which is apparently increasingly ah challenge, so yeah, that's where companies like us come in.

One of the things that we realized through a lot of customer conversations — people are rather surprised that things don't scale linearly. What I mean by that is like there is a huge trade-off between tokens per second on an individual request, to, to the cost, of, to the cost of the request itself. So, for example, if you do not care about the cost, you could run the best, the model — I said a fixed model, on the best hardware, on all eight GPUs, batch size one, and you could get, let's say, ah, hundred tokens per second, two hundred tokens per second. If you like, if you really want, and this is not our specialization, like you really want like thousands of tokens per second, that's why companies like Cerberus is more than happy to offer that as well, and it costs even more.

So, there are definitely use cases where this makes sense, but is but a lot people fall into the trap of thinking, hey, so if I have the speed, maybe it's half the cost. It's not even close, like a rule of thumb that I sometimes say, right: for every two x of speed, it's three x of cost, and it pounds, in this specific example for GIM five point three. It's like one agent on a node. It's like you can see the same server, hundred and shell tokens per second for running individual requests, but if you run sixteen requests in parallel, you get about half the speed for all the requests, but you literally ten x the output total output. You — and that gives you like literally around nine to ten x lower in cost in just like changing is changing the target speed profile. And so when a lot of customers ask us like, how do we deploy things at scale, sometimes the first question is, what's the model quality you want, and what's the tokens per second speed that you want?

And then we start recommending the amount of GPUs, the amount of deployment, and this, this target tokens per second is an imp important trade-off, essentially.

In more extreme cases, right, if you really, really can run things at ten tokens per second — really, anyone does, but there are some weird newspapers that allows it — you can get even more incredible total throughput volumes, respectively. Typically, we see customers choosing between twenty, thirty, which is kind of like you know ChatGPT speed without too much thinking, all the way until like sixty, which is like the faster speed, and if they want 120 to like, it tends to be this range. Most take between twenty thirty to sixty, depending on how long the chain of task is. Yeah.

So, yeah, I kind of mentioned it. You generally only want to pay for the tokens per second speed when it's in front of someone, whether it's in front of your developers, or whether it's in front of your customers. Those are the trade-offs that you want to make, and even within them, right, we have seen cases where even just the, a slight tweak down, right, can easily save two times the cost, but has near minimal impact for the cost, customer, respectively. Hey, once again, background drops. Please don't run it at 120 because it makes no sense when you do that. Yeah.

### 00:31:57 — Dedicated Capacity, Base Load & Cloud Spillover

So, once you have previously, I said measure, right, you are able to see your traffic pattern. So a lot of traffic patterns tend to be bursting some nature, we some baseline because someone's always running something at some right, especially background agents, and this is where you can you can also plan your deployments respectively. I know it since we are dealing financial institute, a lot of financial institute may not have this as an option, but because they are only dedicated instance deployment, but if for some cases where people allow mix, where they have dedicated instances with some first to the public cloud, this is where you have some flexibility.

So like, the biggest cost savings you have right now in the market is deploying a dedicated instance if you know your base workload. So as long as you fully use it, you can have a savings. But it's a bit more complicated depending on your traffic pattern, and a lot of these things can be tested and validated. And this is like essentially what any inference company will work with you. It's not unique us. We'll be like, hey, this is — do you know your traffic volume? Can you share some data? And then subsequently, we'll be like, okay, we propose this much, you can try switching to this, and then we generally we can generally show you your savings respectively.

And in some extreme cases where you have a loss, right, you may want to have limit your dedicated capacity to a certain amount, and let the spikes spill over to the public cloud, respectively. Different use cases have different right sizing, but this is something that, by measuring your token workload once again at the beginning, you can start exploring these options. And the savings can be huge. Like once again, like for the customer service to fluent token, and that was like four point eight Opus Plus model, they are paying less than zero point zero three dollars per million token. Like you can't get that price anywhere. Yeah. So yeah, simple disclaimer is, the spikes really matter. It's very rare that someone has a short spike at the extreme end, where they like run a workload every month at that scale, and then nothing else matter. So usually some base workload exists. It's just how big of a base workload, yeah.

### 00:34:14 — Move Four: Review Bottlenecks & Organizational Trust

So yeah, then the last one is check your processes. This has nothing to do with inference optimization from your words. Some slides may sound that very critical of AI, but I'm generally an optimist. Just, just I just want to lay it out because things can be quoted out of context. So, one of the — a Pandora's box question or problematic question that we are starting to see is that a lot of organizations are starting to ask, hey, I'm ten x or hundred x in my token spend, but I'm not seeing results at that same scale. Respect, like, like we are starting to see reports from various organizations, like, where AI res interprodutividade only has six percent productivity or seven percent, which is not the scale that a lot of us were promised or we envisioned.

And, and unfortunately, this is the case because we are at a stage where everyone is just taking a giant hammer and hammering everything. It's not that AI is not useful. It's just that we tend to just spam it everywhere, and yeah, if there's a — so there are times where I just want to like, can you please not do this as another co-pilot, because it's just too much of it. You need to think of how use it respectively. Ah yeah.

And another major issue that we start to realize is that, while a while AIs are being deployed at a larger scale and more frequently, and people get faster in creating stuff, the organisation processes may not go as fast. The first bottleneck for coding environments would be the CI CD's, and you may heard that CPU becomes important, respectively. Another thing is like it's just the processes in place, respectively: how do you trust? How do you review? I think the previous speaker spoke at length at trust. Yeah, the lack of trust is what slows things down, and this could be rightfully so. Maybe the agent, the AI agent was badly designed and shouldn't be trusted, but it also flips an inverse. What if, hypothetically, that agent was doing everything perfectly, but people weren't trusting it because it didn't had the test in place, it didn't had the validation, and it didn't build that trust? That slows things down, and these things, make these things have a much bigger impact.

So once again, be very wary about how the hammers being used because you need to make your adjustments accordingly.

So, and this goes back into saying like tokens are not work done. You have — and one major example that I would highlight right is, is this happens more often than we like is that sometimes spamming more agents and more tokens sometimes means you get the same task done multiple times. So one of the anti patterns that we started seeing, right — yeah, well, in overall, the engineering organizations improve productivity by clearing more issues quickly, automating clearing of issues. For example, like when so like, when the QA and things like QA things reporting an issue and the agent automatically fixing it without even an engineer, those are great.

But one of the one of the common issues that we start facing is that people get ambitious really fast, and they instead of doing small changes, they start doing huge changes, like changing like half the code bases or one quarter of the code. Sometimes when this is planned with proper testing, it's great, but a lot of times, right, the change is happening at such a large scale, right, the bottleneck becomes the on the reviewer to merge the code, and because everything is done on such a large scale where a human cannot process it, one of the common anti patterns that we start seeing is that okay, someone has implemented the there's issues being done, and because it's so much code been generated, issues that do not have code generated do not get review — ah, did not get picked up by the reviewer.

So someone needs to generate the code using an agent, and then when the agent generate that giant pile of code, which should be a lot less if they were to like properly review prompting and stuff, the reviewer look at it and it's like, huh, this is too many code. Can I ask my agent to highlight what changes are relevant, and then re-implemented, and in the end, they started rebuilding in my head with their agent, so they literally discard the first agent's job. There's multiple good reasons why this happened. For example, like reviewers want to maintain control and quality of the main codebase.

But this is a very common example of a bottleneck where you where if you didn't think the process through, you're just destroying more and more things, respectively, and the speed is not increasing, so review your processes. A good way around this is naturally, to instill better testing practices, and so that it can automate that and filter out. Also, reminding teams to like, hey, split up your merch or make it more modular to make it easier for the review process, like things like these are human processors. It has nothing to do with AI, and yeah, we tend to get too excited about AI sometimes. So.

This is not just a code problem. We are starting to see, like, because AI agents have been deployed to make decks — this deck was entirely, design-wise, was generated by AI. I went through multiple iterations. It's not a zero shot. I went I went through every page like at least thirty times to change the text, but the design was done by agents, and sometimes within your organisation, your son you notice people that just one shot from the entire slides, things like that, and they start passing it around. They start doing meetings, and then it's on everyone else to filter what's the inconsistency, what's the misalignment? Why is these two data points completely different? This is also another example of like, same as the previous example, people generating code, or people generating slides, or people generating report, and the burden shifts onto some human down the road to verify things, so be very aware about this organization change.

Why am I still an optimist, is why I don't think this is necessarily a bad thing in itself because the problem is everyone is to learn, and this is where we just start adapting our processes accordingly, and the other major thing is that, well, there are a lot of common examples of how to use AI, you all know your subject domain for your organization better. And there will be use cases where an AI to be used in place to automate a process, for example, that could save your organisation or millions of dollars, or increase your productivity at a grand scale, that only you will know, and not me as an AI expert will come in.

I like to point to the most famous example that Walmart had, where — was it Walmart? I believe it's Walmart — where they literally started putting a stool, a stepping stool, inside every single truck. And that was like the simplest change, but it lowered their medical bill by the millions because it lowered back injuries across for heavy goods respectively. It it's some it's something specific like that with the use of AI, and it may not even necessarily be ah the best AI model that actually can have the biggest impact. Yeah. Yeah. So.

### 00:41:48 — Slop Grenades, Ownership & Measuring Real Outcomes

Doubling down on my previous point, one the term that was coined by — that there's been trending is called the slop grenade. So generating lots of things with AI and then leaving it to someone else. Do not let slob grenades happen, and this is a cultural thing and the processing. Yep. So, the some guideline rules that I would set and — this is more subjective, people disagree — to prevent the slop grenade from happening, enforce that people should own what they hand over or generate. So, for example, these slides, I own every text that is generated. If there's an issue, that's on me. It's not my AI agent who has an issue. I do not get to hand it off to an agent. I must own the responsibility. Same thing for code. Same thing for anything that is generated, and you also have to measure outcomes, not usage, and so this gets more complicated because I don't want to explain too long to this, but like there is like this simple formula that I think people take as a rule of thumb: tokens times price divided by outcomes. Someone actually used — and this help guides other things. Smaller, more efficient models, faster model, bigger model. Yeah, yeah, yeah.

But, what I don't like about this is that, another analogy that I'll use about AI is think of it more as like supplements or steroids that enhances a team. Think of it as like if let's say I use construction work as an example, let's say I can use it to increase the muscle mass for the vocals, for example. Not that it's ethically to do so. I know that's not the case. But let's just say, as an example, some teams will have it, and they improve productivity. Some teams don't. But even without that, it's quite clear some things are much better than others, and so the AI being a supplement here, right, doesn't necessarily change everything, so you need to be aware that outcomes, outcomes are hard to measure respectively, but it's something that you can start measuring and observing. Most importantly, back to the first step, measure, you know the token spend per department per team, and you can start seeing which game is performing better, and you start seeing how much tokens they use. You can start seeing what works.

Yeah, yeah. As I said, this floor measurement. So yeah, that's the high-level overview, respectively, and if you want to like kickstart your process internally within your organisation, you can you can either talk to me and my team, or like you have six questions for you to start asking, like, who thinks when? What was the what must you test to use a cheaper model? What's your base load? What's the default model for each team? And which workload has a person waiting that you need something faster? Yeah, and how much of what you generate is useful? These are like guideline questions for you to start asking to refine your tokenomics. Yep.

### 00:44:46 — Q&A: Fine-Tuning, Harnesses, Caching & Auto-Routing

So, Q&A. Yes.

So when you mentioned fine-tuning models, is that like SFT, fine tuning, or is there like other sort of funding methods that you have, depending on specific use case?

Okay, so in general, it depends, but I don't think — well, we do provide fine-tuning services and tends to be SFT. I — there is a lot of methods that are, happening right now, currently. I know, R L is very sexy right now, especially in the coding environment, because it's easy to generate the data in the loop. But what we realize for a lot of organizations, especially like in New York, for example, financial institute, things like that, there's not really an RL environment for financial task, for example, and so it tends to be more SFT based, and there are also other organisations like adapted, adapted AI, where they are starting to also automate the refinement of the data sets for you to train the model the SFT base or tinggi tinggal yang machine base.

Focusing on this might be the wrong question; it, to me, generally, what people should focus on is the evils, respectively. So here's a good example. For Jeff, that came when the Jeff model came out, and then we did the open source project called Simple Jeff, which replicates the Jeff API for existing open source model, a lot of people quickly realized that the Qwen twenty seven B was really good at this task. And there were multiple teams that went to fine tune the question assembly. I, it — on outside because we are trying to take the approach like our company, but we want to extend any model, and then we just build the API on top of it. We started taking the approach of like, hey, we see the benchmark, we see the existing model, what does we prompt tune it is it? It's not even fine tuned. And then, and then two weeks later, as Hugging Face, when they do all the benchmarks at scale, like over ten thousand questions and things like that, somehow, my prompt tune twenty seven being beat most of the twenty seven being fine tuned.

So, yeah, this not a flex because I'm quite sure now my models — the recent Cloudflare fine-tuning model that was done on a more proper basis, but, it's more of like, the tuning method might — the tuning method might not be the right question. It's more like, what's your evals and what's the results that lead to it.

Follow up for that. You need — do you need to collect golden traces for the setup data you need if you're doing SFT or whatever it is, or can you do — let's just say, like your for your, the frontier models get like 80% accurate, right? So maybe 20% of the time it makes some sort mistake. If you fine tune on that, won't the fine tune model also get like 20% of it wrong? Are you supposed to filter that out somehow and get the only the golden traces for training, or can you just throw all the traces in and train?

So this is more personal opinion, technically, because so the — generally, I try to split the dataset because people are busy filter. Ah, so I try to at least get a golden trace dataset, and the golden trace dataset maybe is much smaller ah, respectively, and we just ah, and we then we fine tune that with a much heavier weightage, ah, as well. But there have been multiple papers, especially, as, especially, rethinking tokens, that even the non — the failed task, right, the as much model, when you fine tune, tends to benefit from the training because it actually improves the thinking process, even if it got the wrong answer. This is something — this is really a good case of like, try and then measure. And it works seventy percent of the time. Sometimes you just you need to increase the golden trade ratio.

But it's a general rule, I think for a lot of customers, when they're trying to SFT down small model, the general rule of thumb they are trying to keep is: can I keep to the near same level of performance, and maybe the eval is down one percent, or even in some cases it improved by ten percent because the base model wasn't special? More importantly, does it hundred x reduce my bill? So that's usually the motivation, respectively. Yeah. Golden samples are critical if you want to do beyond the baseline, and sometimes that requires humans to actually look at the data, which as much as I love my customers to look at their data more, people don't like to do that.

Hi. How would you deal with code spent that mostly comes from cloud code and all the in — because then your model is fully locked into the about the ecosystem, I suppose. Do you have any advice on how to build this type of?

Yeah, unfortunately, Clockwork is the example of being locked in, so sometimes the first movement is actually getting that was to switch the harness. Harness it seems to be a lot more flexible thing right now. Like, we have seen organizations that are like fully locked in, like, I'm only going to use Devin only or Em only. We have seen organisations that are more flexible, like literally they are running your, they are creating your own agent hunters, and my response is like, why? Why are you creating your own agent hunters from scratch? But like a common one that we see people switching to is, for example, Pine. And this is also because there's a lot of excitement around it. People have shown that, even with the same model, it gets a much cheaper cost per token because of the way it's simple and efficient, and from there I was — the I think the beauty for the AI coding agent harness is that it's easy to adapt because it's so lightweight. We've seen a lot of teams that just start prompting their agents to make modifications, and once they have that, then they start getting the cycle of experimenting and changing.

A common way to get them to be very incentivized to change is to actually introduce a good enough model that is really fast, like hundred tokens per second, and then suddenly they start giving that a try, and some people start like some of the first shift will be, they still use their clock code as their planner, and then they instruct the clock code use this AI agent with this model, to do the implementation, hence the planner implementation sleep. It's more, yeah, and that's where the transitions start to happen. It's not easy to change people's agents.

But yeah. Yeah. So an example is could be perhaps to expose this smaller agent as an MCP in inside cloud, or are having these tools where you can offload? I suppose is that something that you would say could work?

MCP is more tricky because MCP agents, unless done right, don't have access to your office system, so typically it's just something very simple like Py is installed and they run the batch from — but pi is, yeah. That that's usually the case. Thanks.

Can you share, if possible, any case studies about caching solutions being very thoughtful about caching and the outcomes that have been generated for specific customers, even if it's anonymized?

What's the specific challenge you are talking about? You do one caching. You okay? Is your question more like you want annoyance — you do wait, you do not want customer private information to save in the cache? No, so like with customers that you've worked with, have you got any case studies you can share about the outcomes that they've been able to generate because of being thoughtful about how they cache and different sort of caching methodologies?

I think this one we can take offline. But in general, for our larger scale customers, we deploy them with dedicated cash, so it kind of solves a lot of PI issues because it's the cash is only for that customer. Yeah, and obviously there's prompt formatting, but I may need to dig in more specifically, like what you're trying to ask here. The short answer is yes, but there's a lot of way to interpret this. Yeah, sorry about that.

Thoughts on turn or task-based auto routing?

Oh, I'm a — I'm generally more towards task-based, mostly because it's much easier to control expectation at a department, organization level. So this is back to like, hey, a department may have a labeled model, and I kind of auto routed that model for them.

Ah, I know, there's a lot of ah hype right now for turn-based auto routing. One of the issues with turn-based auto routing is that you need to lock down your turn-based auto rotting to let's say at most two to three models, in not your cash will be extremely inefficient to be used. So for example, we have seen ten base for GRN five point three and GRN five point three flash makes sense, but if you put in thirty models and it keeps changing your model every turn, your low cash heat rate will start to wreck your optimization. Yeah, that's what I've seen before. I like it. Yeah, but tomorrow stands to be okay. Task an organization makes the most sense. Yeah.

All right. Okay, thank you very much. And if anyone wants to find me specifically for more questions, feel free to just come up front and ask me or find me by the side, once the next occurs, yeah.