When Will We Train Once and Learn Forever? Insights from Dev Rishi, CEO and co-founder @Predibase

Summary

In this episode of Inference by Turing Post, Devvret Rishi, CEO and co-founder of Predibase, discusses the shift from static models to continuous learning loops and the rise of reinforcement fine-tuning (RFT). Dev explains how the “train once, learn forever” paradigm is already emerging at cutting-edge companies, where production models continuously improve through feedback loops rather than remaining static after initial training.

The conversation covers Predibase’s pioneering work on reinforcement fine-tuning, which allows customers to customize models using reward functions instead of large labeled datasets. Dev shares insights from his experience as a product manager at Google Assistant and explains why agentic workflows are still in early innings, with most implementations being quite brittle when users go off the “golden path.”

Dev also discusses how Predibase pivoted after the ChatGPT moment, betting specifically on specialized and customized LLMs rather than general-purpose deep learning. He shares his perspective that the future belongs to narrow, automation-oriented AI use cases in enterprise, famously noting that “generalized intelligence is great, but I don’t need my point of sales system to recite French poetry.”

Highlights

”Train once and learn forever is actually here today”

yt-dlp --download-sections "*0:37-1:20" "https://www.youtube.com/watch?v=bYsuivSB0qc" --force-keyframes-at-cuts --merge-output-format mp4 -o "bYsuivSB0qc-0m37s.mp4"

“That world is actually here today. Most of the time when we see customers using models in production, they’re taking a model someone else has done 99% of the heavy lifting on and then they’re doing a last mile 1% customization. The trend is going to be a shift where people stop using a static model and instead have a pipeline that allows them to improve the model continuously while it’s in production.” — Devvret Rishi, 0:37

”If you can measure it, you can improve it”

yt-dlp --download-sections "*1:50-3:00" "https://www.youtube.com/watch?v=bYsuivSB0qc" --force-keyframes-at-cuts --merge-output-format mp4 -o "bYsuivSB0qc-1m50s.mp4"

“Reinforcement fine-tuning takes a different approach. Rather than needing large amounts of labeled data, you can actually do fine-tuning with really small quantities of data. Think about a dozen examples or so. And you add in this concept of reward functions. If you’re teaching a model how to write code, you might write a reward function that says you’ll get plus five points if you get the formatting correct and another 10 points if it compiles and another 20 points if the unit tests pass. The goal with reinforcement fine-tuning is if you can measure it, you can improve it.” — Devvret Rishi, 1:50

”90% accuracy becomes sub-50% user experience with multiple calls”

yt-dlp --download-sections "*6:43-7:30" "https://www.youtube.com/watch?v=bYsuivSB0qc" --force-keyframes-at-cuts --merge-output-format mp4 -o "bYsuivSB0qc-6m43s.mp4"

“My view is that right now the way that these agents get built are quite brittle. A lot of times people have built a really compelling demo that works well if you are on the golden path. But if you go off the golden path then the model is triggering a number of different error areas. It’s a very simple thing. If you’re only 90% accurate on a given LLM call and your LLM has to make five different calls, you’re already sub 50% in terms of the user experience.” — Devvret Rishi, 6:43

”I don’t need my point of sales system to recite French poetry”

yt-dlp --download-sections "*11:50-12:40" "https://www.youtube.com/watch?v=bYsuivSB0qc" --force-keyframes-at-cuts --merge-output-format mp4 -o "bYsuivSB0qc-11m50s.mp4"

“My favorite customer quote is generalized intelligence is great, but I don’t need my point of sales system to recite French poetry. Most customer use cases look like something like one of our customers, Checkr, does. They look through employee background checks and are looking to extract out very specific information with respect to criminal codes and violations. They need to be able to do a really high quality job on one particular set of tasks.” — Devvret Rishi, 11:50

”We aren’t going to live in a world where one model rules it all”

yt-dlp --download-sections "*0:00-0:20" "https://www.youtube.com/watch?v=bYsuivSB0qc" --force-keyframes-at-cuts --merge-output-format mp4 -o "bYsuivSB0qc-0m00s.mp4"

“We aren’t going to live in a world where one model rules it all. You’re going to see a mix of open-source models, closed and commercial models, different size parameter ranges, and just like any software tool, people are going to choose the best tool for their individual task.” — Devvret Rishi, 0:00

”Open source is 6 months ahead of schedule”

yt-dlp --download-sections "*21:35-22:30" "https://www.youtube.com/watch?v=bYsuivSB0qc" --force-keyframes-at-cuts --merge-output-format mp4 -o "bYsuivSB0qc-21m35s.mp4"

“Today Deepseek R1 or V3, Qwen 3, Llama 4, these models are not only on par but many times actually in the benchmarks doing even better than the leading commercial models. To me that’s actually 6 months ahead of schedule. I would have thought end of 2025 would have been the earliest that we’d see open source models beat commercial models.” — Devvret Rishi, 21:35

Key Points

• Train Once, Learn Forever (0:37) - The paradigm shift from static models to continuous improvement pipelines in production
• Reinforcement Fine-Tuning (RFT) (1:50) - New technique using reward functions instead of large labeled datasets for customization
• Reward Functions (2:15) - Customer-written rubrics that grade model outputs and incentivize desired behaviors
• Continuous Feedback Loops (3:31) - Healthcare companies using LLMs as judges plus clinician labeling for continuous improvement
• Production Data Collection (4:38) - Predibase feature to automatically collect prompts and responses from deployments
• DPO and Novel Techniques (5:04) - Direct preference optimization and research on using RFT with user feedback
• Agentic Workflow Definition (6:11) - Multiple LLM calls plus tool calling capabilities
• Brittle Agents (6:43) - Current agents work well on golden path but fail when users deviate
• 90% Accuracy Problem (7:05) - Five 90% accurate calls compound to sub-50% user experience
• Weekly Breakthroughs (8:28) - AI space moves faster than cloud or mobile shifts with breakthroughs every week
• Post-ChatGPT Pivot (10:18) - Predibase focused specifically on specialized LLMs starting 2023
• Specialized AI Use Cases (12:25) - Enterprise majority moving toward narrow, automation-oriented AI
• Inference Challenges (15:09) - Production inference requires 99.9%+ SLAs, fault tolerance, and cost optimization
• Intelligent Inference (17:01) - Future where inference pipelines hook into post-training for continuous improvement
• Evaluation Gaps (18:08) - LLM evaluation still largely “vibes” - big open problem in the industry
• Open Source Progress (22:06) - Deepseek, Qwen, Llama now beating commercial models, 6 months ahead of expectations
• Reasoning Token Transparency (22:06) - Open source reasoning models expose tokens, enabling better evaluation
• AGI Perspective (23:48) - Dev focuses on practical specialized intelligence rather than AGI
• Hype Bubble Concern (26:55) - Risk of disillusionment if companies chase far-fetched use cases instead of practical ones
• Happiness Advantage (27:34) - Book recommendation about how happiness drives success, not vice versa

Mentions

Companies

• Predibase (0:37) - Dev’s company, pioneering reinforcement fine-tuning platform
• Google (7:15) - Where Dev worked on Google Assistant
• Checkr (11:55) - Customer using narrow AI for background check extraction
• Marsh McLennan (24:10) - Fortune 200 company example for AI adoption
• OpenAI (8:43) - Competitor in the space
• Anthropic (8:43) - Competitor in the space
• Mistral (8:45) - Competitor in the space
• Deepseek (8:46) - Open source model provider beating commercial models
• Meta (8:47) - Llama model provider

Products & Technologies

• Reinforcement Fine-Tuning (RFT) (1:50) - Predibase’s pioneering post-training technique
• Ludwig (18:20) - Open source fine-tuning framework
• Lorax (18:25) - Predibase’s open source inference technology
• Turbolora (16:25) - Software technique to increase model throughput by 2x
• DPO (5:04) - Direct Preference Optimization technique
• BERT (10:10) - Pre-trained model popular on early Predibase platform
• Deepseek R1/V3 (21:35) - Open source models beating commercial ones
• Qwen 3 (21:38) - Open source model
• Llama 4 (21:40) - Meta’s open source model

People

• Devvret Rishi (0:23) - CEO and co-founder of Predibase
• Sean Achor (27:34) - Author of “The Happiness Advantage”

Surprising Quotes

“The pace here is truly that you will have a breakthrough on expectation about every week.” — 8:35

“In end of 22, OpenAI democratized deep learning more than any of us.” — 9:39

“Some of the most prolific AI engineers are ones that just got started in the field a year or two ago.” — 10:46

“If you ship something that you’re not at least a little embarrassed by, you’ve waited too long.” — 25:50

“AI has probably been a place that for outside of the top 1% of organizations has overpromised and underdelivered between 2012 and 2022.” — 26:25

Transcript

0:00 We aren’t going to live in a world where one model rules it all. But it’s incredible at the rate of innovation that we’ve seen in open source. The world that I see tends to be like rather than artificial general intelligence, it’s like practical specialized intelligence. The pace here is truly that you will have a breakthrough on expectation about every week.

0:23 Hello Dev. Thank you so much for joining me today. Of course. Happy to be here and thanks for having me. Well let’s start with a big picture if you can draw me big picture. When will we train once and learn forever?

0:37 It’s a great question. I think that you know that world is actually here today. Most of the time when we see customers using models in production, they’re taking a model someone else has done 99% of the heavy lifting on and then they’re doing a last mile 1% customization. The trend that I think is going to go towards what you said which is like train once and then learn forever is going to be a shift though where people stop using a static model, you know this one model someone else trained, and instead have a pipeline that allows them to improve the model continuously while it’s in production.

1:10 So we’ve started to see some of our early customers already put these types of pipelines in practice and that’s what you know I’m most excited towards being able to build towards as well. Well that’s super cool. One of the things that you, I don’t know if you initiated it in Predibase, was post training technique RFT. Can you tell us a little more about it? Do you think it’s the big unlock or is it just like another tuning trick?

1:35 Yeah, it’s a great question. I think we were the first end-to-end platform to offer reinforcement fine-tuning RFT when we released it just about a couple of months ago. Sorry for interrupting, if you also can briefly explain what it is for.

1:50 Yeah, so reinforcement fine-tuning takes a different approach towards the way that you can fine-tune or customize models. The real kind of underlying intuition is rather than needing large amounts of labeled data, which is what you need for traditional supervised fine-tuning, you can actually do fine-tuning with really small quantities of data. Think about a dozen examples or so. And you add in instead of labeled data this concept of reward functions.

2:15 Now reward functions are essentially like rubrics that any individual customer can write that helps you grade a model’s output. And the idea is that the model will learn how to be able to actually adapt its behavior towards the types of things that you want to incentivize or reward. So, as an example, if you’re teaching a model how to write code, you might write a reward function that says you’ll get plus five points if you get the formatting correct and another 10 points if it compiles and another 20 points if the unit tests pass.

2:46 In this way, you kind of teach a model how to be able to actually generate its outputs based on really kind of objective criteria. So, the goal with reinforcement fine-tuning is if you can measure it, you can improve it. And then I think you asked, you know, is it a large shift that we see for the future or is it just another tool in the toolbox?

3:04 I think honestly today reinforcement fine-tuning is one of a few different techniques that is really helpful for where customers can start to tune their models. But I think where we’re going to go with reinforcement fine-tuning is going to fundamentally shift the way that people customize models. Today reinforcement fine-tuning is a one-off training process you do. But where I see RFT really going is becoming part of this continuous feedback loop where models are getting better online and I think that is going to be a paradigm shift in how customers are tuning and training models.

3:31 Do you see it already companies implementing this feedback loop? The very early end of cutting edge companies. Yes. So I’m working with a couple of companies in the healthcare domain that are building co-pilots and assistants for their end patients and what they do is they have a lot of interaction data with their end patients and they actually are bringing right now a combination of LLMs as judges to be able to verify how good was that conversation but also clinicians that are able to actually label different conversations and bring that in as kind of continuous feedback.

4:07 So rather than needing to take, you know, the months of labeling that you’d have, just starting to take the labeling that could be done over a handful of conversations and feeding that into an RFT loop. Today, it’s very early and I only see the most cutting edge companies really doing it. But this is the type of pipeline that I think more and more companies are going to do as we get into more continuously improving models.

4:38 What’s on the way for them to start doing it? Yeah, it’s a good question. I think the biggest thing is a feature that we’re working on. You know, there’s actually two things. I think one of them is a feature that we just launched in Predibase which is a very simple thing that allows you to be able to collect prompts and responses from any of your production deployments automatically. One of the key things with tuning and training models always tends to be data. So the very first feature that we have is one that makes it easier to construct these data sets using live production traffic.

5:04 The second thing is making it easier to be able to learn from feedback. So rather than needing large amounts of labeled data, how do you directly nudge a model with small amounts of feedback? Now there’s lots of techniques here in the platform like DPO, direct preference optimization and others that help you learn from feedback, but we’re also working on some novel techniques that are more on the research side of Predibase today for how you use reinforcement fine-tuning with user feedback data.

5:19 Is anything coming soon? I think you can expect that we’re going to be publishing a little bit more about this in the next few months. And really what we’re going to be talking about is seeing what we’ve experienced with real life agentic applications in particular and the type of feedback that users are looking to be able to give. How do you systematize that into a continuous stream? So the very first thing we’ll release is just going to be showing how small amounts of feedback can make performance impacts and then you can scale out those performance impacts with the more and more feedback you get.

6:11 What’s your perspective on agentic workflows and agents in general? My perspective on agent workflows is that they’re in the very early innings and so a lot of things that people are building are a little bit brittle today. The very first thing we need to do when we talk about agentic workflows is really define what an agent or agentic workflow is.

6:21 I think about an agentic workflow as having two key components. The first is having a chain of multiple LLM calls. You know, if you think about a single, let’s imagine you’re doing document classification. We don’t think about that as an agentic workflow because it’s a one-shot like process. Whereas, if you’re having a conversation with a bot, for example, to understand your medical diagnosis a little bit better and schedule follow-up, that involves multiple turns. So, the first piece, I think it’s multiple turn, multi call.

6:43 And then the second piece of an agentic workflow is that it will likely have the agent be able to do tool calling, make calls towards other functions that it can actually use to fulfill a request on behalf of the user. My view is that right now the way that these agents get built are quite brittle. A lot of times people have built I think a really compelling demo that works well if you are on the golden path. But if you go off the golden path then the model is triggering a number of different error areas.

7:02 It’s a very simple thing. If you’re only 90% accurate on a given LLM call and your LLM has to make five different calls, you’re already sub 50% in terms of the user experience. So that last mile of quality with agentic applications becomes really really important. I saw this. I was a product manager on the Google Assistant back in the day which was one of the first AI agents not using generative AI but using more classical NLP methods. The bridge we need to make as an industry is getting to more robust agentic workflows.

7:50 Thank you. Speaking about Google and your previous experience, you are a product first founder in a very heavy research heavy space. Is it hard for you? How do you keep up and what areas of research are you following most closely?

8:05 I really enjoyed actually. So my background was I did my undergrad and masters in computer science and had done some initial research as well in CS before I like to say I sold out and became a product manager. My real interest in product was because I saw it have kind of like the most cross-functional impact. In research you usually go very deep into an individual space whereas in product you get to see how combination of research works with engineering and design.

8:28 And so that was like my main intent. Even when I was at Google though, I was working closely with research teams. And so I’ve always felt very comfortable working in areas that are fundamentally based on things that are still developing and core research that’s happening. I will say this space moves much faster than anything else that we’ve seen. Much faster than I think when we were talking about some of the shifts to clouds, shifts to mobile. The pace here is truly that you will have a breakthrough on expectation about every week.

8:43 When we were planning our reinforcement fine-tuning launch, one of the most stressful things was knowing, is OpenAI or Anthropic or Mistral or Deepseek or Google, Amazon, Meta, are they one of them going to put out something massive on the same week and just suck all the oxygen out of the room. So, it is dizzying to be able to keep up, but I think some of the core background and research that I had helps a lot for just being able to understand some of the core techniques people are employing here.

9:23 I remember working with a few ML companies before the ChatGPT moment and when it happened so many needed to pivot. If we go back to that November December of 2022 when it came out. Right. It’s just so fast. It seems long time ago. What changes did you have to make when everything started to just explode?

9:39 It was a really interesting moment for us because when we started the company we had a mission to democratize deep learning. And so we had built interfaces and a product and infrastructure to make it easy for people to train their own deep learning models. And then the way I like to just reflect on it is in end of 22 OpenAI democratized deep learning more than any of us. And it had done it through these large pre-trained deep learning models that were able to start having conversations in one shot.

10:05 The reason I say it was a really interesting position for us was that as a deep learning oriented company, we had started to see these types of workflows already be popular in our platform but at a much smaller scale. In early 22, the most popular piece of functionality on our platform was you could pick a pre-trained deep learning model like BERT as an example and you could fine-tune that model on your data.

10:23 And so the reason that people were coming to us was already because they wanted to start to adopt some of these pre-trained transformers and adapt them towards their data. But the types of use cases, the user journey and the persona just fundamentally shifted, right? When we were talking in 21 and 22, it had to be an NLP engineer, someone who understood the intricacies of a BERT or a T5 or an image in computer vision like VIT in order to be able to really understand the platform.

10:46 And fast forward to today, I think some of the most prolific AI engineers are ones that just got started in the field a year or two ago. We really needed to, depending on how you think about it, I often think about it as a complete pivot, but it was a real focusing in on our product to say rather than like we’re going to help you build deep learning models across the world in general, we decided to really pick and make the bet at the beginning of 2023 on just this one technology, large language models.

11:07 And we decided to make the bet that the future of LLMs in production were going to be specialized and customized. And so we wanted to build that tuning and post-training stack to make that happen. And then what we quickly found out was inference was going to be a huge part of this game. And so we went really big in that world as well later on in the year.

11:23 There’s so many things to unfold here, but let’s start with more narrow AI. Do you think what is the future for the companies? Is it more smaller models that will use for their specific cases? How do you think about it?

11:38 Yeah, I think that genuinely the entire pie of AI use cases is going to grow. So if you ask me today versus in 2027, you’re going to see more use cases of every breed. The difference that I think you’re going to see is that my favorite customer quote is generalized intelligence is great, but I don’t need my point of sales system to recite French poetry.

11:57 It’s this idea that most customer use cases look like something like one of our customers, Checkr, does. They look through employee background checks and are looking to be able to extract out very specific information with respect to criminal codes and violations, previous things in employee backgrounds. It’s not like they need the model to be able to do French poetry and write Python code. They need to be able to do a really high quality job on one particular set of tasks.

12:25 So in enterprise I think that the majority of use cases are going to move and have already started to shift towards a lot of these narrow use cases that are going to be automation-oriented. Not to say that these general purpose agents where you can talk about anything in the company won’t exist and won’t be great demos, but some of the high value use cases are going to look like being a really prescriptive specialized agent that understands how to be able to solve a series of tasks very well.

12:50 And I think in enterprise that’s going to be true. In consumer I think it’s a little bit harder to say actually. I think the versatility is actually very helpful when it comes towards consumer. But I think regardless, one thing we’ve seen that wasn’t obvious in 2023, but I think is obvious in 2025 is we aren’t going to live in a world where one model rules it all. You’re going to see a mix of open-source models, closed and commercial models, different size parameter ranges, and just like any software tool, people are going to choose the best tool for their individual task.

13:32 In this crazy race, do you have a plan? How long in the future is it, 2026, 2027? What’s your strategy and planning?

13:40 Yeah, it’s a good question. I think there’s the famous quote from the boxer that everyone has a plan until they get punched in the face and that’s probably true in AI as well. We absolutely have a plan that we think about extending through the end of this year and through 2026. But the truth is AI changes on a weekly basis. And so you need to be able to have a framework that allows you to make decisions as things come in more quickly.

14:05 It’s all guided by like I would say a northstar vision which is our vision is to help customers develop specialized AI and help them tune their models and serve and deploy the models. The pieces that are more dynamic is what are the best ways that people will tune their models. We’re not religious to say supervised fine-tuning is the be all end all. We just think it’s the best technique today if you care about getting the best performance out of your models.

14:30 But in a year from now you could see a completely new technique come up. And obviously in 2025, the biggest new technique is reinforcement fine-tuning which we pioneered at the beginning of this year. And so I think from our standpoint our vision is really going to be predicated towards two things: on the goal of helping customers develop specialized AI, tune those models, and then run highly performant model deployments in production.

15:09 Let’s go to inference. Why is it so hard for the enterprises and what could make it easier?

15:15 Yeah, inference is a phenomenal example of something that starts off being very easy and then gets hard as you actually peel back the layers of the onion. Why is inference hard? It gets hard between the different stages that an organization ends up being in. There’s the crawl stage where the difficult part of inference, it’s not so much needing brilliant software engineering, but it is that GPUs for example might be hard to get.

15:40 So if you have the largest possible model, you might need eight or 16 H100s running to just have a single replica of a model deployed, which means you’re going to have to procure them and then decide if you’re able to autoscale them up or down inside of your environment. And you need to set up an initial inference server and framework. Now a number of companies including us have tried to make that easy by open sourcing inference frameworks and servers. So we’ve open sourced Lorax which is the underlying inference technology that we use so that anyone can stand it up themselves.

16:10 I think what I see though is if you’re a smart engineer you can set up your own inference framework and server. You can get that working and it’ll work well to start to feed your initial prototype and application. What’s really hard though and what I think a lot of customers don’t want to do is maintain production inference. And production inference is another ball game.

16:30 That doesn’t mean that the model works and is up 95% of the time or even 99% of the time. It means it’s backed by like 99.9 or 99.999% SLAs. That means you need to have resilient fault tolerance, be able to do blue green deployment updates. That means you need to be able to do multi-region replication in case you have these model deployments set up. And then most critically, GPUs are expensive, which means you need to be able to optimize the models such that you’re actually getting the most for every individual token that comes out.

17:01 And so you’re optimizing for total cost of ownership, whether that’s using a small model or using some of the techniques we have in our platform like Turbolora, which are just software defined ways to be able to increase your model throughput by 2x. All of these factors come in when you make that shift from I’m prototyping and able to get inference going versus I’m actually going to production. I need monitoring, SLAs, high performance throughput, all the other functions that will go with the fact that this is feeding a business critical application.

17:30 The last thing I’ll just briefly say with inference is I think that while all those are challenges, there’s a view that inference is going to get increasingly commoditized as a market. And I don’t disagree with this on base model inference in particular. Like there’s no reason that so and so’s Deepseek endpoint is way better than another company’s Deepseek endpoint.

17:50 What I think is going to be really interesting is the trend towards what we kind of call internally intelligent inference and intelligent inference in my view is back to that initial conversation we were having about do you have an inference pipeline that hooks into a post-training stack that lets your models get better over time. That’s really the future of where we see inference going. It’s super interesting, intelligent inference. We’ll work on the branding and marketing for that one. But it’s definitely the biggest trend that we see get unlocked by having a single place you can do post training and inference.

18:08 If we speak about open source and you are such a proponent of open source AI models, what is missing in the open-source AI stack and what are the gaps between this model zoo and the production?

18:20 I think that the open source model stack has gotten pretty good for some of the core infrastructure that you need to be able to do to set up. So I think open-source fine-tuning frameworks like ours, Ludwig as an example, are pretty good at being able to help people start to run experiments. Open source inference frameworks like ours like Lorax are pretty good at helping people do some of the initial serving. And then when you want to make the shift towards a managed platform, you have a pretty easy on-ramp to a platform that gives you the batteries included GPUs and infrastructure out of the box.

18:50 I think one of the things that is missing is a really resilient way to be able to do evaluations. This is something that has been talked about quite a bit. I think in the LLM industry and the truth is it’s a challenging problem because LLM outputs at times can be objective and at times can be quite subjective. What’s to say if something is a good summary versus if you got decent classification accuracy as in traditional machine learning.

19:15 So I think I’ve seen a number of friends with a number of folks that have started frameworks or even companies in LLM evaluation but I still think it’s an open problem. A lot of companies build very in-house evaluation systems and the leaderboard illusion showed us that crowdsourced evaluation cannot really work at this moment right.

19:35 Yeah, I see people tackle evaluation in a number of different ways. Most people I see do evaluation in house. I’ll just start off by saying that. Most common thing that I see. I’ve seen three versions of evaluations. The first is like they rely heavily on existing data and like some proxies. So a good example is like if you’re doing document classification, you look at some historical holdout data and you’re able to see how did that model perform? That’s the cleanest, simplest way, but isn’t always possible because that works for use cases where you have that historical data.

20:07 The second way I’ve seen evaluations done is to leverage GenAI itself more heavily. And so here we see people use LLMs as a judge as the most common technique, but they start to use larger models as graders to understand this response. Did it answer the customer’s question? Was this a good summary for the output that we were looking for? And so forth.

20:30 And then finally, I would say that when it comes to evaluation, the third way that we see it is it sort of feels like vibes in some way, but you’ll ship the product and you’ll try and collect some sort of product feedback to help you understand whether or not the model was like doing the directional type of behavior that you were anticipating. I mean I think the truth is a lot of evaluation is in house and we should not underestimate how much of it is vibes today but getting good evaluation is going to be critical towards building this kind of continuous improvement loop.

21:05 I agree but it’s so hard especially with the closed models because they publish a new model, it’s a new persona and you need to develop basically new vibe to understand it. So it’s really tricky.

21:18 Yeah I think open source is helping a lot in particular with open source reasoning models. One of the big things was exposing the reasoning tokens themselves. That happens with like if you used Deepseek R1 versus if you’re using the earlier generation of reasoning models from a closed provider. You can actually start to run evaluations not just on model output, but what were the series of steps that it was taking in order to be able to get there.

21:40 Look, I think the point was actually much less obvious in 2023, but I think has become really obvious for most companies now, which is open source is here and going to be here to stay. In 2023 when we talked about the future being open source the best open source model at the time was GPT-J and it was a far gap from where you thought about GPT 3.5 at the time.

22:06 Today Deepseek R1 or V3, Qwen 3, Llama 4, these models are not only on par but many times actually in the benchmarks doing even better than the leading commercial models. To me that’s actually 6 months ahead of schedule. I would have thought end of 2025 would have been the earliest that we’d see open source models beat commercial models. I thought they would be on par this year. But it’s incredible at the rate of innovation that we’ve seen in open source.

22:40 You never speak about AGI. What is your stand on that?

22:45 I think AGI is something that’s far from the world that I see in honesty. Like the world that I see tends to be like rather than artificial general intelligence, it’s like practical specialized intelligence. And so I think that AGI often times think about in the research labs, folks that will come up with good definitions. If we take a definition of AGI is like can a model pass a Turing test. I would suggest that we’re probably in that ballpark already.

23:15 But what’s the practical implication of that? I don’t spend too much of my time thinking about the Terminator style scenarios, but I do spend a lot of my time thinking about what does having this generalized intelligence look like when you actually have businesses processes like a company Fortune 200 like Marsh McLennan or Checkr or any of these other companies that have a lot of productivity that they’ve unlocked via business practices over the past previous decades.

23:45 And that productivity is about to take a step function change and increase. To me that’s really kind of like the interesting area for where this is going to go. I think there’s probably some deeper philosophical questions about what will happen over 5, 10, 20 year period. We found it pretty hard to even predict what’s going to happen 18 months from now in AI. And so I think that’s really where a lot of my focus has been is what’s going to be the practical implication on both enterprise and consumer.

24:15 That’s very nice perspective. What does concern you and excites you the most about the future that you build with Predibase?

24:25 I think what concerns me probably really comes back actually initially to the evaluations piece. So what concerns me is people see an incredible amount of value. I see these statements from analyst reports and others from time to time which often ask like is generative AI a bubble? Are people actually seeing real business value? I never get that question from like a CIO at a Fortune 200 that’s actually working on applications.

24:50 Like I think when you’re on the ground and you actually see what these models can do, I think that the question of like is there going to be an ROI there versus the 50 cents or a dollar that the million tokens are going to cost to process, it’s not even a question in the vast majority of situations. And if it is, it’s really just like a question of selecting the right use case.

25:10 But what does concern me is we might go through a little bit of a hype bubble where people are really excited about these far-fetched French poetry style use cases where the model is doing these amazing multi-agent step demos and then you enter into a little bit of a crash of disillusionment where people have gravitated and attached towards use cases that were probably not the high value business impact use cases that the models can do today.

25:40 You know, the high level thing of what concerns me is if people end up essentially going too far and shooting too far ahead and not realizing the business impact that they can have with LLM use cases today and then they enter a little bit of disillusionment out of that.

25:55 I will say though just given how quickly people have been able to iterate, I think I’m a lot less concerned about that than I was 5 or 10 years ago. I’ve been in the AI space for over a decade. And I think that genuinely AI has probably been a place that for outside of the top 1% of organizations has overpromised and underdelivered between 2012 and 2022. It was an area where we talked about you’re going to have, look at how YouTube does recommender systems, you can bring that to your small and medium-sized business. That never really translated right.

26:25 I don’t think that’s true for the current wave of AI that we’re in right now. And like the thing that would concern me is if we started to adopt paradigms that make people think that way.

26:35 The trend that I’m most excited about is it used to be the way in software development that you’d sort of like perfect and then ship. Like you go ahead and build and then you test test test, you do some dogfooding and then you go ship out your product. As a startup entrepreneur, I think a lot about how do you get really fast feedback from the market as quickly as possible.

26:55 And you know, one of my previous mentors had said if you ship something that you’re not at least a little embarrassed by, you’ve waited too long. And the thing that I think I’m excited about in AI is you’ve actually started to see a shift in how people develop too where they’re putting out honestly 60% solutions today. And the reason they’re putting out 60% solutions is they want to test do I have product market fit with this solution and then also start to collect data such that they can improve those models over time.

27:20 And so I’m excited to see what this more startup way of thinking is going to mean now that it’s being adopted not just by 20-50 person organizations but also some of the larger companies adopting GenAI.

27:34 Thank you. That’s very insightful. My last question is a complete change of gears. What is a book or idea that shaped your thinking and it can be related to machine learning or completely unrelated?

27:45 Completely unrelated. I would say a book that I like is it’s called The Happiness Advantage by Sean Achor. And so it’s a psychologist at Harvard that basically studied behavioral psychology in the context of organizations often times. But what he found was it wasn’t necessarily that success brought happiness in all cases. It was that happiness actually made you much more likely to be successful.

28:05 The book really covered two key things. The first was how just having more positive and happy outlook allows you to be able to do better in terms of the different tasks that you’re trying to do whether it’s work or personal. And the second is like ways that you can essentially put yourself in a position to be a lot happier without having to be relying on these external factors.

28:25 I think the Happiness Advantage is definitely one that I think I really enjoyed kind of as an overall read and kind of an idea that also extends towards both personal and professional work. So you think Predibase is a happy organization? I hope so. I mean I think that the truth is if you’re working in generative AI today it’s a noisy environment. It’s fast moving. It’s competitive. Players including us are well funded which means that you have a lot of cards on the table.

28:50 But I think that we and other organizations will also do our best work if we’re excited about the future that we’re running into rather than if we’re operating out of like, for example, predominantly concern or other things like that. Great. Thank you so much. That was wonderful. Of course. Yeah, I really enjoyed the conversation.