Why AI Still Needs Us? A conversation with Olga Megorskaya, CEO of Toloka

Summary

Olga Megorskaya, CEO of Toloka, explores the evolution of human-AI collaboration from simple data labeling to true “co-agency” where humans and AI agents work together to solve tasks. She traces Toloka’s journey from the early days of training classifiers with simple 30-second annotation tasks to today’s complex work that can take days of expert effort per data item. The conversation illuminates how the role of humans has fundamentally shifted from providing basic ground truth to bringing deep domain expertise that AI cannot replicate.

The most striking insight is Toloka’s counterintuitive approach: they are now training their human experts to “distrust LLMs” - to recognize when an AI’s plan is wrong (which happens about 30% of the time) and apply their own judgment. This moment of human override becomes the most valuable training signal in the system. Megorskaya argues that while engineering limitations will be solved, the fundamental challenge of knowing when to trust AI versus trusting human judgment will always require human wisdom.

The discussion covers synthetic data’s limitations, the importance of benchmarks, and Toloka’s recent red-teaming dataset contribution to the ML community. Megorskaya shares her philosophical perspective on AGI (she’s skeptical it will ever be “complete”) and recommends Umberto Eco’s “Foucault’s Pendulum” as a prescient exploration of how simple sequences of text can generate entire belief systems - remarkably relevant to the age of large language models.

Highlights

”Training people is not much different from training models”

yt-dlp --download-sections "*11:46-12:15" "https://www.youtube.com/watch?v=b5EP6YHMaEg" --force-keyframes-at-cuts --merge-output-format mp4 -o "b5EP6YHMaEg-11m46s.mp4"

“Training people is not much different from training models. People are best trained by examples. So you create a data set of use cases, you show them to people explaining ‘here is right, here is wrong’ and while processing through those examples, people are training their own neural network and then start to understand the logic.” — Olga Megorskaya, 11:46

”We are training experts to distrust LLMs”

yt-dlp --download-sections "*9:21-10:00" "https://www.youtube.com/watch?v=b5EP6YHMaEg" --force-keyframes-at-cuts --merge-output-format mp4 -o "b5EP6YHMaEg-9m21s.mp4"

“The most important skill that we are training our experts right now is to distrust LLM - to see when actually the plan that AI agent has created is wrong. In 70% it would be correct but in 30% it would be wrong. This is the most important and most responsible part of human input - to say ‘no, no, no, I’m not listening to you here. I am applying my own amazing human wisdom to do it my way.’” — Olga Megorskaya, 9:21

”30 seconds became 10 hours”

yt-dlp --download-sections "*4:16-4:55" "https://www.youtube.com/watch?v=b5EP6YHMaEg" --force-keyframes-at-cuts --merge-output-format mp4 -o "b5EP6YHMaEg-4m16s.mp4"

“Many years ago, the average task that was needed to be executed by the human annotator was like 30 seconds. We even had this 30 seconds as a magical number literally hardcoded in our platform. Right now this is no surprise of tasks taking 10 hours, even sometimes several days of human work to prepare one item in a data set.” — Olga Megorskaya, 4:16

”Human expert is a benchmark”

yt-dlp --download-sections "*27:40-28:20" "https://www.youtube.com/watch?v=b5EP6YHMaEg" --force-keyframes-at-cuts --merge-output-format mp4 -o "b5EP6YHMaEg-27m40s.mp4"

“Ultimately the whole concept of AI is that human expert is not an executor. Human expert is a benchmark. AI is trying to keep up with the human expertise. We have in some areas already reached the situation when the average model is performing better than the average human. That’s why it is a specific job to collect the collective human wisdom that would still be higher than the wisdom of the AI.” — Olga Megorskaya, 27:40

”The power of text - Foucault’s Pendulum”

yt-dlp --download-sections "*28:50-29:40" "https://www.youtube.com/watch?v=b5EP6YHMaEg" --force-keyframes-at-cuts --merge-output-format mp4 -o "b5EP6YHMaEg-28m50s.mp4"

“It describes how actually the simple sequence of letters, the simple text can create by itself whole new concepts, whole new societies, whole new religions and ultimately the questions of life and death of particular people. If you think about it, this is what we are observing now with large language models. This is a fascinating power of the text.” — Olga Megorskaya, 28:50

Key Points

• Human as technological resource (2:19) - Toloka’s philosophy: managing human efforts technologically for scalable, replicable production of humanly generated data
• Classical ML era (1:22) - Early days had simple tasks (“this is a cat, this is a dog”) with high variability across thousands of classifiers
• Foundation model shift (2:40) - ChatGPT era needed PhD professors, senior engineers, professional legal counsel - not just any human
• Variability collapsed then expanded (3:30) - Foundation models collapsed variability, but agentic systems are increasing it again across many surfaces
• Human as vector of skills (5:20) - Optimal matching between tasks and experts based on capabilities, integrations, capacity, price
• Co-agency definition (7:58) - AI agent and human agent solving the same task together, each contributing unique strengths
• AI strengths (8:10) - Decomposing tasks, creating plans, ensuring humans follow steps, catching mistakes with fresh eyes
• Human strengths (9:00) - Recognizing when AI doesn’t know what it doesn’t know, applying judgment in that 30% wrong case
• Distrust LLM training (9:21) - The most important skill Toloka trains: knowing when to override AI with human wisdom
• Engineering vs judgment (10:20) - Engineering challenges (integrations) will be solved; judgment about when to trust AI is fundamental
• Train like models (11:46) - People learn best from examples, training their own neural networks through case studies
• ChatGPT impact on Toloka (12:30) - Shifted focus from crowdsourcing to highly skilled experts; AI agents will change tech foundation
• Synthetic data limits (14:45) - Customers can make synthetic data themselves; Toloka provides what they can’t - human ground truth
• Human oversight required (16:00) - Always need humans to evaluate synthetic data quality and develop benchmarks
• AGI skepticism (17:30) - Doesn’t believe in “complete AGI”; there will always be need for human ground truth
• Future excitement (22:20) - AI unlocking new domains beyond laptops and offices into real economy (agriculture, engineering)
• Benchmark importance (24:40) - Public benchmarks get leaked; industrial players design custom internal benchmarks
• Tau and SWE-bench (25:20) - Popular benchmarks for AI agents; GAIA tests tasks easy for humans but hard for AI
• Red teaming dataset (26:40) - Toloka contributed ML Commons red teaming dataset in December 2024

Mentions

Companies

• Toloka (0:33) - Human-AI data labeling and co-agency platform
• Anthropic (15:10) - Customer using purely human data
• Amazon (15:10) - Customer using human ground truth
• Microsoft (15:10) - Customer using human expertise
• Poolside (15:10) - Customer at cutting edge of AI
• Sierra (25:30) - Designed the Tau benchmark for AI agents
• Pinecone (12:18) - Ido Liberty’s company that rewrote architecture after ChatGPT

Products & Technologies

• Tau benchmark (25:20) - Popular AI agent benchmark designed by Sierra
• SWE-bench (25:30) - Popular coding benchmark
• GAIA benchmark (25:40) - Tests tasks easy for humans but hard for AI agents
• ML Commons (26:40) - Red teaming dataset community

People

• Ido Liberty (12:15) - Pinecone founder who discussed architecture changes post-ChatGPT

Books

• Foucault’s Pendulum (28:30) - Umberto Eco novel about the power of text to create societies and religions

Surprising Quotes

“We even had this 30 seconds as a magical number literally hardcoded in our platform as an average time of execution of a task. Right now this is no surprise of tasks taking 10 hours, even sometimes several days of human work to prepare one item in a data set.” — 4:20

“You cannot bring useful signals to the AI systems if everything that you do is being AI trainer for 8 hours a day, 40 hours a week because you need to bring the real insights from the real market. You need to be up to date with your profession.” — 13:10

“I don’t quite like to call it ‘human as a callable function.’ I rather prefer thinking about it as managing human efforts in a technological way to support scalable and replicable production of humanly generated data.” — 2:22

“Discussions about AGI are quite unpractical. I prefer to be more down to earth. I personally don’t think that there is such a thing as complete AGI. There will always be some need in the ground truth which you cannot get from anywhere apart from human wisdom.” — 17:50

Transcript

0:00 Basically training people is not much different from training models. AI agents and human agents - they have lots of similarities. We are training our experts right now to distrust LLM. I personally don’t think that there is such a thing as complete AGI.

0:24 Thank you Olga for joining me. Let me start with this question. You once quoted a phrase from my article about human in the loop evolution and you quoted “humans are another callable function in an AI agent’s toolbox.” So when did you first realize that humans can be callable functions?

0:47 Yeah, thank you. Indeed, when I saw your amazing post it really resonated with me because you have been describing the evolution of human in the loop in machine learning industry but at the same time I felt like you have been describing just the whole story of Toloka and that really impressed me.

1:08 When we started many years ago we started for supporting all the classical machine learning development with training data. These were the days when everybody was developing their own classifiers for different needs. The tasks that required humanly created ground truths were quite simple. It was like to define “this is a cat, this is a dog, this is a pedestrian, this is a car.”

1:37 At the same time, the variability of these tasks was very high because there were thousands of different applications and each of the classifiers required some data set to train on. These were the times when there were literally dozens of thousands of people annotating tasks across thousands of different projects every day.

2:04 These were the times when we first realized this concept that basically led as the foundation of Toloka philosophy - that in order to provide scalable production of ground human ground truth data you need to manage the efforts of humans in a technological way.

2:22 To be honest I don’t quite like to call it “human as a callable function.” I rather prefer thinking about it as managing human efforts in a technological way to support scalable and replicable production of humanly generated data.

2:38 It’s very interesting that after this era of classical machine learning, the new era started - the time of ChatGPT, the time of foundational models, which brought the new level of complexity. It was not enough to be just human to train AI anymore. The complexity of the tasks increased dramatically.

3:00 Now it was the time when the systems needed deep domain knowledge and deep expertise in specific domains of human knowledge. This is when you wanted to attract PhD professors in physics, senior software engineers, professional legal counsels to act as the source of the ground truth, to act as people who actually bring this ground truth expertise to train foundational models.

3:27 At the same time, the variability of the tasks at that moment significantly collapsed because instead of having to train thousands of different classifiers, now the whole industry started to invest into training a dozen foundational models that will later be dealing with any applications.

3:50 Basically, it’s very interesting that now we are entering the new phase of this development with the era of AI agents which basically combines the complexities of the two previous worlds. On the one hand, the tasks are continuing to be more and more complex and demanding more and more deep domain expertise from human experts.

4:16 Many years ago, the average task that was needed to be executed by the human annotator to provide some item in the data set was like 30 seconds. We even had this 30 seconds as a magical number literally hardcoded in our platform as an average time of execution of a task. Right now this is no surprise of tasks taking 10 hours, even sometimes several days of human work to prepare one item in a data set.

4:42 At the same time the variability of the tasks is again increasing because right now AI agents are operating on a large variety of different surfaces. It’s not only a chatbot anymore. It’s already different surfaces, different scenarios of interaction, different modalities and hundreds of different potential scenarios which need to be tested, red teamed, etc.

5:10 And this is when we are again coming back to this philosophy of technological management of human efforts - becomes again even more important than it was before because looking at every human expert as a vector of skills and optimally matching the tasks and the experts between each other becomes a very important part of successful data production nowadays.

5:32 That’s why in a way when you are thinking both about AI agents and human agents they have lots of similarities in terms of - if you describe those essences in terms of parameters - they both have some intellectual capabilities, they have some skills in terms of integrations in case of AI agents, in terms of skills of using certain tools in terms of human agents, they have their available capacity, they have their price.

6:00 And balancing all these parameters in multi-agentic systems, you can basically decide when it is better to call for a human agent or for AI agent. We at Toloka are big believers in this hybrid collaboration approach between human agents and AI agents and we do believe that the future lays somewhere in this field.

6:22 Thank you. Yeah, I also remember that you sketched your evolution at Toloka as data labeling with the crowd, then it was human preference feedback for RLHF, then human evaluation and expertise in niche domains which you describe, and then co-agency on multi-agent teams. So my first question would be what is true co-agency? Can you please define it for me so everyone understands it?

6:50 Well, the major difference of the era of AI agents is in I would say two things. One is this interaction on a large amount of surfaces. It’s not about chatting with a model in a chatbot anymore. It is about using your computer as a surface of interacting with the AI agent, using all your tools that you’re using day-to-day, interacting with the agents.

7:17 And second is iterations - multiple iterations between the user and the system. We are now working with long-term trajectories that start from one point and then can go in many different directions. This is principally from the point of view of collecting training data and creating benchmarks for training agent systems.

7:42 This is the most important difference I would say that we see right now comparing to the previous stage of just creating dialogues between a model and the user.

7:58 So what is co-agency in your definition? For us co-agency is when AI agent and human agent are solving the same task together. There are things where AI agents are much better than human agents. For example, decomposing the task, creating the plan of a task, helping a human validate that the human actually follows the steps of the plan.

8:25 Because humans usually have problems following the plan - not because they are lazy or something, but because we as humans bear much bigger context. Basically all our life experience is our context. And that’s why we quite often tend to skip some steps and some stages because they are obvious for us. They’re intuitive for us.

8:50 But what is intuitive for one person can be not intuitive for the other. And when we are talking about scalable human operations, you need to make sure that none of the steps is missed. And this is where AI agents are helping human agents a lot - then helping them look with a fresh eye at the results of the task performed and helping you see some potential mistakes and problems.

9:13 These are the things where AI agents are helping our human experts in performing the tasks. At the same time, there are things that AI agents cannot do. And one of the most important things is that sometimes they don’t know what they don’t know.

9:28 And this is ironically the most important skill that we are training our experts right now - is to distrust LLM, is to see when actually the plan that AI agent has created is wrong. In 70% it would be correct but in 30% it would be wrong.

9:47 And this is the most important and most responsible part of human input - to say “no, no, no, I’m not listening to you here. I am applying my own amazing human wisdom to do it my way.” And this is actually the very important source of the signal for the whole system - for humans to trust themselves more.

10:05 Yes, in certain cases. That’s the whole source of the essence of the task that we are solving - that you cannot tell in advance when you need to trust AI and when you need to not trust AI. And this is the moment of judgment that brings the useful signal to the system.

10:25 How interesting. Is it the main bottleneck for true co-agency between humans and AI or are there others? This is fundamentally the most important bottleneck. Apart from that obviously there are engineering challenges that will be solved sooner or later - like the amount of integrations that agents have.

10:45 Right now not so many agents can freely use the computer and some other applications but in a year from now it will not be a problem. I would say for the majority of applications it will take some time for AI agents to be able to solve more complicated tasks in more niche domains and more niche applications.

11:05 But I think that these are mostly engineering limitations. I think there will always be this long tail of use cases where agents are not yet engineered enough and when you will need the interaction with a human expert to finalize and to actually solve the task.

11:28 But within all these operations the major essence is to decide when to trust agent and when not to trust agent and to trust human. And this is a fundamentally difficult thing. How do you train people to do that?

11:40 This is the thing that we have learned long ago - basically training people is not much different from training models. People are best trained by the examples. So you create a data set of use cases, you show them to people explaining “here is right, here is wrong” and while processing through those examples people are training their own neural network and then start to understand the logic. I think that this is still the most efficient way of teaching.

12:15 When I talked to Ido Liberty from Pinecone, he said that after ChatGPT they had to rewrite the whole architecture basically for their vector database. Did you have any problem like this? What happened to you after ChatGPT boom?

12:32 Well, for us, ChatGPT per se did not change a lot in terms of technological architecture, but I rather think that now AI agents are something that fundamentally will change the technological foundation of the service.

12:47 The coming of ChatGPT for us marked the major milestone of switching our focus from working with crowdsourcing towards working with highly skilled human experts. And that presumed also some new technological challenges because when you are dealing with systems when it is important to bring in highly skilled professionals in certain domains, you want to invest more into technologically selecting and attracting those people, qualifying and checking their level of qualification.

13:20 And then investing a lot into building the community and building trust with those experts. And this is kind of a separate and not technological part of the business but I do believe that it is crucially important - when we’re talking about people who are bringing their expertise to train AI, these are people who are actually acting as professionals in their domains.

13:50 You cannot bring useful signals to the AI systems if everything that you do is being AI trainer for 8 hours a day, 40 hours a week because you need to bring the real insights from the real market. You need to be up to date with your profession.

14:10 That means that we need to be able to attract highly skilled, highly paid professional specialists and to offer them something that they would be motivated to participate in those kind of tasks. So I think this is what brought the major change for our business with the appearance of ChatGPT, and now AI agents are bringing on top of that this technologically new foundation of co-agency between humans and AI agents.

14:40 What is the role of synthetic data in your data sets? By nature of our business, we are helping our customers with the data sets that are mostly purely human data because since we’re working with some of the most technologically advanced companies in the world such as Anthropic, Amazon, Microsoft, Poolside - just creating purely synthetic data is something that our customers can do by themselves.

15:10 However, there is always a limit above which you cannot gain any substantial profit out of training on synthetic data. And this is when you need human ground truth to evaluate the quality of synthetic data and to provide the new level of the signals.

15:30 That’s why we are focused on mostly purely human data. At the same time, of course, we use different technological approaches into delivering those data sets. You need to be able to use similar approaches for generating synthetic data when you need to care about the diversity of the data set.

16:00 So you need to think of the taxonomy of the data set to ensure that you’re covering all the variety of topics. For example, you are creating a training set in the field of finance. You want to ensure that you are covering all the major topics that are important for the model to learn about finance.

16:20 This is when at first you would invite some human expert to help you define this taxonomy. Then you would generate synthetically or semi-synthetically some skeleton of this data and then you will call for human experts again to validate, update, upgrade this data based on this skeleton. That’s very interesting.

16:45 Synthetic data is a very powerful tool. Obviously a lot of companies are using it. At the same time, I think there is a common understanding in the industry that only synthetic data is just not enough. You always need human oversight - you need to evaluate the quality of the synthetic data at the very least.

17:05 You also need to develop the benchmarks according to which you want to measure the quality of your models. And usually benchmarks require a heavy deep dive of human experts to create them.

17:18 Thank you. Is human-AI true co-agency a path to AGI or is it AGI? What’s your take on it? I don’t know. To me, discussions about AGI to be honest are quite unpractical. So I prefer to be more down to earth looking at what we can do from the engineering point of view.

17:45 I definitely think that the co-agency and hybrid systems where humans and AI are collaborating are the next step. Whether it is the last step or not, I honestly don’t know. I have no idea. I personally don’t think that there is such a thing as complete AGI. There will always be some need in the ground truth which you cannot get from anywhere apart from human wisdom.

18:20 That’s what I love talking to practitioners because basically every time I ask a question about AGI people say “well I try to look more practical on that.” And you create this whole thing - you really know how it works practically. So I think it’s very helpful to have this narrative out.

18:40 I think this is maybe both good and bad because we are developing step by step and every step seems quite small. And maybe within those small steps there is kind of a risk to lose the bigger picture. But at the same time even looking back at the evolution of Toloka throughout those 10 years we see what a giant path there was, but at every moment of time these are very minor and very practical steps.

19:10 Probably 10 years ago it was hard to imagine what it would be like today. It’s a famous phenomenon that first you call something a wonder and then it just becomes software.

19:25 When you think ahead a couple years, 5 years, what excites you the most and what are your concerns? Well, speaking about our engineering and practical day-to-day things we are dealing with, what really excites me is the technological part about this hybrid collaboration.

19:50 Because I think it opens a lot of opportunities for bringing in and attracting much more human experts from a large variety of different domains into this whole AI production world. Because right now we are still in kind of a bubble and there are lots of parts of human economy which are basically not touched with AI at all.

20:15 And it will take us some time to go from the apps that are helping you set up the calendar or book tickets or something else towards some hardcore real economy challenges - to real SolidWorks, to some agriculture, to something else.

20:35 I think this is a very interesting opportunity. And this is what excites me - to see how AI will be unlocking new domains of human knowledge going further from the offices, further from our laptops into the real world. I think that’s an exciting thing to observe.

21:00 Any concerns? Well, I mean every radical change brings certain concerns. I wouldn’t call them concerns. I think it is rather we are in a very interesting position where actually a lot is in our hands to ensure that this technological evolution or revolution whatever you call it actually goes smoothly - that the systems are under control.

21:30 And the beauty of that is that we actually have instruments to influence that. We are doing a lot of tasks related to red teaming to ensure that AI agents are working in a safe and responsible way. We are working a lot in developing benchmarks which basically guide the directions where the models are developing.

22:00 We are actually building a system that allows different human specialists from different professions to benefit from and get additional opportunities for additional income from training AI.

22:20 We have in our hands opportunities to help people not being afraid of AI replacing them but rather seeing new opportunities of AI production offering them new additional sources of income.

22:40 That’s why I think we in Toloka are in a very interesting position where there’s no point in being concerned. Rather there is a point of taking those efforts in our hands and basically shaping the future that we want it to happen.

23:00 How do you work with benchmarks? Because you know the recent scandal with benchmarks demonstrated that it’s very hard to rely on human judgment and it can be tricked, it can be gamed. How do you work with benchmarks? And in a more general sense, what do you think about benchmarks for models?

23:25 I think that benchmarks are super important and I do see that now the industry has already adopted this concept because 3 years ago nobody was talking about evaluation at all. 2 years ago everybody was talking that we need some evaluation and some benchmarks but nobody knew what to actually do with that. Now already this is just a routine thing.

23:55 How we see this developing right now is that from time to time there are some teams inventing some nice benchmarks. Basically inventing a new benchmark is a very serious intellectual effort because you need to come up with a design of this benchmark, you need to understand what are the questions you need to get answers on. That’s why this is a very responsible and very prominent job to design the benchmarks.

24:25 But the problem with public benchmarks is that they get leaked very fast and basically it’s quite hard to rely on them only. That’s why what we see in the industry is that industrial players usually select the kind of benchmarks that they want to rely on and then with our help we are designing specific custom benchmarks for internal use to make sure that they are not leaked anywhere.

24:55 I would say that right now a very popular benchmark related to AI agents is Tau benchmark. It was designed by the company Sierra maybe a couple of years ago, maybe a year ago. Now it is getting a lot of traction. Everybody wants similar benchmarks in order to evaluate their models against them.

25:20 SWE-bench is a very popular one for coding. Also there are some funny benchmarks like GAIA benchmark which is designed out of the use cases which are presumably easy to solve by humans but very hard to solve by AI agents. It’s interesting because it is super unpractical - you would never be dealing with such tasks in any of your real life scenarios - but at the same time they illustrate the limitations of the capabilities of modern AI agents.

25:55 So you think it’s possible to create some sort of a general benchmark or it’s still mostly inside and internal benchmarking that makes sense? To be honest from a practical point of view I do not quite believe in one general benchmark to measure everything against it. I rather believe in a set of different specialized benchmarks.

26:20 They are very useful. They are practically useful because you set your goal against this benchmark - “I want to hit 90% of this benchmark” - and you’re optimizing this model to reach this level. Then you say “okay now I want to choose another benchmark and optimize for it.” And this is basically the guiding steps for the industry to develop.

26:45 There are some interesting initiatives about general across-the-whole-industry benchmarks or general data sets. We for example have contributed with the community ML Commons in creating such a red teaming data set. That’s a recent one, right? Yeah, it was released in December 2024.

27:05 That’s an example of an attempt to create something that should be used across the industry. I think these are noble initiatives but from a real production perspective of course every team would be willing to define their own path and to define specific benchmarks that they want to hit along this path.

27:25 Yeah. Another role for a human. I think that is really a role for a human - to be the source of the ground truth and to be the benchmark because ultimately the whole concept of AI is that human expert is not an executor. Human expert is a benchmark.

27:50 AI is trying to keep up with the human expertise. I think we have in some areas already reached the situation when the average model is performing better than the average human. That’s why it is a specific job to actually collect the collective human wisdom that would still be higher than the wisdom of the AI. But that’s the essence of the whole development and the progress.

28:20 But from the technological point of view - from what we have started - I do think that when we’re talking about human operations we need to think about human operators in the same way we are thinking about AI agents operators. Because technologically this should be a seamless flow that navigates the task between humans and AI and they’re working in collaboration. So this should be a process that does not have those borders - “this is the human part, this is the AI part.”

28:50 Thank you. I believe that books form humans and human experience and human philosophy. So what is the book that influenced your philosophy for the company or in general and that you would like to share?

29:05 You know, I recently was thinking about that and probably this is not the most common answer because my favorite book is a purely fictional one and this is Umberto Eco’s “Foucault’s Pendulum.” This is a book that has lots of cultural references and lots of layers inside it.

29:28 But one of the layers that hit me recently - and I did not think about it when I first read it many years ago - but when I reread it recently, I was just astonished by that. Even though it was written in the 1980s, long before any AI happened, basically what it describes on one of its levels is the power of the text and the power of literally letters.

29:55 It describes how actually the simple sequence of letters, the simple text can create by itself whole new concepts, whole new societies, whole new religions and ultimately the questions of life and death of particular people.

30:15 The whole story of this book is about - it all starts with people finding some small piece of paper with some letters written on it. And depending on how you fill in the gaps between those letters, you can think of it as a starting point of some secret societies and hidden treasures and whatnot. Or it can be a simple note that a wife wrote to her husband to pick up some products on the market.

30:40 And these are two very different trajectories that can be generated out of one simple sequence of letters. And if you think about it, this is what we are observing now with large language models. This is a fascinating power of the text that back like 50 years ago it was a pure fictional intellectual exercise. And now we are actually living in this reality.

31:05 So I think this is something that excites me about the books that I recently read. Thank you so much. Thank you for this interview. It was very insightful.