When Will We Speak Without Language Barrier? A conversation with Mati Staniszewski, CEO @ ElevenLabs

Summary

In this deep technical interview, Ksenia sits down with Mati Staniszewski, co-founder and CEO of ElevenLabs, to explore when universal real-time translation will finally become reality. Mati reveals the three-step pipeline behind conversational AI - speech-to-text, LLM translation, and text-to-speech - and explains why emotion understanding remains the biggest unsolved challenge. The current technology achieves 1-1.2 second end-to-end latency with their TTS at just 70 milliseconds.

Mati shares concrete use cases where the technology is already working: Hypocratic (healthcare patient calls) and Elise AI (customer support with live translation). However, real-time sports dubbing with emotional commentary remains “not there yet.” The interview dives deep into the technical challenges - speaker diarization, emotional metadata annotation, and preserving meaning across sentence boundaries, especially when English to Spanish can be 30-40% longer when spoken.

The discussion also covers ElevenLabs’ strategic decision not to open source their research (yet), their approach of “show, not tell” through partnerships with creators like Lex Fridman, and their vision for the Babel Fish future. Mati predicts real-time translation devices will be viable in 2-3 years, with universal adoption in 5 years.

Highlights

”Real-time translation will happen this year”

yt-dlp --download-sections "*0:00-0:50" "https://www.youtube.com/watch?v=Znm_glAFMUQ" --force-keyframes-at-cuts --merge-output-format mp4 -o "Znm_glAFMUQ-0m00s.mp4"

“I think the next stage will be how do you make it real time? I think this will happen this year… In the next stage ideally you have some device, let’s say it’s a headphone or some other device like mobile phone where you can speak and it live translates to the other person.” — Mati Staniszewski, 0:00

”English to Spanish is 30-40% longer when spoken”

yt-dlp --download-sections "*8:00-9:00" "https://www.youtube.com/watch?v=Znm_glAFMUQ" --force-keyframes-at-cuts --merge-output-format mp4 -o "Znm_glAFMUQ-8m00s.mp4"

“English to Spanish is 30 or 40% longer when spoken. So if you need to keep the length the same, it’s hard. The conversations that Lex has, if we are able to do them semi-automatic or fully automatic in the future, it will be a good proof point that the technology is really there.” — Mati Staniszewski, 8:00

”The emotion problem requires massive annotated datasets”

yt-dlp --download-sections "*4:46-5:35" "https://www.youtube.com/watch?v=Znm_glAFMUQ" --force-keyframes-at-cuts --merge-output-format mp4 -o "Znm_glAFMUQ-4m46s.mp4"

“The first step would be to create a very large dataset and we work with voice coaches and people to effectively create larger sets of audio that then you annotate of how you say things. In this sentence the emotionality is excited, calm…” — Mati Staniszewski, 4:46

”70 milliseconds for text-to-speech”

yt-dlp --download-sections "*13:00-14:00" "https://www.youtube.com/watch?v=Znm_glAFMUQ" --force-keyframes-at-cuts --merge-output-format mp4 -o "Znm_glAFMUQ-13m00s.mp4"

“Current latency is end to end, with everything, with network, with interruptions, between a second to 1.2 seconds depending on the region. The text-to-speech which is the part of generating from text to audio, we have the quickest model which is 70 milliseconds.” — Mati Staniszewski, 13:00

”We want to build the Babel Fish”

yt-dlp --download-sections "*2:13-3:00" "https://www.youtube.com/watch?v=Znm_glAFMUQ" --force-keyframes-at-cuts --merge-output-format mp4 -o "Znm_glAFMUQ-2m13s.mp4"

“I don’t know if you are familiar with the Hitchhiker’s Guide to the Galaxy and the Babel Fish. That’s exactly where that’s going to head. Would love to be part of making this happen.” — Mati Staniszewski, 2:13

”Multimodal is the future, but enterprise needs control”

yt-dlp --download-sections "*14:40-15:40" "https://www.youtube.com/watch?v=Znm_glAFMUQ" --force-keyframes-at-cuts --merge-output-format mp4 -o "Znm_glAFMUQ-14m40s.mp4"

“In the multimodal you will not have as much stability but you will have more naturalness and quicker conversation. For big enterprise companies the three-step solution is usually the best because you have the LLM and you want to make sure it stays on topic, doesn’t go off the rails.” — Mati Staniszewski, 14:40

Key Points

• Three-Step Pipeline (2:24) - Speech-to-text, LLM translation, text-to-speech is the current architecture
• Speaker Diarization Challenge (2:37) - Must identify who is speaking when multiple people talk
• Emotion Understanding Gap (2:47) - Biggest unsolved problem - detecting how something was said
• Context Preservation (3:30) - “What a wonderful day” vs “What a wonderful day, I said sarcastically” - completely different translations
• Healthcare Success (5:52) - Hypocratic AI automates nurse patient calls
• Customer Support Translation (6:30) - Elise AI provides live translation for support calls
• Sports Dubbing Challenge (7:10) - Real-time sports commentary with emotion remains unsolved
• Lex Friedman Dubbing (8:02) - Took 1-2 weeks per podcast, translation was hardest part
• 70ms TTS Latency (13:00) - Text-to-speech is incredibly fast; LLM is the bottleneck
• Interruption Detection (13:30) - Smart mechanisms detect natural end of sentence vs mid-thought
• Voice Embedding Solution (10:15) - Encoding speaker voice to filter noise and detect who’s talking
• Multimodal Trade-offs (14:44) - More natural but less stable; enterprise needs three-step for control
• Not Open Source Yet (20:42) - IP advantage too valuable while still building product layer
• 2-3 Year Timeline (1:40) - Real-time translation devices in 2-3 years, universal in 5

Mentions

Companies

• ElevenLabs (0:29) - Voice AI company building the Babel Fish future
• Hypocratic (5:52) - Healthcare AI automating nurse patient calls
• Elise AI (6:30) - Customer support with live translation
• Palantir (17:00) - Mati’s previous company where he worked on optimization problems
• Deutsche Telekom (21:00) - New enterprise partnership announced
• Google Cloud (20:30) - Partnership mentioned

People

• Mati Staniszewski (0:00) - Co-founder and CEO of ElevenLabs
• Lex Fridman (8:02) - Podcast was dubbed by ElevenLabs into multiple languages
• Piotr Dabkowski (16:30) - ElevenLabs co-founder, “the genius and brain behind all the models”

Products & Technologies

• Babel Fish (2:13) - Hitchhiker’s Guide reference for universal translation
• ElevenLabs STT (4:22) - Speech-to-text model that beat all benchmarks
• ElevenLabs TTS (13:00) - 70ms text-to-speech model
• Conversational AI Framework (13:30) - Smart interruption detection with pre-generation

Surprising Quotes

“English to Spanish is 30 or 40% longer when spoken. So if you need to keep the length the same, it’s hard.” — Mati Staniszewski, 9:15

“My guess is the next two three years we’ll see some of that being out there and hopefully in the next five it will be something that everybody can use anywhere in the world.” — Mati Staniszewski, 1:42

“The text-to-speech, we have the quickest model which is 70 milliseconds. Then the transcription and the LLM is where it takes time.” — Mati Staniszewski, 13:10

“You can have a sentence ‘what a wonderful day’ - okay the translation is easy. But if you had ‘what a wonderful day, I said sarcastically’ - completely different.” — Mati Staniszewski, 3:34

“Initially we were thinking about open sourcing. Given we’ve spent so much time doing our research, we would give a lot of the advantage in IP that we have to others to recreate it.” — Mati Staniszewski, 20:50

Transcript

0:00 I think the next stage will be how do you make it real time? I think this will happen this year.

0:11 You speak Polish. I speak Russian but we speak English. So we can do it. But if we say our parents are here, they will not be able to talk. Yes. When will be that moment when two different people with different languages will be able to talk without any language barrier?

0:29 Such a great question. You might have heard about part of why we started ElevenLabs which was so close to exactly that. When you watch movies in Polish all the voices whether it’s male or female are narrated by just one character, one person, which kind of loses that original emotion, tonality, content. Something that we of course would love to change.

0:51 And as you think about technology we already are seeing that it can preserve a lot of the voice, some of the emotion. Translation is a little bit harder. I think in itself is already showing incredible promise today, like it’s possible with some of the use cases.

1:06 I think the next stage will be how do you make it real time so that you can actually stream that content, whether it’s in media or other aspects. I think this will happen this year. In the next stage ideally you have some device, let’s say it’s a headphone or some other device like mobile phone where you can speak and it live translates to the other person.

1:27 Earpod is good because while you speak the other person can listen so it can stream at the same time. But to make it really adopted at scale, it will need to be a little bit smaller models and a little bit quicker. My guess is the next two three years we’ll see some of that being out there and hopefully in the next five it will be something that everybody can use anywhere in the world.

1:51 I’m very excited about that. I think it’s super needed. But I went to CES this year and I tried a couple of these devices. They don’t really work yet. So my newsletter is about machine learning and artificial intelligence. It’s quite technical audience. If you can tell me what technical challenges do you have to solve before it’s possible?

2:13 Of course. I don’t know if you are familiar with the Hitchhiker’s Guide to the Galaxy and the Babel Fish. That’s exactly where that’s going to head. Would love to be part of making this happen. But currently the process is like three-step process. You do the speech to text. So you try to understand what somebody says.

2:30 But beyond just the speech, ideally in the future, what you would do is you understand who is saying it. If you have many people, you can diarize it and understand. So speaker diarization needs to be there. You need to understand the emotions with which speech is being generated. This problem is still… Yeah. How do you understand emotions?

2:50 Exactly. Like there’s so few data that will have the speech and text but also the metadata of how something was said. So I think that’s a barrier of how do you create that at scale with high quality and something that we are working on.

3:05 So speech to text is the first part, then you need the LLM part or could be other models but probably will be the best solution where you translate it from one language to another. But now the important thing is that you of course want to preserve some of the original meaning and depending on how much of the sentence you take that meaning might be different.

3:27 If you have two sentences and one might be explained in the other one then the meaning and the way you should deliver it would be very different. So you can have a sentence “what a wonderful day” - okay the translation is easy. But if you had “what a wonderful day, I said sarcastically” - completely different. So there’s LLM in the middle and of course a wider set of phrases that people might need to explain.

3:47 And then you have the text to speech step at the end which also needs to take the emotion and the voice from the original and be able to recreate that on the other side. And depending on the use case, ideally in the same or similar length so it feels a little bit more seamless.

4:01 So you have those three steps: speech to text, LLM, text to speech. Where I think the space is today - LLM on the translation for some use cases like more flat deliveries, it’s good. But for a lot of other ones where you have a lot of niche words or phrases, harder.

4:20 I think speech to text is mostly in good spot for understanding speech. We released our model recently that also beat all the benchmarks which we’re very happy about. And of course that helped lower that barrier but still you will have a long tail of other languages that will be harder.

4:35 And the emotion thing, it needs to get fixed which isn’t yet. And the text to speech I think is very good but still we’ll need more of the context understanding from the speaker.

4:46 How would you solve the emotion or the emotional part? The first step would be to just create a very large dataset and we work with voice coaches and people to effectively create larger sets of audio that then you annotate of how you say things.

5:02 So even in our conversation, let’s say somebody would take that sample and then say in this sentence the emotionality is excited, calm, and the speaker starts every so often. And that would help of course to have a lot of those examples that then you can hopefully have the model understand new unseen examples as well.

5:21 So that’s the main blocker and of course then you might have different languages and different languages might have a different description. Whether you can rely on the same technology to translate it is unknown but that’s an interesting one.

5:38 In your conversational AI that combines STT, LLMs, and TTS - on what percentage, on what level have you achieved what we were talking about?

5:45 I think for some use cases it’s there. I think for some use cases it can really help. Let me speak to some concrete examples. We work with a company in the US called Hypocratic which is automating some of the work that nurses need to do around patient appointments that they don’t have time for. Like calling them to ask about how they are feeling, reminding about scheduled appointments - and that works.

6:12 That is something that is possible. And here the same people on the other side frequently aren’t an active mobile phone text user. So that’s perfect for them to be able to reach them. And I think there we are there and that’s just English to English.

6:30 Then we have this other company on customer support side called Elise AI where people are calling in a language A but the responders only understand language B but they need to live translate. And it works. It works very well because people wouldn’t have any way of reaching the other person before. They would literally need to hang up.

6:50 Now they can translate the content. What they cannot do yet is translate all the emotions in the same pattern. But in the customer support use case, that was the first barrier. So these work very well.

7:00 Now there are places where it doesn’t work so well. That variation of the first question you asked - the real-time dubbing or real-time translation element. We are working with some of the media companies where you would want to have a broadcast of a sports event and the listener hears it in their language.

7:20 But given there’s so many different names that are hard to pronounce, the ambience and the emotions of the commentator are the hardest that they can be - you need a little bit of the barrier there. I don’t think we are there yet.

7:35 So let’s say for a lot of call center, customer support, healthcare, education use cases we are there. We’re seeing real use cases being deployed. Whether it’s going to be just a matter of scaling that across - I think we will see more conversational AI agents. And the use cases which have highly emotional dependency, whether it’s across real-time AI dubbing or just conversation, I think that will be in the next 12 to 18 months.

8:02 That’s incredible. How hard was it to do this dubbing with the Lex Fridman podcast?

8:07 It was pretty hard given we wanted to make sure that every word, every sentence is translated in the right way.

8:13 It sounded incredible. Yeah. Thank you. It was also, given the beginning and the mission of actually making it happen and then being able to work with such an incredible person like Lex and the others - seeing that deployed to such a broad audience, that felt very close to heart when we were able to do it.

8:35 But here we did spend a lot of the QA time both on the translation to make sure it’s perfect. Some external parties were coming in to help us on that and on audio as well. We just wanted to make sure that the way the speakers spoke were represented in the same way. So we would listen side by side the previous and the new, compare them.

8:58 And of course the hard thing with podcasts that are distributed on YouTube is the length needs to be the same. So even if one language is much longer than the other one, you still need to keep the length the same. So you end up trying to use the original sentence, you need to maybe paraphrase it a little.

9:15 And only when you paraphrase it then you generate the audio on the other side. But of course now from the original you have longer content, different emotions - they paraphrase it, how do you do that? Very, very hard. I don’t remember the exact number but I think English to Spanish is 30 or 40% longer when spoken. So if you need to keep the length the same, it’s hard.

9:40 Also the conversations that Lex has, if we are able to do them semi-automatic or fully automatic in the future, it will be a good proof point that the technology is really there. It’s like a good stress test for us.

9:52 So it took you a day? It took us longer than a day. I think it took us depending on the podcast between a week or two weeks. The hardest part was translation where we worked with another party to do that perfectly. And then translation in context of audio - so you need to translate but also keep the length. Translation and then the iterations on translation and length, that was the longest step.

10:15 Just the audio part and generating great voice was relatively easier. But then generating great voice with the emotions in that time - much harder.

10:25 Wow. No, I thought it was very bold to go for it. Thank you.

10:30 I don’t know if you know, I think 3 years ago I was reading your work and I think I even tried to reach out. You did? You did! And I reached out to you this December replying to your message on LinkedIn asking for your predictions, but you didn’t reply to me. Okay. Maybe I can still do that. Yeah. We’ll do it next year.

10:50 But let’s do this current situation. Someone walks in and that’s noise. How do you deal with that in conversational AI?

10:58 We have a pretty good model now that’s trying to, based on the volume of somebody speaking, based on the context of what was said before, try to detect like is it a real interruption or a fake interruption - like somebody speaks and speaks to it.

11:15 It works pretty well. It’s not a bulletproof solution. The bulletproof solution that we think will be one of the examples in the future is that you effectively, let’s say I’m speaking to an agent, based on the first sentences that I say then I encode the voice and then I check for that voice.

11:35 Which would be a lot more complex, but then of course you have different use cases. Sometimes you might want to have few people conversing. So what we are thinking is you will have the ability to set it up when you have an agent. Do I set it up that it auto-detects the speaker and then customizes the agent responses just to the speaker?

11:55 So let’s say it detects that it’s your voice based on the sample of what you said. And then in next responses it checks - okay is that still the same voice that said the things before, just by comparing the embedding. If we do it quickly enough then you can just effectively continue speaking.

12:15 And if it detects another speaker and speaker embedding that it processes - is it the same? No, let’s continue with you before interrupting. The real challenge is how do you make it quick enough so you do it still seamless?

12:30 But we think we can do that. Something that some, especially in noisy environments, are requesting with our work. And I think we’ll ship it in hopefully the next quarter. So Q2, which would help.

12:45 And of course sometimes you might prefer to have not auto-assigned to one speaker but have plenty, then we will keep it broader.

12:55 How’s the latency? Did you solve this problem?

13:00 Current latency is end to end, so with everything, with network, with interruptions, from one response to another is between a second to 1.2 seconds depending on the region. So very quick. Yeah.

13:15 The text to speech, which is the part of generating from text to audio, we have the quickest model which is 70 milliseconds. Then the transcription and the LLM is where it takes time.

13:30 But the main thing is of course when do you interrupt - like do I interrupt when a person stopped speaking, maybe they will continue their sentence. So we have behind the scenes in our conversational framework a number of smart mechanisms from detecting whether, based on the context, it’s a natural end of the sentence, is there enough of the silence that followed. We combine all of these signals together and only then generate a response.

13:55 But while we do all those detections we already pre-generate some of the LLM parts so we could stream it much quicker. So that’s why it’s so quick and it feels natural.

14:10 While at the same time, if I’m going too far it would stop me as well. But the alternative to that solution would be to do a true multimodal experience where you train those three models together.

14:25 But if you do that then you lose some of the control you have if you do the three-step solution. Because now you have the LLM, you want to make sure that it stays on the topic that’s relevant, that it doesn’t go off the rails. Especially in customer support or healthcare you need to make sure that it says what you want it to say.

14:44 So that’s what is important there. And all those checks that need to happen might also introduce additional latency. In the multimodal you will not have as much stability but you will have more naturalness and quicker conversation.

15:02 We are also working on that at the same time but from the research aspect. For big enterprise companies the first solution is usually the best and what we recommend.

15:15 Yeah, I was actually going to ask about it because I thought the next step for conversational AI and combining those technologies would be something multimodal at some point.

15:25 At some point for some use cases where emotionality will be so important or the real connection - we think as well will be the case in the next year or two. And maybe to make it more specific - in the interactions where you don’t maybe need to take an action on the other side like issue a refund or cancellation, then the multimodal experience will be potentially more useful.

15:50 Of course you still want to be careful - you wouldn’t want the therapist to be multimodal if you are not 100% clear that it works all the time and there are right safeguards. You can hallucinate every so often with responses both on the text and audio level, it’s a little bit less stable.

16:10 But in the long term, in the next 3 to 5 years, there’ll be more stability and a lot of proof points that multimodal is the potential way to go for those use cases. And then we will see that happening. But in the meantime the three-step solution gives you the pieces that are valuable. Exactly. It’s scalable, it works, it’s emotional, you can build your knowledge base into the agents and do it rather simply.

16:35 You started as a research-first company. How deep are you still involved in technical stuff?

16:42 Me personally? You personally. Oh, much much less. It’s definitely one thing where it does come to us still as a company. Maybe to speak through - my co-founder is incredible. He’s the genius and the brain behind all of the models that we’ve done. He was able to assemble some of the best researchers in our team and audio space.

17:05 So now we have a very strong audio research team that keeps putting out incredible models and benchmarks. So we still do and will do for the next years audio research and we want to build the models that will be the frontier and the best models - whether it’s text to speech, speech to text, conversational agents, whether it’s other aspects of audio understanding or audio generation. That will definitely continue.

17:35 On the personal side, what I still do and what we really believe in is that beyond research you really need the product aspect and what’s the end-to-end solution for the customers. It’s not enough to have a text to speech model that has good narration, but you also need a product or workflow that allows you to do an entire audiobook or create an entire voice agentic experience, bring your knowledge base, bring the functions that might interact with your real life.

18:05 So I’m pretty close to that aspect. And of course as we think about deploying that technology to clients, one thing that we do maybe less traditionally is that we have engineers that will work directly with the clients to try to understand what they need, what’s their process, and then build that solution closer.

18:30 Do you miss being more involved in research? So I was never involved in the research research, but some of the product aspects of building the product itself and especially back in my previous company Palantir, I used to be much closer in pipelines and working with the clients on the optimization problems.

18:50 Some parts of that I definitely miss, especially the mathematics or some of the engineering thinking that was involved. But still the thing that stays true which I love about my current job - I still work with a lot of clients, I speak to a lot of them, try to understand the problem, try to understand what’s the solution we can do, which ultimately then feeds back into how do we build the product aspect.

19:15 And while I don’t do any coding myself now, it’s still nice to be able to see how it can be deployed across.

19:25 Congrats on your recent huge brand. Just recently you had a collaboration with Lex Fridman. This week you had partnership with Google Cloud. ElevenLabs seems to be everywhere. What is your strategy?

19:40 And going back to when you reached out to me back then - I saw that was such a strategic move to work with the newsletters. So what is your strategy?

19:50 One thing that we are trying to stay true to is to really understand what is the problem that we are trying to fix and how we can build the solution around that. But the other piece that really comes here is that even if you know what the problem is and you found a solution, still most of the world or most of the customers don’t know about that.

20:15 So the big question is how do you tell them that the technology is there to finally fix those issues. And given we started with research and built a lot of the research, people don’t really trust or believe if you just say “this is the best human-like model” or “this is the most emotional model.”

20:35 We were looking for ways to show the wider world not by just saying but actually showing the use cases in real life. So instead of telling - show, not tell. One part of the strategy was let’s open the technology to creators, to developers to use in their use cases to show what’s possible.

20:55 And in the process even we were learning of things that we didn’t even know were possible. So it’s both ways - was good. But also part of that strategy was we think the research is such high quality. We know that there are clear problems that this is fixing - from narration of audiobooks and newsletters, through to dubbing of podcasts, to creating voiceovers for movies, all the way to agents.

21:25 But the way to show that to the world was and is working with the creators, the developers, the innovators that then show the use cases in the breadth. And at the same time work with some of the biggest companies in the space to go deep, understand the scale, the security, the compliance that’s needed.

21:45 Last week we also were proud to announce our partnership with Deutsche Telekom which is a completely different angle. But part of the reason we are excited for that partnership is that they already have seen how the podcasts and how people can engage with the voices in the calls and see that the quality is there.

22:02 What we now are focusing on is how can we go even deeper and fix that end to end. And that’s kind of how we’re thinking about it - on one side provide access and wider access to the best of the technology we can, on the other go very deep with very specific end-to-end solutions for the enterprises.

22:20 Thank you so much. Thank you.