Learning From and With AI: Duolingo’s Zan Gilani
27:39 Share

Today we're airing an episode produced by our friends at the Modern CTO Podcast, who were kind enough to have me on recently as a guest. We talked about the rise of generative AI, what it means to be successful with technology, and some considerations for leaders to think about as they shepherd technology implementation efforts. Find the Modern CTO Podcast on Apple Podcast, Spotify, or wherever you get your podcasts. Learning a new language is a complex process.

One app company looks to AI to help with personalization, context, and motivation. Learn more on today's episode. I'm Zangilani from Duolingo, and you're listening to Me, Myself, and AI. Welcome to Me, Myself, and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

And I'm Sherven Kodubande, senior partner with BCG and one of the leaders of our AI business. Together, MIT SMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Welcome. Today, Shervin and I talk with Zain Ghilani, Principal Product Manager at Duolingo. I'm personally pretty excited about today's episode because Duolingo is in a really interesting position. In a lot of our show, we end up talking about how organizations are using AI to augment or to improve human decision-making, and that's great. But Duolingo goes further and focuses on how humans are learning with AI, personalized education at scale, which is something that appeals to me.

Anyway, that's enough from me. Let's hear it from Zan. Zan, thanks for joining us. Thank you. It's a pleasure to be here. Zan, first, can you give us an overview of Duolingo and the company and what your role is? Absolutely. Duolingo is basically an app that teaches languages. We teach 40 plus languages right now. Most of those people, by the way, are learning English.

And most of our learners are also learning on mobile phones or Android and iOS devices. And we are the most downloaded and the most used educational app there is, languages and otherwise. We have around 16 million DAU or daily active users and 40 plus million monthly active users as well. And I've been at the company just coming around to eight years where I am a principal product manager and

for the experimental AI team. My primary role is leading this team that is working to teach more effectively using generative AI and also more broadly I help the company set up for success when it comes to generative AI. One of the main use cases we have now specifically for generative AI is a new subscription tier. Duolingo is a freemium product. The mission is to provide

the highest quality education possible, make it accessible to as many people. All you need is an internet connection. So we have this free version of the app, which is what most people are on. That's really good. Most people at the company spend time on that. And then we have subscription tiers. And we recently just launched a new AI-powered subscription tier called Duolingo Max. And I'm happy to go into the features as well. So experimental AI, it's sort of like your R&D lab, if you will?

Well, I'd say actually more than half of the company is working on R&D broadly, kind of all of the people who work on product. And within the folks who work on learning or teaching effectively, you could think of it as a lab. Yes. Great. Maybe for our audience, you could...

list the variety of use cases or places that AI is being used. I mean, of course, from the obvious one, personalization and gamification that a lot of the apps have, but just tell us how AI is being used.

Duolingo has been using AI for as long as the app has been around, which is coming up to a decade. In fact, it was founded by a computer science professor, Luis Fanon, at Carnegie Mellon University and his graduate student at the time, Severin Hacker. Right now, it's used in a number of different ways.

On the growth side, for example, a lot of work goes into optimizing notifications. The thing about Duolingo that makes it different from other apps is that it really tries to help you stay motivated. For us, pedagogically, we think that the hardest thing about learning a language is actually staying motivated. Because remember, this is something tough to do. It's especially tough to do by yourself.

On a phone, you know, that's really a tough thing. So notifications and the timing when you get them, the messaging that it has really does affect someone's ability to stick with it in the long term. So we put a lot of effort into machine learning for notifications. We also use it on the content generation side. So we are trying to create useful, appropriate content that

We use it there as well. And then in a separate arm of the company, which is the Duolingo English test, which is basically an English proficiency test that students have to take when going to university, kind of a competitor to the TOEFL or TOEIC or IELTS. Most Americans haven't had to experience this. We use it for detecting cheating. We use it for generating content in a bunch of different places as well.

That's fascinating. And on the actual core of the app, which is teaching you language, how dynamic is that? How personalized is it when it adjusts to a person's way of learning? Personalization plays a role.

but a lot of the experience is curated by learning scientists, by educational content developers. For most people, the order in which you want to learn things is going to be roughly similar. For example, everyone learns the present tense before you then go on into other complicated tenses. But where we do use AI for personalization specifically is for something called BirdBrain. And this is basically a system we have

that is optimizing for the difficulty of questions that you are going to see in practice sessions. One thing to note over here about Duolingo is that it's completely interactive. It's meant to feel like a game. So every time you're learning, you're kind of answering a mini exercise and then you get it right or wrong. You always get feedback.

So the point of BirdBrain is to try to come up with the Goldilocks level of difficulty, because basically if something is too hard, it's frustrating and you lose motivation. If something's too easy, it's boring and you lose motivation. BirdBrain is one of those things that we use for personalizing practice in particular. Sure. And one thing I think that's funny about that is that in describing his product, he

We just took for granted the speech synthesis, speech recognition. These were avant-garde things 10 years ago. And now, Zain, you didn't even bother to mention those as applications of AI. I think that's sort of stepping back, seeing what now is normal. That's right. And I think one thing you have that's quite interesting and maybe unique is just the number of feedback loops you must have. I mean, usually in a lot of

other applications, when you think about corporations and retail and banks and telecom, etc. It takes some time to get feedback and you're almost getting instantaneous feedback on so many experiments or mini experiments from so many people. Maybe tell us a little bit more about just the speed of learning and the speed of adaptation of your algorithms.

There's a nice bit of mirroring over here, which is that there are principles for learning effectively that we try to impart into the product itself. And then as a company and as an organization, we are trying to operate on those principles as well. Getting fast feedback really matters when you're learning a language, but it also matters when you are learning

working on the product itself. And from a very early time, we basically built out our own A-B testing infrastructure. And it's pretty core to the product process. That's the central component of how we do product at Duolingo. So at any given moment in time, you have hundreds of experiments being run on things that are very granular, the positioning of buttons on the pricing, on the page for the price of our subscriptions, your classic bread and butter experiments, to also...

huge new features, new tabs. Basically, if it's not a bug, it's probably the case that we're running this in the form of an experiment. We've gotten very, very good and very, very sophisticated at doing this.

Shervin is going to roll his eyes because I'm going to go academic for a minute. But you're a classic case of experiential learning. And actually, I'm going to give you a little bit of segue to talk about your Max product here, because I think there are two pretty exciting parts of that. This role play, which is getting to the academic concrete experience, part of experiential learning. And then this explain my answer, which is a reflective observation and critique. So tell us a little bit about both of those and how those are tied to learning outcomes.

Definitely. I think it's useful to talk about where we were before adding those two features. Pedagogically, Duolingo has always been about implicit learning. So learning without telling you explicitly, this is what you're going to learn and this is how it works. And that is the DNA of the experience because we think...

It's just more automatic when you do it that way. People also get really demotivated and off-put by complicated grammar tables, etc. And then the second is that you're learning on an app and you're doing this interactive thing, these almost game-like exercises. The two things that have been missing or limited is that one, some people do actually want

explanations and that that can be really efficient and effective when done in the right amount and in the right way. So Explain My Answer is a solution to that problem. And the nice thing about Explain My Answer that really only works thanks to modern large language models is that the large language models are really good at providing jargon-free, concise explanations of the mistakes you make.

It's really tough even for teachers to try to do that without using jargon and also without being right on the money. And so we are able to do that with Explain My Answer pretty well. It can give you follow-up illustrations. It can give you examples, all of these things that make large language models, by the way, just good for anyone trying to learn anything by themselves. For me, it's life-changing. Or anyone trying to give a speech. Or anyone trying to give a speech, yes.

And then the second one, role play, is that ultimately, for most people, language learning is not an academic exercise. They're trying to use it as a skill in the real world. And that's a skill in and of itself. So how do you give people the ability to practice that? How do you get them to simulate this? Chatbots are exactly one way to do so, because now what you have is something that can speak to you at the level of difficulty that you're at.

has infinite patience, and can then actually give you feedback afterwards as well. Even if people had access to native speakers, which they don't generally, you'd still need a native speaker who has infinite patience and who really, really can understand what you're looking for. So role play is basically trying to provide that experience into the app. Rather than just having a text interface like ChatGPT and just having you speak back and forth, basically what we've done is we've broken down

or scenarios into these bite-sized experiences where you're talking to a chatbot who's going to be a barista or a friend that you're borrowing something from. And then you try to go to a very specific short experience. Sam, I have a question for you because Sam, as you know, is a distinguished professor and he's been teaching people

including people that are not in his class, like me, a lot of things. So my question, Sam, is in all seriousness, what makes a good teacher in your opinion? Actually, Dan hit on a couple of things right there. And that's part of why I was feeling somewhat convicted in his conversation. This infinite patience that he talks about.

That's really important. And it's also really hard. Yeah. And meeting people where they are is a fundamental problem that we have in education. I mean, I sit in front of a room with 35, 40 people and they are not homogenous and meeting people where they are is exactly what ZAM was talking about. So yeah, you have to know the material well, you have to keep people engaged, but you also have to keep track of where people are and what they know, you know, present stuff that's not too easy and not too hard. And,

I'm very attracted to this model. So then what's next? Well, yeah, that's exactly where I was going. What's next? I mean, if we can do this with language, what else can we do this with? Where does it work? I meant just for language. So meeting people where they are, having very good knowledge of the student and their progress, and then being able to do role play and lots of experiential learning and being able to explain and guide students

in a more jargon-free way, and in a way that you could repeat it, and I don't have to rush to take notes. I mean, all of these things, right? What is missing? I'm hearing this, and this is like better than a regular teacher, sounds like it. But in your view, what is missing for this to be a perfect teacher, in Zan's view? I think we are still a very, very, very long way away from being the perfect teacher.

for a number of reasons. But I'd also say we don't believe, and I certainly don't believe that at the end of the day, all you need is a chat interface and that most people will be able to get really far with whatever they're trying to learn. So to answer the first part about why we're far away from a teacher,

Well, some of this I think is unknown, but there is a real impact to developing a relationship with another human being when you're learning. There is like an emotional part of learning something new. And there is a human connection that can be as simple as the fact that you show up for your tutoring appointments because you don't want to let your teacher down to also the fact that

that tutor can be really engaging with their facial expressions, with the cadence of their voice, with the illustrations that they can tell, the way they can relate to maybe their own lives. So that we're still very far away from. And then when it comes to just, hey, why can't a chatbot teach you everything?

For most people, you still have to be organized and being organized is quite exhausting. For example, just deciding what to eat next is often what you spend all of your time doing or deciding what movie to watch. And similarly, deciding what to learn next is tough. So someone on the product side has to figure out the right sequencing of things for you to learn. Similarly, being creative is exhausting as well.

For a real life tutor, it would be really difficult for them to put on a lot of different personas and do a lot of role playing. Then you'd need a tutor who's also an amazing actor. And AI gives you the ability to do that with a virtual tutor. But even then, we don't want a learner to have to come up with these scenarios all the time themselves.

The last two things I'll mention is that learning is also multi-sensory. It's also contextual. What you'd want ideally is some kind of learning in context happening as well instead of just with a chatbot. And the last thing to mention is people really do need a sense of progression in order to stay bought into learning a language because learning

actual progress is so slow and so incremental, it's really hard to see. And so what you have to do is you have to show people progress bars that are filling up and you have to check in. I think there are probably ways that you could simulate that in a tutor experience, but that's some of the reasons why I think we care about building the experience outside of that as well and building a path that you go along and all of these other things that go into making Duolingo what it is. Zain, I want to come back to one thing you were talking about that I'm pretty attracted is this idea of

figuring out what's the next thing you need to learn. I mean, historically, every class that I had in languages, we had a book and we moved from chapter one to chapter two, and somebody somewhere really thought through the sequence of that. But at the same time, I don't know that that's necessarily optimal for all students. And we think about recommendation systems in Netflix or in music. You

This seems like a great opportunity here. What's the one thing that I could learn right now that would make the most difference? Or what's the one thing that I don't know that's keeping me from opening up this next section? That seems like a huge opportunity. Yep. And I would break that into two pieces as well. One of them is just...

generally what's an effective order to learn in and what's an effective order to teach in. And it's especially useful for people who have some prior proficiency in

which is most people who are trying to learn a language on Duolingo, particularly English learners, because they've done it in school in some kind of way and their knowledge is patchy, as we like to say. And in those cases, the order is going to be quite different person to person because of different experiences. And then the second aspect of it is the things that

that you specifically are having trouble with in the moment that you need to focus on more. Maybe you're just making way more mistakes for a particular concept. Maybe it's verb conjugations are the problem, or maybe you just don't know how to introduce yourself well. And so there's opportunity to say, actually spend more time

in that part of the experience. It's not that every curriculum is custom made for a unique student, but it is that where you put your attention in and your focus and the amount you put it in and the order you do it in is directed by some kind of tutor. I will say that's a tough thing to do. It really is a tough thing to do. We're not there, but that's part of the dream, certainly. So the show is me, myself, and AI, and we didn't

learn a lot about you and yourself. So maybe tell us a bit about your background, how you ended up in this role and what the journey was like. Yes. How far back should I go? Do I start with grandparents, parents, myself? Up to you. Okay. I'll skip the extended biography. I grew up in Karachi, Pakistan. That's where I was raised. And I came to the US for undergraduate studies.

And I actually studied political science and East Asian studies in college because, one, I was interested in languages. So I took Chinese. And when I was learning languages in college, one of the most exciting things was always trying out tools to see what was effective or not.

So it's the interest in languages, it's the interest in technology that got me to Duolingo. And actually, I started in marketing because I'd done some marketing and advertising internships before that. At that time, we were around 50 people at Duolingo. And we went from having just one team that worked on the app

to having teams that work to monetization, learning and growth. And so I started working on growth as a marketing person. And then about a year and a half in, I switched to being a product manager. And for most of my time at Duolingo, I have worked on growth, which is to say user retention, whether that's new users or existing users, growing our presence in Asia. That was super interesting. Things like that and motivation and what gets people motivated and what keeps them motivated has always just been an interest of mine. And then

The switch to working on AI as the focus and leading this experimental AI team came the end of last year because basically we ended up partnering with OpenAI for their GPT-4 launch. And the reason why we ended up doing this partnership was actually because I knew the product manager for the GPT-4 launch at OpenAI. And when we got together, we quickly realized Duolingo is the perfect place

use case for the launch, because for the launch, what they really wanted was to show that companies are actually using this to transform their businesses. And these are also companies that are trying to do good for the world, et cetera. And so we as a company, based on that partnership and getting access to GPT-4 early, shifted a bunch of priorities, started really leaning into AI. And one of those things was to start this team that's trying to work on experimental features.

Very exciting. And actually, from a motivation standpoint, you'll be thrilled to know that my son is very proud of his Spanish language streak. Amazing. We've got a segment where we ask you a series of questions, and these are just rapid fire. Say the first thing that comes to your mind. What's your proudest moment in AI?

We as a company launching Duolingo Max has been super exciting and just paving the way for a new era of teaching. It's been created actually by the Max team at Duolingo, which is led by Edwin Bodge and Bill Peterson. And it's super exciting. I'd say personally, actually, the first product that my team has been working on is actually the experiment is going live. You could kind of think of it like TikTok for reading, but

It's something where all the content is generated by GPT-4. It's giving you a huge breadth of content because people need reading practice in lots of different varieties, and it's also listening cloud. So very excited to see how that goes. Very cool. What worries you about AI? I'm generally very optimistic, but I think the thing that worries me is just that the pace of change is so quick that it will be destabilizing.

And the goal for us as a society is to make sure we can stay stable in the face of very, very rapid change. Because rapid change is what we want, actually. We want progress to come quickly, but without all of the destabilizing effects. So that's the fear. I think it's a solvable problem, but that's the fear. What's your favorite activity that does not involve technology? I play a lot of soccer. Yeah.

Funnily, there is technology involved with that as it turns out, but yes, I play a lot of soccer. That's one of my favorite activities to do. So what first career did you want? Poli Sci, I think you mentioned. What did you want to be when you grew up, when you were a kid? Lots of different things. There was never one answer. I wanted to be a writer, a journalist, a cricket player, a cricketer, work in international development, even wanted to be an architect. You were on the table there. Yeah, a lot of different things.

What's your greatest wish for AI? What are you hoping we can get from this? I think AI is the way that we can create the fastest progress. Where progress here, I just mean as many human beings are living the lives that they choose to live, that they're flourishing. I think that is the highest level goal and that is

the meaning of life for most people. I'm very excited to be part of that effort. Well, it sounds like you are. Zan, thanks for taking the time to talk to us. I mean, I really genuinely feel like this is a hugely different and exciting opportunity for how we as humans can get better. And it sounds like Duolingo is doing a lot of that. Thanks for joining us. Thank you so much.

Thanks for joining us. Next time, Sam and I speak with Jeremy King, Senior Vice President and Head of Engineering at Pinterest. See you then. Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn't start and stop with this podcast. That's why we've created a group on LinkedIn specifically for listeners like you. It's called AI for Leaders. And if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and learn more.

and gain access to valuable resources about AI implementation from MIT SMR and BCG, you can access it by visiting mitsmr.com forward slash AI for Leaders. We'll put that link in the show notes and we hope to see you there.