#18 Innovation in Education: Duolingo's Approach to Immersive Learning with Megan Bednarczyk
50:32 Share

We're competing with TikTok and we're competing with Instagram and Snapchat. If it feels like a textbook, nobody's going to spend time on it. We have to create fun they can do in their five-minute commute or the 20-minute subway ride. We're trying to make language accessible to all.

Looking to these games, seeing how they use animation and delight to get people engaging with it, and we can take some of those principles and apply it to learning, then that benefits both us and the people that are trying to learn a language.

Hello, everyone, and welcome to Experts of Experience. I'm your host, Lauren Wood. And today I am speaking with Megan Bednarzek, a product design expert who is currently the staff product designer at Duolingo, the ever so popular language learning app.

In her role at Duolingo, she leads the Duolingo Max team, which is their latest generative AI-based subscription tier. I'm really, really excited to get into this because there's so many juicy insights about how they have been incorporating AI into their app, which I'm a huge fan of.

Previous to Duolingo, Megan led design at Geneva, a group chat startup, and she also spent time at Digitas. Throughout her career, she's worked with a number of heavy-hitting clients such as HBO Max, Audible, JP Morgan, and Amex. So today we're going to dive into Megan's thoughts on UX and the future of AI in design. Megan, thank you so much for coming on the show. Thank you for having me. I'm excited. Yeah.

So, as I was looking into your background in Duolingo, it was ever so apparent that you've worked on Duolingo Max, which, as I mentioned, is an AI-powered platform.

language teacher. And you worked on this product in collaboration with OpenAI leading up to the launch of GPT-4, which is super cool. And I can only imagine that there was a lot to learn considering GPT-4 hadn't even launched at that point. So I'd love to understand as you were approaching the launch of this product,

Just in general, what did GPT-4 open up for you in terms of what was possible at Duolingo? Yeah, that's a great question. So we found out about GPT-4 about like a little over a year ago at this point.

I still remember the first moment that we started playing around with the prompts. We had like this little Slack bot that somebody had uploaded. And it was funny. We were working on a different project at the time. And our CEO, Louis Van On, came over and he came up to our computers and he was like, you have to see this thing.

So he had us test it out and he had us basically like ask it a bunch of questions. And again, this is, you know, this is the first version of it. This is super rudimentary. So we asked it things like, explain this mistake that I made. We try to like make it practical to do a lingo. We're like, you know, I said this thing and tell me what I did wrong.

And not only did it say like exactly what we did wrong, but it explained like the grammar rules and the reasons why. And we tried some other things were like, you know, write a story about.

A girl who lives in Guatemala that dreams of being an astronaut, right? And it wrote this pretty beautiful, like wonderful story. And then we're like, okay, like trying to play around with like, here's some other things we like to test to do a lingo or like rewrite it at a beginner level. And it was able to take more of that advanced language and rewrite it in like really, really simple terms.

We tried things like add this particular word, rewrite it using like a specific kind of personality. We have these characters at Duolingo that have these different types of personalities. So like that's kind of always one of the things that we like to try. We hadn't obviously translated it to many different languages, you know, explain certain words. And like ultimately, we were just blown away by what it was able to do. For years at Duolingo, we had

been looking at a number of gaps in our product. And what we really wanted was to put like a tutor in front of each and every one of our learners so that they could actually have a really immersive and personalized experience on Duolingo. But, you know, a tutor is really expensive and like a, you know, real and human tutors are definitely the best way to learn a language, but that's not something

that can scale to all of the learners we have that don't necessarily have that money but still want that type of education. So after seeing the quality in this initial model, we were like,

This is something that we thought was probably like four to five years away. And we realized that we were actually there today. Obviously, while this instruction will never be as nuanced and personalized, it's really scalable. And we think that that's really powerful. So that was that big first moment. And honestly, we pivoted our entire team's direction and strategy in the matter of a week. We were like, all right, we're going to build two features using generative AI and we're

So we're the monetization team. So we're like, okay, how do we give our users access to these features? And obviously there's a cost associated with it. It's something that could go down in the future and we expect it to. But we kind of had to package it underneath one of our monetization tiers. And for a while, we'd been wanting to expand monetization.

our tiers to include like a higher tier. We wanted to like basically give people more options and expand beyond the offerings that we already had. And this felt like a really good opportunity and a way of packaging those generative AI features. So saw the demo, pivoted everything, and we're kind of off to the races in a very short amount of time. Yeah.

How long did it take you from that moment of, okay, this is what GPT-4 is and this is what I can do to actually launching a new product? That was about... It was still a very short timeframe. That was about six months. It was between October to March is when we officially started rolling out that MVP. So we basically had...

two quarters to design two brand new zero to one features using this new technology and figure out how to package it as the IR subscription tier. So very short amount of time. And partially that was because one, when we first got access to that demo, we decided we were going to be a partner.

with OpenAI in their launch of GPT-4. As a company, when they first came out with this model, they wanted to release it with a few different sponsored companies because they wanted to give everybody some examples for how this technology could be used for good. They cared a lot about that and they wanted to showcase

important uses of it across a multitude of different businesses. And they were really bought into our mission and what we are trying to do at Duolingo. So it was a natural fit there. So we had to get all of this done by the time GPT-4 was announced to the public. So that was our pretty strict timeline there. Amazing. I'm curious to know if you were also using it to help you build the tool and integrating it into...

your systems and kind of redesigning how you were approaching the development of a product in using this technology? We did a lot, especially in those like first few months, just to like understand the intricacies of the technology and the different ways we could use it, you know, both for ourselves and our own like processes and also in the product and what it could actually solve for our learners. So we did a number of like company hackathons where we'd all come together and just

kind of go crazy with like, what can we do with this thing? And...

We found a lot of interesting ways to incorporate it into our own work streams in the way that like it can support the human employees that are using it, not replace them. Yeah, I think it's a very effective tool for kind of helping you get like the first start to something, right? Like that first draft or that first structure, like I want to do this thing or I'm thinking about this thing or summarize these notes.

So we use it a lot for things like that. But like then you kind of always have to go back through and edit it and apply your own knowledge and experience to like turn it into something that can be used. But yeah, absolutely. We've been thinking about how we can use it in our processes as well as, you know, actually creating a product around it. I think hackathons can be such an amazing way to just go like as far as you can go with something. I

I'm curious. I always like asking this question, like what got left on the cutting room floor that was a really good idea that you might come back to? Oh, honestly, honestly, so many things. We kind of have a few different teams that are working on this technology. We have our team that like actually right now, you know, have has these live features in the app that are built on.

on generative AI and using GBT4 in real time. We have other teams that are almost like R&D type generative AI teams, like thinking about like all these different features and all these different use cases of it.

I think like something that we've like tossed around a lot that could be fun is like creating a type of like Mad Libs type experience where you put in like a lot of the Duolingo things you love, like choose this character and choose this setting and practice this travel scenario to my upcoming trip to Barcelona in, you know, 2024. So like thinking about ways we like allow people to craft their own learning experiences through like plug and play is something that we...

definitely are excited about and maybe will incorporate in the future. That's great. How about the challenges that you faced? I mean, especially not only is the quick timeline a factor, but just using something that is completely new. And at this moment, when you were introduced to the technology, like no one's really had that much experience with it. Maybe you understand it conceptually, but like you're really doing this for the first time and everyone is.

What were some of the challenges that you faced? Yeah, I think honestly, one of the hardest challenges at first was just figuring out at which stage we need to incorporate it into the design process. And I think early on, we started to make some assumptions for like how it could work. And I think we were a little bit limited by like how we approach creating, you know, most features at Duolingo that are not reliant on this technology. And we were like, okay, well,

One of the features that we built was this feature called Roleplay. It's a conversation practice tool. And we're kind of approaching it from almost like a two-structured way. We were like, all right, these are the components of a conversation. You know, we're going to basically like have people navigate this conversation. We're going to give them like three tasks to complete during the conversation. And then once they've completed all three tasks, they'll be done.

Let's take the example maybe of you're ordering coffee from a coffee shop. So we're like, okay, step one is to introduce yourself to the barista. Step two is to order your coffee. And step three is to decide how you're going to pay. So that's kind of how we started thinking about how we'd actually have users engage with these conversations. Like every scenario would have these tasks to complete.

But then once we started actually like testing this out and seeing how it played out and design and using this generative technology, you can't actually plug every single conversation into a formula. And if you said anything like that didn't fit perfectly within how we had structured the conversation,

it just like wouldn't work and it would kind of go off the rails. So like, let's say I say, Hey, I'm Megan. And I want a banana. And then the character starts talking about bananas. And then I never actually get to the task of ordering my coffee so that I can't complete the conversation. But, but I'm

I'm still having a conversation and I'm still practicing my conversation skills. And maybe I want to talk about bananas and I should be able to talk about bananas. And the really magical, cool thing about generative AI is that I can talk about bananas and

And the, you know, the character can respond to me and we can go off in that direction. So we tried to limit it too much and like didn't realize that actually we wanted to like lean away from structure and kind of allow for this freeform conversation. And that's where it would actually start to feel really magical. Mm-hmm.

I mean, it's so true. I actually am currently using Duolingo to practice my Spanish and I am in the very, very early stages of it. And I'm not using Duolingo Max yet. But I'm like, okay, I am learning what coffee is and bread and all these things. And then I go to use it and this exact thing happens of...

Oh, no. Now they're talking about my shirt, maybe. Or they're like commenting on something because my accent was so good. They think that they can go there. So that makes a lot of sense that and such a wonderful application of generative AI.

Yeah. And like, I think the big process learning for us there was that like we had to figure out our feature solution in the prompt before we ever jumped into design. So like whenever we had like a product strategy or idea, we'd create the prompt for it and try it out, you know, just in like a text editor type of interface to play out if it actually worked. And then once we could validate that, then we'd actually move to the next step. So that became like a key part of like

Any feature improvement or update or new feature that we build using generative AI, you have to start with the prompt and you have to like test your product ideas early and often just to validate what works and what doesn't. It's not something that like you can,

imagine with just boxes on a screen, you have to engage with the actual tool. And that's really what the product is all about. So that was an important process learning for us. And now it's something that we incorporate into every improvement that we make to the features and every new thing that we do. Cool. Tell me a little bit more about that process of testing and like, how do you structure it and really make sure that you are pushing the tool to the limits?

to make sure that you're getting those edge cases? So it's definitely something that we've tried a lot of different methods for and have like slowly evolved our processes over time. But I think we've gotten into a pretty good rhythm now where we basically created an internal tool that allows us to test any new prompt update and basically like

simulate, you know, what's we're still talking about our conversation practice feature because this is like specific to that. But we can basically simulate any conversation and any update we make to the prompt and like test that conversation in this tool that we've built. One of the big things that

we've focused on in the last few months or so is we have these characters in Duolingo. And each of the characters has their own personality. Some of them have their own little fan clubs. Everybody knows Duo, but in Duolingo,

this feature called role play. You get to talk to all of our other characters. So we have this, this character named Lily. She has purple hair. She's like probably one of the more popular ones. She's like the snarky moody teen. And you know, we have this other character named Oscar and Oscar is this very like pretentious, like art snob. So they all have these like real,

like worlds built around them and these like really rich personalities. And that comes across like when they show up in lessons, when they show up in stories in the app. And like, it's really kind of one of the things that differentiates Duolingo from other products and differentiates like the features that we create and the way that we approach language learning in this fun and immersive way. And originally,

when you kind of see that character face and you were talking to it and that personality didn't come through, it started to feel really dry. It didn't really have that like quirk and storytelling that made Duolingo special. So one of the things that we did was we basically wrote specific

specific um prompts so i'm using this word prompt because this is like a you know a key part of working with generative ai it's like a series of different prompts or instructions that we write to it to get to act the way that we wanted to act that that's the very simple way of doing it you know when you uh are engaging with chat gbt on the internet you like ask hey what's

summarize this paragraph or give me 10 words that mean stellar. That's, that's a prompt. And our system is a very intricate, complex system of multiple prompts working together that creates our feature. So we created specific prompts for each character that like basically had all these notes about their personality, like

has a son, is really friendly, blah, blah, blah, likes these foods. We compare it to like real life personalities of these characters that like kind of reflected who they were. So like, you know, could actually pull data from the internet on these people and kind of incorporate that into their personality. But we wrote these prompts and then plugged that into our conversations. From a process perspective, it's going to take a little tweaking to get that right. So, you know, we had our...

some creative writers write these prompts for the characters. And then using this testing tool, we basically would have a bunch of conversations with the characters and like see how it was starting to feel. And one of the things that we noticed as we're testing, let's come back to our character, Lily, is that she was so sarcastic at first to the point where she was almost mean. And we were like, this is so demotivating. Like if you're a beginner learner and you're just trying to have a conversation and she's basically bullying you, that's not going to feel good. Right.

So we had to tone that down a bit and get like the right level of like slightly sarcastic, but still encouraging and, you know, friendly enough. So I think that's a use case that like you really just have to get in there and test it. And, you know, we would have we'll have our whole team come together and

sit in a room for 30 minutes. We'll all test out different conversations. We'll leave feedback and then we'll, you know, make iterations and updates to the prompts and kind of go through that like cyclical process until we get somewhere. And I think the reality too, that we've all had to become comfortable with is it's not perfect. It's never going to be perfect. There's always going to be instances where it doesn't work in the way that we want it to. So our goal is

We've kind of formulated these rubrics in our head or like there are certain things that we have to achieve with each different iteration. We're like, this is really bad and this can't happen. This is less bad and let's try to avoid it as much as possible. But we're OK if it happens from time to time and just kind of like make our own.

assessment of like what's really, really crucial knowing that, you know, we cannot control everything with it. We just have to be comfortable with that. Yeah. So it's really a matter of just like pressure testing it constantly over and over again, going in and asking questions, using it, seeing what comes up and just like putting it to the test.

Exactly. And, you know, we test it internally and then we actually test it with our users or our learners, as we like to call them in Duolingo. I'm sure you'll hear me use learners so often. And sometimes I say that and I forget to say like that's that's how we describe users. Well, now we know. Yeah.

And I know. So, you know, we'll test it internally. We'll get it to a point where we're like, okay, this feels good to us. And, you know, let's keep going along with the character personality example. We're like, okay, we feel good. And then we like run it as an A-B test with users. So we'll have like a control group that's using the current one. We'll have this one. We'll obviously outline like this is...

actually what we're hoping this feature improvement does for us. Just like you'd approach any AB test for any feature improvement. Like we want this to improve engagement or we don't care what this does. Like we don't care if this actually improves any metric, but we think it's a better experience and we just don't want to like hurt these other metrics.

So then we'll test it with users, see how that goes. And then, you know, either kind of come back and make iterations and try it again, or, you know, launch it and build off of that latest version. Mm-hmm.

When you're in the phase of not yet giving it to users and you're really putting yourself in the user's shoes, and I'm sure you're doing that throughout the entire design process. As a product designer, what tools or techniques do you use to really put yourself into the user's shoes to build that experience that is going to be ready to even test with them? That's a really great question. And

The lucky thing for us at Duolingo is we have learning scientists embedded on every single product team. So everybody on the team is learning some language and has varying levels of experience, which is a good thing. Like I am personally am learning French. I'm a pretty beginner French learner. Some other members of the team are very advanced French learners. And, you know, we have a multitude of different languages.

So we have some experience just with our own journey in Duolingo and our own stage in our language learning process. But what we really rely on to make these calls is these learning scientists. And these are people that know, you know, have PhDs in how to teach a language and like have an inherent understanding of the needs of

of somebody who's like a very beginner learner to somebody who's a very advanced learner. And they're really the ones that like kind of help us make the call, especially when it comes to these meatier learning questions. It's interesting because honestly, everything is impacting that fundamental piece.

Even character personalities. If we go too crazy with character personalities, they start to use vocab that's too advanced for beginner learners. So we need to make sure we have our learning scientists gut checking that. So that's really the most important thing, especially for a product like ours.

And it actually really started to shape roles of people that we embed on each team. So we have somebody on our team that's a learning scientist and prompt engineer. And they're really the core person that's writing most of the prompts for the products. And, you know, they'll work with engineers and front and back and engineers for how that's actually incorporated into.

into the actual code, but they're a subject matter expert and they also have this prompt writing experience. And that's really the magical formula of how we create prompts that are relevant and, you know, addressing all of our learner needs and actually solving for the goal that we're trying to solve. So.

I think that can be applied to really any company. It's like, I think people writing the prompts need to be the people that are those like subject matter experts on the thing that you do. And whether or not they're the actual prompt engineers or not, they definitely need to be part of that process. So they're kind of like user advocates in a way.

Like they're the person inside who understands the end user the most and they're actually helping to build the product through that experience. So finding people who can kind of have both of that is the sweet spot is what I'm hearing.

Yep, totally. And then obviously we, you know, we'll run user research and run qualitative studies and like actually run it with real learners that give us like a better sample size. But yeah, we always have our learning scientists as kind of the voice of the learner in a lot of these decisions and in every step of the product process.

It is no secret that companies today have more customer data than ever. But how are you bringing together all of your customer data at scale? Salesforce's Data Cloud can help by giving your teams a single source of truth for every customer. Ready to get started? Work with leading Salesforce experts from professional services. Visit salesforce.com slash products slash data to learn more.

How did you do this in other roles? I'm just, I'm curious and, you know, take AI out of the picture because obviously that's come up so quickly, but in, you know, the other work that you've done, how did you approach having that

internally who could really help to architect the product from the user's perspective? Or what strategies do you use to have that person or that mindset, let's say, if that person didn't exist? Well, I mean, I think even beyond generative AI, we still have learning scientists embedded on all teams. And whether they're writing prompts or whether they're just advising the direction and the strategy of the product, they're working hand in hand with product designers on all

pretty much every single initiative at the company. For example, this is related to our specific conversation practice feature, but it's something that could

could exist on anything outside of generative AI. We're like, okay, freeform conversation is very hard. And how do we give beginner learners the tools they need to be able to respond in a conversation? And how I as a product designer work with the learning scientists is we discuss like different methods. Like what is the right amount of support?

that we can give to them so that they don't feel like totally lost or confused, but we're also not giving away the answer. So it's still a learning moment and they can kind of help us like fundamentally form those strategies of like, okay, we could give them this resource or this resource or this resource. And then I, as a product designer can think about how that actually comes to life in the product itself and how that, like how we can make sure that we can represent that in like a clear, simple, easy, easy to understand way.

And then, you know, we'll then work together from that point to like align on an actual version of that, test it out, see if it's like accomplishing what it should accomplish and then kind of go forward from there. So they're embedded in every team regardless of generative AI. And I've seen that model at other companies as well. It's like you really, it depends on the thing that you're building. You know, I guess if you're working on like a social or a chat app, you can kind of put yourself

in those shoes pretty easily since we're all using those apps. But specifically for a learning product or something else that requires that like specialized knowledge, it's really important that that subject matter expert or voice of the user is really, you know, embedded in early product strategy and throughout. Definitely. And I can imagine, you know,

really going and either being that user and being able to experience it yourself so that you can embody that mindset or how I've also seen it done is just spending a ton of time with users and sitting next to them as they're using it and understanding what's going on. So you can really pull that inside. Yeah, I think it's so important. And we do a lot of user research and

I'm a team lead alongside my PM partner and engineering partner. And we all make it very clear to our entire team, like we all need to attend user research sessions. Like this is not just something for product people and designers and all engineers should be there because, you know, we're all working together collectively to build the right experience for our users and to

Ideas don't just come from leads or ideas don't just come from product and design. Like a lot, some of our best iterations and ideas come from our engineers and they're the ones that understand the tech and the limitations and possibilities there so deeply. They can sometimes suggest some of the best product iterations and I think that's why it

It's so important for every member of the team to spend as much time as possible just understanding what the experience is like for our learners. I've definitely seen that in some of the companies that I've worked in before where the engineers like to do what they do best, right? Sit and code and be in their bubble and...

get really deep into something. And I've, cause I typically work on the customer success or the customer experience side of things. And I'm usually the one trying to bridge the gap for the two and like pull an engineer, like get them into a conversation with a user if I can or, or whatnot. And I, I've always found that to be a bit of a challenge, but I'm glad that you underscore the importance of everybody on the team getting to have that real user experience and, and

connect with and empathize with the user. Everyone needs to do it. So one of the things that really strikes me about Duolingo is how sticky it is. It's a super fun experience. And I'm curious, like, how do you think about creating something that is like innately addictive so that people want to be using it all the time? What types of

thoughts or strategies do you put in place there? Yeah. So I think like one of the fundamental things that we kind of all use as a principle and anything that we approach at Duolingo is we don't look at other language learning apps and products as our competitors. We look at any other way people spend their time when they're like on their mobile device.

So we're not really competing with another language product. We're competing with TikTok and we're competing with Instagram and Snapchat and Be Real, although I guess we're past the Be Real days at this point. It was a fast one. But we're competing with all these other very engaging social platforms and

It's really just like what percent of a single person's time is spent on Duolingo. And to actually get them to use our product, we have to create something that's engaging and entertaining and fun to use. And if it feels like a textbook, nobody's going to spend time on it. They're going to spend time scrolling through TikTok. So I think just like

Coming at any product problem with like that mindset is so important for us because we have to create fun, simple, bite-sized experiences that like people feel like they can do in, you know, their five-minute commute or the 20-minute subway ride that they have or

between when they get home from work and their kids get home from school. We know that we're trying to make language accessible to all. And to do that, we need to make sure that people can actually get their practice in in a way that works for their own schedule. So bite-sized element and simplicity is really important there, but also just making sure that it is engaging and entertaining. And I've mentioned the character personalities that we have, but besides that, we have a lot of mechanisms for thinking about how we can continue to get people

to return to the app on a daily basis. And really all of that is for the sake of learning, because to learn a language, you have to kind of constantly be immersing yourself in it. Like if I do an hour of French today and then don't do it for a month and do an hour of French again, I'm not really going to retain anything that I learned a month ago and

The best way to learn a language is to continue to practice it, continue to learn the vocab, learn the new grammar, practice the things that you just learned, and do that on a consistent daily basis. And that's really the only way you can learn and grow. So to do that, we have a bunch of strategies for encouraging people and making it fun to come back to the app.

We try to create these like smaller milestones along the way because learning a language is hard. And from when you start day one, like I'm going to learn French today to like actually being a fluent French learner, that's a really, really long journey. And that's going to take a while. So we need to make sure people are motivated along the way. So one of the things that people really resonate with is obviously the streak. And that's just

The way to get people coming back every day to the, yeah, continuing that streak, keeping that up is something that people are so proud of. Like, honestly, even when, whenever people like will apply to jobs at the company, they'll reach out to me and be like, I'm on my 2000 day streak. And I'm like, that's better than me. It's

And like, but people are so proud of it, which is really, really cool because it gives them this milestone of like, I am taking a step every day to learn a language. I'm taking five minutes away that I maybe would have spent on TikTok. And I'm going to apply that to something that's like, you know, meaningful and useful to me in my life, which is really cool. We've kind of capitalized on that as well. One of the more recent things that has become very popular is our Duolingo widget.

And I don't know if you have it installed on your phone, but we basically have it. People call it the Duolingo Tamagotchi. So basically there's like different chaotic poses of duo that changes throughout the day as you get closer and closer to not completing your streak. So like at the beginning of the day, he's like really happy.

and he slowly transforms to like chaotic and then like very, very angry if it's getting to be about midnight. And a lot of people say, a lot of the users that we talk to will be like, I have to keep Duo happy. Like that's the thing getting me to come in and do my...

do my language learning every day. It's to make sure I don't get angry duo on my phone screen. So things like that and kind of capitalizing on the chaotic and fun side of the brand and using that as a way to like get people to come back to the app is also really important. Yeah. I mean, the gamification is real, right? And I was a huge Tamagotchi user. So that actually resonates with me a lot.

Where do you look to for inspiration with these types of features? Honestly...

all across the ecosystem. And again, like I said, we don't really look at other language learning products. I think obviously to some extent we will redo a little bit, but it's like, what are the things that get people coming back to TikTok and Instagram and other things? What are things that work and how do we apply that to language learning? How do we apply that? And I think the fun thing for us, both from a growth perspective and even a monetization perspective is all of

this is in service of getting people to learn a language or, you know, from the monetization lens, like our Duolingo subscriptions make it possible to have a really, really good free product. So by getting more people who are able to pay for the subscription, it allows all these people to learn languages for free, which is really, really important when you think about that at scale. And, you know, that's really the core of our mission. We look all across

social apps, other apps, like all across the ecosystem, way beyond ed tech and also mobile games too. I think that like, that's something that like, obviously we are known for at Duolingo is how do we apply these fun gaming mechanisms and use them to make learning fun. And one of the things that we looked at recently, I don't know if you saw, but over the holidays, I think Monopoly Go,

was number one in the app store for a period of time. So we're all like, what is it? How did it go? And where did that come from? And if you play it, you understand it's really, really engaging. There's like all these different tactics of

earning money and building properties and like celebration moments where like I win a bunch of cash and there's like a big, you know, cash spiral falling down the screen. So I think like looking to these games and like seeing how they use, you know, animation and delights and like all these other things to kind of get people addicted. But

get people engaging with it. And we can kind of take some of those principles and apply it to learning. Then that benefits both us and the people that are trying to learn a language. A hundred percent. That's so much fun. I mean, it sounds like you get to play a lot of games.

I wanted to come back to something that you were speaking about earlier, and especially being on the monetization team. When you were launching Duolingo Max, the need to bring in money as a part of this new product, obviously using AI adds an extra expense. And there's like a business need there. How do you approach one solving for the business need, but also creating an experience that has...

value that is applicable and value that people want to pay extra for? How do you think about those kind of bridging that gap? It's a great question. And kind of coming back to just technology and technology especially, that's very, very present in news and everything in the current time.

people can kind of hype up these technologies in a way and everybody is like, I need to do AI something. You know, two years ago, everybody was like, I need crypto to be a part of my business, even like when it didn't actually make sense. And I think for us,

We were excited about AI because we were excited about how we could use it to fill these learner gaps or needs that we'd been trying to fill for years, but just couldn't because building a conversation practice feature without generative AI requires so much human labor costs and such an intense content need that it's kind of near impossible to make it make sense as a feature. So we'd been wanting to do it for a while.

And for us, AI was like the key or the tool to unlock this thing that we'd already been hearing for years was something learners wanted and learners needed and learners were saying they would pay money for. So I think for us, it was more like we always start with the need. It's never like, okay, we have AI. How do we make Duolingo AI? Or we have crypto. How do we make Duolingo crypto? That never happens for a reason. Yeah.

There's never a dual coin. But AI was something that we were like, this could help us solve these problems that we inherently know our problems learners have and have said that they would pay money to solve.

That's really the basis of why we built the features that we did, which was this conversation practice feature. And then this other feature that we call explain my answer, which gives people more personalized explanations or like grammar tips behind the mistakes they make in their lessons. So I do think that that is that is so important. And that's the way that we approach this.

any new avenue within our company or business. It's always based on making sure that there's a want or a need or a market for it. And that was the case with generative AI for sure. I think with AI, there's so many use cases. And I think I hear that from a lot of people that this enabled us to do something that we didn't, we weren't able to do before. So it was an unlock that of a problem we were already trying to figure out.

I'm curious if you have any advice, though, for the folks who are maybe in the camp of we just need to use this technology. We have to be everyone saying like you have to be using AI or else you're going to be left behind. And I think I also see people scrambling to try to figure out like a new AI feature just to be able to say that they're using an AI feature, but it might not actually solve a user need. So I'm wondering if you have any advice for folks who might be in that camp of getting pressure to use AI, but not really sure where to apply it.

Yeah. It's a little bit of like a chicken and an egg thing, right? Where you're like, you should understand how it works and really understand the intricacies of it and how it's evolving because it's going to evolve very quickly over the next few years. But also you don't want to build something just to build something. Yeah.

if it's not actually solving anything for your business. So I do think that there's a lot of value in investing time and thinking about what it could do. And like I said a little bit ago, the way that we approached this was doing a ton of different hackathons where we just had people mess around with it, break it, try crazy things that had

no business value whatsoever. It was just like a really cool idea. Try things that were actually like really centered on learners and then kind of come together and share that knowledge. And even the things that were just like crazy fun things that like had nothing to do with our language product helped

us understand like the details and the intricacies of the technology and how we can apply it. So I think it's important to whether or not you're building a product around it or launching a product to users. I don't really, I mean, maybe there's some valuable data you could gain from doing that just for doing it. But I think the better approach is just like consistently staying on top of understanding what it can do, trying to

trying to think about all these different ways you could potentially use it for your business or just use it in general so you have an understanding of how it works and continuing to do that every few months. And I think that's really the best way to at least stay in the know of what it can do because maybe what it can actually unlock for your business is

doesn't exist today, but will exist in five months. So it's good to at least understand and maybe a new need will arise. So that's great.

Where do you go for inspiration or what resources, blogs, podcasts, thought leaders do you like to follow to really stay on top of trends when it comes to AI? I don't know if there's any like one particular place that I go to, but what we do have at our company is we have like a Slack channel that's just like generative AI news. And it's like a bunch of different people in here. And that's kind of where we all share more of like the business news of everything happening in the AI world to like

new models that emerge to like crazy things people have posted on Reddit or whatever that they do with it. And I think it's good to look at it from all angles. Like look at the really like wild, fun things people post on like Reddit or, you know, threads or Twitter. It's hard for me to call it X, but X, I guess. And, and also, you know, stay on top of like more of like the,

the actual news and like the new models that emerge. And I think it's good to like kind of look at it from all those different angles to stay on top of what's happening. Every time I see a new AI product come out, we'll always download it and we'll play around with it and just like kind of constantly get a sense of what are the different ways people are using this technology in different ways. Are they doing certain things that we didn't think of or are they solving a problem in a different way?

I think I always have in the back of my head all of the core problems we still want to solve with it. So whenever I'm downloading a new tool, I'm thinking about, have they figured it out? Normally, the answer is no. Because it's new. It's a new technology. I think we're all still figuring out how to use it. We're all still figuring out how to incorporate it into our products and also just our processes. So...

I honestly think like there's not really one good source. I think you can just like follow people that like you think are coming up with like really interesting creative ideas and just stay on top of it in the news. Myself and my product partner, Edwin Spock,

Spoke at two conferences this past year at Config, the Figma conference out in San Francisco and at Design Matters in Copenhagen. And going to these conferences was also really useful for us, especially the Design Matters conference. There was a lot of people speaking about generative AI. So it was like a really cool gathering. We got to talk

to a lot of other teams that were using similar technologies, both in like similar applications and education and other applications. And we could like really get into the weeds of like how people were approaching problems in different ways. So I think it's really cool to connect with other industry professionals working in the space and just kind of like get a sense of like how they're approaching different problems and that kind of all new things. Yeah.

That's great. There's so much information out there and I love having a Slack channel and just a way of sharing that. And then also connecting with other professionals who are thinking about things in the same way that you are and unlocking opportunities.

taking different approaches to potentially the same problems and sharing that knowledge. That's great. Just one thing I'll add to that. Like, I think the best way to really understand it, you kind of have to just play around with it yourself. Like whether it's for your company or not, like come up with like a personal project and be like, can I use AI to accomplish this? Think about how you would write a prompt to do that. See like what the limitations are. I think it's something that you really need to like

feel and test and see for yourself to like understand what the possibilities are. It's actually a personal challenge that I've made to myself recently of like,

thinking about how I can use AI to solve like every problem I have. Like whenever I'm like stumped about something, whether it's like a personal thing or a professional thing or a client thing, it's like, how can I use AI to do this? And it's been fun to think about it. And most of the time I'm like completely stumped on it. But then you do a little Google and you're like, oh, someone did this. And then you can kind of like inspire yourself on different ways of using it because the possibilities are tremendous.

Truly endless, honestly. So Megan, I have two last questions for you. These are questions that I ask every guest that comes on the show. The first is, I'd love to hear about a recent experience that you've had with a brand and it can be any brand, whether it's an app or a hospitality experience or whatever it may be. I'd love to hear about an experience that left you impressed.

What brand was it and what was the experience? Yeah. So one of my favorite brands or subscriptions is Nully or Newly. I actually don't know how it's pronounced, but it's one of the two. And it's basically it's a clothing rental subscription.

It's like one brand in the same portfolio of consumer brands as Free People and Anthropologie and Urban Outfitters. And I've been using it for the last four years or so, kind of in an effort to reduce reliance on fast fashion and waste less money buying clothes that I will wear once and then never again. And they just sit in my closet for months on end.

And basically what this allows you to do is every single month you pick six items of clothing, you wear it, you can buy it for a reduced price if you want, and then you return it and pick the next set of clothes at the end of the month. And I think the best thing about subscribing to it besides really good price, great clothes, good renter reviews that make it easy to actually find the right fit and size for things is their very low-key approach to the subscription. And I think

there's a lot of consumer mistrust in monthly subscription services because people get this fear that they'll be trapped in it. And I think, you know, a lot of fitness brands or even gyms like Equinox make it like near impossible to quit or very expensive to quit. And then I think that

Also shows up in apps as well. I used ClassPass for a really long time and ClassPass is great, but to cancel ClassPass is like a 20 step user flow that like I almost get like,

so annoyed with that I don't cancel, which is probably what they're trying to do. So I guess it's effective. But I think people have this fear of signing up for these programs because of that. They don't want to get stuck in it. They don't want to waste money or get stuck paying money that they don't realize they're spending. But Nully makes it really, really easy to pause a subscription for

for any month where you don't need a rental or end the subscription if necessary and the click of a button on the website. And they have great customer service specialists that will answer you in less than 24 hours if any issues arise. And if there's any problem with any of the items in your box that comes...

they'll automatically award you extra bonus items the next few months. So it's really just that amazing customer service, this low commitment in a way that makes me more committed approach to subscription services that makes it one of my favorite brands. That's amazing. And I really appreciate you calling out the making it easy or allowing you to actually control your subscription because I am...

a part of a number of subscriptions and it can be just really frustrating. And it makes me dislike the brand when I can't pause it because I'm not going to be able to use it. So that's great. Last question for you.

What is one piece of advice that you think every customer experience leader should know? I'm going to bring it full circle a little bit here. I think the most important thing is that you really, really need to start with the user and make sure that whatever you're building, whatever feature or new product or whatever it is, whatever you're building is actually solving a problem that exists in their life right now. And when I say exists in their life right now, I mean like...

Now, not in 10 years. I think people have been trying to make VR work for a really long time and they should. And we should understand the technology and see what it can do. But it's not really solving a need that we have right now. And I think that like that's,

That is kind of what we all see happen. You know, I mentioned with crypto, you know, now kind of AI is just like the, you know, popular thing to do or the popular thing to use as a company. And you should understand the intricacies of it and you should understand how it works and explore what that looks like.

But as you're building a product, you need to start with the user. You need to make sure that you're using whatever technology you do in a way that's actually solving a problem for them. And you're doing it in a way that's different and unique from other people. I think when you think about generative AI, people can access it through chat GPT. They can access it in so many different ways. So when something is universally available like that,

You have to really think about what is the unique angle or edge that you're adding to it that are going to get people to come to your product.

So, you know, for Duolingo, what that looks like is like bringing in our character personalities. We know how to create a strong narrative. How do we make that clear? How do we tie it to the specific learning journey you're on and make sure that like that particular conversation is suited to your level at the stage that you're at in your, you know, Spanish or French or whatever language learning journey. So I think those two things are really key and honestly can be applied to a lot of different trends, both in tech and

and in business. Mm-hmm. Got to think about the user. Put them front and center. Exactly. Amazing. Well, Megan, thank you so much for coming on the show and sharing all of your knowledge about generative AI, building for the user, product design in general. It's been really insightful and we greatly appreciate it. So thank you so much for coming on the show and we hope you have a beautiful day. Thank you. And likewise, it was fun. Thank you.

We are clearly in an AI revolution. And thankfully, Salesforce is helping customers navigate each new wave of AI. But the question is, how do you actually use AI effectively? And my answer to you is, it starts with your data.

With Salesforce's Data Cloud, you can connect, federate, and harmonize all of your data from any product and system in Salesforce. To learn more, visit salesforce.com slash products slash data.