Ep 63: Khan Academy Founder/CEO on Salman Khan on Classrooms in 20 years, Rolling out to 1.4M Users & Sal’s Hopes for AI Education
50:53 Share

Salman Khan is the founder and CEO of Khan Academy, one of the most influential education platforms in the world. They serve over 150 million learners across 190 countries. And they've also been doing some really interesting stuff in AI with KhanMega, their AI-powered tutoring system. It's already been deployed to over 1.4 million students and teachers. And Salman and I had a really interesting conversation around how AI is transforming education for students, teachers, and school systems. We talked about where these models work, where they don't, and what it takes to build AI that actually makes a difference in the classrooms.

Some of my favorite parts included why proactive AI is the next evolution of tutoring, some surprising ways students and teachers are using AI, what it takes to reduce hallucinations and math errors to near zero, and how AI is reshaping engagement and assessment. This was an awesome conversation for someone who's really doing one of the widest deployments of AI today in the real world for a really good mission. Before we get to the episode, I just want to say a huge thank you to all those that rated the show on Spotify and Apple last week.

If you're enjoying this on either of those platforms and haven't yet left a rating, please consider doing it. Now here's Salman Khan. Well, thanks so much for coming on the podcast. Really appreciate it. Thanks for having me. Yeah, this is going to be a fun one. And I figured I would start with the most like overly broad question I could, which I'm sure our listeners are curious about. Given all the like rapid advances in these models and kind of what you've been building on top of them, what's your current thinking on what a classroom looks like, you know, 20 years from now?

Yeah, it's a good question. You know, I think if we had this conversation five years ago, I would have thought that's a boring question. And now I think that's like too far out because things are changing so fast. Always hard to find the right time period in AI worlds. I'll tell you what I'd like to think a classroom looks like in 20 years or even in five years. But let's go with your 20. Yeah.

I'd like to think that more classrooms are actually going to look like what great classrooms already look like. A great classroom today is one where students are not just passively listening to a lecture. They're engaged. They're interacting with each other. They're actually doing things, whether it's problem solving, working in groups, giving presentations. In a great classroom today, a teacher is walking around quietly

working with the students, getting them to do interesting things. Now, the reason why I think AI is potentially going to play a role there is right now those great teachers...

They have to spend a lot of time making great lesson plans, being very creative about it. These great teachers are able to do that even though they still have hours of grading and lesson planning and progress report writing. These great teachers are the ones that can really observe a classroom and almost intuitively understand what students might need. And those are superpowers. And I think there's a world where artificial intelligence can give more teachers, all teachers,

uh, some time back on their planning side of things, giving them better insights on how do they, uh, well, some ideas and insights on how they can better manage their classroom, better insights on where the students are at any point in time, better interactivity with their students. Uh, so, you know, when these students are all doing breakouts and I can imagine in 20 years for sure, it's not going to necessarily be on your laptop. Um,

I think one of the powerful things about where generative AI is going is there's no reason why it can't be ambient. And it's just observing the classroom and seeing what's going on. And so I think you'll have, right now people try to think very binary. Okay, technology is you're just staring at a screen. Non-technology is you're running around in the real world. I don't think that necessarily has to be a trade-off in 20 years. I think in 20 years, you're also going to have a whole other aspect, which sounds science fiction,

But I think that's a point at which virtual reality, augmented reality will become very mainstream. And that with generative AI and super intelligence at that, who knows, you're going to be able to immersively go into simulations, be part of virtual worlds, go back to ancient Rome and try to

stop or maybe help hasten the assassination of Julius Caesar. Well, whatever it might be, but you know, that it's literally like a magic school bus ride. Yeah. No, I mean a lot of really compelling threads to pull on there. I mean, I think especially as you talk about kind of giving teachers superpowers and kind of augmenting what the best ones already are able to do, probably a good segue, just to talk a little bit about, you know, what you've been doing with Conmigo and the efforts, you know, at Khan Academy, Khan Academy building AI products, you know,

Could you just give a little bit of context for our listeners on, you know, what you guys have built to date, how you've been thinking about this? Yeah, it's interesting because I think in this time of rapid change, it's always important to have like, what are we in this to begin with for? Like, what's our true north? And Khan Academy's true north, I've been articulating it this way.

much more clearly, actually, ever since gender of AI became a thing. But if you go back to the early days of Khan Academy and it was just a hobby, I was tutoring family members. Then I started writing software. I started making videos. Obviously, Khan Academy became much more than me. But everything we've done over the last almost 20 years is...

trying to approximate or replicate some of that personalization that a good tutor would do. You know, not only make hopefully high quality materials available, but make it so that it can be, you know,

to the student. They can get practice. Teachers are an important part of it. How does a teacher with 30 kids in their classroom personalize it more? So we've always been doing that. And even in the early days of Khan Academy, I used to cite Diamond Age, the Neil Stevenson book on Young Ladies Illustrated Primer. I was like, we're going to build that one day. This is what we are building. It's just we're going to be doing it incrementally as the technology gets better. So when we saw what was possible with the latest generation of technology

in particular GPT-4, which OpenAI gave us access to months before even ChatGPT existed. It had issues, but we said, okay, this is going to be able to approximate tutoring. It's going to be able to approximate teaching assistants and other things that we probably hadn't conceptualized before. So that's how we launched Conmigo, which is our AI assistant

as a tutor and a teaching assistant, putting guardrails so that teachers can see what students are up to. It won't cheat it, you know, safety, privacy, et cetera. It's much more Socratic. So we're really trying to lean into the, let's make this really good pedagogy. Let's make this safe. But let's also, let's also make it useful. I think what we've, we've had a lot of successes when we launched, I thought maybe by 2025, we'd have maybe a hundred thousand folks using it as a pilot and

It's now pushing about 1.3, 1.4 million teachers and students. And these are, you know, our mission is free world-class education for anyone, anywhere. But we've had to charge these districts for it because of the compute cost and the support and the training that we've been giving. We charge them about $15 a year. So this is, you know, to have 1.3, 1.4 million districts closed.

paying within a year and a half and the interest continues to be very, very strong. So one thing that we are doing now is realizing that the next phase is making the AI much more proactive. Even if I walked into any math classroom or any classroom and I said, hey, I'm a great tutor. I'm here in the back of the room. If you have any questions, come ask me.

There's probably going to be about 10 or 15 percent of kids who do it. And we're seeing that with the AI. So this next version of Khan Academy you're going to see, we're going to start piloting it in back to school. We're calling it Khan Academy Classroom. But it's from the student point of view, a much more proactive AI that every time you go to Khan Academy, it's like, oh, welcome back, Jacob.

Hey, it's been a little while. Hey, here's what your teacher wants you to do. Hey, how can I help you here? And from the teacher point of view, same thing, much more like a concierge front and center as opposed to being just something to be asked.

Yeah, it's so interesting because I feel like, you know, across AI products, there's like this blank screen problem where you get to a place where you can prompt anything, but it's like, how do you figure out actually what the right thing to prompt is? How have you kind of thought about, you know, just like starting to teach students and teachers as you kind of rolled out? I think Newark was one of the places where you did a huge rollout. Like, what have you learned, I guess, about AI?

you know, teaching folks how to start using these products and, you know, kind of getting started from a cold start. Yeah, so it's a very real thing. And it is where, you know, we've done, we think it was like little dynamic action bubbles where we're suggesting things that people might want to try next. Obviously, there's some training, things like that. But I think the core is making the AI much more proactive and

about things. The other thing I'll say, because it's important right now, there's probably 500 people who are claiming to make some version of an AI tutor in some way, shape or form. I don't think the AI is quite ready yet to

to be by itself and drive learning for most people. I think if you're a curious person, you could go to chat GPT. And if you spend half an hour every night with chat GPT, and just you're really good at prompting it and asking questions, you can learn a ton. But for most people, that isn't most students. And so, and the AIs aren't great yet at creating a

high quality questions that are not going to have errors that can give you, they're getting better every day, but they're still not where they need to be. So, you know, a lot of what we see in places like Newark is the efficacy. And we really are seeing some amazing efficacy numbers coming out of there. It really happens from doing the traditional practice on Khan Academy. And the AI there is a support. It's something to help drive engagement. And

A lot of people in education and ed tech always think about, oh, can I come up with a more efficacious intervention? And that's obviously matters. But it turns out if you engage with anything reasonably, that seems reasonably healthy, it's probably efficacious. The hard part is the engagement. And this is where we are, you know, we're trying to look at every dimension of that pipeline, all the way from how do you get a teacher activated faster? And where's the AI there? How do you get the district's administrators engaged?

in a polite way, holding the classrooms accountable that like, okay, you're really engaged in this tool we're using. The students are really using the AI. And then of course, how does the district leaders and the teachers hold the students accountable? These types of human systems and how the AI can help the humans hold other humans accountable

is actually how you get engagement. Yeah, it's such an interesting point that engagement is actually like really the thing you need to solve for. You know, and on the student side, I'm curious, like, obviously you've kind of released this now to a ton of people using it. Any like surprises in the way that it's been used? I thought, I mean, I've heard you speak on another podcast about, you know, kind of, obviously there's some of the shortcomings of these models is they aren't always right. And you found that like students were explaining their reasoning to the models and then the models were like iterating on that reasoning. And it actually was like a really interesting way for,

like students learn and the models to actually correct themselves with the mistakes they're making. Like anything else you've noticed in the way people use this that might have surprised you? There's definitely cases. These aren't mainstream, but I said, if you're a really motivated person and you engage properly with these models, there's some amazing magical things that can happen. I gave an example. This was a TED talk two years ago. I gave an example, but it's still a fun story of this young woman at Khan World School, which is an online charter school.

school we have and she was in india at the time and she um was reading about the great gatsby and we have an activity where you can talk to ai simulations of literary characters and she had like a lengthy conversation with ai j gatsby uh about every and you know our our um simulations they don't just try to answer questions they actually try to drive the conversation so what have you thought are there things like in your life that you want to and um

I remember her telling me about that interaction and thinking, okay, this is beautiful. This is the kind of thing that we want to see more of. I have, I've talked to my son a lot, my oldest, who's just turned 16. And, you know, he's, he's,

He's doing some pretty advanced math now. He's probably at my level or maybe even a little ahead of me at this point. And he's actually using it all the time to really explore ideas. And I'm like, is it 100% right all the time? He's like, no, but neither are you. And then he's...

And, you know, one of the things that we are putting a lot of resources, and I don't think a lot of other people are, are really trying to not just improve the AI's accuracy, but measure how good it is. And as far as we can tell right now, Conmigo...

when it's anchored on Khan Academy content, you're at about a 2% error rate. And the 2% error rate's about split evenly between just a straight 1% of the time it's a math error and 1% of the time it's an evaluation error. So maybe the answer is one third, you put 0.33 and it said, great job. It should have said, well, close. Are you sure that, you know, do the threes keep going or can you represent that as a fraction? And

we, we obviously would love to get it to a 0% error, but I actually think my son is right. It's actually already when I'm tutoring my own kids now, it's probably one out of every 10 or 15 times that I'm, I'm like, wait, that's not what I got. Wait, let's do that again. And I'm like, oh yeah, you're right. You're right. You're right. You know? And so, um,

I actually think the error rate is actually already better than a lot of human tutors. What about on the teacher side? Like, you know, how have you seen, you know, the best teachers leverage these tools? What's kind of the overall reaction been to some of these things, you know, entering the classroom? I mean, obviously, I guess, in some senses, through ChatGPT's broad release, they've entered the classrooms regardless of whether it's through Conmigo or not. And so what have you kind of noticed on the ground there? Yeah, the ideal, and this is maybe not quite as sexy. I mean, there's some sexy use cases from teachers too.

But the ideal is they just use it regularly. They create a habit around it where they're working with the AI, helping tweak their lesson plans, making a little bit more entertaining, right size for the classroom, then delivering it in the classroom. We have a partnership with Bluekit for those, you know, for students.

Bluekit's, I believe, as popular or more popular than Kahoot. It's like an in-class gaming based on question sets. So our partnership with Bluekit is Conmigo will generate the Bluekit questions for the teacher. So what used to take a teacher maybe half an hour, hour to write, 50%.

15, 20 questions or more, they can now do with like in about like two minutes. And we're seeing a lot of great things there. So we're seeing the planning, the delivery inside of the classroom, and then the getting insights from it. And then that's for another wave of planning. And there's a lot of things that we're building for this coming year that will make that much more streamlined and integrated.

but a teacher who has that kind of habit forming, it's always been the case, who's using Khan Academy to make assignments, hold the students accountable, look at the data, and then keep doing that. They seem to get very, very good results. Once again, it's all about engagement. In terms of

Sexier use cases. We've definitely seen teachers talking about AI simulations. They're definitely opening up a class and saying, all right, everyone, we're going to talk to AI simulation of Harriet Tubman or George Washington. Ask your hardest questions, which is really engaging. We have another tool called Writing Coach, which is our answer to the fears around cheating, etc., where teachers

The teacher creates the assignment with the AI, assigns through the AI. Students do it with the AI, but the AI really acts as an ethical writing coach. And then when the student submits, the teacher doesn't just get the final output. They get the process and they can talk to the AI about what went on. And if

If I copy and paste something from chat GPT, the AI says, I don't know where this came from. So it actually undermines not just AI cheating. We believe all forms of cheating, but teachers are starting to use that reasonably regularly for writing assignments. So that one's I'm pretty excited about. I love that approach for just like, rather than, you know, say, Hey, we're going to just ban these tools completely. It's a unrealistic and be not how people are actually going to interact in the world. It's like,

let's find a way to teach people to use them in a way that still develops their own thinking. I think it's really clever. Do you imagine that being true across other subjects as well? Or like, obviously, in writing, it makes sense, you know, given that in the workforce, people are going to use these tools going forward. Do you think like most of schooling should involve leveraging them to some extent, or are there still places where they should be, you know, you should be able to do math and writing completely without them? I think it's got to be both. I definitely think

if you're going to be managing an AI to do some of your work, you need to be able to write well yourself. So I think especially if we're pre-high school or even early phases of high school, yes, you should do some more writing inside of the classroom with the teacher there. It could be a little short form writing, et cetera, or maybe you do it over multiple class periods. That I think is healthy. At the same time, especially once you get into high school and college, yes, you should have more

opportunities to use the tools to get more productive. You know, I have to give a few commencement addresses later in a couple of months. And

I was like, oh, I better work on this. And I was in an airport and all I did is I recorded my thoughts, my like life advice, you know, like, and then I had it transcribed by an AI and then I had it turned into a first draft. And that first draft, I mean, it speaks to both things.

That saved me a ton of time that I would have otherwise done. But if I didn't know how to write, and if I just took that first draft that just came out of my random musings, it's a horrible speech. But it had the essence of some of the ideas. I was just looking at it last night, and I was like, wow, this one way of phrasing it was actually quite beautiful. And I'm going to tweak it a little bit because it's not exactly how it sounds like me. But going from zero to one,

And really, it went from it went from point three to like point seven. Right. Because I gave it. These are my thoughts. But these things are massive accelerants and people should learn how to how to do this. I yeah, just speaking out. I can imagine when you and I were in school, the term paper, they say you have two weeks. What if a teacher says, you know, your opinion on this matter. But by the end of the class period, I want to I want to see some output.

What about on the district level? I feel like in other categories, obviously, we're seeing tremendous pressure from boards, from CEOs, like, hey, let's adopt AI as fast as we can. Obviously, the schooling system is its own kind of unique universe. Does it feel like there's that same kind of pressure for uptake or momentum? Or what do you kind of observe that like the policy and kind of district level? I think generally speaking, this is a very, very, very, very, very, very, very, very, very,

This is one of the cases where schools might be one of the first places where you see mainstream adoption of AI for productivity and learning and just, you know, doing the day-to-day work, which is really trying to help kids learn.

As you already mentioned, teachers have already leaned into this because so much of what teachers do can be streamlined with AI, especially on the planning side of things and the grading side of things. And as the models get better and can support students better and get more proactive, I think everyone sees it. And

Yes, there is compute costs, et cetera, but it's dramatically cheaper than anything else that's come before. After the pandemic, there was $86 billion that was spent on ESSER, these funds to help kids remediate. And that's like $2,000-something per American student. And

A lot of districts plowed it into fairly expensive paid tutoring, like live tutoring. And there are some exceptions, but for the most part, there's not much to be shown for it. You know, so instead of something that was costing $25, $50 an hour, you're now looking at something that costs $10, $15 a year. And you get much more dosage if you want to. So, yes, I think...

I'm actually seeing more in school districts than I'm seeing as a leader of Khan Academy. And, you know, I've been pushing the Khan Academy team. I was like, you know, when are we going to be able to get, you know, automate some of our bookkeeping? Or when can we do this on this? Or when can we do, you know, I'm constantly pushing the engineers on how much more productive are you getting with the coding? I heard that Company X is, you know,

100%. Why can't we be 100%? But yeah, I think schools are... There's school districts we've talked about. They are...

They're saying it's saving their teachers at least five hours a week, if not more. They're using it as a recruiting tool, retention tool. Yeah, it's really powerful. I mean, I guess from a society perspective, I'm glad that schools have been one of the fast adopters, if any place was going to be. You know, I guess I'm curious in the process of building Conmigo, I mean, you talked about obviously you had to add guardrails in. It sounds like you've tied the models to your own content that you've already had on Con Academy. Anyway,

Anything else you needed to build on top of the open AI models to make this work in the classroom setting? Oh, yeah. Well, I mean, I could go down the list. It is surprising. On one level, a lot of these AI applications, and unfortunately, I think many of them are, are just kind of thin prompting layers on top of a model. But yes, you want to do the safety. You want to do the moderation. Moderation is something that, frankly, we probably were overly conservative about.

to begin with. Makes sense given your audience. Given our audience, but then, you know, there were a lot of false positives. So I think we now have the handle on that one. The math accuracy in particular, this is where there's a lot of work. I mean, just to get the error rate low. And once again, the hardest errors aren't can this thing

figure out what five to the eighth power is. The hardest errors are evaluation errors with the students, especially when there are certain students who are just hammering it, trying to get an answer and they keep switching context, et cetera, et cetera. And so how do you handle all of that? A lot of work, just making the user interface something that feels more natural. Obviously, as I mentioned earlier, we're doing this whole re-engineering of the front end of Khan Academy to imagine a more proactive AI-first approach

approach. And I think when people see that, they're going to see like, okay, this is, this is not just a chat bot. This is the AI integrating it in every aspect of what the writing coach we've been doing, you know, that's where once again, it's not just a chat bot. It's, it's,

There's a brainstorming on your thesis statement. There's an outlining tool that the AI can see and has context. And then when you're drafting, it really is kind of like you're on a Google Doc and it's highlighting parts of it based on the dimensions. So yes, there's a lot to be done that's well above and beyond just a thin prompting layer. There's a lot of, you know, I think the world now doesn't want 20 different

apps on an AI tool. They want an app that's smart enough to figure out and behind the scenes do a little bit of prompt chaining and swap prompts out. So there's a lot of work there and making sure that that's robust memory. I could keep going. Yeah. What are the capabilities that are meaningful to you that are still on the come? Like things that you, you know, breakthroughs, I guess, you know, either in the core LLMs or some of these multimodal capabilities that would be, you know, kind of game changing in what you're able to do in the product.

I mean, memory is a big one. I know right now you could go to chat GPT and ask about, you know, apparently it remembers everything you've done. But yeah, for people like us to have access to that so that we can give the models much more context. It has all the memory, but at the same time, maybe there's ways to reset some aspects of that memory. Know what it's remembering or inferring about you. But I think memory is a big one.

The advanced voice, which is out there, you can use it already in chat GPT and Gemini and all of that. I think integrating that with our platform is going to be pretty cool. We are building for a world where over time,

The human, the Khan Academy written content might become less and less relevant. You know, that breaks my heart as someone who's made 7,000 videos, but it's just a reality. And so, you know, they're all being used to train the models anyway, right? I'm sure they're still, they're still in there. They've had some utility. Yeah. But the ability for the models to create higher quality questions, I think is big.

You know, I think there's some very cool capabilities of models, like the ability to make videos and certain types of images and all of that. Those are fun. I don't think the world has come up with really good pedagogical use cases for them just yet. I mean, there could be some fun project-based learning, things like that. But yeah, the pedagogical side hasn't been... Well, one on image, we do hope, this is probably about a year and a half out, that

For students to be able to, especially if they're on a tablet device, be able to show their work and the AI being able to see their work. I think you had your son demo that, right? Yeah, that open AI demo from last year, which, you know, in fairness to the demo gods, that was the fifth take. It was making a ton of errors, but it eventually did work. But that idea of the AI very naturally being able to talk while seeing your work was

and giving you feedback, yeah, I think that feels almost, if it's robust...

It feels indistinguishable from when I was tutoring my cousins or now I'm tutoring my kids. So I'm hoping if in the next year and a half, two years, we can make that a mainstream thing. That that'd be a big deal. Yeah. One thing I'm struck by when you alluded, obviously to like, there's kind of like fun use cases one could explore. And, you know, to your earlier point that like so much of this is around engagement and like, you know, a lot of the techniques are, there's a bunch of different techniques and a lot of them are effective and it's really about just getting people to engage with any of them. Uh,

I'm curious to see whether we see some, you know, interesting kind of just like fun new ways that we haven't conceptualized that actually drive, you know, drive people to engage. And then obviously you can layer some of this other stuff on top of it. Yeah. And, you know, I'm always, we've been gamifying a bit over the years. And obviously there's people in ed tech who've done very good jobs. People like Duolingo, you know, Duolingo has had the benefit of, they kind of make up the standards, right? And they can, they can, they can, they can, they can make those dopamine hits as long as, as, as much as you want. It would,

When we started, we were on that same journey, but then as soon as we became more mainstream in schools, we're like, okay, we have to align to the standards. And some of these standards are not the most pleasant things to do.

For some students. And so there's always going to be, you know, if you really want to do and learning isn't always easy. So there is always that like productive struggle you have to do. Now, I do think if we can be really creative at about like quest based learning and when people go on scavenger hunts or escape rooms, they are willing to solve pretty hard problems.

cognitively challenging problems because it's part of a broader game. So I think there might be something there, especially if the AIs get good enough. My 10-year-old, his last birthday when he turned 10, I made a scavenger hunt for him and I used the reasoning model to come up with most of it. It didn't do it out of the box. I had to tweak it a lot. But in a few years, you can imagine it being able to create games out of these types of things.

Totally. Do you have a go-to thing you use to test the new models? I remember, I think, famously when you got showed GPT-4, you were using, I think, the AP bio exam to figure out whether these models were any good. Like, what do you use now as like a new model comes out from, you know, Google or OpenAI or Anthropic to figure out, you know, how good it is?

You know, some of the things that we were worried about a year or two ago, I don't worry about anymore. My gut sense is, and, you know, the data seems to back this up, that all of the Gemini, the Clods, the Groks, and the, you know, GPT 4.5, that they're all in the same

plus or minus, some would say they're, you know, one's better than another, but even that's kind of unprompted. I think if you prompt, you could get almost identical behavior out of all of them. So now it really is,

There's obviously a cost performance, but the differences between them are subtle enough that I have to lean more on our own evaluation framework to saying, okay, out of our tough test cases, how many of this is succeeding or failing, et cetera, versus just me playing around with it. Yeah. How do you guys do model evaluation today? And any learnings on that since getting started? Oh, well, there's a bunch of...

We have a series of tough test cases that from the beginning we saw, and it's now several hundred cases that we know these models have had trouble with. And they are some of these classic examples

0.33 is what the student says. One third's the answer. Distributive property is one of these things that sometimes the models back in the day at least had troubles. We have a whole list of it. When we started this of the, let's call it 200 test cases, it was actually failing on about 70% of those test cases. Now these are very niche. It's not like it fails on 70% of interactions, but it's 70% of the hard test cases. Now those numbers, I haven't looked at it in the last month or two, but it's like,

It's like sub 10% of the very hardest test cases is my understanding. So we do some work like that. We also, you know, we have machine labeling of interactions. So, you know,

Can we use an AI to look at the AI interactions and saying, okay, it looks like there might've been an error here. The AI had said some point, it said, oh, my bad. You got it right. Wrong or things like that. But then we also do human labeling because the AI isn't,

AI labeling isn't perfect. And so it's that human labeling work. And we did about 2000 sample conversations. We did this about six months ago. That helped us get a much, you know, when these numbers I'm telling you, like two, 3% of errors is based on that human labeling. We're also doing some labeling on, I would not just errors, but the productivity of the conversation, like what percentage of conversations are healthy and the student is

with it versus what percentage of the students just saying, I don't know, I don't know, I don't know. Have you able to train a model to evaluate that or is it all human labeled today of like what's a perfect conversation? Right now it's all human labeled. In theory, we might be able to train a model to evaluate. In fact, we should be able to over time. But yeah, we're doing human labeling.

It's also it would be interesting because I think in a traditional classroom, you'd probably see similar ratios of the number of kids who are just like checked out and saying, I don't know. I don't know. They're not even engaged. But anyway, we're trying to label all of the above. Do you have like an early, you know, any early thoughts on, you know, I feel people are studying the impact of these models in lots of different, you know, contexts. And I feel like in the workplace, there's been some studies that are like, actually, it's like.

for folks that maybe were in the bottom half of the workforce that were super impactful. And I think it's everyone still trying to figure this out. Any early inklings on, at least to date, for the types of students or where these models have been like, or these products have been most impactful? Or still too early to tell? On the student side, I think for students who are curious and are already engaged, this is

their dream come true i mean they can ask any question etc so i i you know but there's there's the efficacy we're starting to run some and we are seeing some correlation with engagement um and potentially learning uh but i i think the models have to get much more part of the learning experience than just being there to answer questions before we start seeing real movement there yeah make

Yeah.

Yeah. I mean, especially the things I've been talking about, uh, you know, some memory, uh, architecture, um, some, um, robust eval architecture, automated eval architecture, uh, would be pretty, pretty good. Yeah. Those are, those are the, those are the, the, the two. And then, you know, if there's some architecture for some people who've really like nailed it on, on things like math and things, um,

So that it's, it is evaluating well, as opposed to just, we know, you know, yeah. You know, obviously it seems like you're making a big shift to these proactive, you know, making the models proactive. And, you know, to what extent does that end up being like customizable by an individual teacher versus like, hey, there's just general best practices for we're teaching algebra. And so we know the kind of like pedagogy or prompts to do, like, how have you thought about that? Yeah, the teacher in the loop is essential. One of the issues is,

So you can say, hey, I'm an AI here. Ask me any questions. You know, some kids will engage. You can say, hey, I'm the AI. I show up right when you come to the website. Come ask me some questions or I'm here to or you might have this question and maybe that's a little bit better. But actually the best is and this is what we're building for next year. If the AI notices that a student, let's say, notices Jacob is having trouble with the distributive property.

It tells Mr. Khan, Jacob's teacher, hey, Jacob's having trouble with the distributive property. Click here and you will assign a tutoring session with Khanmigo on the distributive property. And Jacob has to do it by tomorrow night. That's what we'll make...

It's his assignment. Jacob has to do it. So when you have teachers assigning AI tutoring interventions and then the teacher being able to hold the student accountable to that, that's when you're going to start finally seeing kids engage.

It's so exciting. And I feel like in hearing that, obviously, we're, you know, in San Francisco, there's all this, you know, techno optimism, and you kind of hear that and you're like, God, like, it almost feels, you know, it almost feels like a disservice to the world that this stuff isn't more widely available. If we have these capabilities today, I mean, especially, you know, even if you look at emerging markets where like, maybe there isn't as high quality of education today, like,

I'm sure there's a lot of techno optimists that are like, God, we should just be throwing these models at students like completely today and let them run. Like, what are those people missing? And I guess, you know, how do you see the next few years playing out like globally? Because when you when you describe that use case, it seems it seems unbelievably compelling.

Yeah, it's like there are some percentage of of of the of students in the world that can just run with things. There was this gentleman, Sugata Mitra, who did the school hole in a wall, school in a wall, something like that. And he put like tablets in rural India, poor kids. And I don't.

They claim that the kids just started going up to the tablets and started learning, et cetera, et cetera. I think there was some. I mean, there's definitely curiosity there. But I think that's limited. And I mean, there probably would be some benefit of just giving kids in villages in India access to chat GPT or something. Some subset will probably start making really good use of it. But once again, if you don't know what you don't know, I mean, you talked about the blank thing.

the blank slate, you don't know how to even structure your own journey. You wouldn't even know to prompt it to say, Hey, can you work me through the Indian national standards in algebra? And you know, you wouldn't even know to do that. So I do think you've got to structure it more. I think once you, and that's what Khan Academy, one of our values is that we've always had that structured content and there are kids, you know, I just met a young girl,

freshman at MIT from Afghanistan. She couldn't go to school. Khan Academy was her education. She used another platform we have called schoolhouse.world to prove what she knew. MIT accepted that and admitted her. So there are people out there who are running with the things that already exist. But yeah, I think you just start throwing stuff out on, you know, you're not going to

But I do think in another year or two, we will have stuff that you could. Ideally, you do have access to a teacher who knows the algebra or the physics or whatever. But if you don't and you're in a container with, you know, Starlink access and some 30, 40, 50 dollar tablets that are shared by five or six kids over the course of a day, I actually think you could get pretty far.

Yeah. Do you think we will see that? Yeah. I mean, there's already people who are trying it out, but I don't see any reason why we wouldn't see it. One of the things that I hope to do in the next year is for Khan Academy, let's call it two years. It really, we need some funding on the transcripting side to automate the transcripting, but I hope to give high school credits and high school diplomas. And I think once you...

offer high school credits, high school diplomas, and eventually college credits and college diplomas. I mean, I don't know when we'll do college diplomas, but college credits for sure. Then all of a sudden, if you're this kid in India, it becomes very compelling for you to have access to one of these devices and at least get a internationally recognized high school diploma. It goes a long way.

Yeah. I guess I'd love your commentary on just like the broader AI and education market. Like, you know, what do you, what have you kind of seen happening? You know, what are some other spaces or areas you pay attention to in that? Obviously it seems like you're really going for the heart of it, but curious, like what else you've seen or what else you find interesting going on? You know, there's, I think we're just, we're, there's a lot of noise right now. There's a lot of startups. Sounds like AI. What I, what we see is a lot of people who've done some

a layer over chat GPT and some prompting. And, you know, some of them maybe get a little bit of traction, but it's, it is, it is actually a hard environment. I mean, you're an investor here. It's, it's a, it's in some ways, it's a dream that you can make apps so much faster than you could before apps of substance. At the same time, the switching costs and the barriers to entry are so low that

I do think some of, not to pick on you guys, some of the incentives of the investor community don't necessarily allow you to take a slightly longer point of view. That's what I tell our team. Khan Academy doesn't, if the market fully solved the problems, Khan Academy doesn't need to exist. We're a nonprofit. We exist now.

I would say in this moment, our value is we actually do have permission to take a slightly longer view. We still have, you know, we put a lot of pressure on ourselves to still be very relevant in the short run and compete with anyone. But we have funders who will give us, you know, a five-year grant to think through assessment.

to think through writing, knowing that it's going to evolve and change. And honestly, some startups don't have that luxury. They just have to, you know, in the next nine months get some product market fit, otherwise they're going to disappear. And I think our other value is the trust that, okay,

you know, people are more sensitive around AI. Okay. Are they really not about, you know, somehow changing, undermining teachers? Are they really about the pedagogy or do they really have efficacy studies here? Is this something that I can feel good about? And, you know, over the years, hopefully we've been building a lot of that trust. So that helps in this AI world. Yeah. Are there any corners of like the education market that maybe you guys aren't focused on, but you're like, oh, I wish there was startups that were trying to build, you know, this area or

or how about with these kind of products? I'm sure there are startups that are doing it. I'm not aware of them and none of them have gone mainstream as far as I can tell. But, oh, I mean, there's just so much

The whole like interviewing piece is so resource intensive and broken and interviewing is assessment. And it's just, it just doesn't feel like assessment because it's so non-standardized and it's so expensive. And, you know, I used to do even before AI, like if I get to a final round with Google and Google tells me, Hey, Sal, you're a great guy. It was either you or Jacob.

And we really had to flip a coin. So we gave the job to Jacob, you know, good luck with your life. And, and then I go to Microsoft and have to start over again. And Microsoft has to start over again and spend all that time. They shouldn't have to like the fact that I got all the way to that end, Google should give me some type of certificate saying like, Sal is a really good,

And then all Microsoft needs to do is do a final round to make sure that I fit in and that I'm aligned with the job and things like that. So there's always been huge, huge, huge inefficiencies. Just even who gets access to an interview, it's a small subset. We have a product manager job out. We get hundreds and hundreds of applicants for the job. We're probably going to be able to interview phone screen maybe 20, 25 people, 20,

I'm sure there's some false negative and I'm sure we're going to miss in that 180, 280 people that we're not interviewing, not even phone screening. I'm sure there was some talent there, but maybe the best applicant is in that 280 that we don't even call back. So anyway, I think there's, there's interesting things there. I think there's interesting things in just corporate training development and

You know, all those things that we all have to do, these, you know, what's appropriate at work type of cybersecurity training or sexual harassment training and all of that. I got to believe a simulation with an AI would be way more interesting. Like the following has just played out. What do you do next? Totally. Well, I mean, you mentioned obviously this, you know, the application of these in the corporate world. And I guess...

you know, maybe zooming out, I feel like one broader question I'm sure you think a lot about is like the skills that will actually matter, just how fast the world's changing, like the skills that will actually matter for students that we're training for like the future workforce as these models keep getting better. Like, how do you think about that? And like the jobs we're training people for and to what extent, if at all, like what we teach has to adjust or change to adapt to that? Yeah, I believe that a smart, cogent,

strong critical thinker is always like, that's always really great baseline skills to have. So I, I think kids should continue to learn to write and read and do their math well, and, you know, have good general knowledge of social studies, history, et cetera, civics. But I think the thing that's really going to be, I think it,

I think economists talk about entrepreneurship as a factor of production, right? This ability to put resources that already exist, but put them in new permutations to create value that didn't exist in the world before. And most people always assume entrepreneurship is kind of like starting Khan Academy or what you invest in. And it is that. But I actually think that's going to be more and more of table stakes in almost any industry.

especially over this next 20 years when there's just like super rapid change. The people who have these, they're otherwise have solid skills, but they're just always saying, hey,

wow, I just heard about this one thing. Let me try to use that and put it together with this thing. And wow, if I do this and then I take that output and I put it here and I can make this, I might be able to make a pretty good, and it might not be the final output, but it might get me 80% there. And then I have to use my skills to tweak it to be those people I think are going to be, um, that's, those are the skills. Those are the skills. And, um,

Those are who I'm looking to highlight in our organization. And organizations that are not doing a lot of that are also going to suffer. If they don't have a lot of that entrepreneurship inside, they're not going to be able to innovate. Their cost structures are going to be way higher than everyone else. I love that. Well, we always like to end our interviews with a quickfire round where we get your take on a set of questions. Maybe to start, what's one thing you've changed your mind on with regards to AI in the last year? It's obvious in hindsight, but...

AI on its own, and I don't think I would have said AI on its own is going to solve all the problems. I wouldn't have said that even two years ago, but I've definitely shifted even more towards a lot of what we talked about, which is like, it's more about how does AI really empower the teacher to hold the students accountable and engage students in other productive things.

What's your favorite way that you use AI today within Khan Academy and then anything on the wishlist where you hope that you'll be using it a year or two from now? When I'm preparing for videos, to make a video, one of my, I guess you could call it, if you want to say superpowers is I ask all of the dumb questions in my head that sometimes people are afraid to ask. And I think sometimes teachers glaze over them because they were afraid to ask them too. But I'm like, wait, how does this make sense? Even if I'm doing like a fourth grade concept, I'm like, wait, wait, that doesn't make sense. Okay.

And sometimes I can figure it out by myself, but a lot of times I, you know, in the past I would, I would do web searches. I'd call people up. I have found that for someone who's asking the right questions, AI can dramatically accelerate, um, that process. I'm always able to make fun images now for, for videos that can visualize what I'm trying to show. So that's been valuable. Um,

Every now and then, I tend to... Most of my speeches, I give off the cuff, but I've had to give a few formal speeches, things like commencement addresses and things. And that's where just doing a verbal dump and then having the AI transcribe it and then at least doing the first pass at the speech and then tweaking it saves hours. I want to try doing some...

you know, I've, I've been talking to some of these people who have these platforms. They say that you can now like prompt the AI and it'll make the whole app and they'll host it for you and everything. I want to try that out. Cause I want to vibe coding, uh, on the, on the docket coding. There's bunkable, which is, I know the founder pretty well. And he was telling me the other day that, um,

no you you can make the whole app and and i even described an app that i wanted to make and he's like yeah that would probably work and i'm like wow okay so i need to try that out i keep telling my son my 16 year old who's really into programming and making games like what you should be doing this vibe kobe and it's so funny because he's like an old man he's like no that's that's that's cheating you know i'm like no but that's the future of coding you got to do a little bit about he's like no i need to learn it properly myself and i guess um you know any uh

What's been like the biggest surprise in building these features? I guess maybe something that you thought would work really well that didn't or something that you weren't expecting much from and has actually been hugely successful in the adoption of Conmigo. I mean, it's not surprising in that almost everything turns out this way when you try to do something at scale and enterprise. But yeah, it's a lot more work. Like when I first saw these models and I'm like, wow, these are magical. And look, I still do worry that...

Will these models leapfrog the apps that are using them in some way? Or will the general apps like the chat GPTs leapfrog the specialized apps? It's still a concern, but yeah, I mean, the amount of work to make it really work well for a certain use case is a lot. Things that have been better than expected,

You know, I was afraid two years ago whether our team was ready for this type of a pivot. And I was worried whether the education community was ready for this type of a pivot. I actually thought we might get a lot of flack for leaning so hard into AI. But it is pretty interesting now. Our team does view itself as an AI-first organization. And it's easy for me to say, but when I really think about ourselves two, three years ago, that was not...

And the obvious thing, and there was some pushback and there were people who didn't like that direction. And I would also say the education community has moved faster than expected. What were the weeks like? I mean, obviously you got the GPT-4 demo, you did your AP bio tests, and then you came back and you're trying to reorient. What was that like first month like post that demo? Well, almost every day we were talking to the OpenAI lawyers. We're like, we have five more people we want to get under NDA or 10 more people. And we had a little, we had an onsite. I remember that first

and we had a little office and every time we got people, we said, come with me. And I wanted to show people because it was such a like, they would, they thought they were getting punked. This was before chat GPT or anything. I'm like, I have something to show you. And they would walk in. I just enjoyed doing the demo. They're like, what's going on? What is this? What is this? What are you showing me? And I was like, this is the future. And

But, you know, a lot of those people, I think initially about half the company was like, this is everything. We have to stop what we're doing. And then half of them like, hold on, makes errors, hallucinates, bias. It's going to freak people out, cheating. And we said, look, we've both sides are right, but we can't, we've got to take all those risks and turn them into features. Yeah.

Well, I love that. Well, look, this has been a fascinating conversation. I want to make sure I leave the last word to you. Anywhere you want to point our listeners where they can go to learn more about Khan Academy, the AI work you're doing, anything else, the floor is yours. Yeah. You know, people want to try it out. You go to Khan Academy, Khanmigo, you can try it at Writing Coach if you're, you know, homeschool or teacher. I also, people should know about Schoolhouse.world, which is a sister nonprofit that I helped start, which is around free tutoring.

We do it through volunteers. So if people want to volunteer and give tutoring or get tutoring, we're about to launch something called the Dialogues Initiative where you're going to have conversations about tough topics and then rate each other on how well the other side listened. So yeah, take a look. And remember, we're a nonprofit, so donate. Yeah, awesome. Well, thanks so much, Sirius. This is a fascinating conversation. Great, thank you.

Thank you.

With the fast-moving pace of AI, we aim to help you deconstruct and understand the most important breakthroughs and see a clearer picture of reality. Thank you for listening and see you next episode.