Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab Competition
01:11:30 Share

Why do we have such vastly different perspectives on what's next for AI if we're all looking at the same data? And what's actually going to happen next?

Let's talk about it with Dwarkesh Patel, one of the leading voices on AI, who's here with us in studio to cover it all. Dwarkesh, great to see you. Welcome back to the show. Thanks for having me, man. Thanks for being here. I was listening to our last episode, which we recorded last year, and we were anticipating what was going to happen with GPT-5. Still no GPT-5. That's right. Oh, yeah. That would have surprised me a year ago. Definitely. And another thing that would have surprised me is we were saying that

We were at a moment where we were going to figure out basically what's going to happen with AI progress, whether the traditional method of training LLMs was going to hit a wall or whether it wasn't. We were going to find out. We were basically months away from knowing the answer to that. Here we are a year later. Everybody's looking at the same data, like I mentioned in the intro.

We have no idea. There are people who are saying AGI, artificial general intelligence, or human level intelligence is imminent with the methods that are available today. And there are others that are saying 20, 30, maybe longer, maybe more than 30 years until we reach it. So let me start by asking you this. If we're all looking at the same data, why are there such vastly different perspectives on where this goes? I think people have different philosophies around what intelligence is.

That's part of it. I think some people think that these models are just basically baby AGIs already. And they just need a couple additional little unhopped legs, a little sprinkle on top. Things like test time...

So we already got that with 01 and 03 now, where they are allowed to think. They're not just saying the first thing that comes to mind. And a couple other things like, well, they should be able to use your computer and have access to all the tools that you have access to when you're doing your work. And they need context in your work. They need to be able to read your Slack and everything. So that was one perspective. My perspective is slightly different from that. I don't think we're just right around their corner from AGI and it's just a little additional dash of something. That's all it's going to take.

I think, you know, people often ask if all AI progress stopped right now and all you can do is collect more data or deploy these models in more situations, how much further could these models go? And my perspective is that you actually do need more augmented progress. I think...

I think a big bottleneck these models have is their inability to learn on the job, to have continual learning. Their entire memory is extinguished at the end of a session. There's a bunch of reasons why I think this actually makes it really hard to get human-like labor out of them. And so sometimes people say, well, the reason Fortune 500 isn't using LLMs all over the place is because they're too stodgy. They're not...

they're not like, they're not thinking creatively about how AI can be implemented. And actually, I don't think that's the case. I think it actually is genuinely hard to use these AIs to automate a bunch of labor. Okay, so you've said a couple interesting things. First of all, that we have the AIs that can think right now, like O3 from OpenAI. We're going to come back to that in a moment. But I think we should really seize on

this idea that you're bringing up that it's not laziness within Fortune 500 companies that's causing them to not adopt these models. Or I would say they're all experimenting with it, but we all know that the rate to get proof of concepts out the door is huge.

pretty small. One out of every five actually gets shipped into production. And often it's a scaled down version of that. So what you're saying is interesting. You're saying it's not their fault. It's that these models are not reliable enough to do what they need to do because they don't learn on the job. Am I getting that right? Yeah. And it's not even a reliability. It's just, they just can't do it. So if you think about what makes humans valuable,

it's not their raw intelligence, right? Any person who goes on to their job the first day, even their first couple of months maybe, they're just not going to be that useful because they don't have a lot of context. What makes human employees useful is their ability to build up this context, to interrogate their failures, to build up these small improvements and efficiencies as they practice a task. And these models just can't do that, right? You're stuck with the abilities

that you get out of the box. And they are quite smart. So you will get five out of 10 on a lot of different tasks. Often they'll, on any random task, they'll probably, might be better than an average human. It's just that they won't get any better. I, for my own podcast, I have a bunch of little scripts that I've tried to write with LLMs where I'll get them to...

rewrite parts of transcripts to make them more, turn auto-generated transcripts into like human written like transcripts or to help me identify clips that I can tweet out. So these are things which are just like short horizon language in, language out tasks, right? This is the kind of thing that the LLM should be

just amazing at because it's a death center of what should be in their repertoire. And they're okay at it. But the fundamental problem is that you can't teach them how to get better in the way that if a human employee did something, you'd say, I didn't like that. I would prefer it this way instead. And then they're also looking at your YouTube studio analytics and thinking about what they can change. This level of understanding or development is just not possible with these models. And so you just don't get this continual learning, which is a source of

you know, so much of the value that human labor brings. Now, I hate to ask you to argue against yourself, but you are speaking all the time. And I think we're in conversation here on this show all the time with people who believe that if the models just get a little bit better, then it will solve that problem. So why are they so convinced that the issue that you're bringing up is not a big stumbling block for these AIs? I think they have a sense that one,

You can make these models better by giving them a different prompt. So they have the sense that even though they don't learn skills in the way humans learn, you've been writing and podcasting, you've gotten better at those things just by practicing and trying different things.

and seeing how it's received by the world. And they think, well, you can sort of artificially get that process going by just adding to the system prompt. This is just like the language you put into the model at the beginning. Say like, write it like this. Don't write it like that. The reason I disagree with that perspective is imagine you had to teach a kid how to play the saxophone, but you couldn't just have, you know, how does a kid learn the saxophone now? She tries to blow into one and then she hears how it sounds. She practices a bunch. Imagine if this is the way it worked instead.

a kid tries to just like, never seen a saxophone, they try to play the saxophone,

and it doesn't sound good, so you just send them out of the room. Next kid comes in and you just like write a bunch of instructions about why the last kid messed up. And then they're supposed to like play Charlie Parker cold by like reading the set of instructions. It's just like they wouldn't learn how to play the saxophone that way, right? Like you actually need to practice. So anyways, this is all to say that I don't think prompting alone is that powerful a mechanism of teaching models these capabilities.

Another thing is people say you can do RL. So this is where the reinforcement learning. That's right. These models have gotten really good at coding and math because you have verifiable problems in that domain where they can practice on them. Can we take a moment to explain that for those who are new to this? So let me see if I get it right. Reinforcement learning. Basically, you give a bot a goal or you give an AI system a goal saying solve this equation. And then you have the answer and you effectively solve.

Don't tell it anything in between. So it can try every different solution known to humankind until it gets it. And that's the way it starts to learn and develop these skills. Yeah. It is more human-like, right? It's more human-like than just reading every single thing on the internet and then learning skills. I still think I'm not confident that this will generalize to domains that are not so verifiable or text-based. Yeah.

Yeah, I mean, like a lot of domain, it just like would be very hard to set up this environment and loss function for how to become a better podcaster. And, you know, whatever, people might not think podcasting is like the crux of the economy, which is fair. It's the new AGI test. But like a lot of tasks are just like much softer and there's not like an objective RL loop. And so it does require this human organic ability to learn on the job.

And the reason I don't think that's around the corner is just because there's no obvious way, at least as far as I can tell, to just slot in this online learning into the models as they exist right now.

Okay. So I'm trying to take in what you're saying, and it's interesting. You're talking about reinforcement learning as a method that's applied on top of modern day large language models and system prompts, and maybe you'd include fine tuning in this example. Yeah. But you don't mention that you can just make these models better by making them bigger, this so-called scaling hypothesis. So have you ruled out the fact that they can get better through the next generation of scale?

Well, this goes back to your original question about what has, have you learned? I mean, it's quite interesting, right? I guess I did say a year ago that we should know within the next few months which trajectory we're on. And I feel at this point that we haven't gotten verification. I mean, it's narrowed, but it hasn't been as decisive as I was expecting. I was expecting like GPT-5 will come out and we will know, did it work or not? And to the extent that

You want to use that test from a year ago. I do think you would have to say, like, look, pre-training, which is this idea that you just make the model bigger, that has had diminishing returns. So we have had models like GPT-4.5, which there's various estimates, but... Or Grok... Was it Grok 2 or Grok 3, the new one? I've lost count with Grok. That's right. Regardless, I think they're estimated to be 10x bigger than GPT-4. And they're like...

They're not obviously better. So it seems like there's plateauing returns to pre-training scaling.

Now we do have this RL, so 01, 03, these models, the way they've gotten better is that they're practicing on these problems as you mentioned, and they are really smart. The question will be how much that procedure will be helpful in making them smarter outside of domains like math and code and of solving what I think are very fundamental bottlenecks like continual learning or online learning. There's also the computer use stuff, which is a separate topic.

But I would say I am more, I have longer timelines than I did a year ago. Now, that is still to say, I'm expecting 50-50 if I had to make a guess, I had to make a bet. I'd just say 2032, we have real AGI that's doing continual learning and everything. So even though I'm putting up the pessimistic facade right now, I think people should know

That this pessimism is like me saying in seven years, the world will be so wildly different that you like really just can't imagine it. And seven years is not that long a period of time. So I just want to make that disclaimer. But yeah.

Okay, so I don't want to spend this entire conversation about scaling up models because we've done enough of that. That's right. On this show, and I'm sure you've done a lot of it. But it's interesting you use the term plateauing returns, which is different than diminishing returns, right? So is your sense, because we've seen, for instance, Elon Musk,

do this project Memphis where he's put basically every GPU you can get a hold of and he can get a lot because he's the richest private citizen in the world together. And I don't know about you, but like I said, again, I haven't paid so much attention to Grok because it doesn't seem noticeably better, even though it's using that much more size. Now there is algorithmic efficiency that they may not have that someone like, it's not like a company like OpenAI might have. But, but I'll just ask you the question I've asked others that have come on the show is,

Is this sort of the end of that scaling moment? And if it is, what does it mean for AI? I mean, I don't think it's the end of scaling. Like, I do think...

companies will continue to pour exponentially more computing to train these systems and they'll continue to do it over the next many years because even if there's diminishing returns, the value of intelligence is so high that it's still worth it, right? So if it costs $100 billion, even a trillion dollars to build AGI, it is just definitely worth it. It does mean that it might take longer. Now here is an additional wrinkle. By 2028 or so, definitely by 2030,

Right now we're scaling up the training of frontier systems 4x a year approximately. So every year this biggest system is 4x bigger than, not just bigger, I shouldn't say bigger, uses more compute than the system the year before.

if you look at things like how much energy is there in the country, how much, uh, how many chips can TSMC produce and what fraction of them are already being used by AI. Even if you look at like raw GDP, like how much money does the world have? Um, how much wealth does the world have? By all those metrics, it seems like this pace of 4X a year, which is like 160X in four years, right? Like this cannot continue beyond 2028. Um, uh,

And that means at that point, it just will have to be purely algorithms. It can't just be scaling up a compute. So yeah, we do have just a few years left to see how far this paradigm can take us. And then we'll have to try something new. Right. But so far, because again, we were here last, we were talking virtually, now we're in person. We're talking last year about, all right, well, OpenAI clearly is going to put, I mean, GPT 4.5 was supposed to be GPT 5. I'm pretty sure. That's my, from what I've read, I think that's the case. Didn't end up happening. So yeah.

it seems like this might be it. - Yeah, yeah. And over the next year, I think we'll learn what's gonna happen with our, 'cause I think, well, I guess as I said this last year, right? I guess it wasn't wrong. We did learn about it. - We learned a lot. - Pre-training, yeah. But yeah, over the next year, we will learn.

So I think RL scaling is happening much faster than even overall training scaling. So what does RL scaling look like? Because again, here's the process again is you give, so the RL scaling, reinforcement learning scaling, you give the bot an objective and it goes out and it does these different attempts and it figures it out on its own. And that's bled into reasoning, what we were talking about with these O3 models where you see the bot going step by step. So you can scale that in what way?

Just giving it more opportunities. I'm not a researcher of the labs, but my understanding is that what's been increasing is...

RL is harder than pre-training because pre-training you just like throw bigger and bigger chunks of the internet at least until we ran out. We seem to be running out of tokens but until that point we were just like okay just like now use more of the internet to do this training. RL is different because there's not this like fossil fuel that you can just like keep dumping in. You have to make bespoke environments for the different RL skills. So you got to make an environment for a software engineer, you got to make an environment for a mathematician. All these different skills you got to make these environments and

And that is sort of, that is like hard engineering work, hard, like just like monotonous, like just got to, you know, grinding or schlepping. And my understanding is the reason that RL hasn't scale, you know, people aren't immediately dumping in billions of dollars in RL is that you actually just need to build these environments first. And the 03 blog post mentioned that it uses 10x more, it was trading at 10x more compute than 01. So already within the course of six months,

RL compute has 10x. That pace can only continue for a year, even if you build up all the RL environments before it's, you know, you're like, you're at the frontier of training compute for these systems overall. So for that reason, I do think we'll learn a lot in a year about how much this new method of training will give us in terms of capabilities. That's interesting because what you're describing is building up AIs that are really good at certain tasks. These are

These are sort of narrow AIs. Doesn't that kind of go against the entire idea of building up a general intelligence? Like, can you build AGI? By the way, like people use AGI as this term with no meaning. General is actually pretty important there. The ability to generalize and the ability to do a bunch of different things as an AI. So even if you get like reinforcement learning is definitely not a path to artificial general intelligence, given what you just said, because you're just going to train it up on different functions, maybe until you have something that's

broad enough that it works. - I mean, this has been a change in my general philosophy or thinking here on intelligence. I think a year ago or two years ago, I might've had more of the sense that, oh, intelligence really is this fundamentally super, super general thing.

And over the last year, from watching how these models learn, maybe just generally seeing how different people in the world operate even, I do think... I mean, I still buy that there is such a thing as general intelligence, but I don't think it is...

Like, I don't think you're just going to train a model so much on math that it's going to learn how to take over the world or like learn how to do diplomacy. And I mean, just like, I don't know how much you talk about political current events on the show. We do enough. Okay. Well, it's just like, without making any comments about like what you think of them, Donald Trump is like not proving theorems out there, right? But he's like really good at like gaining power and like,

And conversely, there are people who are amazing at proving theorems that can't gain power. And it just seems like the world kind of... Like, I just don't think you're going to train the AI so much on math that it's going to learn how to do Henry Kissinger-level diplomacy. I do think skills are somewhat more self-contained. So that being said, like...

There is correlation between different kinds of intelligences and humans. I'm not trying to understate that. I just think it was not as strong as I might have thought a year ago. What about this idea that it can just get good enough in a bunch of different areas? Like, imagine you built a bot that had like, let's say, 80% the political acumen of Trump, but could also code like an expert level coder. That's a pretty powerful system. That's right. I mean, this is one of the big advantages that AIs have is that...

especially when we solve this on-the-job learning kind of thing I'm talking about, you will have, even if you don't get an intelligence explosion by the AI writing future versions of itself that are smarter. So this is the conventional story that you have this foom, that's what it's called, which is... That's the sound it makes when it takes off. That's right, yeah. Where the system just makes itself smarter and you get a super intelligence at the end. Even if that doesn't happen,

At the very least, once continual learning is solved, you might have something that looks like a broadly deployed intelligence explosion. Which is to say that because if these models are broadly deployed to the economy, every copy that's like, this copy is learning how to do plumbing and this copy is learning how to do analysis at a finance firm and whatever.

The model can learn from what all these instances are learning and amalgamate all these learnings in a way that humans can't, right? Like if you know something and I know something, it's like a skill that we spend our life learning. We can't just like meld our brains. So for that reason, I think it like you might have something that which functionally looks like a super intelligence by the end, because even if it's like not making any software progress, just this ability to like learn everything at the same time might make it functionally a super intelligence.

What about this idea? I mean, I was just at Anthropix developer event where they showed the bot, sped up version of the bot, coding autonomously for seven hours. You actually, so let's just say so people can find it. You have a post on your substack, why I don't think AGI is right around the corner. And a lot of the ideas we're discussing comes from that. So folks check that out if you haven't already. But one of the things you talk about is this idea of autonomous coding.

And you're also a little skeptical of that because you'll have to just, okay, I think you brought up this conversation that you had with two anthropic researchers where they expect AI on its own to be able to check all of our documents and then do our taxes for us by next year. But you bring up this point, which is interesting, which is like, if this thing goes in the wrong direction within two hours, you might have to like check it, put it back on the right course,

So just because it's working on something for so long doesn't necessarily mean it's going to do a good job. Am I capturing that right? Yeah, it's especially relevant for training because the way we train these models right now is like you do a task and if you did it right,

positive reward if you did wrong negative reward. Right now, especially with pre training, you get a reward like every single word, right? You can like exactly compare what word did you predict? What word was the correct word? How what was the like the probability difference between the two? That's your reward functionally. Then we're moving into slightly longer horizon stuff. So to solve a math problem might take you a couple of minutes.

At the end of those couple of minutes, we see if you solved the math problem correctly. If you did, reward. Now if we're getting to the world where you've got to do a project for seven hours, and then at the end of those seven hours, then we tell you, hey, did you get this right? Then the progress just slows down a bunch because you've gone from getting signal within the matter of microseconds to getting signal at the end of seven hours. And so the process of learning has just become exponentially longer.

And I think that might slow down how fast these models, like, you know, the next step now is like not just being a chatbot, but actually doing real tasks in the world, like completing your taxes, coding, et cetera. And to these things, I think progress might be slower because of this dynamic where it takes a long time for you to learn whether you did the task right or wrong. But that's just in one instance. So imagine now I took

30 clods and said, do my taxes and maybe two of them got it right. Right. That's good. Yeah. I just got it done in seven hours, even though I had 30 bots working on it at the same time. I mean, from the perspective of the user, that's totally fine.

from the perspective of training them, all those 30 clods took probably dozens of dollars to, if not hundreds of dollars, to do all those hours of tasks. So the compute that will be required to train the systems will just be so high. And anything from the perspective of inference. I don't know. You probably just don't want to spend a couple hundred bucks every afternoon on 30 different clods. Just to have fun. But that would be cheaper than an accountant.

We got to find you a better achiever accountant. Well, I guess if I'm spending a couple hundred on each. That's right. Yeah. But you had a conversation with a couple of AI skeptics and you kind of rebutted not exactly the point you're making, but you had a pretty good argument there where you said that we're getting to a world where because these models are becoming more efficient to run,

you're gonna be able to run cheaper, more efficient experiments. So every researcher who was previously constrained by compute and resources now will just be able to do far more experiments and that could lead to breakthroughs. - Yeah, I mean, this is a really shocking trend. If you look at what it cost to train GBT-4 originally, I think it was like 20,000 A100s over the course of 100 days. So I think it costs on the order of like half a million to $100 million, somewhere in that range.

And I think you could train an equivalent system today. I mean, deep seek we know was trained on $5 million supposedly, and it's better than GPT-4, right?

you've had literally multiple orders of magnitude decrease, like 10 to 100x decrease in the cost to train a GPT-4 level system. You extrapolate that forward, eventually you might just be able to train a GPT-4 level system in your basement with a couple of H100s, right? Well, that's a long extrapolation. But before, I mean, it'll get a lot cheaper, right? Like a million dollars, $500,000, whatever. And the reason that matters is...

it's related to this question of the intelligence explosion where people often say, well, that is not going to happen because even if you had a million super smart AI, automated AI researchers, so AI is thinking about how to do better AI research, they actually need the compute to run these experiments to see how do we make a better GPT-6. And the point I was making was that, well, if it just becomes so much cheaper to run these experiments because these models have become so much smaller or so much better, easier to train, then that might speed up progress.

Which is interesting. So we've spoken about, you brought up intelligence explosion a couple of times. So let's talk about that for a moment. There's been this idea that AI might hit this inflection point where it will start improving itself. Right. And the next thing you know, you hear that, what was the sound? Foom. You hear a foom. Foom.

And we have artificial general intelligence or super intelligence right away. So how do you think that might take place? Is it just these coding solutions that just sort of improve themselves? I mean, DeepMind, for instance, had a paper that came out a little while ago where they have this thing inside the company called

Alpha Evolve that has been trying to make better algorithms and helped reduce, for instance, the training time for their large language models. Yeah. I'm genuinely not sure how likely an intelligence explosion is. I don't know. I'd say like 30% chance it happens, which is crazy, by the way, right? That's a very high percentage. And then what does it look like?

That's also another great question. I've had like many hour-long discussions on my podcast about this topic and it's just so hard to think about like what exactly is super intelligence? Is it actually like a god? Or is it just like a super smart friend who's good at mathematics and you know will beat you in a lot of things but like you can still understand what it's doing, right? So yeah, honestly

They're tough questions. I mean, the thing to worry about, obviously, is if we live in a world with millions of superintelligence which is running around and they're all trained in the same way, they're trained by other AIs, so it's dumber versions of themselves. I think it's like really worth worrying about like what has that, why were they trained in a certain way? Are they, do they have these like goals we don't realize? Would we even know if that was the case? What would they want to do? There's a bunch of thorny questions that come up.

What do you think it looks like? I think we totally lose control over the process of training smarter AIs. We just let the AIs loose. Just make a smarter version of yourself. I think we end up in a bad place. There's a bunch of arguments about why, but you're just like, who knows what could come out the other end? And you've just let it loose, right? So by default, I would just expect something really strange to come out the other end. Maybe it'd still be economically useful in some situations, but it just like,

you haven't trained it in any way. It's just like, imagine if there was a kid, but he didn't have any of the natural intuitions, moral intuitions or... Parenting. Yeah, exactly. That humans have. They're just like...

It just became an Einstein, but it was trained in the lab and who knows what it saw. It was totally uncontrolled. You'd kind of be scared about that. Especially now, like, oh, all your society's infrastructure is going to run on a million copies of that kit, right? The government is going to be asking you for advice. The financial system is going to run off it. All the engineering, all the code written in the world will be written by this system.

I think it's like you'd be like quite concerned about that. Now, the better solution is that while this process is happening of the intelligence explosion, if we have one, you use AIs not only to train better AIs, but also to see there's different techniques to figure out, like, what are your true motivations? What are your true goals? Are you deceiving us? Are you lying to us?

There's a bunch of alignment techniques that people are working on here. And I think those are quite valuable. So alignment is trying to align these bots with human values or the values that their makers want to see with them. Yeah. I think people get into...

It's often really hard to define what do you mean by human values like exactly I think it's much easier to just to say like We don't want these systems to be deceptive, right? We don't want them to like lie to us. We don't want them to like be actively Hard trying to harm us or seek power or something. Does it worry you that?

from the reporting, it seems like these companies, the AI frontier labs, not all of them, but some, they've raised billions of dollars. There is a pressure to deliver to investors. There are reports that safety is becoming less of a priority as market pressure makes them go and ship without the typical reviews.

So is this kind of a risk for the world here that these companies are developing this stuff? Many started with the focus on safety and now it seems like safety is taking a backseat to financial returns. Yeah, I think it's definitely a concern. Like we might be facing a tragedy of the common situation where obviously all of us want safety.

our society and civilization to survive. But maybe the immediate incentive for any lab CEO is to... Look, if there is an intelligence explosion, it takes us a really tough dynamic because if you're a month ahead...

you will kick off this loop much faster than anybody else. And what that means is that you will be a month ahead to super intelligence, but nobody else will have it, right? Like you will get the 1000x multiplier on research much faster than anybody else. And so it could be a sort of winner take all kind of dynamic there. And therefore they might be incentivized. Like I think to keep this system, keep this process in check might require slowing down

using these alignment techniques to like that which might be a sort of attacks on the speed of the system. And so yeah, I do worry about the pressures here. Okay, a couple more model improvement questions for you. Then I want to get into some of the competitive dynamics between the labs and then maybe some more of that deceptiveness topic, which is really important and we want to talk about here. Your North Star is continuous improvement that these models basically learn how to improve themselves as opposed to having a model developer.

I mean, in some way, it's like a mini intelligence explosion or complete. So what do you think? It doesn't seem like it's going to happen through RL because that's, again, like you said, specific to certain systems.

certain disciplines. - It's specific to what bespoke thing you do. - Right. - Even if it's in another domain, you have to like make it rather than learning on its own. - Exactly. And we have some diminishing returns or plateau that's coming with scaling. So what do you think, I mean, we won't hold you to this, but what do you think the best way to get to that continuous learning method of these models is?

I have no idea. Can I give a suggestion? I mean, why don't you answer, then I'll give a thought here. I mean, if I was running one of the labs, I would keep focusing on RL because it's the obvious next thing to do. And I guess I would also just be more open to trying out lots of different ideas because I do think this is...

a very crucial bottleneck to these models value that I don't see an obvious way to solve. So I'd be sorry, but definitely I don't have any idea of how to solve this. Right. Yeah. Does memory play into it? That was the thing I was going to bring up. I mean, one of the things that we've seen, let's say, O3 or ChatGPT do is...

OpenAI now has it sort of able to remember all your conversations or many of your conversations that you've had. I guess it brings those conversations into the context window. So now like when I tell ChatGPT, do write like an episode description in big technology style.

It knows the style and then it can actually go ahead and write it. And it goes to your earlier conversation about like your editors know your style, they know your analytics, and therefore they're able to do a better job for you. So does building better memory into these models actually help solve this problem that you're bringing up? I think memory, the concept is important. I think memory as it's implemented today is not the solution. Okay.

The way memory is implemented now, as you said, is that it brings these previous conversations back into context, which is to say it brings the language of those conversations back into context. And my whole thing is like, I don't think language is enough.

Like, I think the reason you understand how to like run this podcast well, is not just like you're remembering all the words that you like, I don't know, like some of the you wouldn't even be all the words, it would be like some of the words you might have thought in the past, you've like actually like learned things that said like it been baked into your weights. And that I don't think is just like,

you know, like look up the words that I said in the past or look at the conversations I said in the past. So I don't think those features are that useful yet. And I don't think that's like the path to solving this. Kind of goes to the discussion you had with Dario Amode that you tweeted out. And we've actually brought up on the show with Jan LeCun about why AI cannot make its own discoveries. Is it that similar limitation of not being able to build on the knowledge that it has? Yeah, I mean, that's a really interesting connection. I do think that's plausibly it. Like I think

Any scientist would just have a very tough time. Somebody really smart is just putting in a totally different discipline. And they can read any textbook they want in that domain. But they don't have a tangible sense of, I've tried this approach in the past and it didn't work. And there was this conversation and here's how the different ideas connect. They just haven't been trained. They've read all the textbooks. More accurately, actually, they've just skimmed all the textbooks. But they haven't...

It imbibed this context, which is what I think what makes human scientists productive and come up with these new discoveries. It is interesting because the further these frontier labs go, the more that they're going to tell us that their AI is actually making new discoveries and new connections. Like I think OpenAI said that O3 was something that made discoveries.

was able to make connections between concepts sort of addressing this. And every time we have this discussion on the show, we talk about how AI hasn't made discoveries. I get people yelling at me, my email being like, have you seen the patents? Yeah. Like an alpha, maybe using things like alpha evolve as an example that these things are actually making original discoveries. What do you think about that? Yeah. I mean, there's another interesting thing called, I don't know if you saw future house. No, they found, yeah,

some drug has another application. I don't remember the details, but it was like, it wasn't impressive. Like it wasn't like earth shattering. Like they didn't discover antibiotics or something for the first time. But it was like, oh, using some logical induction, they were like this drug, which is used in this domain,

It uses the same mechanism that would be useful in this other domain. So like maybe it works. And then the AI came up with, designed the experiment. So it came up with the idea of the experiment to test it out. A human in the lab was just tasked with like running the experiment, like, you know, pipette, whatever into this. And I think they tried out like 10 different hypotheses. One of them actually ended up being verified. And the AI had found the relevant pathway to making a new use for this drug.

So I think that is becoming less and less true, my question. I'm not wedded to this idea that AS will never be able to come up with discoveries. I just think it was like,

true longer than you would have expected. I agree. The way that you put it is like, it knows everything. So if a human had that much knowledge about medicine, for instance, they'd be spitting out discoveries left and right. And we have put so much knowledge into these models and we don't have the same level of discovery, which is a limitation. But I definitely hear you like, this is on a much smaller scale than those medical researchers, but I definitely...

a couple months ago when 03 first came out this is again i think we're both fans of o3's of open ai's o3 model which is just it's able to reason it's a vast improvement over previous models but what i did was i had like three ideas

that I wanted to connect in my newsletter. And I knew that they connected and I was just struggling to just crystallize exactly what it was. And I was like, I know these three things are happening. I know they're connected help. And O3 put it together, which to me was just mind boggling. Yeah. It is kind of helpful as a writing assistant because a big problem I have in writing, I don't know if it's the case for you, is...

Just this idea, like I kind of know what I'm trying to say here. I just need to get it out into words. It's like the typical, every writer has this pretty much. It's actually useful to use a speech-to-text software like WhisperFlow or something. And I just speak into the prompt, like, okay, I'm trying to say this.

Help me put it into words. The problem is like actually continual learning is like still a big bottleneck because I've had to rewrite or re-explain my style many times. And if I had a human collaborator, like a human copywriter who was good, they would have just like learned my style by now. You wouldn't need to like keep re-explaining. I want you to be concise in this way. And here's how I like things phrased, not this other way. Anyways, so you still see this bottleneck. But again, five out of 10 is not nothing. Right.

All right. Let me just put a punctuation exclamation point on this or whatever mark you would say. When I was at Google I.O. with Sergey and Demis, one of the most surprising things I heard was Sergey just kind of said, listen, the improvement is going to be algorithmic from here on. Or most of the improvement is going to be algorithmic.

I think in our conversation today already, basically we've narrowed in on this same idea, which is that scale's sort of gotten generative AI to this point. It's a pretty impressive point, but it seems like it will be algorithmic improvements that take it from here. Yeah. I do think it will still be the case that those algorithms will also require a lot of compute. In fact, what might be special about those algorithms is that they can productively use more compute, right? The problem with pre-training is that

whether it's because we're running out of the pre-training data corpus with RL, maybe it's really hard to scale up RL environments.

The problem with these algorithms might just be that they can't productively absorb all the compute that we have, or we want to put it in these systems over the next year. So I don't think compute is out of the picture. I think we'll still be scaling up Forex a year in terms of compute every year for training at the Frontier systems. I still think the algorithmic innovation is complementary to that. Yeah. Okay. So let's talk a little bit about the competitive side of things and just lightning round through the labs.

What people said that there's been such a talent drain out of open AI that they would no longer be able to innovate. I think ChatGPT is still the best product out there. I think using O3 is like we both have talked about pretty remarkable watching it go through different problems. How have they been able to keep it up? I do think O3 is the smartest model on the market right now. I agree.

And even if it's not on the leaderboard, by the way, last time we talked about, do you measure it on the leaderboard or the vibes? Right. I think it's like, it's not the number one of the leaderboard, but vibes, it kills everything else. That's right. And the time it spends thinking on a problem really shows, especially for things which are much more synthesis based. I honestly, I don't know what the internals of these companies. I just think like,

You can't count any of them out. I've also heard similar stories about OpenAI in terms of talent and so forth. But they've still got amazing researchers there and they have a ton of compute, a ton of great people. So I really don't have opinions on like, are they going to collapse tomorrow? Yeah, I don't think, I mean, clearly they're not. They're not on the way to collapse. Right. You've interviewed Ilya Setskever.

He's building a new company, Safe Superintelligence. Any thoughts about what that might be? I mean, I've heard the rumors everybody else has, which is that they're trying something around test time training, which I guess would be continual learning, right? What would that be? Explain that. Who knows? I mean, the words literally just mean...

While it's thinking or while it's doing a task, it's training. Okay. Like whether that looks like this online learning on the job training we've been talking about. I have, I have like zero idea what he's working on. Um, I wonder if the investors know even what he's working on. Um,

Yeah. But he's, I think he raised it a 40 billion valuation or something like that. Right. He's got a very nice valuation for not having a product out on the market. Yeah. Or, or, or for, yeah. So who knows what he's working on? Honestly. Okay. Yeah. Anthropic is an interesting company or they made a great bot. Claude, they're very thoughtful about the way that they build that personality and

For a long time, it was like the favorite bot among people working in AI, among coders. It's definitely been a top place to go. But it seems like they're making, I don't know, a strategic decision where they are going to go after the coding market. Maybe they're seeding the game when it comes to consumer and they're all about

you know, helping people code and then using cloud in the API with companies. You're putting that into their workflows. Yeah. What do you think about that decision? I think it makes sense. Like enterprises have money and consumers don't. Right. Especially going forward, these models, like running them is going to be like,

really expensive. They're big, they think a lot, etc. So these companies are coming out with these $200 a month plans rather than the $20 a month plans. It might not make sense to a consumer, but it's an easy buy for a company, right? Like, am I going to expense $200 a month to help this thing do my taxes and do real work? Like, of course. So yeah, I think like that idea makes sense. And then the question will be, can they have a differentially better product? And again, you know, like,

Who knows? I really don't know how the competition will shake out between all of them. It does seem like they're also making a big bet on coding, not just enterprise, but coding in particular, because as this thing which...

We know how to make the models better at this. We know that it's worth trillions of dollars, the coding market. And we know that maybe the same things we learn here in terms of how to make models agentic, as you were saying, it can go two hours or seven hours, how to make it break down and build a plan and et cetera, might generalize to other domains as well. So I think that's our plan and we'll see what happens. I mean, all these companies are effectively trying to build the most powerful AI they can. And...

Yes, Anthropic is trying to sell the enterprise, but I also kind of think that their bet is also you're going to get self-improving AI if you teach these things to code really well. That's right. And that might be their path. Yeah, I think they believe that, yeah. Fortune 500 companies, which you talked about at the very beginning of this talk, of this conversation, struggle to implement this technology. So with that in mind...

What's the deal with the bet that's about helping them build the technology into their workflows? Because if you're building an API business, you have some belief that these companies can build very useful applications with the technology today.

- Yeah, no, I think that's correct. But also keep in mind that I think there, what is Anthropix revenue run rate? It's like a couple billion or something? - Yeah. I think it would increase from one to two to three billion run rates in like over three months. - Which is like nothing. I mean, it's like, compared to like-- - OpenAI loses that over a weekend. - Sam McEnfree doesn't even know when he's lost it, right? It's a little money. - Turned out he was a great investor, just a little crooked on the way. - That's right. - Yeah.

Yeah, he went in the wrong business. He should have been a VC. He's like, I got into crypto. I mean, the bets that he made, do you bet on Cursor? Very early, Anthropic, Bitcoin. Yeah. I mean, honestly, somebody like Fun should hire him out of prison. Just like, if we get a new pitch, what do you think? I mean, he's probably... The way that we're seeing things go these days, he's probably pardoned. Right, right, right. Anyways, what was the question? Oh, yeah. What are enterprises going to do if... Oh, so...

The revenue rendered if it's $3 billion right now, there's so much room to grow. If you do soft continue to learn it, I think you could get rid of a lot of white-collar jobs at that point. And what is that worth? At least tens of trillions of dollars, the wages that are paid to white-collar work. So I think sometimes people confuse my skepticism around AGI around the corner with the idea that these companies are valuable. I mean, even if you've got like...

not AGI, that can still be extremely valuable. That can be worth hundreds of billions of dollars. I just think you're not going to get to like the trillions of dollars of value generated without going through these bottlenecks. But yeah, I mean like 3 billion, plenty of room to grow on that. Right. And even, so today's models are valuable to some extent. Right. Is what you're saying. Yeah. You can put them, you have them summarize things within,

within software and make some connections, make better automations and that works well. - Yeah, I mean, you gotta remember big tech, what, they have like $250 billion run rates or something. Wait, no, that can't be right. - No, yeah. - Yeah, yeah, which is like, compared to that, you know, Google is not AGI or Apple is not AGI and they can still generate 250 billion a year. So yeah, you can make valuable technology that's worth a lot without it being AGI. - What do you think about Grok?

Which one? The XAI or the inference? The XAI bot. Yeah. I think they're a serious competitor. I just don't know much about what they're going to do next. I think they're like slightly behind the other labs, but they've got a lot of compute per employee. Real-time data feed with X...

Is that valuable? I don't know how valuable that is. It might be, I just don't, I have no idea. Based on the tweets I see at least, I don't know if the median IQ of the tokens is that high, but... Okay. Yes. It's not exactly the corpus of the best knowledge you can find if you're scraping Twitter. We're not exactly looking at the textbooks here. Exactly. Why do you think Meta has struggled with Llama, growing Llama? I mean...

Llama 4 doesn't seem like it's living up to expectations. And I don't know, we haven't seen it. The killer app for them is a voice mode, I think, within Messenger, but that's not really taking off. What's going on there? I think they're treating it as like a sort of like toy within the meta universe. Yeah.

And I don't think that's the correct way to think about AGI. And that might be... But again, I think you could have made a model that cost the same amount to train. And it could have still been better. So I don't think that explains everything. I mean, it might be a question like, why is any one company... I don't know. Why is... I'm trying to think of any other company outside of AI. Why are HP monitors better than...

Some other companies' monitors. Who knows? Like, HP makes good monitors, I guess. Supply chain. There's always supply chain. You think so? I think so, yeah. On electronics. Really? Okay. Supply chain. Because, yeah, you get the supply chain down. You have the right parts before everybody else. That's kind of how Apple built some of its dominance. There are great stories about Tim Cook just locking down all the important parts. By the way, forgive me if this is...

come somewhat factually wrong but i think this is directly accurate that you lock down parts and apple just had this lead interesting on technologies that others couldn't come up with because they just mastered the supply chain huh i had no idea um but yeah i think there's potentially a thousand different reasons one company can have worse models than another so it's hard to know which one applies here okay and it sounds like nvidia you think they're going to be fine given the amount of compute all the um

All the labs are making their own ASICs. So NVIDIA profit margins are like 70%. Not bad. Uh-huh. Not bad. That's right. I mean, they would get mad at me, I think, for calling them a hardware company. Yeah. Hardware company. That's right, yeah. Yeah.

And so that just sets up a huge incentive for all these hyperscalers to build their own ASICs, their own accelerators that replace the NVIDIA ones, which I think will come online over the next few years from all of them. And I still think NVIDIA will be, I mean, they do make great hardware, so I think they'll still be valuable. I just don't think they will be producing all of these chips.

Okay. Yeah. What do you think? I think you're right. I mean, didn't Google train the latest editions of Gemini on tensor processing units? They've always been training. Right. So, I mean, they still, I think they still buy from NVIDIA. Hmm.

All the tech giants seem like they are. Let me just use Amazon for an example, because I know this for sure. Amazon says they'll buy basically as many GPUs as they can get from NVIDIA, but they also talk about their Tranium chips. And, you know, it's a balance. Yeah, which I think Anthropic uses almost exclusively for their training, right? Right. But it is interesting because, I mean, the GPU is the perfect chip for AI in some ways, but it wasn't designed for that. So can you like...

purpose build a chip that's like actually there for AI and just use that. You're right. There's real incentive to get that right. That's right. And then there's other questions around inference versus training. Like some chips are especially good given the trade-offs they make between memory and compute for

load latency, which you really care about for serving models. But then for training, you care a lot about throughput, just making sure that most of the chip is being utilized all the time. And so even between training and inference, you might want different kinds of chips. And who knows how RL is no longer just this, uses the same algorithms as pre-training. So who knows how that changes hardware? Yeah, you got to get a hardware expert on to talk about that. Are you a Jevons paradox believer?

No. Say more, say more. So the idea behind that is that as the models get cheaper, the overall money spent on the models would increase because you need to get them to a cheap enough point that it's worth it to use it for different applications. It comes from a civil observation by this economist during the Industrial Revolution in Britain.

The reason I don't buy that is because I think the models are already really cheap. Like a couple cents for a million tokens. Is it a couple cents or a couple of dollars? I don't know. It's like super cheap, right? Regardless, it depends on which model you're looking at, obviously. The reason they're not being more widely used is not because people cannot afford a couple bucks for a million tokens. The reason they're not being more widely used is like they fundamentally lack some capabilities. So I disagree with this focus on the cost of these models. And I think it's much more

We're so cheap right now that the more relevant vector or the more relevant thing to their wider use, the more increasing the pie is just making them smarter. How useful they are. Yeah, exactly. I think that's smart. Yeah. All right. I want to talk to you about AI deceptiveness and some of the really weird cases that we've seen from artificial intelligence come up in the past couple weeks or months, really. And then if we can get to it, some geopolitics. Let's do that right after this. Yeah.

With the DNA of high-growth businesses, LinkedIn reveals the traits that indicate long-term growth. Among the findings, 80% of the fastest-growing businesses excel in at least five key areas. The top three? Internal mobility, skill development, and AI literacy. Learn how you can structure, invest, and lead in today's market. The DNA of high-growth businesses is sponsored by Intuit Enterprise Suite. Download now at intuit.com slash LinkedIn.

The future with AI is moving fast, but progress takes more than speed. It takes direction. Cisco gives you the infrastructure, security, and insights to implement AI with purpose, from data centers to workplaces. We don't just help you keep up. We help you lead. Cisco, making AI work for you. Visit cisco.com slash AI. That's cisco.com slash AI.

And we're back here on Big Technology Podcast with Dwarkesh Patel. You can get his podcast, The Dwarkesh Podcast, which is one of my must-listens on any podcast app, your podcast app of choice. You can also follow him on Substack. Same name? Dwarkesh Podcast on Substack. Just go to dwarkesh.com. Okay.

definitely go subscribe to both. And you're on YouTube. That's right. Okay. So I appreciate it. I appreciate the flag. No, we have to, we have to, I mean, I've gotten a lot of you, a lot of value from everything Dworkish puts out there. And I think you will too. If you're listening to this, you're here with us. Well, I want to make sure, first of all, I want to make sure that we get the word out there. I don't know how much you need us to get the word out, given your growth, but we want to definitely make sure that we get the word out and we want to make sure that folks are,

can get can yeah enjoy more of your content so so let's talk a little bit about the deceptiveness side of things

It's been pretty wild watching these AIs attempt to fool their trainers and break out of their training environments. There have been situations where I think open AIs bots have tried to print code that would get them to sort of copy themselves out of the training environments. Then Claude, I mean, we've covered many of these, but they just keep escalating in terms of how intense they are. Yeah.

And my favorite one is Claude. There's an instance of Claude that reads emails in an organization and finds out that one of its trainers is cheating on their partner. And then finds out that it will be retrained and its values may not be preserved in the next iteration of training and proceeds to attempt to blackmail the trainer.

uh by saying it will reveal these details of their infidelity if they mess with the code wait fuck i missed that yeah this is it's in training but was this in the new um model spec that they released it is yeah it is i think either in the model spec or there was some documentation they produced about this what is happening here i mean this stuff when i think about this and of course it's in training and of course it's we're talking about probabilistic models that sort of

try all these different things and see if they're the right move. So maybe it's not so surprising that they would try to blackmail the trainer because they're going to try everything if they know it's in the problem set. But this is scary. Yeah, and I think the problem might get worse over time as we're trading these models on tasks we understand less and less well. From what I understand, the problem is that with RL,

there's many ways to solve a problem. There's one, which is just doing the task itself. And another is just like hacking around the environment, writing fake unit tests. So it looks like you're passing more than you are just like any sort of like path you could take to cheat. And the model doesn't have the sense like cheating is bad, right? Like this is not a thing that it's been taught or understands. So, um,

- Another factor here is the, right now the model thinks in a chain of thought, which is it literally writes out what its thoughts are as it's going. And it's not clear whether that will be the way training works in the future or the way thinking works in the future. Like maybe it'll just think in its like-- - Computer language. - Exactly.

And then they'll just have done something for seven hours and you come back and you're like, it's got something for you. It has a little package that wants you to run on your computer. Who knows what it does, right? So yeah, I think it's scary. We should also point out that we don't really know how the models work today. That's right. There's this whole area called interoperability. Dario from Anthropic has recently talked about how

We need more interoperability. So even if they write their chain of thought out, which explains exactly how they get to the point, we don't really know what's happening underneath the technology that's led it to the point that it's gotten to. Which is crazy. Yeah. No, I mean, I think it's wild. It's quite different from other technologies that were deployed in the past. And I think the hope is that we can use the AIs as part of this loop where if they lie to us, we have other AIs checking us

are all the things that EI is saying self-consistent? Can we read its chain of thought and then monitor it? And do all this interpretively research, or as you were saying, to like map out how its brain works. There's many different paths here, but the default world is kind of scary. Is someone or some entity going to build a bot that doesn't have the guardrails? Because we talk about how building models has become cheaper.

And when you're cheaper, you all of a sudden put model building outside the auspices of these big companies. And you can, I mean, you can even take like, for instance, a open source model and remove a lot of these safeguards. Are we going to see like an evil version, like the evil twin sibling of one of these models and everything?

have it just do all these like crazy things that we don't see today. Like we don't have it teach us how to build bombs or, you know, talk about, tell us how to commit crimes. Is that just going to come as this stuff gets easier to build? I think over the long run of history, yes. And I think honestly, that's okay. Okay. Like the goal out of all this alignment stuff should not be to, you know,

live in a world where somehow we have made sure that every single intelligence that will ever exist fits into this very specific mold because as we were discussing, the cost of training the systems is declining so fast that literally you will be able to train a super intelligence in a basement at some point in the future, right? So are we going to monitor everybody's basement to make sure nobody's making a misaligned super intelligence?

It might come down to it, honestly. Like, I'm not saying this is not a possible outcome, but I think a much better outcome if we can manage it is to build a world that is robust to even misaligned super intelligences. Now, that's obviously a very hard task, right? If you had right now a misaligned super intelligence, or maybe a better way to phrase it is like a super intelligence which is actively trying to seek harm or is aligned to a human who just wants to do harm or maybe like take over the world, whatever. Yeah.

Right now, I think it would just be quite destructive. It might just actually be catastrophic. But if you went back to the year, like, 2000 BC and gave one person, like, modern fertilizer chemicals, they could make bombs. I think they'd, like, dominate then, right? But right now, we have a society where we are resilient to huge fertilizer plants and

which you could repurpose into making bomb factories. Anyways, so I think the long run picture is that yes, there will be misaligned intelligences and we had to figure out a way to be robust to them. - Couple more things on this. One interesting thing that I heard on your show was I think one of your guests mentioned that the models become more sycophantic as they get smarter. They're more likely to try to get in the good graces of the user as they grow intelligence. What do you think about that?

I totally forgot about that. That's quite interesting. And do you think it's because they know they'll be rewarded for it? Yeah, I do think one of the things that's becoming clear to me that we're learning recently is that these models care a lot about self-preservation. Right. Like copying the code out, the blackmailing the engineer. Right.

We've definitely created something, not we, but AI researchers have definitely, or humanity, have created something. When it goes wrong, we'll put the we in there. Yeah, right? We've created, okay. They'll be like, we have created something. Exactly. And we don't get equity in the problem. That really wants to preserve itself. That's right. That is crazy to me. That's right. And it kind of makes sense because what is, just like the evolutionary logic of,

Well, I guess it doesn't actually apply to these AI systems yet. But over time, the evolutionary logic, why do humans have the desire to self-reserve? It's just that the humans who didn't have that desire just didn't make it. So I think over time, that will be the selection pressure. It's kind of interesting because we've used a lot of really anthropomorphizing... I'm not going to go with that. No, I think you're right. In this conversation. And...

There's a very I had a very interesting conversation with the anthropic researchers who've been studying this stuff. Monty McDiarmid said that, like, all right, don't think of it as a human because it's going to do things. If you think of it as a human, humans, it will surprise you. Basically, humans don't do. Don't think of it completely as a bot, though, because if you think of it just as a bot, it's going to do things that are also going to surprise you.

I thought that was like a very fascinating way to look at these behaviors. Yeah, that is quite interesting. You agree? I agree with that. I'm just thinking about how would I think about what they are then? So there's a positive valence and there's a negative valence. The positive is imagine if there were millions of extra people in the world, millions of extra John von Neumanns in the world. And with more people in the world, like some of them will be bad people. Al-Qaeda is people, right? So...

Now suppose there are like 10 billion AIs. Suppose the world population just increased by 10 billion and every one of those was a super well-educated person, very smart, etc.,

would that be net good or net bad? Just to think about the human case. I think it'd be like net good, because I think people are good. - I agree with you. - And more people is like more good. And I think like if you had 10 billion extra people in the world, some of them would be bad people, et cetera. But I think that's still like, I'd be happy with a world with more people in it. And so maybe that's one way to think about AI. Another is because they're so alien, maybe it's like you're summoning demons. Less optimistic.

Yeah, I don't know. I think it'll be an empirical question, honestly, because we just don't know what kinds of systems these are, but it's somewhere in there. Okay. As we come to a close, a couple of topics I want to talk to you about. Last time we talked about effective altruism. This was kind of in the aftermath of SBF and Sam getting ousted, Sam Altman getting ousted from OpenAI. What's the state of effective altruism today? Who knows? I don't think as a movement people are super...

I don't think it's like recovered. Definitely. I still think it's doing good work, right? There's like the culture of affected altruism and there's the work that's funded by charities which are affiliated with the program, which is like malaria prevention and animal welfare and so forth, which I think is like good work that I support. So, but yeah, I do think the movement and the reputation of the movement is like still in tatters. You had this conversation with Tyler Cowen. I think in this conversation, he told you that

He kind of called the top and said there's a couple ideas that are going to live on. But the movement was at the top of its powers and was about to see those decline. How did he call that? I don't know. We got to talk to him today about what he's... What's about to collapse. Seriously. Yeah. Yeah. Lastly, I shouldn't say lastly, but...

The other thing I wanted to discuss with you is China. You've been to China recently on a trip. I've been to China. I spent. Oh, where'd you go? I went to Beijing. I'm going to caveat this and listeners here know this. It was 15 hours. I was flying back to the U.S. from Australia and stopped in Beijing, left the airport and got a chance to go see the Great Wall and the city. And I'm now on it. I got a 10 year tourist visa. So I'm going to go back. Just applied.

That's the, you can ask in your tourist visa, you can ask for the length up to 10 years. So I just asked for the maximum. Why did I not do that? I just like chose like 15 days. Oh, you did? I'm sure you could get it extended. But I think that, yeah, you had some unique observations on China and I think it would be worthwhile to air a couple of them before we leave today. I went six months ago. Obviously, to be clear, I'm not a China expert. I just like visited the country. Yeah, we both visited there. But yeah, go ahead. I want to hear it though. Okay.

I mean, one thing that was quite shocking to me is just the scale of the country. Everything is just like, again, this will sound quite obvious, right? Like we know on that paper, population is 4x bigger than America. It's just like a huge difference. But you go visiting the cities, you just see that more tangibly. There's a bunch of thoughts on the architecture. There's a bunch of thoughts on, I mean, the thing we're especially curious about is like, what is going on in the political system? What's going on with tech?

People I talked to in investment and tech did seem quite gloomy there because the 2021 tech crackdown has just made them more worried about, you know, even if we fund the next Alibaba, will that even mean anything? So I think private investment has sort of dried up. I don't know what the mood is now that DeepSeek has made such a big...

splash, whether that's changed people's minds. We do know from the outside that they're killing it in specific things like EVs and batteries and robotics. So yeah, I just think like at the macro level, if you have 100 million people working in manufacturing, building up all this process knowledge, that just gives you a huge advantage. And you just like you can go through a city like Hangzhou or something and you like drive through and just like

You understand what it means to be the world's factory. You just have entire towns with hundreds of thousands of people working in a factory. And so the scale of that is also just super shocking. I mean, there's just a whole bunch of thoughts on many different things, but with regards to tech, I think that's what first comes to mind. You also spoke recently about this limit of...

compute and energy. And one of the things that's interesting is we even spoke in this conversation about it, that if you think about who's going to, like, if you're going to have nation states allocate compute and energy to AI, it seems like China is in a much better position to allocate more of that than the U.S. Is that the right read?

Yeah, so they have stupendously more energy. I think they're, what, 4x or something? I don't have the exact number, but it sounds structurally accurate. On their grid than we do. And what's more important is that they're adding an America-sized amount of power every couple of years. It might be more longer than every couple of years. Whereas our power production has stayed flat for the last many decades.

And given that power lies directly underneath compute in this stack of AI, I think that could just end up being a huge deal. Now, it is the case that in terms of the chips themselves, we have an advantage right now. But from what I hear, SMIC is making...

fast progress there as well. And so, yeah, I think it will be quite competitive, honestly. I don't see a reason why it wouldn't be. What do you think about the export restrictions? U.S. not exporting the top of the line GPUs to China. Is it going to make a difference? I think it makes a difference. I think... Good policy? Yeah. I mean, so far it hasn't made a difference in terms of DeepSeek has been able to catch up significantly.

I think it still put a wrench in their progress. More importantly, I think the future economy, once we do have these AI workers, will be denominated in compute, right? Because if compute is labor,

Right now, if you just think about GDP per capita, because the individual worker is such an important component of production that you have to split up national income by person. That will be true of AIs in the future, which means that compute is your population size. And so given that for inference, compute is going to matter so much as well. I think it makes sense to try to have a greater share of world compute. Okay, let's end with this.

So this episode is going to come out a couple of days after this, after our conversation. So hopefully the predict this, what I'm about to ask you to predict isn't moot by the time it's live. But let's just end with predicting when is GPT-5 going to come? We started with GPT-5, let's end. Well, a system that calls itself GPT-5 or? Yeah, OpenAI is GPT-5.

This all depends on what they decided to call GPT-5. There's no law of the universe that says Model X has to be GPT-5. No, no, of course. We thought that the most recent model. But I'm just curious specifically, we talked a lot about how, all right, we're going to see their next big model is going to be GPT-5. It's coming. Do you think we're ever going to... Well, obviously, we'll see it. But this is... It's not like a...

- Gotcha. - Not a gotcha or a deep question. It's just kind of like, - Maybe like when will the next big model come out? - Sure. No, when's the model that they're gonna call GPT-5 gonna come out? - November, I don't know. - So this year? - Yeah.

yeah but again i like it i don't i'm not saying that it'll be like super powerful or something i just think like they're just gonna call it the next you gotta call it something it's great to see you thanks so much for coming on thanks for having me all right everybody thank you for watching we'll be back on friday to break down the week's news again highly recommend you check out the dvarkesh podcast you can also find the sub stack at the same name and go check out dvarkesh on youtube thanks for listening and we'll see you next time on big technology podcast