Will AI Radically Change the World by 2027?... from Risky Business
47:03 Share

Pushkin. In golf, precision is everything. On and off the course, PGA of America chose advanced 5G solutions from T-Mobile for Business for smooth operations and seamless transactions. Together, they enhance ticketing and concessions for better fan experiences from gate to green.

This is pro-level efficiency. This is PGA of America with T-Mobile for Business. Take your business further at T-Mobile.com slash now.

I was joking with my producer Jacob the other day, who's one of Pushkin's most valuable employees. I hired him to be my assistant years ago in the most random manner possible. I think he saw a message board posting somewhere and I interviewed him for basically 10 minutes and said, go for it. I made a wild gamble on someone and got incredibly lucky.

But let's be honest, you can't rely on getting lucky when it comes to hiring people. Lightning's not going to strike more than once. You need a system and you need tools. And that's why LinkedIn is so important. LinkedIn is more than just a job board. They help connect you with professionals you can't find anywhere else. Even people who aren't actively looking for a new job.

In a given month, over 70% of LinkedIn users don't visit other leading job sites. So if you're not looking on LinkedIn, you're looking in the wrong place. Hire professionals like a professional and post your job for free at linkedin.com slash gladwell. That's linkedin.com slash gladwell to post your job for free. Terms and conditions apply.

Do you know who wrote "All of Me" by John Legend? Or "If I Were a Boy" by Beyonce? Or "Big Girl's Song Cry" by Fergie? That's me, Toby Gadd. I'm a songwriter and have a brand new podcast called "Songs You Know" with Grammy-winning guests like "Hosier" producer Jeff Giddey, "Charlie XCX" producer John Shafe, Rihanna and Coldplay producer Stargate, and artists like Jessie J, Josh Groban, and Victoria Justice. We're talking about their lives, their songs, their advice, their tips and tricks, and their most embarrassing moments.

So please tune in at Songs You Know podcast with Toby Gadd. Hey, it's Jacob. There is another Pushkin show called Risky Business. It's a show about decision-making and strategic thinking. It's hosted by Nate Silver and Maria Konnikova. And they just had a really interesting episode about the future of AI. So we're going to play that episode for you right now. We'll be back later this week with a regular episode of What's Your Problem? ♪

Welcome back to Risky Business, a show about making better decisions. I'm Maria Konnikova. And I'm Nate Silver. Nate, today on the show is going to be a little bit doomtastic. Yeah, I mean, I don't know if it's like worse than thinking about like the global economy going into a recession because of dumbfuck tariff policies. This is all about how we're all going to die in seven years instead. No, I'm just kidding. This is a very...

very intelligent and well-written and thoughtful report called AI 2027 that we're going to spend the whole show on because I think it's such an interesting subject to talk about, but that includes some dystopian possibilities, I would say. It does indeed. So let's get into it and hope that you guys are all still here to listen to us in seven years. ♪

The contrast is interesting between like all the chaos we're seeing with tariff policy in terms of starting a trade war with China and then other types of chaos. It's interesting to kind of look at this. I mean, I wouldn't call it a more optimistic future exactly, but like on a different trajectory of like a future that's going to change

very fast, according to these authors, with profound implications for, you know, everything, the human species. These researchers and authors are saying that everything is going to change profoundly. And even though there is some hedging here, this is kind of their base case scenario. And like,

So base case number one and base case number two differ. There's kind of a choose-your-own-adventure at some point in this report. But they're both very different in the status quo, right? And the notion that we hear from AI researchers is that everything becomes different if AIs become –

substantially more intelligent than human beings. People can debate, and we will debate on this program, what that means. But yeah, do you want to contextualize this, Marie? Do you want to tell people who the authors are of this report? Absolutely. Absolutely. So the report is authored officially by five people, but I think unofficially there's also a sixth. We've got

Eli Leiflund, who's a super forecaster, and he was ranked first on RAND's forecasting initiative. So he is someone who is very good at kind of looking at the future, trying to predict what's going to happen. You have Jonas Vollmer, who is a VC at Macroscopic Ventures. Thomas Larson was a former executive director of the Center for AI Policy. So that is a center that advises AI.

sides of the aisle on how AI is going to go. And Romeo Dean, who is part of Harvard's AI safety student team. So someone who is still a student, still learning, but kind of the next generation of people looking at AI. And finally, we have Daniel Coctello, who basically had written a report back in 2021. He was a

AI researcher, and he looked at predictions for AI for 2026, and it turns out that his predictions were pretty spot on. And so OpenAI actually hired Daniel as a result of this report. Previously. He's now left and complained about it. Yes, and then he left. Exactly. And importantly...

There's also Scott Alexander. Exactly. So I was saying he's the person who kind of is in the background, and you guys might know him as the author behind Astral Codex. And I know Scott. He's one of the kind of fathers of...

what you might call rationalism. I think Scott, when I interviewed him for my book, was happy enough with that term and accused me or co-opted me into also being a rationalist. These people are somewhat adjacent to the effective altruist, but not quite, right? They're just trying to apply a sort of thoughtful, rigorous, quantitative lens to big picture problems, including existential risk, of which most people in this community believe that

AI is both an existential risk and also kind of an existential opportunity, right? That it could transform things. You talk to Sam Alp and he'll say, we're going to cure cancer and eliminate poverty and whatever else, right? And Scott's also an excellent writer. And so let me disclose something which is slightly important here. So I actually was approached by some of the authors of this report,

A couple of months ago, I guess it was in February-ish, just to give feedback and chat with them. So I'm working off the draft version, right, which I do not believe they changed very much. So my notes pertain to an earlier draft. I did not have time this morning to go back and reread it.

So I was not on the inside loop, so I did not get an earlier draft. And I've read this draft. And basically, just to kind of big picture sum it up, it outlines two scenarios, right? Two major scenarios for how AI might change the world as soon as 2030. Now, important note like that.

That date is kind of hedged. It might be sooner. It might be later. There's a confidence interval there. But the two different scenarios, one, basically...

He doomed 2030, humanity disappears and is taken over by AI. The positive report is in 2030, basically we get AIs that are aligned to our interests and we get kind of this AI utopia where AIs actually help make life much better for everyone and make the standard of living much higher. But the crucial turning point is before 2030. And the crucial kind of question at the center of this is,

Will we be able to design AIs that are truly aligned to human interests rather than just appear to be aligned and kind of lying to us while actually following their own agenda?

And how we handle that is kind of the linchpin. And it's actually interesting, Nate, that you started out with China because a lot of the policy choices and a lot of what they see as kind of the decision points that will affect the future of humanity actually change.

hinge on the U.S.-China dynamic, how they compete with each other, and how that sometimes might basically clash against safety concerns because no one wants to be left behind. Can we manage that effectively? And can kind of that transition work in our favor as opposed to against us? I think that this is kind of one of the big questions here. And so it's funny that we're seeing all of this trade war right now as this report is coming out.

Yeah, look, I think this exercise is partly just a forecasting exercise. I mean, you know, obviously, there's this kind of like fork at the bottom where we learn to have an AI slowdown or we kind of are pressing fully on the accelerator, right? Like in some ways, scenarios are like not predictable.

that different, right? Either one assumes remarkable rates of technological growth that I think even AI, I'm never quite sure who to call an optimist or a pessimist, right? Even AI believers, you know, might think is a little bit aggressive, right? But what they want to do is they want to have like a specific fleshed out scenario for how the world would look like. It's kind of like a modal scenario. And like, I think they'd say that like,

We're not totally sure about either of these necessarily, right? I don't think they'd be as pedantic as to say, if you do X, Y, and Z, then we'll save the world and have utopia. And if you don't, then we'll all die, right? I think they'd probably say it's unclear and there's kind of risk either way. And we wanted to go through the scenario of fleshing out what the world might look like, right? I do think one thing that's important is that whatever decisions are made now, we're

could get locked in, right? That you pass certain points of no return and it becomes very hard to decelerate like an arms race. This is, you know, what we found during the Cold War, for example. I mean, one of the big things I look at is like,

Do we force the AI to be transparent in its thinking with humans, right? Like now there's been a movement toward the AI will actually explicate its thinking more. I'll ask it a query, open AI. The Chinese models do this too, right? And they'll say, I am thinking about X, Y, and Z, and I'm looking up P, D, and Q, and now I'm reconsidering this. It actually has this chain of thought process, right? Which is explicated in English. You know, one concern is that what if the AI just kind of communicates to one another that

in these implicit vectors that's inferring from all the text it has. It's kind of unintelligible to human beings.

beings, right? Maybe kind of quote unquote thinking in that way in the first place and then does us the favor of like translating back so it goes from English to kind of this big bag of numbers as one AI researcher called it, right? And then it translates it back into English or whatever language you want really in the end. What if it just cuts out that last step, right? Then we can't kind of like

what AI is doing, then it can behave deceptively more easily. So, you know, so that part seems to be important. I want to hear your first impressions before I kind of poison the well too much. Well, my first impressions is that the alignment problem is a very real one and an incredibly important one to solve. And what I got from this is that actually the problem that I've had with like these

Initial AI LLMs is the kernel of what they're seeing there. So you and I have talked about this on the show in the past, and I've said, well, my problem is that when I'm a domain expert, I start seeing some inaccuracies and I start seeing places where it either...

just didn't do well or made shit up or whatever it is. Now, I think it's very clear that those problems are going to go away, right? That that is going to get much, much better. However, the kernel of it's showing me something, but that might just be, you know, I have no way of verifying if that's what's going on, what it's reading, like how it's

I don't want to say thinking about it, even though in the report they do use thinking, but it's... No, no, no, no, no, no. I hate anthropomorphizing too much. I think thinking is correct. I think it's...

Okay. We'll stick to that language. Yeah. Okay. So how it's thinking about it, that those little problems and like the little, the glitches and the things that it might be doing where it starts actually glitching on purpose are not going to be visible to the human eye. And so one of the main things that they say here is that as AI internal R&D is

gets rapidly faster. So that means basically AI is researching AIs, right? And so internally, they start developing new models. And as they kind of surpass human ability to monitor it, it becomes progressively more difficult to figure out, okay, is the AI actually doing what I want it to do? Is the output that it's giving me

its actual thought process? And is it accurate or is it like trying to deceive me, but it's actually kind of inserting certain things on purpose because it has different goals, right? Because it is actually secretly misaligned, but it's very good at persuading me that it's aligned. Because one of the things that actually came out of this report, and I was like, huh, you know, this is interesting, is if we get this

remarkable improvement in AI, it will also remarkably improve at persuading us, right? Yeah. So this is one part I don't buy. Yeah. So this is... I'd never even thought about that. I was like, okay, fine. But one of the things that I do buy is that it's going to be very difficult for us to monitor it and to figure out, like, is it truly aligned with human wants, with human desires, with human goals? And

the experts who are capable of doing that, I think are actually going to dwindle, right? As AI starts proliferating in society. And so to me, that is something that is actually quite worrisome. And that is something that we really need to be paying attention to. Now, just to fast forward a little bit,

In their doomsday scenario, in 2030, when AI takes over, it basically suddenly releases some chemical agents and humanity dies, and the rest of the stragglers are taken care of by drones, etc. I don't even...

It's a quick and painless death, I will say on the positive side. Let's hope. We don't know what the chemical agents are. It might not be quick and painless. Some chemical agents are actually a very painful death. So let's hope. Let's hope it's quick and painless. Quick, at least. Yes, quick. Okay, hopefully. Some chemical agents are not quick.

Let's hope it's quick and painless. But if they're actually capable of deception at that high level, then you technically don't even need them to do it. If we're trusting medicine and all sorts of things to the AIs, it's pretty easy for it to actually manipulate something and actually insert something into codes, etc., that will fuck up humanity in a way that we can't actually do.

figure out at the moment, right? Like the way I think of it, and this is not from the, this is not from the paper, but this is just the way that like my mind processed it, is like, think about DNA, right? Like you have these remarkably complex, huge strands of data. And as we've

found out, but it's taken forever, one tiny mutation can actually be fatal, right? But you can't spot that mutation. Sometimes that mutation isn't fatal immediately, but will only manifest at a certain point in time. That's the way that my mind tried to kind of

try to conceptualize what this actually means. And so I think that, you know, that would be easy for a deceptive AI to do. And to me, like that, that's kind of the big takeaway from this report is that we need to make sure that we are building AIs that will not deceive, right? That they're

capabilities, they explain them in an honest way, and that honesty and trust is actually prioritized over other things, even though it might slow down research, it might slow down other things, but that that kind of alignment step is absolutely crucial at the beginning, because otherwise, we're

Humans are human, right? They're easily manipulated. And we often trust that computers are quote unquote rational because they're computers, but they're not. They have their own inputs. They have their own weights. They have their own values. And that could just lead us down a dark path. Yeah. So let me follow up with this pushback, I guess, right? Like, first of all, I don't know that humans are so easily persuaded. This is my big critique with like all the

misinformation. People who say, well, misinformation is the biggest problem that society faces. It's like people are actually pretty stubborn and they're kind of sound pretentious. They're kind of Bayesian in how they formulate their beliefs, right? They have some notion of reality. They're looking at the credibility of the person who is telling them these remarks. If it's an unpersuasive source, it might make them less likely to believe they're balancing whatever information with their lived experience, so-called, right? And

You know, part of the reason that like I am skeptical of AIs being super persuasive is like, you know that it's an AI, you know, it's trying to persuade you. You know what I mean? So like if you go and play poker against like a really chatty player, Phil Hemmuth or Scott Siever or someone like that, right? You know on some level that the best play is just

to totally ignore it, right? You know that they are trying to sweet talk you into doing exactly what they want you to do. And so the best play is to disengage or literally you can randomize your moves if you have some notion of what the game theoretical optimal play might be, right? Or salesmen or politicians have reputations for being, oh, he's a little too smooth. Gavin Newsom,

A little too fucking smooth, right? I don't find Gavin Newsom persuasive at all, right? He's a little too, from the hair gel to the constantly shifting vibes. I mean, I don't really find Gavin Newsom persuasive at all, even though like in AM, it's like, boy, Gavin Newsom's a good looking guy. He's a little gravelly throated, but you know, whatever. I mean, look, the big critique I have of this project, and by the way, I think this is an amazing project.

project, in addition to wonderful writing, if you view it on the web, not your phone, so these very cool little infographic that updates everything from the market value. They don't call it open AI. They call it open brain, I guess is what they settled on for a substitute, right? Yeah, they call everything something else just to make sure that they're not stepping on any toes. So they have open brain, and then they have deep scent from China. I wonder which one that could be. I wonder. Yeah.

But it's beautifully presented and written. And like, I appreciate they're going out on a limb here. You know, I mean, I think they have. It's been fairly well received. They've gotten some pushback, both from inside and I think outside the AI safety community. Right. But they're putting their necks on the line here. They will look if we if if things look pretty normal, if we're looking pretty normal in 2032 or whatever. Right. And they will look dumb for having published this.

Well, and they actually have that right as a scenario that you end up looking stupid if everything goes well, but that's okay. Now, can I push back on the persuasion thing a little bit? Just on two things. So first of all, the poker example is not actually a particularly applicable one here because you know that you're playing poker and you know that someone is trying to get information and deceive you.

The tricky thing, so this is kind of when I spent time with con artists, the best con artists aren't Gavin Newsom. Like they're not car salesmen. You have no idea they're trying to persuade you to do something. They are just like nice, affable people who are incredibly charismatic. And even in the poker community, by the way, like some of the biggest grifters who like it comes out,

later on we're just like stealing money and doing all of these things are charming, right? They're not sleazy looking. Like they have no signs of, oh, I'm a salesman. I'm trying to sell you something. The people who are actually good at persuasion, you do not realize you're being persuaded. And I think people are incredibly easy to kind of

You subtly lead in a certain direction if you know how to do it. And I think AIs could do that. And they might persuade you when you don't even think they're trying to persuade you. You might just ask, like, can you please summarize this research report? And the way that it frames it, right, the way that it summarizes it just subtly changes

changes the way that you think of this issue. We see that in psych studies all the time, by the way, where you have different articles presented in slightly different orders, slightly different ways, and people from the same political beliefs, you know, same starting point, come away with different ideas

of what kind of the right course of action is or what this is actually trying to tell you because the way the information is presented actually influences how you think about it. It's very, very easy to do subtle manipulations like that. And if we're relying on AI on a large scale for a lot of our lives, I think that if it has like a quote-unquote master plan, you know, the way that they present in this report, then persuasion in that sense is actually going to be

But you'll know you're being manipulated, right? That's the issue. No, you don't. No, that's the thing. You don't know you're being manipulated. But people will adjust and adapt because it's AI. I know, but...

I don't know. I honestly, like Nate, I applaud your belief in humans' ability to adjust to this, but I don't know that they will because I've just seen enough people who are incredibly intelligent fall for cons and then be very unpersuadable that they have been conned, right? Instead doubling down and saying, no, I have not. So humans are stubborn, but they're also stubborn in saying, I have not been deceived. I have not been manipulated when in fact,

they have to protect their ego and to protect their view of themselves as people who are not capable of being manipulated or deceived. And I think that that is incredibly powerful. And I think that that's going to push against your optimism. I hope you're right. But from what I know, I don't think you are. I'm not quite sure I call it optimism. So I guess maybe we do like different views of human nature. But like, there's not yet a substantial market for like AI driven art

And I'm sure there will be one eventually. Right. But like people understand that context matters. Right. That you could have a creator rip off the Mona Lisa, but you can also buy rip off the Mona Lisa on Canal Street for five bucks. Right. And like, you know, so it's the intentionality of the act and the context of the speaker. Now I sound like superstitious.

super woke, I guess, right? Where you're coming from. I think that actually is how humans communicate. Like art that might be pointless dribble coming from somebody can be something different coming from a Jackson Pollock or whatever, you know? Absolutely. I think that that's a really important point, by the way. I think it's a different point, but I think that that is a very important point. I think context does matter. We'll be right back after this message.

Are you a trailblazer? A risk taker? Someone with countless tales of epic adventure? Well, I'm not quite there yet, but I'm working on it. Even the boldest among us started off small, daring themselves to reach greater goals each day.

If you're looking to take on a challenge like that, the Defender is too. It's a vehicle built for drivers capable of great things, whether they're headed towards uncharted territory or just a weekend getaway. And this may meet other adventure seekers at Destination Defender. This incredible weekend festival in Port Jervis, New York, was created for those who embrace the impossible. It includes outdoor activities, live music, chef tastings, and more. The Defender is a vehicle that, like you, is capable of great things.

It was engineered to meet challenges head-on so you can explore with confidence. Created with durable, lightweight monocoque architecture for extra strength. Explore the full Defender lineup at LandRoverUSA.com. The last thing you want to do when you're sick is go to the pharmacy to pick up a prescription. Because then you're standing in a long line with a whole bunch of sick people. And everyone is sick of being sick around other people who are sick.

Amazon Pharmacy will deliver right to you, fast, so you can get meds without congregating among the contagious. Healthcare just got less painful. Amazon Pharmacy.

Together, T-Mobile for Business and industry leaders are innovating with our advanced 5G solutions. For Walt Disney Studios, we transformed movie making by syncing teams in California with a remote production hub in Hawaii, enabling picture-perfect collaboration to help bring Lilo & Stitch to theaters this summer. For PGA of America, we deliver pro-level efficiency with connected security and ticketless entry for smoother operations, seamless transactions,

and better fan experiences from gate to green. And for tractor supply, we put 5G business internet to work across 2,200 stores, cultivating AI-driven customer experiences to keep things running seamlessly inside, curbside, and countryside. We're helping industries redefine what's possible because with a partner that's as committed to your business as you are, there are no limits.

Discover how our advanced 5G solutions can take your business further at T-Mobile.com slash now. I was buttering up this report before. My big critique of it is like, where are the human beings in this? Or put another way, kind of like, where is the politics, right? They're trying not to use any remotely controversial real names, right? So you have open brain, for example. So where is politics?

President Trump. Let me just do a quick search to make sure their name Trump does not appear. No, the name Trump. So they do actually, I don't know if this existed. Maybe they took your criticism in this, but they do have like the vice president and the president. Like they do put politicians in this version of their report. They don't have names, but they say the vice president in one of these scenarios, you know, handily wins the election of 2028. We have one vice president. They have general secretary, I think. I'm not sure if they, I mean. General secretary. General secretary,

She resembles she. Yep. And the vice president kind of resembles J.D. Vance. Right. I don't think the president resembles Trump at all. Right. It's kind of. No, they don't. Character. And if they tried to sidestep that as much as possible. If I think it's all happening in the next four years, then, you know, presidential politics matter more.

Quite a bit. I mean, I don't know. I was jogging earlier on the east side and I was listening to the Ezra Klein interview with Thomas Friedman. It's such a fucking yuppie fucking thing, right? It's okay. It's okay. You're allowed to be a yuppie. Sometimes you just got to embrace it. Not a huge fan of necessarily, but like, you know, is well-versed on like geopolitics and like China issues. Like, yeah, China, he's just been back from China. He's like, yeah.

China's kind of winning, you know what I mean? And like, I'm not sure how how Trump's hawkishness on China, but like kind of imbecilically executed hawkishness on China. Like, I'm not sure how that figures in to this. Right. If we are reducing U.S.-China trade, that probably does produce hawkishness.

An AI slowdown? Maybe more for us if they're not exporting their raw earth materials and so forth, but we're making it harder for them to get NVIDIA chips, so they probably have lots of workarounds.

And things like that. Maybe Trump tariffs are good. I'd like to ask the authors of this report because it means that we're going to, like, have slower AI progress. I'm not joking, right? On the other hand, it, like, increases the hostility between the U.S. and China in the long run, right? I mean, if we rescinded all the tariffs tomorrow, I think we would still permanently –

or let's not say permanent, let's say at least for a decade or so, have injured U.S. standing in the world. And so I don't know how that figures in. And I'm like, I'm also not sure, like, kind of quote what the rational response might be. But one thing they tracked, let me make sure that they kept this into their report, right? So they actually have their implied approval rating for how people feel about open brain, which is their...

Not very self-proxy for OpenAI. I think this actually is some feedback that they took into account, right? They originally had it slightly less negative, but they have this being persistently negative and then getting more negative over time. It was a little softer in the previous version that I saw. So they did change that one thing at some stage. But the fact that AI scares people, it scares people for...

Both good and bad reasons, but I think mostly for valid reasons, right? That the fear is fairly bipartisan, that the biggest AI accelerators are now these kind of Republican techno-optimists who are not looking particularly wise given how it's going with the first 90 days, whatever we are, of the Trump administration. And the likelihood of like a substantial...

political backlash, right? Which could lead to dumb types of regulations. But like, you know, part of it too is like, okay, AI, they're saying, can do not just computer desk jobs, but like all types of things, right? And like humans kind of play this role initially as supervisors. And then literally within a couple of years, people start to say, you know what, am I really adding much...

much value here, right? You kind of have like these legacy jobs and there is a lot of money. I think most of these scenarios imagine very fast economic growth

Although maybe very lumpy, right? For some parts of the world and not others. But we're kind of just sitting around with a lot of idle time. It might be good for live poker, Maria, right? All of a sudden, all these smart people, their open earth or open brain, excuse me, stock is now worth billions of dollars, right? And like, it's nothing to do because the AI is doing all their work, right? They have a lot of fucking time to play some fucking Texas Hold'em, right? Yeah.

That is one way of thinking about it. Let's go back to your earlier point, which I actually think is an important one, because obviously they were trying to do, as all super forecasting tries to do, is you try to create a report that will work in multiple scenarios, right? You can't tie it too much to the present moment, otherwise your forecasts are going to be quite biased.

However, I do think that what you raise, kind of our current situation with China, et cetera, has very real implications given that this is kind of the central dynamic of this report that their predictions are based on. I think that it's incredibly valid to actually speculate, you know, how will, if at all,

This affect the timeline of the predictions, the possible, the likelihood of the two scenarios. And I will also say that one of the things in the report is that all of these negotiations on like, will we slow down? Will we not? How aligned is it? This all takes place in secret, right? Like we don't know. We, the humans, don't know that it's going on. We don't know what's happening behind the scenes. And we don't know what the decision makers are kind of thinking. And so for all we know, you know, President Trump is meeting with Sam Altman and

trying to do some of these things. And it's funny because we were kind of pushing for transparency in one way, but there's a lot of things here that are very much not transparent. Yeah, it's kind of the deep state, right? But also...

Also, a lot of the negotiations are now AI versus AI, right? And look, I'm not sure that AIs will have that trust both with the external actor and internally. I'm skeptical of that, right? If that does happen, they kind of think this might be good because the AIs will probably behave in like a literally game theory mindset.

optimal way, right? And understand these things and make, I guess, like fewer mistakes than humans might. If they're properly aligned, like that's a crucial thing because in the doomsday scenario, AI negotiates with AI, but they conspire to destroy humanity, right? So there are two scenarios. One, it's actually properly aligned. So AI negotiates with AI, game theory works out and we end up with, you know,

democracy and wonderful things. But in the other one where they're misaligned, AI negotiates with AI to create a new AI basically and destroy humanity. So it can go one way or the other depending on that alignment step, first of all. I mean, the utopia didn't seem that utopian to me, right? I'm not sure. No, it didn't. It actually seemed quite dystopian to me. Like it seemed incredibly dystopian. But at least we're still alive. We'll probably live longer. And again,

Lots of poker. The AI, all you're writing is Silver Bulletin and hosting our podcast, right? Let me back up a little bit because I think we maybe take for granted that some of these premises are kind of controversial, right? So they have a breakpoint, I think, in 2026. Well, it says, why aren't certain increases substantially beyond 2026, right? So that's kind of the breakpoint. It's like 2027 is this inflection point.

I think I'm using that term correctly in this context. You know, so I'm reading this report and up to 2026 and like thumbs up. Yeah, this seems like very smart and detailed about like, you know, how the economy is reacting and how politics are reacting in the race dynamic with China. Maybe there needs to be a little bit more Trump in there. I understand why politically they didn't want to get into that mess. Right. But like, so there's.

Kind of three different things here, right? One is a notion of what's sometimes called AGI or artificial general intelligence. And if you asked...

A hundred different researchers, you get a hundred different definitions of what AGI is. But, you know, I think it is based like being able to do a large majority of things that a human being could do competently. Assuming we're limiting it to kind of like desk job type tasks, right? Anything that can be done remotely is sometimes definition that is used or through remote work, right? Because clearly AIs are inferior to humans and like...

sorting and folding laundry or things like that, right? That requires a certain type of intelligence, right? If you use the kind of desk job definition, then like AI is already pretty close to AGI, right? I use large language models all the freaking time and they're not perfect for everything. But like, you know, in terms of like being able to do the large majority of desk work at levels ranging from complex

competent intern to super genius, like on average, it's probably pretty close to being generally intelligent by that definition, right? If you're the one using it, I just want to like once again point that out. Because one of the things that they say in the report is that as it gets more and more involved, what we're asking AI to do, it's like the human process to evaluate whether it's accurate and whether it's making mistakes will get longer and longer. And I think they say like for every like

basically one day of work. It'll take several, it's like a two to one ratio at the beginning for how long it will take humans to verify the output, right? So you think like, you think you save time by having AI do this, but if you want it to actually develop correctly, then you need a team and it takes them twice as long to verify that what the AI did is actually true and actually valid and actually aligned, et cetera, et cetera. Now you're not asking it to do things that

require that amount of time. But there do need to be little caveats to how we think about their usefulness and how you are able to evaluate the output versus other scenarios. When I use AI, the things that it's best with are things that save me time, right? Where I feed it a bunch of different names for different college basketball teams. We work in our NCAA model. I'm like, take these seven different naming conventions. They're all different. And

create a cross-reference table of these, which is kind of like a hard task, right? You need to have a little context about basketball and it did that very well, right? That's something I could have done. It might have taken an hour or two, but, you know, instead I could do it in a few minutes. It gets faster. It's like, oh, I've learned from this, from you before, Nate, so now I can be faster doing this type of task in the future. I was at the poker tournament down in Florida last week and, like, uh,

You know, I ask open research. Excuse me. God, is that? See, exactly. Right? It all becomes. Deep research. Deep research. Deep research. I ask deep research. Reading too many of these 2027 reports, your brain gets all over. To pull a bunch of stock market data for me, and then I'm playing a poker hand and, like, I make a. . .

really thin, sexy value bet with like fourth pair. No one knows that means, right? I bet a very weak hand. I thought the other guy would call with an even weaker hand. And I was right. And I feel like I'm such a fucking stud here, value betting fourth pair. And well, AI does the work for me. And then of course I like bust out of the tournament an hour later. And meanwhile, you know, deep research bungles this particular task, but in general, AI has been very reliable. But the point is like, there's like a inflection point where like, I'm asking you to do things that like,

just a faster version of what I could do myself. I would, at the moment, ask AI, be like, I want you to design a new NCA model for me with these parameters, because I wouldn't know how to test it. But anyway, I'm being long-winded here. So AGI, we're going to get AGI, or at least we're going to get someone calling something AGI soon, right? Artificial superintelligence, where it's doing things much better than human beings, I think this report

takes for granted or not takes for granted it has lots of documentation about its assumptions but it's saying okay this trajectory has been very robust so far and people make all types of bullshit predictions so the fact that these guys in particular have made accurate predictions of the past is certainly worth something i think right but they're like okay you kind of follow the scaling law and um

And before too much longer, you know, AI starts to be more intelligent than human beings. You can debate what intelligent means if you want, but do superhuman types of things and or do them very fast, which I think might be different, right? And AI can do things very fast. Maybe it's certainly a component of intelligence, right? But I don't take for granted that, like, quote unquote, AI can reliably extrapolate

beyond the data set. I just think that it's not an absurd leap of logic. It may even be the base case or close to the base case, but it's not assumable from first principles, I don't think. We've all seen lots of trend charts. If you look at a chart of Japan's GDP in the 1980s, you might have said, okay, well, Japan's going to take over the world and be in people bought this.

And now it's kind of hasn't grown for like 40 years, basically. Right. And so like we've all seen lots of curbs that go up and then it's actually an ester where the fuck you call it, where like it begins to bend the other way at some point. And we can't tell until later. Right. The other thing is like the ability of AI to plan and manipulate the physical world. I mean, some of these things where they're talking about like

You know, brain uploading and Dyson swarms and nanobots like, you know, there I would literally wager money against this happening on the timescales that they're talking about. Right. They double the timescale. OK, then I might start to give some more probability. And look, I'm willing to be wrong about that. I guess we'll all be fine.

dead anyway, 50% likelihood in this scenario. In this scenario, 50% P.D. But like AA, you know, the physical world requires sensory input and lots of parts of our brain that AI is not as effective at manipulating. It also requires like being given or commandeering resources somehow. By the way, this is like a little bit of a problem for the United States. I mean, we are behind China by like

quite a bit in robotics and related things, right? So I don't know what happens if we have the brainier, smarter AIs, but they're very good at manufacturing and machinery. So what if we have the brains and they have the

brawn so to speak right and they have maybe a more authoritarian but functional infrastructure so i don't know what happens then right like but the ability of ai is to command your resources to control the physical world to me seems far-fetched on these timelines in part because of

politics, right? I mean, the fact that it takes so long to build a new building in the U.S. or a new subway station or a new highway, and the fact that our politics is kind of sclerotic, right? And look, I mean, I don't want to sound like

too pessimistic, but if you read the book I recommended last week, Fight, I mean, we basically did not have a fully competent president for the last two years. And I would argue that we don't have one for the next four years, right? So like all these kind of things that we have to plan for, like who's doing that fucking planning? Our government's kind of dysfunctional and, you know, maybe that means we just lose to China, right? Maybe that means we lose to China. At least we'll have like nice cars, I guess. We'll be back right after this.

Are you a trailblazer? A risk taker? Someone with countless tales of epic adventure? Well, I'm not quite there yet, but I'm working on it. Even the boldest among us started off small, daring themselves to reach greater goals each day.

If you're looking to take on a challenge like that, the Defender is too. It's a vehicle built for drivers capable of great things, whether they're headed towards uncharted territory or just a weekend getaway. And this may meet other adventure seekers at Destination Defender. This incredible weekend festival in Port Jervis, New York, was created for those who embrace the impossible. It includes outdoor activities, live music, chef tastings, and more. The Defender is a vehicle that, like you, is capable of great things.

It was engineered to meet challenges head-on so you can explore with confidence. Created with durable, lightweight monocoque architecture for extra strength. Explore the full Defender lineup at LandRoverUSA.com. The last thing you want to do when you're sick is go to the pharmacy to pick up a prescription. Because then you're standing in a long line with a whole bunch of sick people. And everyone is sick of being sick. Around other people who are sick.

Amazon Pharmacy will deliver right to you, fast, so you can get meds without congregating among the contagious. Healthcare just got less painful. Amazon Pharmacy.

Together, T-Mobile for Business and industry leaders are innovating with our advanced 5G solutions. For Walt Disney Studios, we transformed movie making by syncing teams in California with a remote production hub in Hawaii, enabling picture-perfect collaboration to help bring Lilo and Stitch to theaters this summer. For PGA of America, we deliver pro-level efficiency with connected security and ticketless entry for smoother operations, seamless transactions, and

and better fan experiences from gate to green. And for tractor supply, we put 5G business internet to work across 2,200 stores, cultivating AI-driven customer experiences to keep things running seamlessly inside, curbside, and countryside. We're helping industries redefine what's possible because with a partner that's as committed to your business as you are, there are no limits.

Discover how our advanced 5G solutions can take your business further at T-Mobile.com slash now.

I think that your point about the growth trajectories not necessarily being reliable is a very valid one. The Japan example is great. Malthusian population growth is another big one, right? We thought that population would explode and instead we're actually seeing population decline. So, you know, the world does change. The thing that I think they rely on is that

that AIs are capable of designing just this incredible technology much more quickly so that our building process and all of that gets sped up hundredfold from what it is right now. But it still, at least at this point, needs humans to implement it and needs all of these different workers. And so, yeah, I think there are some assumptions built into here that I hope that that timeline isn't feasible. And I do think that there are things that are holding us back.

All the same, I think it's interesting. One of the reasons I like this report is that it forces you to think about these things and try to game out some of these worst-case scenarios to try to prevent them, which I think is always an important thought exercise. I do want to go back to their good scenario. Bad scenario is we're all wiped out by...

by chemical warfare that the AIs release on us. Good scenario is that, you know, everyone gets a universal basic income and AI does everything and no one has to do anything and we can just live, you know, happily with everything.

Play poker, yeah. And that just, as you suggested, that seems actually like a very dystopian scenario where people can become much more easy to brainwash, control, et cetera, et cetera. It's like a dumbing down, right, where we're not challenged to produce results.

good art to advance in any sort of way. Just to me, it does not seem like a very meaningful way of life. Well, I mean, the question is, there's another way, another, if you read AI 2027, which I highly recommend that you read it, there's also another post by a pseudonymous poster called L. Rudolph L., who wrote something called A History of the Future 2025, which

to 2040, um, which is very detailed, but goes through kind of like what this looks like at like more of a human level, how society evolves, um,

how the economy evolves, how work evolves, right? And like very detailed, just like AI 2027 is, but kind of focused on the parts of AI 2027 that I think it kind of deliberately ignores, maybe you can call them mild blind spots or whatever, right? But like, but that's interesting because that kind of thinks about like, what types of jobs are there in the future? There are probably lots of layers actually, right? Because, you know, the law is very sluggish to change, especially in a constitutional system where there are lots of

veto points, right? Probably high-end service sector, you know, you go to a restaurant because everyone's, or a lot of people are rich now, right? And you're flattered by the attractive young server and things like that. So it's kind of highly kind of like catered and curated experiences. I guess I have some faith in humanity's ability to fight back, quote unquote, against like

Two scenarios that it might not really like either one. You know what I mean? And like the scenario where like AI is producing like 10% GDP growth or whatever, right? I mean, it's great if you own stocks that are exposed to AI and tech companies probably, right? But it's also making that money on the backs of mass producers

And, like, economists are confident in the long run that human beings find productive things to do and, you know, mass unemployment has been predicted many times and never really occurred, right? But it's not occurring this fast where they think the world ends in six years or whatever they're predicting or we have utopia in six years. And, like, just the ability of, like, human society to, like,

deal with that change at these timescales leads to like more chaos than I think they're more predicting. But I think also, and I told them this too, right? I think also leads to more constraints, right? That you hit bottlenecks. If you have,

five things you have to do right um and you have the world's fastest computer etc etc but there's like a a power outage in your neighborhood right and that's a that's a bottleneck right maybe there are ways around it if you're like go to home depot and and buy a generator or you know what i mean but like but the point is that like you're often defined by like the slowest link and and politics are sometimes the slowest link but also but like also you know i think the report um

maybe understates and I think kind of in general, the AI safety community, like maybe understates the ability of like human beings to cause harm to other human beings with AI, right? That concern kind of gets brushed off as like to, you know,

pedestrian or like, like, yeah, the exact word I was thinking of. I think that's a good, I mean, I think that's a great place to end it because yes, we do need to be concerned about all of these things about AI, but like that, that phrase I think is very crucial. Like do not underestimate the ability of humans to cause harm to other humans. And I think that that's, you know, it's not a very, it's not a very,

pleasant place to end, but I think it's a really important place to end. And I think that that's a very valid kind of way of reflecting on this. Or to trust AIs too much, right? I generally think that concern is like somewhat misplaced, but like if we're handing over critical systems to AI, right, it can cause problems if it's very smart and deceives us and doesn't like us very much. It can also cause problems if it has, um,

hallucinations bugs in critical areas where it isn't as robust and hasn't really been tested yet that are outside of its domain um yep or there could be espionage anyway we will have plenty of time although maybe only seven more years actually to like explore explore these scenarios

Yes. And in seven years, we'll be like, welcome back to the final episode of Risky Business, because the prediction is we're all going to be dead tomorrow. But yeah, this was an interesting exercise. And I think my P. Doom has slightly gone up as a result of reading this, but I also remain optimistic that humans can do good as well as harm. Yeah.

Yeah, my interest in learning Chinese has increased as a result of recent developments. I don't know about my P.D. All right. Yeah. Let's let's do some language immersion. I'm with you.

That's it for today. If you're a premium subscriber, we will be answering a question about whether MFAs can ever be plus EV right after the credits. And if you're not a subscriber, it's not too late. For $6.99 a month, the price of a mid-beer, you get access to all these conversations and all premium content across the Pushkin Network.

Thank you for listening.

Do you know who wrote All of Me by John Legend? Or If I Were a Boy by Beyonce? Or Bickle's Song Cry by Fergie? That's me, Toby Gadd. I'm a songwriter and have a brand new podcast called Songs You Know with Grammy-winning guests like Hosea producer Jeff Giddy, Charlie XCX producer John Schaaf, Rihanna and Coldplay producer Stargate, and artists like Jessie J, Josh Groban, and Victoria Justice. We're talking about their lives, their songs, their advice, their tips and tricks...

and their most embarrassing moments. So please tune in at Songs You Know Podcast with Toby Gadd.

Elon Musk, Doge, and Donald Trump are weaving a web of technological corruption. Suddenly, the eyes of the industry are open to things that had been obvious to lots of other people for months. Isn't it a conflict of interest that the president of the United States who regulates crypto has his own coin? I'm Lizzie O'Leary, the host of What Next TBD, Slate's podcast about tech, power, and the future.

What Next TBD covers the latest on how Silicon Valley is changing our government and our lives. Listen wherever you get your podcasts.