‘We Have to Get It Right’: Gary Marcus On Untamed AI
41:43 Share

Hey everyone, it's Daniel. Before we start, two announcements. Our very own Tristan Harris is going to be in conversation with Yuval Noah Harari the evening of Thursday, October 3rd at the Meyer Theater in Santa Clara to discuss, among other things, Yuval's new book, Nexus. If you're interested, see the link in the show notes. As a special gift to listeners, you can use the code CHT at checkout for $15 off tickets.

And a quick reminder that we want you to send us your questions. Anything you've been wondering about, please tape yourself on your phone and send us the voice memo at undivided at humainetech.com. Thanks. It's a confusing moment in AI. On the one hand, it can feel like we're on a runaway freight train. Companies are pouring billions of dollars into the technology, promising new models every few months that are vastly more powerful than the ones before.

Some experts in Silicon Valley predict that we'll have artificial general intelligence, AI that's smarter and more capable than most humans, in just a few years. The hype is massive, and the promises are huge. On the other hand, the AI that we use today is often full of hallucinations and other unexpected behavior. And those same AI companies have seen their stocks take nosedives in recent weeks.

And you have equally knowledgeable experts predicting that we're about to hit a wall in AI model capabilities, that the bubble will burst, leaving society with a half-usable technology and so many broken promises. One of those people is Gary Marcus. Gary is a cognitive psychologist and computer scientist who's become a leading voice in the public AI debate. He built his own successful AI startup, which he sold to Uber.

And recently, he's argued on his substack and in the press that the AI race we're seeing today is going to slow down considerably by the end of the year. He says we've seen the end of exponential growth in the power of large language models.

But in his new book, Taming Silicon Valley: How We Can Ensure That AI Works for Us, Gary argues that regardless of how quickly the technology progresses, our society is woefully unprepared for the risks that will come from the AI that we already have. And here at CHT, we couldn't agree more. That's why I'm so excited to have Gary on the show today to talk about this head-spinning moment in AI. As you'll hear though, we don't always agree on the details, especially when it comes to questions of where this technology is headed.

We'll explore those disagreements today. So Gary, welcome to your Undivided Attention. Thanks, and that was a really lovely introduction. So I imagine that a lot of listeners are struggling to hold two things in their head at the same time. And I struggle with this too. On one hand, you have people in Silicon Valley who are really worried about the pace of AI development, that it's going to be smarter and more capable than most humans in just a few years.

And then we have other experts, including you, talking about the limitations of the fundamental AI technology, expecting that we're about to hit a wall. Help me with this. How do we square that circle? My view is AI will someday be much smarter than the smartest human. I think that's an inevitability.

But it's not inevitable that it will happen in the next two years or next five years. And I don't think that it will. I think it's at least a decade away and probably longer than that. I come to this from the perspective of cognitive science and how the mind works. And I look at all of the things that a human mind needs to do. And making analogies is one, doing statistical analysis is one. But reasoning is another, for example. It took us...

you know, 70 years of AI research to get to the point where we can really do well a kind of statistical inferencing kind of thing.

But there are other things that we just haven't made that much progress on, like reasoning from incomplete information. And there are ways in which current systems are just really, really bad. One of them is sticking to facts. And you can't be smarter than the smartest human and fail on basic facts. We still don't understand abstract knowledge, reasoning, and things like that. We still don't know how to put those into our AI systems. And we will need fundamental innovation. Some people look at a curve and they think it will follow indefinitely. And sometimes

And jaded people who have been around for a while realize curves don't always go that way. So Moore's law went for a while, but it's not a law of the universe. And eventually it ran out. And in a similar way, it looks like there was genuinely exponential progress between GPT and GPT-2, between GPT-2 and GPT-3, GPT-3 and GPT-4. You could actually fit a curve and say what the exponent was depending on what your measure is.

But that hasn't continued. And some people just sort of want it to continue. And other people say, yeah, but it's been two years since GPT-4. We haven't really seen something that fits with that. And so that looks more like an asymptote where things are starting to be at the point of diminishing returns rather than continuing exponentials.

And the third thing is, my view is the executives all want to project very fast progress. And some of the people in some part of the AI ethics world also want to project it. But if you look at it from a cognitive science perspective, it's just not happening. Let's talk a little bit about where you sit in this AI debate. I think it's sort of fair to say that you're skeptical of a lot of the claims coming out of the big AI companies and their models. And you've sometimes been called AI's loudest critic.

But in your book and in your background, and when we talk, I see a more nuanced picture. I mean, your book, you say, quote, I work on AI every day, and I quarrel with those who wish to take risky shortcuts, not because I want to end AI, but because I want it to live up to its potential. What's your reaction to being called an AI critic? And why do you think you get that label?

That's a complicated question about information warfare. The quote that you just read is something I've also said on Twitter, but people ignore it when I say it on Twitter. So, and you know, probably the reputation as loud as critic primarily comes from Twitter or X.

I have repeatedly said there that I actually love AI. I have sometimes pointed to pieces of AI that I think are really great. But I don't like generative AI. I genuinely don't like generative AI. And for the last five years, that's almost all that anybody has wanted to talk about.

And it's very hard in a place like Twitter to make a nuanced argument. I have certainly tried and said, you know, my target here is generative AI, not all AI. You know, one example of this that just baffles me almost is that the accelerationists all hate me. They all hate me and they don't understand I'm actually trying to accelerate AI, but I just have a different view. Their view is the way that you accelerate AI is to have no regulation on it and to scale things as fast as possible.

My view is that that would be a catastrophe for AI and for society. So if we scale the stuff we have right now, it just makes it more and more dangerous as a weapon of misinformation and cybercrime and so forth. It doesn't solve the truthfulness problems. It's a disaster waiting to happen. And if we have no regulation, then the disaster really will happen. And then there'll be an enormous public backlash against AI. And so if you want...

AI to succeed, you have to realize that there has to be a regulatory environment for it. And it has to be done right. I mean, imagine what would have happened if we had stuck to dirigibles after the Hindenburg crash, because there'd been a bunch of dirigible accelerationists shouting down the other people who'd be more like me saying, "Maybe we should try a different approach. Maybe we should have some regulation. Maybe we shouldn't build them this big unless we know how to put out the fires." And they'd be like, "You bastards, why are you slowing us down?"

And in fact, that would never have worked, right? Eventually the public would have lost heart. In fact, they did lose heart after the Hindenburg. And so we're going to wind up, I think most likely, in fact, with something similar, which is going to lead to a pause on AI because the public gets really sick and tired of seeing what they're seeing. And because the industry itself is not taking, I think, appropriate responsibility. There is a different scenario, I think a more utopian scenario where...

We look at both the moral and technical failings of current AI, of generative AI, and we build something better by improving the technical stuff, which I think requires a different approach than we're taking right now. And by having some kind of regulatory framework, you know, look at airplanes. Like the reason commercial airlines are so safe is because we have multiple layers of regulation. It's very well thought through and so forth. And we're going to need something similar for AI. And if we get to that, then I think there's a chance we will get to the utopia where people

AI helps with science, medicine, agriculture, and all of those kinds of things that we have long talked about. But we have to get it right. You could make an analogy, by the way, to driverless cars. Like if you just said tomorrow, hey, Tesla, just put this stuff on the road. Let anybody use it. Take humans out of the loop. There would be an enormous number of accidents, and there would be congressional investigations, and the whole thing would be shut down.

But generative AI has been a little different. It's just been thrown out there to the entire world and it is in fact causing problems. There is in fact a developing backlash. The patience that we've seen in the driverless car industry has not been there.

The hard part in talking about AI is it's one of those catch-all terms like sports, right? It means so many things to so many people. And there are so many different kinds of AI with different strengths and different weaknesses. Of course, the one that's getting all the attention now is generative AI like ChatGPT and Gemini and Claude. So let's really start there. In your view, the tech industry is putting way too much attention and energy on generative AI. But you point out that it has all of these unique weaknesses. Can you talk a little bit about what those weaknesses are and how they show up?

Yeah, maybe even as level setting as they sometimes say in the business. Let's just be clear. AI is incredibly useful. So, you know, nobody's getting the press that generative AI is, but generative AI is just one small part of AI. You know, we also have, for example, the GPS systems that help us navigate in cities we don't know that work fabulously well and that I'm not complaining about. You know,

It's generative AI in particular that I think is quite flawed. I'll give you some examples of why and so forth. But it's also somehow gotten all of the money and all of the attention. I think it's gotten all the money and all the attention because it's anthropomorphic in a way that others aren't. So people don't really fall in love with their GPS nav systems. There was a little period a few years ago where you could use a voice and people kind of like that. But they don't really fall in love with it.

with the system, they really have, many people, not all, have fallen in love with generative AI. They relate to it as a person. And from a technical perspective, it just isn't reliable. So, you know, you can compare. A calculator is 100% reliable within its limits. GPS system is really like, you know, 98% reliable within its limits. Generative AI promises to answer literally any question about the world.

it almost never says, I don't know, or that's not in my scope. There's a few tiny circumstances where it'll do that. So basically, it promises the moon, but it can't really deliver. On almost any question that you might ask, it might be right, or it might make something up. And it sucks because you actually get different answers on different occasions, and it depends on the exact wording and the exact context.

And so there's no inherent reliability in the system. It's really not very good. It's not what we dreamed of with AI. I mean, you wouldn't want the Star Trek computer to be like 50-50 or ring your dealer to see what you get. That's an old Ali G joke. Well, I hear that. I mean, I really do hear that, right? But

If there is an analogy to generative AI, it is sort of that we've developed machine intuition. And like our intuition, it's inscrutable. You can't look at it and understand it. It sometimes gives you really wrong answers, also dependent on whether you're trained right in the area. And sometimes gorgeous answers, right? I mean, there's no doubt about that. But it is unchecked. The only way that I feel comfortable with it ever is when there's a human in the loop.

Because there has to be someone there to check the intuition. It should never be trusted on its own because it cannot do sanity checks. Well, I want to push on this a little bit because some of the experts that disagree with your analysis of hitting a wall will say things like,

increasing the amount of tool use that this tool uses, better prompting of the LLM to do self-reflection. So when you say something, it looks at its own results and it sees what it comes up with. Different experts and mixtures of experts looking and rewriting each other's work. You fundamentally don't believe that this is going to get us to a more beautiful future either, right? Whereas I think the other experts say that will put the intuition in check. Not in its current form.

it is not going to work in its current form. So the real problem is in the translation. So what you have from a large language model is it gives you an output sentence, and then you want to feed that to your tool. But if the output sentence isn't the right output sentence, your tool is not going to do the right thing. And so if you take a word problem for kids and you feed it in,

You may get it to transform it into the right piece of mathematical stuff, but you might not. It might just come up with the wrong thing entirely. And because it's a black box, it's just a sentence, you don't really know what's there. I actually like to use an analogy here, which is an old joke.

A guy says, I've got a perpetual motion machine. And the other guy says, well, you know, I mean, really? And he says, yeah, I've got the whole thing going. Really? Well, okay, except for one thing. What's the one thing? It's this part that goes back and forth and back and forth and back and forth. And so the point of the joke is like you can think you're really close, but if you don't have the critical element, you've got nothing. I partly agree with what you're saying, and yet I wonder –

there's sort of these two biases. I'm having a hard time adjudicating how stuck I think we are. And, you know, according to you, we're actually quite stuck until we come up with a new architecture. And another part of me says, you begin putting better memory on these models. You begin having other models checking each other's work. You begin having tool use saying, oh, run that down for me. Is that really a thing? Is it not? And then all of a sudden it gets just so much better that even though we can't prove anything about it, it's like I can't prove how Tylenol works or I can't prove that my taxi driver will get me to my destination.

but it ends up working well enough for me to really lean on. So let me see how to put this. At some level, I agree with what you just said, and you agree with part of it. We won't get into all the levels of meta-agreement. But what I'm saying is to do all of those things where you say, well, I will just add an external memory, for example. Let's focus on that one. It turns out that each of these problems is actually a serious research problem that people have worked on for decades on and not made much progress on.

And so I think it can be done, but it is deeply non-trivial to do it. There are, you know, many bodies, you know, along the path. You have a lot of problems like this where people are like, well, you just need an X, you just need a Y, you just need a Z. And those of us who've been in the field for a long time know that those X, Ys and Zs are A, really hard, and B, they're really hard to integrate with massive, uninterpretable black boxes. And

And so somebody is going to find a way through this forest, but it's not like we're, you know, two minutes from the light and the water and like, it's just not like that.

Okay, and maybe just to pull us back out, maybe whatever technical disagreements we might have about how to get there, and I could talk to you about that for hours, but the disagreements may not matter, right? Because I think we agree that there are these real and these present harms coming from this tech now. So let's slow all this down. So can you say more about those specific dangers that generative AI has on society right now? And regardless of how fast this goes, why should we be concerned about the AI that we have right now?

So I actually have a chart in the book of a dozen risks that are immediate. There's also discussion about less immediate risks, and we should certainly talk about them. But immediate risks include things like disinformation for political purposes. That's clearly already a serious problem. This year in the U.S. elections, it's a problem in the German elections and so forth. There's also...

similar problems around stock markets and stuff. And, you know, I warned about that when I spoke to the Senate a year and a half ago. And I was like, hmm, should I mention this one? I don't think it's actually happened yet. A week later, we saw the first evidence of that in the U.S. And now we've seen a bunch of evidence in the Indian stock markets. And you're talking about like pump and dump disinformation within the stock market. So the first one was the picture of the Pentagon being on fire, which drove down the market and

I don't know that that was deliberate as a test pilot of all of this, but it may have been. And certainly people took note. And then, yes, there've been some pump and dump stuff in India. Just a quick note about what we're talking about here.

Anonymous bad actors have been using AI deepfakes to manipulate stock markets, like a recent audio deepfake of the CEO of India's National Stock Exchange telling people which stocks to buy, and a fake photo that circulated recently of an explosion near the Pentagon, which affected the US stock market. This kind of thing is super dangerous, not just because it destabilizes stock markets, but because it funnels money to the people who are using these tools for personal gain.

That's right. It's not hard to imagine, in fact, people taking AI agents and trying to hack the infrastructure in order to cause harm, in order to make money. I will be surprised, in fact, if that doesn't happen in the next five years.

And one thing, this is a bit of a sidebar, but one thing that's really interesting is that bad actors often don't need AI to work as well as good actors. So a good actor who wants to use AI to solve medicine is just going to realize that it's not really there yet. And they're also going to see that the accuracy is a huge problem for them.

I mean, that doesn't mean people shouldn't try it, but it's an issue. Whereas bad actors often don't care. I mean, think about spam. They send something out to a million users. It only convinces like five of them, but that doesn't matter. If they get one in 200,000 people to send them whatever, $1,000, then they're doing really, really well.

And so bad actors often don't care. Take another example. Current AI is very bad at factuality. You mentioned hallucinations. You don't want to write a newspaper with that because people will start to complain. CNET tried it and failed and got pushback. But if you're a bad actor, you don't care. You run these websites, you make money off fake news or you distort the public view of reality, which we've seen an enormous amount, I think, recently on X.

and probably in other outlets, TikTok and so forth. And so the bad actors don't care if like 80% of what they put out is viewed as garbage and people ignore it, as long as 20% of it suits their purposes. For sure, because all you need to do is convince a subset of the population. You don't actually need to cure something. You don't need to find a new solution. It's a lot easier to scare a bunch of people or deceive a bunch of people. You just have to find some suckers. Yeah.

Yeah, and the bar for suckers is lower. So, okay, so there's disinformation, there's misinformation, there's defamation. You know, I'll leave it to the book for people to see all dozen, but the point is there are a lot of risks right now. I'll just mention one other, which is deepfakes, both for their misinformation potential, but also the deepfake porn, the way teenagers are using it against each other as a weapon now is just really tragic. It's horrible. It's horrible.

And speaking about your book, I was really excited to see you open the conversation talking about the parallels with social media. And in fact, we started paying attention to it because we saw the same incentives that played out in social media play out in AI.

Why did you start your book with social media? Because it's a really good example of how the tech industry screwed up and because there are a lot of parallels. So I should say, I've been a gadget head all my life. I've always loved technology. I think social media is one of the places where we really screwed up.

Social media has been an example where we have a powerful technology. It's a dual-use technology like AI in the sense of that technical term, which is it can be used for good and can be used for evil. Social media can be used to organize people for reasonable protests against repressive regimes. And that was one of its earliest uses and why I think

people were positive towards it initially. But it can also be used to addict people. I mean, it's crazy for me to even explain it on your show since I'm sure your listeners know it very well. But just to make this a little bit freestanding, this episode has caused enormous harm presumably to teenagers. It has caused enormous polarization to society. I'm sure you have many episodes detailing all of that.

So that's an example where a dual-use technology probably on balance has done more harm than good and certainly hasn't done anywhere near as much good as one would have liked it to have done. Well, and from our perspective, it wasn't the fault of the technology per se. I mean, the technology, it's just beyond the idea that the technology is a platform that can be used in different ways. It's that...

Really, the shape of the technology was driven by those incentives, by the business models, by all of it. So it wasn't so much a failure of the technology itself. It was a failure of the societal institutions and the incentives that we wrapped around that technology. 100%. So, you know, in principle, we could have wrapped a different set of incentives around it and it might have worked better.

But we got caught up in things about anonymity and rewards for news feeds and things like that. I always think news feed was one of the worst moments in all of this history when suddenly news feed combined with Section 230, which is

a legal thing that basically gives immunity from liability to these companies, was a horrible combination because that's the point at which these companies suddenly really started shaping the news and not having responsibility for it. So coming back to AI...

I see a lot of the same dynamics at work. So I see inadequate regulation. I see the government doing too little. I see the lobbyists being extremely effective and well-funded. There are a lot of parallels at that level. There's a surveillance aspect to social media that we may see even in a deeper way. And I'd like to take a second to talk about that with respect to AI.

We see a possibility of scale causing enormous harm. Just on the surveillance point, it turns out that large language models can do crazy things like instill false beliefs in people. And so the amount of power that the social media company had to kind of shape our thought is maybe even greater with the large language models. Probably is even greater. I mean,

You ran over that so fast, but it's a very deep point. Can we slow down on that one? The idea that interacting with a large language model can instill a false belief. That's right. There was a new study that just showed this. Elizabeth Loftus, who's done most of the most famous work on false beliefs, was an author and it was a collaboration with a group of AI researchers. I'm blanking on who it was. It just came out a couple weeks ago. And they showed that output of large language models

could persuade people that certain things happened that didn't. I mean, it's as simple as that. And the result is very simple, but the implications are profound.

I mean, first of all, the first layer of that is you have software that not by design, but by inherent property hallucinates. So large language models hallucinate. Nobody knows how to solve that. So first layer of this is that you have software that can hallucinate and they can then persuade people that things that didn't happen happened. That is terrifying for software.

democracy and for the fabric of society, right? And you have these things at large scale. The next layer is there's no law about any of this. It's the Wild West. So if the large language model do this, it's not clear that they have any liability. And then there's another layer that's even more frightening, which is that the large language model companies could do this deliberately. So let's say somebody...

has a lot of money, we won't name any names, and wants to build their own large language model, and they have the means to distribute it, and they want a certain bias in it. That bias might put false memories in people. And so the people running these companies

could easily influence the structure of society. Oh, and it doesn't even need to be a mustache-twirling person who wants to inject a political belief. It could be banal capitalist reasons, right? It could be that you're being paid by a shoe company and you ask the LLM some question about how do I get a job and it starts saying, well, you know, first of all, you got to wear the shoes for the job. You got to wear the shoes for the job you want to be in. And like that kind of world with that kind of persuasion, I think it's a really risky world to be in.

Before I went to the Senate, I was told that I should modulate how many times I was a wise ass. I wasn't told that in those words. Still good advice.

I gave Sam Altman a really hard time when he ducked the question of what was he most afraid of. So Senator Blumenthal said, are you most afraid of jobs? And Sam Altman said, no, he wasn't really worried about that problem because there's always been jobs. And I said, Senator Blumenthal, you might wish to get Sam Altman on the record as to what he is most worried about. And that was the point at which he acknowledged that these systems could cause great harm to humanity.

I don't know if I'm allowed to do this, but I will note that Sam's worst fear, I do not think, is employment. And he never told us what his worst fear actually is. And I think it's germane to find out. Thank you. I'm going to ask Mr. Altman if he cares to respond. Yeah. Look, we have tried to be very clear about the magnitude of the risks here. My worst fears are that we cause significant, we, the field, the technology, the industry, cause significant harm to the world. ♪

So then there was a later thing where I passed on my wise-ass moment since I had used it up. But what I wanted to say, because you probably know the famous line when Zuckerberg was asked about his business model and he said, Senator, we sell ads. And so Sam was asked, do we sell ads? And he said, well, we don't right now. I forget exactly his word. But I wanted to say, what he's saying is, Senator, we don't sell ads yet. And in fact, there's a much more...

more pernicious maybe is the word, more subtle thing, which is not just selling ads, but tilting the scale as you just answer queries, which is kind of what you're talking about, right? It doesn't have to be because you're working for the Russian government. It could be just like the temptation to do product placement in large language models is huge. I think people are already playing around with that. I don't know if it's happening in the commercial domain, but I think there's been experimentation.

So the temptation here is going to be enormous. Then you have the side note that all of these things are now essentially available open source, which means anybody can open up shop to do their version of push polling or we need a new word for it, like push prompting. It's not quite the right word, but tilting the scale. We're going to see a lot of that and it may be very effective and yet under the radar so people don't know that it's happening to them.

Yeah, I mean, I'm very scared by that future in terms of the lack of transparency, right? If you have lack of transparency, even on what's happening, on the way that it's being prompted, in how you're moving from a place where it's a query and a response to kind of a relationship, and then you're beginning to have that relationship change.

manipulated for reasons you can't tell. And there's no transparency at any level. So there's no transparency, first of all, on what the systems are trained on. And there's no transparency on what might go into the, for example, reinforcement with human feedback stuff, which also can shift the scale. There's no transparency at any stage. I guess maybe the question I have for you is,

So how much of those harms do you think comes from this sort of semi-broken, unreliable LLM technology per se? And how much is coming from the bad incentives? And the lack of regulation. I think it's three things. So some of them are from the technical limitations. Some are from bad incentives and some are from the lack of government oversight that we could have hoped for and still should hope for. So

So I'll give some examples of each. I'm not sure I can put, seems like a hard question. A good example where the failings of the technology kind of are the problem is disinformation because the systems simply cannot handle truth. They don't know what's true and so they cannot filter what they're saying at all. And that makes them highly vulnerable to misuse. Phishing attacks are another example like this. It's just very easy to craft them to do very deceitful things.

And that's in the nature of the technology. A third thing like that is, you know, people have talked about the risk of AI killing us all and, you know, what if the machines are misaligned with us and so forth. They certainly are misaligned with us. Even if they don't kill us all, they at least...

It's just very hard to specify to a machine of the nature of the machines that we're talking about what it is you actually want them to align on. They don't understand, you know, don't do evil to human beings or don't do harm to human beings and so forth. That is in the nature of the architecture that we have right now that they can't be very well aligned. And that is a very serious, deep problem.

You could imagine regulation that might incentivize people to wrap all three together to build architectures that didn't have those problems. But right now we don't have the regulations, we don't have the incentives. And so people build stuff that is really very difficult to align.

Well, in your book, you have a wonderful set of interventions about how do you mitigate these incentives that I think we agree are driving a whole suite of risks. Can you talk about those interventions you think are the most effective at mitigating these risks? I mean, this is like Sophie's choice, except I have 11 children, right?

There are 11 different recommendations. And the first recommendation is don't expect one recommendation. And it's kind of depressing. It's much easier to say, here's the one thing you should do or shouldn't do. Especially in my position as a guest on a podcast. But the reality is that we are now messing around with a kind of general purpose AI. It doesn't work very well in all the regards that I've talked about, but it can be used in so many different ways to

that it does pose many different risks and it's just not realistic to expect one silver bullet. The closest would be stop building AI, but that's never going to happen. And I'm not sure it should happen because I do think there are positive uses. So the next thing would be keep it in a lab and don't deploy it. And I could understand an argument for that. It's also probably not going to happen.

So once you move past that, you have other interventions. Probably, if I had to prioritize, the number one would be anything that's going to be used at large scale should in fact have pre-deployment checking with some kind of actual teeth if it doesn't meet the pre-deployment check. So we do that for drugs, right? You cannot release a drug if you can't make a case that it is not going to cause massive harm and that it is going to cause good results.

We have a whole procedure with the FDA and you can argue about whether it's perfect and too slow and whatever. But we have a procedure there to try to keep people from being harmed by random charlatans making drugs. And it works reasonably well, I think, if you actually kind of look at what could have filtered in and didn't and so forth. I'm not saying it's perfect, but it works pretty well. We need something like that.

for AI, and especially if somebody's going to deploy something for 100 million customers or something like that. Now, I'm going to assume for the sake of argument that GPT-4 was a small net positive, let's say, to society. We'll just make that up. It doesn't really matter because the point I want to make is, what about the next system? The next system that is significantly different from GPT-4, whenever that comes, is going to have a new set of harms,

possibly more severe harms, who makes that decision? How does that decision get made? I think a reasonable thing to say is that somebody with some knowledge outside the company, some independent oversight should be there and there should be a process. So with drugs, it is a process. So it's not like the FDA says go away, never come back. It's more like come back when you fix this problem.

Maybe to fix this problem, you just need to put a warning label. Don't take this on an empty stomach because it could rot your stomach. And like...

now we're good to go. And sometimes it could be worse. So we do need, in fact, warning labels on large language models. And a remarkably large number of people don't understand that they lie. And the little fine print at the bottom is not sufficient. But probably for the next generation, we're going to need more than that. So if I had one thing, and only one thing, it would be pre-deployment testing. But honestly, we also need auditing tools

We need transparency. One of the things we need transparency around is the data that goes into the systems for all the reasons we talked about earlier. We also need transparency into what internal testing was done. We don't want to be sort of replicating what happened with the Ford Pinto where those guys knew that the gas tanks would explode, but they made a financial decision.

There's nothing stopping OpenAI or any of these companies from making financial decisions, even if they know that there are certain harms for next generation models, from releasing them anyway. And there's nothing so far obligating them even for disclosing what they've done internally. Yeah. And in your book, you talk a lot about how you think there needs to be a new center of power, a new agency to be able to handle those responsibilities. Are you modeling this off of the FDA? Yeah.

So part of what an agency should do is, and let me rewind one sentence, the point of the agency is to be more nimble than Congress can be. And Congress obviously can't be very nimble at all, so that's not a high bar. But the point of an agency is you have people with the expertise and the power to

to move quickly, right? So we have an agency for health, education and welfare, for defense and so forth. Like, you know, you could imagine a world where we don't have a defense department and we just kind of like distribute the responsibility around and the health, education and welfare department has to kind of like do the work of the defense department. Like that would be ludicrous, right? You could imagine doing it that way, right?

Or you could imagine like having no defense department and just like the president sits around in the Oval Office and says to his or her buddies, hey, you know, what do you think? What if we invade Poland today? How do you think that's going to go? Like, that's absurd, right? You want a defense department. You should have an AI department. And the AI department, one of the things it should do is like, what has happened this week? You know, is there something, because things are moving very quickly, even if there's diminishing returns on the,

absolute sort of competence of the models, people find new applications.

There's a lot of fighting over turf about what to do. We should have somebody running point. What do you do about these various risks? There should be an agency looking at all of these things, having it all on the radar. There should be a person in the United States with the staff in order to do that. Well, I mean, I too want help sense-making on some of this. And I too want a body to help collate the news, to help figure out where things are ready and where they aren't. But there's actually a lot of voices who say, you know,

that will get captured, it will get either overtly corrupted or it will just grind into the dust like some of the other institutions. What do you say to those people? I say those people are absolutely right to have that concern.

Every agency, in fact, faces the risk of regulatory capture, which means that the companies that we're trying to regulate co-opt the process. And boy, the companies that we're trying to regulate are good at that. They did a phenomenal job in social media of co-opting the regulation that we had. Section 230, as I mentioned earlier, has been a complete disaster. So I 100% think that we need to

for regulatory capture. I think one of the ways to address that is to have independent scientists in the mix. I have squirmed every time I have seen a major government have a meeting in which they have, you know, three or 20 members

representatives of big companies in the room and no major scientists. And I have seen that over and over and over again. I've seen it in the United States. I've seen it in other countries. And this tendency to treat the leaders of the companies as rock stars and treat scientists as an afterthought is a huge mistake. And it's also a mistake optically, I think, although people don't care enough about it. But it just looks wrong. And I think...

We cannot succeed if the only people at the table are the big companies because they will co-opt things. They have a long history of doing that. So I'm extremely sympathetic to that and

My only really good answer is to have independent voices. I tend to emphasize scientists because I am one, but independent ethicists and legal scholars and so forth. We have to have multiple voices at the table looking out saying, what are the ways in which this could get co-opted or just go wrong? So it's worth getting it right. That's right.

Gary, this is obviously a really deep topic that we could talk about forever. And we haven't even talked about international competition and a bunch of other things. But I would really encourage people to read your book where you lay a lot of this out. Speaking of which, you end your book with a call to action. And I really want to give you the chance to make that call to action straight to the listeners of this podcast. To everyone listening right now, whether you're a policymaker or a computer scientist or someone completely outside of this world, what do you think we all need to do to make this go better?

We need to speak up. We can't leave it just to the companies to self-regulate, which is what they want. And we can't leave it to the government, especially in the United States with Citizens United and so forth. We're just not going to get the government on its own without people speaking up to take care of us.

We need the citizens to speak up. And that can be in the form of direct action, indirect action. One direct action that I don't know that everybody will do but we should think about is actually boycotting certain products until they are ethical, until they're reliable and so forth. So I'll give an example. There are generative AI programs that make images. They're super fun to play with. But they are using the work of artists and they're not compensating the artists.

It's like using coffee that's not fairly sourced. We could as a society say there are so many negative externalities of these products that if you use them, you're helping build a kind of new form of capitalism that is very exploitative.

So, we haven't talked much today about the environmental cost, for example, but they're large and they're getting larger because the industry's only idea about how to fix its problems, and everybody knows these problems of hallucinations and so forth, is to try to build even bigger models that are going to cause even more harm to the environment, take an even larger fraction of the energy of the world, probably drive prices up. We could say, as a society, enough.

We're not saying never build AI, but why should we use your products now when you don't have an answer for who's going to pay the cost? You don't have an answer for how to make it reliable. And if we incentivize the companies by insisting that they make their products better and not release them until they are, we could actually force them to a position where the AI would be better.

I honestly think we're at a kind of like knife's edge between a dystopia here and a utopia. The utopia is we could make AI that really helps with science, really helps the many and not just the few, or we could wind up with an AI that's basically just

of intellectual property to a small number of companies that cause all of these harms, discrimination, disinformation, and so forth. And which way we wind up depends on what do we do now? Do we do anything like a boycott? Do we at least insist that as part of the election that people tell us their position on it? I mean, the thing that flabbergasts me is like,

Every day people talk about immigration and economy and so forth, and nobody is talking about AI. There's been no discussion so far on policy and AI. I hope that this is outdated by the time it comes to air, but I don't know that it will be. The very least that people could do is to call their senators, their congresspeople, and say, what are you doing about AI and all of these risks? The American public is, in fact, worried about AI, but they're not loud enough about it.

Gary, thanks for laying all of this out. And thank you so much for coming on your Undivided Attention. Well, you had my Undivided Attention. And I think the connection between what you think about in this show at CHT and what I'm talking about in AI is profound. Everything that you have been worried about applies in AI and possibly even worse. And so I really appreciate the chance to speak to your listeners.

If you enjoyed this conversation and you want to learn more, you can find Gary's book, Taming Silicon Valley, wherever books are sold. Your Undivided Attention is produced by the Center for Humane Technology, a nonprofit working to catalyze a humane future. Our senior producer is Julia Scott. Josh Lash is our researcher and producer.

And our executive producer is Sasha Feagin. Mixing on this episode by Jeff Sudeikin. Original music by Ryan and Hayes Holliday. And a special thanks to the whole Center for Humane Technology team for making this podcast possible. You can find show notes, transcripts, and much more at humanetech.com. And if you liked the podcast, we'd be grateful if you could rate it on Apple Podcasts because it helps other people find the show. And if you made it all the way here, thank you for giving us your undivided attention.