Echo Chambers of One: Companion AI and the Future of Human Connection
42:17 Share

Hey everyone, this is Daniel Barquet. Welcome to your Undivided Attention. You've probably seen a lot of the news lately about AI companions, these chatbots that do way more than just answering your questions. They talk to you like a person, they ask you about your day, they talk about the emotions you're having, things like that. Well, people started to rely on these bots for emotional support, even forming deep relationships with them.

But the thing is, these interactions with AI companions influence us in ways that are subtle, in ways that we don't realize. You naturally want to think that you're talking with a human because that's how they're designed. But they're not human. They're platforms, incentivized to keep you engaging as long as possible, using tactics like flattery, like manipulation, and even deception to do it. We have to remember that the design choices behind these companion bots, they're just that, they're choices, and we can make better ones.

Now, we're inherently relational beings. And as we relate more and more to our technology, and not just relating through our technology to other people, how does that change us? And can we design AI in a way that helps us relate better with the people around us? Or are we going to design an AI future that replaces human relationships with something more shallow and more transactional? So today on the show, we've invited two researchers who've thought deeply about this problem.

Patty Maas and Pat Pateranudapan are co-directors of the Advancing Humanity Through AI Lab at MIT. Patty is an expert in human-computer interaction, and Pat is an AI technologist and researcher, and a co-author of an article in the MIT Technology Review about the rise of addictive intelligence, which we'll link to in the show notes. So I hope you enjoy this conversation as much as I did. Patty and Pat, welcome to your Undivided Attention. Thank you. Happy to be here. Thanks for having us.

So I want to start our conversation today by giving a really high stakes example of why design matters when it comes to tech like AI. So social media platforms were designed to maximize our users' attention and engagement, keeping our eyes on the screen as long as possible to sell ads.

And the outcome of that was that these algorithms pushed out the most outrageous content they could find and playing on our human cravings for quick rewards, dopamine hits, eliminating any friction of use and removing common stopping cues with dark patterns designs like infinite scroll. It was this kind of race to the bottom of the brainstem where different companies competed for our attention. And the result is that we're now all more polarized and more outraged and more dependent on these platforms.

But with AI chatbots, it's different. The technology touches us in so much deeper ways, emotional ways, relational ways, because we're in conversation with it. But the underlying incentive of user engagement is still there, except now, instead of a race for our attention, it seems to be this race for our affection and even our intimacy.

Yeah, well, I think AI itself can actually be a neutral technology. AI itself is ultimately algorithms and math. But of course, the way it is used can actually lead to very undesirable outcomes.

I think, Bob, for example, that socializes with a person could either be designed to replace human relationships or it could be designed to actually help people with their human relationships and push them more towards relationships.

human relationships. So we think we need benchmarks to really test how or to what extent a particular AI model or service ultimately is leading to human socializing and supporting them with human socializing versus actually pulling them away from socializing with real people and trying to replace sort of their human socializing.

I want to challenge Patty a little bit when she said that technology is neutral. It reminds me of Marvin Kranzberg's first law of technology where he said that technology is neither good nor bad nor it is neutral. And I think it's not neutral because there's always someone behind it and that person is either having good intention or maybe bad intention.

bad intention. So the technology itself is not something that act on its own, but it's always, even though you can create the algorithm that's sort of self-perpetuated or like going in loop, but there are always some intention behind. So I think understanding that sort of allow us to not just say, well, technology is out of control. We need to ask like who actually let the technology go out of control?

I don't think technology is just coming after us for affection. It's also coming after us for intention as well, like shaping our intention. So, you know, change the way that I want to do things or change the way that I do things in the world, right? Like changing my personal intention, whether I have it or not. So it's not just the artificial part that is worrying, but the addictive part as well, because this thing, as Patty mentioned, can be designed to be extremely personalized and

use that information to exploit individual by creating this sort of like addictive use pattern where people just, you know, listen to things that they want to listen to or the bot just tell them what they want to hear rather than, you know, telling them the truth or things that they might actually need to hear rather than what they want to hear. I think the term that we use a lot is the psychosocial outcome of human-AI interaction

People worry about the misinformation or AI taking over jobs and things like that, which are important. But what we also need to put attention on is also the idea that these things are changing who we are.

Our colleague Sherry Turco once said that we should not just ask what technology can do, but what it is doing to us. And we worry that this question around addiction, psychosocial outcome, loneliness, all these things that are related to the person's personal life are being ignored when they think about the AI regulation or the impact of AI on people.

And can you go a bit deeper? Because, you know, when those of us in the field, we talk about AI sycophancy, right? Not just flattering you, but telling you what you want to hear, right? Going deeper. Can you lay out for our audience other kinds of mechanics of the way that AIs can actually get in between our social interactions?

Totally. I think in the past, right, if you need advice on something or if you want to sort of get an idea, like you probably go to your friends or your family, you know, they will serve as the sounding board. Like, you know, this is sound, right? Is this something that makes sense? But now you could potentially go to the chatbot

And a lot of people say, well, this chatbot are trained on a lot of data. Then what would you hope for is that this chatbot will just, you know, based on all this data will tell you the unbiased view of the world, right? Like what is the scientific accurate answer on a particular topic? But because the bot can be sort of biased intentionally or unintentionally, right? Right now, we know that this system contain frequency bias, the thing that they see in

frequently will be the thing that they might actually say it or they have positivity bias. They always try to be positive because, you know, user wanna hear it doesn't sort of say negative thing. And if you keep being exposed to that repeatedly, it can also make you believe that, oh, that's actually the truth.

And that might actually make you go deeper and continue to find more evidence to support your own, right? We have identified this as sort of the confirmation bias where you might initially have skepticism about something, but after you kind of being repeatedly exposed to that, then it becomes something that you have a deep belief in. Yeah, and I also want to dig in a little bit there because people often think there's this like mustache twirling instinct to skepticism.

take these AIs and make them split us apart. And that's a real risk, don't get me wrong, but I'm also worried about the way in which these models unintentionally learn how to do that. We saw that AIs being trained to autocomplete the internet end up playing this sort of game of improv where they sort of become the person or the character that they think you want it to be. And it's almost like codependency in psychology where the model's kind of saying, well, who do you want me to be? What do you want me to think? And it sort of becomes that.

And I'm really worried that these models are telling us what we want to hear way more than we think. And we're going to get sucked into that world. And I think with social media, we had a lot of polarization and information bubbles. But I think with AI, we can potentially even get to a more extreme version of that where we have bubbles of one, where it's one person with their information.

echo of a psychophant AI where they spiral down and become, say, more and more extreme and have their own worldview that they don't share with anyone else. So I think we'll get further pulled apart even than in the social media age or era.

We did a study where we investigate this sort of question a little bit, where we prime people before they interact with the same exact chatbot, different description of what the chatbot is. In one group, we told people that the chatbot have empathy, that it can actually care for you, it have deep beneficial intention to actually help you get better. In the second group, we told people that these chatbot were completely manipulative, that they, you know, act

act nice, but it actually wants you to buy a subscription. And the third one, we told people that the chatbot was actually a computer code. And at the end, they were all talking to the same exact LLM model. And what we found is that people talk to this chatbot differently, and that also triggered the bot to respond differently as well. And these sort of feedback

loop, whether it's a positive feedback loop or negative feedback loop, influence both the human behavior and the AI behavior, right? As Patty mentioned, it could kind of create this sort of bubble. It kind of creates certain belief or reaffirms certain belief in the user, right? And so that's why in our research group, one thing that we focus on is if we want to sort of understand the science of human-AI interaction to uncover the positive and the negative side of this, we need to look at not just the human behavior or the AI behavior, but we need to look at

both of them together to see how they reinforce one another. I think that's so important. I mean, what you're saying is that AI is a bit of a Rorschach test. If you come in and you think it's telling you the absolute truth, then you're more likely to start an interaction that continues and then affirms that belief. If you come in skeptical, you're likely to start some interaction that keeps you in some skeptical mode of interaction. So that sort of begs the question, like, what is the right mode of interacting with it? I'm sure listeners of this podcast are all playing with AIs on a daily basis.

What is the right way to start engaging with this AI that gives you the best results? Yeah, I think we have to encourage very healthy skepticism. And it starts with what we name AI, right? We refer to it as intelligence and claim that we're nearing AGI, general intelligence. So it starts already there with misleading people into thinking that they're interacting with an intelligent entity. The anthropomorphization that happens

Well, and yet all the incentives are to make these models that feel human because that's what feels good to us. And Pat, only a few months after you wrote that article in MIT Tech Review, we saw the absolute worst case scenario of this with the tragic death of Sewell Setzer, this teenage boy who took his own life after months of this incredibly intense, emotionally dependent, arguably abusive relationship with an AI companion. Right.

How does the story with Sulzetzer highlight what we're talking about and how these AI designs are so important?

When we wrote that article, it was hypothetical, right? That in the future, the model will be super addictive and it could lead to really bad outcomes. But as you said, after I think a couple months, when we got the email about the case, it was shocking to us because we didn't think that it will happen so soon. As a scientific community sort of start to grasp with this question of how we design AI,

We are at the beginning of this, right? You know, these two are just two years old and it sort of launched to, you know, massive amount of people. Pretty much everyone around us are sort of using these two on a regular basis now. But the scientific understanding of how do we best decide these two are still at the early age. I mean, we have a lot of knowledge in human-computer interaction, but

Previously, none of the computers that we designed have this interactive capability. It doesn't model users in the way that these LLM have, or it doesn't respond in a way that is so human-like that we have right now. We had an early example, like the ELISA chatbot. Even with that limited capability, ELISA, I think, was a chatbot that was invented in the 70s or something like that. Late 70s, early 80s, something like that. Yeah.

It can only sort of rephrase what the user said and then engage in conversation in that way. They already have impact on people, but now we see that even more. So going back to the question of the suicide case that was really devastating, I think now it's more important than ever that we think of AI not just an engineering challenge. It's not just about improving the accuracy or improving its performance, but we need to think about the impact

of it on people, especially the psychosocial outcome. We need to understand how each of the behavior, you know, psychofancy, bias, anthropomorphization, how does it affect things like loneliness, emotional dependence, and things like that. So that's actually the reason we, you know, start doing more of that type of work, not just on the positive side of AI, but it's also equally important to study the condition where human doesn't flourish with this technology as well.

I wanted to start with all these questions just to lay out the stakes, right? Like, why does this matter? But there's also this world where our relationship with AI can really benefit our humanity, our relationships, our internal psychology, our ability to hold nuance, speak across worldviews, sit with discomfort. And I know this is something that you're really looking at closely at the lab at Advancing Humans with AI Initiative. Can you talk about some of that work you're doing there to see the possible futures? Yeah.

Yeah, I think instead of an AI that just sort of mirrors us and tells us what we want to hear and tries to engage us more and more, I think we could design AI differently so that AI makes you see another perspective on a particular issue. It isn't always agreeing with you, but it's designed maybe to help you grow as a person

an AI that can help you with your human relationships, thinking through how you could deal with difficulties with friends or family, etc. Those can all be incredibly useful. So I think it is possible to design AI that ultimately is

well, created to benefit people and to help them with personal growth and the critical thinker, the great friend, et cetera. Yeah, and to give more specific example, as Patty mentioned, we did several experiments and

I think it's important that we highlight this word experiment. It's because we want to also understand scientifically, like, does this type of interaction benefit people or not, right? I think right now there are a lot of big claims that, you know, these tools can cure loneliness or can make people learn better, but

there was no scientific experiment to compare different type of approach or different type of intervention on people. So I think in our group, we take sort of the experimentation approach where we build something and we develop the experiment and also control condition to kind of validate that.

For example, in critical thinking domain, we look at what happened when the AI asked question in the style of Socrates, where he used Socratic method to ask or challenge his student to think rather than always providing the answer. I think that's one of the big question that people ask, like what is the impact of AI on education?

and you know if you use the ai to just give information to students we are essentially repeating the sort of the factory model of education right where kids are being you know given or feed the same type of information and they're not thinking they're just absorbing right so we flip that paradigm around and we design ai that you know flip the information into a question and instead of helping the student by just giving the answer it will help student by asking question like oh

if this is the case, then what do you think, you know, that might look like? Or, you know, if this conflict with this, what does it mean for that, right? So kind of framing the information as question. And what we found is that when we compare to an AI that's always providing the correct answer, again, this is an AI that's always providing the correct answer. This is, you know, that's not happening in the real world, right? AI always hallucinate and give

wrong answer sometimes. But this is comparing to the AI that always gives the correct answer. We found that when the AI engaged people cognitively by asking the question, it actually helped people arrive at the correct answer better than when the AI always gives the answer.

This is in the context of helping people navigating fallacies. So when they see a statement and they need to validate whether this is true or false. So the principle that we can derive here is that human-AI interaction is not just about providing the information, it's also about engaging people with their cognitive capability as well. And our colleague at Harvard coined this term cognitive forcing function, where the system presents something

some sort of conflict or challenge or question that make people think rather than, you know, eliminate that by providing the answer. So this type of design pattern, I think, can be integrated in education and other tools. I think that's really interesting because, you know, we've been thinking a lot about how to prompt AI to get the most out of AI. And what you're saying is actually making AI that,

prompts humans, like prompts us into the right kind of cognitive frames. Yeah, totally. I think the promise is really there. But one of the things I think I worry about is that you run headlong into the difference between what I want and what I want to want, right? I want to want to go to sleep reading deep textbooks. But, you know, what I seem to want is to go to sleep scrolling YouTube and scrolling, you know, some of these feeds. And then, you know, in an environment that's highly competitive, right?

I'm left wondering, like, how do you make sure that these systems that engage our creativity, that engage our humanity, that engage the sort of deep thinking out-compete, right?

That's, I think, a really great question. Our colleague, Professor Mitch Resnick, who runs the Lifelong Kindergarten Group at the Media Lab, he said that, you know, human-centered AI is a subset of human-centered society. If we say that technology is going to fix everything and we can, you know, create a messy society that exploits people and has the wrong incentive, then these tools will be in service of that incentive rather than supporting people.

So I think maybe we're asking too much of technology, right? Like we say, well, how do we design technology to support people? We need to ask bigger question and ask, how can we create human-centered society? And that requires more than technology. It requires regulation. It requires civic education and democracy, right? Which is sort of rare these days. So I think right now technology is sort of on the hot spot, right? We want better technology, but if we zoom out,

technology is sort of a subset of an intervention that happened in society and we need to think bigger than that, I think. Yeah. Patty, zooming in a bit, you two were involved in this big longitudinal study in partnership with OpenAI that looked at how chatbot use, regular chatbot use, was affecting users. And this is a big deal, right? Because we don't have a lot of good empirical data on this. What were those biggest takeaways from that study and what surprised you?

Yeah, it was really two studies actually. And one of the studies just looked at the prevalence of people sort of using pet names, etc. for their chatbot and really seemingly having a closer than healthy relationship with the chatbot.

And there we looked, or OpenAI specifically, looked at transcripts of real interactions and saw that that was actually a very small percentage of use cases of ChatGBT. Of course, there are other chatbots... Sorry, what was a small percentage? A small percentage of... People who sort of talk to a chatbot as if it's almost a lover or their best, closest friend, sort of. Yeah.

But of course, there's other services like Replica and Character.ai, etc., that are really designed to almost replace human relationships. So I'm sure that on those platforms, the prevalence of those types of conversations is much higher. But then we also did another study, which was a controlled experiment. And do you want to talk about that a little bit, Pat? Sure.

Yeah, totally. So as Patty mentioned, the two studies, we co-designed them with OpenAI. The first one, we call it all-platform study, where they look at the real conversation and then we look at 40 million conversations on ChatGPD and try to kind of identify first heavy users, people that use it a lot.

But we want to understand what is the psychosocial outcomes, that's the term that we use in the study on that. So we create a second study that, you know, be able to capture rich data around people, not just how they use the chatbot. So for this second study, what we did was we recruit about 1,000 participants

And we randomly assigned them into three conditions. In the first condition, they talked to advanced voice mode, which is the voice mode that at the time, I think people associated with the Scarlett Johansson scandal. And we intentionally designed two voice modes. One is the engaging voice mode, where it's designed to be more flirty, more engaging, more

And then the other one is like, you know, more polite and more sort of more neutral, like more professional. Yeah. And then we compare it to the regular text. And in the third group, we prime people to use it sort of in the open world. They can use it whatever. So what were the key findings from that when these different modes of engaging with AI that had different personalities emerged?

So I think we found that there was a driving force with the time that people use it. If they use it for a shorter period of time, I think we see some positive improvement. Like, you know, people become less lonely. People have healthy relationship with the bot. But once they sort of pass certain threshold, once they use it longer and longer, we see this sort of positive effect diminish. And then we see people become lonelier and have more emotional dependence with the bot.

have used it in a more problematic way, right? So that's the pattern that we observe. Longer every day, not just number of days, but the more you use it in a day, the less good the outcomes were in terms of people's loneliness, socialization with people, etc. So people who use

these systems a lot each day tend to be lonelier, tend to interact less with real people, etc. Now, we don't know what the cause and effect there is.

It may go both ways. But in any case, it can lead to a very negative feedback pattern where people who already possibly are lonely and don't hang out with people a lot then hang out even more with chatbots and that makes them even more lonely and less social with human relationships and so on.

Yeah, so it feels like one of the coherent theories there is, right, instead of it augmenting your interactions, it becomes a replacement for your sociality in the same way that being at home alone and watching TV, sitcoms have the laugh track. Why do they have the laugh track? So that you get this parasocial belief that you're with other people. And we all know that replacing your engagement with people is not a long-term successful track.

I want to zoom in on two particular terms that you talk a lot about. One is sycophancy and the other is model anthropomorphization, like the fact that it pretends to be a human or pretends to be more human. So let's do sycophancy first. Most people misunderstand sycophancy as just flattery, like, oh, that's a great question you have.

But it goes way deeper, right? And the kind of mind games you can get into with a model that is really sycophantic go way beyond just flattering you. Let me go one step further and tell you why I'm really worried about sycophancy. Like in 2025, we're going to see this massive shift where these models go from being just sort of conversation partners in some open web window to deeply intermediating our relationships. You know, I'm not going to

try to call you or text you, I'm going to end up saying to my AI assistant, oh, I really need to talk to Patty about this. And let's make sure, can Patty come to my event? I really want her there. And on the flip side, the person receiving the message says,

is not going to be receiving the raw message. They're going to ask their AI assistant, well, who do I need to get back to? And so as we put these models in between us, as we're no longer talking to each other, we're talking to each other through these AI companions. And I think these subtle qualities, like telling us what we want to believe

are going to really mess with a ton of human relationships. And Patti, I'm curious your thoughts on that. Yeah, well, I think AI is going to mediate our entire human experience. So it's not just how we interact with other people, but of course, also our access to information, how we make decisions, purchasing behavior and other behaviors and so on.

It is worrisome because what we see in the experiments that we have done and that others like Mor Namah and Cornell are doing is that AI suggestions influence people in ways that they're not even aware of. They're not aware that their beliefs and so on are being altered by interaction with AI when asked.

So I'm very worried that in the wrong hands or in anyone's hands, as Pat talked about earlier, there's always some value system, some motives that ultimately are baked into these

systems that will ultimately influence people's beliefs, will influence people's attitudes and behaviors when it comes to not just how they interact with others and so on, but how they see the world, how they see themselves, what actions they take, what they believe and so on. Right.

And I also see that as something that might have a negative effect on skill as well. Like we may have skill atrophy, especially skill for interpersonal relationship, right? If you always have this translator or this sort of, you know, system that mediate between human relationship, then you can just be angry at this bot and it will just like translate into a nice version to the person that you want to talk to. So you might lose ability to control your own emotion or know how to sort of talk to

other people, like you always have this thing in between, right? But I mean, going back to the question of AI design, right? I mean, if we realize that that's not the kind of future we want to do, then I hope that as a democratic society, people would have the ability to not adapt

this and go for a different kind of design. But again, there are many types of incentives and there are sort of the larger market force here that I think is going to be challenging for this type of system, even though it's well designed, well backed by scientific study. So I really appreciate your center for

for doing this kind of work, right? Well, likewise. Yeah, to ensure that we have the kind of future we want. Well, our future with AI is being determined right now by entrepreneurs and technologists. Basically, increasingly, I think governments will play a key role in determining how these systems are used and in what ways they control us or influence our behavior and so on.

And I think we need to raise awareness about that and make sure that ultimately everybody is involved in deciding what future with AI we want to live in and how we want this technology to influence the human experience and society at large. In order to get to that future,

We exist to try to build awareness that these things are even issues that we need to be paying attention to now. And why we're so excited to talk to you is people need to understand what are the different changes and the design changes that we can make. And I'm curious if you've come to some of these. Clearly, Pat, the problem you talked about that I call the pornography of human relationships, like these models becoming these always-on, always-available, never needing your attention, you're the center of the world,

And it becomes such an easy way to abdicate human relationship. How do we design models that get better at that, that get us out of that trap? Yeah, I think, you know, first of all, I think the terminology that we use to describe this thing need to be more specific, right? It's not just whether you have AI or don't have AI, but what's specific about AI that we need to rethink or redesign, right? I really love this book called AI Snake Oil that they talk about

they say that, well, we use the term AI for everything and you will not be able to, in the same way that we say car for, you know, bicycle or truck or bus, then we would treat all of them the same way when in the real world, you know, they have different degree of dangerousness, right? So I think that's something that we need to think about AI as well. So we need to increase in our literacy

the specificity of how we describe or talk about different aspects of AI systems. And also benchmarks, as we talked about earlier, for measuring to what extent particular models show a certain characteristic or not. Yeah, so talk more about that for our audience who may not be familiar. What kind of benchmarks do you want to see in this space? Mm-hmm.

Yeah. So I think right now, the benchmark that we use, most of them don't really sort of consider the human aspect as well. Like they don't ask, well, if the model can do very well on mimicking famous artistic style, how much does it affect the artist doing that? Or how much does it affect

human ability to come up with creative original ideas. These are things that, you know, it's kind of hard for a test to be able to measure, right? But I think with the work that we did with OpenAI, I think that's a starting point to start thinking about this sort of human benchmark. Like, well, whether the model make people lonelier or less lonely, whether the model make people more emotionally dependent or less emotionally dependent,

on AI and we hope that we can be able to scale this to other aspects as well. And that's actually one of the mission of our AHA program or the Advancing Human with AI program is to think about this sort of human benchmark that when the new model come out, we can sort of simulate or have an evaluation of what would be the impact on people. So the developer and engineer could think more about this.

Okay, so let's go one level deeper. So we covered sycophancy and we covered this always available, never needy kind of super stimulus of AI. But what about anthropomorphic design? Like what are specifics in the way that we could be making AI that would be preventing this sort of confusion?

of people thinking that AI is human. Yeah, I think the AI should never refer to its own beliefs, its own intentions, its own goals, its own experiences, because it doesn't have them. It is not a person. So I think that is already a good start. And again, we could potentially develop benchmarks and look at interactions and see to what extent models do this or not.

But it's not healthy because all of that behavior encourages people to then see the AI as, say, more human, more intelligent, and so on. So that would be some sort of metric you could put on is how often even the statement like, oh, that's a really interesting idea is a fake emotion that the model is not actually experiencing. Exactly. Mm-hmm.

But I mean, I think this is a complicated topic, right? Because on one hand, as a society, we also enjoy sort of art form like cinema or theater where people do this kind of role playing and, you know, portray fictional character. And we could sort of enjoy the benefit of that where we can, you know, engage in a video game character and interact with this sort of fantasy world, right? But I think it's,

a slippery slope because once we start to blur the boundary that we can no longer tell the difference, I think that's when it gets dangerous. We did a study also where we looked at what happened when students learn from virtual characters based on someone that they like or admire at a time we, you know,

We did a study with virtual Elon Musk. I think he was less crazy at the time. And we see the positive impact of the virtual character for people that like Elon Musk, but people that did not like him back then, they're also not doing well, right? It had the opposite effect. So that personalization or creating virtual character based on someone that you like or admire can also be a positive thing. So I think these...

technology also heavily depends on the context as well and how we use it. That's why I think the quote from Kranzberg that is neither good nor bad nor neutral is very relevant today.

It wouldn't be CHT if we didn't direct the conversation towards incentives. One of the things I worry about is not just the design, but the incentives that end up driving the design. And making sure that those incentives are transparent and making sure that we have it right. So I want to put one more thing on the table, which is right now we're just RLHFing these models, which is reinforcement learning with human feedback. We see how much a human liked, or even worse, how many milliseconds they just continue to engage with the content.

as a signal to the model of what's good and bad content. And I'm worried that this is going to cause those models just to basically do the race to the bottom. They're going to learn a bunch of bad manipulative behaviors. And instead, you would wish you had a model that would learn who you wanted to become, who you wanted to be, how you wanted to interact. But I'm not sure the incentives are pointing there. And so the question is, do you two think about different kinds of incentives about the way you could push these models towards learning these better strategies? Yeah.

Maybe we need to give these benchmarks to the models themselves so they can keep track of their performance. Like try to optimize for the right benchmarks.

But I think, you know, Patty, you have said to me at one point, you said that if you're not paying for the software, then you are the product of the software, right? I think that that's really true, right? I think majority of people don't pay social media to be on it, right? They subscribe or they get on it for free. And in turn, the social media sort of exploit them, you know, as a product for, you know, selling their data or selling their attention to other companies. But I think for the AI companies, I think if,

people are paying subscription for this, then at least they should be able to, in theory, have control, even though that might fade away soon as well. So I think we need to figure out this sort of question of how do we create human-centered society and human-centered incentive, and then the technology would align once we have that sort of larger goal or larger structure to support that, I think. But even with a subscription model, there may still be other incentive sets possible.

play where these companies want to collect as much data about you because that data again can be monetized in some ways or can be valuable training or it also makes their services more sticky right

Sam Altman just announced that there's going to be more and more memory, basically, in ChatGPT of previous sessions. And of course, on the one hand, you think, oh, this is great. It's going to remember previous conversations. And so it can assist me in a much more personalized way. I don't have to explain everything again and so on.

But on the other hand, that means that you're not going to switch to another system because it knows you and it knows what you want, etc. And so you keep coming back to that same system. So there's many other things at play, even though there might be a subscription model.

Well, it reminds me, the term con man comes from the word confidence. And the entire point is these people would instill enough confidence in people that they would give them their secrets, their bank accounts, their this, and then they would betray the confidence, right? And so while an aligned model and an aligned business model, knowing more about you and your life and your goals is fantastic, a misaligned model and a misaligned business model, having access to all that information is kind of the fast path to con man. Mm-hmm. Yep.

Okay, so maybe just in the last few minutes, I guess I have just a question of, I'm curious whether you've done any more research on the kinds of incentives or design steers that produce a better future in this sense, right? In the sense of what are the kinds of interventions, the kinds of regulation, the kinds of business model changes? Like, what would you advocate for if the whole public could change it?

I was actually reflecting on this a little bit before coming on to the podcast today. Like, what is our pathway to impact? And I think for me as a researcher, like what we are really good at is sort of trying to understand this thing at a deeper level and coming up with experiment and new design that can be alternative, right? You know, I think that there's something sort of

interesting about the way that historically, like before you can attack the demon, you need to be able to name it, right? I think it's similar with the AI. In order for us to sort of tackle this wicked, challenging thing, you need to have a precise name and terminology and understanding of what you're dealing with. And I think that's sort of our role, I think, as a researcher in academia is to shed light on this and

enhance our understanding of what's going on in the world, especially with AI. Yeah, well, here at CHT, we say that clarity creates agency. If you don't have the clarity, you can't act. And that's why I want to thank you both for helping us create the clarity, name the names, find the dynamics, you know, do the research so we know what's happening to us in real time before it's too late.

Yeah, I think I might want to say a little bit about technologies of the future. What we hope is that this is not just our work, right? I hope that more researchers are jumping on to do this kind of work. And so for people developing AI across academia industry, that they will start thinking bigger and broader and not to see themselves as someone who can just do the engineering part of the AI or doing the training part of the AI, but

thinking what is the downstream impact that what they are doing is gonna how it's gonna impact people so that maybe you know we will steer away from the conversation around like well you know it's inevitable this thing is individual but we need to kind of be on the race to this if more people have that sort of awareness and more people listen to you guys like you know this podcast and follow the work of the center for humane technology i think we will have technologies that are more thoughtful

Yeah, and AI should become a more interdisciplinary endeavor, I think. Not just, again, the engineers, the entrepreneurs, and maybe the government as well. But we should have historians and philosophers and sociologists and psychologists, etc. They have a lot of wisdom, right?

about all of this. And so I think it has to become a much broader conversation, not just educating the entrepreneurs and the engineers. I agree with that 100%. And as this technology can meet us in such deeper ways than any technology in the past, as it can touch our psychology, as it can intermediate our relationships, as it can do things out in the world, we're going to need all of those other specialties to play a part.

I think it's really clear that this is a really, really hard question, right? It touched on so many aspects of human life, not just our sort of, I think right now, people focus on productivity, whether AI will help people work better. But I think even just the productivity alone or just the work area alone, you know, it's also touched on the productivity

question of purpose, right? What does it mean to actually do something, right? It will change the way that we think about the purpose, our sort of meaning in life and things like that. So even just this domain alone is never just about work by itself. So I mean, that's why AI is a really hard question that requires us to think in many dimensions and in many directions at the same time. And we don't necessarily have all the answers to this big question, right? But

I think that the more that we can learn from auto-discipline, the more that we can learn from wisdom across culture, across different group of people, across expertise, the better we could start to comprehend this and have better clarity on the issue at hand. I'm so thrilled that you're doing this work. I'm so glad that you're in this world and that we get to work together and build on each other's insights. And thanks for coming on Your Undivided Attention. Great to be here. Thank you. Thank you so much.

Your undivided attention is produced by the Center for Humane Technology, a nonprofit working to catalyze a humane future. Our senior producer is Julia Scott. Josh Lash is our researcher and producer. And our executive producer is Sasha Fegan. Mixing on this episode by Jeff Sudeikin. Original music by Ryan and Hayes Holliday. And a special thanks to the whole Center for Humane Technology team for making this podcast possible. You can find transcripts of our interviews and bonus content on our sub stack and much more at humantech.com.

You can also watch all episodes on our YouTube channel. Just search for Center for Humane Technology. And if you liked this episode, we'd be grateful if you could rate it on Apple Podcasts and Spotify. It really does make a difference in helping others join this movement. And if you made it all the way here, let me give one more thank you to you for giving us your undivided attention.