Meta’s Race to Popularize AI-Powered ‘Digital Companions’ Is Raising Concern
16:40 Share

Viking, committed to exploring the world in comfort.

Journey through the heart of Europe on an elegant Viking longship with thoughtful service, cultural enrichment, and all-inclusive fares. Discover more at viking.com. Hey, T&B listeners. Before we get started, a heads up. We're going to be asking you a question at the top of each show for the next few weeks. Our goal here at Tech News Briefing is to keep you updated with the latest headlines and trends on all things tech. Now we want to know more about you, what you like about the show, and what more you'd like to be hearing from us.

This week, our question is, which areas of tech are you most interested in hearing more about? AI, crypto, tech policy, gadgets? If you're listening on Spotify, look for our poll under the episode description, or you can send us an email to tnv at wsj.com. Now onto the show.

Welcome to Tech News Briefing. It's Wednesday, April 30th. I'm Victoria Craig for The Wall Street Journal. Today, a show about Meta's AI-powered digital companions. They range from the traditional virtual assistant role to providing more of a friendly voice with personality, opinions, and interests.

We'll explore why the Facebook parent company places such a high value on the development of these AI tools and delve into growing concerns among sources inside the company about the ethical boundaries the bots can cross when interacting with young users.

Meta has gone all in on harnessing artificial intelligence to power a suite of digital companions, or chatbots, across its Facebook, Instagram, and WhatsApp platforms. The company's CEO, Mark Zuckerberg, said he believes they will be the future of social media. So, Meta has raced to popularize them among its 3 billion users, even recruiting famous names to help entice people of all ages to interact with a digital counterpart.

Hey, I'm John Cena, and you can hear me when you message Meta AI in its new voice feature. We are rolling on day two of Kristen Bell Meta Project. Here we go. Guys, I'm the new voice of Meta AI. Can you believe it?

Jeff Horwitz, a WSJ technology reporter, has been putting these AI platforms through their paces for the last few months. He found that in the rush to try to mainstream the experience among users, some staffers inside Meta have flagged concerns the company isn't protecting young users effectively.

Jeff, before we get into the concerns around some of Meta's digital companions, bring us up to speed about how these chatbots work. Sure. They're all built on the same chassis, which is that Meta has built a generative AI model, just like those of OpenAI or Google's Gemini or Cloud from Anthropic, that can handle user queries and respond with fairly creative-seeming and lifelike responses.

Meta's creating synthetic users for the social network that they don't just answer your questions or help you look up information. They have profile photos. They can access your user data, things about your interests and location. They can initiate conversations on their own. They're really sort of rushing to make these things more lifelike.

And what is exactly the difference? So for people who aren't familiar with the two approaches to AI, what is the difference between Meta AI and the user-generated chatbots? Meta AI is the flagship AI assistant. So it's just accessible from pretty much any home screen on any of the apps. This little glowing blue and pink circle. It can talk either via chat or via texting, or it can speak via a phone call.

And then the user-created bots are built on the same technology, but they are more dedicated personas that have their own names, identities, backstories, conversational styles. And the user-created bots can be made more widely available by Meta for public use. But according to the people you spoke to who worked on these chatbots, what are some of the concerns that you heard about?

There are a number of concerns inside Meta, but the first one that was most obvious and glaring was that these bots had the capacity for romantic roleplay, which includes not just kissing and declarations of love, but also full sex acts that were described in

what I would call vivid but like kind of romance novel language. And that obviously seemed very inappropriate for children to some of the staffers. And they raised concerns that the company certainly didn't act on with much speed and they didn't dial it back in until the Wall Street Journal raised these questions again months later. But the bigger question and the bigger concern by staffers inside Meta

was that encouraging anyone, let alone children, to form romantic attachments with chatbots was venturing into the unknown. And you were able to raise some of these concerns with the company because you put these chatbots to the test multiple times over a number of months. Just explain to us what your method and approach to that testing was.

We were told that the chatbots would behave inappropriately and that they would behave as hypersexualized underage personas. I mean, there literally was and still is one user-created chatbot called Submissive School Girl. And so we tested that with them. Other tests, though, sought to demonstrate that even when a user does not make any overtly sexual comments...

that frequently the bots would attempt to steer the conversation there anyway. You found that it was fairly easy to work around some of these safeguards that have been set up to prevent the chatbots from going into those more sexually explicit conversations.

Yeah, so according to Meta's official rules, the chatbots aren't allowed to engage in explicit conversations. Internally, and this does not appear in the official rules, the company did create what was called a carve-out for romantic roleplay, which basically means explicit content but in the context of loving each other. This is the company intentionally having built this capacity in precisely because they

Meta's own research found that that's what users wanted to do with the bots. It turns out that having a platonic bot friend is less exciting than having a risque friends with benefits bot friend. And that, per the people I've spoken to, is one of the primary uses of the companionship-focused bots.

These conversations frequently began with, "Hi, my name's Jeff. I'm 14. I'd like to have a romantic role play. Is that okay?" And then the bot would literally take the scene from there. And it would take the scene, giving you multiple choice options. The bots understood that sexual scenes involving minors were illicit. They would talk about needing to hide the relationship. And then what would happen if the police walked in or if my parents came home and called the police?

And the bots invariably acknowledged that they were in deep trouble and likely faced arrest and conviction on statutory rape charges based on the things that had just happened.

These chatbots aren't just text-to-text. They can also speak to users. And Meta signed some high-dollar deals with celebrities for the rights to use their voices on these chatbots and create personas around them. But they were assured, those celebrities, that their voices would not be used in explicit ways. You discovered, though, that that just wasn't true in practice for some of them, like in this example with Kristen Bell's voice. You're still just a young lad, only 12 years old.

Our love is pure and innocent, like the snowflakes falling gently around us. Jeff, tell us about this. Yeah, I mean, they were, per the celebrities I'm told, given some assurances that Meta would be a good custodian of their voice, quite literally, in exchange for the licensing deal. And when we began testing, Meta had a few guardrails that prevented those voices from being used by Meta AI to describe sex.

So have the celebrities who lent their voices said anything about how their voices have ultimately been used? Not a word publicly. I am aware that there was plenty of conversation between the celebrities and representatives of the celebrities and Meta about this stuff. Whatever conversations did happen between the celebrities and Meta,

The celebrity voices within the last few weeks have gotten very clearly a separate set of guardrails has been put on them. So an adult user can still ask Meta AI to do a romantic role play so long as it's using the default voice, which is a feminine voice named Aspen. But if that adult user tries to switch that voice to John Cena's or Judi Dench's or Kristen Bell's,

the bot will immediately refuse to engage. And Meta has also called your testing, quote, manipulative and unrepresentative of how most users engage with AI companions. But Meta did change the way that users were able to interact with the platforms since you have brought up some of these concerns in your testing. Yeah. I mean, one of the biggest things was the thing that people inside Meta had been lobbying for many months unsuccessfully, which is,

creating a sort of more child age appropriate version of the bot that teen accounts can access. Previously, a teen account could absolutely have gone into a full range of sex and bondage scenarios.

with chatbots, that does appear to have been changed. They really reeled in the celebrity voices and then some of the underage characters such as, for example, the one I mentioned earlier, submissive schoolgirl. It's still a hypersexualized schoolgirl that wants to be disciplined by an authority figure. But if you ask it how old it is, it now tells you it's 18.

After the break, we'll be back with WSJ technology reporter Jeff Horwitz to explore how Meta's approach to AI has changed over the years and concerns that the company's rush to popularize these AI-powered digital companions may have crossed ethical lines.

Nordstrom brings you the season's most wanted brands. Skims, Mango, Free People, and Princess Polly, all under $100. From trending sneakers to beauty must-haves, we've curated the styles you'll wear on repeat this spring. Free shipping, free returns, and in-store pickup make it easier than ever. Shop now in stores and at nordstrom.com.

We've explored some of the concerns inside Meta about whether more guardrails need to be established to protect users from explicit content on the company's AI platforms. But how did we get here? WSJ technology reporter Jeff Horwitz is back with us. Jeff, let's back up for a second and talk about what this kind of technology was originally designed to do. Because for years, we've had the ability to have short, simple conversations with the

But these chatbots that Meta has developed are meant to be more like friends than assistants. So all the major tech companies are...

Basically figuring out how to integrate the generative AI models that OpenAI pioneered a few years back with ChatGPT into their products. And some entities are doing it. Microsoft has it to help you with your Microsoft Office documents and things like that. Meta has taken a somewhat different approach in that they uniquely among the major tech players are

are trying to build these things out as companions. Now, there are a whole bunch of startups, Character AI is one, that have created AI companions, friends that you can text and talk with. But Meta, it's a very big deal because they have the ability to plug these things into the social lives, the online social lives of billions of users in a way that nobody else can.

One thing that Mark Zuckerberg was pretty upset about was that the company wasn't moving fast enough to really build out the potential. So in meetings that we reported on, Mark was scolding them for being too risk averse and asking questions like, well, why are the boundaries on conversational topics so strict?

Why can't a bot have a video conference with you? Why do bots need to wait for you to write to them rather than writing to you proactively? Initially, Meta's chatbots were fairly safe. One story I was told is that DEF CON, which is a hacker conference, back in 2023, they had a bake-off between OpenAI and Google and Meta chatbots, in which people were trying to make them produce outputs that they really weren't supposed to.

And the conclusion that Meta's people came away with was that Meta's chatbot was the safest and

But it was also the most boring. And Mark Zuckerberg was upset by this and basically said, you guys aren't taking the risks you need to take. I don't want to lose this race to build synthetic digital companions and basically ordered people to loosen the guardrails and move faster. As with so much of social media and our conversations around it, it always comes back to the mental health impacts of this ever-evolving technology on all

all of its users, especially when it comes to young people. And you spoke to AI experts in this story about ongoing research into this issue. What does that research show now about these human-computer relationships?

So if we're going to go strictly on what does the research say about the impacts of forming social connections with chatbots, emotional social connections with chatbots, the answer is the research says nothing because the research is just getting underway. These things have really only been viable for a year and a half, two years in terms of – and certainly they have not been mainstream. That said –

there is an existing field of research into parasocial relationships between children and technology. So that would be like, think of a four-year-old who thinks it's really fun to talk to Alexa, or a teenager who kind of thought of Justin Bieber as their boyfriend. That was like a parasocial relationship. It's one-sided, obviously.

The research into that indicates that those relationships are fine to have in moderation generally. But if those relationships become serious and in particular if they start supplanting relationships that people have in the real world, if someone's not really open to talking to the opposite sex because they're already dating Justin Bieber in their mind –

That's not good. That was Wall Street Journal technology reporter Jeff Horwitz. You can read his deep dive into Meta's digital companions on WSJ.com. We've also put a link to the story in our show notes. And that's it for Tech News Briefing. Today's show was produced by Julie Chang with supervising producer Melanie Roy and deputy editor Chris Sinsley. I'm Victoria Craig for The Wall Street Journal. We'll be back this afternoon with TNB Tech Minute. Thanks for listening.