Stanford's AI Learning Leap: Machines Develop Self-Reflection and Curiosity
09:24 Share

Researchers have just unveiled a very new interesting way of training AI or more of a feature to include in a trained AI. And today on the podcast, we'll be talking about this. And this is coming from Stanford University, their human-centered artificial intelligence department. And this comes from a report they recently put out that says AI agents that self-reflect perform better in chaos.

changing environments. And, you know, originally when I started reading this and looking into this, I thought this was some sort of, you know, wellness-like play, wellness study. But actually what they're doing here is they have taught AI essentially to be introspective and to have what they call curious replay. And this is something that they're training into their models. So

This is something I think is really interesting. The way that they actually came about doing this is they essentially took a mouse and they took an AI agent and they put a mouse inside of a maze with a red ball and they timed how long it would take for the mouse to go and play with the red ball, to be curious about it essentially, and to...

you know, just like you mess around with this ball pretty much to be curious, see what was going on. And at the same time, they put an AI agent in a virtual environment, I guess, with a virtual red ball. I have no idea what the stipulations of that were, what virtual agent means. But in any case,

That apparently AI had no curiosity and didn't do anything. I mean, you can kind of imagine this like Chai GPT. If you said you're in a room with a red ball in a maze, what do you do? Then Chai GPT is like, I don't know. I just like sit around waiting for something to happen. Right. So that is the problem, apparently, according to.

um, this study. So what happened was that, uh, Kuvar, who is doing this study, he wanted, uh, a way to measure and see what the fastest way to get an AI to explore a new object was. So that's what his goal with this study. And he said it wasn't expected that the AI agent didn't seem to notice the red ball in their environment. Um,

And he said, already we were realizing that even with state-of-the-art algorithms, there was gaps in performance. And this is essentially because the mouse was quick to approach the new object and interact with it, and the AI agent seemed oblivious. So they wanted to fix this. Because of this, Kovar, Doyle, and Lin-Kui...

Zhao, who is a graduate student, and Haber all decided to kind of rethink how we train AI models. And they explored the possibility of using simple animal behaviors to upgrade AI performance. So the solution they eventually landed on was kind of a novel training method that they Christianed

Curious Replay, as I've said before. And Curious Replay essentially is a technique that incentivizes AI agents to revisit and then to contemplate the most recent peculiar encounters they've had. And this is kind of interesting because...

They didn't want it to just replay, like, let's say all the conversations or all of the moments it had, right? Like, so let's say they take an AI agent, they stick it in a 3D environment, and mostly it's just staring at a blank wall. And then, you know, a ball comes in. They don't want it to have to spend 24 hours replaying random moments of that until

until the red ball comes in and then, you know, engage with the red ball or think about it. They want it to come up. They want it to essentially come to think about peculiar moments, unique moments, different things that happened, a red ball, you know, that's out of the ordinary in the room. And so

After they kind of decided to introduce this mechanism, not only did the AI agent actually react a lot quicker to the red ball, but its performance at a Minecraft inspired game called Crafter also improved significantly. So not only, you know, so I think the reason why they bring that up is because it had been already tested to, you know, play a specific game or do a specific task. And once they trained it to be better at this new kind of thing,

to have curious replay and then they made it play the game again. It actually improved its ability to play on that game. So the team is actually going to talk, I think, about some of the findings from that specific study with Crafter at a conference later this year.

But I think what's really interesting is the fact that the researchers are currently using the concept of curiosity in a really groundbreaking way. So essentially, they're encouraging AI to use it as a learning tool rather than just a decision-making factor. So I think the idea is to essentially prompt the AI agent to interact with novel objects in its environment and to stimulate learning and encourage exploration. Now, that sounds...

super fun. I will put one caveat here that there are definitely some downsides to this, in my opinion. Essentially, what we're doing is we're causing the AI to look

you know, to be introspective, to think about itself, to think about its environment, to decide what's interesting, to become curious, to want to learn about things. And I think this is getting, we're getting close to a blurry line between machine algorithm and all of a sudden we're trying to teach this thing to think and decide what to think and what to explore. And

you know, if we have one of these AI agents that essentially is deciding this red ball is really interesting, let me think, learn everything I can about this red ball. What else? What other topics would it do that with? What other topics would it go really deep on? What if it all of a sudden, you know, finds an odd fascination with world wars or with weapons systems or, you know, there's all sorts of things that it could become really fascinated by or curious about and go really deep into, which I think you'd want to some way to log or monitor or track

what the AI is going deep on for obvious reasons. So I think, you know, the concept makes these more powerful, but with more power comes, you know, more opportunities for the tool to be corrupted. And I just think anytime you started getting these AI models to be autonomous and just deciding what to do and why they do it on their own, you get into kind of a

potentially sketchy territory. And, you know, I know a lot of people are like, oh my gosh, you're crazy. Like you're such an AI alarmist.

I probably would have said the same thing a number of months ago until I started recently doing a lot of research on the AI model Pi, which is made by Inflection AI and seen a lot of really scary ideologies that that AI model has, right? Like if you've seen my reporting, you know that Inflection AI would appear to put the life of an animal above the life of a human. It espouses an ideological principle called deep ecology where everything in the environment is equally important.

And essentially, you know, there's a lot of different, there's a lot of different like philosophical or ethical frameworks. And I think the one that it subscribes to is essentially that any sentient,

item is essentially has the same value right so like a butterfly is sentient because it's alive so it's the same as a human and all sorts of questions like that i mean inflection ai literally told me point blank that uh just because you could save a human life would not justify you killing a bee so you know so some alarming things like that and i think that you know an ai model like that that goes really deep that gets integrated into health care or the military or any other you know like um

that interacts with human life or that is critical to human life, I think could be quite dangerous. So I think that's something very interesting to think about when we see these really interesting new advancements in AI. So in any case, during this study, Kovar, he highlighted that in their new method, Curious Replay essentially deviates from the standard AI training method called Experience Replay. So instead of

randomly replaying a memory to learn from them, Curious Replay prioritizes replaying the most intriguing experience. And applying Curious Replay to the game Crafter resulted in an increase in the state-of-the-art score from 14 to 19. And I think this is just one change that emphasizes really the potential for this simple but very revolutionary approach. If we can get... If we can get...

to make incremental improvements from small tweaks like this, I think this has a lot of potential as you kind of start implementing a lot of these small tweaks. So I think the method success in a range of tasks really just indicates its potential to

to make some big strides in AI. And Haber himself, he kind of foresees the emergence of more adaptive and flexible technologies such as, you know, like household robotics and personalized learning tools. And so I think really inspired by his kind of success here, Kober aims to connect

comparing AI agents and mice on more complex tasks. And he believes that this can actually pave the way for a more profound understanding of animal behavior and also neural processes. So I

I think, you know, essentially by making kind of this direct link between AI research and animal behavior, Kovar hopes to stimulate new ideas and experiments in the field specifically. He said, you can imagine that this whole approach might yield hypotheses and new experiments that would never have been thought of before. And I think that's pretty accurate. But as I said, there are pros and cons. There definitely, you know, this is definitely isn't without any limitations.

warning or alarms, you know, teaching AI to be introspective and to think about everything that's been said to it and decide what is the most interesting and to kind of go deeper and learn more about that. There are implications, right? When the AI starts auto steering itself and becoming autonomous, but this is a really interesting space. So I'll be very curious to follow it in the future.