Machine Learning + Procedural Content Generation with Julian Togelius and Sebastian Risi
47:59 Share

Hello and welcome to SkyNet Today's Let's Talk AI podcast where you can hear from AI researchers about what's actually going on with AI and what is just clickbait headlines. We release weekly AI news coverage and also occasional interviews such as the episode today. I am Andrey Karenkov, a third year PhD student at the Stanford Vision and Learning Lab and the host of this episode.

In this interview episode, we'll get to hear from two professors who are co-founders of a company, Model.ai, which delivers AI tools for video game studios and combines profound knowledge from game development and machine learning to accelerate game development and enhance player engagement. And these two professors also co-wrote a paper, Increasing Generality in Machine Learning through Procedural Content Generation, which will be the focus of our conversation.

Firstly, I'll introduce Professor Julian Togelius, who is an associate professor at the Department of Computer Science and Engineering at the New York University Tandon School of Engineering. He works on artificial intelligence techniques for making computer games more fun and on games for making artificial intelligence smarter. Or in other words, he asks what AI can do for games and what games can do for AI.

He is also the co-author of a book, Artificial Intelligence and Games, and of a book, Procedural Content Generation in Games. We also hear from Professor Sebastian Risi, who co-directs the Robotics, Evolution, and Arts Lab, or RIEL, at the IT University of Copenhagen, and who aims to make machines more adaptive and creative, a way of research that is focused on computational evolution, deep learning, and

and crowdsourcing with applications in robotics, video games, design, and art. His work tackles questions such as, can we create life learning machines that continuously acquire new knowledge and skills? Can we create machines that learn from and work together with humans to solve tasks that neither humans nor machines can solve by themselves? And again, both of these professors are co-founders of the company Model.ai.

So thank you so much, Professors Tokelius and Risi for making the time to be on this episode. Thank you for inviting us. Yeah, thanks a lot for the invitation.

Okay, great. So let's go ahead and dive straight in. So you have both worked on a ton of interesting stuff, but today we are mostly focusing on this recent survey paper, Increasing Generality in Machine Learning through Procedural Content Generation, which came out in Nature Machine Intelligence earlier this month and a version of which is also publicly available in Archive.

and in which you review existing work on procedural content generation or BCG and its current overlap with efforts in machine learning and promising new research directions such as procedurally generated learning environments. So first, maybe you can set the stage with a quick explanation of procedural content generation as you understand it, and also kind of explain what was the motivation for writing this paper at this point for both of you.

Yeah, so procedural content generation is a practice and a set of techniques and tools that have existed in game development since late 70s, early 80s of different kinds.

Basically, the idea is that a lot of games, for some reason, need parts of the games to be generated by the game engine itself.

So this started out back in the days when we just didn't have enough memory to create, to store complete game worlds and not enough developer time to create those game worlds. So for example, the game Rogue, which was created by two students at the University of California, Santa Cruz back in 1980, they wanted to create Dungeons and Dragons on a computer.

But they didn't want to program the adventures themselves. And they didn't have enough hard drive space to store the adventures. So they had to come up with an algorithm to create new dungeons that could be explored, complete with various enemies, with treasures, with differing layouts and so on. And then you created this game so that every time you played it, you had a completely new adventure to play that you'd never seen before.

So Rogue was a sort of underground hit, and it became, well, it's not what you would call a cult hit, and it became the predecessor of a whole series of games called Roguelikes. There are other games, such as Elite, a game from 1984 where you can traverse thousands of worlds,

and you're a space trader or space adventurer, space pirate. And it's really a vast sort of galaxy you can travel around in. And there are missions, there are other sort of species to discover. There's trading you can do.

You can be chased by the space police and the space pirates and so on. And this all fits in the memory into these very, very limited machines of the time. So this is basically because every time you come to a new star system, it's generated from a random seed.

And these set of techniques, they have been refined over the last four decades or so. And we now have a lot of games that feature some kind of generation of some part of the game, either levels or the vegetation or terrain or quests or something. And there has been a lot of research on that in academia.

In commercial game development, it's one of these various forms of procedure come generation are just like basic and standard forms of game development. But in the research world, there's been a lot of work to see how we can extend this and create completely new games or create games that can adapt to human player, create games that are endless in various ways. So, yeah, Sebastian, maybe you want to talk a little bit about the connection to machine learning.

Yeah, I can do that. So yeah, our motivation basically for then writing this paper was twofold. So the PCG community and the machine learning community have existed for quite some while, but they basically existed in these kind of parallel streams where PCG researchers did work and machine learning, but there was not too much intersection maybe between the two. Now it's more converging. And what we see is that both basically communities have

can benefit from each other a lot. So on the one hand, we're starting to incorporate more and more machine learning techniques in our PCG algorithms. So to learn, for example, some work we did with Model AI is working with the company King to learn automatically from existing levels, Candy Crush levels, and then create new levels using machine learning algorithms like generative adversarial networks. So that's one reason is using machine learning in PCG

Another big reason is to actually the problem of overfitting in machine learning. So we see impressive results of deep reinforcement learning in games, in Atari games, in games like StarCraft. But what we have found and other researchers also found out

is that oftentimes these solutions drastically overfit to the task at hand. So if you train an agent to play, for example, Breakout, and you slightly change the pedal size, it will completely fail. Or if you try to have a bot that plays StarCraft is trained on one level, if you change the level, it will completely fail. So even slight modifications will break these agents.

And what the research is pointing towards is that we can actually use PCG to make these systems much more general. So instead of, for example, training on one level, maybe you can procedurally generate the levels. And every time the agent plays, you give the agent a new level. And this forces basically the agent to learn a more general policy. And we have seen these for games before.

like Atari. We have seen these for more robotics tasks. So this kind of forcing the agent to not being able to exploit some elements in the domain, but actually learning general policies, PCG has helped a lot in those regards. And both of us basically believe that both fields could benefit a lot from each other. And the article was kind of to start a more closer dialogue between these two fields because they can both

help each other and making on one hand machine learning more general and on the other hand we can actually increase the power of our PCG algorithms by incorporating machine learning methods.

I see. Thank you. That's a very clear answer. For listeners, you can actually find the paper just by Googling for it. And fun fact, there are some example games that incorporate PCG in there that you can find in one of the figures. And overall, it's quite a readable paper. So I would say you can definitely go ahead and dig in yourself.

One of the games cited is actually No Man's Sky, which I've put in a decent amount of time and is a good recent example for sure. Yeah, No Man's Sky is like it's quite a few people were unhappy with it because they thought it would be something else. And it is like basically infinite mass effect. But what it is, it's just a fantastic art piece, isn't it?

Yeah, it's a wonderful place to just explore. Yeah. Maybe for people who don't know what No Man's Sky is, it's a game. It's a game about space exploration and space colonization, which is...

There is literally more planets in there than anyone could ever explore in a lifetime because it's all generated and they all have different types of fauna and flora. They have like things discovered that are not available on any other world. It's really a masterpiece of generation.

Indeed, and it's a great example of both strengths and weaknesses to some extent at PCG where you do get an infinite variety. But at the same time, the kind of difficulty with PCG is, as far as I understand, is you can only vary things so much because you have algorithms with a set of amount of things that can change.

And compared to hand crafting levels or things like that, you get more repetition and more sort of similar themes coming up. I think the challenge is basically you need to strike some balance between that you can easily search the space, but you also want it to be expressive.

So of course, the more you include, the more things you want your PCG generator to make, the larger is the space you have to search and the more also like junk you basically find in that space. So that's one of the kind of the trade-offs you have to do. And in this sense, this kind of

what we now call data-driven encodings or variable evolution is an interesting example where you can learn a lot from existing level that designers created and learn with machine learning from that. And that kind of narrows down the space of things that you might want to vary inside your game. And

And then it becomes even more complicated when you, so some work maybe Julian can talk about. Actually, Julian made some work where he actually also evolved the rules of the game itself, which I find quite interesting. Yeah. Yeah. So there's like, there's different levels of ambition and, and the more you want the generated content to be able to surprise you, the sort of

The bigger the search space, the more open-ended representation you need to create, and the bigger the search space you need to give it. But that also makes it very hard to control. Now, interestingly enough, in No Man's Sky, they didn't actually perform much in the way of search, as far as I can tell. So the algorithms powering the generation there mostly come from the graphics family of PCG algorithms.

So there's a lot of different noise and fractal algorithms. So things that come from the general same background as Ken Perlin's famous noise algorithm from the 80s and other fractals. So one message we really want to get across in our paper is that

For people who are doing game development, they should really start looking at algorithms that perform some kind of search or constraint solving in combination with other methods. So I've worked, and Sebastian has also worked, some of it in common, some of it separately for a long time,

on methods that use search to search for them as a generation method. Search could be things like standard simple greedy search algorithms, or it could be more advanced stochastic algorithms, evolutionary algorithms, particles for optimization and so on, to find content in a search space that optimizes maybe a whole family of criteria.

Recently, there's been a lot of advances in quality diversity search, for example, for this. There's a rich variety of interesting search algorithms to use here, and they should be used both for game development and also for creating new sort of environments for training for machine learning.

in particular reinforcement learning. And if you really have to boil it down even further, what is the main message for people who do reinforcement learning from a paper? Well, it is that, yes, many of you have started thinking about environment variation and use simple randomizers to randomize environment.

But there are so many richer and more useful algorithms out there, both from game development and from games research, that stand ready to benefit the reinforcement learning research.

I see. Yeah. And as someone working in robotics and keeping track of RL, I've certainly seen some of these works you cite as examples of this evolving trend of co-evolving the task and the agent that's trying to learn the task. And a lot of, yeah, I would say a decent amount of people are very excited about this direction. And that's part of what you cover in the paper. Yeah.

And you also cover you cover quite a bit, of course, so you won't get into all of it. So the paper covers a bit of a history of PCG and classic PCG covers data augmentation and domain organization. So also a bit of relation to machine learning.

AI-driven PCG and PCG for creating learning environments. And again, listeners can go ahead and Google for the title of the paper, Increasing Generality in Machine Learning for Procedural Content Generation, if you really want to dig into everything.

But maybe we can narrow down things a bit. And I'll ask in terms of the intersection of AI and PCG, you just mentioned RL and PCG as one example. But if you have any other kind of examples of a trend or even some specific results or papers that you found particularly interesting and exciting and real progress in that intersection.

You also mentioned it, and I think that's a really interesting, very promising direction, is basically the idea of generating the environments together with the agents themselves. So one really interesting example, I think, is by some of our colleagues. This is work by Uber AI Labs' poet,

that generates, um, the, the learning challenges like the environment, the 2d environment for robot, uh, and together with the, the, the controller and Julian followed up on this work and extended it to also, so games. And I think that's one really interesting direction because normally we always, um,

learning environments for these agents. And so we have to make sure that these environments have the right difficulty, they're not too easy and not too hard for an agent. And then we manually have to kind of maybe make a curriculum and deep reinforcement learning to allow these systems to learn more and more complex tasks

but the idea is in these automatic generation that you the algorithm itself will generate the challenges and make the challenges in the right way for the agent to solve so you have kind of the scaffolding where the environment allows the agent to learn and then you have more and more complex environments so you're trying to get this kind of

co-evolutionary system going. So similar to like a generative adversarial network that you have kind of both the generator that generates the environment and the control of the agent gets more and more complex over time. And you as the experimenter can kind of at some point ideally lean back and see these two systems get more and more complex with the agent ultimately solving challenges that you might not have even thought about.

And I think that's a very, very interesting challenge, not just only for games, but also for robotics tasks where the

could design new robotics tasks for the agent to solve. And that's also, I think, where we see that that's also an interesting intersection of algorithms from evolutionary computation that both Julia and I have a long history with, which kind of naturally lend themselves towards generating these challenges. And then you have a machine learning system, on the other hand, that can try to learn like a reinforcement learning that learns

to solve these challenges. So I think there will be a lot more to come kind of in this area, and we review also some of this in our paper. Yeah. I mean, the core idea is something...

that is sort of intuitively understandable and attractive to a lot of researchers is that, well, I mean, in order to learn a complicated behavior, complicated, well, the generally competent behavior is really crucial that you have the right tasks.

And where do you get the right tasks from? Well, during the development of life on Earth, I mean, there were no tasks given to us. All the tasks came from the rest of the natural environment, which co-evolved. There were a number of species that evolved together, and they, in a sense, provided the task for each other.

So the challenge here is that we can't just reproduce the evolution of life on Earth because we don't have the CPU power for that. And we're likely to never have that. But how can you sort of create a somewhat efficient algorithm that co-generates the tasks and the solutions? And that's where Poet is one of many possible answers to this.

The general thinking here of co-generating tasks and solutions, of course, has a lot in common with, for example, generative adversarial networks themselves, which most people listening to this are probably going to be familiar with. You train a generator, you train a critic,

Generative adversarial networks in themselves are just a variation in a sense of competitive coevolution, which has been around for a long time in biology and in evolutionary computation, where you have two or more species that are in an adversarial relationship with each other. There's some really cool work...

We go back to Danny Hill's work from 1991 on co-evolving sorting problems and sorting networks, which, for example, exposed this dynamic. I've been trying to use this work, for example, for game-playing agents. We have car racing neural networks that learn to race against each other and against each

and against attractive racing to try to get as far as possible. So this dynamic is really important. I think there's something really deep in there that we need some variation of co-generating problems and environments in order to get to more general intelligence.

because we just are not good enough at creating good tasks and environments ourselves. And we don't have resources for it. We need help from AI with it. But getting this going, I mean,

GANs are notoriously unstable. There are a lot of tricks you need to use to make GAN training work well. Competitive coevolution, you have very much the same problems. So PoET is one way of dealing with this by introducing a kind of diversity and matching certain problems to match to certain tasks. But I think there's a lot to be discovered in this vein. I see.

Yeah, definitely. It's actually now that there's always discussion of PCG for AI, it makes me think of how much I've used in a sense, procedural generation of

heaps of objects in the bin and simulation for robotics. We've used that. I've also been a co-author on a paper that used procedurally generated tools to learn tool use. So we had all these weird looking hammers for a bunch of variety. So I definitely see the excitement and I'm excited myself to not have to craft so many similar levels.

And on that note, I'm actually curious. So that was sort of an example of how PCG can be useful for AI and how it can further AI research by making more environments and contexts to learn in.

Are there any examples of the opposite of sort of AI for PCG? How is procedure generation being made more powerful or easier or faster leveraging machine learning and AI? Definitely. Yes. You want to start, Spencer? Sure.

Oh yeah, I can just start. So one example that also Gino mentioned is basically that you can use AI techniques in PCG to learn from existing content. So there's this class of methods now sometimes called data-driven encodings or latent variable evolution where you basically already have a lot of content and now how can you best

learn from this content to produce content that is different but still similar in a way. And so there you of course can take advantage of all the recent machine learning techniques that are able to learn from existing content. So one approach is

this latent variable evolution that Julian and one of his students invented is that you can train a GAN, for example, on fingerprints or we use in our work on Mario levels. And then the GAN kind of learns some of the structure in those levels. And then when it is able to produce those levels...

You can basically, you train this generator and then you can throw away the discriminator. Then you have a generator that you can feed, you know, a random vector and it outputs a random level segment in the case of Mario. And then what you can do is you can use actually a search method to search the space of what this generator learned. So you can search inside of the space of a GAN through, for example, methods like artificial evolution, right?

So you're searching for a specific latent vector that you feed into the GAN that can then, for example, generate a specific type of level segment with a certain type of difficulty. So this way, basically, you can generate it. You don't need to hand code the representation, but it's actually a representation that is learned from the data itself. And every level segment you generate is kind of a reasonable level segment and not some total garbage segment.

And this way, you're kind of benefiting from having this kind of machine learning system that can learn these regularities in the levels themselves. And you can apply this to, we apply this to Mario, you can apply it to fingerprints, you apply it to Candy Crush levels. And this really allows these PCG algorithms to search through the space of interesting content and also functional content information.

much more easier. So in case of, you know, if you generate a face, maybe it's not so important if there's a pixel wrong. But if you try to use PCG, then you may have to make sure that the things you generate are actually also in the context of games that they're actually playable. And so machine learning can help you there that you can, instead of designing these encodings yourself, you can actually learn them from the data if you already have existing levels available.

Yeah. So this is what Sebastian just mentioned is this interesting combination of using, you learn a representation, right?

from looking at existing content, such as levels. But we often talk about levels because we often do levels. It's nice and simple and 2D, but you could also be, you know, quests or characters or puzzles or something like this. Now and then you use search within that representation. So technically you in evolution of invitation speech, you use the generator as a genotype to phenotype encoding.

But you can also do this without search. In many cases that actually works relatively well. So we've seen examples of using, for example, autoencoders to generate different layers of StarCraft levels. So from the topology you can train an autoencoder to predict where the resources would be with some success. We've also seen simple things such as engrams being used to produce Mario levels.

recast them as like a string generation problem. We've also seen the LSTM networks being used in Mario levels in the same way.

The big problem here with just using pure machine learning for generating game content is that you often don't have enough game content to train from. You don't have 5,000 phases. You have like 20 Mario levels or something like this. So there's been a number of different attempts to sort of deal with this.

We had a paper out recently where we combined a kind of generative adversarial networks with self-attention. And we also combined this with a bootstrapping algorithm, where if you produce new levels that fit some kind of playability criterion, they get added back and that are different enough from what you already have, then get added back into the training set.

So you start with only maybe five levels, and then suddenly you can create a large diversity of levels that are all playable and look reasonably good. So that's another way of dealing with a level generation problem using supervised learning. Now, another direction that we've also taken recently is using reinforcement learning.

So the way you recast the content generation problem here is that you are now controlling the level generation agent and learning a policy for the level generation agent is a reinforcement learning problem.

And we've successfully been able to train generators for, for example, Sokoban levels and a simple version of Zelda. And they take a lot of time to train, but then they generate very, very fast. The problem with this is that you don't... It's very easy to get the kind of a collapse where you train a generator that only generates the same level again and again and again.

So for that, you have to sort of force it to be forced to generalize. So we sort of in a weird kind of inception thing here, we are facing the over the overfitting of reinforcement learning problem again in trying to train agents that could create environments that would reduce overfitting and reinforcement learning. Sorry for the circularity of that sentence.

But you can do this. One of the main tricks for making this possible is to limit the episode length to very, very short episodes and drastically randomize the starting positions. And you're training an agent rather than to create complete levels. It's through training it to make small improvements from any given position.

And then you can end up with something agents are actually quite generally competent. So these are like as we summarize, we can talk about supervised learning and unsupervised learning with or without search. And we can also use reinforcement learning for generation. So there's a lot in it here.

I see. Yeah, very, very interesting to hear that sort of overview of the ways in which AI is being leveraged for BCG. And it sounds like sort of there's progress being made and really new inroads in that sort of work.

You also outline five main open challenges or opportunities in the paper. So relating to, as you noted, there are some kind of problems or difficulties. And so some of these you mentioned are challenges.

In a paper, learning from limited data, generating complete games, lifelong generation for lifelong learning, new PCG-based RL benchmarks, and from simulation to the real world. And as a roboticist, the last one, from simulation to the real world, definitely does make sense. So probably we can't go into all of these challenges so much, but...

I will ask when people who haven't worked with PCG see new and exciting things like Poet or the Rubik's Cube results from OpenAI, what do you think they kind of miss in terms of the challenges and limitations of PCG for creating more environments and opportunities for learning?

So I think one thing we currently don't have yet is a system that can really create things in a truly open-ended manner.

So I think even with Poet, which I think is really cool, but of course you're still kind of restricted in the types of levels, like the terrains that Poet can make. It will not suddenly make a 3D world or something because it's constricted in the type of encoding that you use. So I think that's one thing, and that's also a big...

or like a lot of people in the evolutionary community are interested in this kind of this open-ended evolution, open-ended creation. Can we really create a system that can, in an open-ended manner, create new and new, more challenging tasks and agents that can design its own, maybe, representation and coding? So that's definitely...

we're definitely not there yet. And there's some very interesting, um, you know, work in this, in this area, but it's, it's far from, from being, um, basically, basically solved, uh, like creating a system that can also create this, this really large diversity while still being somewhat tractable. Uh, that, that's, I think a really interesting research challenge. And this idea of open-endedness, uh, is a, is a challenge. I think that also people like, uh,

Ken Stanley worked a long time with him. There's some really interesting keynotes of his way he's advocating for this challenge. So there's a lot of room there for interesting research. Yeah, I don't think this is a challenge that we're ever going to completely solve. It's not that kind of problem that we're going to have a solution. It's like AI. I mean, obviously, we're never going to

solve AI. I mean, not more than we already did many times. So open-endedness is like, it's this huge meta problem, creating spaces in which agents can really learn to, there's no end to the complexity of the behaviors they can learn. And environment generation is a key problem of that.

And I love this, and I expect to be working on this until the day I die in some way. If we look at more tractable problems, one big issue in how to use ML to improve PCG is that we...

is how to deal with the scarcity of data. If we want to make this available for game developers, or even if we want to make this more useful for... We have designers, but we also have people who design learning environments for reinforcement learning, for example.

So what you would want to do is that you would want to basically sketch two or three different game artifacts, say levels or say quests or something. And then you want to many, many, many more for them like that. And the problem is that most of our current self-supervised learning algorithms expect thousands of instances of data to train them.

How do you reliably capture a style of something and what makes it important from, say, three examples? So that's a big challenge. And there is a lot of work in few-shot learning, of course.

throughout machine learning. But a big difference in trying to use machine learning for, in particular, supervised learning and self-supervised learning for game content is a game content usually has

some kind of functionality requirements. So if you train your GAN to generate faces and your faces are slightly off, they still look like faces. They're just slightly off. The eye row is not there or the ears are asymmetrical or something like this.

If you generate a Super Mario Bros. level and there is suddenly a wall you can't get over or a gap that is too wide, then the level is not slightly off. It's unplayable. Its value is not just like 99 out of 100. Its value is zero.

So learning, teaching models to respect functional constraints is a very, very hard problem that I think there's a lot of progress that could be done, especially in learning these functional constraints from just a few examples. I don't know how to do that, but it's a relevant research problem.

I think that one way forward on this is some kind of pre-training. So if you look at language models, you often talk of pre-training and then fine-tuning. And one thing that I would love to see is being able to pre-train a level generator on...

levels from 100 different platform games. We scrape everything from the Nintendo NES, and then we fine-tune it on one of them, Kid Icarus, or something like this, or Mega Man. And then from just a few examples, we learn the style and the specific functional constraints of that particular game. I think that will be very useful both for game development and for machine learning research. And it's very much unsolved.

Makes sense. Yeah, that would be very cool to see something pre-trained on a whole bunch of games. And then you can really, with just one example or a few examples, make a lot of new content for a particular type of game. That is a paper I would like to read for sure.

Now, you mentioned this kind of thing probably being useful for game development. So maybe we can switch gears a little bit as we wrap up here. And I wonder if you could actually share anything about what you've been able to do concretely outside of research in your company in Model.ai as far as actually building tools and

building new techniques for game developers to leverage AI or PCG or the combination. If there's any examples you can give. So one thing we're quite excited about, I think, is this idea of this. We also posted something on Twitter about this kind of glitch finder.

Or we also call it sometimes flexible play trace is basically the idea that can you automatically play test a large game where there are a lot of different possibilities of what you can do. And so there we also, so for example, you want to know if you change something about the agent, can it still complete the level? You change something about the level, can the agent still complete it?

You have a race car game, you change like the friction of and you want to test all your hundred cars, you know, can they still complete the race track?

And so that's one product we have that automatically tests these games using like a combination of evolutionary and machine learning techniques to quickly explore the space and see is the game still playable or otherwise report back to the designer. Here, there's a problem. You change something and now the level is not good.

anymore. And we're seeing a lot of interest in those kind of tools because they can really cut down on the number of hours you have to spend debugging your system and the number of hours you have to playtest them by having some part of it at least automated. Yeah.

I think the other interesting project that we can talk about, we have a bunch of things we unfortunately can't talk about yet. The perils of dealing with game developers is our work on using exactly the latent variable evolution approach that Sebastian talked about before with Candy Crush.

So we've been Candy Crush as a game. I guess everyone knows about this game because it's been so wildly popular. It still is wildly popular. There is something north of 4,000 levels for this game, and many people would assume that that would be enough.

But it's not enough because people keep playing Candy Crush and they keep demanding new levels. So we've been working and we have a paper that's actually being presented at this year's conference on games, which is going on as we speak.

where we were working on a method for learning from existing Candy Crush levels to generate new Candy Crush levels. And it is quite exciting because these went through a bunch of internal playtests, and developers are happy with it. The levels have the right level of difficulty. They have the right...

They have the right sort of patterns in there and so on. And the use case for this is that developers inside will basically ask the system for new level sketches and then they will go through it and edit it a bit before they release it to the game playing public. And it's very cool to be able to build a solution to contribute to such a successful game.

Definitely. Yeah, very cool. I think if any of the listeners are playing Candy Crush or Candy Crush fans, that's an exciting thing to hear. Now, I guess I'll just throw in one last kind of more fun question, which is more personal as someone who plays a good deal of video games and even dabbled in game development. Something I've often wondered about is right now, most games have a

characters, you know, non-playable characters who have, let's say, static AI. So they use various AI techniques, but they are pre-coded for certain behaviors. They don't necessarily evolve or learn using any sort of machine learning, reinforcement learning. For the most part, it hasn't been used in game development as far as I've seen. So I'm curious if either of you can speak to if you think maybe

this will become more of a thing now that AI has been making so much progress? Are people going to start experimenting with it more? Or is this just kind of too hard and, you know, anything that involves learning is too chaotic and that's why it will maybe not happen? I have like, you know, too many opinions on this. Yeah. I think for a long time, um,

Any kind of AI that involves learning or even search in a more advanced form, states by search, has largely been absent from commercial games for controlling, in particular for controlling NPCs, largely because the current game design thinking and the current game design patterns don't allow for it.

So we have for a relatively long time been able to train behavior that could have made their way into games, into video games of various kinds, but we don't have the design for it.

So a lot of designs are based on the absence of AI. If you look at when all the main sort of game genres we play from like, you know, real time strategy to first person shooters to turn based strategy or room playing games and so on, they were all developed back when we did not have enough AI methods.

to train agents or to build agents. I mean, we've also had huge advances in search, like what they call a tree search, in only the last 15 years or so. And therefore, I mean...

there's no real use for it in many games. Take a standard sort of shooter, a campaign model standard shooter, you will typically see an NPC on screen for like five seconds or so, and there is no room for like any kind of intelligence there. It won't really show. It's sort of wasted energy to use reinforcement running there.

Or if you take other games where basically reinforcement learning happening during gameplay would actually...

It jeopardized the whole design of the game because the sort of behavior that the designers can't predict could be really, really bad and could sort of make the game unbalanced. So I think we need, now that we have these mechanisms and we have better understanding of them, we need a new design that builds on the existence of reality. That's my main goal.

feedback on this. My main reason for why we haven't seen this in games. But things are about to change. Sebastian, do you have like maybe complimentary perspective?

Yes. So I think also, I also think, I mean, one is like designing completely new games with these machine learning techniques, I think is a really exciting area. Like, for example, one thing I think is very exciting is games like AI Dungeon, which you could not imagine, you know, would be possible without it. But I also think that we will have better AI techniques for existing games like that can fast adapt to the player. Like there's this

I mean, I can imagine recent advances in fields like meta-learning that learn based on a few examples, could allow agents in games to learn from a few examples from humans to adapt.

And then there's also some recent work of our lab that I'm quite excited about, actually, that started in robotics. But I can see that very much applied to games where we basically combine machine learning, reinforcement learning, with evolution, where we give these agents instincts of actions they should not perform.

So you as a human know, you know, there's a certain, you have a certain instinct and you know that performing this action would likely not lead to a positive outcome. So I could imagine in these video games, you can give agents co-evolving this instinct that would...

you know, hinder them in breaking the game in a certain way, like killing the main protagonist or doing something that would basically end the game and then still allow them to learn, but allow them in kind of a safe way of doing exploration in reinforcement learning. So I think

That's an approach I could imagine that we applied to robotics tasks before that could be used also in games to allow the agent to learn, but allow them to learn in a kind of safe way that would not break the flow of the game. But in general, I agree with Julian that we need to just also rethink

How do we design games using these AI techniques? And because we can make so many cool new games and we're kind of stuck right now in making all the same games over and over again, while with these new techniques, we can actually make something really innovative now and that I'm quite excited about.

I see. Yeah, that makes total sense. Hopefully the democratization of AI tools will enable some indie developers or younger game developers to try out really radical new things and see if we can actually leverage AI in games.

So with that, we'll go ahead and wrap up this episode. Again, we've been discussing the paper Increasing Generality in Machine Learning for Procedural Content Generation, which you can look up and dig deeper into yourself.

Thank you so much for listening to this episode of Skynet Today's Let's Talk AI podcast. You can find articles on similar topics to today's and subscribe to our weekly newsletter or similar ones at skynettoday.com. Subscribe to us wherever you get your podcasts and do not forget to leave us a rating if you like the show. Be sure to tune in to future episodes.