Machine Learning for Art with Google's Emil Wallner
29:57 Share

Hello and welcome to Skynet today's Let's Talk AI podcast where you can hear from AI researchers about what's actually going on in the AI and what is just clickbait headlines. I am Andrey Krenkov, a third year PhD student at the Stanford Vision and Learning Lab and the host of this episode. On this interview episode, you'll get to hear from Emil Wollner, an internet educated independent machine learning researcher and a resident at the Google Arts and Culture Lab.

As a resident at Google, he's using machine learning to explore art and culture. And part-time, he applies machine learning to logical tasks such as programming and mathematics. Thank you so much, Emil, for making the time to be on this episode. Cool, yeah. Thank you. And thank you for reminding me.

I'm excited to have you. So just to get going, kind of an exciting incident for why we wanted to have you on is that you created this project MLArt.co, which is a showcase with creative machine learning experiments. So maybe you can just give us a quick overview of what this project is and kind of what motivated you to create it.

Yeah, no. So it started out as a research project. I was doing research to figure out what to do next. And I thought I had a pretty good overview of what the kind of creative machine learning scene was. But I started making this list in a Google sheet. And after I reached a couple of hundred projects, I realized, wow, I only knew about maybe 10 or 20% of what people are doing. And then I realized that, hey, this might be useful for other people. So...

Then I decided to turn it into a website. So I started Googling no-code websites because I didn't want to spend too much time developing the actual software. And I found a website where I can basically take a spreadsheet and turn it into a website

And that's when I thought that it was very useful because you can get a visual overview and also the descriptions. And since I had all these categories and everything, it was fairly easy to navigate from from the start. Fantastic. Yeah. So listeners, while you listen, if you have a moment, you can actually go and visit MLArt.co and see it for yourselves.

It's basically a big curation of various ways in which people have used AI, let's say for creative purposes, often art. And you can see sort of a grid of different things people have done. So I'm curious when you got started, what sort of things did you discover that were new to you and what sort of things do you expect and sort of already familiar with? So I think the new things that I saw were

I mean, in general, a lot of the kind of the key categories I was familiar to, but there was a lot of projects within these categories and also a lot of different technologies that I weren't aware of. So, for example, Raphinec Anadol, he has these very lovely animations. But first, I didn't really know what kind of what's going on down under the hood. But most of it is based on using guns to produce

interpolations over different data sets. And then he uses 3D animation software such as Houdini to kind of create these very lovely visualizations that you can project on buildings in high quality. There were other projects that, for example, with Irvin Dressian and Maria Varspanen, who uses facial

face recognition on pieces of sand grains. So they have a robot arm that takes different pieces of sands, look at it and takes a picture. And if it recognizes the face, it kind of adds it to the collection. So they have this weird looking people that are made out of sand randomly, which I thought was very lovely. And another product that I didn't know about and kind of a niche is also different kinds of robots. So

Jouzy Lou has a robot that takes a picture, so it's an image to text model, so it translates CNN features to RNN features, and it lives on a beach. So it's kind of looking at the ocean and the beach, and then it's translating these images into text. And then inside this little robot, it has a pencil that draws these kind of poems on the sand, and then it just moves along. I thought that was very poetic and beautiful.

Definitely. Yeah. And I think the website has what, like hundreds of these unique examples where each one is very different. And it's very interesting to see how many unique things there are. Having created the spreadsheets and this website, I'm curious to

Do you have some sort of high level categorization of all of these? Are there any sort of like, you know, three, four or five discrete types of ways in which people use AI for art or creativity?

I would say that a lot of the art disciplines that we have, kind of normally have an equivalent in machine learning. And I think the most basic ones are vision, language, audio, motion, UI and tools. So if you look at vision, which is the most popular one at the moment, you see style, gowns and first-order motion. Those are kind of the three categories that are most popular at the moment.

And then you have a lot of small subcategories within vision and within language.

We see a lot of GPT-2 and 3 from OpenAI and still a lot of LSDMs are used. If you look at the trace, what's nice, there's about 370 projects on the site. You can categorize by when they were created. And if you sort them, you can see that in the early days, it was more Marco chains, LSDMs, and then you can kind of see the quality improve there.

over time. And if you look at sound, it's not as popular as vision. So we don't have kind of the gun moment in sound yet, which I'm still hoping for. So what we can see with the vision is that it's really developed a new aesthetic and a lot of new things that didn't exist before, before machine learning kind of entered the art scene. In sound, there are examples of

For example, the latest one Google developed is called DDSP, differentiable digital signal processing. And that's when you can kind of translate dog sound to a trumpet sound.

And I think that has a lot of potential to create new sounds, just like GaNS created new aesthetics. And I think also WaveNet and variational autocoders in terms of sound synthesizing has potential, although we haven't really seen that taken off yet.

On the other side, this kind of sound is split in two, I would say. There's one, the act of creating a sound itself, and the other one is organizing the song. So that could be melodies. And often these melodies are constructed in MIDI files. So these are often constructed with language models such as LSDMs.

But I think there also, sometimes it can be hard to really understand what's novel within these songs. You need a very deep understanding of music to really appreciate what's going on there. And I think in motion, the most popular one right now is PostNet. So we see a lot of fun things within dance.

So Maya Mann has made a lot of these really fun experiments where she dances in front of the camera and then you have words popping up and she creates poems with it and all sorts of fun things. And then when it comes to web experience and user interfaces, early on, say five, six years ago, UMAPs

were super popular, so taking a lot of high dimensional data and making it accessible to a lot of people. And now kind of what we're seeing more and more is a lot of APIs and especially tools like Runaway ML are because they expose these different models and people can play with them. We see a lot of creativity in terms of user experiences.

Another example of those is Basenet. Cyril Dijon at the lab as well, at Google Arts and Culture Lab. He made recently an app where he kind of drags and drop anything. So you can take a picture of, say, a tree and it just extracts the tree and then you can just drop that on the screen. So things like that are becoming more and more popular. And in terms of, I think, physical tools,

we see a lot of automated classifiers to do a lot of fun things.

So a basic example is that you sit down and you press a button. It takes 100 photos. You stand up, it takes 100 photos and then it automatically creates a classifier and then you can do turn off lights or create games with these automated devices. Yeah, as you say, it's interesting how nascent a lot of these applications are and how the whole thing is quite emerging and

Really, only maybe visual art and GANs have evolved quite a bit already, and it's interesting to see their progress over the last few years. So I'm curious, having put together a spreadsheet and this website, has your view of this intersection of machine learning and art changed at all? Did you learn anything new from this project?

So I think the first thing I realized is that there were so many things that I weren't aware of and kind of getting to know them and digging into the details, I'm able to create a vocabulary. So I can kind of use everything that I see and learn and apply to new projects. And the reason why I find creative applications very useful is that you can actually see that it works in practice. Often papers can be overused.

overfit on a small data set, so it's very hard to understand if it's really working or not. But as soon as you see other people using these models and creating creative applications, you can at least know there's some good technology that you can apply to other problems.

I think another thing that I realized just looking at a lot of projects is that they can often be very hard to understand. I think that's a mix of the art language and a technical language. And once you mix them, they can both be confusing. But oftentimes, these projects are very interesting, but you have to spend sometimes 20, 30 minutes or several hours just to understand one project.

So I think in general, like a quick feedback to people doing these projects, when I learn about a project, I first want to understand what's going on, maybe in like one or two lines so that you can have a feel for what the technology is and what you hope to achieve. And then you can add more context in terms of creative applications or a narrative and technology to kind of...

add more context. That's very interesting. And yeah, I've definitely been kind of trying to keep up with artistic applications of art and often it's sort of a home brewed method, I guess, is what it sounds like. And so it's kind of hard to figure out how they do it. And it looks amazing. So it's super cool to have a compilation and to be able to sort of browse the vocabulary, as you say.

I'm curious, in putting this together, have you interacted at all more with the community of people doing these sort of works? Have they kind of liked the project? Did they comment that it's great to have this sort of thing? Or maybe did other people comment, stuff like that? Yeah, no, so in this process, I reached out to all the artists that I put on the site, and I had a very warm response. And I think...

I think it was a problem that most people didn't really know they had until they saw the website. I think the NeurIPS creative workshop every year kind of makes a very good summary of projects. But there's not a bigger context in terms of the past 10, 20 years. And it can also be hard to find specific projects

I think that's also one of the problems I had. I was like, it was this lovely idea and project, but I can't remember what it was. And I can't really know or kind of remember what to Google for. But once you have the site and you have the categories, it's a lot easier to find these kind of gems that you've forgotten about.

Exactly. Yeah, it's and just for myself, I appreciate a lot of this creativity. So being able to browse and see many that I haven't come across because sometimes they're quite niche. It helps a lot, as you say. I'm curious. Wait, sorry, I forgot my question. Let me. Oh, yes.

I'm curious, actually, do you have any favorite things from the site? Maybe like a few things that really stand out in your memory as being very cool or unexpected?

So I think there's, I love a lot of art and creativity. So but if I had to choose a few of them, I would say that one trend I find really fascinating is synthetic statistics. So Terence Broad made an art project where he created a gun without using any data.

So we just kind of had an heuristic for how the generator and discriminator work together and he created this kind of visual art out of it. And I think that's very interesting in a lot of different ways. One is the kind of idea of...

vision from first principles because kind of the world we're in was created from a certain statistic. But I think it's interesting to kind of entertain the idea is that if you can create a new visual statistic from scratch, what can they look like and will they be different from kind of what we're experiencing right now? And that can not only be applied in vision, but you can create similar experiences in

say language or audio. I know Joel Simmon, who runs Artbreeder, he also made an interesting project where he invented a visual language. So it was essentially a variational out encoder. And then in the middle, it had visual elements who could visualize the communication between two models.

I think another area I'm very interested in is the cellular automata. So Alexander Morgenstern, who created DeepRing, which I also find very fascinating, but his latest project is based on cellular automata. So this is, if you're not familiar to cellular automata, you can think of Game of Life. You have these kind of rule systems that create these very lovely and complex patterns.

And Alexander has applied this idea to CNN and gradient based approaches. And you end up with this kind of cell like design so that you can remove a piece of the design and artwork and reconstruct itself. And I think this idea of reconstruction and kind of organism like design is very interesting and appealing.

And I think the last area broadly that I'm interested in is restoration. So I think most recently you have the Art Breeder projects where you translate... So Art Breeder is kind of based on two things. You can create things with guns, but you can also reverse them. So you can take a picture and turn them into the latent vectors.

And what we've seen people done recently is that they take, for example, a picture of the Statue of Liberty and they convert that to the latent vectors. And then they find the latent vectors that correspond to reality. And then they change them and then reproduce it. And then you kind of have a real world looking person for the Statue of Liberty. And also in terms of restoration, I think there's still so many things that

that are in our past that we can access now, just like the Statue of Liberty. There are other statues and there are other artifacts. Jason Antic, who made the deal to colorize pictures. And I think there's a lot of similar tools that are kind of taking our rich cultural history and making it accessible and interesting for today's audience. And I think that's very important.

Very cool. Yeah, just browsing the website before this, it was just so many cool ones. I'm thankful you could pick out a few to highlight. I'm curious, having gone through this and probably discovered a lot of new kind of forms of things, has that informed the work you're doing as a resident at Google, at the Google Arts and Culture Lab?

Yeah, no, I think it's for sure. I mean, we're doing... I think our lab, we ideate maybe 20 or 30 projects per year and we publish a lot of projects. And in all those discussions, you want references to other projects. And I think one...

I kind of create a vocabulary, but also a lot of people are submitting projects. So I think over the past few weeks, I have about 150 submissions. So I get to see what everyone is creating. And I think if you're a lab and if you want to be at the kind of cutting edge of things, you really need to know what everyone else is doing.

Makes a lot of sense. And actually, I personally am not too aware of the Google Arts and Culture Lab, and I'm not sure if our listeners are. So maybe you can give us a bit of an overview of what does the lab do? What kind of projects does it work on? So the main kind of goal of the Google Arts and Culture Organization is to make

art and culture more interesting and relatable to the main audience. And the Google Arts and Culture Lab were more focused on the kind of technological aspect of this area. So I think the most famous project is Art Selfie.

which is you take a picture of yourself and because google arts and culture has a database of several hundred thousand paintings it can match the painting that looks most similar to that person and this one super viral a couple years back we do a lot of collaborations with magenta so working with a lot of their kind of the latest research and understanding how we can kind of

get these, create interfaces, interact with these models and understand how the general public can benefit from these different innovations.

I think early on, so it's been about five years that the labs existed. Early on, it was a lot about using U-maps and testing maps to visualize data and make that more accessible. So we're working also with a lot of museums. And Google also has an arm where they help

archive information so they can go to a museum and they can archive several hundred million images. And then they kind of want to get an overview, the museum and the researcher in museums want to get an overview of what they have in their data sets. So we can kind of automate that and make that information accessible to researchers.

Very cool. Yeah. And you mentioned VooLab has been around for five years, and that makes me wonder, actually, having put together this site and you mentioned how it's in a way almost sort of cataloging a history of how AI has been evolving and its applications to creative purposes. Do you think...

Things are accelerating. Are we getting more and more sort of people playing with technology or AI to make art over the years? Or is it pretty steady? I think we, if you look at the timeline, there's kind of a huge explosion around

And I think that's a combination. One, you have technologies that are good enough to produce interesting new areas to explore. And I think on the other hand, it's a lot more accessible. I remember a couple of years back when just to create a cloud instance and use a GPU used to take several days, but now you can just click on a button and access a model.

And I think also the kind of the creative community is always looking for new tools and ways to express themselves. And I think we're just at this edge where kind of AI is coming in into the mainstream narrative. So a lot of creatives want to use this tool in this form of narrative. And you have the technological side. And I think this is really, really making machine learning art

something that's exploding at the moment. That's great. Yeah. And it's very cool to see it happening almost in real time. On that note of accessibility, I'm curious kind of on the backgrounds of the artists and creators on the site, is it still mostly people who can do coding as well as art or has it gotten to be enough

accessible where artists that maybe aren't technical, that don't do programming, are able to also incorporate it? Or is that just starting to happen maybe? I think it used to be a lot more technical. I would say probably

Before 2015, definitely, it was a lot more technical. And now you have things as what I mentioned earlier, Runaway ML, that is kind of a web browser with different models. And it has a visual interface that you can use to use these different models.

So I think in terms of the popular approaches such as StyleGAN or different GAN variations, there are easy ways to access them. But I think a lot of the interesting work still comes from kind of the bleeding edge of machine learning. And then you need to be able to work with models in a way that most people can't. That makes a lot of sense. Yeah. And it's also sort of what seems to be the case for me. But yeah,

You mentioned RunwayML and maybe for the benefit of our listeners, let's say they go to the site and they like some of the Gantt style art where you can do style transfer, you can do deep dream, you can do various effects.

Would you say for these listeners that maybe aren't too familiar with coding, they can go ahead and try and use sort of the materials on your site and then start playing around and creating their own versions? Would that be kind of doable?

Yeah, no, I think that's how I started when I started learning machine learning. I started with colorization. And one reason for that is that it's kind of, it's visual, it's playful.

I think that's a great way to get into the kind of machine learning scene. And the website is created so that I've categorized everything. So as soon as you see something that's interesting, you should be able to kind of take the keywords and the models and Google them. And then if you find a collab or a notebook,

Often you can click on them and you will be able, it might be a little bit hard if you're not technical, but oftentimes if you spend maybe 30 minutes or an hour, you can understand how to use these models and create different artworks or projects. That's great. Yeah. So listeners, if you check out the site and see something cool, you know, feel free to try and play around. It's actually not that hard. Usually there's instructions and you can just copy paste, which is what, and

AI researchers do anyway, a lot of the time. Yeah. So, um, yeah, I'm curious, any next steps as far as this project that you maintaining it, is it actually growing pretty quickly with the number of submissions?

Yeah, no, so it's, it's, uh, I only launched it a couple of weeks back and it's, I'm still getting a lot of submissions. Um, yeah, no, the goal is to kind of every month do a deeper review to kind of understand what people are doing and add projects to the site. And then I have a newsletter where I send out to, to highlight a few of the products so people can, can stay up to date with what's going on. Oh, I see. And that newsletter, uh, people can sign up for it on the site presumably? Yeah.

Exactly. Yeah. Very cool. So I'll have to go ahead and sign up myself because getting the latest and greatest certainly sounds fun. Yeah, I think we covered a lot of cool stuff related to this project. Last question I typically go with is kind of open ended. Do you have any other thoughts that's been on your mind with respect to AI and art and creativity that you've been mulling over and kind of would like to share with our listeners?

Well, I think what's interesting is kind of the clash between the AI art community and also the general art community. And I think there you see kind of people that don't know technology, it's easy to anthropomorphize. Yeah, it's a hard word.

Yeah, it's so long. I wish there was a better word for that because it's such a typical kind of... It's such a useful word for AI, yeah. Exactly, it's such a useful word. But so people kind of attribute human heuristics to these different models and things they don't understand.

And I think one that's kind of part of the technology's magic narrative, but I think in a lot of aspects, it can be dangerous and create harm. And I think one area where you can create confusion is that people...

think that these can replace jobs in a fairly short amount of time. So there's a lot of projects coming out with guns and videos and kind of the people in the cinema scene are getting worried. You see authors getting worried because of the work that OpenAI is doing. So I think it's...

There's still a lot of work that has to be done there to help the broader audience understand how good are these, what can they do, and have a more honest narrative around these technologies. Absolutely. And the site has some interesting examples I particularly like.

the music album by the band Yacht, where they were a normal band, right? That didn't use much AI, but then they actually incorporated it into their creative process. And it would be, it's very cool to highlight the ways in which artists can use AI and be replaced by it.

Hopefully others will see it like that. I think that's something that I've been seeing a lot too, is that you have people who are really good at something. For this case, it was music. And then you find people in machine learning and you collaborate.

So I think that's what we're seeing a lot is that people who are cutting edge and say motion graphics or 3D animations, they work with someone who's really good at machine learning and then together they can create something that's never been seen before. Fantastic. So on that note, I think we are going to go ahead and wrap up. Thank you again, Emil, for joining us on this episode. Awesome. Yeah. Thank you for inviting me.

And thank you so much listeners for listening to this interview on this episode of Skynetoday's Let's Talk AI podcast. You can find articles on similar topics to today's, including art, on the website and subscribe to our weekly newsletter over at skynetoday.com. Subscribe to us wherever you get your podcasts and do please leave us a rating or a review. If you like the show, we could use your feedback. Be sure to tune in to our future episodes.