Images and Inspiration With AI: Pinterest’s Jeremy King
20:20 Share

Today, we're airing an episode produced by our friends at the Modern CTO Podcast, who were kind enough to have me on recently as a guest. We talked about the rise of generative AI, what it means to be successful with technology, and some considerations for leaders to think about as they shepherd technology implementation efforts. Find the Modern CTO Podcast on Apple Podcast, Spotify, or wherever you get your podcast.

How does one image-based platform use AI and machine learning to continually inspire users? Find out on today's episode. I'm Jeremy King from Pinterest, and you're listening to Me, Myself, and AI. Welcome to Me, Myself, and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College.

I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

Welcome.

Our guest today is Jeremy King, head of engineering at Pinterest. Jeremy, thanks for taking the time to talk with us. It's great to be here. Thank you very much. It's always good to start with an overview. Maybe tell us a bit about what Pinterest does and what you do at Pinterest.

Pinterest is, we like to say, the destination on the internet for inspiration. We have hundreds of millions of people that come to us every day to figure out what they want to wear, what they want to do this afternoon, or what they want to make for dinner, or how to decorate their kid's cake, or remodel their kitchen. And so we want to build a platform that allows anyone to create a life that they love.

It's a great place to work with that kind of mission. And I have a lot of people who, when I tell them where I work, get pretty elated about their own Pinterest boards and what's going on in the world. Great. Tell us more about working at Pinterest. Yeah, I've been at Pinterest now, the head of engineering, for just over four years. And

Just like any leader, I'm trying to figure out how to unblock my team and hire great people. And I tell people all the time that when I came to Pinterest, I was just so impressed with the level of talent. It's a relatively small team. We have 4,500 employees in the world and about 1,400 engineers. And as a result, competing against some of the biggest companies in the world, this team has to be extremely high caliber. We have some of the best graph database people in the world. We have the best computer vision people in the world.

these people are really fun to work with. None of that so far, you've mentioned artificial intelligence. So what role does artificial intelligence play with those 1400 engineers?

Yeah, that's an excellent point. One of the things that is true at Pinterest and now at many other companies is Pinterest was actually born with machine learning and AI from the base. It doesn't exist without it. So talking about AI and ML separately from Pinterest is almost like a misnomer. It really doesn't happen without it. So nearly everything that we do touches our machine learning systems. We do AI.

millions of inferences every second. Every single request comes all the way back to the graph and publishes back out a very specific use case to every single person. It's not cached. It's every person is a specific result. I mentioned that we have about 1,400 engineers. We have about 350 machine learning engineers

As a percentage, when I tell other CTOs that I work for, they're like, that's a really high percentage of machine learning people compared to my company. That's true also on the data platform side. We have a wonderful leader, Dave Burgess, who runs our data platform. And you can imagine our data platform is about as important as any other part of our capabilities.

And so I mentioned the graph, but it's also, you know, the sort of normal SQL databases and the real-time systems that keep this thing running. Jeremy, you said graphs a couple of times. What's a graph? Essentially what a graph is, is if you take an entity, an object, and you say what's related to that object. So in this case, we're using a nearest neighbor graph here.

which we've written lots of documents on, which is saying, hey, I have an image in this case, and I have another image that's related to this image. So that allows me to build this sort of network of images. And the reason the graph works at Pinterest is because

every pin is essentially added to a board, which essentially makes it a node in a graph. And so it continues to increase the graph. So it allows you to, unlike a SQL database or a relational database, where you have to sort of tag indexes and indices, the graph can be indexed. Essentially, any image could be indexed to any other image. It allows you to do searches on things that are related incredibly efficiently. That's the power of Pinterest.

Jeremy, you mentioned AI has been part and parcel as part of Pinterest's journey and has been ingrained in Pinterest's tech stack from its very early inception. So it must be that generative AI is all over the map for you guys as well. And I'm sure that it's on everybody's mind these days. So maybe tell us a bit about what you're doing there.

We have a state-of-the-art machine learning environment that interconnects these datasets across all the surfaces. We call it home feed, the search system, the related things that drives personalization and recommendations and engagement across the Pinterest. While we're growing really fast from monthly active users, including Gen Z is growing even the fastest.

We added 13 million monthly active users, so we're really focused on how to increase engagement. And not surprisingly, a lot of the improvements in machine learning, even in the last two years, have been driving that engagement. We have, of course, built new features to allow Pinterest to become more shoppable. But as we're seeing the capabilities of the machine learning models getting effectively 10 or 100 times bigger, which is relatively common today, we're

we're seeing that giving an outsize increase in our results as the result gets more specific. And frankly, Pinterest is also about looking at vast different things. So it's not every single item is about what you want to cook for dinner tonight. It's also, I know you were looking at birthday cakes last week or New Year's is coming up and we're going to interplay some of that.

So we've been taking advantage of some of the advanced AI capabilities that have only come to life over the last couple of years. And that includes GPU work and all the things that make it cost effective to do these things. So tying back to your question of generative AI, not surprisingly, given that we're an image platform, generative AI was very interesting to us. And I typically break it up into three buckets.

When we're thinking about large language models, LLMs, we talk about number one is how do I make my team more productive? We've got a couple of pilots running. We haven't decided exactly which way to go yet, but it's looking really promising. And like lots of other CTOs, we're really excited about that. That's interesting. Tell us how generative AI increases efficiency.

I was talking to a number of CTOs and one thing I thought that was really interesting is, there was one particular CTO saying is that in general, it's increased productivity by 10 to 15%, but there are a small set of users where it's increased their productivity by 50%. That's fantastic. I was just going to say, it also feels like if you go back a decade ago, there were lots of images and unstructured text and that kind of stuff that

that for AI to train on that data and make sense of it, human intervention was needed, both to tag the content, but also to make sure that the outputs made sense. And it feels like now with Gen AI that a lot of those human-like judgment calls are going to be made more and more with Gen AI.

particularly with lots of unstructured text and images and video and that kind of content. Is that right? Or am I going too far?

I think that's right. I mean, the Pinterest system is built on embedding. So you take an image or a piece of text and you essentially tag it, essentially. And to your point, this is what Pinterest has really been great at. You take our computer vision technology and effectively build these embeddings to detect people or content or couches or birthday cakes. We've been really good at that and have really led the industry for a long time on this. And

And you're not wrong that what happens is just gets better. It's just more accurate, more specific. Like in the old days, I could say, I know this is a lamp and I know it's got pendants and it's made of crystal and it's gold and that sort of thing. But now I can say like, I know exactly how many times it has on it. I know what kind of light bulbs they are. I know probably who manufactured it and these sorts of things. I can get way more specific. In Pinterest, we really think about these as two different things. People come to Pinterest and

because they don't know exactly what they want. And this is where Pinterest thrives. And I think why we have a long-term differentiation in the market is because you know what you want. You can always go to Amazon or Home Depot or Wayfair to go buy it. But if you don't know what you want, you start with Pinterest or do like a million different searches because I don't know how to describe...

you know, classic barnyard kitchen. I don't know those words. So even putting it in something like ChatGPT, I don't know how to say that. You know what I mean? Sure, that'll get better over time, but a lot of it will be image-based too as well. Are you finding that the foundational models that are available do it for you or you're building your own domain-specific models?

Both, I guess, is the answer. What we're finding, and it's still relatively early, in sort of the generative imagery, what we're finding is that the smaller models actually are much more specific. And so I think that's what's going to happen. And my CTO friends are saying the same thing, which is like, each one of these models are going to be very specific to the use case. And that helps it not only be more accurate, but also makes it much, much cheaper to implement as well.

I think it's interesting that you mentioned several times cakes because decorating cakes is, I think, practically my only use of Pinterest. And it ties to Shervin's comment about generative because I deeply suspect that those cakes that I see there, no human could create. I certainly can't create them. So to what degree are you, one of your four, three use cases wasn't that fourth one, which is generative.

people putting images that don't really exist or products that don't exist. What's Pinterest's perspective on that?

Yeah, excellent point. I think, yes, and we see this a lot at one of our biggest categories we call art. Art includes a whole bunch of things, including things that you would traditionally put on walls and paintings and that sort of thing. But it also includes things like body art and we have tattoos and these sorts of things. And again, lots of those are great cases where people need a little bit of inspiration and that can be generative, right? And it may not exist in the world, but you want to see what it looks like on a human face.

you can see this coming where you'll be able to print out something that's generative and this is already happening, right? Yeah, that really aligns with the idea of inspiration. Yeah, absolutely. And things like cakes and that sort of thing are very interesting. One of the harder parts is I've seen lots of work for home and home improvement and

But what it's doing is it's generating a whole bunch of ideas that can't be realized where I can say, hey, people have yet to really break out the camera and take a picture of their room. People do that, but we call it a 1% feature. Like, how do you get people to actually engage the phone or the phone camera?

But, you know, if you can upload an image, you know, I've seen some beautiful renderings of like, here's what my room is, rearrange the furniture in my room, the furniture I already have and things like that become really interesting. Or you say, I want to replace this couch with this couch from Wayfair. I want to replace this lamp from some other lamp. And here's 20 different combinations. You can kind of click through. And those are like,

You're taking a virtual experience, almost AR-like, but you're putting real products in there, right, versus them being generated products. But it allows you to build user experiences that are much more enhancing. Last year, we built AR Try On, which allows you to try on makeup. And what we're seeing, again, while it doesn't get a ton of usage, the people that do use it are 60% more likely to buy something, right? It's, like, amazingly engaging when people actually get there.

What's tough at Pinterest? You mentioned a lot of good use cases where you're doing a lot of things with these technologies. Certainly, it can't all be wonderful. What's hard? We spend a lot of time on inclusive search and results. And in 2021, we launched hair pattern search. It was kind of this first of technology. When you're searching for hairstyles, how do you identify hair that looks like yours, if you will, right? And this is hugely important in order to refine what you're looking for.

We had the same kind of problems with skin tone where originally we were trying to use the early models for skin tone detection. And what we found is like it was more on face detection. And so our team had done some great work like doing skin detection. Like it could be a side shot or a back shot or an ear shot or a hand shot. You have no idea what kind of skin. So like how do I detect what skin is? And then you can detect what kind of tone you're looking for. Same thing had to be applied to hair and

you know, we've got all kinds of paired. We do have shaved slash bald. We have straight, we have wavy, we have curly, we have coily, we have protective, you know, and these kinds of things make the results dramatically better, right? You can save your hair pad and your skin tone, and then we'll tailor your results to that.

And as you can imagine, this is a complex problem. It's not just the USA thing. Every different country has different types of hair patterns and skin tones and different kinds of fashion. And so these kind of things are really hard, but they're the great projects to work on. And we've had a wonderful team, the advanced technology group here in particular. We've been working on inclusive technology and all these advanced models, and they're definitely hard, but they're wonderful when you get them right.

We have a segment here, five rapid questions. I'll just ask you a bunch of rapid fire questions. Just tell me the first thing that comes to your mind. What's your proudest AI moment? The inclusive product feature and actually hair pattern was my proudest one because not only was I involved from the product from the beginning, but it's also first in the industry. And the amount of response that we got from our pinners was incredible. So yeah, it was a really good one. What worries you about AI aside from bias and some of the ethical issues?

It's funny. It's on every single forum. It says like, hey, are you worried about bias? Let's see. What worries me about AI? Yeah, I'm not worried. I don't worry about too many things, but that's a long time CTO's thing. It's like, if you worry about too much, you can't survive on this job. What's your favorite activity that does not involve technology?

Mountain biking. Very cool. I try and go at least two times a week. I go with a crew of Silicon Valley tech on Friday mornings and then usually on the weekends with my brother and a few other people. The first career you wanted, what did you want to be when you grew up? I think my five-year-old baby book says fireman, but both my grandfather and my father were engineers. So I actually thought I would build houses because I loved working with my hands outside. So that's where I thought I was going to go, architecture or house building.

What do you wish AI could do that it currently can't? Teleportation. Get me through a security line at the past year. Maybe that's coming. You know, I've always been super excited about translation. I took over the translation team at eBay recently.

way back when. And there's some wonderful people that were working on this. And I've always thought about a universal translator as something that it would just be so amazing. And it's getting so close, but it still seems like so far away before it'll be like a consumer version of this device. But why do you think that is? Because we've got now LLMs and we've had for a long time audio to text. What's the limiting factor here?

I think it's form factor. Like, how do I, you know, get out my phone, hit translate from Spanish to English, you know, that kind of thing. You know, I don't know if I'm walking up to a vendor in the street in San Francisco and, you know, I don't speak the same language and I want something, you know, that it's got to be close, but it's a form factor thing. But I think it'll come. It'll come shortly. It seems solvable. Yeah, solvable.

Jeremy, we really appreciate you taking the time to talk with us. It's been very interesting to learn sort of behind the scenes what's happening between all those images that we see. But moving on, I appreciate all that goes into making something that looks as easy as Pinterest work. And it's been pretty fascinating to see how much artificial intelligence and machine learning is behind that. Thanks for joining us. Of course. It's been wonderful.

Thanks for listening today. On our next episode, Sam and I meet with Damini Satija and Matt Mahmoodi from Amnesty International. Please join us. Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn't start and stop with this podcast. That's why we've created a group on LinkedIn specifically for listeners like you. It's called AI for Leaders. And if you join us, you can chat with show creators and hosts, ask your own questions, share your insights,

and gain access to valuable resources about AI implementation from MIT SMR and BCG, you can access it by visiting mitsmr.com forward slash AI for Leaders. We'll put that link in the show notes and we hope to see you there.