Marketing With Generative AI: Harvard Business School’s Ayelet Israeli
34:33 Share

Today we're airing an episode produced by our friends at the Modern CTO Podcast, who were kind enough to have me on recently as a guest. We talked about the rise of generative AI, what it means to be successful with technology, and some considerations for leaders to think about as they shepherd technology implementation efforts. Find the Modern CTO Podcast on Apple Podcast, Spotify, or wherever you get your podcasts. How can using generative AI help us understand consumer preferences?

On today's episode, hear from a professor about her market research study. My name is Ayelet Izraeli from Harvard Business School, and you're listening to Meet Myself and AI. Welcome to Meet Myself and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

And I'm Sherwin Kodobande, senior partner with BCG and one of the leaders of our AI business. Together, MIT SMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Hi, everyone. Today, Shervin and I are thrilled to be joined by Ayelet Israeli. She's associate professor and co-founder of the Customer Intelligence Lab at the Data, Digital, and Design Institute at Harvard Business School. Ayelet, thanks for taking the time to talk with us. Let's get started. Thank you so much for having me. Often we begin by asking guests their professions, but what's nice about being a professor is that people kind of have an idea of what that means.

But I still think it'd be nice to hear a little bit about your background in bio. So can you take a minute and introduce yourself and tell us what you're interested in? Sure. I'm a marketing professor at Harvard Business School. I'm really interested in how we can better leverage data and AI for better outcomes, if it's outcomes for the firms, for customers, for society at large.

Some of the work I'm working on is around gen AI and how can firms use that to gain better access to consumer information and preferences. In other work I do, I think about how we can eliminate algorithmic bias in our decision making. I saw your talk a few months ago about using generative AI. And it really struck me as interesting because lots of people are talking about generative AI.

But we don't have a lot of evidence yet. Evidence is not saying it's not there, but it's just forthcoming. But you're starting to get some evidence through this research that you're doing. What can we do with GPT and generative in market research?

Me and two of my colleagues that are at Microsoft, Donald Way and James Brand, started thinking around, can we actually use GPT for market research? The idea was some people have shown that you can replicate very well-known experiments, including the famous Milgram experiment using GPT by just asking it questions.

And we were thinking, you know, we work so much as researchers and as practitioners to better understand customer preferences. Maybe we can use GPT to actually extract these kind of preferences for our clients.

large language models, the idea is that they will give you the most likely next word. That's how language is produced. And we were thinking maybe if we ask GPT or induce it to make a choice between two things, maybe the response, which is kind of the most likely next word,

will actually reflect the most likely responses in the population. And in that sense, we will essentially query GPT, but get kind of the underlying distribution of preferences that we see in the population. And we started playing around with that idea. And we focused on consumer products because we assumed that,

The data that GPT is aware of is mostly around consumer products, maybe from review websites or things like that, to see can this idea actually work. And does it? Yeah.

Kind of. That's wonderful. Yeah. So tell us more. Our first rush was like, okay, let's see if it can generate very basic things we expect from economics. Like when the price is higher, does it know to reject an offer? Does it know to make this trade-off between price and choice? And we do see kind of a downward sloping demand curve, which is what you would expect to see when we query GPT, you know, thousands of times to get

answers. We also see things like, oh, we can tell it something about its income and it reacts to that. When it has higher income, it's less price sensitive, which makes sense, is what we expect from people as well. We also see that it can react to information about itself like, oh, your last time you bought in this category, you bought this particular brand.

makes it much more likely to pick this brand in the future. So those are kind of our tests of does it actually react in a way that humans would in surveys? And then we took it one step further and we were trying to get willingness to pay for products or for certain attributes. And then we basically compared the distribution of prices to distribution of prices we see in the marketplace, which

is pretty consistent. And a really interesting and exciting thing for us was the ability to look at willingness to pay for attributes, because it's something that we all as marketers want to find. In our example, it's toothpaste, and we're trying to figure out how much

People are willing to pay for fluoride, which is something that is difficult for us to think about. If someone would ask you that, I don't know. I do know that I prefer to buy this toothpaste, but I don't know what is the number. So it made us more curious to see if GPT can provide us this number in the same way that we ask consumers. And the way that researchers have shown over years, the best way to ask these questions is through conjoint studies. Essentially, you provide fluoride

people with 10 to 15 choices. And through their different choices, you are able to understand the trade-offs that they're making and actually quantify the difference that they're willing to pay. We essentially did that. We did a conjoint type analysis with GPT.

And we compared the outcomes to human studies that a forthcoming paper just ran and got pretty similar results. So we were very excited about that. Of course, the results are not identical. We need to do a lot more to figure out where some of the issues are and how much does this generalize. But just the fact that we were able to get it was incredibly exciting. So it seems exciting for firms because...

I'm guessing that the cost of doing a market study on a lot of people is much more than doing it just through a bunch of API calls with ChatGPT. That has to be the appeal. Are there other appeals?

Basically, these type of studies are time-consuming, costly, complex. Ideally, you would like to ask people to make a lot of trade-offs, but you're limited by the human ability to do that. With GPT, you can query it a lot of times, but...

At this point, I'm not going to tell anyone, replace all your human studies with GPT or with another LLM because there is a lot more work to be done to figure out how to do that right. One of the things around GPT is that it's pre-trained. It will give me preferences, but these preferences are relevant for the time period in which it was pre-trained. And a firm wants to know what are the customer expectations.

interested in right now. So that's kind of a limitation. What we're testing now is maybe we still have to query people, but fewer people than you would normally have to. So usually when you run these studies, you need thousands of users to get something that would be robust and statistically significant from an academic or statistical standpoint. We're trying to look at maybe I can collect

information from much fewer humans and combine it with LLM through fine-tuning and generate something useful. But really, a big advantage would be cost-saving and time-saving. At the time, it was a big one. Yeah. And we're talking so far about consumer products, but you can think about business-to-business type surveys, which are way more expensive and harder to do. So perhaps there is potential there as well. We haven't tested that yet.

I love the idea, right? I mean, like, because when you think about most use cases for

generative AI, there was a lot about taking grudgery out of the work or creating images and content and summarizing text. And then there's more advanced ones around planning and inventory management. But the one you're talking about is literally replacing humans with this, right? I mean, that's basically what it is. And it's a beginning of something that could be quite interesting because you've proven at least that

It's sort of rational, right? I mean, you're asking it all these questions and it is economically, I guess, rational. But then as a marketer that you are yourself, not all marketing strategies are based on rationality. In fact, many of them are based on completely irrational desires. What are your thoughts on the...

non-rational choices that many people make that creates these big brands and $20,000 handbags and all kinds of stuff like that. How do you tap into that? Before I answer your question, the first thing I was nervous about as an academic is when you used the word proven. Proved? Yes. I heard it. I smiled when I said it. I would say we showed evidence consistent with evidence.

that. And we also know that these models are still evolving and maybe something we showed a month ago will not be relevant in a month from now, which is also a reason why you shouldn't just go and implement it without testing. So I want to be careful about that. Yeah. So, you know, there is the more rational view of what is a product, but

brands have value that is created that is kind of not measurable to us and hard to quantify. But that's

Almost like the example I gave with fluoride. Like we don't know how to quantify fluoride. We might find it difficult if I would ask you, oh, how much are you willing to pay for a brand name like Colgate versus a toothpaste that I just made up? Actually, the same model of conjoint study will be able to infer those differences. And we see preferences, for example, for Mac over a different computer type. So it's already embedded in there.

in a way. Now, how accurate it is, it's an empirical question. Yeah, no, you're so right. Because as I hear you respond to this question, I also realized that my assumption that

what you showed some evidence for vis-a-vis proven isn't necessarily rationality. It's that it's got a ability to sort of encapsulate what most people do or what many people do, which is embedded in stuff that it was trained on. So then my second question is, how do you get this to be more segmented or more specific or more nuanced? Yeah.

Because when you do focus groups, you're looking maybe for a particular flavor, a particular nuanced mix. Yeah, yes. And also a lot of the uses that we have seen, you know, when GPT and other LLMs were just introduced, a lot of the excitement was, I'm an engineer, I can just ask it a question. It gives me the most common thing. That's exactly what I want. And actually what we are doing is the other side of that. We don't want the most common thing. We want to understand the distribution of

That's why when we query GPT, we ask it every question many, many times because we want to get many, many different consumers. In our analysis, we only varied income and what you bought before, but we can in the same way vary...

gender, race, anything else that you want, age. And I've seen other researchers do that for, there is a really interesting paper by colleagues at Columbia and Berkeley that use GPT to create perceptual maps.

So how close two brands are to each other. And they showed also differences by gender and age and things like that around cars, which is a market where we expect to see these differences. So you can definitely do that too in a similar way.

It was also shown in political science for politics. I can give someone an ideology and their voting behavior makes sense, their text generation on different topics makes sense. That's also very exciting as a marketer who cares about heterogeneity and understanding the differences between different consumers. Yeah. Only if we could use this for clinical trials. Yeah.

I saw some paper on like better bedside manner of LLMs relative to doctors. So maybe there is still something there. That's GPT-5 maybe. Yeah. As you're saying that, though, think about the way these work is a probabilistic estimate of the most likely next word, the most likely next. And you've segmented out, all right, given that you are low income, high income, given that you are this attribute, that attribute.

That's interesting. But where do we come up with the weirdness then? If everything is based off of the most probables, particularly from predefined, not that you're not brilliant about coming up with a nice search base, but how are we going to find the things we don't know then? Isn't that something that comes out of market research and focus groups? Certainly.

And that's part of the challenge. Obviously, GPT learns some kind of distribution, but there are people that, you know, if let's say all that it learns is from reviews, there might be a lot of very extreme consumers that don't write reviews online or don't have access to internet, but have these interesting extreme ideas. And even if I tell GPT, you know, I want the randomness as possible, very high variation, I will not get to those people. So...

that will definitely be a problem. I know already of some startups that are trying to solve this issue and identify these extreme consumers and then take them to the next level by

using LLMs to maybe predict what they will do in another case. But at the same time, there has been some work on kind of creativity of GPT and that it creates very creative ideas, which, you know, is not exactly what you're asking for. Some of those creative ideas are unconstrained by reality. I think we've all seen some of it, the way that it plays chess and decides that that rule is a little bit too confining. Right.

Right. So that's also the problem of hallucinations, which should be tested in different contexts. But I think the way that we induce it to make a choice is less prone to hallucination problems because it provides a choice and you're not asking for facts or something like that. I'm not trying to say that GPT will outperform any customer survey or anything like this. All I want to see is if it's as good as humans.

And even with human customers that we talk to, we have to work really hard to find people to do these surveys. And sometimes we miss them. We might be able to get the distribution of some people, but still have to work hard on the extremes without AI, but with just human conversation.

What I find really interesting here is you said something like it's not as good as a consumer survey. And now I want to challenge that because what I find interesting in this idea that you have is that when you think about other AI or Gen AI use cases, there is a sort of a burden of proof that you say, okay, so like I'm a human, I'm an engineer, I have a task, right?

Let's ask GPT or any generative AI system, whether it's a knowledge kind of work, right? Whether it could do it as well as a human does. Okay, great. Or can it code better than a human does? Or can it create a video or a document or something that you would read and you say, wow, this is nice. So then you could do it. I don't need to do it, right? So that's sort of a burden of proof is very clear.

On this one, I'm not so sure that you even have to have a burden of proof because in many ways, we're assuming that a focus group of 500 or 1,000 people or any survey, I mean, there's no focus group that big that I know of, but like a survey of that kind is somehow gospel or like that's like what GPT or whoever, whatever. Can you talk to the reviewers of our paper? No, but because the reality of it is, if you think about it, it's that...

If the only way to know, so go back, because look, I mean, your premise here is like, we are going to save so much money on all this market research by augmenting this with that, which is a true premise, and for sure it is. But like, I also find the burden is lower. And even if you don't stop a single market,

human-based market research or survey, you've still added a ton of value by broadening the universe of responses and options. Because I would argue, how do you know 1,000 people or 2,000 people are representative at all? Or they have all those nuances. And so this thing is actually bringing in signals that you know for a fact exist, because otherwise it wouldn't be there. And I find that actually quite inspiring to a marketer.

I'm happy to talk to your reviewers. I think as academics, we are used to a certain level of kind of rigor and robustness and ability to say like, oh, to actually prove things. And the fact that this tool can provide a simulation of something is nice, but can it actually replace humans is a higher burden because of this question of,

Is it actually giving me meaningful, updated responses? Will it match something? And you're saying, well, maybe humans aren't that great in the first place. So why do we try to... No, I'm actually making a different point. I was trained as a scientist and I get the burden of proof is much higher in science than in academia. And I wasn't trying to argue that you've proven that this replaces humans. I don't think it replaces humans, but...

What I was trying to say is the value of this is that it dramatically augments the signals and insights and ideas available to a marketer. And because there is no survey or focus group that by definition isn't limited,

And this isn't limited because it's got everything that's there. So my point simply is that not that the burden of proof has been met, but that I don't even know if there should be that kind of a burden of a proof because it is addressing a limitation of focus groups and traditional research.

So it doesn't necessarily need to replace it. They're not perfect to begin with. Nobody would argue with that. Yeah, I think at the very least, I feel comfortable saying that we show that it could be very informative about preferences and what is going on, at least within the data it's trained on. And that could already change a lot for a lot of firms, given the type of research and the problems with market research and access to humans and all of that, for sure.

Yeah, so there's multiple different signals coming in here. And I think we've addressed this first from the idea of does this signal replace the other signal from a focus group? But the dependent variable here might be do people actually buy a product? Do people buy the fluoride? Do they buy the non-product? And if this signal...

adds some information to that prediction, then we've got a new information source. If it completely supplants it, then we have a different thing. Right. And now we're going to the problem of these surveys of stated preferences versus revealed preferences that are actually based on what people do. Now, I would argue that GPT might have less problem than human because

It's not subject to things like experimenter bias or trying to appease me. So it's giving me probably something closer, but it's still giving me something closer likely to stated preferences if it brings the data from review sites or market research and not necessarily what people would actually buy. But that is also true about the focus groups and the surveys. So we think about this as a new source of signal.

That, you know, there's lots of different signals out there and it has some overlap perhaps with one signal. And I think that's by itself is fascinating, but it also may have new signal. Yeah, yeah. The other thing that I find fascinating here is that AI solutions have been trained on data. And then when they're put in production, they are then trained on data or they get feedback from data in production and they get better.

With generative AI, so much of that feedback also needs to be human-driven versus data-driven, right? Like, this is what it tells you to do. Does it resonate with you? Yes, no, etc. So it also feels like this kind of a technology where generative AI can be a user of another generative AI's output, right?

So let's go to the paradigm of, look, it's replacing a human in a focus group. But we can also replace a human in a company that's a marketer dealing with a response from generative AI on like, how do you design a campaign for this? And so this idea of maybe multiple generative AI agents going at each other to improve the overall quality, what do you think about that?

I think it's an interesting idea, but I also think that the evidence so far suggests that you still need at some point at least one human in the loop. For sure. Because of all of these hallucinations, unrealistic things that come out.

But certainly, if these models are getting better and better, more efficient, higher quality, then why not? But as we kind of implement these type of things in our organization, we also need to think about how do we, I don't know if the word is exactly validate, but how do we ensure that the process still makes sense and that we're not just wasting everyone's time with these models?

agent talking to each other. No, for sure. You're 100% right. You need humans in the loop and probably for many decades at least. But you may not need so many of them if you have some kind of an output that is supposed to be helping, let's say, a group of 20,000 customer service reps. Right.

And it's going to get better based on the feedback, based on their usage in a pilot of, let's say, three months. Maybe you don't need to pilot this to 5,000 people. Maybe you could pilot it to 100 people plus two or three different Gen AI agents so that you dramatically accelerate the adoption time. Yeah, that's cool.

Although I have to say, when I heard you say in that, Shervin, what it made me think of is when people hold a microphone too close to a speaker and we get these feedback loops, amplifying feedback loops. You do worry that if the two sources of data are too co-aligned, we'll get squelched. We won't get craziness. Skip to the back of the chapter here. Give us the answers. If people are listening to this and they're working in companies and they have these tools available,

Right now, not 20 years from now, like you're thinking as an academic. What should people be doing right now with these tools? Play around with them. Figure out what do you want to know about your customers?

We provide in our paper like a whole list of prompts of exactly how to prompt for these types of things and start getting this information. And like Shervin said earlier, what is it exactly? We're not sure, but it's a signal. There is information there that we can start finding out. Right. And so by playing with it, that helps people discover what information is there.

I think testing and discovering, but starting with a concrete question is really helpful because you will just get down so many rabbit holes. You can have these conversations forever. Ayelet Izraeli, you're the only guest we've had.

that has the AI initial, which nicely fits into me, myself, and Ayelet Israeli, which is me, myself, and myself. But tell us more about yourself and your background and how you ended up where you are and what got you interested in all this stuff. Sure. I'm originally, as my last name might indicate, I'm originally from Israel. Israel is known to be a startup nation.

And when I came through to think about, you know, what I want to study in university, there was a special program that was geared toward improving Startup Nation by giving people kind of managerial tools. So it was a bachelor's in computer science and an MBA combined program in kind of five years. And I started doing that.

And I like computer science. I actually majored in finance and marketing, but I especially was interested in marketing and particularly making sense of a lot of data in this context that is so kind of fun and applied. And then I decided to get a PhD in marketing and research.

Over the years, you know, I figured that consumer products or things around customers and transactions are interesting to me. It's just a fascinating world. You have a lot of data around that because as we move more to online and digital, we can see more and more data. And then the question is, how can we actually leverage that data more efficiently and also in a responsible manner, which part of my research is about as well?

So we have a segment where we will ask you a series of rapid fire questions. But to put you on the spot, just answer the first thing that comes to your mind. OK. What's the biggest opportunity for artificial intelligence right now? Biggest opportunity. This is not rapid. Next question. Yeah. Next question. I'll think about it. What's the biggest misconception that you think people have about artificial intelligence right now?

I tend to be around people that work in this and understand this, that it's just a model, but a lot of people still don't and still envision robots and this magical thing that happens. And that's why I like to explain very clearly, oh, it's predicting likelihood of next word and choosing among the distribution. And that's all that is happening. So I think we're still maybe not as bad as it used to be, you know, 10 years ago, but it's still this like magical artificial thing that happens.

And it's not. It's still magical, I guess. It's pretty amazing. Or it can be. What was the first career that you wanted? I don't know. You know, in Israel, you go to the military. I was in the military. I was a lieutenant in intelligence. I don't think it's a career I necessarily wanted. It's something I did. Okay.

There's a lot of discussion and excitement about artificial intelligence. Where are people overusing it? Where are people using it where it doesn't apply? I think one of the challenges I've seen is actually using it to ask it factual questions because that's not what it's about. It's not a truth-finding mechanism. And that's just the wrong usage.

Okay, is there something that you wish that artificial intelligence could do right now that it can't do? What's the next exciting thing? What announcement tomorrow would make you happy?

I'll take that question slightly differently. I think what excites me about AI in terms of my research on responsible use of data and algorithmic bias is that, yes, a lot of people have shown that AI can generate biased outcomes. We also know for many, many years that humans generate biased outcomes. And what excites me about AI is that it's much easier to fix

biased outcomes by a machine and to generate processes that will eliminate bias. And it's so much more difficult with humans. And that's something that I'm really excited about. I love that point because we got all this bias and misogyny in our world, not by the machines. The machines are not the people who put us in this situation in the first place. And the fact that they

maybe do a little bit of that at the beginning before we've trained them, that we shouldn't just throw them out for starting down that path because we can adjust the weights and model. We can give feedback to model to improve those.

in a way that we can't with bazillions of people. Right. So I think that's a huge point. And we've seen like the first models of Gen AI images. If you say doctor, we're only photos of men or things like that. And over time, this improved a lot. So that's really exciting, right? We can try to think about how we fix

some societal problems using these things because yes, machines can be manipulated more easily than humans. Of course, that's a risk, but that's for some sci-fi podcast, not for this one. Yeah.

The example of the doctor in the image is spot on because I think so many people were fascinated by how accurate these models are because they felt right. They confirmed our stereotype. You ask for this image and it gives you exactly what you think of as that image. But that's just feeding into the problem again. And that's going to perpetuate it if we don't. But like you say, there has been improvement there.

Ayelet, thank you so much. This has been really insightful and quite interesting. Thank you for being on the show. Thank you so much for having me. This was fun. Thanks for joining us today. On our next episode, Shervin and I speak with Mictad Jaffer, Chief Product Officer at Shopify. Before you do your holiday shopping, please join us to learn more about how little bits of AI everywhere can add up to big value for all of us.

Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn't start and stop with this podcast. That's why we've created a group on LinkedIn specifically for listeners like you. It's called AI for Leaders. And if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and learn more about AI.

and gain access to valuable resources about AI implementation from MIT SMR and BCG, you can access it by visiting mitsmr.com forward slash AI for Leaders. We'll put that link in the show notes and we hope to see you there.