Ep518 - Ravi Bapna & Anindya Ghose | Thrive: Maximizing Well-Being in the Age of AI
01:01:04 Share

Welcome to the Talks at Google podcast, where great minds meet. I'm Shay, bringing you this week's episode with AI professors and researchers Ravi Bhatna and Anindya Ghose. Talks at Google brings the world's most influential thinkers, creators, makers, and doers all to one place. Every episode is taken from a video that can be seen at youtube.com/talksatgoogle.

Ravi Bhatna and Anindo Ghose visit Google to discuss their book Thrive: Maximizing Well-Being in the Age of AI. The book explores how AI can positively impact many aspects of our daily lives, from health and wellness to work, education, and home life. Artificial intelligence is a powerful, general-purpose technology that is reshaping the modern economy.

But misperceptions about AI stand in the way of harnessing it for the betterment of humanity. In their book Thrive, Ravi Bhatna and Anindya Ghose counter the backlash by showcasing how AI is positively influencing the aspects of our daily lives that we care about most: our health and wellness, relationships, education, the workplace, and domestic life.

The authors explain the underlying technology and give people the agency they need to shape the debate around how we should regulate AI to maximize its benefits and minimize its risks.

Bringing over two decades of experience with cutting-edge research, consulting, executive coaching, and advising to bear on the subject, Bapna and Ghos demystify the technology of AI itself. They offer a novel "house of AI" framework that encompasses traditional analytics, generative AI, and fair and ethical deployment of AI.

Using examples from everyday life, they showcase how the modern AI-powered ecosystem fundamentally improves the emotional, physical, and material well-being of regular people across the globe. Moderated by Mary Maruyama. Here are Ravi Bapna and Anindya Ghose. Thrive, maximizing well-being in the age of AI.

I'm so excited and so thrilled to talk to you all about Thrive, maximizing well-being in the age of AI. So we have Ravi Bhatna. You founded the Analytics for Good Institute. I can't wait to talk about that. The Analytics for Good Institute at the University of Minnesota.

and you're the inaugural INFORMS IS Practical Impacts Award winner for your work in analytics and digital transformation. Welcome. Thank you. And Anindo. Anindo Ghos is an award-winning professor of business at NYU Stern. Pretty famous. And a globally renowned expert in digital transformation and the applications of AI in business.

You're also the author of the best-selling book " Unlocking the Mobile Economy." So after you read this one, also read "Tap." Welcome to Google. Thank you for having us. You're welcome. So I believe you're going to be starting with a presentation, which I'm very excited to listen to. So after the presentation, we're going to have a Q&A, talk a little bit more about what you presented and additional questions from the book.

And then after that, we're going to open up to questions from Googlers around the world and here in person. So get ready with your questions and feel free to ask in the Dory or come up to the microphone live. So thank you. RAVI BAPNA: All right. Well, good afternoon, everyone. Such a pleasure to be here and to tag team and do this with Auroindo. My name is Ravi Bapna.

The motivation for us for writing the book was twofold. Anindya and I have been dealing in this world of getting companies to use their data to make better decisions for the last 20 years. So there was kind of like this collective 40 years of experience of looking at how companies are thinking about using the data to get smarter and to become more intelligent.

And one of the recurring themes that we found, which was actually backed up by industry studies by companies like Gartner, was that actually companies are using only a tiny fraction of the data that they have. So this number is like 3%. So literally like 97% of the data that they have, they're not truly using it to get smarter and to make better decisions.

And there are multiple reasons for this which we can get into. We start off by thinking about this as the three I's of inertia, right? People are doing something in a particular way, let's say doing a demand forecast in a particular way, and they don't want to change, right? Change is hard. Ignorance or lack of awareness of the art of the possible, right? And so that was a big motivation for us to demystify how to do this. And then also then, you know, the lack

of imagination and kind of the consequent lack of innovation that comes out of that. So this was one big motivator. And if data is the new oil and AI and analytics is the engine that's going to leverage that oil and convert that into energy, it felt really important for us to write something that helps not only the layperson but also executives and leaders in companies. Speaking of AI, the other big motivation was that

In comparison to prior general purpose technologies, you can go all the way back to the steam engine or the printing press or even a computer or electricity, those are all tangible. AI is much more complex, it's layered, it's these algorithms lurking in the background influencing who you date or what books you read or what movies you watch

or whether you get a loan or not, or whether you get screened by a resume screening algorithm for a job or not, right? And that's hard for people to digest, right? If you can't see it, you don't have a relationship with it, that becomes hard. And that then has downstream consequences, including all the way to policy and to think about the sequence of how we can use this really powerful technology to automate or to innovate or to regulate, and we'll talk more about that as well.

So that was the whole motivation for writing this book. And as a part of doing this, the idea of giving people agency and understanding and leveraging AI

we essentially ended up exploding a lot of myths around AI that are out there. So myth number one, it's all about Gen AI. This is the shiny new object, everybody's talking about it, it's caught the imagination of people. Actually some of the smartest minds in the industry are still betting on the fact that our traditional AI, and we'll define that in a second,

in the next three years, that is still going to generate close to 70% of the value. And that's why, like any good academics, we start our book with a framework. We call this the House of AI. Everybody is welcome into the House of AI, which has a foundation of data engineering. So this art of cleaning, aggregating, transforming data, making it interesting and useful to then do things like descriptives,

where you might be using machine learning to detect anomalies if you're a financial institution. Of course, prediction is a huge pillar. We emphasized this whole art and science of separating correlation and causation as being a really important part of the overall House of AI, the framework, optimizing things using prescriptive. So these are the basic pillars.

I will spend more time today in the layer above that which gets into more richer data. So using more complex models, deep learning models to go beyond simple tabular numeric data in an Excel spreadsheet to getting insights from images and audio and video and text to get into environments where we might be making sequential decisions and optimizing over a long-term reward function like reinforcement learning. Obviously, we'll talk about generative AI.

and then doing all of this in an ethical, equitable, ideally explainable and fair manner. So this is the core of the book. And let's take a quick tour of the top layer or the second from top layer of the House of AI.

with a couple of interesting use cases, right? And I'll start by just showcasing the value of rich and granular data that we get in today's day and age, right? So in our book, we have this example that we talk about, a use case from Whoop, which is again, a company right here in Cambridge, Mass, which is tracking our body's metrics

at a very, very granular level, right? Getting, for example, heart rate multiple times in a second, right? And it turns out, for example, in this picture that you see out here,

The x-axis is time until birth, right? So imagine a woman going through pregnancy and there is time until birth. The y-axis, the metric here, is heart rate variability, okay? And it turns out there is this pattern of decline in the variability of the heart rate till about seven weeks from birth, and then there's an inflection point, right? So this is the regular pattern, right? Now imagine if you're living in rural North Dakota

And the nearest obstetrician that you have is maybe 100 miles away in Fargo. In fact, 35% of US counties today are what are called maternity deserts, where they don't have access to maternity care. Now the WHOOP pregnancy coach, if it detects that this inflection happens in week minus 10, will give you a warning.

We'll say that, you know what? You are probably on track to have a premature delivery out here. Go and see a doctor today. And that can save lives. So this is just-- we haven't even built a model out here. We're just getting new insights just from that rich data that we have in today's environment. On top of that rich data, we can make models. So again, a really fun example from our health care chapter in the book.

goes all the way to Budapest and Hungary, right, where Mama Clinica built these deep learning predictive models to detect, take images from mammograms and detect whether a particular cell is malignant or benign. And

The AI tool that they built was given to the doctors. It often agrees with the doctors, but it has flagged areas which the doctors in the mammograms, the doctors missed out. And since 2021, there was a New York Times story that said that the AI had 22 cases of cancer that it diagnosed, which the doctors had missed out. So this is literally like 22 lives that were saved using the particular model.

Thinking about other use cases of deep learning, this picture is from in the front of my house in Minneapolis, which has a lot of lakes in the city. So one of the fun things for us to do in the lake is swim in the summer in these lakes,

till we see this kind of a sign, right? Temporary beach closure due to high bacteria levels. And I walked over to the beach one day and I saw this lady from the city, the Parks and Recreation Board putting up the sign and I asked her like,

how frequently do you check this? Like, what's your process, right? And she said that, oh, you know, we come in, we take a water sample once in two weeks, and then if the bacteria level is high, the sign is going to be there for two weeks. OK? Imagine.

Now, water is a dynamic thing. It changes literally every second, right? So a better way to do this, and again, there's a BBC story now talking about this new use case, where we're taking sensors embedded in the water, we're taking satellite imagery, we're taking land use records from public sources,

And we can do this in real time so that the sign can be there maybe for an hour in a digital format and then it can go away because the water's clean now, already out there, right? So simple example on the environmental side, but deep learning models have been used to detect fires, wildfires in California much faster than the current process as well. So lots of use cases on the environment side. So that's an example of using richer, in this case, image data.

Thinking about reinforcement learning, let's go to the world of people analytics, and let's think about companies like Google and Amazon and others, hiring for tech jobs. I'm sure when you post a tech job, you probably get 1,000 people applying for that job. And it's pretty time consuming to screen those 1,000 resumes that come in

and figure out, okay, who's the first set of people to have a phone call with or to have a Zoom interview with, right? So the resume screener was a very, very popular use case for AI about five years back.

And it turned out when Amazon rolled out its first version of the resume screener, it got called out in the media because it associated being good at tech with being male. So women were basically kind of getting screened out by that AI algorithm, and Amazon had to shut it down. But the modern way of doing this is saying that, you know what? Wait.

we can be smart about taking the data that we have and instead of focusing on those candidates that are right now being scored the highest by the algorithm, let's strategically explore and exploit.

Okay, so reinforcement learning algorithms will say you know what 20% of my capacity out here 20% of my of my decisions I'm gonna I'm gonna figure out a way to explore other candidates that might not be the top score right now But it may it'll allow me to bring in some underrepresented minorities. It might allow Allow me to bring in some some women and when I talk to them I discover they're amazing at tech jobs

So this reinforcement learning algorithms in the context of thinking of hiding as exploration turns out to be a very good use case of an advanced model, in this case reinforcement learning, to eliminate bias and hiding. Which brings us to AI myth number two.

The media likes to hide the negative around AI. In fact, that was another motivation for us to write the book. They will find these corner cases of societal biases that occur and those become headlines, but actually it turns out that why do these biases occur if you take a step back

These are because of data generation processes that have, you know, that reflect social norms, that reflect historical trends, and that is, in some senses, if that's the starting point, right, that's why,

we had that case with Amazon, right? There were not enough women perhaps even signing up for STEM courses in high school. That then causes a sort of sequence of events. But the interesting part is, and our previous example showed, that more AI and smarter AI is actually gonna help us deal with these biases, right? So that's myth number two that we wanna explore. Speaking of Gen AI,

Another myth that's out there that's percolating right now is that, hey, if you can get really good at prompt engineering, you're gonna get all this kind of interesting value for you and your company from Gen AI. So all you need is to get really good at prompt engineering and there are all these, there's quite an industry of gurus out there telling you how to do good prompt engineering.

Actually, I think the evidence that's out there is rather mixed. If this is the only thing that we think about, right, Anand and I started looking at what's the research telling us about the relationship between, you know, gen AI usage and productivity and work,

And for every solid research paper that basically claims that generative AI benefits non-experts more or reduces experts' performance due to hallucination, there are other studies that are telling us actually that it benefits experts more since they know the terminology and context better, right?

For studies that talk about how it improves performance on complex tasks like software, there are other studies that have shown actually it reduces performance when tasks get too complex. Studies showing it enhances diversity of solution. In another context, studies are showing that it actually harms the diversity of solutions out there, right? So the lesson here is that at this point in time,

our understanding of the fundamental forces is somewhat limited. Our ability to deploy this correctly in our organizations is also limited, right? And a lot more needs to be done to truly get value out of generative AI. And in particular, we encourage companies that we work with to go beyond prompt engineering, right?

Think about integrating the proprietary data that you have. Let's say the records of the call center agents that are your highest performers,

If you can marry that data with the intelligence of these large language models and fine tune a solution like a chatbot for customer service, and even doing something as simple as providing more context in a rag format, this can start creating much, much more value. So there's a pretty big runway for companies to think beyond prompt engineering and starting to get true value out of

of generative AI, right? So the final myth that I want to explore before I hand it over to Anindya is kind of this idea that when we say, oh, you know, you're advising, let's say, a Fortune 500 company outside the tech sector that, oh, you need to think about fine-tuning, one of the common complaints we get is, oh, we don't have the talent in-house, right, to do that fine-tuning, right? So there's another kind of myth out there that the bottleneck in getting, you know, value from AI is the lack of data science talent, right?

I'll tell you a story. I was at dinner with one of our former students from our graduate programs in analytics and AI that Anindya runs at NYU, I run at Carlson School, one of the co-founders, and I asked the student, how's it going? Are you enjoying your new job? She said, yeah, prof, everything is amazing. We learned a lot in your program, but to be honest,

while you taught us all these fancy reinforcement learning algorithms, what they're making me do in the company is running reports and visualizations in Excel.

And that actually points to the fact that it's not the supply of data science talent that's a problem, it's more actually the demand, in this case internally from this Fortune 100 company, from senior VPs and directors and leaders who don't have the imagination to exploit the house of AI. They don't know the use cases, right? So in fact, in our book we call for a Marshall Plan to re-educate leadership and executives

in the art of the possible. So with that, I want to hand it over to Anindya and he's going to talk to us a little bit about other angles including innovation regulation.

Thank you, Ravi. So what Ravi set up for us was sort of what the world looks like today, especially with respect to organizational demand and skill sets. And I want to sort of take us to the next stage, which is what might the world look like tomorrow or in a few months or a few years from now. And I think that conversation is hard to have without at least talking about regulation, right?

We often joke that there are three kinds of people in the world. When they hear the word AI, one kind says automation, the second says innovation, and the third says regulation. So which are you? So I think here's sort of the context. Part of the reason why we wrote the book Thrive is to sort of not just give ourselves into this fear mongering that's happening about AI, which is what Ravi was saying,

We are not dismissive of the negative aspects. We are just saying that nobody really talked about the positive, compelling other side of it. With that being said, I think it's also hard to ignore the possibility of deception and the possibility of collusion, so I'll talk a little bit about that.

And then the other part of what I'll talk really about is the world of antitrust. And being at Google, you guys are very well familiar with what's happening. Governments around the world are chasing you guys, trying to figure out whether we should break up the company or not. But rather than single out a single company, I will give you my predictions, our predictions on what we think the world of antitrust and AI will look like.

Okay, so first let's start with deception and collusion. I think you'd agree with me that defining deception is incredibly hard, right? Especially in the world of AI when the thing is not tangible, right? For example, if the consequence is negative, whatever the consequence you're trying to measure is negative, how do you know if it was deliberately done or was it an unintended inadvertent consequence? It's not very easy to tease out that part.

But before any consequences are actually measured and even some actions are taken, you have to figure out was it deception or deliberate or not.

For example, if there is liability that will be imposed on the organization or the company that let's say has engaged in deception, how do you know whether it was intended or negligence, right? Like sometimes, you know, the best data scientists also may have missed out some flaws in the training data, right? It's not like they were deliberately trying to mislead people, it's just that, you know, they couldn't spend enough resources figuring out what the flawed data could be.

And so what happens in such cases is,

Traditional doctrines of liability don't really work as well, okay? Because they're not contextually relevant for AI. So in the past with product marketing, when there was mislabeled advertising, we would look at issues like misrepresentation and then there's a very clear liability part. If a brand has intentionally mislabeled the benefits of its product, then that's misleading advertising, right? And there would be consequences. But it's not very easy with AI, right?

The next issue that arises with this is, how do you account for consequences and who is responsible? Is it the creator of AI? Is it the deployer of AI? Is it the end user? Who's actually responsible for this? And finally, is it possible that algorithms in the world of AI can actually lead to unlawful collusion, like pricing collusion?

So these are all like open questions and I think we are still kind of scratching the tip of the iceberg with respect to what we can actually measure and then what kind of consequences can we take. And finally, like what is a liability over here?

One of the most interesting use cases of AI that you're seeing is reduction of barriers to entry. So now I'll take you to the world of antitrust, where in all of these large antitrust cases being battled out right now, a big issue that happens is that larger companies have access to lots of data which create barriers to entry, which prevents new companies from entering. That's generally sort of the agreement.

What's really interesting with AI is that now we are seeing that even in like traditional digital advertising, that access to data as an advantage is rapidly diminishing. We no longer see that the larger firms have that advantage as much as they used to. And part of the reason is advanced AI techniques like reinforcement learning, transfer learning, is making it possible for smaller entrants to come and compete head to head with incumbents.

Intuitively, what is really driving that is that if an incumbent that specializes in digital advertising is about to compete with a new entrant, the new entrant essentially is in the business of figuring out how to create a new digital ad. And these pre-trained JNI algorithms make it really, really easy for them to enter the space.

And getting more into some of the details, like Ravi talked about rag models, right? One of the features we see is that these AI algorithms actually have diminishing returns to data, okay? That means your advantage with more data starts to plateau over a certain point in time.

A, we see that these pre-trained models work really well. And once you fine tune them, then you can make even more progress. So put it simply, smaller startups that had a cold start problem in entering the space no longer face as much of a hurdle. Because now they have access to the algorithm, the data, the embedded intelligence, the weights. So what used to be a real stumbling block is no longer the case.

So that's right, that's a really important feature of AI which changes the barriers to entry, which then has huge implications for how the government should be thinking about antitrust. The second aspect of what I want to talk about is the infrastructure layer itself. So the bottom layer, this is a model layer that many of us, and especially you guys in Google are very well familiar with, right? These are the large language models from DeepMind or Facebook or Hugging Face and OpenAI and so on.

On the top are essentially the application layers. So the reason we see a lot of companies entering into this application layer is because they have access to this infrastructure. In the past, they didn't have access to the infrastructure. Now they have it. And you already see that. We already see that significant amounts of activity in each of these layers with new startups.

The problem is not so much in entry. We are seeing a lot of entry. The problem is also not so much in revenues. On an average, a lot of these companies have already broken into $100 million in revenue. Problem is in profitability and retention.

And so what is interesting is that while Gen AI is actually letting new entrepreneurs and startups enter, because they have access to very similar infrastructure and similar training, they're not able to differentiate their products. So these verticals are heating up with a lot of new entrants, but they're all using the same infrastructure. They have very similar data sets. And so retention and profitability is a big issue.

So we'll see how that goes. But taking you further, this is really interesting. And this is, I would say, a fairly simplified but yet complete version of the Gen AI stack.

Here, in the bottom layer, you have semiconductor companies, chips, right? And I'll give some examples. On top of that, you have the cloud computing companies. On top of that, you have these large language models, AI models. And on top of that, you have the applications, right? So in the previous slide, I talked about these two layers. But below this exists some very important infrastructure with cloud computing and semiconductor chips. Now, that's where things get very interesting, right?

I'm going to show you a more detailed version of this on the next slide to give you a sense of who are the companies, what products they're building, and how that starts to cause issues. So this, each layer over here on the right corresponds to the layer on the left, right? So at the bottom you have cloud computing companies, semiconductor companies like I mentioned, NVIDIA, everybody heard of NVIDIA with the GPUs, but Google also has TPUs, Transit Processing Units, right?

But then there's also Amazon. How many of us knew that Amazon actually produces its own semiconductor chips? They're called Tranium, and they're competing with the likes of Nvidia and Google and AMD.

Then you have the cloud computing platforms. So the big three are Google, Microsoft, and Amazon. And this is a big CapEx investment for them. These three companies on their own, they spend $100 billion in cloud computing just in maintenance. Now there's been some entry by a company like Oracle, but they haven't really done as well.

On top of cloud computing, like AI models, there are two kinds of models. There are the open source models with the way it's being released, and then the closed source models, right? You know, OpenAI is closed source. Lama from Facebook is open source. And then finally, you have the application layer, okay? So now, I'll sort of raise a couple of possibilities. Take a look at some of the main companies who are competing in this space. In chips, you have NVIDIA, AMD, Google, Amazon.

In cloud computing, you have Google, Amazon, Microsoft, Oracle. In AI models, you have Google, Meta, Hugging Face, Microsoft, Amazon. And in applications, again, you have Google, Meta, Microsoft, Amazon, Apple, and of course, many other startups. So if you just eyeball this, what do we start to see? We start to see the same and often similar names appearing in each level.

So now if you're an economist like us, you're half economist, half data scientist, you're probably thinking, okay, this can lead to some really interesting and important regulatory issues, right? I'm not saying that these are happening. I'm not saying these will happen. But what we are saying is these might happen, okay?

Okay, so when the same handful of names appear at each level, right? So what are some possible regulatory concerns? Number one is bundling and tying. Okay, so classic economics, right? Bundling essentially means that multiple services are bundled and sold as a package. And tying essentially means, well, you know, if you want to use my product B, you also have to use product A.

So what are some possible issues that might arise? Number one, is it possible that larger firms bundle their Gen AI products overshadowing the smaller firms? Maybe. Is it possible that the control over the AI Text Act by a handful of firms, even though they are competing, they still have control, they can leverage that to limit opportunities for some of the new entrants? It's possible, right?

Number three, could there be some upstream firms, like going back to this deck over here, could some of the upstream firms, right, practice exclusive, you know, dealing exclusive practices to stifle competition? Maybe, right? But these are all, while I'm proposing these theories, these are possibilities that's hard to discount, and so that's why we think these are regulatory issues that are likely to come in the next year or two.

In addition, market consolidation, if again, a handful of companies are able to vertically integrate, like what do you do integration is a possibility, right? Let's say, I'm not going to name a company, but let's say company X says, hey, if you want to use my Gen AI models, you also have to use my cloud computing and you have to use my chips too.

So that's an example of what we call vertical integration, where again, A, B, C, D, or E, they're like right now five or six companies, they might say if you wanna use our downstream products, you have to use the upstream products too.

And lastly, I think we are going to start to see these interesting battles between first mover advantages and second mover advantages, like first mover disadvantage and second mover advantage, because these things are rapidly changing over time. And so yes, there might be some positive feedback loops for the initial entrance, but they're also making mistakes, which thus the latter entrants are realizing, and they're fixing that before they make the same mistakes.

So I think these are classic economics, IO 101 questions, you know, top of the mind for the world's premier regulators, including the DOJ, the FTC here, European Commission, and we think this is gonna be an issue. And lastly, I'll sort of close by saying that this is also where economics would meet geopolitics, right? Why is that? And I'll just point out one thing. Look at this last stack over here, right?

There is one company that you may have heard of called TSMC.

TSMC is basically the fabrication factory for everything. All the chips that are being produced, right, with the exception of Intel has some of its own fab factories, but TSMC produces chips for all of these companies too, right? Because essentially they are just putting wrappers around the fabrication and every major superpower existing or future wants their hands on TSMC, okay?

So every time you hear this imminent battle between US and China, it's about Taiwan, right? It's about Taiwan because Taiwan owns TSMC. TSMC is building chips for every single company over here. Whoever has their hands on TSMC is going to be in a very powerful position.

All right, so I think I'll close by saying hopefully between Ravi and I, you get a good sense for what has happened so far, what is happening now, and what we think will happen in the future. And you have a copy of Thrive, so hopefully you all keep thriving. - Okay, so first question. How much AI did you use when building your presentation? - Surprisingly little, actually. - Or none. I mean, you know, I think, I actually don't even know how to build a PowerPoint slide using AI. I should learn.

So there's so many questions that can be asked. What are the most common questions that you get? Because I'm sure that this is not the first time that you've given a presentation on AI because it's so thoughtful. And I'm so glad that you gave this talk at Google because you went in even deeper than what you had in your book. So first of all, read the book before talking to these guys. But second of all, what are the most common questions that you get?

I can get started and then you have to. So I think for me, you know, we both talked about this book in various parts of the world. And obviously there's some similarities in the questions but some differences. One question I often get, and you may get the same, is are you guys saying that, you know, there is like no negative, no downside to this widespread implementation of AI?

And it basically goes back to the reason why we wrote this book. We wrote this book not because we are dismissive of the negatives and the downsides. We recognize it. But we wanted to clarify, just like any general purpose technology, it can be used or abused.

And there's been a lot of good that has come out of AI. We've documented that in the book. And so we are not dismissive of the negatives, but we just basically want to share the other side of the story, which is that there's already been a lot of positives. Yeah, and I think actually, Sandra's question on Dory is probably very similar to a question that's very common, which is, how does somebody who's new to this start?

So, leave aside the top 50 tech companies in the world, if you go into even the rest of the Fortune 100, Fortune 500, you go to the medium-sized companies, those companies, their capacity to absorb the innovations in AI actually is still very, very, very nascent.

And honestly, there is really no shortcut out here to education and learning. So embrace lifelong learning. Of course, our book is a great place to start because it gives you a high level picture without getting into the weeds of the actual technicalities.

But if you read this book and it excites you, then you can follow that up with-- there's some wonderful courses by really smart people on Coursera that you can start saying that, you know what? Let me learn how to build a predictive model. Let me learn that process.

And I think, and by the way, even some of the smartest CEOs and chairmen of the boards, people who, like Risto Salisma who turned around Nokia, somebody asked him, what's your secret to success? He said that actually it's going back to school. So he said that, he was the chairman of the board of Nokia, and he said that I took a month off and I decided to go back and learn how to program machine learning systems. So I think there's really no substitute to that.

And we're both educators, so we'll plug our programs. At Carlson, at Strong, we have great business AI programs. So anybody listening in, please consider applying. So why don't you as professors stop doing what you're doing and just build your own models?

- You wanna take that? - Yeah, I mean, so we actually do, so both in our research context and consulting, we are constantly working with companies and helping them build models to solve whatever their most pressing challenges are, right? It could be around predicting customer churn, it could be around detecting early machine failure,

These are use cases that we see a lot. It could be about figuring out, okay, what's the best design of your referral marketing program and how do we optimize the call to actions? So if you look at Anindya's Vita, or look at my Vita, I mean, his is 10 times mine, but there's always active research going on. And in fact, as we speak right now,

we have developed a bot that is going to allow executives and MBA students, managers, who don't want to actually code but still want to use machine learning. So generative AI is going to democratize access to the house of AI in some senses.

And we're already kind of at the forefront in building a solution that a manager can talk. So English is like new programming language they say, right? And a manager can talk and

and say that, you know what, can you help me build a model to, or help me build five different models and compare their accuracy to predict whatever they're interested in out there, right? So that's kind of going on as we speak. - And just to piggyback on that, I would say that yes, he's right. We are both very active in helping companies build these models, whether it's marketing mix models, price optimization, advertising.

ROI measurement. I think in any given day we probably spend a good chunk of our time every day helping real world companies do this. But I think when you ask the question why don't we build modules, it occurred to me that maybe it's helpful to talk about just the big picture which is

in the stack that I talked about, right? So the model building comes at the application layer mostly, and maybe we leverage the LLM layer. But here's why most of the entrepreneurial activity that we will see will continue to be in that top layer,

not so much in the subsequent layer because of the cost, right? So when a company has to build its own LLM, even going beyond fine tuning and pre-training, there was a graph that Ravi had, that's 40 to 50% of the overall cost. And this is the cost that includes cloud computing and all that. So bottom line is it's very expensive for an individual or a smaller company to get into the bottom layers.

right, semiconductors, incredibly, incredibly high capex, cloud computing, incredibly high capex, even building LLMs very high capex. So people like us will mostly be in the top layer, in the application layer. - Which is where innovation happens, right? Because that's where we're solving real problems that people are facing. It could be in their supply chain, it could be on their marketing, yeah. - So with regards to models and

what you're talking with your customers with, because they're also consultants. They're not only writers, they're not only professors, they're not only on multiple boards, I'm sure, but you're also consultants and you're talking with enterprises on a regular basis. It seems, when you look at the news, it seems like some of the biggest models that people talk about are Gemini, Anthropic, OpenAI, of course, but

In your experience, what do you actually suggest to people when they say, I don't even know how to get started? Or what do you use out of the box? Like, how do you approach that conversation? Especially now, because we are talking about right now board-level AI strategies are super important.

I think it's not like a one size fits all. So different companies whose boards I'm on, I first try to understand their context, their infrastructure level, their capabilities. And there's a wide spectrum of heterogeneity in how they actually are. And it's not like the larger companies are necessarily the most sophisticated. In many ways, the smaller companies are actually more. They can move fast. So I think you have to first understand the context.

And then it becomes sort of who is the end user and what they are personally most comfortable with. You know, the general rule of thumb, some people will say, well, if it's textual content creation, maybe you should use Anthropic, because Anthropic is really good at writing essays.

So I think there's some truth to that, but typically I would advise them to use multiple models and look for what we call robustness. Like are different models giving you the same or a highly similar core set of results?

And if you think that, then that's good. So rather than having them just adopt one right away, and I personally would, at least on the boards I sit on, I would advise them, like, use multiple models, look for robustness, and then see. And then adding your own contextual data to it, you know, either in a RAC format, right? Maybe going to the extent of fine-tuning, because that's when things start getting really interesting, right?

And I think the real scarce resource, again, going back to a point we raised earlier, is imagination.

in the executives, knowing the art of the possible, seeing all these use cases, seeing enough repetitions of these so that they can identify a new opportunity to say that you know what, I would love to be able to predict the sentiment in this type of text and hey, I can do this really fast now if I can use an existing model out there. So the speed to do some of these things has really come down because of advances in Gen AI.

Now, going along this same talking point to a level deeper of the data, which I'm, as you all are data scientists, I love talking about data because it is really, really important, especially as people are looking at strategies and where do they want to move, how do they want to move, what do they use to move it.

What is your best practice when it comes to the underlying data? Like how do you, because right now we actually have an ML and data workshop going on with some customers across the way here. But do you see a commonality of people shifting to more open clouds? Or how do you see the data supporting all of the movement going on to support AI?

So I think there are two schools of thought out here, right? So there is a set of companies that come to us saying that, you know, we have this ton of data. Can you help us get some-- you know, sort of-- and oh, by the way, we actually invested tons of money in setting up a data lake, right? And can you help us get insights from it, right?

And often that's actually not a really good approach because the other approach which is more maybe driven by business, somebody in finance says that you know what, we are a large hospital and we're having great difficulty scheduling our nurses out correctly, right, and our demand forecasting is way off, right, and can you help us solve that problem?

And then we say, oh, you know what? Like, yeah, I can use your time series data to do your demand forecasting. But if I add these seven other covariates, maybe other, let's say, macro factors, it could be even something as simple as the climate. If I can get in that temperature or other data, then my forecasting accuracy, that data combined with some advances in models, my forecasting accuracy-- and we've shown this actually can go up pretty significantly--

There now, instead of spending a lot of time creating these data lakes, fishing for all kinds of data, fishing for insights, you have started with a business problem and you've solved it actually, right? And there's value right there to be seen, right? So I always encourage people to think about, I think again the scarce resource is people who can identify good problems to solve, right? And once you do that now, we are at the stage where we are starting to see

like many of the, you know, like the coding difficulty is going to go down because of Gen AI, right? It's a, you know, good co-pilot in many contexts, right? Well, maybe not mention co-pilot.

Well, that's right. Yeah, sorry, we're at Google. Yeah, yeah, yeah, yeah. I mean, you know, I love using Colab these days when I'm writing my code because there's always, you know, it basically, you know, finishes my prompt, my comment before I even think about it, right, which is amazing. Yes, everybody use Gemini, please. Thank you. So just sort of going back to your question, you know, how do I think about data when I talk to companies? I think the first thing I tell companies is,

Every data set I work with in its raw format is dirty, is messy, there's a lot of issues. Whether it's a small company or a big company. And so if you remember the framework where Ravi was presenting, right, in our House of AI first chapter, the very first layer is data engineering. And what we tell companies is that before you start getting into any of the cool, really sexy modeling stuff, please use 70% of your resources in cleaning your data.

Because in AI, in the world of AI, the classic problem is garbage in, garbage out. If you do not spend a lot of time-- and 70% is our recommended time--

in cleaning the data, then you are not going to be doing justice. So you can't really make any inferences from any intervention you create afterwards. And so the first point of conversation is when you think about data is recognize it's going to be raw, dirty, messy. It has missing observation, missing variables. It has issues with formatting, processing. - Unstructured. - Yeah, so you've got to clean that data. That's not very cool because it's a very tedious job.

But that's the first thing we tell companies. That's why that's in the bottom layer. Foundational, yeah. And even in the programs we both run, in the Capstone program, that's the very first thing we tell students. Do not come to me in the first six months with any super interesting modeling. Focus on the data cleaning part. So that's the conversation to have. Do you think either generative AI or AI in general is going to...

automatically clean it up as we, like the new models that are coming out, be able to recognize unstructured data or be able to use the data mess that's already underneath it. That's a technical term.

- There might be some of that, but at the same time, I personally at the moment with my team in the industry or my PhD students or my co-authors, I still encourage them to do the manual human intervention, which is

Look for outliers, look for these missing variables, missing observations, try to understand that. Before you automate the cleaning part, try to understand why these are being created. And this goes back to our de-biasing issue, right? In theory, you can let Gen AI figure out some of this, but I personally am not there yet, despite being...

you know, an AI advocate. So I would say step one is human intervention and then step two, once you figure out the reasons and the patterns and then maybe you can automate it.

- Yeah, I would agree. And I think the good thing is that, you know, Gen AI can help you do some of this work faster, right? So it can write the code for you, but you have to know what to do, right? And that's where I think that's the key part, I think, out there. So it's gonna improve the productivity of data scientists going forward. They should be able to solve bigger problems, actually, in some senses.

There's so many more questions I want to ask you, but I do want to recognize that there is a question on Dory. So what resources do you recommend for someone who wants to learn how to use generative AI? I believe you mentioned it a little bit, like using Coursera and some other programs that are out there. I think there's a whole series of offerings that are out there.

I honestly haven't evaluated all of them myself, so I can't say one is better than the other. I think our colleague at Wharton, Ethan Malik, I think runs a blog or a sub stack that I think points to a lot of

interesting resources out there. So I would refer people to there. That's a good starting point in some senses. Yeah. And you as professors, how do you keep up to date? Because I do AI for my day job, so I'm very lucky in that way that I can use my time and resources to always stay up to date. But how do you do it and how do you inspire your students and your enterprise companies that you work with?

I mean, for me personally, I don't think there's a one-stop shop. You basically just have to keep your eyes and ears open. We both are on Twitter. I follow certain individuals on Twitter who often share knowledge. Basically, it's free knowledge, free Substack.

So I personally don't have a one-stop solution for this. So just basically keep your eyes and ears open. There's a lot of, obviously, unhelpful information on the internet. So I think over time, maybe it comes with experience. Over time, you're able to figure out what's really helpful and what's not.

But again, for me, it's like, yeah, I don't think I recommend one source. And one of the things we talk about in the book is that if AI can do lots of different intelligent knowledge tasks,

what should humans think about, right, in some senses, right? And it goes back to this point that actually, like, both of us are very fortunate to be embedded in very rich and diverse networks of colleagues, of coworkers, of industry partners, many of them who are outside of academia,

And I think both of us, fair to say, I think both of us spend a lot of time and energy in nurturing those networks, right? Those human connections. There's really no substitute for that, right? I mean, I came in a day early here for this talk. I could have come in this morning, but I came in a day early because I wanted to go and have dinner with my colleague, Gord Birch, who's one of the smartest people in this space. Every time I spend an hour with him, I learn something.

And I'm working on a project right now that is trying to look at the effect of managers using generative AI in their ability to learn about AI. So there's a study that I was getting into

And yeah, within five minutes, you know, God was able to point me to the most relevant paper that has done, that has findings that are opposite of mine, right? And that conversation, that human interaction, actually, that is going to become even more valuable over time. And I think people underestimate the time of developing these human connections, of, you know, developing these relationships, nurturing those networks. They don't happen automatically. Right?

So there is a section. Everybody follow God, because that's the one-stop shop. And then everybody follow Ravi. I'll pass it right back. But I think we're also fortunate being more senior in the community. We've had this long entourage of really smart PSA students. So believe it or not, even though I'm the advisor, I still often reach out to my ex-PSA students who are now professors. Like, dude, how do I fix this problem? And God is one of them, but there's many others. Panos, Vilma, and so on.

Yeah, one of the things I wanted to talk to you about since you mentioned it, I feel like education is definitely changing in this world of AI, generative AI. Students are using it. Students are thinking, I'm not going to say thinking that they're smarter than professors, but sometimes they feel like with the technology in their hands, they can get those answers. So how do you see education changing and what would you suggest the industry does to continue to support itself and more importantly, to support the students?

Yeah, I mean, I can take a shot at this. I don't know if you'd pipe in. I mean, I think we really are at the cusp of something fundamental, I think, in education, right? With the technology that we have already today, and entities like the Khan Academy have shown this with KhanMego, me and some colleagues have deployed similar kinds of technologies, basically personalized,

either virtual data scientists or personalized tutors in the context of managerial education. So we have this ability now to basically give each kid, like literally every kid that's out there that has access to a smartphone, access to a personalized tutor.

Something that was only accessible to really elite families in super high, upper middle class neighborhoods maybe. We talk about Kanmigo in the book. The way that is structured is that if a kid is stuck in calculus and let's say asks for the answer to solve a particular calculus problem, it's not going to give you the answer. It's going to give you the next step

the next question to ask to then eventually get to the answer, right? Or conversely, let's say another kid is stuck, doesn't like to read literature, right? Or doesn't, you know, has got to read a book and doesn't want to engage with that.

Maybe you can have a conversation with one of the characters in the book, right? So I think what, you know, that's at the grade school level, at the high school level. I think we're going to start seeing, and we are already at the forefront of experimenting with this in managerial education in the world of, you know, business schools, right? And schools that adopt this are going to, you know, create managers who are better trained, who are going to go out and be more productive, right? And as we always say, like, you know,

AI is not going to replace managers. Managers who use AI will replace those that don't. I think the other message relevant here is-- so your question is great, because I often get this from prospective students, that how do I keep up with the latest things in AI? And how does your program keep up?

And what I'd remind them is that, you know, we in our programs, the NYU Business AI program, similar to Carlson's MSBA program, we focus on the concepts, not so much on the next new shiny object, because that thing will always keep changing.

And so how we keep abreast with the latest and the greatest in the space is by teaching in class the fundamental concepts. Like what is deep learning about? What is econometrics about? Those things. Because the tool that will use the concepts, that will keep changing. Six months from now you'll have the next new tool that is even better at experimentation. But we don't really focus so much on the tool.

It's almost a means to the end. We really want students to focus on the concept. So I think that's sort of the message to anybody who's interested in figuring out how do we keep track and stay updated is focus on the concepts.

- Yeah, and you mentioned something when we were speaking earlier. It's interesting because sometimes as students who graduate or for those of us who are in the field and we think and we talk about it every day, we wanna talk about a specific aspect of AI and then when you go to the person who's maybe responsible for decision making, they ask us, oh, wait, wait, wait, we don't wanna talk about that yet. Start with this instead. So can you talk about that a little bit too?

- Sure, I mean I would say, Ravi and I talked about this the other day. We often in our consulting work meet organizations who claim they're into AI, but when you go under the hood, they're running a logistical regression.

That is not AI. It's a very small part of the journey in analytics. So the flip side of your question is, we ask our executives and students to recognize what is snake oil from what is bona fide. Because the fact of the matter is, whenever a new technology comes in, there will be many organizations and companies who will be selling snake oil. And so a big part of what we try to teach our students and executives and companies is, please spend enough time recognizing it.

And it's a process, so I think over time this conversation becomes even more deep. But it's a good starting point now. Thank you. We only have a couple minutes left. Are there any questions from the audience?

If not, I have one last question, if that's OK. So your book was written in about 2022, 2023. Now it's about December of 2024. And as you know, things are changing, and there's new updates and new technologies out there. So my question is twofold. First, what is your favorite new product that you've seen come out since you wrote the book? And second, what is the next mountain you're going to climb?

You can take the first one. All right. So I love listening to music. And my younger brother is actually doing a PhD in generating music out at a really top university in London.

So I've been playing around with music generation and merging classical with jazz, and he keeps sending me stuff. So personally, that's really exciting for me, just based on what I like to do. But I think what I'm, for me, the next mountain to climb is helping companies, again, outside the top 50, outside the Googles and the Airbnbs and the Ubers of the world,

In Minneapolis we have 19 Fortune 500 companies, in retail, in biomedical devices, in supply chain.

how do these companies start climbing the AI summit? Because their ability to absorb all this technology, even if they start using 20% of the components in the house of AI, they are going to get multi-million dollar returns. And this is something I'm not just making it up because we've been doing this for the last 10 years. So we know that. So getting these companies to get smarter about leveraging this technology, I'm really excited about that actually.

I will answer your question literally, because I literally am a mountaineer. I'm a certified guide as well, so I climb. So two weeks from now, my family and I are going to be in New Zealand. So we're going to be climbing over there. And then in the spring, my daughter and I are going to be in Nepal, and we are going to be doing some pretty cool trekking, some climbing stuff in Nepal in MLS.

What kind of devices are you going to be wearing for those climbs? Yes, Garmin all the way. Oh, not Google's product.

So for life and death situations, Garmin's pretty good. Yeah. Your book was so interesting, how it combines not only the base level understanding of AI, but ways that it really is helping in the medical field, helping save lives. And again, this is what we love to do at Google. So thank you so much for coming and for talking and for answering so many questions that I have. And I'm sure everyone else has a ton of questions too.

But thank you again. Thank you so much for having us. Thank you for hosting us. Thank you. Thank you for having me. Thanks for listening. You can watch this episode and tons of other great content at youtube.com slash talks at Google. Talk soon.