This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life. This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life.

Could this year be the year that autonomous vehicles become the norm, right? You know, I know we've been hearing about how AI is going to, you know, improve autonomous driving and it's going to bring us

you know, fully autonomous vehicles. We've been hearing this for many years. But I think today's guest is going to help really open our eyes and our ears to maybe some of these new technological breakthroughs that have actually happened

changed the research going into this and might make this a possibility. And we are here at NVIDIA's GTC conference, a lot of new announcements when it comes to autonomous vehicles, and we're gonna be going over those and talking about how this might impact

all of us, right? The roads we drive, the safety of the future of autonomous vehicles, and I'm very excited for today's conversation. I hope you are too. So hey, what's going on, y'all? My name is Jordan Wilson, and welcome to Everyday AI. This is your daily live stream podcast and free daily newsletter, helping us all keep up with the

ever-changing world of AI and how we can actually learn from the experts, you know, helping build it all so we can grow our companies and our careers. If that's what you're trying to do, you're definitely in the right place. Make sure if you haven't already, please go to youreverydayai.com. Sign up for the free daily newsletter. We're going to be recapping all of the important insights from today's interview there, as well as everything else that you need to stay ahead. All right. So

Enough chit-chat. I'm excited for today's guest and talk about everything new that NVIDIA is working on in the autonomous vehicle space. It is a lot. So please help me welcome to the Everyday AI Show, Marco Pavone, an associate professor at Stanford University and the lead autonomous vehicle research at NVIDIA. Marco, thank you so much for joining the Everyday AI Show. Thank you for having me.

All right, I'm excited for this conversation. This is, I think, one of those hot topics that people love talking about. But before we dive in, can you just tell everyone a little bit about your background and what you do here at NVIDIA? Yeah, so I'm a roboticist. So I'm a faculty at Stanford, and I also lead AD research at NVIDIA. My work is in the field of autonomous robotics. So how do we make robotic systems capable of making decisions on their own?

especially in high-stakes applications, like for example self-driving vehicles or aerospace vehicles. And prior to joining Stanford, I was a roboticist at another Jet Propulsion Lab. So still working on self-driving vehicles, but on Mars instead of Earth. Yeah, yeah. It's not every day you get to talk to someone that's helped the autonomous... Which project was that on Mars that it was?

It was one of the Mars landing missions. There we go. NASA sent a rover to Mars and the mission was a success. Love that. So yeah, we're bringing, you know, some research from Mars to, you know, your streets here. So, you know, there was a lot that's been announced so far at NVIDIA GTC when it comes to autonomous vehicles. But, you know, one of the things I wanted to talk about is Halos. So can you tell our audience what that is and how it's ultimately, you know, going to impact NASA

everyone else on the roads? Yeah, sure. So, Handles is a full-stack system that comprises hardware, software, tools, and safety principles to combine all of these elements into a safe driving stack. And it's exciting because it's basically unifying all the investments that NVIDIA has been made in the past

in the past few years on the topic of automotive safety into a unified program. And I believe this program is going to boost safety both with respect to the, you know, own AD program internal at NVIDIA, but also helping partners in making their program, their programs even safer.

So, you know, can you just bring us up to speed, right? Because, you know, even people, you know, who've listened to this show, we've talked about autonomous vehicles and AI, you know, in years past, but bring us up to today. What, you know, what has been successful? Because there's obviously, you know, fully self-driving cars on the road, right? In certain states where it's allowed, but, you know, where are we at, you know, in the, you know, fully self-driving autonomous vehicles, what's working? Yeah.

What's not? Well, autonomous vehicles are becoming a reality. And I'm sure that you have heard this sentence many times in the past few years. But like, you know, if you come to San Francisco, you will see robot taxis providing rides to customers without any safety driver on board.

And very advanced driver assistance systems, like for example the Tesla Autopilot, are becoming available on a massive scale. So my point is that the self-driving technology is now graduating into becoming a consumer technology. That's why I feel confident by saying that autonomous vehicles are becoming a reality. We still have challenges. So we have solved the problem yet. It's

I like to draw a parallelism with respect to aviation. It took us like a hundred years to get to an industry that is as safe as it is today and as efficient as it is today. So this is a marathon, it's not a sprint. What are the key technological challenges? Well, in the context of a full set of driving vehicles like robot taxi systems,

their deployments are still relatively limited in few cities in the world. And scaling up those deployments still represents significant technological challenges

as it requires scaling the algorithms, allowing the algorithm to generalize to new situations in a much more effective way. So we need some innovation there throughout the development cycle from simulation to training to algorithm design and so on. Same thing for semi-automated systems.

The sematomated systems are already available worldwide, but of course we want to increase their availability. That's again, that requires technology.

And can you maybe just, you know, help our audience better understand NVIDIA's footprint right now in the, you know, not even just the autonomous vehicle industry, but just the auto industry in general, because, you know, it's something I've found out through the years by getting to talk to, you know, really smart people such as yourself. But, you know, people don't know, you know, you probably don't know.

have or might have NVIDIA hardware in your car, right? Like Tesla is probably using NVIDIA's data, their data centers. But can you just bring us up to speed? What is NVIDIA's footprint right now in the auto industry? Yeah, absolutely. So NVIDIA, first of all, is both a product company and an ecosystem company. So NVIDIA has a substantial investment in developing its own autonomous vehicle solution.

which we call NVIDIA Pride. And it is doing so in collaboration with partners such as, for example, Mercedes. And it's also helping, since it's also an ecosystem company, it's also helping other AV companies to develop their own AV programs in many different ways. There is no like a unique recipe. It could be by providing the NVIDIA automatic grade chip

many of the autonomous vehicle companies out there are using indeed NVIDIA hardware. It may be by providing data centers to train the AI. It could be by providing simulation technologies and so on and so forth. Every company is a bit different, but this is also what from a researcher like myself makes it exciting because I have an opportunity to really scale up my contributions even beyond the confines of NVIDIA to really the entire ecosystem.

So, you know, speaking of different brands or different companies, also some exciting news, NVIDIA and GM. Talk about this partnership a little bit and, you know, when we might see, you know, that partnership actually out there on the roads, right? I'm not going to hold you to it, right? But like what's coming in this partnership? Well, in terms of timing, as you can see, there's a bit of sensitivity. Of course. For men on that.

I would say though that this is a super exciting partnership. I was also involved in the discussion, so I'm really happy to see that coming to fruition. One of the interesting aspects of it is that there's a partnership regarding automotive, but also manufacturing and potentially in robotics. So it's a very broad partnership. Then also,

plays well with NVIDIA's broader ambitions. Of course, as we said before, NVIDIA has a very strong autonomous vehicle program.

But indeed, it's also scaling up this problem to what we refer to as a physical AI program, whereby cars are just one instantiation, one embodiment of a broader concept that is that of physical AI. In addition, for example, to humanoids or autonomous mobile robots and so on. And so this collaboration with General Motors can also involve this kind of broader vision.

So like, simply like, does this just mean in the future, you know, are we going to see, you know, different versions of GM's vehicles have autonomous capabilities or is the long-term goal with this partnership to have maybe most or all of GM's vehicles? Like, what's that going to look like? Is it just going to be kind of like certain vehicles in the future are going to, you know, kind of benefit from this partnership or is it just kind of all vehicles mostly?

like longer than I'm alive.

Okay. No, no, that's fine. That's fine. No, okay. All right. All right. We'll, we'll, we'll just have to follow up on that when the, when the news does come out. But, you know, so I'm curious. It seems like, you know, very fresh out of the press. Yeah, it's extremely fresh, extremely fresh. So yeah, we'll, we'll follow up with that once, once that news is officially released. But, you know, one thing that I do want to talk about is, you know, and we even mentioned this, right? Like there's been a lot of excitement, right? In this space for three, five,

10 years. What specifically do you think has changed over the last couple of years that leads you and your team to believe that, you know, now is this time, you know, that, you know, we are kind of hitting that moment when this might become much more common, autonomous vehicles on the road?

So for as well even more than 10 years, right? It's true. Yeah, doing my PhD between 2006 and 2010 at MIT. It was the time where autonomous driving technology was starting its very steps. I would say it's almost been 20 years. Okay, there we go. So the way we build autonomous systems today is very different from what we were used to do 20 years ago, of course. There have been

of course a lot of lessons learned but most importantly the technology has changed and there is no like this

single technology that has really changed the game completely, but a convergence of technologies all the way from hardware, so basically having dedicated chips and dedicated sensors, all the way to, as you can imagine, AI becoming pervasive in the design of autonomous systems. And in that context, I think one of the most exciting

opportunities, it's still a little bit debated here in the community. It's the opportunity of leveraging so-called internet pre-trained models. You might be familiar with GTPT, for example, with the idea that with this type of models, we can

we have an opportunity to bring internet-scale knowledge to the task of driving. Think about how you learned about driving. It took you a few hours probably to learn how to drive a car simply because you brought a lifetime of experiences to the task of driving. Well, that's a part of this about behind using this kind of interpretive models to bring multiple lifetime of experiences of generalist knowledge to the task of driving. So that's another AI in general and photocopying

potentially, internet pre-trained models in particular provide the key opportunities to improve the technology. And another, I would say, big technology that has made an amazing process in the past two, three years is the simulation technology. And we actually had a number of announcements related to simulation. Simulation has always been a holy grail in robotics.

And now finally we are simulators that we can use throughout the development lifecycle from the training of the vehicle, the training of the AI, all the way to the testing of the AI. So the key, the challenge with simulation historically has been the so-called simulation to realism. Yeah.

And this gap is becoming increasingly closer along a number of dimensions in terms of visual realism, in terms of behavioral realism, how faithfully we replicate the behaviors of humans in the road and so on and so forth. So long story short, I wouldn't say it's a single technology that is really pushing this industry forward. It's really a convergence of technologies from the chip all the way to the algorithm or to the simulation that all of this, you know,

Now, I'm finally coming together. Yeah, and I do want to talk about that a little bit more. So kind of this concept of using, you know, NVIDIA's new generative AI technology, Cosmos, correct? So like walk us through that. And, you know, I know this might be difficult, you know, to imagine on the podcast. So in the newsletter, we'll link to some of these videos, you know, and how Cosmos helps

But walk us through these simulations and how specifically, right, maybe, you know, since we've hit this generative AI wave, how does that help with, you know, NVIDIA's ability to use more diverse simulations? And how does that make ultimately the autonomous vehicle sector safer? Yeah, that's a great question. So,

Typically, simulation is restricted to the scenarios that are authored by a human. So a human saying, okay, I want to test a vehicle with a particular intersection, so I'm going to draw a map with respect to which a vehicle has to drive.

It's fine, of course it doesn't scale to millions of cases, right? Or there are new technologies referred to as neural reconstruction technologies that allow you to reconstruct in 3D scenarios out of drives that are recorded. This is all fine, but for autonomous vehicles it's really a game of the last 5% or 1%. It's all about thinking about very complicated corner cases.

And that's where generative simulation comes in. So this new technology, and Cosmos is one of the prominent examples, allow you to simulate, allowing you to generate a simulation out of textual prompts or images. So this allows you to create a completely new simulation scenarios to, for example, stress test your vehicle. So it really allows you to

automatically and in a way it is highly scalable, generate a plethora of corner cases that can allow you to better more robust systems and also test those systems. Now it's still a technology in development, so there are still challenges. For example, physics realism is a challenge to what extent is generated simulations, for example, obey the law of physics.

But there's been quite a bit of progress in improving the physics realism. So I'm very hopeful that this technology will be yet another tool that has an autonomous engineering leverage in order to build more capable and safer autonomous vehicles. And it's not just for NVIDIA.

I believe that the cosmos and broadly generated simulation is going to have a significant impact in the whole industry, but more broadly in the whole robotics industry.

Yeah, and I think that's really important to bring up because, you know, especially if you've been to a city like San Francisco or, you know, I've seen, you know, Waymo's and in Austin, Texas, right? So these vehicles and this technology has, you know, it's been out on the roads, right? It's out there in the wild, which, you know, allows NVIDIA and other players in the space to gather that actual data.

real life data, right? So, you know, I'm wondering, you know, if the simulation side is improving and you're able to, you know, simulate more scenarios with the Cosmos platform, you know, what are still some of those bigger hurdles, right? Aside from, you know, just more time and, you know, more data from the real world, from the cars, right? What are some of those other big hurdles that the space is still looking to maybe overcome?

Well, data is a big one. So simulation is going to help, but you still need to have data, real data, to ground your system. So the hard role is to make technologies, AI technologies, that can adapt with an increasingly lower amount of data to new areas. And it is a problem that maybe is a bit difficult to appreciate, but it's crucial. Transportation is a very location-specific phenomenon.

I don't know if you have Italian followers. I'd like to give an example that actually this is also a tip for you. I'd love it. Please, please. I have to. I have to soon. If you go to Italy and someone blinks the lights at you, typically that is a kind sign. It means that you can cross in front of me. I'm looking at you.

If you go to the south of it, that is typically blinking, it's an aggressive sign. It means that don't you dare cross in front of me because I'm not going to stop. And if you cross me, we're going to crash. So it's the same country, a few hundred kilometers apart with two completely different behaviors. So this is what I'm saying. You still need to have some location-specific data to train your AI. So then the game is...

but that is expensive to acquire. So how through better simulation and better algorithms we can decrease the reliance on real data. There always will be some need for real data. The question is how we can reduce it so that we can really quickly expand

to new domains, technically we refer to those as operational design domains, to really make this technology financially viable. We know it's technologically feasible. Waymo is developing a robot accident in San Francisco, but to make it financially viable, we have to be able to scale it up. Scale it up means to be able to kick a data flywheel that is not too onerous in terms of how much data we need.

Simulation is one tool, but better algorithm design is another tool.

driving down the cost of some key sensors. Like for example, as you might have heard, there's quite a bit of discussion in the community in terms of to what extent you want to have a sensor system that is very much camera-centric or maybe also relying on ladders and so on and so forth. So these are additional discussions. Again, it's not just a single technology, it's a combination of technologies, but I would say

the capability of scaling operational design domains more seamlessly is the major challenge which will be solved through a combination of a number of technologies from redundancy at the sensor level to simulation to algorithms that adapt more quickly to new scenarios with the best

That's, that's a fascinating example, right? Because I never thought about that, that, you know, it's not just a one size fits all approach for autonomous vehicles. Yeah, yeah, you're like, definitely think about that. So like, like, as an example, right? Like, even I'm thinking in the United States,

people have different driving styles right from state to state city you know big city urban areas right so you know I'm curious you know what are some maybe uh successes that you've found in addressing those or how do you even go about knowing those things right aside from like you know you you know because you know you you have lived there you know but for everyone else and for all these other challenges that maybe the industry hasn't thought about I mean how do you you know start to

these issues? Is it just, you know, maybe, oh, there was an accident and we don't know why because, you know, all of our data was right. And, you know, is that kind of how you discover these things? Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on GenAI. Hey, this is Jordan Wilson, host of this very podcast.

Companies like Adobe, Microsoft, and NVIDIA have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for chat GPT training for thousands or just need help building your front-end AI strategy, you can partner with us too.

just like some of the biggest companies in the world do. Go to youreverydayai.com slash partner to get in contact with our team, or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on Gen AI.

Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Gen AI. Hey, this is Jordan Wilson, host of this very podcast.

Companies like Adobe, Microsoft, and NVIDIA have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for chat GPT training for thousands,

or just need help building your front-end AI strategy, you can partner with us too, just like some of the biggest companies in the world do. Go to youreverydayai.com slash partner to get in contact with our team, or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on Gen AI.

Yeah, so that's one of the reasons why the EV industry has been increasingly shifting from a paradigm whereby most of the possible cases were hypothesized by humans and then coded into the brain of the autonomous vehicles to a data-driven paradigm where we let the autonomous vehicle learn from the experience, basically from demonstrations, because that is more scalable as a technology.

So, all these new behaviors essentially are to a large extent learned from data that is acquired through other test vehicles, even maybe dash cam videos. The good thing is that videos are bound on the internet. Yet another opportunity to more seamlessly scale up the AI to new operational design domain. So, bottom line is that these days, many of those behavioral nuances

are learned through data, which again means that we need to have technologies that allow us to make as much use, as much efficient use of this data as possible. The good thing is that data abounds. So we do need data acquired from a test fleet, for sure. But one thing that abounds on the internet is videos, and especially driving videos. And that is again a new technology

modality that as an autonomy engineers we have at our disposal to allow the scale of this technology in an even more efficient. You know, and I'm curious because on the data point, I may be wrong in this assumption, but I'm guessing very early on a lot of the data that you might get, you know, from those driving videos, let's say if there is a bunch from five years ago, I'm guessing that the other cars for the most part were not

autonomous, right? They were driven by humans. So, you know, I'm curious, like, what are you all at NVIDIA? And again, maybe just the broader industry doing to account for that, right? Because what if in five years, it's 10% autonomous vehicles? And, you know, how can you say, oh, in the training data, this was an autonomous vehicle versus this is a human? And, you know, what challenges does that

you know, bring in the future? Will it just be, you know, autonomous vehicles be able to more communicate with each other so they know, okay, you're an autonomous vehicle, you know, you have the halo system, so this is what you're going to do.

That's a great question and there are a lot of sub-questions. So, first of all, as you alluded to, now that we're moving toward a more data-driven paradigm, then the brain of the autonomous vehicles becomes very much dependent on the data that you use to train the brain. So it becomes imperative, and this is one of the hallmarks of KELOS, to develop AI-equated workflows that allow you to remove

unsafe behaviors or biases from your training set. And interestingly, this is another domain where we use internet pre-trained models now not as drivers, but as judges that allow us to judge whether a given demonstration is

is a demonstration of safe driving or not. And of course, there are humans in the loop to align the judgment of these AI models. So we as humans, we think it's acceptable driving, but we train these basically AI models to serve as judges at scale to remove all those biases that you were mentioning. Now, moving forward, AD is a very weird technology because we are solving the hardest problem first.

going back again to aviation you know i have this kind of dual uh aerospace uh and earth-based background it's like as if the black brothers at the first problem they wanted to solve was a supersonic flight that's the state of the av industry we are solving the hardest problem first because we're solving a problem where we there are only few automated vehicles everybody else is a human like you have a human driven vehicle so there is no dedicated infrastructure

But is this basically what it is? Now, in the future, there will be a higher penetration of autonomous vehicles. So your question is how that will change the technology. Definitely provides an opportunity to make the technology even safer. But as with everything, you have an opportunity, but you also have a challenge. What is the challenge?

Well, there are multiple. So let's assume that, for example, autonomous vehicles could communicate with each other. In principle, that is great because it allows some level of coordination, which has clearly an immediate impact on safety. But it exposes your decision-making capabilities to external interference. So, for example, cybersecurity becomes much more of a threat than it is now when the system is basically very much confined.

within the vehicle. It might introduce latencies. Sometimes when you have a call on your phone, you don't care if your communication drops for a little bit. That could be fatal in the case of an automated vehicle. So not to say that these are impossible challenges, I'm just saying that it's not as simple as people might think.

that vehicle-to-vehicle, vehicle-to-infrastructure might simplify the problem. Let alone the challenge of who is going to place that infrastructure, how you're going to standardize that infrastructure. So yes, when there will be higher penetration autonomous vehicles, there will be opportunities to make this technology even safer.

But the exact mechanics about how we were doing is still subject to our discussion. So Marco, we've covered a lot in this conversation. So, you know, as we wrap up, what do you think is maybe the one most important thing for our viewers and listeners to know about

you know, specifically even new advancements that were announced here at GTC and how that is seemingly going to quickly change the future of autonomous driving on our roads.

I think there were two announcements broadly that are going to have a significant impact in the field of vehicle autonomy. First, all the announcements related to simulation and second, announcements related to foundation models, internet training models that would be used in the context of physical AI.

One of the big announcements, actually this was a month that was made at CES and then it has been defined at GDC, is that of Cosmos, which is by now a sort of umbrella term where we cover both video generation models, particularly useful for simulation, and reasoning models, particularly useful for autonomous driving in the real world.

So these are definitely technologies that are worthwhile to keep in mind if you are an autonomy researchers. Many of these technologies are available open source on the Alpenface and at NVIDIA. That's one of the reasons why I'm excited about being at NVIDIA.

We publish a lot. We share a lot of our knowledge. And again, this is because it is both a product company and an ecosystem company. So we want to make sure that as we grow, the entire ecosystem really grows. I love it. I think I just became so much more informed about

on everything, you know, autonomous vehicles, what NVIDIA is working on. And I really hope that our audience did as well. So Marco, thank you so much for your time and coming on the Everyday AI Show to share with us. We appreciate it.

Likewise, and if you want to become an autonomy engineer, you're welcome to take some of my courses. There we go. Hey, at least I'm part of the way there, right? I went from zero to one, I think. So, hey, that was a lot of fantastic information. NVIDIA is working on a ton of advancements. So if you missed anything in there, if you want to know more, we're going to be recapping today's conversation in the newsletter. So thank you for joining us. If you haven't already, go sign up for that.

newsletter. Read it today. It's going to be a great one at youreverydayai.com. So thank you so much for tuning in. Hope to see you back tomorrow and every day for more Everyday AI. Thanks, y'all.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.