883: Blackwell GPUs Are Now Available at Your Desk, with Sama Bali and Logan Lawler
01:04:03 Share

This is episode number 883 with Sama Bali from NVIDIA and Logan Lawler from Dell. Today's episode is brought to you by ODSC, the Open Data Science Conference, and by Adverity, the conversational analytics platform.

Welcome to the Super Data Science Podcast, the most listened to podcast in the data science industry. Each week, we bring you fun and inspiring people and ideas exploring the cutting edge of machine learning, AI, and related technologies that are transforming our world for the better. I'm your host, John Krohn. Thanks for joining me today. And now, let's make the complex simple.

Welcome back to the Super Data Science Podcast. Today, we've got not one but two exceptional and complimentary guests on the show. Sama Bali is an AI solutions leader at NVIDIA that specializes in bringing AI products to market. Prior to NVIDIA, she held a machine learning solutions role at AWS. She's focused on educating data scientists and developers on AI innovations and implementing them effectively in enterprises. She holds a master's in engineering management from San Jose State.

Logan Lawler leads Dell ProMax AI Solutions. If you haven't heard of ProMax before, we'll cover that in this episode. Over his 16-year tenure at Dell, Logan has held positions across merchandising services, marketing, and e-commerce. He holds an MBA in management from Texas State. Today's episode will be particularly appealing to hands-on data science, machine learning, and AI practitioners.

but it isn't especially technical and so can be enjoyed by anyone. In today's episode, Sama and Logan detail why data scientists are camping out at 6 a.m. to attend NVIDIA's GTC conference. They talk about the killer specs of NVIDIA's next-generation Blackwell GPUs, how Dell and NVIDIA have joined forces to bring server-level AI power right to your desktop, and how microservices are revolutionizing AI development and deployment. All right, you ready for this excellent episode? Let's go.

Welcome to the Super Data Science Podcast. It's awesome to have two guests, not just one on the show today. Logan Lawler, where are you calling in from today? Hey, John, thanks for having me on. So I am calling in from, well, I'm going to say Austin, Texas is technically Round Rock, Texas, Dell's corporate headquarters. Very nice. And then we also have Sama Bali on the show. Sama, where are you calling in from? Hi.

Hi, Joan. I'm calling from San Francisco Bay Area. Unlike Logan, I'm actually at my home. Nice. And for our YouTube viewers, they can enjoy Sama's beautiful background. It's outstanding. It's like it's, you know, 99th percentile of guests we have on the podcast in terms of, you know, it's got it's very peaceful, but it's also a bit officey. Just really nice colors. I love it.

And it's NVIDIA green. It is NVIDIA green. You don't like the carpet color back there? The carpet color? It's pretty good. Logan's in a cubicle. It looks very corporate. It looks like he's in Austin, Texas in a cubicle. Yeah, exactly. And he is. And I am. Do you have any fun coworkers around you that you like, you have like Nerf gun things or whatever shooting at them in the cubicles?

Honestly, part of my earlier career, there was a lot of that at Dell. Like there was a lot of hijinks and shenanigans. I would say as I've progressed in my career and got a little older, those hijinks have kind of went away, which is kind of, you know, hurt my heart a little bit because I quite enjoyed it. Mm.

That's too bad. I remember the last time I worked in an office with cubicles, some people had like remote operated. It wasn't Nerf brand, but it was that same kind of idea of firing a completely harmless dart with like a suction cup. But they had turrets that you could remote control that were positioned on top of their cubicles. I do miss that. That is something I miss.

about that kind of corporate setup. Anyway, we're not here to talk about desk arrangements. We are here to talk about amazing innovations. And so at the time of recording this episode, we are just a week out from NVIDIA GTC, which is one of the biggest tech conferences in the world. I don't know if you guys have some stats. Maybe it is literally the biggest. So tell us about that experience. Maybe we'll start with Sama since you are formally at NVIDIA and

And so tell us about GTC. What's that like for, I haven't actually been to GTC myself and I bet a lot of listeners haven't either. What is that like? We see so much of it on social media and the news. What's it like in person?

The way it was described, it describes the Super Bowl of AI. And I don't think anybody could have described it any better. San Jose is my college town. So I was really excited to see that we really painted the entire town green with all the innovations, not just from NVIDIA, but also our partners, everyone from cloud partners to enterprises to startups, all of that.

It's amazing to see, and we are in the center of Silicon Valley, but to see that kind of innovation come to life,

It was a great experience for me, I will say that. You could see all kinds of developers filled with data scientists, practitioners, tons of opportunities to really network with people working in all kinds of tech companies at this point in time. Some amazing, amazing... I mean, I never thought I'd be having conversations with so many robots in my life in just these kind of five days. Yeah.

But that happened. So my favorite moment, although we'll be seeing Jensen, who is our CEO at NVIDIA, would

Within the Denny's food truck, I never in my world imagined that there is a food truck from Denny's as well. There was one. He was serving food right before he went and actually presented the keynote. So that was my favorite moment for sure. I'm also a nerd in a sort where I've seen people camp out, you know, Apple offices whenever they're announcing the new iPhone products.

For the first time in my life, I've seen people camp out to get into a keynote venue as well. People were lined up at 6 a.m., which was, again, insane in my mind. So it definitely was a festival in San Jose, I will say.

Very nice. Logan, any highlights for you? You know, I think you, I think Sama more or less covered it. I mean, I, it was my first GTC that I've ever been to. And I think it was a lot. And when I say this, this is meant to be a positive, but in terms of the learning, the education, just the booth set up,

all the options, it was a bit overwhelming, right? Like I think the key, the key thing is to go in kind of with a plan of like, here's what I want to accomplish. Here is the, for example, the learning track that I want to go down, whether I'm a data scientist or, you know, whatever is to have that plan going in because it's very easy to go stand in the middle of the trade show floor and just kind of look around and just be in wonder for hours. But I loved it. It was great. Tons of traffic, very packed wall to wall. It's not necessarily my jam, but hey, we were there.

Fantastic. That sounds like an amazing experience. I'll have to check it out in a future year. Maybe someone from NVIDIA or Dell will invite me and maybe I can contribute in some way actually in the future. I don't know. I hadn't thought of that before, but that could be a lot of fun. All right. So particular to what both of you do,

Dell unveiled at GTC last week, the pro max PCs. So what are those? And the, I mean, maybe that's just, I don't need to have an end. What are pro max PCs? Okay. So we, and when I say we, I mean, Dell, um, we kind of pre, I wouldn't say launched, but we announced these at CES, right? So overall, give me a quick bit of background is Dell, you know, coming this year for many years had, um,

you know, lots of disparate brands, right. You know, from professional to personal use precision OptiPlex in XPS, the list goes on and on. So really the rebrand kind of starts there is how do our customers shop? And that's really broken down into what we've launched with a new brand kind of nomenclature, which is Dell, which is, you know,

for home and basic work. You have Dell pro think of your traditional consumer or your traditional professional, you know, knowledge worker workflow thing like that. And then Dell pro max, which is what I support is really designed for heavy ISV type workloads. Think like, yeah, ISV like independent software vendors. So think like

you know, Katia or just so software Adobe where it's designed for specific workflows within industries that really rely on like heavy GPU compute and acceleration to get their workflows done. Um, so we announced them at CS, we launched them at GTC, which

Is to be honest, more of a kind of an ISG type show. And I was really excited with the amount of, you know, press, but also, you know, the feedback because we've been in the, uh,

you know, supporting data scientists. We've been supporting, that was our precision line before, but with the launch of Dell Pro Max, I think the key thing to take away is we still have our traditional, you know, our towers, you know, our mobiles, that hasn't changed, you know, all accelerated by Blackwell GPUs. The difference is we did, and I know we're going to talk about this, so I won't like, you know, jump the shark too much, but we launched two systems that are specifically designed

for data scientists and developers with kind of system on a chip with Grace Blackwell designs, which really was kind of a difference. And I think a recognition from Dell and the market to say, we know where the market's going. You know, we need to have a purpose built device for data scientists that is easy turnkey that brings the power of a server to the desk site, which I know sounds crazy to say, but that's really what's happening. So that was kind of the big announcement, which we're going to get into more later too.

Very cool. And so I guess the reason why that is announced at GTC is because of the inclusion of NVIDIA GPUs in the Pro Max PC line. Yeah, I mean, absolutely. I mean, that is correct. And

I mean, we're great partners. I mean, that's from kind of an, I won't say inception of AI, but over the last several years, you know, the Dell AI factory with NVIDIA has been kind of a cornerstone to kind of our go-to-market and how we make AI real. So it made perfect sense because workstations are very GPU dependent. It's the only thing that has the ProViz cards in the lineup. So it makes perfect sense to launch a GTC. Cool. And so, Sama, then what's your involvement from the NVIDIA side with these Pro Max PCs?

So I at NVIDIA lead our AI solutions go to market. So my job is to see, you know, when we are taking these new GPUs to market with our partners like Dell, how does that full solution come together? And it becomes a solution once you really add in that layer of NVIDIA AI software to it.

right? And that's what really truly becomes NVIDIA, Dell AI factory with NVIDIA. We've got that entire hardware lineup from Dell, but then along with that NVIDIA AI software, which is kind of consistent, you've got consistent experience. If you're starting from workstations, moving to server, moving it to your data center, moving it back to your workstation as well, it's that software layer which really helps you, one, harness the power of GPUs because this entire software layer is really optimized.

So your data scientists, your developers don't really have to do that fine tuning between the software or the AI models they're using and the hardware at that point in time.

So that really is my job, is to really talk about and manage the go-to-market for that entire full solution of Dell Pro Max PCs along with our NVIDIA AI software. Fantastic. And so we will get to that software aspect in a moment. First, Logan, I think, is the person to direct this at mostly. Let's spend a bit of time talking about the hardware. So explain for my listeners what

what the latest in actually, you know, I'm not, I'm not certain that this is maybe for Logan necessarily, but, um, you know, explain what's up with the latest in GPUs from Nvidia, as well as the different kinds of chips that are required to make something like a pro max PC be such a success for AI developers. Yeah. So I'll, I'll take part of it. I mean, I know Sama will probably provide a lot more detail than I will and probably better detail, but I think we're,

When it comes to, you know, Dell pro max, you know, acceleration by Blackwell, right? The car is at the core of it.

and really any data science workflow, the acceleration, depending on what library was pandas or Polar's or whatever, it's really about how much can you load on the GPU? How quickly does that GP work? And ultimately how quickly can you run that workflow? Now there's some surrounding things around that, that, I mean, GPU is core to it, but there's other things within Dell pro max that, you know, also add to the experience and add to the ability to accelerate a workflow. I mean, our T2 tower, which is even though it was announced last week at launch today, I mean,

people probably won't find this very interesting but the size of that tower increased a little bit with not necessarily increasing the footprint that much so you can add in for example more hard drives for data storage you can add in extra cards if you want to run multiple monitors things like that and doing a few other things being able to run on different applications whether data science or not being able to run kind of the new intel processors at kind of a 250 watt

sustained workload, which is very unique in the industry. Cause that really wasn't the case before. So, you know, all of this has really been personally built when like you take the word max for Dell pro max, it means taking everything legitimately to the max, right? Like how, how, how far can we push this thing? And that that's really Dell pro max in a nutshell.

Very cool. So let's talk a bit more about the GPUs in particular. I get the importance there, of course, of having CPUs. It sounds like ones there that are taking more power than ever before. And so you have power requirements in this Pro Max line.

Let's talk about the GPUs specifically. What is the difference between Blackwell and predecessor GPUs? Why is having a Blackwell GPU from NVIDIA more helpful to someone who's training or deploying AI models than GPU predecessors? I can start, Logan, and then I'm pretty sure I'll miss a bunch of things. You'll be fine. You'll do great. You're great, Salma. Go ahead.

I'll add on. Thank you. Perfect. So I think one of the things we would talk, when we were talking to a lot of our data scientists in-house within our customer base as well, we started

soon realized in the last few years there was a lot of development, AI development happening in the cloud, in the data center. But AI is now becoming mainstream, right? Everybody is trying to now fine-tune a model. Logan and I are not technical people. We are in the product marketing. Or guess what? We are fine-tuning our own models nowadays. We are running these models locally as well.

And we soon realized that the systems we're using, you still need a little bit more horsepower in your system where you have the ability to actually

have these data sets running locally, have these models running locally, have the ability to fine-tune, run maybe a small drag model just for yourself and things like that. I remember talking to a data scientist last year during summer, and he was talking about any time that he is trying to do a job, they actually have to, they had a Slack channel. He had to put his name down and he had to wait in line to get an instance where he could actually run his data set.

So we recognize the need where now a lot of enterprise customers are struggling to get a lot more horsepower. And it's not possible to give away cloud and data center just for learning, just for experimentation.

And it's interesting how AI is still evolving, right? Almost every day there is a new model you want to try. There is a new technique you want to try. So you need to have that local sandbox of experience where you can just do your learning, your experimentation. If I am a developer who's building an AI-based application, I probably want to continue doing all my testing because getting a data center resource is becoming more and more scarce.

So that was a thought process with the NVIDIA RTX Pro Blackwell GPUs. And then we've got a full lineup in desktop and laptop format for you. But the biggest feature for me has been that we've doubled the GPU memory. So we've gone from 48 gigs to 96 gigs per GPU. You can actually have

four of those in one workstation as well. So that's a lot of memory right at your desktop. Wow. You're doing any running, any kind of model locally, fine tuning that model, any kind of inference application that you want to run. You've got a lot of power right there within those GPUs itself, along with our usual, more and more of our advancements we make with the GPU. But that massive memory size to actually run these things locally has been a game changer. Yeah.

Excited to announce, my friends, that the 10th annual ODSC East, the Open Data Science Conference East, the one conference you don't want to miss in 2025, is returning to Boston from May 13th to 15th. And I'll be there leading a four-hour hands-on workshop on designing and deploying AI agents in Python.

ODSC East is three days packed with hands-on sessions and deep dives into cutting-edge AI topics, all taught by world-class AI experts. Plus, there are many great networking opportunities. ODSC East is seriously my favorite conference in the world. No matter your skill level, ODSC East will help you gain the AI expertise to take your career to the next level. Don't miss. Online special discount ends soon. Learn more at odsc.com slash boston.

Yeah, so 496 gig memory. So when I, years ago, I'm kind of dating myself. The last time I built a server, I was buying 1080Ti's, NVIDIA 1080Ti GPUs, which were at that time impossible to get. Everyone was using them for Bitcoin mining. And so like, I'd have to like take an Uber. Living in New York, I'd have to take an Uber to like some distant Brooklyn warehouse to get one

NVIDIA 1080 Ti GPU, and they'd be like, that's maximum one per customer. And then I'd have to try to source one somewhere else so that I could have two in my server. And those had 13 gigs of RAM. So I could get 26 gigs into this server that I'd built. And for a while, actually, with the size of models...

Even in early kind of large language model era, myself as well as a team of three data scientists, all four of us were able to share. I built two of those servers in the end.

And that was sufficient for us. And it would, you know, compare to trying to run something on our laptop with just CPUs. You know, it was crazy, crazy. You're talking about, you know, that 10,000 X kind of speed up to be doing that. And so it's interesting now that the paradigm that you're describing is,

is kind of being reversed because now somebody locally can have their own in, you know, in that kind of system that I was just describing there, you know, we were a small, uh, small AI company with a relatively small data science team, but there was no option for, for us to be buying individual machines that we could be fine tuning our own models. So we were, we shared a couple of servers and because large language models weren't absolutely massive like they are now, that was sufficient. But as you're describing, uh,

as LLMs have become gigantic, it has become very hard to get allocated cloud compute to be, uh, training or deploying AI models. And so it makes so much sense to me that you can have now a local box that is just for you. You can run whatever experiment you want. You can learn on it. Uh,

But you can also do really heavy lifting, especially when you're talking about something like, so if you have 96 gigs on each one of these Blackwell chips, you can fit four into a Pro Max. You're talking about almost 400 gigs. I'm putting you guys on the spot here, but do you happen to know what that would correspond to in terms of model weights in an LLM? So I'll give you general, as long as you don't hold me to it and no one listening comes after. But as a general rule of thumb, every billion parameters in a model requires two gigabytes.

So that in essence, say an 8 billion parameter approximately. Right. And there's some things, quantitization, what precision you're running at FB 64 FB four, it all kind of varies. But I mean, the model size, I think you hit on a huge point is that the model sizes are

you know, like say llama three, four Oh five, like before, not really possible to run on a workstation. And it wasn't that it didn't want to, or didn't have the desire. It just, the technology wasn't there. I think that's, you've hit on a great point is the technology between, you know, Dell pro max, as well as the video blackboard GPUs is enabling people to do things.

And that's kind of a takeaway from GTC. You'll be able to do things you were never able to do before, which I think is super cool. That is really cool. It's interesting. It is kind of a full circle. I mean, I'm even thinking back to...

Back when I was doing my PhD, which is before really the most recent AI era, my PhD finished in 2012, which is when there was a big explosion of interest in deep learning as a result of AlexNet, the machine vision model released out of Jeff Hinton's lab at the University of Toronto. And then all of a sudden, everybody in academia and then in industry was taking notice of deep learning. And actually, it's interesting to have someone from NVIDIA on this call because it's the insight

from Jensen Huang or whoever at NVIDIA at that time to say, whoa, we're building graphics processing units for rendering video game graphics or allowing editors to do video editing, that kind of thing. But we're going to invest a huge amount of money in...

money and time and hiring in specializing in this deep learning revolution that seems to be coming. So the story that I was going to tell, I'm just going to really quickly wrap that up. But back then in a pre-AI era,

I remember I was working at the University of Oxford. We'd have servers that lots of us would be competing for. And it's cool now to think that in that same lab, people could be buying. I mean, I had a very generous, as part of my PhD, I could spend something like 20,000 pounds, so about $25,000 on hardware. And so I could be buying one or more of these workstations online.

and be locally in full control of any LLM stuff that I want to do. So it's a very cool world that we're in. And then I want to get back to the NVIDIA story from around the time and kind of this visionary nature of what NVIDIA has done and reflected in their share price.

is this idea that, okay, deep learning is going to be gigantic, or let's assume that deep learning is going to be gigantic. And so let's build a software ecosystem, going back to your point earlier, Sama, that supports that. So yeah, so tell us about things like CUDA, TensorRT, maybe a bit of the history and why those are so important in this GPU ecosystem and in this AI era.

Yep. I'm actually going to start first with NVIDIA AI Enterprise, right? Just completing the story of how we're doing things, especially with Dell Pro Max AI PCs. So think of NVIDIA

of NVIDIA AI Enterprise as our version of end-to-end software development platform, which is helping you not just accelerate your data science pipelines, but also really helping you build next-gen. It can be generative AI applications. It could be computer vision applications. It can be speech AI applications. And it has a lot of components. We've got NIM microservices. This is

how we are delivering all kinds of AI models as containerized microservices. So literally think of any other, any AI model in the world. We work with open source partners, proprietary partners. We have our own NVIDIA AI models as well. We're taking each of these AI models, putting them into a container and then adding our, you know,

I won't say secret sauce because everybody knows about Tensor or TLLM and all kinds of services which are really helping you get the best inference possible on NVIDIA GPUs. And we're offering them as microservices. And the reason being, and you'll soon start seeing this from NVIDIA perspective that we are providing almost all of our AI software as microservices is because

Things are changing quickly. I'm a developer today who built an application with Lama 3 and guess what? In two months, Lama 3.1 comes and then another two months, 3.2 comes up. So we want to make it really, really easy for people to just swap in the model as quickly as possible without really disrupting that entire pipeline.

So that's NIM microservices. We've gotten all kinds of models from if you want to build a digital human to actually building speech-related applications to now we also have NIM microservices for our reasoning AI models as well. So that's the first component of NVIDIA AI Enterprise. Really quickly before...

It's going to be obvious for sure to you, to both of you, as well as to many of our listeners, exactly what a microservice is. But could you define that for our listeners that don't know just so that they understand what it is and why it's important, why it's helpful? I actually don't have a definition of microservice. Well, I'm going to give you not like a textbook definition, but I'm going to give you a practical definition, right? Cool. Let's say you're a data scientist and you have created...

Let's just pretend a chat bot with llama three. And you create that without a microservice without, you know, an NVIDIA NIMH.

Like Sama said, every time that that model updates, if there's security, all this stuff, you're doing a ton of, I hate to say it, but background tedious work to get that to a point where you can deploy it. Where when things change, for example, if you don't, like that's the whole point of a microservice with NIM is you basically can load that to literally one line of code and the LLM part of it is really done for you.

It is containerized, it's packaged, it's ready to go. So a data scientist can focus on, well, how am I going to customize it or building whatever application wrapper around it versus like, ooh, I need to update the code here to get this to connect. Like that's really the point of a NIM is how quickly can I leverage the power of an LM vision model, whatever, with one line of code. That's the power of a NIM.

And it runs on a workstation too. It runs on Dell Pro Max servers. It runs pretty much everywhere. Yeah, that was going to be my point, that the key point being with these NIM microservices, you don't have to make sure that the AI model is tuned to the GPU, right? We've done all of that work for you. So as soon as you're downloading this locally on your Dell Pro Max PC, it already understands the kind of GPU it's running on. The only thing you have to make sure is

you know, the model you're downloading fits onto your GPU memory size now, but with 96 gigs of memory, you've got, you've got the entire world for you here. Nice. And so I've been trying to, as you've been speaking, I've tried to kind of look up quickly online what NIM stands for. It doesn't seem to stand for anything that I can find easily. It just sounds, Oh, I'm going to let the secrets out. It's actually stands for NVIDIA inference microservice, but then we also use NIM microservice. It's like, it's like chai tea kind of a thing.

they mean the same thing. Potato potato. Yeah. A potato potato. A potato brand potato. Exactly. Cheese queso. That's what I would say. I go ahead to a restaurant, I'll say, I want cheese queso. And then my wife always gives me a hard time. But yeah, cheese queso. Nice. Yeah. Now I understand perfectly. Thank you for giving us that insight. It's interesting. It isn't, it isn't something that's very public. So people really are getting the inside scoop on NIM. And yeah, it's just spelled N-I-M for our listeners who,

who are wondering what word we're saying. It's exactly like it sounds. And I am in all caps. And I'll have a link to that in the show notes, of course. Anyway, so I interrupted you. Oh, go ahead. Oh, I was just on the same topic of name microservices. I was going to say, we've got a website called build.nvidia.com. That's where we host all of these name microservices.

It's a good website to not just go try out these different kinds of AI models. You have the ability to prototype on the website itself. There are no charges for it at all. You can see models, again, by all kinds of partners that you work with, including NVIDIA models as well. They're segregated by the industry you work with or the use case you're trying to build. So it's easy to kind of maneuver around, find the exact model you want to work with.

And then once you want to download this, we've made it easier. So if you really sign up for our NVIDIA developer program, we actually let you download these models and then continue to do your testing, experimentation, free of cost. There are no charges at all. So you can continue. As a developer, I would want to go try out different kinds of models, see what's working with my application. So we let you do that as well.

Ready to take your knowledge in machine learning and AI to the next level? Join Super Data Science and access an ever-growing library of over 40 courses and 200 hours of content.

From beginners to advanced professionals, Super Data Science has tailored programs just for you, including content on large language models, gradient boosting, and AI. With 17 unique career paths to help you navigate the courses, you will stay focused on your goal. Whether you aim to become a machine learning engineer, a generative AI expert, or simply add data skills to your career, Super Data Science has you covered. Start your 14-day free trial today at superdatascience.com.

Fantastic. That was a great rundown. What I was going to say, and I'm glad that you had more to say on NIM microservices, because my transition was going to be that the last time I interrupted you, you were about to, I think, start talking about other aspects of the AI enterprise. So now I'll let you go on that.

So outside of the microservices, we've got Nemo, which really helps you build, train, fine-tune your own models, but also gets you the ability to add guardrails to your model so that whenever you're deploying an application, you are making sure that the application gets used exactly the way that you want to do it itself.

We've got AI blueprints. Think of these as reference AI workflows. We give you the ability to build different kinds of AI applications. Think of this as a recipe. You've got the step-by-step process to actually build an application. There's a reference architecture,

But we also get you the ability to add your own data to it. And that's what gets every company their own edge, right? You want to add your data, which is your differentiation at this point in time. So you have the ability to build different kinds of applications. What else do we have? Oh, we've got different kinds of frameworks and tools. So we actually do support different kinds of AI frameworks like PyTorch, TensorFlow. We also have our CUDA library. So I think this is a good time to kind of talk about CUDA as well.

which really stands for Compute Unified Device Architecture. I didn't know that. I've been using that word for like a decade now. Thank you.

So this really has been playing a crucial role in AI development by enabling efficient parallel computing on NVIDIA GPUs, right? So the idea was its entire architecture really helps you train different kinds of models significantly faster, which means that you can, in some scenarios, actually reduce your training times from weeks to days, right?

It is also helping you get better and better inference. So you see higher inference performance on NVIDIA GPUs because of this architecture of parallel processing if you're comparing it to just CPU-only platforms. We now have, and I'll have to look up the right number of how many CUDA libraries we have, but we've got...

Tons and tons of these CUDA libraries, and these are GPU accelerated libraries. So a good example I'll give you is of RapidSCUDEF, right? So the idea, and Logan touched on this earlier as well, is

the way RapidSchoolDF works is that it tends to mimic the APIs of a lot of data frames like pandas, polars. So if you are in that process of pre-processing your data in your data science workflow, it can actually accelerate that entire process by 100x on our 6,000 GPUs.

without any code change. That's the beauty of it, that as a data scientist, all I'm doing is adding that one API line of code and then it actually

accelerates the entire process by 100x. So that's like massive time saving from a data scientist perspective. At GTC, we announced QML, which is again, one of our CUDA libraries as well. This is helping you accelerate your machine learning tasks as well. So if you're using Skitlearn, you have the ability to go up to 50x acceleration for your ML tasks as well. So each one of these libraries, and as I said, we've got tons of these right now,

But depending on the data science tasks that you're doing, these are all designed to then offload that work to the GPU so that you can see that massive acceleration. Nice. QML is a new one to me as well. I inferred correctly that the Q isn't like the letter Q. It's like the beginning of CUDA. And so it's C-U-M-L. And yeah, I'll have a link to that in the show notes. Looks really cool. GP Accelerated Machine Learning Algorithms.

designed for our listeners, designed for data science tasks. Thank you for the tour, Sama, of all of the amazing things that NVIDIA is doing on the software front for people who are training and deploying ML models. Logan, can you fill us in on how that relates to the Pro Max systems that you are so involved with?

Yeah, I mean, absolutely. So it was, we kind of talked about before, right? All the new NVIDIA kind of Blackwell GPU architecture, the 6,000 on the way down really is designed for Dell pro max purpose built for Dell pro max. And it's a little bit beyond Dell pro max, but let me give you a perfect example. Like, you know, Sama was talking about, you know, AI enterprise, right. Which is really kind of at the heart of any kind of data science workflow. Well, Dell, we

we sell pcs but we also sell servers right and where it really fits and ties in is kind of two parts one is that if you're using for example a dell pro max t2 um you can do that work ai enterprise you know clean your data sets refine do some fine tuning experimentation all of that you know leveraging qdf everything through there but let's say you want to go then deploy that that is where it becomes very seamless using ai enterprise to take it from a dell pro max

to the in-house server for deployment or taking it up kind of for bigger experimentation, right? Like that's really the layer that connects kind of everything that we do from the desk side all the way to the data center, which makes it very seamless. But then I'd be remiss not to talk about, we talked about, we'll probably talk about it more, is the Dell Pro Max systems that are really purpose-built for developers and data scientists being the GB10 and the GB300. Those are,

Where, you know, if you were to buy a Dell pro max T2, you know, there is a cost of AI enterprise, right? But if you look at those two systems, all of those are designed really with all the Nvidia stuff preloaded, ready to go. So it's out of the box, you plug it in and you're off to the races. And that's very different than the other Dell pro max systems where I would say maybe not too technical, but let's say someone here is a data scientist for, you know, meeting entertainment company. They're out there. They exist.

And if you're doing anything outside of data science, you're going to want to be in a traditional Dell Pro Max system.

But if you're doing only data science, that's where you're looking Dell Pro Max GB10 or GB300. Because, for example, Creative Cloud doesn't really work with Linux. Like it's just not designed for it. So you have to really make that distinction. But that software package is kind of the connecting glue from desk side all the way to deployment, whether you're doing it on a server cloud, etc. Okay, so let's get. Oh, sorry. Go ahead, Saba. I was going to do your part, John. And I'm going to ask Logan to actually describe how GB10 looks like.

Like, that's how much you'll be dead. Cool. Yeah, let's do that. But I have got one thing that I want to get. Before we start talking about these specific PCs, you gave an example there of something that was a creative suite. What was it that you couldn't, that wouldn't run on Linux? Yeah, so perfect example. So Adobe Creative Cloud, which is, you know, think Premiere Pro, photo editing, video editing, all of that, is...

that's something that's unique is that all of our precision workstations previously or workstations in general, which could do data science or they could do any other traditional kind of media entertainment workflow, video editing, et cetera. But there is kind of a line in the sand, which I think,

would probably make most developers on this call and data science in this call very happy is this is a purpose-built system and the people have to kind of think like hey if i'm only doing data science we really need to lean towards the gb10 or the gb300 if i'm doing anything on top of data science then i really need to be because you have to kind of look because that system only comes with linux which like nvidia gdx linux

GB10s, GB300s, which are models of Dell Pro Max, which we'll talk about in a second, those are Linux-based systems. That is really interesting to hear. The question that I had in my head that was starting to come up was, I have personally been programming or doing data science on Unix-based systems for a long time now. That is an interesting... What I was going to ask, and actually, so here is still a question that I think

is hopefully really interesting for a lot of listeners and is definitely interesting for me, is why should I consider switching from a Unix-based system to perhaps a Windows-based system as a data scientist? Well,

I mean, a couple of things is that you don't necessarily have to with, with the GB 10 and the GB 300, because that is kind of Linux space. And that's why, right. At the end of the day, we know, I mean, I'm not a data scientist, never claimed to be one, but y'all have Linux and that's great. It works well, accelerates well. I'm a windows guy. I've always been a windows guy.

Um, I mean, there is some optimizations, not that we can really talk about with WSL two. It makes that seamless transition a little bit better. I personally like windows, not trying to change your mind, but if you're doing anything outside of that.

It does sometimes make it easier to run on Windows, depending on the applications that you're actually using and just from a compatibility standpoint. But yeah, if you're a data scientist, you like Linux, that's the whole reason for GB10 and GB300 is you like it, it works, you're used to it. There you go. And I would say you don't even have to choose, especially with GB10, because...

I'm going to steal your thunder here, Logan. It literally fits in the palm of your hand, John. It's this tiny small box. Logan actually has maybe a phone. Well, I had one. I brought one back from GTC, but this is a representative example. Probably actually bigger. Box of Kleenexes? Probably chop off that. Exactly. I mean, legit, like...

That's actually useful given that most of our listeners are only listeners and not viewers of YouTube. It's actually maybe even more useful that you just brought out a Kleenex box. That gives our audio listeners an idea that you took a Kleenex box and cut off about a third of it. Yeah, but going back to my point, you can actually be using a Dell Pro Max AI PC with Windows running for all of your productivity apps and have the GB10 attached to it

And you can let it be, if you're training a small model, find you, it can be on the side, right? You can actually daisy chain two of them together. So you can actually have a Dell Pro Max, like a laptop, if you're a mobility person and have two GB10s network together. I can't, is it NVIDIA X Connect or whatever the connection is. Connect X. Yeah, Connect X, yeah. So you can actually have two and it's smaller than pretty much every, even that setup would be smaller than pretty much every desktop tower that we sell.

And that kind of gets you the ability of not, you don't have to choose. You can continue doing all of your productivity workflows on Windows if that's what you're used to. And then if you're also doing your data science or developer tasks, you can easily do that on Linux by having best of the both worlds together. Nice. So this is similar to the situation that I was describing earlier where I was talking about, hey, you know, I hand built these servers with NVIDIA GTX 1080 TIs in them years ago.

and four of us on the data science team would log in through a terminal. So that's a similar kind of idea here where you could be using whatever operating system you want on say the laptop you're typing on, but then when you want to do something with an LLM,

You open up a terminal window, some kind of window for accessing that machine, and you run from there. Okay, that's really cool. And so now we understand that a GB10, you can blow your nose with it, and it's very compact. Tell us about the GB300, the other kind of Linux-based. So the GB300 you are not blowing your nose with because it is –

I don't really have a representative example, but it is a traditional, you know, sized kind of fixed tower. What, and what I think is so exciting about this is, um,

One, it's kind of run, it's running on the grace Blackwell, like ultra super chip. So when we say GB grace Blackwell, right? So this is a kind of an integrated system on a chip design that has 784 gigs of unified memory, 288 specifically for GPU. 496, hopefully my math is correct of kind of CPU memory and GPU.

Tops wise. I know that that's a term maybe audiences heard of or not, but it's basically Tara operations per second is at FP4. That is 20,000 tops.

And let me just give you a set kind of context of that is that within the RTX cards, the Blackwell cards, just the singular 6,096 gigs, that's about 4,000 tops approximately. Right? So this is a very, very server level, powerful system that is designed for, I mean, you could really put this thing in a data center and it would act like a server, but it is at the desk side and the power of

And I mean, just the whole thing and the whole design, I think it's not going to be for everyone at the end of the day. Right. But for those that are heavy data scientists working in an enterprise, this is going to be a system of choice for you. Just,

I mean, honestly, with the horsepower that it's built, it's kind of insane to be completely honest. I actually see teams using this where they can easily access multiple, multiple people can access at the same time. John, the way you were talking about how you had a small team of AI developers and data scientists, they could be using it at the same time, plugging in, getting their work done. But it's again, on the desktop, it's on-prem. You get to keep your data private right there with you.

Exactly. And I want to add on to that because that's a good point is that, you know, the multi-instances of that, it's very important is that at the end of the day, just being very transparent from the knowledge that I've gained is that when you look at companies that have, you know, gone on, you know, that have started down kind of an AI approach.

you know, path or journey, right? It's usually the bigger companies because you know, there's costs associated. It takes time talent where I, what I think, and then just generally, I'm not a server guy. I've never been a server guy. It is not that they're bad. I just don't know much about them. Right. That's a skillset that I don't have. Um, everyone kind of has a client.

you know, desktop, you know, skillset. And I really think with the GB 300 kind of like to Sam's point will bring is if you are maybe in a smaller company, you're more of a mid market and you don't have servers and you don't want to mess with it. This gives you the ability to really bring AI to, you know, your company, whether it's, you know, rag model, fine tune, something, build up some adjunct gaps, whatever you want to do. And you're not having to go out and, you know, get racks and cooling and all the other things that come along with servers.

This episode is sponsored by Adverity, an integrated data platform for connecting, managing, and using your data at scale. Imagine being able to ask your data a question, just like you would a colleague, and getting an answer instantly. No more digging through dashboards, waiting on reports, or dealing with complex BI tools. Just the insights you need, right when you need them.

With Adverity's AI-powered data conversations, marketers will finally talk to their data in plain English, get instant answers, make smarter decisions, collaborate more easily, and cut reporting time in half. What questions will you ask? To learn more, check out the show notes or visit www.adverity.com. That's A-D-V-E-R-I-T-Y dot com.

Amazing. That is actually exactly the next question I was going to ask was kind of around cost and when it would be appropriate to use servers or this kind of system. And so you just nailed it there. I don't know if anyone else has anything to add on that kind of, you know, this kind of builds on the point I was making earlier about this transition from, you know, there was a point, like when I think back to my PhD ending in 2012, there's a, are

there are a fair few number of things that I could at least test locally. Maybe I'd be like, okay, now I'm going to scale this up over a larger data set and so I'll use a super compute cluster or something. But in the world that we're in now, it's a completely different world where I can't just have a gigantic Lama model on my laptop and do anything with it. It's impossible. So it is an interesting, it's a completely different kind of world that we're in now. So we went through this transition from

being able to do a lot of things as a data scientist locally on a laptop than to getting used to doing a lot of things in the cloud. And now folks like you two literally are bringing new kinds of solutions, a new kind of paradigm where you can have either a Kleenex box or a desktop tower that can supercharge your ability to be using all the cutting edge AI models yourself with

without having to wait in line for anyone else. So I kind of, I guess I just summarized a lot of points there, but I don't know if you guys have anything to add around, you know, who this is a great solution for, you know, in terms of a company or an individual, you know, particularly maybe with respect to cost or efficiency. Okay.

I mean, I'll take a stab at it. So GB 10 clean Xbox, we're just going to call it the clean Xbox. I mean, that's, that could be scaled down to a student or it could be scaled up and, you know, put inside a Dell. I personally am going to get one and use it. You know, the GB 300 is, you

you know, really, you know, regardless of enterprise size, it's people that are doing very, very heavy workloads, right? I mean, it is a server ask now pricing is TBD. So I'll probably get in trouble. Let's just say that Nvidia, um, and their, you know, spark DGX spark product, I think is it three to 4,000 Sama? Is that right? Okay. I would assume that

that Dell's probably in that range for the Kleenex box as well. Pricing hasn't been released for GB 300, but considering you're going from a thousand AI tops all the way up to 20,000, it will be more expensive, but it will not be the cost of a server.

Cool. That's a great, that's a great way to keep me out of trouble. Way to keep me out of trouble. Yeah. You're really, really nice. Yeah. I was wondering if we get into specific pricing and I think that was, that was directional enough that it's helpful to me and our listeners. All right. So one last kind of big technical question for your, for both of you.

You guys are sitting right at the forefront of AI. You're going to conferences like GTC, which also I should clarify again just for listeners, when any time on this show that anybody said last week or today, they're talking about at the time of recording, which is about a month before this episode is published. So you're not on a time warp. GTC was, whatever, five weeks ago, not last week when you're listening to this.

But so with the ear on the ground that both of you have, where do you see, like trying to look into, I realize it's very hard to look into a crystal ball with how fast AI moves. But if you could try to make some predictions about what being a data scientist or an AI developer, or maybe even just what life will be like,

in the coming years, the coming decades? I would love to have your thoughts. Big question. I don't know who wants to go first. You can kind of just rush into that. I mean, I'll take a bit of a stab at it. And I mean, I have wild theories about what the world might be like with the advent of AI. And I don't think it's going to be robots taking over, right? I don't believe it's that. I believe is that if you think about it,

you know, any sort of technology or software, there's some sort of, you know, a seminal moment, like where think about, you know, first time you had a cell phone in the palm of your hand, right? Like, and yes, there is AI and different applications and stuff like that. I, I think what you're going to start seeing is kind of two, two big shifts is that one, you're going to see AI get into the hands of a lot more people. And what I mean by that is not, and I'm going to be

I am not a developer. I mean, Sama will attest like when I was hired for the job about a year ago, I mean, I was not great, but I was able to go out, learn, educate myself, pull down different SDKs from GitHub, other things to go out and train my own lores for an animation studio. I was able to do X, Y, Z. And I think you're going to see that become.

you know, a lot more accessible and easy and popular is kind of the years go on. And then I think the other thing that you're going to start seeing is AI kind of make its way into our daily life. Like for example,

complete crazy. But my mom, my mom's kind of old school. They're like 70, but she has a recipe book like in these little recipe cards thing. And I was like, Hey mom, um, you know that pumpkin pie recipe that you have? I'd really like to get it. And she's like, Oh, let me find the card. I mean, there's thousands of cards, dude. Like I, there's no, there's not alphabetized or whatever. And I was like, wow. Like what if my mom had a rag model of all

all of her recipes where all she had to do was really type that in and just say pumpkin pie. And it would just deliver and be able to tell you that. Could we go set that up? I could go set that up for her, but that would be the work. And I think what you're going to start seeing is some of these technologies and things like that

make its way into the mainstream where it's going to simplify our lives. Like, you know what I mean? And I think that that's what you're going to start seeing over time. I'm going to repeat what Jensen kind of painted that picture in his keynote as well, that we've gone from really the years of generative AI

to now being in the world of agentic AI, right? You've got an agent, an AI-powered agent for everything. So if you are, let's say, in a factory setting, you've got an AI agent, which is managing your incoming raw material and how much that is coming. Let's say you got less raw material. So this AI agent is telling that AI agent, which is managing the floor, guess what? We've got less raw material. So your end product is going to be less. And

And then this AI agent itself is telling the transportation one that we've got less, you don't need that many trucks today, right? So you've got, we definitely are entering that world with a lot of these reasoning AI models coming into being as well of AI agents where you can build these systems which have the ability to learn, perceive, but then also act.

And I think what the future is, is all about physical AI, right? You have a lot of these autonomous systems now, which are able to, again, learn, perceive, but then accordingly act. But this is in our physical world itself. So if you think of autonomous vehicles,

I'm in the Bay Area, I get to see a lot of these driverless cars all the time, but then every week I see them getting better and better at it because they're learning, they're perceiving different kinds of conditions of the road, if they're seeing somebody walking on the streets as well. So I definitely bet AI can kind of get integrated a lot with our physical world.

as a personal opinion, I hope there are a lot of guardrails and regulations just to make it safer for everybody to use it then. That was well done. And I realize, you know, you're using the CEO of

you know, NVIDIA, which is this hugely important AI company globally. And so, you know, I really appreciate you bringing those insights. It makes it so clear to see where we're going. I think you two worked perfectly, not only on that question,

But on this whole episode, I've loved this. It was so much fun. And I hope that I can get both of you on an episode again sometime soon. Yeah, to dig into another topic is so great. Yeah, nice. We nailed it. Before I let you guys go, I always ask my guests for a book recommendation. Have you got one for me? Maybe Sama go first because I think I saw you hold a book up earlier.

Yes, this is definitely one of my favorite books. It's called The Forest of Enchantments. It's by this Indian author called Chitra Banerjee. I highly recommend this. It's based on Ramayana, the mythological story around it. It's great, especially for any...

I want to say woman who's trying to come up, create their own career in any field. It gets you a lot of self-confidence in you as well. But I do love Gary Gennetti if he's listening. I am a really, really big fan. His books are amazing. They make me laugh. They make me cry at the same time. So I am a very big Gary Gennetti fan. I love all his books.

Those would be my recommendations. I'm sure famed author Gary Gennetti is a listener to the Super Data Science Podcast, no question. I'm sure he's really happy to hear it. Thanks for those great recommendations, Sama. Logan, what have you got for us? You know, I thought about this and

I'm not a huge reader. Like I'm more of a talker. If you haven't, you know, you can't see this, but I've got my get started manual for my Dell, uh, GB, you know, my XB webcam. Um, you know, it's riveting reading. No, I'm just kidding. I mean, it's, it's been a while. I'll, I'll admit, um, one book that I do, uh,

Uh, do not have in front of me like Sama, but I really did like, and as cliche as it, you know, it might be was the art of war by tons or Sun Tzu, like in the context of, you know, business there's been, you know, I've read it over

over the course of a couple of times, you know, and it was just, I like it because it definitely relates back to business and things about like the importance of strategy and thinking about what you're doing before you go out and tackle things and talking about, Hey, what is kind of a tactical way I want to do things or thinking about, Hey, when I'm under fire, how do I handle that? And

not the promoting war or anything like that, but just in the context of business, it's, it's very interesting. And I've definitely used a few nuggets from that book, um, over the course of my career for sure. Awesome. Great recommendation. I've read some excerpts from it and it does seem, it's not a very long book. I mean, I've heard that as well. It's about like the Dell pro max webcam. It says pretty close, not pretty thin, like a tissue. If we're keeping on the tissue box, it's about a tissue. Yeah.

How many arts of war can you fit into a single GB 10? Probably three paperback. Yeah. Nice.

Awesome. Thanks so much, both of you. If people want to hear more insights from you or just have a laugh, perhaps, because you've both been really fun and funny on the show today. How can people follow you? Sama, do you want to go first? I think the best way is LinkedIn. I'm probably the only Sama Bali you will find there. So that's the best way. Send me a message. I'm happy to connect. Nice. Yeah, we'll have a link to your LinkedIn in the show notes and maybe Logan's too.

Yeah. I mean, mine as well. It's LinkedIn's the best. Uh, that's where I'm most active. Um, it's just, you know, Logan Waller. I mean, pretty straightforward. I think I'm the only one actually know there is a couple of other ones, which is shocking. Uh, but I'm the one, uh, yeah, that kind of looks like this. Yeah. The one at Dell that says Dell pro max and their profile.

Yeah. And for our audio only listeners there, you know, find the LinkedIn profile that sounds like this. Yeah, exactly. Exactly. Perfect. Hey, John, one other thing I want to talk about before we close out and we really appreciate you having us on the show is that for everyone that's kind of listening is I'm putting together a program. Easiest way to describe it.

is we're looking for data scientists that have heavy industry knowledge experience to kind of help us do two things. One, to

help us kind of test products, right? Like the GB 10, the GB 300, as well as, you know, in our product design process, right? Where we at Dell, we work kind of directly with the ITDMs, but we're always looking for the end users that are actually using the workflows, whether it be in data science or M&E or engineering, it doesn't really matter. Being able to understand the workflows that impact them, what they do on a daily basis and what they need

in a Dell pro max. So I'm good starting kind of a developer advisory council. So we say, and if you're interested in it, want to hear more, um, I

I'm throwing out the offer. Happy to chat. I'm very fast at email. It's just Logan underscore Lawler Adele.com. And it will also be John, I think said will include it in the show notes, but reach out to me. Would love to hear it. Would love to meet you, hear about what you're doing. And if it's a right fit and it makes sense, then would love to have you a part of the council. Love that. That's such a nice call to action for my audience who are, it should be spot on the money for that kind of thing. Well, that's what I need them because it's not me. Yeah, it's perfect. Thanks Logan. Of course.

All right. So, uh, so awesome having both of you on the show. Uh, yeah, I really enjoyed it as I've already said. Uh, yeah. And hopefully catch you again soon. Thank you for having us. Thanks, man. Appreciate it, John. It was a great time.

What a fun, informative time with Sama Bali and Logan Lawler. In today's episode, they covered NVIDIA's Blackwell GPUs and how they offer unprecedented memory capacity of up to 96GB per GPU, with the ability to install 4 GPUs in one workstation providing nearly 400GB of GPU memory, sufficient to run LLMs with about 200 billion parameters.

They also talked about how NVIDIA AI Enterprise software creates a seamless ecosystem between workstations and servers featuring NIM, NVIDIA Inference Microservices, that allow one-line implementation of AI models without requiring manual GPU tuning. They filled this in on Dell Pro Max PCs and how they are designed specifically for heavy computational workloads that require GPU acceleration and a Unix-based operating system.

They talked about how the GB10 workstation is compact enough to fit in a palm yet powerful enough for significant AI workloads, while the GB300 delivers server-class performance with 20,000 AI tops and nearly 800 gigs of unified memory. And finally, we talked about how the democratization of AI is accelerating as technologies like these make powerful AI capabilities accessible to smaller organizations and individuals without requiring enterprise-level server infrastructure.

As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Sama and Logan's social media profiles, as well as my own at superdatascience.com slash 885.

And if you'd like to engage with me in person as opposed to just through social media or this podcast, I'd love to meet you in real life at the Open Data Science Conference ODSC East, which is running from May 13th to 15th in Boston. I will be hosting the keynote sessions and

along with my longtime friend and colleague, the extraordinary Ed Donner. I'll be delivering a four-hour hands-on training in Python to demonstrate how you can design, train, and deploy cutting-edge multi-agent AI systems for real-life applications. Thanks, of course, to everyone on the Super Data Science podcast team, and a warm welcome to Nathan Daly, who just joined us

as our head of partnerships. In addition, we've got Natalie Zheisky on partnerships as well, our podcast manager, Sonia Bryovich, our media editor, Mario Pombo, our researcher, Serge Massis, our writer, Dr. Zahra Karcheh, and our founder, Kirill Aromenko. Thanks to all of them for producing another

excellent episode for us today for enabling that super team to create this free podcast for you. We are deeply grateful to our sponsors. You can support this show by clicking on our sponsors links, which are in the show notes. And if you yourself would like to sponsor an episode, you can get the details on how at johnkrone.com/podcast. Otherwise, share the episode with folks who might like to listen to it as well or view it as well. Review the show on your favorite podcasting app or YouTube.

Subscribe if you're not a subscriber, edit the videos into shorts if you want to, but most importantly, just keep on tuning in. I'm so grateful to have you listening and I hope I can continue to make episodes you'll love for years and years to come. Till next time, keep on rocking it out there and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.