842: Flexible AI Deployments Are Critical, with Chris Bennett and Joseph Balsamo
17:18 Share

This is episode number 842 with Chris Bennett and Joseph Balsamo.

Welcome back to the Super Data Science Podcast. I am your host, Jon Krohn. Today's episode features the highlights of a session I recently hosted on flexibility in AI. And this one has not one but two guests. The first is Chris Bennett, who's Global CTO for Data and AI Solutions at Dell. And the other is Joseph Balsamo, who is Senior VP of Product Development at Eternal Technologies.

Today's episode should be interesting to anyone in it. Chris and Joe detail why it's critical to be flexible in our deployment of AI models and why the term generative AI is overhyped.

Ready? Let's jump right into our conversation, which was recorded at this ScaleUp AI conference in New York a few weeks ago. The interview you're about to hear came hot on the heels of my interview with superstar Andrew Ng. You can check out his interview in episode 841. So that's what I'm referring to at the beginning of this conversation. All right, let's cut right into it.

Welcome back, Second Stage. I hope you enjoyed that session with Andrew Ng. I'd say tough act to follow, but we've got great guests now as well for this session. We've got Chris Bennett from Dell and Joe Balsamo from Eternal, which if you're Googling that, it sounds like eternal, like eternal paradise, but it's with an I, kind of like IT at the beginning. Although I tried to dig into that and

They apparently made it no basis to that etymology that I've hallucinated. So that's an LLM hallucination from me. So let's first talk about how Dell, a market leader in GPU-enabled systems like AI systems, and Eternal, a comparatively small firm, came to work together. And what do you guys do together today? So thank you, first of all, and thank you, Scale.

Scale up AI. Scale up AI. Yeah. Wow. There was my LLM moment. So we have been working together for just under a year, I think, maybe more. Actually, now that I think about it, we've been working together for a very long time because they are behind a proposal creation platform that Dell has been running for quite some time. So bringing AI to our sales teams and helping them create individual personalized proposals. Right.

But the interesting thing to me about Eternal, and we have an entire ecosystem of partnerships with the smallest of the small ISVs all the way up to the largest of the large companies that you can imagine in IT and beyond.

One of the things that we are trying to do is we're trying to build a stable, if you will, of trusted providers where we have worked with them in the past. We've delivered exceptional results to our customers on top of our infrastructure or in Cloud or in a hybrid mode, however that manifests itself.

Nice. Anything you'd like to add, Joe? Yeah. I mean, I believe the business value of AI is still really in its early stages, and you kind of need that complementary scale and niche or specific use cases in order to

to align to where you are, whether you're crawling or running. And I think the combination of Dell and combination of an internal could really help you with those types of cases, whether they're small or large. Very nice. Chris, for those of our listeners who are not already familiar, can you explain the AI services that Dell provides? I

I think when people think of Dell, it's a no-brainer that this is a hardware company. You can instantly see the Dell logo on the top of a laptop or on a server, and maybe some people out there aren't immediately thinking of Dell and the AI services they provide. So fill us in on those. Yeah. The interesting thing about our company, and I've been here for a long time, I'm not going to say how long, but this might help. His beard. My beard, yeah. So the interesting thing about Dell

is we've been built from a college dorm room all the way up to the data center and consumer services provider that we are today over the last 40 years. The last 40 years of execution and strategy and acquisition have led us to this sort of pivotal moment in this industry. We've been through many epics in the IT space. And one of the things that we've done deliberately is we have built...

Again, back to the ecosystem, we've built a massive ecosystem of partnerships and we've also built up capabilities in our own consulting business that complement our partners to bring services to our customers no matter where they are in their continuum. So if they're just trying to figure this AI thing out, maybe they're a little bit terrified of it. Maybe they have some use cases identified, but they're not exactly sure how to identify the data that they're going to need to use.

One of the things that we can bring, if we don't have that discrete service in our own stable, we've got partners that do.

So I think from an end-to-end solutions company, I would pose that we have unique capabilities that no one else in the marketplace has. Whether you think about the desktop with AI-enabled devices to the edge computing devices that are AI-enabled and GPU-enabled to the data center to the world's largest large language model training farms, we're the infrastructure behind most of that.

So very cool, very cool. One of the advantages of working with Dell, I understand is the flexibility when you're using these kinds of AI services, flexibility between

doing your AI training or inference on-prem fully in a cloud or in some kind of hybrid model between your own premises and some third-party cloud. And Dell also offers, if you are working with a cloud partner, flexibility in your vendor choice. So tell us a bit more about that advantage of flexibility.

I think when it comes to flexibility, there's a couple of dimensions to that, right? The first dimension to that is what tool chain are you going to use, right? We're not a tool chain company. We're the infrastructure that powers it, right? And when you think about the platforms that need to be built on top of the infrastructure,

the platforms that need to be built need to be, let's call it portable, right? So if I'm building something in cloud and I'm piloting things and I'm testing it in cloud, that's fine, right? We've got software-defined platforms that run cloud and by hyperscaler cloud environments. We've got software, that same exact software-defined platform runs on premises and data centers and colos.

And I think the important thing in my mind is as you're considering building these AI powered applications, right, which encompasses a very large suite of infrastructure, both software and hardware defined. You have to think about it in a way that if I want to move this one day, if I want to pick it up from my hyperscaler cloud platform and move it to a co-location provider or go the other way.

Right. I've got to be able to do that. So if I use a bunch of proprietary tooling and it leaves me stuck in a particular consumption model, that's going to require refactoring, which is going to significantly increase the cost.

to move and it decreases your flexibility. So that's kind of what we're all about. We're all about flexibility. We're obviously have a strong bias toward on-prem, but we interoperate pretty much where our customers live. So, and that's our motto. We preach bring AI to your data. So if all of your data is in a hyperscaler cloud, let us help you with our partners and with some of our software defined solutions. Let us help you realize that vision in cloud.

and then run your applications and your workloads where it makes the most sense. You said that it's obvious that you'd have a bias towards on-prem, and I guess that's because you're a hardware company. If I'm thinking about an application that I'm building, why should I be thinking about potentially having an on-prem application or a hybrid situation as opposed to just using a third-party cloud?

I think a lot of it really just dictates to, again, where that customer is. If they're all in on hyperscaler cloud, it might be career limiting for me to say this, but maybe you should run your AI applications in hyperscaler clouds because that's where your data is. We believe that fully the large majority of data going forward and even historically is being produced at the edge.

And what that means to us is not in a traditional data center location. We think the data warehouses, et cetera, live in consolidated spaces, whether it be in cloud or on-prem. But really, where kind of the action is happening now is at the edge, right? We're seeing in almost every industry where, you know, new data is being created in volumes, very large volumes. So I think, you know, again, we're...

our customer is, is where we want to meet them. We don't want to dictate that you must and you shall and you will, right? It's where are you and how can we help you advance yourselves? It's kind of a new dimension for us, right? We're kind of used to backing a truck up to a loading dock and saying, hey, I've got 3000 servers in here. How many can I leave with you, right? So it might be a little bit of an uncomfortable conversation for us to have. And it's conversations that we're having in new areas, but it's an important conversation.

Nice. Very cool. Do you have examples, Chris, probably, of examples where a client was inflexible, unlike Eternal? And so we've talked about how important it is to be flexible. Do you have examples of where a client was inflexible and how that led to disastrous results? From a flexibility perspective, I think there are so many different ways to get this wrong and there are so many different ways to get it right.

Right. So if I tie myself to a model, what I might find is as my user count goes up or my interaction with my AI system goes up, that model might have a premium over something that might be hosted on-prem. Right. So if I'm using an API-based model to host my application, and it's very, very difficult if I've written my application specifically to that.

to that model. So I think, I won't call out any companies, but I think

From a flexibility perspective, we would encourage you to look at all of the options. Look at every option that's out there. Understand what works the best, but also understand what is the performance cost? What is the resiliency cost? And what is the actual dollar or euro or yen or wherever you happen to live cost? What is the cost of implementing it specifically to one ecosystem? We are all about open. We have a Silicon Diversity strategy.

strategy. We have a platform diversity strategy. We support open source. We support fully integrated stacks from our silicon providers, and we support integrated stacks from software partners. So I think when it comes to success stories, I prefer to think that we have far more success stories. I will give you one very quick example, and it'll be internal facing.

We published internal to Dell. Very early days, we built an application.

And the idea of the application was back when the usage and deployment patterns for augmented workloads, workflows were not well understood, right? So RAG is what we call it today. Terrible term, I hate it, but it's a term. And it's a year old in August. Retrieval augmented generation. Yes, imagine it's actually mature. So we deployed this platform in a pilot stage to about a thousand users inside of Dell.

It was a very easy to understand format. The UI was incredible. It worked really well until you asked it a question. What we learned was the dataset, the corpus of knowledge that we use to train the model, because we thought you had to train a model, and then the systems that we were pointing to that contain the enterprise data, they were all behind what I'll call a paywall. If I, as the user,

had access to that paywall, I could ask it anything I wanted to and it would answer me authoritatively. But if I didn't, it would literally hallucinate a response because the model would respond in the best way it knew how. And at the time, again, I'm not going to tell you what model this was. It's an open model, but it's a model that was in its infant stages. It didn't really understand how to say, I don't know the answer to your question. Right.

So it would fabricate the question. So, and again, and this all goes back to when you're doing this, you need to understand the consumer of your application. You need to understand the rights that the consumer has to access the data that this application has access to, right? Don't use system as the login for HANA when you're using a HANA agent to connect your reg solution to your AI application.

Nice. Yeah. Great story. I'm glad you were able to give us something. If I asked about a disaster and you didn't have any, it would have been a bit of a dull question. The Eternals never had a disaster. And so, yeah, one last one for you, Joe, before we get to it, we've got some great audience questions that have come up. But one last one for you is, do you think that we're using the term generative AI excessively? It is a term that we hear a lot in

especially in data science and the AI world, but in enterprises in general. At a conference like Scale Up AI today, you hear a lot about generative AI. Do you think it's overhyped?

I think that business leaders want to generate business outcomes, not AI. Is it overhyped? I mean, it's getting attention that needs to be there. So I like that aspect of it. I like that people are paying more attention to it. Not everything is generative. And you might not need generative for what you need. You need to look at your business cases and find a platform that is right for you. Find a company that's going to allow you to get as close to turnkey as possible.

with the technology and try to realize small results before you try to realize big results, right? And again, make sure you know what your data is and make sure you have the confidence in your data before you go at it. That's where I've seen most disasters. Let me just add one quick thing to that. You know, the adage of rising tide lifts all boats is very, very true here, right? When generative AI, generative AI, first of all, it's been around for a long time.

But the practical application of it sort of exploded two years ago-ish. And when that happened, all of the other AI platforms that people had deployed, machine learning, traditional AI, whatever, and however that manifested itself, they all almost instantly became more capable. They didn't all instantly become better.

but they all became more capable as a result of the innovation curve that was happening at such a massively fast speed. So I think that's...

You know, it's not really overhyped, but I think when people think of generative AI, they think of AI in general. Maybe I should trademark that. But, you know, it really is a question of, you know, we now have capabilities with machine learning to use data sets in a way that we hadn't even imagined before, right? Use cases across businesses are exploding because now it's no longer, it's actually left to imagination now, right? If I can imagine it, I can probably do it.

And it also provides you access to all kinds of unstructured data, which is most of the data that we have. So that was a big point at the end of our conversation with Andrew in the preceding session,

was that with generative AI tools, now you have access to all this unstructured data. And he gave a real-time demo of typing in some keywords and then pulling out specific frames in video fully automatically without any labels being explicitly trained. So yeah, lots of flexibility thanks to generative AI.

So yeah, Chris Bennett from Dell, AI and Data Solutions CTO over at Dell, and Joe Balsamo, VP of Product Development at Eternal. Thank you both for your time today and your excellent answers on flexibility in AI systems. Thanks so much.

All right, I hope you enjoyed today's conversation with Chris Bennett and Joseph Balsamo on why it's critical to be flexible in our AI model deployments. Be sure not to miss any of our exciting upcoming episodes. Subscribe to this podcast if you're not already a subscriber. But most importantly, I just hope you'll keep on listening. Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.