Snowflake's Baris Gultekin on Unlocking the Value of Data With Large Language Models - Ep. 231
32:10 Share

Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. Today I'm joined by Barash Gultecan, the head of AI at Snowflake. At Snowflake, he's driving the development of cutting-edge AI and ML products, including Snowflake Cortex AI and Arctic, their new foundational model. But Barash has also had a remarkable journey through AI himself,

having co-founded Google Now and led AI efforts for Google Assistant. So we've got a ton to talk about background, the president's snowflake, and of course, we're not going to let Barash off the hook without asking him to opine about the future of AI and, you know, the hot topic today, what's going to happen to all of us in the age of AI? So I can't wait to get into it. Barash, thank you so much for joining the NVIDIA AI podcast and welcome. Thanks,

Thanks a lot, Noah. I sometimes wonder if the pods we tape on a Friday, let alone today, a Friday afternoon, have a little bit looser of a feel than some of the other ones. So we'll have to do an exit survey at the end here. But I'm super excited to talk to you. We were talking offhand. I've been tracking Snowflake, especially over the past few years, as interest in AI and data has exploded. A good friend of mine has been with Snowflake now for a few years, so I've been tracking her journey as well. So

I'm thrilled to have you here, Baris, and to learn more about Snowflake and your own journey. Maybe we'll start with Snowflake and we can get into your background as we go. Can you start by telling the audience a little bit about what Snowflake is, how long the company's been around, what you do?

and obviously your role in the burgeoning AI explosion that we're all part of. - Of course. So Snowflake is the AI data cloud. We started the journey over a decade ago, focusing on how do we make data a lot more easily accessible? How do we make data processing a lot more easily accessible to companies?

As the data volumes were growing, there was a big innovation of separating the storage of data from compute that allowed a massive unlock in the data space. And since then, we've been evolving from a provider of a data warehouse to a data cloud where that data can be easily shared. And now we're in the third phase of that journey where you can now unlock a lot more value of that data with AI for a lot more users.

Right. Maybe just to set baselines, can you quickly just talk about terms like data warehouse, data lake, the AI data cloud, as you described Snowflake, and what those terms mean and maybe how they've evolved a little bit? Of course.

Customers have a lot of structured data. They need to have a way to bring all of that data and then manage it, govern it, be able to run analysis on it. So Data Warehouse allows our customers to very efficiently run massive scale analysis across large volumes of data. Right.

And that's for structured data, for kind of tables. With Data Lake, you kind of expand that to bringing in kind of more unstructured data, structured data into the mix.

Right. And so there's issues at hand of the way the data is stored itself, obviously, the physical media it's stored on. But then, as with all things technology and certainly all things AI, machine learning, deep learning, the software plays a role. The interconnectivity plays a role. Obviously, if it's a data cloud, there's all of that.

Everything that happens between your cloud and my workstation, wherever in the world I am. So there's a lot going on. So maybe we can dive in a little bit to some of the breakthroughs

that Snowflake has ushered in, is currently working on, and kind of how data plays a role, what the data cloud's role is in the modern ALML stack, if you will. So let's talk a little bit, or if you would, talk a little bit about how Snowflake works with enterprises in helping them unlock their data, as you put it.

Yeah. So first of all, we're super excited about what AI is capable of, what AI is bringing. And of course, when we think about AI, there is no AI strategy without a data strategy. Of course.

get the most out of this data by bringing AI compute right next to where the data is. So believe deeply in being able to bring compute to data versus bringing massive amounts of data out into where the compute is. So the way we work with our customers is that we built a AI platform and with this AI platform, our customers can run kind of natural language analysis, can build chatbots, can talk to their data in natural language.

And we make this platform available and our customers are using it to build all sorts of AI applications. And so the customer stores their data with you, but then also the training, the inference, all the compute operations are done on your side as well. That's right, exactly. So basically our AI platform runs fully inside Snowflake. So for Snowflake, it's really important to govern the data as well as AI that's running. Right.

We run everything inside Snowflake. So maybe you can get into a little bit of some of the product offerings. Do we want to start with Snowflake Cortex? Yeah. So Snowflake Cortex is our managed service. It is our offering where we're running a series of large language models inside Snowflake.

Our customers have super easy access to these large language models. But the way we think about AI is we want to make AI easy, efficient, and trusted. With Cortex, it is incredibly easy because AI is running right next to where the data is. So our customers don't have to build data pipelines to manage data in multiple places and govern it.

We're running it in a secure environment and we have a series of very efficient models from our own to a series of partner models. We make that available. And then Cortex also makes it very easy for our customers to build, talk to my data experiences, if you will, build chatbots, both for documents as well as for structured data.

What are some of the use cases that, you know, are kind of most popular, most seen? I've heard people talk about, we're recording in late June of 2024, and I've heard people reference 2023 as being the year that generative AI took over all the headlines and everybody was

talking a lot, but not really sure what to do. And 2024 maybe being the year that businesses start to actually develop applications or use applications others have developed, but really start to do things

using, leveraging AI on their own data to whatever it is, improve processes, you know, try new ways of working, all that kind of stuff. What are you seeing on your end as the things that customers and enterprise customers, other Snowflake customers, you know, are interested in doing? And then maybe on the flip side, some of the things that, I don't know, they're concerned about or don't quite understand or still trying to kind of wrap their collective heads around.

Yeah. So I agree with you. In 2023 was, I'd say, the year of proof of concepts. They got their hands on AI and then started building demos. And this year, we're starting to see these turn into real production use cases. I can give a couple of examples. We're working with global pharmaceutical company Bayer in building demos.

an experience where bears internal teams and sales organizations, marketing organizations can ask questions of their structured data. Okay. So believes that, you know, dashboards can only take them so far. Dashboards tend to be very rigid and, and,

First thing that happens when you see a dashboard is you have three questions, four questions where you want to drill in and figure out why something isn't the way you want it to be or you. So now we give that power to not only the analysts, but to business users. So business users...

ask questions of their data, drill in in natural language. And that's super powerful. We've been working with a series of companies and like Bayer, they're finding it very valuable to give democratized access to that kind of data. Another interesting one is we're working with kind of Siemens. They have a large research organization

They've just recently built a research chatbot that has 700,000 pages of research that's now unlocked and available for this research organization. So, you know, instead of kind of figuring out how and where to get that data to continue your research now, the team feels a lot more productive. How many tokens is 700,000 pages? Yeah.

It's a lot of tokens. It's a lot of tokens. So when you're working, and I'm sure the answer is different depending on the customer, but when you're working with a customer to do something like take 700,000 pages of documentation and turn it into something that the average employee, the average user can just speak to using natural language.

What's the process like in terms of what you're doing on the technical side? Are you fine-tuning the model? Are you building custom rag pipelines?

And again, I'm sure it's different with different use cases, but what are some of the things that Snowflake does with a customer that they couldn't get just by sort of brute force uploading these documents to a publicly available model? - So when we actually surveyed our customers,

as they think about going from these demos, proof of concepts to production, usually three big things emerge. One is they are concerned about quality hallucinations. The second one is they're concerned about security of their data, governance of that system. And finally, the cost. So those are the top three things that always emerge. And then we try to address these three concerns head on. Yeah.

Cortex Search is a new product offering that we are just recently releasing. And we've tuned Cortex Search to be the highest quality in terms of a RAG solution. So we implement a custom RAG solution. We have our own embedding model. And we've built a hybrid search engine that can provide high quality. And we will tune the system so that it knows when not to answer questions, reducing hallucinations.

Got it. Hybrid search meaning it combines RAG functionality with Internet search or? Exactly. Hybrid search is basically combining vector search with the traditional keyword-based text search. Right.

On the hallucination point, any interesting learnings or insights when it comes to, I would assume it's more sophisticated than manually just line by line telling the model, you know, don't say this, don't say that. But how do you sort of coax a model into hallucinating less? Oh, yeah. So first of all, there is definitely model tuning. That's important in this. But also, again,

We just touched on the hybrid search element. The nice thing about hybrid search is it can give you meaningful information about whether

the set of documents is relevant to the question. And, you know, usually LLMs tend to hallucinate when they are not grounded on data. So the system can know that the match to that question is low. And rather than trying to answer the question, it should just reject it. Got it. Speaking of models, there are a bunch of Snowflake products. We don't have time to get into all of them, obviously. And I'm going to leave room to bring up things that I didn't ask about. But I do want to ask you about ARTIC.

So Arctic is a, is it an LLM that Snowflake built or how would you describe it? Yes. Arctic is our own language model. It's actually a family of language models. We have the Arctic LLM as well as an embedding model and a document model.

So the LLM is an open source, large language model that is quite unique in its architecture. By combining both a mixture of experts model with a dense architecture, we were able to have a very efficient and high quality model. So we focused on what we're calling enterprise intelligence, being able to

to follow instructions, being able to do well in coding and SQL. And then we were able to achieve the highest benchmarks in these categories amongst the open source models while being incredibly efficient. We've trained Tastic at about, you know, one eighth the cost of kind of similar models. And that means, you know, when we train custom models, for instance, for our customers, we can be very, very cost effective while delivering very high quality.

understood if these are trade secrets you don't want to divulge. But how did you figure out how to train the model so much more efficiently?

- So we actually really pride ourselves in our openness. We released cookbooks of our, not only the model weights, but also the research insights as well as our kind of data recipes. - Oh, fantastic. Okay. - All of those are available and we've shared some of these insights. It boiled down to having some of the best researchers that have pioneered MOE models, mixture of experts models, you know, back in the day.

along with some of the VLLM, the original team members and all working together to architecture. - Very cool. Again, for folks who might not be familiar and I'm familiar, but I don't fully understand how it works, so I'm gonna ask.

What is a mixture of experts approach? What does that mean? What does it entail? Why is it different, better than other approaches? Yeah. There are two major architectures that we're seeing. One is what's called a dense model. In the dense model, all of the parameters are active and they're being used when you're doing inference. So also during training, all of these parameters are active. Mixture of experts model,

has larger set of parameters, but only a subset of them gets used. So you have different number of experts essentially that are getting activated to answer one question. That tends to be very efficient. Right, got it. So you can hone in on the accuracy that you're looking for, but then also it's more efficient because you're turning things on and off as you need them instead of just leaving all the lights on. It's efficient to train as well as efficient to run inference. So it tends to meet you.

because it has lower number of active parameters. Got it. Any other specific products, innovations that Snowflake has put out during your time that you're particularly excited about? I am excited about our Cortex Analyst product that we just recently announced. As I was alluding to, there's a lot of data gem that is locked in very large amounts. And bringing...

allowing more people to have easy access to that data is really important. So far, data teams have to run kind of SQL analysis to get insights from these data sets.

Large language models have been exciting to see, hey, can we turn language into SQL? And turns out that is a really, really difficult task because the world of data tends to be massive. You have tens of thousands of tables, hundreds of thousands of columns, really complex abbreviations of column names and so forth. So we work really hard to have the world's best text-to-SQL experience.

And then we've achieved it. So we have the state-of-the-art when it comes to text-to-SQL. And that now becomes available to business users who can now ask natural language questions. And then we turn that into SQL. We run that SQL and generate an answer. So something like, how is my revenue growing in this region for this product now becomes an easy question to ask for a business user. Right. Fantastic.

My guest today is Baris Gultekin. Baris is the head of AI at Snowflake, the AI data cloud. And we've been talking about, well, the role of data. We always talk about the role of data on this show because data is what fuels AI. But in particular, Snowflake's approach is to everything from fine-tuning customer data, unlocking structured and unstructured data so that

the customers, the developers, the folks in Snowflake's ends can turn the data into insights and applications. Then also some of the innovative approaches that Snowflake has taken to fine-tuning models, building models, converting text to SQLs we were just talking about. Let's switch gears for a second, Baris, if we can, and talk about your background in AI at Google even before Google.

Have you always been a data nerd, to put it that way? You've always been interested in data, computer science. Where did your journey start? Yeah, I have been actually, before it was cool to say AI. I started this journey at Google a long time ago. And at some point around 2010, 2011, we started building Google Now.

It was the traditional 20% project at Google, where we basically thought, our phones should do better than what they do today. They should be able to give us the information we need when we need it. So it was this proactive assistant. And we built that product, even though at the time, the technology wasn't quite there. Right.

where it is now we were able to give helpful information like uh hey there's traffic on your commute and you should just uh you know take this alternate route or uh your flight is delayed and all of that information felt magical because of bringing context with uh kind of prediction right right even though it was a set of heuristics it felt like oh there's some something intelligent here and that was the beginning and i love yeah yeah and um and then after that i uh you

I worked on Google Assistant, and Google Assistant is, again, exciting because it understands language. It can respond in natural language. Early on, it is just a series of use cases that are just coded one by one, right? And now we're at a point where finally computers can understand language, and you don't have to kind of code each use case one at a time. Right, right, right. Exactly.

When you're working on all the things that you work on at the scale that you do now, well, you did at Google and now you do at Snowflake. How much do you trust the answers given by

by a generative AI model and what's sort of your own, I don't know if it's a workflow so much as just kind of a mental, like, do you go back and verify results that you're not sure about? Have you kind of gotten to, you know, do you have a feel for when something's grounded versus hallucinated? And this is a little bit more of a, um,

I don't know, metaphysical question perhaps than the other stuff. But I'm just wondering, someone with as much experience and day-to-day working with this stuff as much as you do, what your kind of feel is for where the systems are right now? I try to see where we are, this generative ability, the creativity, if you will, a feature to a certain degree, right? So the types of use cases that are great are when you ask...

the language models to generate something, to generate content. And if my question is a factual question, then I know to be careful. But if it's more of a help me brainstorm, let's think about this. How do you say this differently? Those are the types of

things where you're leaning into the creativity, into the hallucination as a feature, if you will. Right, right, right, right. And so then how does that translate to enterprise customers you're working with? I would imagine there's a, you know, they sort of run the gamut from folks who are really excited to work with this, who maybe you have some customers who are a little more reticent, but feel like they should be

How do they relate to this whole notion of hallucinations being part of the deal? I think it's incredibly important to know that right now, where the technology is, we need to build systems and these systems need to have grounding in them. So we work hard to provide technology to help our customers to make their systems, their products, their chatbots a lot more grounded.

with the data that they provide. If an LLM is provided with a grounding, if an LLM is provided with the data, it does not hallucinate, right? Only when there is lack of information, it then kind of makes it up sometimes. So we work hard on those solutions. We work with our customers. We also want to make sure our customers are able to evaluate these models. We've acquired a company called Truera just recently. Truera is a company that focuses on ML, LLM observability.

Being able to evaluate whether a chatbot that's built is grounded, whether the quality is right, whether the cost is how they want it. So those are the technologies, tools that we'd like to offer to our customers and we work closely with them.

Right. And so along those lines, so that was an acquisition, obviously, but Snowflake's partnering, you mentioned that kind of your company's openness and transparency. And there seems to be a spirit of that. And perhaps because everyone's laser focused now on this, you know, frontier technology that inherently we're all sort of figuring it out, whether as a user or developer to some degree.

What's the nature of some of the other partnerships or sort of what's Snowflake's role in working and partnering with some of the other tech giants and companies out there working at the leading edge of AI and ML? Yeah, we have very close partnerships with NVIDIA, with Meta.

as well as Mistral and Rekha, you know, the large language model providers. We've invested in some of them as well. We basically see our platform as a way where we provide choice, but we work very closely with our partners in kind of helping us build specific solutions.

when it comes to kind of making sure that our rag solutions are grounded, making sure that we have the world's best class Texas SQL experience that requires partnerships. We work very closely with our partners. So in terms of openness, openness matters for many of our customers. Understanding what kind of data was used to train a model is important.

We also partner with some of our providers to have high-quality proprietary models as well. Snowflake, as I understand it, is a global company, has around 40 offices worldwide? That's correct, yeah. Right. And how many data centers do you run? So we're running on the three clouds, AWS, ZP. Okay, got it. Right, right. So kind of looking...

at the present, but looking forward a little bit. I'm going to put you on the spot now, like I said I would. We touched on some of these things during the conversation, but are there major trends you're seeing in business adoption and, you know, sort of real-world use cases that your customers, Snowflakes customers, are, you know, adopting now? Or really, are there trends and areas they're really interested in, you know, exploring with the power of AI?

And then kind of piggybacking on that, where do you see the industry headed, sort of in broad strokes, if you like, over the next, say, three to five years? So we're seeing a lot of exciting use cases. I've mentioned a couple of them, but our partners are, again, building production use cases. Some of them are our bread and butter, running large-scale analysis across their data inside Snowflake. So we're seeing a lot of super simple, like,

just using English language to be able to create categorization, extract information, and kind of make sense of a lot of data. For instance, one of our customers, Sigma, that's a BI provider, they are

They are running analysis on sales logs from sales transcripts, sales calls, and figuring out, understanding why do we win? Why do we lose deals? Right. So being able to run this now across a large data set of all sales calls in a period of time is as simple as writing English language. So that's fantastic.

Fascinating to me. - No, it's amazing, right? Yeah. - And then as I mentioned, of course, the bread and butter, high quality chatbots, as well as being able to talk to your structured data for BI type use cases. Those are the use cases that we're seeing. What I'm seeing, of course, the world of AI evolves incredibly fast. Week over week, we get a new announcement, something new, exciting. - I know, three to five years, I should have said months or weeks even. That's on me.

Also, it feels like the year. Of course, the next big phase that is coming that's already kind of getting traction is the world of agents. So not only are we seeing the ability to answer questions by looking at documentation, but being able to take action.

And these agents are, these agentic systems are coming, this ability to reason, this ability to self-heal, the ability to take action for agents to talk to each other, collaborate. And that is the next evolution of the technology.

Right. Are there any agentic frameworks on Snowflake that customers can access? So very soon. Right now, the agentic systems that we've built are kind of behind the scenes. The Text2 SQL BI experience uses a series of tools to deliver the product. Gotcha. And we'll make that available to our customers. Right. Very cool. Looking back on your time at Snowflake,

Further back, I won't prescribe timeframes here. I learned my lesson. Is there a particular story, moment, something that springs to mind as, you know, an important, perhaps unexpected learning that's kind of really impacted, you know, how you view your work and the landscape today? Maybe a problem that, you know, the solution turned out to be something unexpected or something you thought was going to be hard and turned out to be simple.

So I'll give two examples. One is very early on, as we were building Cortex, we talked to a customer. This customer is a longtime Snowflake customer. They've built a pipeline to take their data out and then get it processed by an LLM running elsewhere and then bring it back. And of course that pipeline took two months or so to build and it was quite expensive to maintain and they were concerned about it. And our early prototype was,

was able to replace the full thing with literally a single line of code. So we are onto something.

When you bring compute, when you bring AI right next to where the data is, it makes everything a lot simpler. And when it's a lot simpler, it just unlocks a lot of usage. So I'm super excited about kind of just ease of use, simplicity. The other example is just realizing how kind of demos are easy to build, but production systems are hard.

You know, you have, especially when it comes to working with structured data, generating SQL is difficult. So we work really hard on how do we build a system that together creates a very, very high quality response. When you're essentially asking revenue questions, it's not enough to be, you know, 80% accurate, right? So that's another big, important area that we focus on. All right, getting into the wrap up here.

I always ask this question anyway, but I have a child who seems to be getting older every year and now he's in high school, interested in computers, computer science, physical science. What advice would you give to either a young person kind of looking out maybe on the edge of graduating, you know, a little older, maybe graduating college?

Or maybe somebody who's older and is just interested in AI and sort of keeps hearing the things that we've been talking about, which is both that things are changing so fast, but also, you know, there are things that we can do in the present moment and still plenty of problems to be solved.

So where should they go? Is studying computer science still a viable path? Is it better to just dive right into the work world and start working on, you know, as you said, prototyping is one thing, building a production scale system is quite something else. What's the advice that you give to, you know, young people or maybe older people looking to dive further into AI?

So I think everyone has their own unique path and everyone is drawn to something. And it's important to be able to connect to what you're drawn to. And it'll be different for different people. But I just focus on that, just listening to that inner voice, which is hard to listen to sometimes, especially given there's so much noise out there. But I will say, even though AI sounds intimidating, oh, there is this kind of archetypal

kind of artificial intelligence that sounds very complex. And it is complex when you start doing, going down the rabbit hole and start doing your research. However, the use of the AI is going to unlock, it's incredibly easy. All of these systems are now an API away and they're incredibly powerful. So I think, you know, creativity is going to determine

all sorts of super interesting technologies to be built next. So I would say don't be intimidated with technology, just dive right in. And it's incredibly easy to use and really looking forward to what's to come in the next two years or so. Love it. Love the optimism. More audio only, which is a shame because your face lit up.

Smile got big when you were talking about that. Baris, you alluded earlier to cookbooks and other resources that Snowflake makes available. Maybe we can divvy this up into two parts. Potential customers who want to learn more about what Snowflake does, what the offerings are, how to maybe engage with you. And then folks...

practitioners working in AI wanting to learn more about, you know, what Snowflake's been doing, research, some of the techniques we talked about. Where can people go online to learn more? So our website, snowflake.com. If you are trying to figure out how do I use AI just in seconds and bring my data, analyze my data, we have a solution for you. Nice. So snowflake.com is the place. Perfect.

Barash, thank you. This was great. As with many of these conversations these days, I feel like this was kind of the warm-up and we'll have to get back in touch down the line to really dig into where things are headed. But the Snowflake story, you know, is a great one and it seems like it's just getting started. So congratulations on the work so far. All the best to you going forward. And, you know, look out for my unnamed friend I mentioned earlier if you see them around campus. That sounds great. Thanks for having me, Noah. ♪

*music*