#113: Why you need Knowledge Graphs for your AI chatbot | ft. Aniket Mitra
45:13 Share

Hi, welcome back to another episode of Real World Serverless. And today we're joined by Aniket Dmitra, who is the founder of SaveA, a software agency that focuses around AI and ML and data engineering. And he's been doing some really interesting work around the knowledge graphs with LLMs. Hey, Aniket, welcome to the show. Thanks, Jan. I'm glad to be here.

Yeah, so I heard about your work through a mutual friend of ours, Daniel Lu, and I've actually come across a few people recently talking about the use of knowledge graphs with LLMs. And that's something that you have spent a lot of time working with. I wanted to get you on the show to talk about what it is, what knowledge graphs and how it kind of helps with the AI and LLM stuff that we're working with nowadays. Before

Before we get into it, do you want to take a moment to talk about the work you're doing at the SaveA and what's your history and background?

Yes, absolutely. So currently, as you said, I am the founder of Saveway. And what we do at Saveway is basically we execute on projects which are heavy on computational sciences in general. That's the idea. But, you know, more specifically in the sub-branches of AI, ML and data engineering and data science, for instance.

And basically, you know, my background has typically been for a long time in the field of data science at YAML. And more specifically on the knowledge graph part, because in the last couple of years, I was mainly involved in building enterprise knowledge graphs for enterprise customers, ranging from, you know, customers in the mapping space,

which is basically the location-based services. And currently I'm executing projects that require, again, building knowledge graphs for large retail customers, for example.

Right, okay. Yeah, so talking about the knowledge graphs, I know they've been around for quite some time. And the last time I think they were in the mainstream consciousness was when Web 2.0 and the whole semantic web was kind of around. And I've spoken with a guy, I forgot his name now, that was really big on this idea of a mirrored world where you essentially build up a metaverse, well,

not the metaverse, but like a mapping of the real world properties and whatnot, buildings, locations, so that you can start to query that information about things happening in the real world and connect them, how basically measure impact of one activity have on something else. And we've seen something like that with some of the software that

I guess companies like Palantir and it's building for defense and whatnot, but maybe not so much in the general sense that the people like you and me are using day to day perhaps. And so yeah, Knowledge Graphs, tell us about it. What's the history behind it and what's happening today?

Yeah, absolutely. And, you know, as you also mentioned, knowledge graphs have been around for a long time, or at least the idea of knowledge graphs have been around for a long time. Because at some point of time, the genesis of it or the core idea about it is being able to organize the data, for example. And one might say that, okay, when we, you know, organize the data in its

traditional form the way as we know it that okay every record is a collection of attributes for example or values associated with attributes and when you represent that in the form of rows of a table that's that's the you know de facto traditional form of organization but what that does not allow you generally is to be able to add

add semantic meaning, for example. So which typically means is that if I am recording information about students in a classroom and I have information with respect to their grades, you know, the classes or the courses they have taken, those are attributes. But

What does grade mean in the real world? So what you then want to do essentially is you want to associate meaning to the attribute that is great. So grade is a value or grade is an attribute associated with the course, for example.

Now, course is an intangible item that is also a subclass of a thing, for example. So also the grade is associated with the course, but the course also has a name and the name is also associated with something. It could be the name associated with some other class that is again a subclass of a thing. So essentially when you start drawing this out or if you take a moment to start thinking about it, you will see that the nature of

of how this evolves is in the form of a graph as such, right? So you will see that, you know, multiple paths arise from one concept and they again join at a higher level at some point. So when you go up

up the attribute space and you start assigning meaning to that attribute space, the natural form of that or the natural structure of that is the form of a graph. And that is where the concept of the semantics comes into play. The concept of what we call in the knowledge graph space is the concept of an ontology comes into place. So when you look at that kind of an organization, you are also able to now

attach meaning not only to what is recorded, but also something that is not explicitly recorded, for example. And that is where you are now gradually transforming information to knowledge. And that's where these two things come into play that you have now been able to organize information in the form of a knowledge and the structure of that knowledge is in the form of a graph, essentially. That's why you get to the concept of the knowledge graph.

Okay. Yeah, oncology, that's a term I haven't heard for a long time. Can you remind me what does that mean again?

Yes. So whenever we talk about semantics, typically, and the whole idea of associating meaning, for instance, right? So this is defined as a vocabulary. So the vocabulary is like, okay, when you say grade, what does grade mean? And the grade, you know, in the hierarchical organization of grade in high

in its representation in the real world, for example. And that is what an ontology defines. And ontologies can also be very general purpose like schema.org, for example, but they can be also very domain specific. Like you have ontologies in the medical domain, you have ontologies in the retail domain, for example. But the core idea is to be able to assess

assign or associate meaning in that metadata space, for example. And that is what the notion of an ontology is. It allows you to give meaning to the notion of a grade, for example, the notion of name, for example, the notion of person, for example. And typically, if you look at ontologies, ontologies, because it attempts to assign meaning, it takes the form of a graph.

in Excel. Right, okay. And when talking about graphs, obviously I think you were kind of hinting at this earlier that the traditional way of representing data and relationships in relational tables doesn't quite cut it. And that's where I guess graph databases comes in and

The graph databases, I'm a big fan of them. I used graph databases like Neo4j in the past and I'm a big fan of what you can do, the kind of queries you can run. It's kind of mind-boggling. You can do pattern matching against your entire graph and find things that have specific relationships. And also, like you said, I think you're able to add attributes not just on the entities themselves, but also on the relationship between the entities. And that's something that...

Yeah.

becomes really powerful. I think you mentioned it. You can tag intangibles, things like relationships and how they're connected to two different entities. You can have attributes on that relationship itself, and you can actually query against those properties on the relationship as well as the entities. Tell us a bit about graph databases, what's been your experience of working with them in terms of using knowledge graphs and

how do they differ from relational databases yeah absolutely and uh you know graph databases again uh when you look at graph databases there is a subcategorization because the traditional form of creating knowledge graphs typically would always be under the category of relation resource description format essentially so which is rdf for example right and any ontology if you take

pick up any ontology for that matter as it exists. Most of these ontologies are basically, they fall under that category of RDFs. And RDFs are a simple organization of concepts by

in the form of triples right so you have a subject you have an object and you have a relationship between the two which is in the form of a predicate for example and there have been a number of graph databases that uh you know support this rdf like format for example but then the problem as you naturally see is that when you start organizing these things uh you know in the form of triples all of a sudden the number of nodes and relationships that you need to start modeling that

just blows out of that becomes unmanageable, for example. So for very simple data sets, for example, which would be in order of magnitude of a few thousands, the number of nodes and relationships that you would get would be in the order of magnitude of millions, for example. And that's that's the, you know, the blow up in proportion that you would see.

So what, you know, as a succession, what graph databases like Neo4j did was they said, okay, do you need to represent everything as a separate node, for example, right? So say, for example, the notion of a student in a class wherein, you know, the student has a name, the student has a grade and things like that. Do each of them have to be represented as separate nodes or that information can be collapsed into one single node?

or a collection of notes, for example. And that is where the notion of label property graphs come in, wherein they allow you to attach, you know,

properties or labels to the node itself wherein you know every the the mapping of a label or a property uh to a node is not always one to one right so one node can map onto multiple properties for example and also as you also highlighted that okay you could also start having having relation properties to relationships for instance right so that adds another layer of benefit because then retrieval becomes easier for example so

Neo4j actually has allowed us to, in the space of graph database, allowed us to optimize this model further for more efficient representation as well as retrieval for that matter. And of course, the way it is different is because now you are able to

model semantics and when you are able to model semantics in contrast to a relational database, you are also able to capture things that are not explicitly mentioned, for instance. But what is also important to know is that when you start building knowledge graphs at a large scale, at an enterprise-wide level,

it is generally a bad idea to think of a knowledge graph as being a one stop solution. It is good at doing some things, but it is not supposed to be good at doing everything, for example. So, for example, if you want to do fast text searches, you would still go to something like an elastic search, for instance, which is very specialized to do that kind of work. So the whole idea is that when you look at

an overall system architecture for this you look at knowledge graph as being fitting into that overall architecture as one of the components that supports a collection of use cases in conjunction with other things rather than you know being a de facto replacement for

everything because then that's a design misstep in some sense. Because you will see that if you want to do, let's say very advanced text search, maybe a graph database is not something that you would want to go for because of performance issues.

Right, yes. And I think you were kind of bordering the use case for graph databases with LLM. But before we get into that, I remember when the vector databases became like a thing with RAC and other things like that,

One of the problems that it solves was the nearest neighbor problem. It's a traversal problem that was typically quite difficult to implement in other types of databases. But I remember very clearly that when I first discovered graph databases, that was one of the classical examples of, okay,

These kind of problems is just trivial to implement with graph databases because you start with your center point to find the nearest neighbor, it just traverse the graph.

It's literally the next step and then the next step and then until you find a node that matches your criteria. And in the Neo4j in Cypher, they literally have an example that shows you, okay, one node with some, I guess, I forgot the syntax, basically says any number of nodes in between these two nodes, so long these two nodes have the right properties they're looking for, like a student and a class, then you can find a

part of your graph that matches that syntax, and that basically kind of solves your nearest neighbor traversal problem. And so when the databases became a thing, and oh yeah, the thing that people keep talking about is it's really good for finding the nearest neighbor, I was thinking, yeah, but so did graph databases, so how come nobody was using them? And so...

You were talking about how it makes traversal easier when it comes to LLMs, but also the fact that you can capture more context when it comes to the raw data. So can you talk us through how you're using graph databases and knowledge graphs with LLMs and where does it fit into that whole ecosystem?

Yeah, absolutely. And, you know, just to also continue on that chain of thought that you mentioned is that, you know, if you...

because knowledge graphs have existed even before we had the notion of embeddings typically. And as you mentioned, Jan, was that nearest neighbors was something that would still be done, but then it would be a function of how things were organized in the database and not typically the semantic similarity in a high dimensional vector space.

But what happens is if we take this as two extremes, wherein we say we'll only do semantic similarity based on an embedding-based representation,

there will still be certain things that will be missing because those things are easy to do as part of a natural organization of things in the form of a graph, for example. If you take the other extreme, for example, when you say, "I will never use an embedding based system, I will only use a graph based system," you will be able to find similarity, which is a function of the connectedness in the graph. But what about the similarity based on the attributes on the node of a graph, for example?

Right. Because that is not something that you can model just based on connectedness, for example. So the very good thing that has happened is because of the advent of embeddings typically or similarity search based on vectors, is that somewhere we are increasingly seeing these two fields come together. Right. And one being able to benefit from the other.

And when you are trying to, when you actually start working in that space, when you use embeddings to be able to find similarity on attribute values, for example, and you also use a knowledge graph to augment that similarity on nearest neighbor search based on the topology of the graph, based on the connectedness of the graph, you start seeing really good results, for example. So it's basically at the combination of the two. Yeah.

- Yeah. - Okay, so let me just digest this and see if I can explain this in my own words. And you can tell me if I misunderstood what you just said. So what you said is that with vector databases, with the way LLM sort of sees data, it sees, I guess, more kind of correlation. If the two words appear normally adjacent to each other, then they are considered close to each other in terms of neighbors in the vector space.

So when you're doing embeddings and doing reg and, you know, you're looking at the data and creating a vector space of the data, the kind of relationship between the different data points is based on how they appear in the source data. But those doesn't necessarily represent the semantic distance between the two nodes, which may not appear all together, you know, in a JSON format.

positions in the training data, but semantically they are similar. And so you can capture that semantic similarity with graph databases

but you can't capture the, I guess, the correlation of position in the training data with graph databases. And that's where vector databases can do. And so you are able to, in that case, combine both so that you have both the semantic similarity, but also the, I guess, training set similarity in terms of the positions of the tokens as they appear in the data. Is that correct? Understanding? Absolutely. And see how you combine it.

could also be that you said that fair enough. I can, because the concept of graph neural networks have existed for quite some time, right? So when you generate embeddings based on how the structure of the graph looks like for instance, right? So you can think of combining both of them again in the embedding space, right? You take the embedding of the

based on the values as such, you also embedding, take the embedding based on the structure as such and you combine them. But somewhere that combination is necessary to yield or extract maximum benefit for the problem you are trying to solve. Because otherwise, each side, if you look at either extreme can only get you so far.

Right, right. Okay, interesting. Okay, so in that case, can you walk us through an example problem and how you may apply this with LLM as part of, I guess, as part of augmented retrieval? Which step does the graph database come in? Which step does the vector database come in as you process a user query or as you, I guess, process the data?

Yeah, absolutely. So, you know, things like I'll take the example of, you know, the location domain or the domain in which we were building the knowledge graph as part of my previous work. So imagine, you know, you are collecting information for restaurants in a city, for instance, right? Now you have information about those restaurants. For example, what is the category of the restaurant in the sense, you know, what kind of cuisine it serves, for example.

What are the reviews of that restaurant? What are the, you know, what is the description of the restaurant in the in textual form, for instance, right? Also, how is that restaurant located in the vicinity of other POIs? If there is a parking lot nearby, for example, how far is it from maybe the main highway, right? So when you start looking at all this information, there are certain things, you know, like the description of an

restaurant which is purely text data for instance right let's say the kind of what is on the menu for that restaurant that is also text information for example so typically to be able to find

semantic similarity between, let's say, what two restaurants offer as part of what's on their menu. Maybe you can do that through a better database, right? So when you create embeddings on both of them, for instance, and you try to look at similarity between the two.

But the fact that, okay, let's say, you know, if two restaurants are part of the same subcategories of cuisine, for instance, right? Then there is also the organization in the category hierarchy of the restaurants, right? The cuisine, for example. The proximity of the restaurant to a parking lot. That is a relationship, that is a semantic relationship between a parking lot and a restaurant. And

that it is better to model always in the form of a graph, in a knowledge graph, for example. Now, when you have modeled that and somebody asks this question from an end user point of view, I am looking for a restaurant that serves, I don't know, maybe sandwiches, for example, but I can park near the restaurant.

So the notion of being able to find a parking space near the restaurant, for example, that kind of information, that semantic relationship between two entities, that will come from the knowledge graph.

The fact that you are looking for a sandwich, for example, that will come from a semantic similarity based on vector search. And then being able to combine that too. So this is basically the kind of information that you start trying to model and see what fits where, essentially. Right, okay. So in that case, when you are constructing the prompt...

as a part of the reg. So you first query graph database, understand the different entities that should be included as part of the query, and then as part of the problem, you include documents from the vector database that you retrieved and send it to the LLM. Is that how it works? Is that the steps?

Well, so now we have talked about the organization. Now let's talk about, you know, typically when a query comes in, there is a very important part, which is the query interpretation, right? And LLMs play a very big role as such. And this also ties back to the point that we started off with is that what has changed, you know, knowledge graphs have been around for so long, but in the last couple of years, what we are seeing an increased interest in knowledge graphs, for example, and one of the bigger problems,

challenges with knowledge grasp was always that okay when you have been able to and when you say i have transformed information into knowledge one of the natural extensions of that is that you want to start

you know conversating with that system you want to ask it more contextual questions rather than you know a text-based search for example right so that's the next step that you would generally do and in information retrieval but being able to do a very good job at information retrieval is not just a function of organizing data it's also being able to understand what is the user looking for

So what is that hidden context in the user's query? So which is a layer of query interpretation for instance, right? So what then becomes also important and what is happening as of today is that can that layer of LLM actually understand as step one can actually understand what is the user looking for. In our case, the user is looking for proximity to a parking space.

the user is looking for a specific type of, you know, food, which is a sandwich, for example. Now it could be different kinds of sandwiches of different cuisine, for example, but that's separate. And then once you have done that interpretation, which typically means that you have taken your query and broken it down into a structured form.

And then you will have a group of agents, which basically, or maybe one agent, which actually acts as one kind of a dispatcher. And that's why you see this whole topic of multi-agent systems as of today, right? When you dispatch that query to one, basically you do the query interpretation, you send it to an agent, and then that agent knows as to where should it dispatch the query to based on, or which part of that query to get the best information out of.

because that you know kind of a routing information is part of that agent essentially so it will now know that proximity to uh parking lot that information it is best retrieved from the knowledge system uh you know the information with respect to you know what is the closest uh for example uh the

semantically similar uh food or representation to a sandwich that goes to a vector database it pulls out that information from both the two combines them together and gives you that result for example that would be one way to do it there are other approaches wherein we say that look we are going to we don't need you know kind of a dispatching mechanism but we have a common

common let's say embedding generate out of generated out of both one that is generated out of the organization or the you know the connectedness of things one is actually the semantic representation of things for example and then you combine them into a common embedding and you just hit that embedding and get that information from so that becomes the second one is you know

typical RAD, for example, which combines both of these. The embeddings are a combination of both or a function of both. The first one is as a next step, essentially, wherein you actually use a multi-agent kind of a system to get information from specialized information sources.

Okay, okay. I'm still having trouble kind of visualizing the actual flow in which everything happens. So imagine I'm going to... Okay, one example that I go to Google Maps and I'm searching. I want to have some kind of...

say, Chinese pastries, for example. Go to Google Maps and type in Chinese pastries. It comes up with a lot of things that are somewhat relevant. Some are just completely irrelevant. So how can we use vector databases and graph databases in this-- like your maps example, how can we combine the two of them when the user types in, say, Chinese pastries?

into into your chatbot yeah so uh let's say you know chinese pastry is probably a base example because that's the transition that you know when you want to uh let's say use this kind of systems you're as a user and this is again an active area of work that is ongoing is that how do you you know help the user transition to a state wherein they can ask more

more conversational questions, for example. So rather than asking Chinese pastries, you say Chinese pastries for Friday evenings, for example, right? So then you need, the system needs to understand two things. Of course, Chinese pastries, which restaurants are open on Friday evenings and what is the occupancy rate of those restaurants on Friday evenings, for example. So the decoding has to happen

into a form that, you know, all of this information, first of all, interpretation of this information and then extraction of this information. So now let's say, you know, let's start with Chinese pastry, for example. So when you go and search for Chinese pastry in Google maps, for example, one simple thing would be, you know, just to do the traditional text search, for example, right? Which means, okay, Chinese pastry is what you are looking for. You would do a text-based match and then it will give you something, whether if it is there or not.

But then with more semantic search, you will also be able to figure out what are other things that are similar to Chinese pastry, for example. Maybe a bun, for example, or something else. But maybe within the Chinese cuisine, for instance. Specific items that is an example of a Chinese pastry.

Yeah, but then now the question is the fact that when you do semantic search, should you only get results that are semantically similar to pastry but within Chinese food as a cuisine or should you also be shown pastry, let's say a French pastry for example.

Now, that is something you can, of course, if you really want to go about it, you can probably add constraints as part of your embedding generation and you can model that. But you can also simplify it saying that, look, I am able to identify that Chinese is a kind of a food category, right? So I will use the knowledge graph to only look for information or food types that are within the Chinese food category.

And piece tree is something I will do a semantic search on. So what you will do is you will find semantically similar results using a vector search. But in this case, you will use the knowledge system or the knowledge graph to build a constraint on top of that and start filtering things out as opposed to processing layer, for example.

Right. Okay. Okay. Gotcha. Yeah. Cause that's, that's one of the problem that I keep finding with things like, you know, searching for specific food cuisines on Google maps. You just get the, I think that they don't do the constraints. So you get just, okay. Pastries or Chinese restaurants. And they never quite understand the context of the query, which is, I'm looking for a specific kind of cuisine. Well,

specific kind of food within a category of cuisines. And so it doesn't use, like I said, using the Chinese category to limit the results that comes back. And so I guess in that case, what would you say are the current state of art of knowledge graphs with LOMs? How much further can we go with this?

Yes, a number of core problems are also being solved or at least we envision that it will be solved. One thing is there are two parts when you go and build a knowledge graph. One is being able to bring information into the knowledge graph. And this was always non-trivial. And this is where adoption of knowledge graph as mainstream system.

for example, was tricky because how do you ensure that first of all, you can add continuously keep adding information, for example, and secondly, also maintain sanity of that information. So these were typically two of the things.

So what has started happening with the advent of LLMs is that ontology, typically. Because you can just take a general purpose ontology if you are building something for, let's say, something which is very generic, for example. But if you want to build something for a very specialized domain, for example, say, for instance, in the medical domain, for instance, or even in the location space, for instance, that

the ontology has to be specialized that ontology has to be specialized for that domain

And that specialization of the ontology, it is an upfront time consuming work. So there weren't a lot of automated ways for doing this, for example. And when, let's say, additional information starts coming and say, for example, in the medical domain, if there has been a new publication of a certain trial of a certain vaccine or a medicine, for example, how do you bring that information?

and still be able to bring that information in the knowledge graph under the constraint that you might have to extend to ontology because you know ontology is never finite in as such right so ontology also grows over time based on you know how information comes in so

Traditionally, a big part of this was with a lot of manual intervention. Now with the advent of NLMs, what you can start doing is because you can take the whole step backward and say, okay, can I use the ontology to do some kind of a rag on the input data source? Or and say that, look, I have the ontology. This is how I am organizing things.

This is the source information, feed both of them to the LLM and tell the LLM that now structure the information from the input source into this form.

And that is where, you know, of course, will it be completely automated? No, because you still need guardrails. For example, you need still need supervision. But a lot of the upfront manual work will start going away so that you can start, you know, managing the ontology in a more semi-automated fashion, for example. So that's one of the, you know, niche parts, I think, where we are looking at, wherein you can, or state of the art that we are looking at is that how can you start building

more semi-automated ontology management with LLMs, for example. That's one. The other side is typically the query interpretation, right?

Even though earlier, a couple of years back, if you had an LLM, for example, building that query interpretation layer was not trivial. You had to put in a lot of effort to build that query interpretation layer. And think about it from the complexity of if you had to do it multilingual, if you had to do multi script, for example. So it added to those layers of complexity.

Now, a lot of that effort. So if you were actually building a knowledge graph, typically your effort distribution would be, you know, earlier days, let's say 50 percent, I would say 50 to 60 percent of the knowledge, building the knowledge graph itself. But also the rest of the 50 percent or 40 percent would go into the query interpretation system.

in building that query interpretation system that actually identifies, understands that Chinese pastry query and breaks it down into a structured form.

Today that equation is changing, for example, because now what is happening is you can focus a lot more on the part about building the knowledge graph, really engineering it into a proper system. The query interpretation part, you can offload a lot of it to an LLM and say that, okay, because that LLM has that knowledge to be able to understand what Chinese space tree means, for example.

If it does not still, you can still do a small transition or take a small iterative step and say, okay, I will prompt the LLM.

based on the information that I currently have. I have category information with respect to pastries, what kind of different cuisine types those pastries occur in and then feed that as a prompt to the LLM and then we are actually seeing the LLM is still actually able to do a very good job at being able to interpret the query and break it down into two subparts saying Chinese is a food category

pastry is a particular kind of food, for example. And what is happening is that the user is looking for pastry under the Chinese food category. So this is typically what an LLM is able to do.

But cases wherein, again, you want very specialized knowledge, for example, the LLM has to be fine-tuned. You cannot just do it with an iterative step of prompt engineering or relying on a specialized or a general purpose LLM. So you will have to start fine-tuning the LLM based on that domain knowledge as such.

So how do you fine tune that LLM to be able to do a better, you know, for example, query interpretation. So these are, you know, two of the things I think are some of the research topics or the directions in which at least the industry is going forward. Yeah. Yeah.

Yeah, so you're describing that I was just thinking that because you're talking about ontology, you're talking about knowledge graph, those are very much domain-specific I guess, you know, problems and

And RAG normally is, well, I guess RAG is probably not for specific domains. It's probably useful for extending the LOM with some additional information data. But if you want something that's really specialized for a particular domain, you do want fine-tuning. So in that case, have you seen some example of using RAG

Are people doing this with fine-tuning as well? Is that a case of using the knowledge ground to identify which documents and what information goes into the fine-tuning process so that you don't over-tune, you don't put too much data into the fine-tuning process?

- And that's a very good question because this is also something that we have looked at it. See, one of the very important things that you also run into with any of these LLMs is the issue of context length, for example, right? And you run into the issue of context length if you are trying to either prompt engineer or fine tune an LLM with large textual documents. But imagine if you have been able to break down that textual doc or text

form of the document into a more structured form.

into simple organization of subject, object, and predicate, you have to a great extent cut down on the context length requirement because you have been able to compress that information because of the fact that you have been able to organize it. And then you are able to feed in a much more compressed information to the LLM, for example, which also has implications in terms of how soon or how

efficiently can typically the llm fine tune for example um so this will also have an impact there because now instead of passing it uh you know sentences which are uh maybe

I don't know, thousand characters long, you're sending an LLM information or giving it information, which is very small, which is think of it as small phrases. So, you know, if we have to do a thought experiment, if you were able to fine tune an LLM with phrases rather than sentences or rather than a collection of sentences, that phrase based fine tuning is obviously going to be more effective.

efficient in a way, right? Because of all these reasons. So yes, absolutely. This is again a reason or one of the things that you will probably all of us will see that. And also probably the reason why knowledge graphs have become so popular in the last, everybody has been talking about it, is that you can also bypass the constraints of fine tuning in many ways by through the simple organization.

Right. And the fact that you also mentioned that you can use LLMs to organize input data into knowledge graphs and make it easier to better, more automated way to process input data. It feels like that's this like almost like a

recursive effect where you can have better LLMs at organizing the input data into a knowledge graph and then use the knowledge graph to then make the fine-tuning more efficient and producing better fine-tuned models. And then you can use that to, again, better organize the source information into knowledge graphs.

I guess, I mean, that's probably going to be a cost involved with iteratively fine-tuning the model, but that feels like that's the path that you can lead to incrementally improving the quality of your fine-tuned model. Yes.

Yeah, absolutely. But also the thing is that, and that is something that will need to be figured out is that what is the comparison in terms of the cost? So how much does it really take to fine tune a model with large documents versus how much does it take to fine tune a model with more structured information when it is organized in this form? And the offset that you get,

is it good enough that you are still able to take that overhead or being able to again take that llm and do this recursive process or iterative process wherein you go back to the source use the llm to again extract information in the structured form bring it and figure out you know what is wrong and again feed that to the llm for example but you are spot on that this

there is a closure of the loop, right? That the LLM that you are actually fine tuning based on the information of the knowledge graph, that LLM is again actually used to extract information, the information that is coming in form of unstructured data, different kinds of documents and everything. That information to extract information out of it in structured form and bring it to the knowledge graph, which again will go into the LLM to fine tune. Yeah.

Thank you, Aniket. This has been really, really interesting. Something that I haven't, I guess I haven't heard too much about in terms of how people are using Knowledge Graph for LLMs. First, some, you know, top, the high level LLMs

mentions of this but never quite like a detailed walkthrough on how it might actually look but yeah thank you so much for your time before we go is there anything that you'd like to share with us any upcoming work any maybe you mentioned at the start before we start recording that the

you're looking to expand your business. Is there anything that you want to kind of mention there in terms of, you know, if someone wants to work on knowledge graphs with LLMs, how can they reach out to you and maybe join your company?

Yeah, absolutely. So we are actively in the process of working on a number of these problems as such. The fact that being able to extract information using an ontology from the source system, for example, and when the source information is in the form of, let's say, text documents, and then how do you structure it? How do you do a better job at query interpretation, for example?

And these are active work products that we are currently working on. So if somebody in your audience is interested and if they want to collaborate, we would be more than happy to collaborate on that. Okay. So where can people go to find you and find out more about what you're doing?

Yes. So I think currently it would be best to reach out to me on my email ID. And based on that, you know, we can see how the next steps evolve. Yeah.

Okay. All right. I'll include those in the show notes so anyone who's interested can reach out to you and chat about how you guys can collaborate together. So, Enneket, thank you so much again and I hope you show the best of luck and I hope to see you some other time. Thank you. Thanks a lot, Ian. Thanks for having me on the show. Thank you. No worries. And everyone else, I'll see you guys next time. Okay. Bye-bye.

So that's it for another episode of Real World Serverless. To access the show notes, please go to realworldserverless.com. If you want to learn how to build production-ready serverless applications, please check out my upcoming courses at productionreadyserverless.com. And I'll see you guys next time.