Packaging MLOps Tech Neatly for Engineers and Non-engineers // Jukka Remes // #322
55:30 Share

You can put the, we can extend on this later, but basically you can put the expectations for things on our platform and then you can take it to the ML operations side and then you can also follow up on how those expectations are met.

So I'm Jukka Remes. I'm a senior lecturer of ICT and AI at Haaga-Hellie University of Applied Sciences and also the founder and CTO of A2Wave AI. And yeah, mostly I drink tea. We were just talking about open source platforms. You mentioned that is very, very top of mind for you, almost passionately top of mind. Why is that?

Well, yeah, I actually am a quite late entry to the, like actually contributing to the open source. I've been mostly using those, but then we like found this new life for this platform that we created at Silo AI for MLOps, a certain type of like open source platform. And now like I've been working since Silo,

in academic setting and then like I've been in the research world before and I really when I started things there weren't these kind of like stacks available so I'm kind of coming the full circle back to the research world where which would benefit from MLOps and then this like having the open source platform which we can develop for that purpose specifically then

It's now one of my contexts that I'm living out. Can you explain to the listeners what was Silo doing and why did you need to create an MLOps platform? Yeah, so I joined Silo originally because Silo was like, now it's acquired by AMD and the realities might be different. But back when I joined it in 2020, just after the COVID time started, I

It was a consultancy largely. And I joined as a solutions architect. So I did client cases and big and small. And many of those actually that I ended up working with MLOps. So there was like this...

demand at that time to build practices for the MLOps as well as then the platform settings and then we also like then launched as one phase of Silo to have this like technology areas and I also took up on leading the Cloud AI and MLOps area that we had at Silo and it then entailed like both

competence development because we had like people with very various backgrounds at silo especially coming from academia so very strong like deep research-based AI and kind of data science type of like skills but varying levels of engineering related to the AI and MLN

Then partially we started like putting effort into the competence development, having like information sharing sessions, workshops, like all kinds of things, visiting lectures and such. But we also started developing that because we had a certain amount of like internal R&D activities at Silo even at that time. So we started developing in my team then workshops

this platform based on open source components and the it was that it's quite like loosely it's kind of integrated but quite loosely so we are basically we're collecting those kind of like open source technologies that made most sense and we're going to extend that and

make the platform for certain kind of client cases so that we can, for those clients that completely, for example, outsource the ML development, that we would have a platform to do the development on and deploy the models and such. But then also regarding these kind of cases where we have

clients who are a bit more mature or even really mature and they want to build their own platforms or parts of the like MLOps setting, then we can like leverage, for example, even just part of that platform setup that we have for those purposes. So that was kind of the origin of the platform work in itself. So classic story, you have AI researchers and these ML researchers that

don't have as strong of engineering chops and they probably want to see their models and their work get into production. So you create a platform to help that process. You mentioned something there that I want to hit on, which is you not only created the platform, but you created more than the platform, the processes around that. Can you explain what those were too? Um,

Well, actually not internally that much. We didn't like standardize internally that much. We pretty much focused on the platform development itself. But of course, like in the client cases, there was like what would be the workflows and these kind of things for various clients.

Take me back to that moment when, especially internally, when you're saying, okay, we need to start building an ML platform. Where do you even start? What do you go with? And why is that the place to start and then fan out from? Well, it was like based on the kind of the demand that we were facing. So it was very practical in that sense that it had to do with the, like what we thought that we would need when engaging the silo clients at that time.

So we would need the technical platform specifically. Yeah. Which piece of the technical platform? I mean, is it a... Like end-to-end. So we basically incorporated, which is still like the extent that we got at that point, that silo. We got the pipeline orchestration and then the model management. So supporting both training and development and the...

model and metadata management and for the experiments and then the deployment, but also included some components for the monitoring part. So that was the kind of the intent and we covered the whole extent end to end from all the way from training to deployment and operating the models. Now, I'm curious, did you also do stuff with the data?

No, we didn't actually go that way then ultimately. So yeah, that was one idea that we would extend to that direction, but never got to that. So the kind of the reality at Silo as in any like this sort of like fast evolving and fast growing industry.

company was that the direction changed in somewhat like how we do things and so so and like new things came along and then we we kind of like did develop the platform forward in in this R&D collaboration that we had with international consortium a bit but then then we didn't

that much yet anymore like put the R&D efforts there but there was actually a separate like R&D entity within Silo established and so the kind of the most of the R&D went the kind of like a bit to other direction so and ultimately this also led to the possibility and the idea that let's open source the platform because the within that consortium like University of Helsinki there was like a

interest for that also Fraunhofer Institute in Germany and some of the companies involved with that. And so if I'm understanding this correctly, you saw for the external clients of Silo, you needed to create some way that these folks could standardize and get these models into production. And so you stitched together

a few open source tools to make sure that you were landing on these key pieces within the ML development life cycle. And after you had done that, you recognized, well, since this is pretty much a full platform in itself, why don't we take what we've built and put it out there into the world?

Yeah, exactly. And then, of course, like continue with that, because then we could involve the University of Helsinki has continued developing certain things on top of that actually in the research like context, because they have been making a setup, extended setup with that platform. We are soon like also merging that effort into the

into the main repo of the platform. But they have included support for this high-performance computing or supercomputing in the centers that are available in Finland for research use. And we are still actually continuing to extend on that. And then I have now continued in the university context myself with the students to work on the project.

So tell me what use cases this supports, because I imagine it is certain types of models that are being built, or it is certain types of use cases that you're supporting, or is it across the board you're trying to help support all different models? I haven't yet myself looked that much into how the components keep up with the large model model.

like context so have they like as far as I understood from the University of Helsinki that the because we have basically Kubeflow and then we have MLflow and then we have Kser from the Kubeflow ecosystem so they are pretty generic right so so you can basically do anything with those but then like how practical it is or like how efficient it's to like

do the trainings or do deployments with those regarding the launch models. That's, for example, one thing that it seems to have been tested by University of Helsinki, but that's one new thing that I'm looking into with them. But in general, it's pretty much meant for like to cover all kinds of like cases. Of course, like

Often, like if I go present to some, especially certain fields of research, they bring up some like more like, I don't know if they are exotic or edge, but perhaps like a bit more rare cases, like what about the federated learning here and this kind of like...

set up. But the basis, I mean, that's very generic. But then, of course, it needs extra work to perhaps cover certain kinds of cases. Since we're talking about researchers getting a bit more platform jobs or using an ML platform, have you pondered the value that

an ML platform brings to a researcher? Yeah, so like said, actually, I need to comment still on the previous one that you can, of course, run any bigger workloads as well with any GPU-requiring ones, and of course, with the

with the kind of the additions that we are now doing there's a supercomputer environment that has a lot to do with leveraging the GPU super clusters that are available like the AI factories even the clusters that are within the scope of the AI factories that the European Union just funded and which are being launched at the moment so yeah so the intent is to go to that direction that we can also train large models

also with this platform. But yeah, so the value of the platform, so in research, so like I said, I was like, I was working pretty early on, like before the whole, even not just before the deep learning times started, but before the data science kind of like era started as well. I was at University Hospital of Oulu in Northern Finland and

it was like that time analytics and ML I was running, uh, certain, uh, pretty ready-made, uh, algorithms through, uh, or like, uh, data from, uh, brain imaging through, through those. And there was like a lot of like, uh, within that research, a lot of like variation as to like, uh, what kind of like parameters do you use with those analytics tools? Uh, and, uh,

like what kind of like filterings of data or other like data preprocessing you need to have there. And then when I had myself, for example, that kind of like setup where you have these different options and you need to like somehow orchestrate running a very big set of like different kind of experiments basically or analytics jobs.

with different kind of pre-processings then like handling that those all those combinations and the results therein or even like creating the running of those then I ended up doing that without any of the tools that I had so far so that's largely kind of the basis of my appreciation for the current stacks because it was really tedious and I see a lot of like opportunity I think

I'm not sure if you are familiar with KNIME. I guess it has found its place in bioinformatics at least. So KNIME was kind of like the, I think the first one emerging at that time when I was doing this. And after that came a lot of different things. But yeah, even the Python, it was used a bit, but it wasn't in that big of a use, especially in those fields that I was in yet.

at that time. So the kind of this bringing the consistency to complicated research processes like increasing the quality we are supposed to trust the information to scientific like results that we publish from there right so they are supposed to be the

creme de la creme of like all the information that we have in the world even though I'm not sure like how much fake science there is also nowadays with the but still like the quality criteria are very high right so you if you are doing the complex stuff then you need something to deal with the even executing that complex stuff but then making sense of the complex stuff and the results this is a fascinating statement how the reproducibility of

researchers work potentially comes into question if you do not have a standardized way or at least a way to actually go and put that model into production. If it's just research that's being done in the lab, then you write a paper on it, that's great, but there could be some

problems when you go down the line and you say, all right, now let's actually make this something that the industry can use. Yeah. And even, even like, even if we stay in just the science, uh, then like the kind of, uh, how well can we assess, like this goes back to the, again, like more generic things than, than rather ML alone, but it reflects on the ML as well.

Because there are reviewers that need to be able to review the results. And then it's much more credible and it's much more efficient to review if you can share everything. So you can share. Nowadays, of course, there's more and more like you are able to share at least the results.

experiment set up code and parameters and these kind of things the data is a bit tricky given what kind of like constraints might be related to that use but it's even like better if you can like share it in exactly the same way as you run so like you can have the full pipelines and the full experiments like as such and somebody can replicate that then it's much more like accessible in some sense.

But yeah, like you said, also kind of like taking the research results into anything, any practical use, then if there's that sort of like applied research angle that the next step would be the commercialization or the productization, then there's a huge gap if you cannot take it. Like I'm always myself like reflecting, for example, I was working at Nokia Technologies for a while and

then I saw there and I saw at IBM, at which I was working after the Nokia, that, okay, yeah, there are these, both of these big corporations, they have the research centers, but the research is actually a lot in many cases, like even there in that sort of corporate thing, disconnected from the kind of the product development. So they are kind of like different, all the different stages. So perhaps they are like a,

lifted and shifted in certain manner the results then to the product side and taken further there but there could be much smoother transition to actually commercialize and I think now with the advent of the or even it's not even an advent anymore but the coming of the regulation like AI act and this kind of things then it's if you need to have all this like provability of that you have done your

do things when developing the model. So how can you prove that if you don't start that from the research stage, if you try to catch up with that in the product development phase? Yeah. I don't know how it was at IBM, but I know that a friend of mine, Danny, was working at Google from, I think it was 2015 to 2020,

And his whole job was to figure out how to take the research that was being done in the ML sphere and plug that into products that Google had. And so he was working closely with the researchers and

anything that was coming out that seemed promising, he would go around to different teams and say, you think we could use this? You think we could maybe plug this in here? Would it be useful if... And then he would sit with the PMs and try to figure out if there was products there. Yeah, that's also like one perhaps like aspect, like how, what is the driver? I'm really for like this sort of like kind of

unlimited like innovation if that's possible and kind of researching like something that is not constrained by the by the by the business but then then yeah you have like this kind of like a bit like tech first kind of approach there right so

that it's not necessarily like addressing, you're kind of going the other way around because the results might not be based on the real world problems, right? And then they are not addressing exactly some problem. Yeah. And sometimes you can create something very cool that people don't figure out is actually valuable until five years later, like the transformer models.

And it's not even you who figures out really how to productize that and bring it to market. So it's a fascinating one. And he was also the same guy that told me when ChatGPT came out, he was like, yeah, I remember when transformers were at Google and I was trying to figure out ways to put them into different

different like google sheets and figure out if we could add them to google docs and or the translate app that they had so it's a fascinating one to think about now i i also wanted to touch on this platform that you're creating a bit because of what you have under the hood you mentioned that you have cube flow you've got some ml flow and then you've got k-serve

Yeah, and then Prometheus and Grafana, basically. Those are the kind of the five things we were supposed to like extend on them. It's completely possible because it's pretty like simple, but it's packaged together. I guess that's the kind of the beef of the whole thing in terms of like making it easy to use. And so is it another abstraction layer that sits on top of these different tools? Or is it that these tools now can just...

really easily be deployed as one cohesive unit. That's kind of like the one major thing, because I think that the, like playing around with the platforms line and trying to build them like some separate set of like instructions,

like according to some separate set of instructions and then manually that's or you know like perhaps automating that but in a custom way it doesn't make sense like for many users so first of all it's it might be like tedious even for some people that deal with the infra and deal with the platform if it's a new thing to them then let alone let's say that we have that

university student or like researcher from any field that just wants to run the ML so like if they if they want to take this kind of like setup into use and they are not they are not they don't have like access to some cloud service as the for example universities not often have and not on all the like in the companies is even have then then this kind of helps with that but the kind of the

philosophy originally also around this would be that then you would like develop this kind of like more of pipelines on top of that and of course like when we are like using kubeflow as a basis then of course like then you if you have only kubeflow pipelines or kubeflow there then you end up defining them

in the way that you define things for Qflow with the DSL and that kind of things. But that was the kind of the idea that we, and one could provide like these pipeline templates in a sense that that would be the kind of then the, in a sense, the more integrating things that you would have like the end-to-end pipeline for all the different stages and using

all the different parts of that. And you could aggregate this like library of pipelines. So it's a bit similar as what you can have, for example, with AWS SageMaker and Azure ML, right? You can define pipelines there and then the whole team or community or whomever can share them and use them as a basis easily to launch their own things on top of those platforms. So that was kind of the philosophy here that we have pretty loosely developed

coupled services so those services are not that much yet at least like integrated within the platform more what is more is the kind of like in the package that integrated package in a sense that it you can install them easily to put in place of course like this work then that my students at my university has been have been doing

On top of that, like with Git repositories and CICD pipelines to go with this or any other platform, that's kind of like an extra layer also there. Well, you've given me a lot of confidence in the youth. When I hear about AI researchers using Git and CICD, that is amazing, first of all.

Yeah, I'm not sure like how many actually use that. I would guess that many use some notebooks just from the corner of their own laptop's hard drive. Yeah, but this idea also is something that makes a lot of sense to me in that I can, as a researcher, go and grab a pipeline off the shelf

and not have to worry about standing up these different tools and taking the time to figure out why did installing Kubeflow just crash my whole system. You know, like it is a much easier type of workflow if I can now go and grab Kubeflow

these pipelines off the shelf. They come with everything that they need. And I don't know how it works, a chicken or the egg. Is it that I get the pipeline and by using the pipeline, it installs everything? Maybe it terraforms out everything that I need as part of that pipeline? Or do I

do I get everything I need and then I can start using the pipelines? Yeah, it's more like the latter one. Yeah. So at the, at currently, so it's not like that it would provision it any place that would be kind of like extra thing to do as well. But the kind of the installer that we have for the, the platform is, uh, in the open source. Like we have from the silo times, the Google, uh, Google cloud, like, uh, manage Kubernetes, uh,

like how to deploy it there but the main kind of installer with the open source version of this platform is such that you can it's basically running with the kind so the kubernetes on docker or in docker and so then you can basically like provision the platform in your own computer as we are doing actually there's a lighter weight with just the key flow pipelines there

Or you can do it to basically any machine. So we are, for example, using the virtual machines that are available in the Supercomputing Center or this IT Center of Science at Finland for that. But you could basically put them anyway. So that kind of like need to make them pretty generic in that sense that has like been the approach that we have been using so far. But then, yeah, after that, then...

with this Git related stuff that the students have been doing. We are kind of like providing tooling with which you can set up like in a sense like configuration related like Git repositories and then ML project related repositories. So you can establish this configuration repository

for, it doesn't perhaps make that much sense for one person, but you could establish that for the whole team of researchers or data scientists or whomever. And then the tool, with the tool you can set up the projects for the individual people or per project. So those are the kind of the ML repos.

And those link back to the configuration repo. So whenever you have like some configuration that you need to update, everybody can just like basically pull much of the changes to their own repositories. For example, where do the common platform instances reside? So that if the endpoint addresses are changing or this kind of things are changing, so you can pull the updates to all the repositories that the actual ML developers are using.

are working on. And then the tool also set up the CI/CD pipeline so you can kind of like hide away kind of this sort of like platform interaction or platform API interaction stuff behind those CI/CD pipelines. So it like you can configure what kind of like CI/CD workflows and what kind of like file structure like a directory structure you want in the

in the repo, but after you have that and we have like a default of course like to go with that then it just setups that for you and then you can start introducing your Python code or whatever to the different steps of the for example the ML pipeline and then when you introduce that and if you commit to certain transit

like compiles the ML pipelines within the CICD pipelines and then deploys that to wherever your platform instances are that are connected to that branch in the Git. So this is kind of the way. It's basically like you're like saying this is a self-serve productionization tool. Yeah, yeah. But of course, like you can use that for, like I said, also to do the research. Yeah.

So after that you would have in the research context also, or in company context, if you don't have some cloud platform to use, or even if you do, you can set up the platform easily to some environment or environments, different instances of that. And then you can work with the, as usual with the ML Kit repo, but

Like focusing on the kind of the contributions that you want to do to the ML part and not to worry too much about the engineering. And what are some of the pipelines that you've been seeing folks set up? It's also training pipelines? Yeah, yeah. Of course, like with the

We have only like the example pipelines at the moment. So yeah, so those cover the training parts, but also in those you can have also like as part of the same like Qflow pipelines, you can have the deployments. But the idea like especially originally was that you could like launch whatever. So you can launch just separately the training pipelines. You can

do the deployments as separate things and so forth. Now I understand why you were saying it's also the ML part, not just the productionization, which makes a lot of sense. Okay, so now there is a bit of a juxtaposition that we talked about before we hit record where you mentioned teams desperately need MLOps, but...

It's not until there's this pain that they feel. And so you in a way are hanging out in a world where you can be accumulating tech debt and accumulating tech debt and not necessarily have to bring on certain practices or platforms anymore.

to help you get rid of that technical debt until you hit a certain level and then you're crossing the chasm in a way and i was just talking to a friend of mine on another podcast uh how ai built this about this very same thing how there's ml ops is very hard as a practice and so it

inevitably will turn people away from it because it is so much work but you get to a point where you can't deny that work and you absolutely need to do it and so if you are a company in the beginning stage and you have not hit the pain where you need to figure your shit out you can kind of live ignorant and blissfully yeah i guess like uh the

I think a nice comparison that I heard was to software DevOps just recently, that now it's kind of like pretty common stuff in the software development itself. It still wasn't like 10 years ago or so. But with the MLOps, even though there has been like several years already with that, I think it's the...

even the awareness, let alone then the kind of the adoption, they are pretty still like, not perhaps in infancy anymore, but at low level. And well, as you said, you can like, one can go like, go about your business, like, or go about your things like ignorantly and such. And then you at certain point, like end up catching up that awareness.

or needing to catch up with that data. And I guess like in research, actually, that's a bit like a perpetual thing because there is certain kind of like, perhaps like threshold that there's a kind of this, like we discussed, like this standing chasm between the research and then the kind of the utilization of the research results in some product context.

Yeah. And the product hitting a certain scale. And then once it's at that scale, then you start to have to make the product velocity go faster. And so to get that velocity that you need, you are figuring out, all right, well, let's implement some processes around this. But also like, I think just for example, if we consider the lineages and the tracking of the results, then I think...

I suspect that now the AI Act might actually force to consider this that we talked about previously, that how can you actually, how can the utilizer of the, for example, models done in the research, if they are utilizing those in a commercial setting, how can you

like really comply with the regulation necessarily unless you have the kind of the whole story behind those models I'm actually a bit skeptical about like this some of the foundation models like how how will that turn out in in terms of the AI act because there you have okay you have model cards and you have certain information there but does that ultimately suffice so

that might be this kind of like traceability of like how things have been done, like one factor that will actually lead to more MLOps adoption in the research stages as well. But I don't know what kind of incentive that is to say academic research then, like perhaps there will be some funding instruments guiding ultimately that if you want to get the funding for the academic research, then you should also have some sort of like...

You should somehow deal with these things as well. I don't know. Well, yeah, you already see it with... At the time of recording this, Llama 4 just came out this week. And...

I think there's a asterisk in the whole release notes that say, oh yeah, this is freely available for anybody, except for if you're in Europe, then you really have to be careful with it. And we're not going to take any, or we just almost want to step away from that whole shit storm and having to put our model out there, open source, and then potentially get sued or have the regulations, uh,

beat down on us. So in that regard, yeah, it's a little bit of you need the practices, but you also need the transparency to show what you're doing if you really want to put it out there. With the EUAI Act, I think a lot of people are

unclear on exactly what you need to do because it is written in language and it is not written in code and so the engineers that are creating the actual ml and ai or the researchers that are creating it are sitting around thinking okay how does how does this apply to what i am actually making yeah and i i think it's there's a still a lot of like as far as i understand it now

there's many things like up to both like interpretation as well as pending like definitions so even some parts of the or some certain things related to AI act they are coming along only after a certain while and I have let myself to understand from people that understand the relationships of like standards to the regulations also that the

probably the certain standards will be actually defining the many of the technical details ultimately for those but that's also like a longer process in a sense because like yeah but there are many of those standards already so those will be probably like being leaned on when

when these things get actually interpreted or set in more set in stone. I have been working with the regulations in the medical field because that's the, besides the software and ML, the third kind of like, or more like domain specific, like part of my whole career. So, but not only to a certain degree with regarding the models in that context, because it was like earlier.

but now I'm working with the AI actor because that's partially the context of the company that I have been founding lately. What are you working on these days? Yeah, so besides these like MLOps things that I'm... Those are largely like this kind of like basic MLOps related

And for example, MLOps related DevOps things, those I'm doing in the academic context. But then I have now been founding this new company, 8Wave AI. So we are putting something more on top of the MLOps. So we are basically like connecting the AI operations to both like from the training as well as the deployment side to the

Like how do you guide and lead the AI project so that they actually provide both value as well as then are compliant with the regulations. So yeah, that's what we are building product for. And how do you build a product that fills that need? Because it feels like it is inherently...

a service, not necessarily a product. Well, it's a, yeah, we are, well, we are building basically a digital service, but it's a product development, so we are not, it's not consulting, right? And so you're helping folks that are using AI and ML understand how the AI Act in the EU works

has actual repercussions for them is that it it's more like that you can you can put the uh we can extend on this this later but basically you can you can put the like the uh expectations for for things on our platform and then you can take it to the ml operations side and then you can also follow up uh uh on on the like how those expectations are met so

So we are providing like in a sense certain type of like missing management layer on top of the ML Ops. So connecting the kind of the needs that for which the AI features or AI for processes should be built for. So and it doesn't concern only the AI but the also the broader business

needs so that you actually have some return of investment from the AI project. This is actually something that we saw like both at Silo with people and as well as then before that with some of the people that are involved with the company that often the projects fail because of that so they are a bit disconnected from the

So it feels to me like what you've created is almost a connector between the PRD or it is in a way it's a very strong and battle-tested PRD for the ML efforts because you're helping spec out what is needed legally and for compliance reasons and then for business reasons. And so you were saying that

The reason that a lot of different projects or ML products failed was because they didn't have that insight into building out everything that needed to be built out on the compliance and different stakeholders. Yeah. Perhaps not before because of the compliance, but before because of that, like they are not

like necessarily building exactly towards what is needed by the business. So there's a broken phone. So everybody are working in their own corners of the organization and we are trying to change that. So, yeah. Yeah. I've seen that. I've heard that many times on this podcast. It's a broken phone or you're not really each person

Each expert is trying to optimize their little silo and they're not looking at it as a bigger product. And so you have someone who is trying to optimize the accuracy score of the model. And later they realize, actually, I'm building a model that doesn't do what needs to be done. Exactly. Yeah.

And so you're coming at it from a different angle, I think, than I've seen and heard other folks talk about coming at it from, which is you're building this middleware in a way that is helping the business folks, basically every stakeholder, because there are so many stakeholders in this process. You're helping them all add their issues or their requirements to the document or to this process.

software that you've created so that at the end of the day when a product pops out the other side after that birthing journey you have the right product not something that has been optimized in a cave without any feedback for many years and you know exactly what has been happening there so you also like keep track of all the things that happen so you have the like constantly

the understanding like how it has come to be. So you're also tracking the kind of the operations there and then you have the confidence to report for audit purposes or you have also the confidence to actually take it to production, right? Because the business decision makers don't need to be afraid anymore because they can see that, okay, yeah, we are here, we have done this.

It fulfills this kind of things. So, yeah, we're good to go. Now, isn't this what most project management software is trying to do? And why do you feel like it falls flat? Like a JIRA or a ClickUp is... I think they are applied on a certain level. We have a broader ambition there to really cover all the stakeholders in a way that makes sense to them.

Of course, like you could like, it's again like the same thing as with the ML of sales. Yeah, you could build your own platform. Yeah, you could like use some tool or perhaps like you could do like use some individual tool from some place. You can, for example, track experiment information in Qflow as well, but

then you often people want to use ML flow more, more to that, even when they have the Qflow. So you, everybody could like build them whatever, like, uh, ways of applying, uh, things. But what we are trying to do is like productize that so that it, you would need to do that. Of course, like that's the whole value proposition of the product, productization there to begin with, that you don't need to have like this, uh, hacks or like, uh,

like custom things that you end up like also then needing to that much

maintain i see i okay so the the whole thing here is that because i was trying to figure out and play it back in my head why is jira not good enough for this or why is click up not good enough for this but you're saying not only are you gathering requirements which is what you could potentially be doing in jira or click up you're plugging into the platform so that

It is tracking the experiments and it is helping document each step of the journey in this whole ML product lifecycle so that it's not like you have to go into JIRA and try and figure out a way. Are you uploading a CSV file to your sprint to showcase, hey, here were all my experiments and here's the one that we're going with.

Exactly. Yeah. You're natively building it into MLflow and you're saying we've got this connection with MLflow. So if you want to go and then figure out what the experiments were, you can click into it and see all of that inside of MLflow. Well, you can like see the like, I think it's still different, like use that if you want to have like MLflow,

for hosting the models and such that that's a like as part of the MLOps that that's a different thing but with our platform then it's a kind of additional thing to which you integrate and then you put those kind of things there so that kind of like metadata there that the different stakeholders need to see there in terms of like lineages and and then the

performances and hitting the targets and this kind of thing. It took me a minute to fully wrap my head around it, but I love that idea of you have this

buddy that is taking the journey with you to help make sure that you're documenting everything that you're doing. But before you even get that far, you have all of the stakeholders gather requirements and plug it in so that you're building the right thing. And then when you do take that journey, you are

automating a lot of this documentation lifecycle and making sure that that metadata gets put in the right place and then is not lost or you have to go back and try and sift out two years later and figure out why exactly did we make that decision and have that model out there in 2023, et cetera, et cetera. Exactly. Yeah. Yeah. Yeah.

Cool. Is there anything else that you want to hit on that we didn't talk about? I'm not sure. Perhaps this covers pretty much what I've been at. And I just wanted to say that the, of course, like with the, there are other options like to the open source platform that I was talking about. Like, I think as far as I've looked at the flight, it covers certain bits, perhaps even a bit more like,

approachable manner. It doesn't necessarily cover everything as far as I understand, so perhaps the deployment side. But I guess if you are interested in the platform, please come try it out. If you are even more interested, come make a contribution, join the project. We are pretty early on with the open source one, but I think there's good stuff going on there and

for example, the software DevOps related part, that's agnostic. So even though we have this Kubernetes in Docker kind of setup for the Qflow and MLflow and so forth, then I think the DevOps part where you can set up the Git-based CI-CD operations for your ML projects, that's applicable even more widely. So there's a

different kinds of things and of course like if you happen to be in Finland or interested in the or needing to use the AI factories and the supercomputing we are extending to that like said so now in Finland and later on perhaps in Europe you can utilize this to launch jobs into the supercomputers. It was actually interesting to hear about the one of the first like AI factory launches that was in Finland because the

They claim that in this European HPC consortium, at least with this Lumi supercomputer that we have in Finland as part of that, the capacity now going forward in relation to these AI factories, it should be free for all small and medium size companies.

like companies in the union, which is quite a tall promise, I think. But I think it's very like, just for the kind of the product development part, there's capacity apparently available when you need to train the models. So perhaps in the future, you can utilize that capacity through this platform.

free GPUs for everyone. That's what we like to hear. And big clusters of them, so you can really do the large model trainings as well.