I Am Once Again Asking "What is MLOps?" // Oleksandr Stasyk // #308
01:07:22 Share

Hello, I'm Sash. I'm an engineering manager at Synthesia and I take my coffee out of the machine and then drink it. Folks, welcome back to another MLOps Community Podcast. I'm your host, Dimitrios, and Sash schooled me through so many things in this upcoming conversation. He has truly done MLOps in many different places. Right now, he is creating the ML platform. And as you probably know,

The requirements and expectations are quite different when you're creating a video generation type of product versus what he did at his last job around chemistry and creating an ML platform for a chemistry-led ML initiative. He broke down the different parts of this and where he has seen

initiatives fall flat. He also talked through how much of a cultural thing this is, not just a technical thing, which we harp on here many, many times. But this really felt like the man has...

gone through the trenches. He has some scars and he is sharing his wisdom with us. I hope you enjoy. And if you are listening on one of the podcast player platforms, not on YouTube, I've got a little treat for you because all these folks that are joining the community are giving me some incredible recommendations when it comes to music.

And I get to pass that on to you. This next song is from a band called The Voids. And the name is Leave It In My Dreams. I literally just posted something that was more or less a meme saying, like, pour one out for the homies who thought they could vibe code their way into a million dollar product. And...

It really strikes me as funny how much I'm seeing folks talk about how now you don't need to be a software engineer to become a millionaire with a SaaS product because you can vibe code your way into a SaaS product and all that fun stuff. Yeah, sometimes I guess people have to learn the hard way. Yeah.

But even I do respect that there's a lot of great things that you can now do if you understand software engineering. I 100% agree. To me, it almost feels like you know when the vibe coding has to stop. That's the key skill. Vibe coding is all good, but you need to understand when what was generated is unmaintainable. And that's when you kind of say it's enough of the vibes for now. Getting too many vibes. Yeah.

I gotta stop with the vibes. Bad vibes, bad vibes. Yeah, that's so funny because for me, I laugh because when you watch videos of folks saying like, oh, I'm just going to vibe code it. The prompts are very much coding prompts. And if you have known nothing about coding, you can't

prompt your way into that type of thing like you're prompting with very coding specific language even if it is in english it still is like you're using algorithm yeah or describing the exact things that you want and understanding what you need and the requirements there it's uh

It's pretty difficult in my eyes to be able to do that. And then the whole reason that I came up with this was because there's the meme going around of someone on Twitter that said,

Something along the lines of help me. I'm under attack since I've been sharing about how I vibe coded my product. Now people are like DDoSing it or they're finding security breaches and I don't know what I created and I don't know how to make it more secure. Some problems should not be solved with code. I feel like.

And, you know, vibe coding kind of implies that you're writing code and you've got, you know, it's text in, text out. And that's what developing software is all about, right? You just write some symbols and the computer interprets it. And outcomes value, and most importantly, Benjamins. But in reality, like, you know, sometimes you go, aha, maybe we should solve this with infrastructure. Or maybe we should just actually move the file over here and I'll make things a lot simpler rather than brute forcing it with vibe coding.

Sometimes you don't need code at all and just a spreadsheet will do. Yeah, or you have to think if someone has no idea about

security best practices and they forget to ask their yeah they're putting their secrets everywhere or they aren't even thinking about secrets and they aren't even thinking about the most basic common things that you learn as you're going through the trials and tribulations you you're going to learn the hard way yeah yeah yeah it reminds me once i asked a lawyer uh you know i'm you know

in a very sick way, I'm kind of interested in law, you know, like how hard is it just for me to do like, you know, a cheeky little bit of lawyering without any of the background, any of the experience, any of the education, like what could possibly go wrong, right? You know, just read the book. It's like, you know, it's just like a giant document, the law, right? And you go, yeah, there we go. Yeah.

And I'm sure the lawyer was very fond of that question. I was like, yeah, give it a shot. Just said, and call immediately. Yeah. Why would you ever need to pay a lawyer or a software engineer when you have ChatGPT? Absolutely. Or VibeLaw, Defend Yourself, all of that. On the other hand, I think...

The Brexit is... It lowers the bar for people to actually explore. There is definitely a light side to it, I think. Not to just completely shit on vibe coding. I think it gives people the opportunity to actually get into stuff because...

Like, last thing you want to do is be a gatekeeper, obviously. Like, no vibe coding for you because you need to have four years in uni to study how what n of n squared is, you know, o of n squared and all the complexities. But this way, you know, people can get stuff going quite quickly, even though it might be wrong. I think guidance is always needed.

But the fact that people are actually enabled to do this, that's almost a completely different world to live in. Sorry, I just wanted to bring a little bit of light to the situation. Yeah, we got to steel man it because like I said, there is some incredible stuff that's happening with it. I think it just needs to be taken with a pinch of salt. And like you're saying, the...

ability to explore and hopefully pique your interest and then you go deeper down the rabbit hole and you're now becoming more and more familiar with when you should do things when you shouldn't or how to explain yourself how to know when this vibe coding is going off the rails i think exactly yes yeah the key knowing when to stop and ask for help

Or knowing when to hire someone that can actually do it. Oh, but that's an even bigger situation. Vibe coding aside, knowing when you need to hire someone, that's a huge thing, isn't it? Yeah. Vibe hires. The chat GBT, the test-driven development doesn't tell you, hey, you should probably hire someone.

Now, that's classic. But dude, I wanted to talk to you. And I know that you've also been interested in chatting because you have this idea that MLOps is more important now than ever. And so it's great to hear that because in a way, because of all the vibe coding, because of all the AI hype,

you don't hear about how important it is to have these foundation and foundational pieces. Indeed.

Yeah, the best practices, the idea of whether it's Gen AI that you're putting into production or it's traditional ML, you still need those foundational pieces that are hopefully going to be a part of everyone's project. And so let's just start with that. I know you wrote an awesome blog post like two, three years ago, all about what is MLOps?

It still holds, in my opinion, but I could definitely write another one on a similar topic. Exactly. So let's just hear your thoughts on this and then we can dive into it. So the blog post specifically was incepted in my head. There was a scenario. I was...

given an MLOps team, I got told go do MLOps and that was happening. And in the meantime, there was another department who was kind of on the side and they're saying, oh, we would also like an MLOps team. And it was definitely not something that we could both hold. So it's definitely time to hire.

And the interesting question was, and to be fair, this was me quite new to MLOps, et cetera, et cetera. There's more details in the article behind it. But the interesting question was, what is the skillset? What do we put on the job description of, you know, you need to have X, Y, and Z to be MLOps. And boy, oh boy, I sat there stunned. You know, I don't have an answer to that because if you list everything, you're never going to find that person. Like good luck. So the blog post kind of,

reflects upon the idea thinking about so I myself big confession I don't have a PhD I don't have a research background which I sit on a chair of imposter syndrome and I see all these people with their chevrons and their masters and their professorships and but on the other hand I can claim that I got some things working in production I definitely know pods are failing and I got them fixed and I looked at logs and I know tracing and I know graphite is pretty sweet so

I have that to support me. So I imagined that the skill set of an MLOps engineer kind of takes the idea of a T-shaped skill set. So you have one long or deep specialization in something and can divides it. So one of the legs becomes your foundation where you came from, be it either a software engineer or a researcher.

And it kind of drives you towards the other side. So you start off with a solid software engineering background, and then you get told, yo, make some models in prod and get them served. And then you kind of dive into that world and immediately get hit with, oh, cross-key validation. And you go, where are these names? I don't understand. But slowly but surely, hopefully you're surrounded by people who can help you learn that this is a model, this is a feature, this is Drift.

This is what inference does. And then you can see, ah, I can see there's a problem here in scaling. Let me just do some microservice magic and make that scalable. Right. And there's a vice versa, which is, I think almost just as beautiful because the combination of these works really well together. If you're a researcher and you go, right, I've got a super cool model in my Jupyter notebook. I get my 99% accuracies. This is super sweet.

"But boy, oh boy, how do I give it to my customers?" You can't just walk around with your laptop letting people use it to influence your model, right? So you gotta do the deploys, right? And then you go, "Google, how do I deploy things?" And then all of a sudden, Dockers and Kubernetes and everything starts falling onto you and you kind of go, "Whoa," right? But then what I think the beauty is, and when those two skillsets, two little pie-shaped engineers or ML officers meet

And that's when the real magic happens because they can support each other, right? There's also arguably a third little leg between them. No one you end this year, please. This is a family friendly channel.

is uh the actual domain specialization right so for example yeah yes you have ai and yes you have software engineering but what you're doing right are you doing chemistry are you doing computer vision you're doing finance yeah like so that's the little little little thing that is this is the genuine you know domain expertise so you can argue that like you know it could be

super duper knowledgeable about the domain and then you can kind of grow out the way. So like you can stretch this analogy left, right and center obviously but like the main idea is that when you say MLOps you kind of go what does that actually mean? Like who's this MLOps unicorn that can do all these things together? But in my opinion it's not about who it's about the strength is in the team because then you have

economy of scale, you have the sum is a greater than the part of the sum, sum of the parts greater than the whole, sorry. And like that's when the real beauty happens. It's about your 10x environment. Yeah, the other

role or special specialization that I would think about is a data engineer and being able to get the data to where you need to go. And and this is it's interesting that you're talking about it in this way because it's very much, in my eyes, something that you would potentially need more of these specialties in the traditional ML world.

Do you feel like now in the Gen.AI world, you can have a bit less or is it just that it's different? Well, that's a good question. I do have some opinions about this current Gen.AI world, but what I do think the real winner is money making.

Because money making is a much more important thing now because you want to get that stuff going faster. VCs, please check out my cool video that I posted on LinkedIn. You have to sacrifice some things. You have to, basically. And it's pragmatism at the end of the day. So when it comes down to, yo, by the way, did you know data is important?

um like that is a foundation like i have to put it's essentially tech that you have to pay that pay down at some point um and if you don't pay it down it's going to be bad news bears um and then i think i think you could definitely take steps towards it but as you said as you just mentioned data as aside from mlops um oh yes my my personal opinion and it's almost like

a mantra that I always hold in the corner of my heart and my head that like can't have AI without good data. It's just that sometimes you get lucky enough with data to get going. But other times you have to like, you know, pay down the debt and get better at it or get better data essentially. You can MVP something really quickly. And it goes back to what we were talking about with vibe coding. You can MVP something

By vibe coding your way into the MVP and having some one shot or zero shot prompting and you're using AI with your product and it's just hitting an API. And then when you want to go to a deeper level of production, which you want to validate first, you don't want to get there. And I imagine you've seen this meme where it was something along the lines of, well, my

product has seven concurrent users so I guess we need to switch to Kubernetes now. Yes. And so you don't want to over-engineer before you have to really. Absolutely. Yeah. Inside of you there's two rules. One of them wants to do the right thing. The other one wants to do the thing right. Yeah. Exactly. And the thing that is constantly going through my head is like okay well

You MVP it. You get something out as quickly as possible. You test it. And then once you see right on, there's legs here. Let's go for it. Then you're doing what you're talking about. Very much like we're building the team with their specific specialities so that we can leverage each piece of the puzzle. And I'm just thinking back to what you told me before we hit record. And that is,

ml ops is more important now than it's ever been and so i i want to explore that idea a little bit more like why now more than before vibe coding vibe organization building if you dare if you will allow me to expand the analogy um and the question is what is ml ops at the end of the day nitrous right yeah what are those questions i'm sure we get asked that every day

But I'll put it straight here. Rather than being mysterious about it, it's no different than essentially DevOps, but with extra things, right? So you have to understand that you need to break down the walls. You need to be able to think about your feedback loops in your organization. It's, as you said, validating what you're doing, right? The more organizational barriers you put in place,

the worse the time you're going to get in terms of getting your feedback faster. And that is 10 times as true for ML because now you have a little bit of, you know, research going on over here and a little bit of production going over here. And if you let that grow into two separate things,

what you'll end up with two different companies essentially one is publishing papers the other one is saying yeah we've got a model that we made originally but we haven't updated it since so MLOps to me is making sure that well it's not about making sure but essentially trying to undo that

to create tunnels, to create bridges, to create windows, but ultimately to melt away the distinction completely. Even though sometimes it's very, very hard because researchers got to research and not everything in production is about ML. But what's most important is your value stream map from an idea in terms of, I would like to predict something to does it actually give us the money, right? And then the shorter you can make that feedback loop,

the better off your business will be for sure. So it's about being able to iterate on your ideas, being able to validate them and as close to the customer as possible, essentially, right? And then make those feedback loops tighter and tighter and tighter and tighter. Bring teams together, melt the concepts of research and engineering together,

If you need to, depending on the skillsets you have, you might need to have a product organization step. You might need to rewrite some code for sure. Refactorings. Maybe the Kubernetes is managed by DevOps, right? But you can try and build your organization towards in such a way where the process is actually streamlined, right? Right.

So this is kind of my spiel in terms of what I think MLOps is about. It's about enabling your business to actually iterate on your machine learning efficiently. Yeah, efficiently and quick. And I think that you point out something that is inherent in these projects, especially the bigger the company gets.

is how you need to break down those walls between the functions. And you want to look at it like we've said it for so long, but it does feel like it needs repeating. And that is recognizing this is just as much cultural and company culture, organizational, as it is technical. And so even if you have the coolest tech approach,

But if each team is siloed, like you said, you're going to end up creating two different companies where you have one model that's been in production and hasn't been updated for a while. And then meanwhile, you have another team that is just creating research papers. Yep, indeed. And recognizing that this is where the vibe organization building has to be kind of like, you know, put on pause. You can go, right, okay, now we might think of a reorg. We might think of this, we might think of that, but...

All of these little decisions come from, you know, sometimes arbitrary creation of a team even, right? Say like, oh, we need to build a team that does this, right? And it's little by little, these little actions and these little decisions that can solidify this difference essentially. Yeah, that was where I wanted to go next was the idea of the organization and the team structure and what you've seen happening

work well and not work well when it comes to platforms, platform engineering? Do you need... Is it the ML platform? Is it a data platform? Is it just a platform engineer? And how...

organizationally does that look are you embedding teams are you centralizing these platforms are you trying to do the hub and spoke model like have you tried it all each of those questions are almost worth the podcasting themselves uh i'll i'll start with i'll start with what i think works and what doesn't um i think what's important is the skill set sets that you hire

And this is very hard to speak about what's good and what's not, but imagine, let's say you just only hire backend devs and only hire frontend devs, right? If you just keep going like that, what does your organization look like after, if you like extrapolate in time, right? You have a whole bunch of people who can do this and a whole bunch of people who do that. And then hopefully you have GraphQL to solve the hard problem between them, right? And then...

I see this almost like equivalent, but you don't just have backend and frontend devs now. You have AI researchers and you have this and you have DevOps and you have that. So,

I think, and this is where I try myself to be as I don't like to use the word like full stack dev and all that kind of stuff. Cause like I'm crap at front end. I'll put my hands up. Right. I do try, but like I get laughed at every time my friends see what I do. I'm a classic backend doing front dev scenario. But what's more important is that even if you might not have the skills on the other side, right.

what you must at least have is the empathy, the empathy to see things from that side, right? What are the problems that the front end devs face when you do stuff in the backend, right? That's the first step to be able to actually work together, right? Same thing with research versus product engineering, et cetera, et cetera. Like you need to understand, hey, like researchers value their freedom to be able to experiment things. And that's very important because you need to drive that innovation because if they're not researching,

no, they're not, they're not really doing their job. So you need to enable that. And this is where the platform stuff comes in. Um, hard to say in terms of what's, what's good and bad. Uh, I did ask, uh, one of my good friends that I work with, what is MLOps? You know, one of those, one of those philosopher after questions and his answer, I still hold close to my heart, which is, you know, it could be anything in every organization is different. And, um,

I think there's just so much context in there is that like, yes, you can kind of vaguely see like this, this, these things you should do these days and you shouldn't. So I've already kind of described them, but when it comes down to actually practical things, what do you actually do? Right. How do you actually organize your teams? Yes. Sometimes you need people who like, you know, look after infra and maybe call them ML infra people. Right.

But what's more important, I think, is as a person who is in charge of hiring and building this organization, that you really, really, really need to know your people well, what their skill sets are, what their preferences are, and understand this dynamic. Because if you then just kind of go, and here comes an ML platform team, and they will do this, and here comes a research team, and you'll do that, you just miss so much nuance. Like that kind of goes out the window, and

it's it's very easy to get things wrong that way so like you have to kind of be very careful and do it step by step and definitely don't be afraid of a of a reorganization just in case because like you know you might need to change things sometimes when you build a lego house you do need to like go back a few steps right um in in terms of what not to do

It's also kind of hard to say because sometimes things generally just work well, even though they don't represent best practices. Yeah, they shouldn't, but somehow it's working. Yeah, yeah, yeah. I'm sorry, I don't have any clean answers here. Something, something, it depends. And please, my consultancy fee for that. Are there specific things that in your experience haven't worked? I don't have any glaring examples of like, please don't ever do this kind of thing.

But I can definitely say like, this could be improved, right? I think it's which skill sets you inject at which stages that can improve interaction. So let's say some, like to give another non-MLOps actual example is that, you know, somebody wants to make a microservice and they need a database, right?

Do they make a ticket to DevOps to say, please, can I have a Postgres instance, blah, blah, blah, blah, blah. And then DevOps go, oh, that's nice. Please join the queue, right? And then we'll see you in 20 days, you know. And you just want a database, right? But imagine you're enabled to write the Terraform to actually create your database, right? That little nuance there can like unlock massive, massive innovations. So when you have two teams trying to do something together,

and you have enough skillset to kind of cover all of these gaps and people with enough initiative to actually follow up on that, then you can definitely have things progress. Because sometimes it's almost not necessarily that things are complicated to achieve. It's that there'll be a blocker here because somebody just doesn't know how to do something, right? And the right people are not in the conversation. So end of the day, the answer here is probably have the right people, the right skillsets, and the right conversations. Oh, I like that.

Because you're looking at things in this dimension of time also in the maturity. And as it matures, you want to make sure that the right person is there. Yep. And I also like the idea of understanding through empathy that you called out. I know that one of my friends, Chad Sanderson, talks about this a lot and how when software engineering folks will make changes to data,

It affects so many different things downstream. And it's usually the data engineer that has to deal with that. And there's products being built by the analysts and by the ML engineers or the data scientists that are not working. And so...

they are putting pressure on the data engineer like, hey, why did this not work? Like my model is throwing all kinds of errors now. What is happening? My heart goes out to my data friends. They'll probably be watching this. Exactly. I have all the empathy in the world for them. And it comes down to what you're saying that if you can empathize with what

is happening so far downstream, just as much as an ML engineer can empathize with the software engineer for why those changes need to continuously be happening, then maybe there's some way that you can create a very low friction idea of when changes are made, they get

double checked on downstream. Yep. And you never know how much time you might save someone else by just doing that little check. Oh my God. That is such a great point. You just think about all those wasted hours because of some change that, oh yeah, this event, we don't need that anymore, do we? Yeah, let's get it out of here. Meanwhile. And sometimes it could be, so that's an example of Yagni, right? Yeah.

I ain't going to need it for people who don't know. Don't repeat yourself is also one of my favorites is just somebody trying to do the due diligence and refactoring something and saying, yo, let's, let's, let's just abstract this together. And then downstream,

team-wise, downstream time-wise, downstream code-wise, for example. All of a sudden you go, oh my God, we're coupled now and we can't do X, Y, and Z because reason, because reason, because reason, because this drying has happened. You just got into a marriage you had no business being in. Sorry, Romeo and Juliet. Your great, great, great, great grandparents were at war and that's just how things are. That's how it works. And you probably don't

don't even realize it until two hours later or five days later when you recognize because this, because this, because this, and then you track down the root cause and you go, uh-oh. As the victim, unfortunately, as the culprit, you might not be in the company anymore. That's the classic scenario is that, hey, wait a minute. Why did this? Oh, I should start looking for a job too now. Yeah.

Let's cheer up. Let's talk about something fun. Yeah, exactly. Let's talk about the good stuff too. But that is...

A great point. It's not necessarily all doom and gloom. It is talking about this so that hopefully it becomes more and more commonplace to recognize that it is good practice if you can figure out when changes are being made upstream. How can you track those changes so that downstream products are not affected by them? Absolutely. Sometimes all you need is a cheeky little test in CI to shift things left and make people aware.

Yeah, that shift left. Yeah, that's such a good point. And even the ideas on, like, it's not only traditional ML that is affected by this, because if you have this data and you're creating Gen AI products from it, you can still get into the same scenario. And I think that's why the idea of the ML ops being more important now than ever, and hopefully bringing those barriers down

is part of what we're talking about today. So this is fun, actually. You mentioned to me before we hit record that we're almost living in this post-agile, post-DevOps. Oh, yes. Why do you think that? Oh, that's...

Oh, I feel quite strongly about it because in a way, as I grew up in my career, it was all about Agile good, DevOps good, and these buzzwords and we almost religiously cult-like followed this. But this is Agile, but this is DevOps, right? And, you know, maybe sometimes it worked out, maybe it didn't. Maybe sometimes you got hit in the face with the capital Agile and you had to go work out story points.

DevOps not so much, but I think with DevOps it's a little bit more sad than painful because you kind of go, oh, DevOps is just Terraform and AWS. Cool. But what I've experienced is that people kind of deny, people see Agile as only ever existed as capital A. They've never experienced a good Agile with a small a. And there were some blog posts about Agile, dead, nice and click-baity.

But in my opinion, the spirit still lives on. Like I actually worked with one quite senior engineer one day who was very much anti buzzword across the board. But when I talked to him about agile, he just said like, you know, I don't get this agile thing. It just means go do the thing. Right. It literally described this. Go do the thing. And I'm like, you know what? Like that's, that's literally it. Right. Like don't mess around with this. Definitely don't waterfall for sure. Right. But go do the thing.

just needs a little bit of feedback on the way back. And there you go. That's all Agile is, right? Just your very frequent feedback loops, right? But when I say post-Agile, I see people saying that first and foremost, Agile is dead, but also saying we don't need any meetings, right? Just delete all meetings and like have one sync once a week and say,

How are things? They're not delivered yet? Getting delivered. Right. That's your meetings. I'm blocked on XYZ. Yeah, exactly. So in a way, it's kind of like this rejection of all these ceremonies, like no stand-ups, no...

sprint plannings sprint plannings have very little defense towards that right but sometimes it's like no to Jira and I look at them and go yeah you know I can't say I love Jira ain't nobody ain't nobody here gonna say stand up and say I love Jira I'm sorry Atlassian if you're watching this

But Jira's not about... They know. Just a little bit of a sidetrack on that. One time, one of the ML engineers that works at Atlassian came into the community and introduced themselves and said, yeah, I'm working on Jira. And then put in parentheses, sorry, everyone. So they know. Fair enough. But when I look at... My recent team begged to get Jira.

because we had nothing, right? How do you track work? How do you represent your process discipline without some sort of can banish like tracker, for example, right? You can do it in a whiteboard if you want. You can literally type it in Slack every day. Every day I did this, this is in progress, this is to do and this is completed, right? But then why not just use Jira, right? It's up to you how you configure it.

But it's almost kind of like, you know, this big reset, like, oh, no, no, no, no, no, right? We're going to go super fast because we're just not going to do all those things that people did, right? So I think it's kind of like this revival. But the heart of it, right? The heart of Agile still lives on because it's not like,

like you don't waterfall things right which is like the main other alternative so like if you don't waterfall things that just means like you do small things quickly and it's just the size of these things that varies right but at the end of the day like i think this post agileness is mostly about rejecting the capital agile the capital a agile and you think it's because there's too much friction that comes with the capital a agile absolutely so first of all was yes but it's

It's the survivors of this friction that kind of, it's almost like the Agile Apocalypse, capital A Agile Apocalypse has happened. And now we see all the mammals that survived the meteorite and they're all kind of like, there's a whole new world now, right? We're not doing Agile, right? We're not, right? Even though like, you know, life goes on, right? But I do find it quite interesting because

a lot of the concepts are reinvented just under different names. And I remember speaking to some senior engineers back in my life where they just said, like, you know, things go in cycles. You know, this was like, you know, Docker containers were a thing back in the day. They were just called something else. Like, it's just things keep going. It's very similar with DevOps. DevOps obviously is a different impact because, yeah, you know, DevOps equals AWS and Terraform. But...

DevOps could just literally equals reorganization, right? It could just say like, it's not quite right. These teams should work like this, right? Boom, DevOps, you know, like out of nowhere. Or, you know, how about we hire a platform team and not call it DevOps? That's already a step forward, right? Like this concept of platform teams, it's just the same thing. It's just different names. So it's not like concepts disappear, right? You don't go, oh, DevOps is dead. So we're not doing DevOps, you know,

But, you know, feedback loops still exist, right? And once people figure out that, oh, god damn, these feedback loops are good, right? There you go. There's your DevOps culture of feedbacking, right? So I think it's more our case of people having had enough of buzzwords and new people come in and they are not familiar with this whole

education almost back in the day like because as i said at the start like you know it was almost like you know pray to the agile gods and pray to the devops and this is the way things are and that means good at least that's how i felt right and now we have this new generation here like you know that's a devops for those boomers you know like you know that's you know we have new stuff right we just get we have vibe coding yeah vibe coding and getting shit done you know um so

I feel like I'm getting to that stage of that senior that I've spoken to so many years ago about how they said, like, you know, I've seen all this before. And slowly but surely, I'm like, oh, God damn, I'm starting to get the same feeling. Like, I don't... Like, it's obviously slightly different, but I'm kind of like, you know, like, this was a thing already, you know? And, like, you know, sometimes it's just a case of learn from history, if that makes sense. Well...

Have I got news for you on the hype train, man. You want to come into an area where there is more hype and buzzwords than any other area? Oh, yes. Oh, yes. I know just where to point you to. But at the end of the day, the idea or the concept of how do we create

really fast feedback loops. Seems like it is a huge piece or thread that you're pulling on right now. And you're saying, hey, look, I don't care if you are a full stack dev, if you're a data scientist, if you are a data engineer, how can you create as small of iteration loops or as fast feedback loops as possible to get that

in place so that you can know you're going in the right direction. Whether we call it DevOps, whether we call it Agile, whether we call it MLOps, it doesn't really matter. We can call it LLMOps if you want. That's another one of those buzzy words. But at the end of the day, what you really want to do is try and... I really like how you said it. And so I'll reiterate it. It's breaking down those communication barriers and recognizing...

how to get folks talking when they need to be talking in the right ways. Yeah, and having like important conversations earlier. Yeah, and whether you do that with Jira or you do it with ClickUp or you do it in Slack, it doesn't really matter. It's whatever works for you. And hopefully you and your team and your company have found ways to lower friction. I know there is probably...

an AI project at every company internally that is trying to figure out how can we use AI to get rid of JIRA? I can almost guarantee you that every company over 500 people has an internal team working on that. And I can just about guarantee that nobody's been successful so far. By coiling yourself into a whole new JIRA. Yeah. And you would think like you kind of...

imagine oh it can't be that hard right it's got to be can we just get a AI note taker to hang out on all of our stand-ups and can we get an AI whatever can we like install a slack bot so that it's privy to all of our slack conversations and then can we have it read Jira or ClickUp or whatever and hopefully it can just start doing what it needs to do

That is always... You think it's going to be that easy. And then you talk to people who have tried it and it's like, yeah, there's a few problems with that approach. There's always the no plan survives the first contact with enemy or getting punched in the face story. The enemy always has a say. Yeah, and...

but this world that you described prior to this problem is that like it's almost like you know if we could just use ai we could all live like the the people on the wally ship just sitting there you know barely like oh we'll just use ai to not even say our stand-ups right it'll just reach into the brain right but yeah sometimes sometimes you just have to do the work and that's what makes you stronger right like it's

over-reliance on the vibe coding again when does the vibe coding end i think that's that's a cool title to the to the talk when should you stop vibe coding yeah well it's funny because there's the product folks who are probably thinking it can be done we can get rid of jira we can get rid of this friction if we have a better approach and we try and think ai first instead of

bringing AI into Jira how can we make it so that our agile for lack of a better word approach is now going to be AI first and I was talking with my friend Floris about this on one of the podcasts that we did a month or two ago and he was saying because they tried the whole hey let's bring an LLM to

to Jira. We've got this Jira agent. It's going to not only understand the context and everything and understand where we are with projects, but once certain things get updated and checked in and the PRs get merged, they're going to just update the Kanban board itself. What they found is that

The LLMs do not understand the context on these projects because when you're a human writing for another human, you write as the or the least amount of information as possible.

So that the other human can understand because you're on the same wavelength. You've been working on this project maybe for the last two weeks or two months, who knows. And so you don't need to go and explain each little ticket and what you've done and how you've updated it.

that makes it very hard for the ai agent to understand what the hell is going on and so i asked him i said you know well what would you do differently would you try and create some kind of knowledge graph and slack so it can get all of these more tech yeah of course man like why this is a problem that for sure can just be solved by how else would you get paid yeah

You know what this project needs? More AI. That's what we need on this project. So if you throw a few more LLM calls at it, it's going to work, right? And he said, I think about this a lot. And what I would do is I would try to probably have each engineer be interviewed by a LLM note taker.

And they get asked all the questions for all the context on the project so that it can now be in the loop. And he was saying, this is something that we think about a lot is that are we trying to jam AI into an existing product or are we trying to reinvent the workflow with AI because it is now fundamentally changed?

In this AI hype, obviously everybody started doing it. I was just thinking, you can't even tie your shoelaces without AI now, can you? And to be honest, again, it's genuinely very cool, right? And even I have something like, oh, can I? So just a little hobby aside, I'm very, very deep into Warhammer and I'm like, can I just AI some Warhammer? Almost doesn't matter what it is, but it would be cool to LLM this and do that. But

At the end of the day, just kind of sit down and go, right, no, I'm just going to do the human thing. Like, you know, just bash scripts, man. Yeah. Just bash scripts. What did I just see? Someone wrote a blog post that I was reading this morning, and it was talking about how more LLM calls or more work doesn't mean better work. And just because you are...

creating the illusion of doing more by prompting something and then getting output if the output is mediocre and at the end of the day you have to do a lot of work on that output or scrap it all after giving up and saying I don't think it's going to get there then that is not progress that sounds like exactly like my project last night I was just talking about writing this parser trying to vibe code as far as I can and

And just realizing I have to stop, you know, and just do it the proper way. Yeah. And so it's, it's funny just going back to the idea of, Hey, can we use AI to help us break down friction, help us break down these communication gaps that there may be, or help us create faster iteration loops. That's,

an interesting one to think about. But the thing that it comes back to in my eyes is like you can't substitute doing the work. That's true. In a way, you can... I was just thinking, is it just the case that our brains are so much more multimodal than any LLM could possibly be? That like there's no LLMing. Like as you say, like, you know, how do we use AI to solve our organizational problems? I'm like, well, do you need to have like a little...

little pet on your shoulder that kind of watches your everyday work life and kind of goes oh there's a problem there i think you know a little clippy that suggests have you considered reorganization and bringing these teams together but first of all how the hell are you going to train it yeah like how would you find the data to actually tell her that this is this is what should be done on the other hand is yeah the multi-modality is going to be but as you said yourself sometimes you just have to do the work yourself as a human

Yeah. How are you going to give it that reward function? How are you going to create evaluation sets? Cool. Right. So I'll see you tomorrow. We're going to make a new startup. Surely going to get a lot of funding for this idea. I think we could go pretty far with this, even though it's complete vaporware and it will never work. It sounds like a really good idea. Oh, yeah.

It would sell to at least a few companies and you would have that overarching LLM that can see into all of the different channels and see when folks are getting stuck and then suggest you should talk to John from the platform team. Might work.

We never know. We're never going to know until we vibe code it. Absolutely. Yeah, that's true. Yep. We got to just get that MVP out real quick. So before we go, since I think I would be doing a disservice to not talk about like problems that you've seen in

in going to production with ML and AI. And maybe you have some confessions. We started doing these confessional stories in the ML News, ML Ops newsletter that goes out. And it's hilarious what people are writing in with. They are talking about how one time, you know, they took down the Etsy homepage because the recommender system, they just kind of like poured over some code from one of the smaller sub pages and it

exhausted the CPUs and all of this low-level APIs were just getting bombarded. I wish I had something as dramatic. Like, the ones I can think of...

especially mlops related were only as bad as oops the machine's gone let me reboot it so please use a new ip address like it's unfortunately it's very um very mundane in terms of confessions obviously very painful for researchers right yeah but not as not as epic as in pakistan

Yeah, another one that I heard was from Flavio talking about how for 18 days straight, their recommender system was recommending the same item to every single person that went on to the webpage. And they did not catch it until 18 days later. And they realized, ooh, yeah, we may have lost out on a couple hundred thousand dollars worth of products sold. Actually, I do have a chemistry one. So...

in Panda's world and all that tabular goodness, very favorable to do things columnarly. So you've got your labels and you've got your values, right? And when you're jumping across umpteen microservices here and there, you better hope that those labels and values are in the right order. And in chemistry, it's beautiful because

You have no idea if this is good or not, right? You're like this compound, like I have no idea what this compound is and this number for this property. Like, you know, how acidic is this random compound? You never, you're not even a chemist, you know, you barely know. Oh, acidity, probably, you know, something like lemons, right? So you kind of go this number kind of maybe okay, but like, you know, you never know. So imagine like, you know, I'm going to say like a million of them, right?

And then you go, please, please, please, microservice hops, do not change the order of these. Because if you do, nobody, nobody will ever know. So of course, of course, there were some instances where like, you know, through these umpteen hops, through all these microservices, pop comes out the answer for like all of these compounds, for all of these properties. And, you know, the numbers, we don't know if they're in the right order or not. And at some point, the chemists,

kind of go like, something's not right because they obviously know if this should be acidic or not. But the way for them to prove it is that they'd have to like reverse engineer the AI to kind of say like, you know, this is what they predicted should actually be. And they go like, this doesn't align. And then one of the software engineers kind of goes, oh, we've sorted it by accident when we should have been sorted.

Oh no. Yep. So like, that's definitely pretty painful as you can probably imagine. Yeah. Um, in terms of, uh, my current place, it's all this video generation and, um,

uh the ai problems themselves oh man they're it's almost kind of like you hope that something goes wrong because their results are absolutely hilarious like literally today i've seen the wrong skeleton being applied for a render and somebody just got a chad chin and it's almost like you know why don't we just ship that as a product just these random creative hallucinations yeah exactly yeah

That's the premium community that you can sell. They get access to all of the ones that you don't use. Yeah. I think one last message I do have is that we talk about MLOps and organizational practices, but software engineering best practices are just as applicable. So imagine pre-MLOps, almost kind of like pre-DevOps things, but like continuous integration, continuous delivery,

These things aren't just Jenkins, right? So a lot of the times people go, do you do CI, CD? They go, yes, we have GitHub actions. So I'm a big fan of kind of returning to thinking about what continuous delivery actually is and

That is so crucial in terms of implementing these feedback loops. This is all about the shifting left. This is all about thinking about when is the sweet spot for this automation, right? Like obviously not straight away because you have to prove things, but at the same time, if you leave it for too long,

you'll grow way past that moment. Now, this might be a whole nother podcast, so I don't want to delay things too much, but I feel like that's almost kind of like the foundation of all of this, right? Like, you know, it doesn't matter if it's ML Ops or DevOps at this point or whatever, but like these things have their place and obviously like, you know, doing them too early could be very damaging, just as much as doing them too late could be very damaging. Have you seen signs that it may be too late?

Or it may be too early. Well, it's never too late, as they say. But I've seen signs of the lack. So in a way, if you have some code that isn't tested in the production-like environment for too long, it might not even be integratable. It might even just basically say, like, we can't even ship this because the effort of actually productionizing it is way too high.

And that's how you know, okay, we need to do something here. Yes. And this is where shifting left is such a key activity here. And can you explain what shift left means for you? Well, first and foremost, what does shifting left mean to begin with? What is left and what is right? So imagine right is production. Left is your head and then hands and then keyboard and then commits and then pushes. So shifting left, hopefully you can visualize it by

some sort of thing initiative check test that is moved closer to your head you know the most immediate best shift left is like you know you talk with someone and you say let me rubber ducky an idea with you and that gives you a way to say wait a minute this was a bad idea all along or this was a good idea all along right next thing is you write some code

And you have type checking. Type checking is one of the next shift left things that can happen. Bad type, please don't do this, please do that. And I think usually what happens is that the further right you go, the more and more value derived from these checks, for example. So like at the end of the day, you go production goal, where's my money?

And either people start going, yes, this is a great product or you go, oops, it didn't work out. Right. So this is like the rightest, the most right check you can possibly have. Right. So the idea is kind of like shift as many of these checks as to the left as possible. But obviously like some of them you have to pay in cost of times. Like the best, the most practical examples is integration tests. Right. There's this nice seesaw of how much do you test versus how much time it takes.

You obviously don't want to implement the most end-to-end test possible, which takes three hours. And then every time you want to push the smallest change, it's a three-hour CI job. And you kind of go, well, at least I know it works, right? So you have to be very careful in terms of balance of value versus time. And you can extrapolate that. It's not just integration tests. It's either conversations or it's...

anything further down the right. Let me try and think. Sorry, I left my head. But like, or for example, some sort of scientific check, you know, you can say like, you know, this model produces something that's not good. You can't quite always check the NCI, for example. It's interesting that you talk about that because I've seen it relayed as the closer you get to production, the more expensive it is to catch a problem. Yes. And the most expensive is

is when it's in production. And as you start going left, it gets less and less expensive if you catch that problem. If you catch it right at its source, that's great. You can quickly iterate and hopefully fix it.

If you catch it downstream, then it's going to be a little bit more expensive. Sometimes things are not as easy to catch as well. So there's that dimension as well. Of course, if you could catch the fact that nobody's going to buy your product and you've just spent like $10 billion in AWS bills, if you could write a unit test for that, that would be some unit test. That is the product that we need to create. We need to vibe code that product into existence because...

I can tell you what, if we could unit test whether or not someone is going to buy your product and sell that as our magic snake oil, woof, talk about a billion dollar company. Oh, yes.

Yeah. And it's cool too that you're talking about shifting left because that idea, I've seen it gain more popularity. I'm actually going to be hosting the Shift Left Data Conference. And so it's bringing the idea and the fundamentals of shifting left, but to your data pipelines and your data CICD and bringing

whether or not your data is, it has analytics built on top of it or it has ML built on top of it. That same idea of can we catch problems at their source, but also how do we balance one which is, I don't want to have to have three hour CICD check every time I make the smallest change versus I don't want to just not have anything and be willy nilly and kind of pray every time I put something out there. It depends. Yeah.

Yeah. Well, what in your eyes would be different when it comes to shifting left for code versus shifting left for data? Oh boy. Oh boy. Once again, my heart goes out to my data friends. So recently enough, I had to do quite a lot in terms of setting up an entire data platform. And that just...

gave me so much respect for that area the best thing i can say is that it's a lot harder right a lot harder because code is stateless first and foremost data is state and um that in itself is already super hard like it's i i kind of struggle to explain that from first principles but please trust me data harder than code um so because of that i think like

how do you test data? Well, you can have data validation and integrity tests and all that kind of stuff. But implementing that and running it and operating that, like, you know, that is so much harder than, for example, testing if, yo, my inference works. That's great, right? Yeah. And also, like, the sheer amount of possible knock-on effects, because if you're going to dive into data, like, as a holistic view, to kind of, like, take it from bird's-eye view, right?

if you're going to start thinking about, you know, here's my source, this is the entry, right? And this once again comes down to how much value do you want to derive from your test is like how far down the stream of all the possible ways your data can go, are you going to test things and the nature of your data? Like, you know, a lot of the times we speak about data, we implicitly assume tabular, right? My current, in my current place, I don't have such luxury. There is tabular metadata for sure.

But boy, oh boy, the majority of the actual data is media. So testing that is even harder because you can imagine like working with media is not the same as, you know, having something that you can fit a lot of in the container. So I'm sorry to give you like a very hand wavy answer, but like shifting left in terms of data just seems so much more harder than in code. In code it's,

I don't want to say easy, absolutely not, but it's a lot more easier to visualize because you can kind of say this is right or this is not right. You know, your unit tests pass because two plus two does not equal five and you kind of see red unit tests, right? And you can kind of extrapolate and extrapolate and extrapolate, right? With data, it's just so many different cauliflower level fractal problems that you can expand to.

that I'm finding it hard to visualize. Sorry. Maybe I just suck at data. Maybe that's... No, it's also true that with code, it's a much more mature field. And these ideas have been iterated on for years. Sorry, I've got a guest. Yeah, what a tootie. For anybody that's not watching, we have a dog that just jumped on screen. A really good looking dog. So...

Data is 100%. I forgot where I was going with this, but yeah. One thing is for sure. Yeah, the dog. I'm looking at the dog so chill. So one thing is for sure, though, data has had the idea of shifting left for much longer. So it's almost like this mature idea, or it's much more mature than the idea of shifting left. Did I say data? I meant code. Code, yeah, yeah. I assumed that, yeah.

so one thing is for sure code shifting left is a much more mature idea because it's been around for so long and data shifting left is a newer idea and so it almost is it needs those iterations and it needs diverse minds to come at it and try and attack the problem from different ways so that it also can become a bit more mature and in my own experience i definitely see that like when you just

try and tackle that from a code perspective like that's when things just stop lining up quite as neatly because it's almost like a different mindset you have to have as well oh that's good yeah that's a very interesting one so