Kubernetes at Google with Ben Elder
42:53 Share

Containers make it possible to standardize the deployment of software to any compute environment. However, managing and orchestrating containers at scale is a major challenge. Kubernetes was originally created by Google and solves the problem of scaling container deployment.

Ben Elder is a senior software engineer at Google and is an elected member of the Kubernetes Steering Committee. Ben joins the show to talk about why Kubernetes became the standard for container orchestration, Kubernetes control theory, how he runs his home infrastructure, and more. This episode is hosted by Lee Acheson. Lee Acheson is a software architect, author,

author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale, is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business, produced for people looking to build and grow their digital business. Listen at mdb.fm.

Follow Lee at softwarearchitectureinsights.com and see all his content at leeatcheson.com. Ben Elder is a senior software engineer at Google and an elected member of the Kubernetes Steering Committee, and he is my guest today. Ben, welcome to Software Engineering Daily.

Thank you. It's great to have you here. And, you know, Kubernetes has really taken over the world of container orchestration, but it really isn't the only option. Why has Kubernetes done so well compared to things like Docker Swarm or AWS ECS or any of the other container orchestration services out there?

You know, I think it's a complicated topic. I think some of them, it's because we have an open portable standard across providers. Some of it's about community building and some of it's about the design and approach to it. The way that Kubernetes is a distributed system where you declare the state that you want and controllers reconcile towards what you want builds a resilient approach to doing these things. But I think it's also about the community building that people focus on just so many aspects.

You know, I think that's a great point. When we talk a little bit later on, I want to talk about the APIs and the mechanism that uses for APIs and how all that works. But let's talk a little bit about the community aspect first. What is it that keeps Kubernetes going today? Is it the community? Is it Google? Is it a combination of both? What is it that's driving the adoption of Kubernetes?

Well, you know, I think that some of these things do come down to the technology to build something useful and people will come, but there is a lot to the community building and that includes people from the CNCF, from Google, from other vendors and independent folks participating in the community. I know within the Kubernetes project,

We have whole groups dedicated to aspects of community, like the contributor crumbs group that works on sending out communications through social media platforms about what we're working on, or the docs team that works on the blog that we host ourselves within the project.

There's a lot of work that goes into that sort of thing. We host our own discussion forums and we participate in others. It's really important to have all that because it's about ecosystem. I think all of the technology and so on aside, the biggest thing that Kubernetes has at this point is the whole ecosystem around it. And that comes from community. Yeah, and that ecosystem is...

Not only, you know, plugins and things like that, but it's also, you know, companies that have built their entire corporate strategy around supporting people who are using Kubernetes environments. And so there's a lot of support from third-party companies with products and offerings that are compatible with Kubernetes.

Yeah, and that's a good point. I mean, one of the things that I've been close to that we've worked on has been the conformance program that's about making sure that, you know, we actually do have compatible offerings from so many vendors and that they meet a certain expectation of standardization. Well, when I think of, you know, some of the greatest foundational value that comes from Kubernetes and what is it that Kubernetes provides, and, you know, it's kind of a long and kind of a complex list, but one of the big things that

overriding things, if you will, that I think of when I think of the value of Kubernetes is it's really an enabler of infrastructure as code for infrastructure management. Now, you know, the cloud certainly was a big part of that and Kubernetes is an independent dimension from the cloud, but Kubernetes on its own provides the ability to build infrastructure as a code offerings that can help companies, whether they're in the cloud or not,

achieve code-based infrastructure. Is that a valid statement or...

Do you agree with that statement that that's a foundational aspect? Yeah, I think that's totally fair. I think that, you know, one of the things it does is ensure that you have a set of common APIs available that you can use for core infrastructure concepts like your compute workflows, aspects of networking and storage that you can target, be it bare metal or some on-premise solution all the way out to all the different cloud providers and pretty much anywhere that you can get compute storage and network infrastructure.

It's more than, you know, the ability to get it. It's the ability to manage and control it. And the real value of IAC is the manageability of the infrastructure that you're building and the consistency of the infrastructure you're building and the, you know, ability to treat it like you would any other piece of code that's managed and deployed and viewed in a standard way. And that's something that Kubernetes supports and actually encourages, I should say. Is that correct?

Yeah, I mean, I guess what I was getting at was having the APIs gives you room to build IAC, but it certainly has it in mind from the beginning that, you know, that you're going to even a more particular flavor. Kubernetes really pushes the declarative approach to infrastructure config. And that's kind of interesting because, you know, you don't have to use that. You can use

imperative code to emit declarative, but it's much harder to go the other way. And it creates room to build this IAC ecosystem around, okay, you're going to have these APIs to do IAC for

on-prem and cloud and so on. The whole system works that way and everything that we run for ourselves kind of meta in the project, which is one of the things I participate in, uses pretty much every form of IAC you can imagine from bash to terraform or Kubernetes YAML, depending on the scope of what we're running. You mentioned declarative versus imperative. And I know

Many of our listeners understand the differences between those, but do you want to talk a little bit about that and about what the difference is and why declarative is better?

Well, I should say, I think declarative, you know, it fits for this use case. There's plenty of reasons you might want to do imperative. But so declarative versus imperative with declarative state, you're spelling out as data as config. This is what I want the world to look like. And with imperative, you're like writing code, trying to drive things towards it yourself. I think the really powerful thing about declarative is that

It allows us to centralize that controller logic to take sort of a control theory approach to we are going to get things to the state that you desire. And we're going to implement that with all of its quirks once as a common controller for something like Kubernetes services or the kubelet that runs pods. And then all of these other tools can just tell Kubernetes this is the state we want.

And those controllers can continue to drive it. It's an eventually consistent system with large complex environments. If you try to do everything sequentially and imperatively, you can run into problems with

ordering on dependencies between things or transient flakiness, by moving that sort of logic out into a daemon, a controller that continues to drive towards that state, you'll eventually get to what you need with the resiliency that comes with that. I think that's a really powerful way for running infrastructure, and it's much harder to get right with a purely imperative approach. Yeah. Arguably, I think you mentioned this, it's

Using imperative techniques, it's almost impossible to get a declarative system. It's a little bit easier the other way around, but it's almost impossible using just a declarative style to truly get what you're trying to get towards. While it's so much simpler to say, these are the set of systems I want, this is what I want it to look like when it's done, and let the system figure out the optimal way to get there.

Right. And it allows us to hand that off to like common upstream APIs and in many cases to common central implementations that people can work on together to get to a really reliable state where we continue to squash bugs and handle quirks of these systems. So how does Kubernetes work with

other, you know, IAC tools or the IAC infrastructures like Terraform, et cetera. How does it work with them to create a system as a whole? They're not competitive. They're tools that work together. Is that correct? Yeah, absolutely. As I said, within the project ourselves, we use a combination of all these tools, which

whatever seems effective for what we're trying to do. For example, if you're trying to bring up a cluster itself, you know, you might want to use bash or terraform or something, but once you have the cluster up, maybe you can do it with pure Kubernetes YAML. Maybe you want to have it combined with the rest of your stack. There are terraform providers or other tools. We have a whole ecosystem of,

of tools even built since Kubernetes was created that let you do this the other way and start from Kubernetes style APIs and move them out to all the rest of your application. We call it KRM, Kubernetes Resource Management for all of your external things. And Kubernetes has built a platform where if you want to extend it to support all these external

APIs and infrastructure. We have common tooling for that and you can build APIs and Kubernetes, but we also see people going the other way and building tools, you know, using tools like Terraform that were already existing, that they were already targeting all of these external things with and adding Kubernetes to it. There's a really, really rich API ecosystem

and tooling ecosystem in both direction. And it's a combination of Kubernetes itself providing good APIs, Kubernetes providing extension mechanisms for people to build their own APIs.

and just the ecosystem around it. And that's really the difference between Kubernetes and, for instance, ECS, just throwing out one of the examples. It truly is an open environment that has community support and lots of third-party integrations that work with it and make it work better. Well, and it's something that you can...

Depending on exactly what you're trying to do, you may be able to define a common way to run some application and have it just work on cloud, on-prem, on another cloud because of that neutral API that isn't specific to one vendor. Are there classes of applications, size or type or whatever, where you think that Kubernetes is the wrong answer for and another tool is better?

Oh, yeah, I've actually written about this before. You know, when I was newer to these things, I was excited to run Kubernetes everywhere for everything. And my home environment was all Kubernetes and super complicated. And I've since realized that, well, I'm not running very many workloads at home. They're really simple things. And they run just fine as services on one appropriately sized box. I'm not that worried about availability. This isn't a business. And it's not worth my time to overcomplicate it.

If you're running a very small amount of things and you know how to run them well, you might be able... I mean, you can get a surprisingly large VM...

I think within this community, probably a fair number of folks have heard about how Stack Overflow runs things and how they just have a handful of big machines that they run things on. If you're doing that and you don't have too many services, you might not find that you need this whole complex system. I think where Kubernetes really shines is where you want to leverage the ecosystem and its tooling, or if you're running lots of workloads or a lot of services that are talking to each other and that sort of thing. But if you've got like

one service, you might not use it. I mean, the Kubernetes project itself, we haven't quite gone to the point where literally everything we run is Kubernetes all the way down. Because, you know, same thing, it's operated by volunteers, and we don't have the resources to run an ops team. So

You know, I think where Kubernetes also really shines for larger organizations, and we might get to this with some other discussion later, it's a little bit of a tangent, but Kubernetes lets you sort of build your platform. And if you're not at the point where you need a whole platform, if you've got like a few small services, it's probably overkill.

Is there a sweet spot? Yeah, I think once you're starting to expand to some handful of services and you're starting to think about how you need to manage them as IAC, then you want to be there. If you're running one or two things and you think you can get it fine, just kind of

running them by hand, you don't need. So like my home lab, you know, I've just got a couple of Debian packages on a Debian box and they run a system B units and I almost never touch it. And it's just not running anything big, you know, or if you can get a managed service that solves your needs, it may be a good fit. You know, we use some, again, we use some of those for running Kubernetes because they're just, they're just the right fit. Like because of the scale, the amount of people we have in the, in the sort of workloads we're running, if we've got,

blob storage. We're not going to spin up a blob storage implementation when it's just easier for us to use one off the shelf. And it's not that important to us because there's already something that has a de facto standard and we can swap vendors and so on as we need. But

But when you're really actually starting to run lots of things and you're starting to think about, okay, how are we rolling things out? How are we running things together? How are we managing our infrastructure that isn't just one box in a corner? Then you want to start thinking about it. But if you're a startup and you're just running on one machine and that's working fine for you, you probably don't need the overhead yet and you'll be fine. But when you start thinking about how you're going to do op

and automation and you want to leverage all those IAC tools or you start to scale up to more services, then I think you're going to start to find Kubernetes valuable. So is Kubernetes by itself more of a straightforward infrastructure play or is it more of a platform? And I know that's kind of a loaded question because it depends on the definition. So feel free to define those terms however you feel appropriate.

I think a lot of these things are relative to part of the stack that you work with. But I'm not sure who originally said it, but someone smarter than me once said that Kubernetes is a platform for building platforms. And I really like that way of putting it. I think Kubernetes itself doesn't give you a full stereotypical platform as a service expectation. It isn't quite...

quite that level of opinionated and batteries included for, you know, point your repo at it and you're good. You're going to do more of that. But what it does give you is a portable set of APIs for managing your infrastructure. When you want a platform as a service, you want a little bit more than that. You want...

more opinions around how you're going to do things like application rollouts. And Kubernetes provides a lot of the like base needs for that, but the ecosystem around it or what you choose to build yourself in some cases is what will provide the rest that gets you all the way to a full like pass style thing. So I think Kubernetes allows us to build, build your own paths anywhere and,

Yeah, so it's plain in the PaaS area, but it itself is more of an infrastructure as a service. Right. It itself, you know, provides all the foundational APIs you need for I'm going to run workloads. I need to have network for them. I need to have storage. And this is how the workloads need to scale across nodes and things like that. But when you start talking about things like blue-green deploys or something like that, then that's going to be your...

toolkit of choice from the ecosystem that's building on top. Maybe in some cases, companies are going to build something themselves. In some cases, they're going to look to other CNCF projects or ecosystem tools, you know, projects like Knative. So, you know, let's start talking about the Cloud Native Compute Foundation. We've mentioned that briefly. You know, the...

Google played a major portion in the starting and the creation of the Cloud Native Computing Foundation. And, you know, Kubernetes obviously was, you know, the foundational product or foundational technology that was part of that. Do you want to talk a little bit about how the Cloud Native Computing Foundation started and how Kubernetes fits into all of that?

Yeah, so I actually kind of missed this time, but obviously I've been involved for a while. I've talked to a lot of people about what happened here. To give a little bit of context, just to frame where I'm coming from, I worked on Kubernetes as a summer of code project the first time Google has a program where they have students...

work on open source projects over the summer under a mentor. And usually that's just open source projects that Google wants to support and the mentors are coming from somewhere else. But with Kubernetes being early, it wound up being a project heavily backed by Google and actually worked with a Googler on it.

And then, you know, I finished school and I've since been working on Kubernetes in some form full time. But that time window where the CNCF happened, I was back in school, I've come back. So my retrospective on this then is that the CNCF people involved in Kubernetes are

Kubernetes early on, they wanted to make sure that it got moved out to a vendor-neutral foundation. When I first contributed, it was actually under the Google Cloud Platform GitHub org, and that wasn't the right place for underscoring. This is about APIs that are

independent of any particular vendor. So they were looking for a home for it and folks didn't feel like any of the existing groups were quite the right fit and wanted a space to host it that was centered around what would become this space and this ecosystem. And so the CNCF provides space for take what you will of the...

I think people have different definitions, but to me, they're about, in a way, the space around Kubernetes. It's about having your applications built like they're going to run on cloud and scale and so on, but not necessarily actually on cloud. You know, it might be your on-premise, but it's about that approach to how you run things and the whole ecosystem of open source tools there.

I find it interesting that the cloud-native term, if you ask anyone what it means to be cloud-native, and especially if you ask through CNCF, the definitions you get never include running in the cloud. They always include methodologies and philosophies and structures and techniques, and it doesn't necessarily talk directly about being in the cloud. Right. I think it's more about approaches that...

came out from the sort of hyperscaling that you see in cloud. But for a lot of users, they're not necessarily actually going to hyperscale and they're not necessarily leveraging cloud vendors. But we can share the approaches to how you run applications in this very API and driven way and in this way.

way where you're not thinking about individual machines as much as possible. Another way that I've heard people put it for Kubernetes itself is that it's sort of like the operating system for the data center. And I think that's another good way of putting it. Cloud Native is sort of about

thinking about things in that way. And they don't necessarily involve, you know, a lot of it does, but just the way that you approach things to that, you should take the approaches that were developed against cloud, but really they're useful all the way down to in an individual edge installation to your small or large on-prem and up to cloud.

Yeah. I always talk about the dynamic infrastructure as the API-driven infrastructure. And that's really what we got from the cloud was that. But like you say, it doesn't take a cloud to make that happen. Well, and a cloud is someone's data center, right? Yeah, exactly. So maybe you're sort of running a small one for yourself. Yeah. Other people ask me, what's the difference between private and public cloud? And I said,

The broadness of the applications running on it and the broad needs of the applications running on it is the only difference. And that is important when it comes to, you know, availability, instant availability of resources and things like that. You know, it's got value in those areas. But other than that, the actual mechanisms are exactly the same. It's no different.

Yeah. And, you know, in some cases we see people, you know, it's not necessarily dichotomy. We see some people that are using both. One of the most memorable demos at KubeCon, the CNCF conference for me, involved CERN showing running some huge scale data compute with the Large Hadron Collider data from their data centers that they operate and then expanding out and spinning up temporarily a huge amount of collider data.

cloud resources running the exact same workload. Yeah. So what are some of the alternatives to what the CNCF does? The CNCF has a very specific philosophy of how you do things. What other alternatives are there? And again, why is CNCF, why is the cloud native model so preferred or so popular? Choose whatever adjective you want there. You know, I think

It kind of depends on what you're actually trying to do, but there's obviously lots of vendor-provided solutions. And I think there's been various other projects that are... It's not like Kubernetes is the first thing anybody's ever built for running infrastructure that runs on cloud and on-prem. So there's a lot of projects that exist. I probably haven't even encountered all of them. I think that

Like I said, it's very complicated how we got here. But I think if you look at it today, the CNCF and the Kubernetes space gives you a really energetic standard ecosystem to build around that you can use. But for certain things, you

you might find that the scope of what you're doing is small and there is some maybe more niche solution that targets your needs or maybe you're still happy running something that you were using before all this or, you know, I never like to try to tell people like, oh, this is a silver bullet and everybody needs to use this tool for this thing. And I think that one of the really awesome things about open source is that we have such a low barrier to entry to try out these things. You can spin up

something on your laptop and try it out. Don't have to pay anybody anything and see if it looks like a fit for you. And if it's not, then I mean, hopefully we get some feedback from you about what it is that we're not solving well enough yet. But we know that we're not solving everything perfect for everyone. So like I said, even Kubernetes itself is

when it comes down to it, sometimes we have trade-offs between the amount of resources we have from people who are going to run applications that the project itself uses. And it just makes more sense to use a managed offering. And sometimes Kubernetes has tools to help build

build around those common things too. Like I believe, you know, there's open source projects out there for things like I want an S3 compatible API and I'm going to run it myself. And then now you can think about like that itself being some common API between on-prem and cloud, but that didn't have anything to do with us and was built out by Amazon in that case. Right.

So you used a term that, you know, Kubernetes is often called the OS for the data center. And I've heard that too. And that makes a lot of sense. I've also heard it being called the, you know, the standard API for application infrastructure, right?

There's value that comes from Kubernetes from the standpoint of cloud agnostic, right? You can build applications and the infrastructure that works with applications by having less knowledge about the specific infrastructure you're running on, whether it's on-premise, whether it's cloud, whether it's whatever. And that's really helped.

promote the ability to make applications, you know, hybrid cloud or poly cloud or any of the other different models available there has made that feasible and viable. Is that a fair statement? And what would you add to that? I think that's fair. I would probably point out that, you know, anytime somebody says, oh, this is the standard. Well, of course, someone heard that and just started a new standard. But I think it's

As I've been saying, I think it's useful that it has become such a big standard, even if it isn't perfect for you. It's a place where you're going to find offerings with pretty much any infrastructure vendor one way or another, and you're going to find integrations with all these things. And we have a place where people can come together and help us evolve the APIs.

And there's room to, you know, one of the interesting things about Kubernetes is it's not just itself the standard API. It's also not just a platform for ruling platforms and a set of standard APIs. It's also a set of ways to build extension APIs. A good chunk of the ecosystem is projects like Istio, where they're creating more APIs

using Kubernetes extension mechanisms to create standards for things like, here's how you can do service mesh across clusters and vendors. So Kubernetes itself is also guilty of creating a space for more API standards to build. And I think that's useful, but having one that's really popular and has a lot of tooling around it means that, for example, maybe you actually really know a lot about the cloud you're running on, and yet you still might want to use

Kubernetes, even aside of any of the other pros and cons, just because of all of the things that people have already built for you around it. Yeah, you're right. There's value to using a platform that's popular, even that's not a perfect fit for what you're doing because of the ecosystem that it provides. That's what you're saying. Yeah.

Yeah. And I mean, it's really hard to be a perfect fit for everyone, but being a good starting place for everyone and having room for people to customize what they need or to come back to the project and say, hey, this isn't working because I'm missing this functionality and getting that signal from everyone and then kind of building out. OK, now we have a standard API that covers a bit more. The momentum is really useful, I think. And that's kind of the power of open source in general.

So now you're a member of the steering committee for Kubernetes, but you're also a senior engineer at Google. How much involvement does Google have in the day-to-day operation of Kubernetes and CNCF in general? And how much of it is very much a hands-off? I'm talking about how much outside of Google actually goes into the standards. Well, yeah. So at the CNCF level itself, there's a lot of...

and folks that have representation in the meetings for running that, but it's its own independent foundation under the Linux Foundation with their own leadership and things. And so they, you

They obviously take input from all of the many vendors that are funding them, not just Google, but they're still running their own foundation with their own governance for things like we have the technical oversight committee at the CNCF level with representatives from all over who are determining when a project makes sense for inclusion into the organization. And then as for Kubernetes itself, I mean, again, Google is a major contributor. I've gotten to spend almost...

six and a half years now contributing to the project in one way or another as an engineer at Google. But it's a multi-company project and there are people from all over. And another important thing is that they're not all just like big company who has people working on it. There are a lot of engineers

of independent contributors, people who are interested and want to participate in open source, people who do contracting in the space and are contributing their feedback and working on the project. So it's a really big space and a really big project, and there's a lot of contributors from all over. So let's talk about modern applications in general, okay? You know, what's next in

in the development of modern app infrastructures beyond Kubernetes? What's the next big thing that's coming up? Or that's already there. You know, it's always hard to predict the future, but I think we're going to continue to see more work and more projects and more ideas around how we...

provide the open toolkits for extending to other APIs, you know, projects like a crossplane or for projects for how you kind of build your paths on top. You know, a lot of people don't really want to use Kubernetes. It's not quite the right approach.

level for them. And it can be frustrating to be thrown at, you know, using these APIs that are like a lower level than you're thinking about. And so I think there'll continue to be a lot of work on how can we build out projects like native, can native that provide a bit higher level abstractions for running your workloads. Also having just come from KubeCon, I mean, there's a lot of money, excitement, energy around all of the

AIML workloads people are trying to run. And, you know, it's not something you can ignore. I think the project has already been evolving in a direction to expand and improve APIs for things like integrating accelerators, things like graphics cards for your workloads. I think we're going to see a lot more around that space probably in

you know, projects for serving inference workloads is going to be a really popular one. So when I, you know, the people I talk to and the problems they worry about in application modernization, um,

It seems like there's four main things that people talk about. They talk about security, they talk about AI, they talk about data and the complexities of data management, and they talk about scaling. Now, Kubernetes really is a tool that helps a lot with the scaling aspect of things.

Either what does it do or more generally, how does the CNCF help fit with these other three things? Security talks a little bit about AI, but data management. Yeah. So I think the compute part is something that we've really got in Kubernetes and the

networking storage and accelerators have had and security controls have had a lot of API evolution and things. And some of these are like sub projects when they org or external projects, things like gateway API for building the future of how you manage networking. But for the AIML, for the security, for the storage,

Some of those things wind up being part of the sort of Kubernetes umbrella, and some of them wind up being under the CNCF, and they're all being supported. There's people working on various approaches to standardizing storage, and we have a bunch of groups doing security work. The Kubernetes project itself has a

special interest group for security and also the security response committee. So we have things like a bug bounty funded through the CNCF and their partners for the project itself. But we also have reports about where there are foot guns and talks about how we can improve that and what folks should do. For AIML, I think, you know, we've had some projects in the CNCF space that

the broader ecosystem like Kubeflow. But I think that if we go and look at the technical oversight committee's backlog right now, there's probably at least a few projects looking to join in that space and there'll be more to come there. Right. Yeah. I fully expect that to be a growth area for CNCF as in short to medium term anyway. Yeah.

So AI will be big. What else will be big for CNCF besides those things I mentioned? I mean, I do think that some of the area you talked on, like the storage, the networking and computer things that were still evolving. We just had gateway API go GA and it's sort of

It covers a lot of the things that, you know, Kubernetes already had APIs for to some extent, but people were looking for how we can we do better for folks who are still listening. Hopefully that have used Kubernetes, they'll know that the Ingress API in Kubernetes is a bit underspecified and it doesn't,

pan out as portably as you would actually hope. And so we've had a lot of energy behind, okay, we need to have common networking APIs that cover more of this and are better standardized. And there's been a lot of work on that. Storage, that's been getting more mature for a while now with the transition towards

CSI and having a standard way to implement drivers and storage interfaces. And to cover more than what, again, more than what Kubernetes had built in, the storage APIs we had were pretty much like, here's how you can connect up having some storage. But operations like snapshots for backups just weren't in scope. But with these newer APIs that we've been evolving, like CSI does have operations.

does have managing those sorts of things built in and sort of expanding the standardization and API coverage. And I think there's going to be a lot of discussions about how we continue to do that and leverage Kubernetes extension mechanisms to build more common APIs.

So that's where we're going or where it looks like we're going. Where do you want to see it go? You personally now, not you Google or you Kubernetes or you CNCF, but you. Well, one of the things I've been really focused on is sustainability. I think one of the things is, you know, as a project matures and is more widely accepted and the core has matured,

most of what people are looking for and things are being expanded out of the core. It can get more boring to folks and it gets a little bit trickier to staff and fund. And when you're coming down a little bit from the peak of really fast development in the core project, you know, making sure that we adapt to that and that things keep moving forward at

the right balance between keeping the system stable and reliable for people without big surprises or changes while still moving forward. I mean, security is one of those tricky ones where if we had kind of a bad default from a security perspective, we also have a commitment to stability. I think there's going to be more conversations about

how we can continue to move forward on these things. I want to see a sustainable space. I also want to see more of...

how we can look at all this work people are doing for all these extension APIs and things, and how we can bring some of it back and say, you know, this should be a standard API. I talked a little bit earlier about we have this conformance program. And so if you want to be a certified conformant vendor, you have to run some tests that we maintain and make sure that your cluster matches all the expected behavior and has all these APIs aligned.

But that only applies for core built-in APIs. I'm interested in what efforts we can do to expand sort of the common expectations anyhow, as we've continued to iterate on things like storage and network APIs. I'm also excited about some of the work we're doing that seems a little more boring, but it's trying to take some of the rough edges off. A couple of my teammates have been contributing on an effort

to use a language called cell to, uh,

configure validation rules for your APIs to take away for a lot of users the foot gun of running a blocking webhook on your API where you have the choice between the webhook doesn't block and so things that are invalid can get in or you can configure it blocking. But if you have a bug in that webhook and it goes down, then you take down your whole API because it's waiting for the response from that webhook. Instead, giving people a safer, more scoped way

built-in way to specify simple rules about what is valid. I think there's a lot of work like that that doesn't sound as exciting, but will just help take away some of the rough edges and make things easier to use. And I hope that a lot more of that continues. I hear people about how hard...

it can be to run and use it depending on what you're trying to do. And I want to keep seeing us make that easier to do. Yeah. You, like you say, arguably if you're building a large complex application built with thousands of containers, there's nothing paired Kubernetes to make it work. But if you have a small container,

home system, small, very small number of containers involved, Kubernetes is overkill. To me, it would be great to make Kubernetes less and less an overkill for the smaller environments. And that sounds like one of the things you're talking about. Yeah. I mean, in some ways, being able to...

have common apis for all these things people are looking for and build an extensible system means that it's it's always going to be complicated but there's ways that we can take some of the sharp edges off or we can improve upgrades or just make it easier for people to use and a lot of what's happening right now i think is people are looking to differentiate in the space but i want to

help continue to drive that we also have to drive forward the core and, and make sure that some of these improvements and better ways of doing things, make them back into the project and are available to everyone. Yeah.

Well, thank you. This has been a great conversation, Ben. I very much appreciate your time and energy on this. Thanks for having me. I mean, this is a topic I'm really passionate about. Like I said, I started working on this as a college student just contributing to open source. And I've come back and I've spent all my time around this. The community is really important to me and I've really enjoyed getting to work in this space. I think it's something special. Okay.

Great. We look forward to hearing and talking to you more, and maybe you'll come back again someday. Ben Elder is a senior engineer at Google working on Kubernetes, and he's also an elected member of the Kubernetes Steering Committee. And he's been my guest today. Ben, thank you for joining me on Software Engineering Daily. Thank you.