#111 - EventCatalog Revolutionizes Governance in Event-Driven Architectures | ft. David Boyne
51:00 Share

Hi, welcome back to another episode of Real World Serverless. Today, we're joined by David Boyne, who is the developer advocate at the Winglang and also the creator of Event Catalog. Hey, man, good to have you here. Hey, Ian. Yeah, thanks a lot for inviting me. Yeah, we caught up recently, actually. So yeah, great to be here. I've always been a fan of this show, watch the stuff. So it's awesome actually to be on here and just talking to you. So pretty

Appreciate that. Excellent. Yeah, it was really nice to see you in Manchester and for the Manchester Compsom, well, AWS Compsom in Manchester. It was good to hear your thoughts around the event-driven architecture as well because I know that's been your bread and butter for quite some time when you were at AWS and maybe even before that as well. Yeah, that's right. Yeah, I just naturally kind of fell into the whole thing, to be honest. I was actually, before all of that, I was a customer there at AWS and then I

building just so happened to be building like you know sns sqs all this lovely stuff and then event bridge come out and i kind of fell in love with it and hit the rest is history i just ended up working working for aws and advocating this stuff but i just yeah i love uh love event driven architectures so um and i've just been double downing on that since just naturally i don't yeah

I'd love to learn a bit more and also to get you to talk about the event catalog and maybe your vision for where that's going to go. But I guess to set the background for why the event exists in the first place, can you talk about some of the common challenges that you've seen people face when it comes to event driven architectures? Sure.

How long have we got? No, but there's a lot. There's a lot. There's a lot from like a, I don't know, a business point of view, a technical point of view, and all sorts of things. I remember when I first started, you know, just cracking on, right, and just implementing stuff. So I remember when I first started, EventBridge literally just come out, and how easy is it to raise an event on an event bus, right, especially these days, right?

And that barrier to entry is reducing more and more. So one problem I often see, and I've had it to myself, is like this implementation first mindset. We just jump in and we start raising events in our architecture. We have customers making orders, we have people subscribing, all this stuff. Let's just raise events and handle this over here.

And without necessarily thinking about the consequences of any of that, it's just like, yeah, we can have a Lambda function or we can do something over here. So, you know, I think...

There was a lot. There was a lot in this. There's complexity. There's all the way from design all the way down to implementation. And through that, you got things like, you know, duplicating of events, idempotency, all this kind of stuff. And then as we go, you know, at a higher level, it's like, well, what about event design itself? What about schema evolution strategies?

and standards and governance. I mean, governance, we'll probably talk about that later, but that's where event catalog kind of sits. But it's a huge, huge problem in this area.

but yeah i don't know if there's any of them you want to start at um otherwise i can just talk to keep talking sure maybe let's let's look at the the whole development workflow starting from the point of okay you're thinking about building event architecture uh and what are some of the first things you got to think about i think like you said a lot of people just go straight into it and start sending events and only to realize later uh well we don't really have a way to kind of

the reason about this, the schema, making sure that people don't break other downstream systems with contract changes. How do we version them? How do we capture information about the events so that we have some way to, I guess, to replay them, to catalog them, to visualize them in some way, things like needing to know the

Something even simpler, what do we use as event ID? Do we have some kind of a unified format or do everyone just kind of use whatever source event ID you get from say, from EventBridge? And what happens when we leave EventBridge and go somewhere else? What do we do then? And also how do we map events, contracts from different services and domain design and all of these other things? Let me start from the very beginning. Where do you think...

What are some of the first things you really should get right? I think you talked about your schema and some of the versioning and stuff like that. I think, yeah, I think start at the top. I think the easiest thing to do is get started, as I said, especially these days. But I just highly recommend the people I speak to is just actually stop and think for a minute. With a friend of an architect, as you know, we've got distributed systems. We've got messages that are communicating across different boundaries and systems and

And you've got to really kind of think about where are these boundaries? And you mentioned domain-driven design there too. And there's this huge correlation between, I think, between domain-driven design and event architectures. Because where are the boundaries of our systems? And if we use messages, not necessarily events, but messages to communicate between these systems,

what do these messages look like who owns the messages and the downstream consumers and things like that um and there are a lot of there are a lot of things out there to help one a couple of things that i recommend looking at or or starting out is like event storming or event modeling uh very popular things but you can actually talk to business owners talk to people which is you know funny as as techies talking to people uh but actually talking to um

and actually identifying some events in your architecture. Because events themselves, there's different levels of events. We could have a low-level private implementation event. Maybe we're on the same team and we're in the same boundary. We're creating an order system so we can create an order, a placed event or whatever.

But that's completely different to events that we use across boundaries, across these systems that I was just talking about. But as you as a developer, or the audience listening to this, I assume, there's only so many events you're going to think about, but there's also business events, like domain events too. And I think it's good to start off just by creating a holistic picture of what is our system actually doing?

We have 10 microservices, 20.

Things with event architectures at the start, you might think this isn't a problem. We only have me and Jan on the team and we're a startup or whatever. But, you know, you hopefully want to grow this. You want to grow the teams. You want to grow into a big organization, you know, if things are going well. And things get chaotic very, very fast without thinking about, as we mentioned, schemas, about events, about forwards and backwards compatibility, around documentation or validation or testing, right?

So these are things you want to be thinking about, even things like retries. And there is a lot to think about, but equally there's a lot of benefits. Some people might be listening thinking, wow, okay, why the hell would I do this if you're just telling me all the bad stuff?

But there are a lot of benefits to this whole approach, this whole evolutionary architecture approach, the whole we can raise events and kind of consume them in the future, which I love. And I've been on teams before implementing this stuff, and it's just an agility that I've never seen before in like over 20-odd years of doing REST-based stuff, API stuff. And don't get me wrong, our architectures will have this, but it requires a different mindset, I think, is the key to all of this. And that mindset has to be...

We have to be thoughtful of this, not just dive in, use EventBridge, use SNS, or use RabbitMQ, or use Kafka, or use something. Because you will see the value. You will see that initial value, and you will probably be getting your dopamine hits from this, and things are looking great, and then all of a sudden, things aren't great. We have to retry events. We have to replay events. Now, how do we handle that? What side effects is our Lambda function?

or our code doing now because we've replayed the event. Oh, we didn't think about that. These are consequences of this type of architecture. But if you go into this architecture type, knowing about a handful of these things, I just think it can really help you down the line, you know, and just set yourself up, set yourself up for success. And if you don't,

if you don't set yourself up i don't know i feel there's a common theme in this industry that um there are common problems that emerge in you know and uh i believe people probably would uh blame the architecture type and the technology when necessarily i don't know it's more of a mindset a lot of it's a mindset change and how you deal with that

Yeah, I think this is something that a lot of the team topologists folks often talk about, the fact that organizations are socio-technical organizations and you can't just rely on technology to solve all your problems. Your organization, right, your...

The social part is just as important as the technical part. And if the two of them are not aligned, you're just going to create more problems. So at the end of the day, things may not work. And it's not because the technology itself doesn't work, but it's because of the way that you have organized yourself. This is not working with the technology, but rather working against how the technology works. And I've seen this a lot with people that are trying to adopt serverless practices.

where they still have this rigid gatekeeping all over the place. And so you have, I think, again,

Team topologies often talk about the flow, having fast flows. But if you've got gates everywhere, that means that you have all of these feature teams trying to move fast. They can get their code ready quickly, but pretty soon everyone is going to hit the same wall. The gatekeeping, the infrastructure teams, who basically hold the keys to the kingdom to your AWS accounts and environments, and everyone just gets stopped right there. So you're not going to see any value as a

business if you've got things that are blocking all the flows in the organization and same thing goes like you said with event-driven architectures yes you can you can go you can have this ability to be really flexible and build systems that can you know

evolve over time easily without hitting with this contract, API contract, the problems and things like that. Different services can evolve independently. But if you don't think about how you are going to, how that's going to work with your organization and how it's going to work with your practices, because the way you write code, the way you write tests, a lot of them have to change and adapt to what you're doing, what technology you're working with. And the same goes to service as well, that I think, you know,

A lot of things that I used to do, I do them differently now because the serverless works differently to running stuff on servers and running modern lift applications. So I need to adopt how I do development differently. And a lot of people don't want to do that. They want to do exactly what they have done previously, but just make it fit into the new paradigm that they're working with. And that is just trying to scale up.

squeeze a square peg into a round hole and it's going to run into problems all over the place. And so how does that, in that case, how does the event catalog come into the picture? What are some of the specific problems that you're trying to solve with event catalog?

Yeah, sure. Yeah. So there's a big problem with event-driven architectures is the point where at the start, as I mentioned, everything's fine, kind of. And as you scale, you have more teams firing events. The common questions turn up like things like, where are the events? Like, how do I know what events are out there in the ether in your organization? You know, you're talking, some of these teams are huge. Some of these organization enterprise users are just huge.

Just huge. And then equally, it's like I'm about to make a breaking change. Who's going to be affected? So now we're talking about schema evolutions. We're talking about discoverability. We're talking about governance and standards. And it's a weird one, especially with event architectures, because we seem to be lagging behind in this area. So

Examples would be if we were tasked to build an API for an organization, I'm pretty sure we would get a good grasp of how we're going to do that because people have been doing this for 15, 20, 30 years or whatever. Open API specs and all this kind of stuff, and there are governance and strategies out there to do this. But I feel like the EDA space is lagging behind, and the more people I speak to and

and talk to, the more the common problems start to emerge. And a lot of it, to be honest, is in this governance bracket. So Event Catalog solves that. That's the kind of vision of it, really. It's technology agnostic. It doesn't matter if you're using Rabbit, if you're using Kafka, if you're using SNS, SQS, EventBridge, whatever.

It's discoverability, bringing discoverability to a venture of an architectures and allowing your teams to basically say, I have this event and who's consuming this? Who's producing this? What is the schema of this event? Am I about to break someone or not? Who owns this event? Is what team owns this? And even taking a step up is like, I have a service.

And this service raises 10 events and consumes four events. Okay, what 10 events does it raise? What version of the events does it raise? Is it still listening to an old version of this event that this other team is listening to? And when you start to map this out, you can actually start to

I don't know, just skip over this massive governance pitfall that many, many people are falling into. So I started the VEG catalog two years ago just as a side project, just like I think this idea is cool and see if anyone else actually, see if it solves any problems out there. And yeah, fast forward two years and more and more people are using the open source project. It is an open source project for those listening. We have Discord members. Feel free to kind of join that.

But over the last two years, the traction is gaining more and more. And now, in fact, the reason I actually left AWS is actually to focus more time on event catalog because it's at the stage now where it requires more effort, more time, more management issues are coming up and things like that. But there's huge amounts of value still to be had out there in this governance space. And

And yeah, it just doesn't seem to be there. There are things out there like cloud events is one of them, a specification that allows you to document your events in a standard way, which is very, very interesting. Because even like how you design your events is a minefield and it can be a minefield.

So Cloud Event Specifications is out. I got graduated this year and now we can start using a standard specification for our events. It doesn't enforce, you know, how you do certain things.

but it tells us here are some particular standards you might want to follow. And the great thing about this is once we have standards, we can start integrating and writing tools around this stuff. So if we fast forward into the future, if cloud events is adopted, which it already is in a lot of places, when we want to integrate with streams or other business events and things like that, if our broker or technology knows how to integrate with cloud events, we can start getting that value.

I think there's just, I think this EDA world is just still exploring these standards and these specifications and this, you know, we seem to be, I don't know, it feels like, to me, it seems like we're 10 years plus behind the API space, you know, and there's a lot more to do in this space. There is huge value to be had. Another specification worth shouting out is async API. That's a relatively new specification.

But this allows you to document channels and messages and what does my service do and all this kind of stuff. And it just allows, you know, once you have these specs, we can start writing tools, code generation, models across all of this stuff.

So, yeah, I kind of, I'm blabbing on here, but how event catalog helps is just, my vision for the project is just bring discoverability and governance to your chaotic mess. And a lot of people are coming out of the woodwork at the moment saying, hey, we've got this problem. Thank you for event catalog. And it's helping them. But I still feel like I'm on step one out of 100 on this project. So, yeah.

Okay. Is that something that you can actually demo? Do you have anything that you can just showcase the environment that you've got something that you can kind of show people what we can do? Because obviously you mentioned the different, I guess, the specification for how you document your events, well, the schema of your events, but also you've got, you know, other things that can be

do something similar to a lesser degree. Things like event bridge has got its own schema registry, but it's just a simple registry. It doesn't do a lot of the things that event catalog can do in terms of, you know, keeping track of who are the subscribers and potentially, you know, because you know that relationship, there's a lot more you can do there.

Exactly that, yeah. So I'm just sharing my screen, but I guess people will also be listening, so I'll try and explain what I'm doing. But if you go to eventcatalog.dev, you'll see the website here. And this gives you just an overview of what's happening and what's doing. But Event Catalog basically is generated by markdown files, okay? So if you can write markdown files, which I'm hoping most people can do,

Then you can use Event Catalog and it uses something called MDX, which is this custom component library thing, just marked down on steroids basically, that allows us to do custom components and all sorts of things. So if you want to get started, come here, but I'll quickly show you a demo of what you get so you can see the demo.

So everything you hear is generated by Markdown. It's a static site at the moment. It's a static site website, which has its pros and cons. The pro of this is it can be hosted anywhere. It can be hosted on your machine. It can be hosted on S3. It can be hosted literally anywhere. It comes with a Docker file. And it comes with search and all sorts of weird and wonderful features.

But the whole idea is it's a documentation tool for event-driven architectures. So it takes a lot of leafs, a lot of thoughts from domain-driven design and event-driven architectures and kind of marries these up to hopefully provide some value. Everything here you see in event catalog is optional, has the ideas of domains. So again, this is domain, going back to domain-driven design where we have a bounded context.

But let's say we're creating a system, we have this orders domain. And we could just start documenting this for people. We can start having who owns this domain, what services belong to this domain. We have visual representations and graphs where we can click on things and drag things around to actually start seeing what's going on. Sports, things like Mermaid,

diagrams and flows, all sorts of things. Flows are basically

If you think of like an orchestration end-to-end piece, right? So your product owner or your business might say, okay, what happens when a user unsubscribes from our system? Now, realistically, probably what, five events, six events, two, three different services, emails being fired out, all this kind of stuff. And normally this is documented somewhere in some Wiki somewhere, if you're lucky, or you just go read the code.

But flows is another way in event catalog to just document, okay, end-to-end, what is this particular functionality doing so I can kind of go in and dive deeper.

so that's the idea of there but everywhere everything you see on these pages are just marked down you can add as much or as little as you want but the whole idea too is you can start linking things you know the idea of as a domain has services so if i go inside a service this is our inventory service so this service itself could be api gateway lambda whatever you know dynamo db this is the service and as i said event catalog is

technology agnostic, doesn't really care what you're using. But we can see from this page here,

We can see that the inventory service, this is what it does. It has a change log. Event catalog supports change logs. As you make changes to your schemas, as an example, as you make changes to your service, it can actually write a change log for these resources, these entities. An example, Jan, you might join the company and you're tasked to do something with the inventory service and you might want to know some history about the inventory service, why these particular fields are changing in the event.

And we can actually do that with event catalog. We can actually say, okay, here's version A and here's version B. And this field changed for this particular reason. Now today, none of this context is captured. Right?

Right. And many companies don't capture this context at all. So when people join and leave and our systems evolve, the questions, the value is, you know, why has it evolved? Why have we gone from this to this? And what do we need to get it to the next version?

As I said, Event Catalog just comes with some basic features and some components that allow you to do things as you see on the screen here, like step components. I can come in here and copy code and maybe this is a code snippet to raise an event onto EventBridge using this particular schema and all sorts of things.

So I just want to, hold on, before we go any further, you mentioned that all of this is marked down. So does that mean that as a user of event catalog, I should have to write markdown to generate these pages and all the instructions here all have to be manually maintained by someone on the team? Correct. So there are options. So number one is, yeah, so this is all marked down. So you have to make, someone has to maintain this.

Now, there are two other options if you want to automate this. So event catalog actually also has an SDK. So now you can use these functions to basically say, okay, I want to add an event. I want to, I don't know, add a service, remove a service. I want to version an event.

Now, the interesting thing here is like, okay, so now if we have an SDK into your event catalog, we can start to automate some things. Now, what do I want to automate? Okay, let's say you have a Kafka cluster somewhere and you want to read the topics off of it. Okay, so I can read the topics. Now I want to document these topics. I can start using this SDK to basically build up the event catalog. And in fact, I can start writing scripts that get generated

Every time my event catalog wants to build which leads me on to the next thing which is different integrations and

So event catalog has this idea of generators, which is like a pre-built script, I guess. But these generators take a particular resource like async API, open API, and EventBridge was launched a couple of weeks ago. And it will go to the source, which is your files or your schema registry, pull in the information that it needs, and then document this for you in your event catalog.

So why is this cool? Because it allows you to keep in sync with the schema registry or your async KPI files, but equally allows you to add additional context onto that. So let me explain that quickly.

Amazon EventBridge has a schema registry and it has a schema discovery feature. So what you do is you can enable this on EventBridge and events coming into your bus and out the other side will get automatically documented for us. They would automatically get a schema in the schema registry. Now, unfortunately, this isn't enough because we have a schema. We can do code models and things like that. But

The context is lost. What is this schema? What does it belong to? What is the service? What is the, all this stuff, it just evaporates. So all you can do with event catalog and event bridge in this example is we can pull in schema for multiple schemas registries. Like we can pull in the information and we can generate our catalog data.

The interesting thing too is we can add additional context to our events using the in the markdown. So let me explain that again is we pull in the information. This information is stored in catalog and then we can hydrate that information with additional markdown context. So what does this mean for your teams?

is okay we're using event bridge we're in event bridge house we've got multiple event buses whatever's going on um and um here is an event catalog that tells you about all the different events now just out the box you won't get too much you'll get a schema and that's about it

But if your devs or your teams want to document that particular event, what is this order placed event or subscription canceled event? What does it mean? How can I raise it? What's the schema? Why is the schema changed? You can start to add this context. And then from a discoverability point of view, you have this thing constantly generating from the registry and you have the additional context added by your teams.

And I won't go too much into it, but there's lots going on here. You can even, I'm just showing on the screen here, you can map particular events into the catalog, particular services by like source, prefix, suffix, detail type. So you can basically say, okay, I have like a thousand events in my registry, but I only want to map

the events with the source of myapp.orders because that's a convention we're using in our organization and I'm going to map this to this, I don't know, order service as an example. So you can start to play around and mix and match and whatever your conventions are for your events, you can actually start mapping these into your domains and services. And again, this is only one of the examples.

There's a lot of people, a lot of users at the moment using things like async API to generate their catalogs too. So in this example, you have three services, three teams. Each team have documented their service. They've used async API to document the messages of that service. This message could be an event, a query or a command in this example. And event catalog here will just go in and pull all these things

spec files from all these different teams and give you something at the end of it it will say okay here's your discoverability tool and um your teams can add different contexts onto the files the same as event bridge and again this is all automated um reads reads from the files you can add manual stuff if you want or you can use the sdk to yeah you know um do what you want um

Yeah, lots going on. And for the actual flows, I think that's quite interesting because the saying that we often hear in the event-driven architecture space is that as a publisher, you should know who the consumers are. And that's the idea of an event versus a command. But as a developer of that system, you kind of do need to know who is listening to your event, or at least if anyone is listening to your event.

And so if there are going to be changes you're going to be making, you want to maybe communicate with them ahead of time and whatnot. And the flow is one way to show you who are the subscribers of the event. And I'm guessing this is manually maintained or is there some way to automatically for a subscriber to let event catalog know that, hey, my service is now subscribing to this event so that it will augment and update the visualization.

Correct, yeah. So at the moment, so the flows itself, so here are the flows. The flows at the moment are manual. So the flows, it's kind of like this YAML kind of format. But there's a feature at the moment open on the SDK to add SDK support for this. Now, why that's cool is then I can use the SDK to create flows, edit flows, change flows, etc.

Flows are a different kind of mindset. If I'm a consumer and I'm now consuming a particular event, does that mean I've changed the flow? Flow is the end-to-end business flow.

who don't know, I don't know, you know, in that case. But in the service case, you would see, you would definitely just automatically see that a consumer has come on board and the consumer is now, as you can see here, you know, the consumer is now consuming this particular event or this message. And I can actually drill down into different events, different services, different domains. And we can kind of click on these and, you know, zoom in.

zoom in, zoom out and see what's happening. The same with flows. So I don't know, did that answer your question? Yeah, kind of, because I'm trying to mentally compare it and compare this with kind of some of these more runtime approach someone like say PostNL has taken with their custom built event broker which sits between

I guess the publisher and the subscriber and so they know okay when the when a subscriber is subscribing to an event and so they're able to produce this you know this dependency graph and that graph is actually really useful I don't know if they're actually doing anything with that right now and that's one of the things that I find when it comes to implementing business workflows

Even though the individual component themselves may not need to be aware of who's listening, as a business owner who's trying to implement this business workflow, how can I make sure that all the right people are listening to the right events and there's no breakage in there in terms of the

the different filter rules that you're using. I can't tell you how many times I've had problems where upstream swears that they're sending the events just fine, but then downstream doesn't get any events. And so the whole business workflow is broken. And it's because, okay, some small change in the contract perhaps, or the filter rules were set up incorrectly so that the events never fires downstream. And so how can I visualize this as a

at the top level. And that's kind of the things that kind of drives me more towards something like an orchestration approach using step functions for business workflows, because, you know, I can't design that. Um, and there's no, there's doesn't leave a lot of room for, I guess, um,

That kind of mistakes or visualizations becomes a lot easier when you can see okay, that's workflow That's what's gonna happen as for not using event is much more Okay, everyone's doing their thing. But if we you know correlated Choreograph everything correctly. We're gonna get the right behavior. And so now being with you, I guess bring that visualization and that Well

their ability to understand the workflow and visualize it when it comes to event-driven architectures. I think that's the thing that I'm trying to get in terms of the workflow visualization you've got.

Yeah, I totally agree. I remember Step Functions, super, super powerful for just sitting with someone and kind of visualizing what's going on and things like that. The event catalog is an interesting one. So originally started as a static site generator and being able to just bring discoverability in a different way.

I'm speaking to a lot of users at the moment and the community around what would next steps look like. And I think Event Catalog, if Event Catalog knows the schemers and the producers and the consumers, think about the tools you can build on top of that. As an example, it could be validation, it could be code models or whatever else. Or in your case, it could be, could I do more runtime stuff with this tool? Could I do more...

things to help out there or could my service use event catalog to go get the latest schema and we can validate it against this and all sorts of things. One thing, features that come to my mind I spoke to people about in the past was

you know, like if I'm consuming an event as a consumer, that's one thing. But you're only interested in particular fields of the event too. There's granular kind of consumption, I guess. This is kind of going back to Postal's Law where he says, you know, be conservative, be liberal in what you do and conservative in what you expect from others.

And I think there's huge value in that. You know, I imagine a tool, I imagine event catalog to be like, okay, I am subscribed to your event, Jan, and I'm going to, I'm only interested in this one or two fields. Like the rest I don't care about. But can I be notified if any of these fields are about to change? Can I be notified if that field disappears? Can I be notified if it went from an int to a string? And that notification, where is that going to go? Is this going to go in Slack? Can I rerun my tests or my systems and things like that?

As far as I'm aware, there's nothing out there at the moment that kind of solves that level. But when I bring that up with event catalog users, it's quite an interesting thing. And I think event catalog, you know, if it's used as that core governance tool, I think there's just huge amounts of runtime and automation and extra value that can be added on top.

Yeah, absolutely. The whole contract testing piece is going to be interesting as well, because this is where I know that we've got the PACT, and the PACT can technically be used for contract testing for event-driven architectures as well. But as a space, it's just not very well documented or well understood in terms of how to use it for testing event-driven architectures as opposed to APIs.

This is something that I've spoken with Luke from PostNL, as well as Sarah from Lego, a little quite a bit recently, and the different approach they take with PostNL, the fact that they've got a broker, and the publisher and subscribers have to register their subscription and their schema for publishers,

to the broker first. So the broker has the time and space to then say, "Okay, you're changing a field from int to string. That's a breaking change, so we're not going to allow that." Because like I said, you want to be strict about what you expect. So the broker is there to protect the subscribers from breaking contract changes. But at the same time,

you know, if there's no one subscribing to this event right now, why should I not be allowed to change schemas, right? And, you know, you want to enforce that when you know for a fact that there are people listening to that event. And so this can be a breaking change for them.

But also there are the behavioral changes. For example, okay, this is a string, fine. But there's no guarantee that on a runtime, someone's going to send like empty string instead of like a proper ID or whatever. Mistakes can still happen. They can still be breaking. And so there's still other things that you can do.

you want to be able to catch and that's where hopefully something like a pact and consumer driven contracts can kind of fill in that gap even though right now it's still rather unclear how exactly that works I've been trying to figure out how the pact works for EDAs for a while and just haven't had much success there and the approach that Lego has done I think did you see Sarah's talk at the Compsom I did actually today I watched it this morning right okay

Yeah, so they had a really interesting approach where they kind of take a lighter weight version of kind of PACT that just they publish the schemas to NPM and then the consumers write tests against that. It doesn't have that enforcement element to PACT where you talk about where when upstream changes a schema, some kind of notification can be sent out to downstream because in that approach,

the downstream subscribers are responsible for pulling the latest version of the schema package from the publisher, run their test. But in large enterprises, there's going to be lots of systems through team reorganization, attrition, and what have you. The system does not being actively worked on. And so how do you ensure that when you make a change, those systems are running production,

No one's actually working on them by now, but you still don't want to break them, right? So how do you ensure, how do you provide the governance, the enforcement elements to contract? I feel this is where even catalog, being there in the middle and more of a design time, sort of tooling, can provide a lot of value in that particular space around testing as part of maybe overall governance.

Totally, yeah. It's, as you kind of alluded, it's a minefield, I think, out there.

with this stuff. And there are strategies, there are other strategies you can hopefully enforce, but it's more of a governance thing. It's things like, okay, what's our scheme revolutionary strategy? Are we backwards compatible, forwards compatible? And if you're thinking about that itself, you're in a good space, but most people don't. Most people just kind of YOLO this stuff and that's where the headaches emerge. But I think you're totally right. I think this is the vision. One of the visions for Event Catalog really is

It's exactly that. I think it has the potential to solve some of them problems because it knows who the consumers are. It's technology agnostic. It's schema agnostic, actually. It supports Avro, JSON, whatever you want to use. So if it had this kind of plug-in way or automated way where we could define some of these rules as a user and you could get notified of

changes incoming you know you could have a catalog in test or in staging or production um then you know it's um it becomes an added value i think for sure because as you say today we've packed and i saw sarah's schema stuff which is cool um it still still feels very um uh you know we we can we can prove this area a lot i think um yeah absolutely can do that

Yeah, and so I'm really glad that you're working on it full-time because I think this is definitely a space that's looking, that's screaming out for a solution. And ideally, one that you can just pay and use as opposed to having to host anything myself. Yeah, yeah, yeah. Yeah, this is, yeah, it's still early. As I said, it's still early days for Event Catalog.

the project itself is open source it's free um still trying to make things sustainable um so exploring ways but like even hosted catalogs uh plug-in dual license i'm not sure what it looks like yet um but there are problems out there that need solving so hopefully uh you know hopefully we can fix them and figure out the rest when it comes at some point

Yeah. Yeah. I mean the whole, yeah, I really liked a lot of things that the DDD space has been doing. You know, one thing we haven't even talked about is anti-corruption layers and all these other patterns that the DDD gives you. Yeah.

uh this you know patterns and and observation i guess if you call it that no the ddd space gives you and i'm really keen on seeing more people kind of adopt these kind of practices and like i said the first step would be to just be aware of them and think about it and rather than just going in just blind and just oh just you know just gangbusters and this is working great and then suddenly holy crap this is terrible what's happening yeah what's happening and then

Yeah. And then to be honest, you know, we all seen it before, just you end up in big rewrites and people leave and I don't know, but you're totally right. It's just being conscious of, you mentioned before the serverless mindset, right? And I think there's essentially an EDA, my EDA mindset. Um,

which is it's not a typical way of doing things. Any distributed system isn't really. And it's just understanding some of these pitfalls in like, okay, you know, as you mentioned, you know, the bounded context mappings with domain-driven design, like understand what they are, like don't necessarily go use them all.

But there'll be points where you consume an event and do I want this events model to be in my domain or not? And do I need to change this to something that we understand as a team? Okay. You know, we're talking about ubiquitous language here, you know, some standard stuff, but the easiest thing to do is just consume and go. And it's like, wow, I've got this order model everywhere inside my notifications email system now and what's happening and,

It's just separation of concerns. And it's an interesting one because when we're in one code base, one monolith, people have been doing solid and dry stuff for years and years. And you see all these nice models and patterns and things. But as soon as we kind of break that up into this distributed world, I think the same principles apply, but we just got to be aware of...

you know, these boundaries. And by default, unfortunately, none of that's there. And it's up to you to sort that out. And if you don't, you'll probably be, I don't know, I guarantee you'll be in a mess in a couple of years at least.

Yeah, absolutely. I've seen so many people run into problems of that description. And even something as simple as we talk about domain quite a few times, domain events, what does domain event even mean to a lot of people who don't understand that. And even the boundary context is another good one. The actual definition of boundary context is within which you've got one set of ubiquitous language, one unified understanding.

But that's not how most people use it. And I'm guilty of that as well. A lot of time, I'm just kind of lazy and use boundary context to mean a system. You kind of just falsely assume that one system is one domain, one set of team ownership, one set of language. But oftentimes, the boundary context is a lot bigger than just one system. Maybe it's an entire business area with many, many systems and whatnot. And so

So even though the way we can talk about it sometimes, it can be tricky. It can be difficult. And yeah, there's a lot of confusion in that particular space.

Another space, I guess that's still also, even though it's quite mature, but still a little bit, you know, feels a little bit Wild West, is infrastructure as code. And so, you know, you've been working with Wing Lang for a little while now. Can you maybe give us a high-level pitch? You know, why do we need something like Wing Lang? Why is it, you know, why is CDK or CloudFormation or Terraform not sufficient? Yeah.

Yeah, for sure. You know, I joined Wing Lang a few months ago, back in June now, and I kind of been looking at what they were doing when I was at AWS. And I reached out to Alad and Shai, the founders there, and said, you know, do you need any help? And I kind of joined them on that, on some of this journey. But Wing Lang is very interesting, like stuff I've never seen done before, right? Kind of merging the, you

you know, it's a brand new programming language that kind of merges, you know, your application, your infrastructure as code. Like, you know, for example, why can't I, in my code, why can't I declare a bucket?

and I had permissions to the bucket or whatever, like Wing will handle all this for you. But we have this concept of pre-flight, in-flight, and I think this is where the magic is myself. But on the pre-flight, this is provisioning code, right? So I have a bucket, I can do particular things on the bucket in this particular part, right? This is things like, I'm trying to think of some,

Maybe grant some permissions or add some properties to the bucket or whatever that is, right? But inside of in-flight, this is runtime code now. We can actually start interacting with the bucket. So I can add a particular object to the bucket and things like this. But the mindset is very interesting. The mindset is like once you get that understanding, it's like, okay, all I have to do is focus on...

just focus on a particular piece of logic or application code. What Wing will do as an example, it will understand in your in-flight, in your runtime that you need this function needs certain permissions and it will handle that for you. I don't know long enough to worry about this kind of stuff either.

it's it's yeah it's very very interesting to recommend people if you're interested come on got their examples on the website come to the discord and but if you're curious about you know i feel it's we need the next uh step of you know cdk and things like this and um to how do you simplify the cloud and i think i think i think this is a step forward to that for

sure and the guy people that work there the people I've worked with there it's just super smart people friendly people so yeah if you're interested come come along or ask me any questions I'll happily help but you know there's a there's there's somewhat of a learning curve there it's a new programming language like you know rust or if you're going to go or something else like that it's a it's a brand new brand new thing so yeah

And with the programming language, the thing that fascinates me the most is you can actually... You're...

I guess your playground is it's up to you to define the playground because it's a new programming language itself. So that kind of took me a while to get my head around, but we can start doing things like local simulation. So Wing actually supports local simulation. So if I'm building an application like APIs or buckets or functions or containers or whatever it is,

I can actually run a command called wing it and I can actually run this on my machine because we have the concept of different platforms and all sorts of things. I don't know. It's a whole, it's a whole talk in itself. But, you know, if you're interested in a programming language for the cloud, that's not tied to AWS, you know, there's Azure and,

and GCP, even custom platforms, which is very interesting. So if you're building, like if you have, let's say you're an enterprise and we do functions in a particular way, we do this in a particular way, you can actually create your own platforms and then give your engineers the code to deploy against these platforms. So yeah, it's great. It's wild. It's cool.

Right. Yeah. So I guess it reminds me of something that Gregor Hopp was talking about before he left AWS. He said this grand vision of architecture as code. So, you know, not just because the way most people use the CDK is that they just use a level one, level two constructs.

at best, whereas the real power is with level 3 constructs. And in the right hand, you can probably achieve that grand vision of just custom, maybe almost business-specific language for describing your architecture with code.

Greg was talking about how he sees people should be using CDK, and what I actually see people use CDK, which is a very different thing. So I guess for me, this is kind of where, okay, Wing Lang has got that advantage because it is a purposely designed programming language.

I mean CDK probably had that opportunity with constructs, but because it has to work with everybody, I guess, so you have to support those low-level constructs, level 1 and level 2 and whatnot, and that loses a lot of the expressive power that you're supposed to get with general-purpose language like what CDK is trying to do. And so I'm

I'm quite hopeful for what the wing Lang is doing. And I'm, you know, I'm kind of, I've kind of be keeping an eye on the, what you guys are doing. And I hope to get the yield out on here sometime at some point as well to maybe, you know, like I said, do a whole talk around the wing Lang and, uh, how that, uh, how that's going to revolutionize, uh, I see. And, uh,

But, yeah, David, thank you so much for taking the time to talk to us today. And it's really good to get your insights on event-driven architectures. And I hope you continue to improve event catalog and look forward to seeing what you do next. Yeah, thanks, Jan. Yeah, thanks a lot for coming on and inviting me. It's been awesome and great.

Yeah. I guess I won't see you at the re-invent this year because you're going to be busy working on your event catalogue. But I hope to catch you some other time. Take it easy, mate. Yeah, for sure. Always. Cheers, mate. Cheers, guys. See you later. See you.

So that's it for another episode of Real World Serverless. To access the show notes, please go to realworldserverless.com. If you want to learn how to build production-ready serverless applications, please check out my upcoming courses at productionreadyserverless.com. And I'll see you guys next time.