 All right, hi there, and welcome to my talk today at the Reactive Summit. I am here to talk to you about how we are building a pluggable cloud native event driven serverless architecture within just. This talk will also be a walkthrough of what we're building, how we're building it, and how you might get involved. I hope this generates some ideas for y'all about how you might build or approach your own systems. I also hope that it might get you interested in what we're doing at Ingest. So I'll start off with a brief outline of what I want to cover today. First I'll start off with explaining what Ingest is as a company and as a project. I'll share what our goals are for the project and what we set out to build and why. Then I'll talk to you how we approach building the system, including the principles of the system and how we designed it to be pluggable. Then I'll share where we're at today and where we're going in the future. Before I jump in, I'll share a little bit about the person speaking right now. My name is Dan Farrelly, I'm from New York, but I actually live in Metro Detroit, so it's kind of a hometown conference for me. Got to drive here this morning, which is nice. I am a founder at Ingest.com, brand new company, we've been around for about a year or so, and I was a former CTO at Buffer.com. And I've broad range of interests from event-driven systems to restoring old homes and just moved to Detroit to restore a hundred-year-old home, so working on really old things and working on brand new things. Hit me up on email or Twitter if you wanna chat or if there's anything you wanna talk about after this. Love to hear from you. Now, we'll start with some backstory. What is Ingest? Ingest is an event-driven platform. Its aim is to make it easy for developers to build, test, and deploy serverless functions to triggered by events. The goal is to not have to worry, but the platform is not to worry about infrastructure, queues, or stateful services. So one user called it an event-driven architecture box as we provide features that most folks building an event-driven architecture will need like schema registries, logging, metrics, SDKs, and various tooling. So Ingest is both a developer platform and an open source project. So you'll see me talk about both today as I go through this, but as I talk about the architecture and the system, I'm gonna lean into the open source and how it actually works. So our company's mission is broadly to accelerate the adoption of event-based architecture. Well, why this goal? Why is it important in the context for this talk? And I don't think I need to convince this audience because y'all are here, but I think that event-driven can be a beautifully simple way to build. You might say, simple, what are you talking about? You need to know CQRS or event sourcing or you need to understand item potency in your system deeply. You need to master systems like Kafka or NAT. No, these thoughts bring me to my next point. Jar, I think how we think about event-driven or reactive systems is often uber-technical and is very deep into system design, which is fantastic, because I think in various use cases, you need to do that, but I think this can push introducing the basic event-driven concepts to the average developer to being a non-starter. You know, it can feel sometimes like it's only a senior architect or an enterprise architect type thing. And sometimes architecture diagrams alone are enough to scare off most of the average developers, small teams working on this. And this brings me to my next point, which is that I believe that since it's such a large learning curve for most, event-driven in the basic sense has a lack of awareness. People have mentioned this morning, there's been a lot of talks in this kind of space. And what is it? How can you experiment with event-driven? How does someone learn over time? How does someone grow into those advanced concepts? And we're aiming to introduce event-driven systems to more developers in a more approachable way. You know, I think Mary mentioned this this morning in the keynote that there aren't a ton of people building event-driven and we wanna help change that. So we wanna take what other products and technologies have done for things like deploying static websites, building web apps, shipping APIs, working with databases and bring that to event-driven. This is why we focused on a fundamentally simple approach, system that can receive events and schedule work based off those events. We wanna enable software engineers to build but not battle their systems. So with all that in mind, we set off to have some simple design ideas and goals. It should be pretty fundamentally simple. We're not aiming to build something specific for streaming data or IoT. It's general purpose. It's aiming to cover a lot of use cases for most software products out there. Remember, we're aiming to reach the most people. As I mentioned, we need to make it approachable for all sorts of developers. To do this, we built an abstraction layer with common needs and things like handling failures and handling retries. As Jonas talked about this morning also in the keynote, most teams don't need every knob, he said. I totally agree with that. Doing this approach, I think that we can enable more teams to build reactive applications. And some folks might even take, think about this and think about ingest as a framework in addition to the platform side of things. But that's up to you. We don't really have a hard stance on that. But at its core ingest combines event streams with job scheduling into one simple system. So reiterating the goals from the previous slides, it must be easy to build applications for. So there can't be a high learning curve. With our mission, we broke down the ideas into two areas to focus on for this project. There needs to be a great developer experience or DX for short. We needed to be the most approachable system that we can build. You know, we wanted to reach more devs, wanted to be fun to experiment with. Want to remove those hurdles and provide great tooling. So it's not having to learn all these new concepts and build all new tooling and learn all these new workflows, set up script everything for yourself. And it had to be cloud native and it also had to have a cloud native serverless offering from day one. And the reason why we needed that serverless hosted version from day one is to enable people to build on experiment with provide feedback for and maybe a lot of businesses that are small. That's a great way for them to get started rather than self hosting. We wanted to get people from zero to shipping as soon as possible. And we feel like sometimes systems can require infrastructure teams to get approval and to figure out how they run this in production and like in production and it can take months to get these things going. A lot of times, sometimes at like mid-sized companies that can just render projects dead in the water. So we really had to design it to be ready to run in the public cloud for the start also because that also is potentially a non starter for some businesses. They want to be able to operate the software at some point. They may not right now, but they want to know that they can. They want the eject button. So we set out to ensure that the cloud native solution also had a pluggable architecture. The backing services should be able to be swapped out and run on different clouds. We don't want developers to program to a specific cloud provider or a specific underlying event stream system. So we just needed to be agnostic to where the platform is running. And the system had to of course be open source to support all these use cases. Just wouldn't be possible without this. So for good developer experience, we just think that it needs to be simple to start building. Heard me say that. You know, I consider myself a highly average programmer and the initial cost time costs to writing say producers and consumers and making all the various necessary decisions from the start can create a lot of friction for people. And I know it has for me in the past. You know, we worked through several iterations of this concept to our current state. You know, we started with multi step publishing of runnable containers as actions and job configuration files and, you know, custom DSL that we have through a bunch of user feedback. We've developed a simplified SDK for enabling people to define functions and easily publish events. You can see just quick sneak. It's probably pretty small on there on the right. But, you know, we want people to be defining functions not consumer processes. It should be ready for serverless and designed to be portable. Dependent, you know, it doesn't matter where the actual code should run. The person should not be worrying about the underlying event stream abstraction. Like I mentioned earlier. So the SDK should also be able to then generate configuration for that file. We want to minimize the amount of YAML or JSON configuration, configuration hell that teams can go through to just get something out. So we think the system might have to require some external configuration at some point for functions, but we want to minimize that as much as possible. So to prove all these concepts, we started with JavaScript and TypeScript SDK because that was where the initial demand was. Some folks may actually look at this and think, is this low code? You might be right. I think it's a highly subjective concept or a term to be using. But, you know, if that's what it is and people can just focus on the business logic as we have seen also highlighted in multiple talks today, then I think that's a great thing. You know, as a prior CTO at our business, I always was trying to get our engineers to focus more on the business logic and how we're delivering for our customers and less on the infrastructure and the actual architecture of the system. I wanted us to deliver things or else we didn't grow. So that was my primary goal. And, you know, so coming back to this, I think that we also wanted to keep it simple. We don't want to, I know myself, I've wrangled with ProtoBuff in my day and we just wanted to keep events simple. So we just used JSON from the start. It's just the JSON API that we have that enables developers to also just do things like point a webhook at ingest. And I think the benefit of that is that oftentimes earlier backend developers often the first event driven concept that they might encounter might be webhooks. So that's a great entrant into the space of thinking about how to build these systems. They're all events. So, you know, in addition to that, things like versioning of functions, throttling, item potency, concurrency should all be included in the system and should be easy to configure. And the last thing is for this part is that we think it should be runtime agnostic. You know, I talked about that portability of how you define the functions. They should be able to be runnable on HTTP. Doesn't matter, right? If someone just wants to deploy it there, if they want to get something out really quick, that's fine. It's kind of RPC then. They could also run it in Docker runtime in any code really at that point. And they should also be able to run it in serverless environments like Lambda. And because of this, the SDK needed to try to create a right once ship anywhere approach. And developers should be able to also use multiple runtimes. They should be able to mix and match depending on the use case or needs of their job or their system. So the portability really matters there. Another component that we thought it was key for these event-driven systems to have a simple event schema registry. So some developers that are just getting into building systems like this, like reactive systems, don't necessarily fully understand the value of schema registries and whatnot and types for their events. So we wanted to design it in a way that they could kind of easily adopt it and like learn the value over time. So what we built in was generating the event schema automatically from events that the system has seen. Because we're just using JSON. We have a system, I won't go into detail in this talk, but about how we parse these types and generate and version different event schemas. So we think that could be a way to onboard them and get them trying out event schemas from the first place. And what we also added was the ability to export the schema as types. And in the SDK's case, typescript types. So the SDK then can leverage those same types. So we think of this approach as a way to enable developers to learn the benefits of this and be able to leverage types and kind of have that consistency rather than have to wrangle with things or have to write them all first and decide them. So the system then and how we want developers to use it needs to have the tooling or API so that they can share types across teams, repos, project services, what have you. So I included just a snippet on the bottom where there's a one command that can just generate all the types for the entire system into your files. You can build this into your CI CD process or a local development flow. So we wanna do things like this that may take some time to do just very easily and then make that easily to easy to consume. So shifting from the end user experience for building we'll now talk about the backend system architecture. So on the system side, the first principle should be able to run on any cloud or on your machine. And these are different ideas but we think that both need to be approached in a similar way. The system should be able to build around the idea of drivers that can support a few different things. In the cloud, they should be able to support as many open source or popular cloud managed services and should be able to be spun up on multiple clouds. We should be able to use things like Postgres or Redis or Kafka or Nats. And locally, drivers should be able to run in memory for what we think is single dependency, local development and then also end to end testing. So I'll touch on this more in a little bit. And overall, the system should be pluggable, it should be portable. Now I'd like to share a basic high level architecture of the system and here it is including all the necessary backing services and components. So fundamentally, it's an event stream and a job scheduler. So that's what its core is doing. It's not data streaming or processing and whatnot, more advanced things. So there are four services that you need to run here. The event API, the runner, the executor and the core API. And the backing services are an event stream, a data store database, a job queue, a job state database. Pretty simple names right here, but we also wanted the architecture to be pretty simple for the average developer to also grok and understand. Runtimes as you see to the right, where the jobs are, are external to the system, but also have drivers. So I'd like to briefly break down this system and share what each component does. And starting with the services, the event API. It receives events, handles authentication and authorization of these clients that are publishing events with their own keys. This asset has a buffer to the event stream and it also ensures that the system is highly responsive. Then we have the runner, which basically just consumes and records and does multiple things. It also communicates with the event schema registry and it then creates and decides what events cause, what functions to run and creates jobs. The executor does just that. It executes the function steps from the job queue. And then it also, as it's processing these jobs, manages the state of the jobs. And lastly, there's the core API, which is basically a CRUD API for system data. It's used as an interface to deploy function configuration. You need to know what code runs on what event, what configuration, what are those retry policies that you might have, as defined by the SDKs. And overall, all of these are designed to be run in replication and in production environments to be more resilient. And we have some benchmarks on this. We haven't built in auto scaling just yet in the open source version, but all of these are designed to be a resilient elastic system that can auto scale. So we've covered the core services and now we'll check out the four backing services. Probably these are things that you all are familiar with. And we're using drivers to enable different configurations from say low volume staging to high volume production environments. So we don't aim to reinvent the wheel, right? There are amazing projects out there and amazing tools as we've heard talked about earlier today as well with the talk on NATS. We don't aim to do that. We just aim to provide that abstraction layer and to be able to leverage those technologies. So it all starts with an event stream. It's the backbone of any event driven system. You can use NATS, PubSub, Kafka, which are all great, but they'll have varying levels of operator friendliness. Potentially you can use managed offerings as well. And all of this persistent data in the system is currently stored in one data store. That's the functions and the configuration, all the keys. In a production environment, this would be split out into a separate history store which you could throw in something like a Cockroach DB or a MongoDB or Elasticsearch depending on your needs or preference. So this is definitely an area where you'd have to decide if you're running this system yourself, how would you wanna run it? And, you know, sorry. Then the next then there's this job queue which is basically just a simple job queue where you can handle some back pressure from the entire stream and have multiple executors that are executing those jobs. So the last component there is the job state which is the state of every single job that is in process, maybe how many times they need to be retried, how many failures, all the successes. What happens in the system? So this is basically sums up the simple take on the system. This isn't that revolutionary, but we think that it's a very simple and easy to grok approach and all the system ends up wrapping all these components so the end user doesn't necessarily need to know that they are writing a consumer for NATs. They just need to write a function. So as I mentioned earlier, the goal is to support multiple function runtimes. You know, HTTP, we could even support Wasm in the future. Anything that can really run in a Docker container can run as well in our system. That's how it was originally designed. And the function should not, the function code should not have to change to support these environments. So these are all drivers. So you can have your runtime of Docker and your actual environment of Kubernetes set. So we wanted it to always be portable. And there's one thing that I told, I mentioned earlier that I come back to was the in-memory implementations. And we think that this is an important approach because at the end, at the end user level, you're gonna be building applications. You're gonna need to be testing it. You're gonna be needing to sending events and consuming events and running functions. And oftentimes you might encounter some situation where you're gonna have to Docker compose a bunch of containers and your teams are gonna be complaining about their laptops slowing down and not working right or some configuration changes and the system doesn't match production. That can be really difficult. So because we're an abstraction layer, these can all be in-memory implementations. The end user doesn't have to care about how it works. They seem to know how their code reacts and how it will behave in production. So you can use all that in a local environment and then you can also run it in CI CD and do things like integration tests and for your actual application code. So another thing that this architecture hopes to enable is the ability to run cloud hybrid configurations. So operators can customize their deployments based on those decoupling points back end services. And the idea is that these at some point could potentially also enable cloud hybrid. So some things that are hosted, some things that are not hosted. So they're all these different lines where they could interface with the rest of the system. So we aimed to make it to be portable or to decouplable so that you can have multiple event streams or multiple job queues and maybe separate where clusters are executing the jobs so that maybe you can handle unique situations like data sovereignty. So how we're building it. For those interested in the code, we're building it using Go. You probably could have guessed this, but we are leaning into packages, things like the Go Cloud Development Kit so that we can just roll with amazing community drivers and switch between all these underlying things. So we're not reinventing the wheel there as well. So we're also taking the approach of learning how to scale and all the failure modes of the system by operating cloud service that uses this code as well. So we're using the same thing so that we can understand how to build this, what features people need, what features people want. Where we're at today. I talked about our system goals and our design principles but where we're focused at today is we have our in-memory implementation that you can use. You can run on your laptop. We've written some initial drivers. We have commonly, for commonly used bagging services, Postgres, Redis, and we wanna add more and more and more like some of those Go Code libraries. We can adopt a lot more depending on user demand and what people need. And we've also created our first cloud stack. So you get a Terraform file and you can point it at AWS. And you can see this configuration file for ingest right here and basically aims to be as simple as possible. You point it at those bagging services, maybe some advanced configuration in the future but you can run something yourself pretty quickly. So this AWS stack ends up using things like SQS and elastic cache. So that allows you to maybe get going and shows the portability of the system, the plugability nature of it. So our priority though is getting feedback from people. We just need engineers to work with the system, provide feedback and we fully believe that great systems are not designed to be built in a bubble. We need feedback from people to understand what they need and what they want. So our roadmap, I'll quickly run through here. What we're focusing on, we'll keep working on the developer experience. We're streamlining this thing so that people can just write against this system. No matter where it is, run. We're also working on step functions which you have highlighted on the right here and they're things that we're developing to make even easier for people to run advanced workflows and also say coordinate between multiple events. And these are concepts that maybe also other people might find difficult. So when one event comes, you may wanna wait for another event and then trigger some code. So coordinating on two different things that would require more state or scheduled jobs or delayed things in other architecture if you're rolling it yourself. So we're very excited about that and we'll also be working on supporting other function run times like Lambda or GCP's cloud functions. So this talks about what we're doing today and our long-term vision. And our goal is to ensure that Ingest Core is all open source. This project has to be open source that needs to be to provide that developer experience that I've talked about and the workflow that we think is required to build systems and bring this to more developers and small teams and startups. So there's still a ton of work to be done. We have some code that also needs to be extracted from our core. We have some stuff that's open source that needs to be upstreamed into our own cloud system. So we need to dog food it a lot more. And that's all I came to cover today. I hope you found it interesting. I'd love for you to check out some of the code, feedback, run it yourself, let us know what you think. There's some links up here. If you're interested in self-hosting or just joining our community and asking questions or providing feedback, we're eager to hear from you. All right, thank you. Does anybody have any questions? Okay, the question was about putting thoughts around versioning of functions and versioning of schema and thinking about schema evolution. So if I were to ask you about that, you mean like certain functions handled certain schema and only are compatible with that? Yeah, so that's definitely a thought that we've had and because we've encountered other systems that we use that I won't name that are difficult. You almost have to support every single function version that you've ever had to run certain versions. Some of the controls that we don't have in place yet are like rejecting events based on the schema is not matching. So that is definitely something we need to do. It's something that we have encountered with some people that are using the platform and the open source, but it's something that we haven't decided on how we're gonna do it yet. But I think we could approach that in a way where functions could define what versions of the events that they can handle and tie that relationship closely. Right now we just have a function can just choose what event or events that it handles. We do have support that's a very general thing. We've added some general features like expressions. So a function can define an expression that reads that event payload. So hypothetically, if you wanted to roll it yourself, we wanted to make it flexible so you could write something like event.payload.v equals 2020-10-24.1, and that's your version. So you could handle certain of those things or detect that, write that in your code. So you could filter that and the function just would not run. But there's definitely a lot more advanced things that we could do there. We hope to also enable things like running multiple live function versions at a given time so you could have blue green deploys or gradually as you're rolling out changes to a system including the producing of events that one function would take over from the other as those new events came over and once it got to 100%, the old function would stop running. So definitely there's a lot of interesting things there, but we wanna make sure that we can do it in a way that isn't too over complex. But yeah, if you wanna talk more about it, I'd be happy to hear what you think. Great. Well, thank you. And if anything is interesting, hit me up, I'll be around all day. Thanks y'all.