 hold it, that's fine. So hi, my name's Greg, as you said, Poyer. And I'm gonna talk about an infrastructure, architecture pattern that's probably my favorite, and also my 14th favorite cocktail, the Sidecar. But before we get that, let's talk about me for a second. Some quick background about me, for those of you that don't know me. I currently work at SENSU on an open source monitoring platform that maybe some of you have heard of. Prior to that, I spent about 15 years building systems and building software for internet service providers, the United States government for a little while, big data companies, and most recently, several software as a service platforms. So what ams a Sidecar? At this point, I kind of was working under the assumption that maybe everybody in the world had heard about Sidecars, and this wasn't all that interesting of a talk, but as recently as yesterday, I was totally proven wrong that still people don't really understand or know about the Sidecar pattern, so I'm kind of encouraged and hope that at least some people here get something out of the talk. Last year, this is a Sidecar with a dog in it, for those of you that don't know what this looks like. I love this picture, it made me happy this morning. I've been sick for the last couple of days, so I need some pick-me-ups in this talk. Last year, Brennan Burns and David Oppenheimer at Google published this paper at USENIX, called Design Patterns for Container-Based Distributed Systems, and in it, they talked about the Sidecar pattern, and they said that Sidecars extend and enhance the main container in a unit of deployment, for example, in Kubernetes like a pod. And in this case, really just main container is another word for the actual application that's running in the pod, and the Sidecar provides enhanced functionality. But I wanna frame our discussion more generally in the context of a holistic software architecture. But first, a couple of trigger warnings. I use some words in this talk, like software architecture and monolith, and I think I might get some people like angrily adding me on Twitter, because monolith specifically, I'm not using this as a pejorative to describe any software. I'm working on a monolith right now and I'm super happy to be doing so. I'm not talking about a monolith. I'm not even saying that monoliths are terrible. I'm just saying that this isn't a talk about why one architecture is better than another. It's just about how they are different, why, and maybe some of the benefits. I look at software building as a choose-your-own-adventure book, but it's like choose-your-own-architecture, right? Where it's this super expensive book that only ends when you run out of money. So my suggestion to anybody, right? So don't do that, don't run out of money. My suggestion is to choose an architecture that makes sense for your organization so that you don't end up engineering yourself into a hole that you can't dig yourself out of. And while we may be super thrilled to be in that hole that we dug ourselves, I mean, I've been there and I really enjoyed my time, but we didn't get out of it. So in this choose-your-own-adventure book, you might find yourself on this page where you say, like, your founding technical team has left post-IPO, and your sprawling organization of 25,000 people is now faced with addressing the mountains of technical debt that made you wildly successful, right? So choice one, find more tires, hire more people to carry them to the fire and light them, right? Like, just keep piling on. That's choice number one. Turn to some ridiculous page number. Number two is like, okay, let's reorganize our people and code in a way that makes not just necessarily microservices, but when I say microservices, I mean like a services-oriented architecture, right? But not like capital SOA, just services first in a decomposition based on business units, logical segmentation, that kind of thing. So either way, honestly, like congratulations, you live. Your business is not going to end because of your architecture choices. I mean, it could, possibly. But I haven't seen it happen yet. Actually, no, I'll take that back. I've seen it happen once. And it was cool. I didn't help build that one. No, yeah, I did. I helped, I've seen it happen twice. I built one of them. So, yeah, we'll say for the sake of this presentation, though, that you decided to choose number two. You and your leadership team got together and you said, okay, look, let's take a look at our product and take a look at our business needs and let's start to decompose this very obvious, coherent and clear architecture diagram into something that makes a little bit more sense and something that we can scale. And it's just like, you know, you're all staring at it and then you have this moment and you're like, aha, yes, of course. I see how this easily decomposes into services. Why don't we reorganize our teams around this very, very obvious decomposition so that all of our development efforts are aligned with this great new shining city on the hill that we're going to get to. And you get right away to work, making a new architecture diagram that looks very similar to a Death Star. And you figure out how, like, how are we going to get from, you know, horrible nightmare into, like, something that's terrifying to our competitors that we use to destroy entire planets. So you come up with these small units that you use to build small components of functionality, right? So we say, okay, this is our service. This is what a service is going to look like. All of them will look the same, but like this one will be yours, right? So we're going to do a very simple HTTP transport. So we have an HTTP listener. The next thing that the stack hits is like some rate limiting, right? Because you don't want somebody just hammering the hell out of your service and running you over. Then after that we have like a coherent authentication and author, I'm like describing something that's totally pie in the sky everywhere I've ever worked, but, you know, we'll get there someday. But a coherent sensible authentication and authorization layer that's like, has the ability to have accountability for the people using your internal services so that you know, like, who your customers are internally and how much resources they're consuming. And also, you want to prevent, like, unauthorized access between services for things that should not ever talk to each other, you have some compartmentalized sensitive information. I did work in an organization with classified sensitive information and the way that compartmentalization works there, it's very, very much like physical in some senses and moving across boundaries is like a big no-no and it's very heavily tracked and like declassification is this serious process. So like, this is a real need that people actually have and not even just in the United States government, right? But anyway, I digress, this is like a whole other talk that I have. Then you hit like a request logging layer that's like, okay, well, we've authorized this, now let's go ahead and log it and say with some kind of structured, sensible event format, right? Like, this is the request, this is the response, et cetera, et cetera. Then we hit our business logic, right? We have a persistence layer so that we have, you know, some backing data and then we have some like core service components like service discovery, configuration and metrics. And everyone's really happy about this brave new world and you get to this point where you're like, okay, but how do we build this now? And you have some choices, right? And so you hire Nick Jonas to come in and look pensive because I really like Nick Jonas and I'm consulting for you, right? And the thing I like about him in all of his pictures is that he always looks like he's like trying to make a decision between two things and in this one, he's clearly trying to decide, okay, do I build with shared libraries or do I use external processes that add functionality to each of our organizations deployed applications? And then Nick Jonas has this aha moment and he's like, hey, that's a sidecar. So even Nick Jonas is like recognizing the amazing, I don't know why that's flashing. I guess it's the aha moment is that powerful and also Nick Jonas' face. So everybody gets really excited about sidecars. We get excited about microservices. We're doing a little bit of conference driven development but like it's pretty well accepted at this point and we're at this pivot point where we can be architect. So you get your well staffed organization, well organized organization together and you say, well, we all want these things to look the same. Let's start out all building them in a single language for example, what if we had, I know, that's almost laughable these days. What if we step back and we actually like build these shared components once and dedicate and staff resources to maintaining them over the course of their lifespan and allow those to be easily reused between each of our deployed applications. And at this point, I wanna say that like this is, notice I have not yet said like, oh, and also we're using containers because that doesn't actually have to be the case, right? Like we've actually been deploying sidecars to VMs like since the dawn of computers. So recall in our architecture diagram, the HTTP listener and rate limiting, right? Possibly even authentication and authorization and request logging, that sounds an awful lot like a reverse proxy, right? Like engine X is terminating SSL and maybe doing some rate limiting and it happens to be local to the host that the service is listening on instead of like a loan balancing layer. I've deployed engine X like this before I'm sure other people have successfully. And engine X is providing a lot of really important stability for your application with buffering, caching, rate limiting and load balancing in some cases. And that's additional functionality that you didn't have to implement yourself that you can reuse across every single service. It's uniform, it's dependable and you're not writing it, right? And that's the first big thing that I like about sidecars is reusability and I think that in the world of the DevOps like we talk a lot about reuse and instead of re-implementation, right? Because the more you end up rebuilding the same thing over and over again like it's never exactly the same and if you're implementing these libraries in multiple languages, the implementation is fundamentally different in every language. So by deploying this application alongside another application we add this functionality to it and we didn't have to rebuild it. And this is where like in working on this talk I was talking to a lot of people that I really respect and I got a lot of nerd anger because they were like but shared libraries. And yeah, okay, sure, sure. You work at a unicorn where there's only one language and I'm really happy for you. I've never worked there. So maybe don't judge me for wanting to like solve a problem that I've actually had and let's take a step back and just talk about the merits of this architecture, right? So sidecars don't care what language your application is run in, right? They are very implementation specific and the interfaces are not language bound. And the other thing that's really interesting about this is that sidecars become runtime dependencies instead of build time dependencies which means that they're independently built, tested and deployed outside of the scope of your application, right? So when a critical vulnerability hits and it happens to be affecting one of your client libraries you'd have to like build and deploy the world. And that's like a really scary button to press even in a land of unicorn continuous delivery. I still think that building and deploying the world is a terrifying button to press. So if you only have to deploy this one piece of functionality like if you're only addressing an open SSL vulnerability in Nginx like nobody's ever done that here you only have to deploy these reverse proxies instead of every single app. And solid integration regression testing of your sidecars gives you that sense of confidence where you don't have to like worry that you're going to introduce a breaking change because you clearly define these interfaces and document them and you say we will adhere to this interface like this sidecar will behave a very expected way. And you can do like end to end full suite integration testing that way. And it makes it like the level of confidence you can get with these kind of deploys I think is significantly higher than if you're building and deploying because of a change in your client library that you're maintaining every other service even more causing every other team in your organization to do it for you, right? Like hey everybody has to go deploy it's just a scary moment. The other big I guess conversation around sidecars arguments I got into was well why does it have to live on the same host? Why can't I have a pool of shared resources? Like think of service to service routing. It's not uncommon for you to have like your Nginx SSL termination layer and then an HA proxy routing layer behind that or maybe we're going direct to HA proxy now because it does SSL okay and buffering but I still prefer to put Nginx in front. Anyway, you'll have like a pool of load balancers, right? Like think ELBs or I think there's a GCP equivalent some kind of service proxying load balancer. And you can go that route and that's perfectly legitimate. There are two key things that a sidecar provides that that architecture doesn't I think but before we like really talk about those let's kind of dig in and work our way toward them. So as a concrete example we'll talk about a configuration service. In our imaginary architecture configuration could be a client library and I don't know if you've ever done like runtime configuration I had not until just a couple of jobs ago, right? So you have a library who's responsible for talking to either a simple configuration service via HTTP or maybe it talks directly to something like ZK or SCD and this library does all kinds of failure handling for you and it's very friendly but there's a problem that exists with this that you end up having to solve. So your durable configuration data does have to live somewhere in both worlds. There's no arguing that. On the left we have our ZK or some kind of configuration service and it's centralized. We're scaling it like a SAS, an internal SAS basically. I would argue that like if in a services world you do kind of run a bunch of internal sasses in my mind like that's how I think about it because you have all of these services that have to be incredibly well behaved and have to respond in like a uniform manner and handle all of the like failure cases that an actual SAS does. So maybe you're getting really good at building those but I would argue that maybe some of them don't have to be full sasses. So this could just be a key value store or something. On the right we have like the sidecar equivalent of that and your configuration sidecar might behave fairly similarly to a configuration service where you have like a very simple interface which is like put and get to a key, a path to your configuration data and maybe it's one or more keys but for our example we're gonna simplify it and it's gonna be a single key. So your requests are gonna look like HDD put, HTB get and they will have a request body of just JSON, right? We'll use JSON for our configuration. So our interface for a sidecar might look a little bit different. You could build any interface you want but for our example we'll just do a simple CLI that's the exact same binary that goes into the sidecar that you just go get onto your laptop and you conf put dash i as your input file, right? And then config service order service and that puts it into the key value store. You don't have to know what it's putting it into, how that works but you have a very simple clean interface into that configuration service and then the same thing basically runs inside the sidecar but we might, we could maybe fancy it up a little bit, right, because like something that just runs once and gets a config file and puts it on disk isn't very interesting. So let's say that we are in containers because I happen to use the word volume in here and maybe we don't have to talk about it in container land but let's say we're a pod, right? And let's fancy up our little binary a little bit so that when you run get it pulls. Well just it'll pull. It doesn't even like watch for ZK or SED events, it just pulls at a fixed interval that the service builders chose just arbitrarily. So we've sacrificed a little bit of consistency for availability here because we're not hammering our config service or key value store all the time with requests for updates. So regardless of the interfaces that expose though this affords us a little bit more flexibility than an HTTP based service because it's located on the same host as your application so you have every communication primitive available to you to build around and in our case I kind of want to just make it talk to the file system, right? Because this is really simple, all our application has to do is look on disk at a pre-configured path for a configuration file. It knows the format, the configuration service is completely unaware of the format, it's just stalling bytes like a byte array into this key and then you as a service owner like interface with that byte, that byte array on disk in a file. And but how is that different, right? Than a client library really. Well, again this is about like crossing language boundaries first of all because you're not having to re-implement that. It's right there, it's available for you and it's just super simple and you can get into this situation where like let's build a new application and go, right? Like we're a JVM shop, there's a wonderful client library that lets us talk to our configuration service and but we have a business reason to build something and go and so we're like okay well now we have to go basically some other team has built this library and they manage the configuration service. We work with them to implement to their spec and we don't have whatever service building framework is available for the JVM so we kind of have to build our own little runtime thing and we put configuration loading at the very first thing before we test for availability of any runtime dependencies. So what ends up happening is that our app starts and we're in Go so we're failing fast if we don't hit a runtime dependency, right? We're just gonna let ourselves crash and then we'll get restarted by our process manager. So we go to our config service, we crash on the next runtime dependency and we restart and let's say we got really gung-ho about our deploy and we started a hundred instances of this and all of them are crashing in a really tight loop and we didn't even bother configuring our process manager to do any kind of back off whenever we're crashing even though we're using system D and we could totally do that but we end up dossing the config service because it's crashing, a fin's not being sent, you've got a bunch of sockets on your config service hosts that are in fin weight forever, you hit SOMaxCon, cast gaining failure, a total configuration service outage for your organization and now all of your stuff is failing because it won't start because of this one bad actor, right? And maybe you are better than the company, I was that one that's happened and you have some things in place to mitigate this problem but the point is that with a sidecar this wouldn't have happened in the first place because we have this process over here whose job is to be very dependable, very simple, it handles like managing the state on the disk and all your service had to do was like open a file and read its contents so even if it's like spinning tightly in that loop on the disk, it's really not a big deal and that file's contents are getting updated only whenever things change in the key value store and those are like the two key final benefits of sidecars, locality and isolation. Sidecars isolate the interaction between processes and localize them to a single host. Your application may abuse the local copy of the sidecar but it's only abusing its instance of that sidecar, not somebody else's. These aren't multi-tenant processes, right? Like you have two processes that are tightly coupled on a single host that have their own life spans and every instance of the application having its own instance of the sidecar means that we're scaling out with demand immediately and you're gonna have to take that into consideration like this is an operational cost on the owners of the configuration service, you're really gonna have to think about that. Okay, well now we have like 50,000 copies of this sidecar service, how do we scale that? But that's the thing you can do, like you are scaling out just your key value store at that point, you're not also scaling out this whole service layer as well. You're not really running a SaaS, you're running a very simple application that other people use and consume. And those are the whys of sidecars, reuse, locality and isolation. And I think that these are really, I don't think I can state the importance and like the magnitude of these benefits adequately but I hope that at least some of it came through. And now for like the fun part of the talk, at least for me, in my proposal, I promised some like practical examples of things that you could do with sidecars. I mean the config service is a totally practical example, I've done that one myself, I actually use it for like my little crappy Kubernetes cluster. I don't know, maybe I'll open source it or something, it doesn't really matter but it's just super dead simple. And in working on this talk I was like, well I should like get good sidecars, like things that people might get excited about because I know that I learn about things best when I'm super excited by a particular concrete example of an idea. So I wanna start with Prana from Netflix and it's funny because this was like the thing that sparked the whole title of this talk because Prana is literally like the Netflix Paz stack lifted out into a sidecar JVM with an HDP proxy. So like your Go application at Netflix can talk to all of their Paz components without having to re-implement like a rich suite of JVM libraries in Go. And I know that they have done Hysterix in Go and I've actually talked to the person that's working or worked on Hysterix. And I think that they are actually building out some client libraries for like first class citizens in Netflix. But like this exists and this is super dope, like this is the stack taken out of like a monolithic JVM application and put into a sidecar. And I just, I don't know, I just got super excited by that one in particularly. Jess Rizel tweeted about this like last week and this one was like, this is maybe my second favorite one. They're sort of in order of favoritism. It's an OAuth2 reverse proxy. I don't know if any of you have ever done an OAuth2 workflow in like a Rails app or something. It's a huge pain in the ass. Like I hate OAuth2, I don't remember how it works and I swear to God I'm never doing it again because this just does it for me. Like this is so dope. You can do an Nginx for SSL termination and then you've got this like OAuth2 proxy that's sitting there in the middle as middleware between Nginx and your app. So like the request comes in for authentication. It just goes and talks to Google or GitHub or whoever you configured does the workflow for you and then passes the token to your application identifying the user. And it's like, I'm done, I'm done. I don't even have to like consider authentication for anything I build ever again because not only does it do the workflow, it lets you use your own CSS and like static assets to customize it for your organizations like look and feel on the web. I don't know, yeah, yeah. I thought that was super dope. This is a super, super also rad sidecar that I found just the other day. It's a Let's Encrypt certificate manager for Nginx in a sidecar. I don't know if any of you have ever built like internal certificate management systems before and you've had to roll certificates everywhere. It's a huge pain in the ass. And Let's Encrypt is probably one of the coolest things that has happened in the last few years because it's like API-driven certificate management. Like how dope is that? So this sidecar, not only does it do like the initial certificate creation, it rolls your TLS certs for you when they expire before they expire. So yeah, I really like this one too. This one, like this was literally my face when I saw it because I was just like, what? So I mean, like there's some really cool stuff out there. Service to service routing is like a near and dear topic to my heart because I'm like a super microservices nerd. I'm sorry, I really am. I get a whole startup. I mean literally like employee number one and I'm like, we're using Kubernetes. I'm like, why? Cause I can. So you have to solve this service to service proxying and routing stuff whenever you do that. And so I don't know if you know buoyant.io but they're the people that do linker D and I think at least one of the founding members worked on finagle at Twitter. So you have all of this like really rich configuration for your service to service routing, including things like rate limiting. I think they do off in, off the Z. I know that they do circuit breakers right out of the box and that's like a really powerful construct to just have like in your pocket when you go to build these systems. This is the thing that's really hard to get right and to do well. And speaking of the team that built finagle, some of them are now at Lyft and they worked on Envoy there and Envoy is also super cool. So linker D is to Kubernetes as Envoy is to VMs sort of, cause you can still use it in containers but like it feels like a lot of Lyft stuff feels very VM centric. Even though I know that they do a lot of container stuff but also very similar. It uses a middleware approach. So it's less of a black box than linker D is. You can actually insert middleware like custom middleware that you built at your organization into the request chain. And that's pretty cool. Speaking of Lyft, they have an AWS metadata proxy and this is like, I fell in love with this so I worked at a company that did, like we were a monitoring company specifically for AWS. And so literally everything we wrote had to have IAM credentials. And so we were using like STS to assume role all the time and like you wanna test that. Like you wanna on your laptop be building this stuff and you wanna know it's talking to the metadata service and not just using like your IAM credentials and you wanna know exactly which credentials it's using and you can use the metadata proxy to do that. And also what I know that Lyft is using this for in some cases if they're still using it is to do per container or per pod IAM roles in like a container paths like swarm or Kubernetes or something. So you trick Docker into routing requests to the like 169 whatever address that the metadata service is listening on locally to the metadata proxy process so that it will expose things like the container name or some kind of metadata and then the metadata proxy will use that to identify the service and STS assume role to a specified IAM role like in an environment variable or something. And that's like super rad. And if you're using the EC2 container service now it actually does this for you. It didn't when we started so we were using the metadata proxy but it'll do this for you and you can do per container IAM roles but if you're building or using your own container paths on AWS definitely check this out because it's super sweet. SciCars are cool okay. I'm Grappler on Twitter. All of my references and my slides are at this URL. So I'll tweet this crap out too. If you have any interest in it if you wanna talk to me about SciCars that's cool. I'm sick if I don't shake your hand I'm really sorry I just don't wanna kill anyone. And that's it. Yeah that's all I got. Right okay well I mean fundamentally there's not really that much difference between a SciCars and a microservice except that like microservices don't necessarily have the locality or isolation component that a SciCars does because you can just as easily have microservices traversing the network to each other but like the specific thing about the SciCars is that it lives like the locality really is the big component there. Like think of like a Kubernetes pod right? So you'll have your app in a container and you'll have the SciCars in a container and those get scheduled onto the same host by Kubernetes. So that's I mean that's really the key difference. Yeah yeah sorry I'll repeat the question. The first one was like I still don't really understand the difference between a SciCars and a microservice and I was saying you know there's not much of one. The follow up was if you're not traversing the network like what IPC are you using right? Well that's the thing you can use anything. Like this could be any communication primitive offered by the VM itself so you could use UNIX sockets, you could use files, you could use shared memory but don't. But I mean you could use like mmapped files, you could do whatever you want right? Like that's the point. Like that, it's just a richer set of communication primitives. A lot of times it's like files on disk so like shared data volumes or something. Okay thanks. Give it up for Greg.