 So, thank you everyone for coming. I've been, so DataWire we've been doing microservices for almost three years now. We've built some microservices internally. We've also consulted with organizations doing microservices and we also make open source software for making microservices easier on Kubernetes. And so this talk is kind of a distillation of everything we've tried to learn and sort of compile into something that kind of makes sense hopefully around building developer workflows on Kubernetes which is a big topic and what one of the things we found is sort of a underrated portion when you start thinking about adopting microservices. So, totally fine interactive so you don't need to wait till the end to ask questions and if I think I already answer it in the presentation later on, I'll just tell you and then you can always ask the question again. So, so with that actually before I start actually, how many people here actually build cloud software versus on-prem? Okay, so like half maybe? Maybe more than half? Okay, so the question I think everyone always faces is how do you actually build better software faster? The framing of this talk is really around cloud software because I don't really know very much about building on-prem software anymore. It has left my brain. And so, in the answer that everyone knows is all hot and exciting is instead of one rocket launch you do multiple asynchronous launches which people call microservices, right? And when you think about microservices what are you actually doing? And what we find that companies that are successfully adopting microservices they take your standard development process, right? Where you have a product organization that defines product requirements and they talk to customers hopefully and then you have some developers and a development team that actually writes the code and then you have a QAT maybe sometimes and maybe these are different roles and not necessarily different people, right? And then you have a release team, release engineering and then you have an ops team that sort of runs it in production, right? So you take this flow that you're running your entire engineering organization through, right? You have a release defined for the Q1 release if you're using time-based releases you have this release train concept, right? And you code to this release day and then you go to a feature freeze and then you test this thing. This entire cycle that everyone here is super familiar with, right? And what you do with microservices you make it distributed, right? So microservices you eliminate that central release cadence and you have each person or each team working on a service operate independently. And what this means though is a huge organizational cultural shift because your team actually needs to have all the skills and knowledge to operate all aspects of that development process, right? So your team is now responsible not just for just writing code but they also have to figure out how to test it. They need to figure out how to release it and they also need to figure out how to operate it in production, right? Because that's what you're actually shifting their responsibilities to, right? And by doing this you actually improve velocity because now no team is actually bottlenecked on some other team, right? So and the key is you need to think about this distributed nature of it's distributed systems all over again, right? Because if your team is actually bottlenecking on a central resource that team is no longer able to operate independently. So the definition I like to use for microservices is that it's a distributed development process for cloud native software, right? So a lot of people will say microservices architecture. I'm like, I don't really like microservices architecture because that implies that's where you need to start. But really your architecture should actually support your process and not the other way around. So really think about how you're actually doing development, right, how you'd like to do development and then figure out an architecture to actually support that. So we certainly talked to lots of organizations and they start thinking about microservices and they start thinking about Kubernetes and they start thinking about GRPC and Istio and all these sort of fancy technologies that are very cool. And we try to sort of turn that around a little bit and say, well, how do you actually start thinking about your workflow? Because that's actually what you're trying to do. You're trying to make development faster, right? And that starts with your workflow, not your architecture. So how do you actually, how do we actually suggest you actually get started? So if microservices is distributed development process, then you should take advantage of that and actually create a new team, right? And I like to think of it as a spin-off, right? So you take your existing engineering team and you create a spin-off, whether it's one person or a team or whatever and you say, just create a service. You figure out what that service is actually supposed to do and you access it through an API. You'll probably have existing code with your monolith and just like you access any other API service, whether it's Twilio or Stripe, you actually access that new API service through its API. You already have a way to access Twilio, you already have a way to access Stripe, so just use the same systems to access that new service, right? And that team creates a new API and they're allowed and empowered to use whatever process they can't, right? They're not beholden to your existing process, right? And that's the key. However, of course, if you just put a bunch of developers and say you're just in charge of operations, you've kind of got a recipe for disaster, right? So you actually want to think a little bit about how you actually make them successful. And there are two things that you actually need to think about to make this team successful. One is this team needs to be self-sufficient, right? So as soon as this team needs to talk to someone else, they're no longer self-sufficient. And if you think about being super productive, when you have to ask someone else another question, then you're not being as productive now. It's great to interact with people and I'm not discouraging that, but every time you have to open a ticket with someone else or wait for QA to do something or wait for a customer to get back to you, it slows you down, right? And so you really want to think about how do you actually be self-sufficient so that you can maximize your forward velocity? But self-sufficiency isn't actually enough, right? You also need to think about the fact that your team doesn't necessarily have expertise in every single one of these functional areas. And just like Karen's really amazing talk a little while ago, you actually need to figure out how to provide safety in your systems, right? Your systems need to provide safety so that if you have an ups in production and everyone makes an ups in production, hopefully you minimize the impact of that ups, right? So you really want to figure out how do you build systems that provide self-sufficiency for teams as well as safety? So of course this sounds very complicated and a lot of work and so the question is, well, do you have to build all this stuff from scratch? And luckily the industry has actually come a really long way. So we're big fans of the CloudNative Compute Foundation which is part of the Linux Foundation which has done an amazing job of identifying all sorts of emerging technologies that CloudNative Accompanies should really look at. And I'm gonna highlight three of them that we believe are sort of foundational components to provide that safe self-sufficiency. I like alliteration. So the first one is Docker, right? So how many of you guys use Docker today? Okay, so everyone knows about Docker. Most people know about Docker, right? It's got a massive community ecosystem, lots of third-party tools. It lets you build and package and deploy your service and its dependencies in sort of a consistent way that you can run in a whole bunch of different places whether it's your laptop or your VM or in Kubernetes, right? So Kubernetes, so Kubernetes, how many of you guys run Kubernetes? Okay, not as many. So Kubernetes, for those of you folks who aren't super familiar, is it lets you run Docker containers in the cloud or on your own bare metal. But one of the ways I like to think about Kubernetes is it's not just a container scheduler. It's actually more of more kin deposits, right? For the cloud. So by this, I mean it's actually an operating system for how you run all of your cloud infrastructure and it provides this YAML-based syntax defining how all your cloud infrastructure is supposed to run from how many containers you're gonna run, how do you actually update these containers, how do you store secrets, how do you mount persistent volumes. It does a lot of things that you might not necessarily associate with container scheduling, but it actually lets you basically define all of your cloud infrastructure in this hideously complex YAML file. And a lot of people ask, well, isn't there other stuff other than Kubernetes out there? There's MISOs, there's Nomad, there's Docker Swarm. There's a whole bunch of things to which I think that Kubernetes has actually won this sort of container scheduling war. First of all, it's the fastest growing project in the space in terms of contributors. So it has the most community momentum. It's got some massive companies behind it. Google started it, but it has branched out so that Red Hat is a core contributor. There is Microsoft as a major contributor. Oracle just announced a couple of weeks ago that they are all in on Kubernetes, right? And what's astonishing is that if you look at sort of the commit ratios, no company actually contributes more than 30% of the commits to Kubernetes. And the big company that's not on the Kubernetes, like Bandwagon, really kind of is Amazon, but 62% of production Kubernetes workloads are actually on Amazon. And that's because the reality is, back in the day, all these cloud software companies did all the stuff on EC2 and they used RDS and all these fancy Amazon stuff and that's where the data is stored and now they're moving to Kubernetes and so they don't wanna migrate databases. So therefore they're deploying Kubernetes on Amazon. And MISOs is another company they just announced, I think it was two weeks ago, that they've been engineering for however many months Kubernetes on DCOS. So in other words, with maybe the exception of Amazon, every other major player in the cloud ecosystem is embracing Kubernetes and even Amazon, inevitably they're actually blogging about how you do Kubernetes on AWS. So I think from my perspective, if you're actually thinking about container scheduling, Kubernetes is the obvious choice. And the third technology, which people are probably least familiar with, has anyone heard of Envoy? Okay, so actually a good number of folks. So Envoy is a layer seven proxy. It was written initially by the engineers at Lyft. You can think of it as sort of a modern updated version of HA proxy or Nginx, but it provides a lot of the functionality you need for a distributed cloud architecture. When you have multiple services and multiple containers talking to each other, that all happens typically over layer seven and so you actually need some support for doing more sophisticated things at layer seven that you previously didn't. Things like global rate limiting or circuit breaking or monitoring of those metrics. And so Envoy actually provides all that and Envoy actually has also been gaining a lot of momentum and was just accepted into the CNCF a few weeks ago. Lyft obviously is a big contributor. There are actually a couple dozen engineers at Google actually based in Kendall Square here that are actually working on upstream enhancements to Envoy so they can swap all of Google.com to be running on Envoy. There's IBM is actually investing in it. So there's a whole bunch of different companies actually working on Envoy. And so I'm gonna actually show how the combination of these three technologies actually let you do something interesting. So what I've been talked about so far. First is microservices is a distributed development workflow that hopefully helps you go faster if you think about it. You should start your microservices journey by building an efficient workflow for your first services team and then think about your rest of your organization. And then if you combine Kubernetes, Docker and Envoy it provides the foundational components you need to build that workflow. Okay, so let me talk a little bit about a workflow using these things, right? So typical developer workflow starts with you need to bootstrap your service, right? You need some sort of template or something like that. Then you actually write some code, then you actually run your code, maybe in a development cluster, maybe somewhere else, right? And you kind of do this over and over again until you actually have something that is ready for broader testing, right? And so at that point you actually deploy to a production Kubernetes cluster and then you hopefully have some sort of monitoring or analytics to actually see how well that thing is running and then that itself becomes your production workflow where you go back and you get some feedback from customers or metrics that crashes in production or whatever and then you write some more code, test it and you go through this over and over again, right? So how do you actually maximize productivity in this? And the answer is you wanna actually go fast, right? The faster you can go, the more productive you are. It's kind of like going from five minute compile times to five second compile times. There's sort of a asymmetric jump in developer productivity. Similar concept here. I think the point I'd emphasize is that the more self-sufficient you are, that's actually where you're driving the productivity. It's not necessarily just about just how long it takes for you to actually mechanically deploy something in production. It's like how self-sufficient you actually are. So I'm gonna walk through each of these steps in a little more detail. So with an emphasis on how do we make these things faster, right? So the first step, you bootstrap the service. Typically what people do is they create a template, right? And you don't need to use anything super fancy. You can just create a git repository with your example code and you can just tell everyone just clone this repository, right? There's no fancy technology here. And typically in your template, you'll need a Docker file which specifies how you actually build your code in container. You'll need a say a Kubernetes directory which stores your Kubernetes manifest, which is a fancy term for how you actually run your particular service in Kubernetes. And there might be more than one file which is why you put in a directory or you can put in one big file. You might have another directory which specifies how you actually provision non-Kubernetes resources so you might have a Terraform directory. You would have your application stub and then frequently you might have a YAML file that sort of specifies metadata about your application self, like the name of your particular service or whatever, right? So that's the first thing. So you wanna create a service template that people can just clone. The second thing you wanna do is you wanna build a productive development environment. And this is actually kind of tricky because if you're doing a multi-service, multi-container environment, you usually might have like five, 10, 15 services that are running around that your service might need to talk to, right? And so how do you actually set up a good development environment for that? And there's usually this trade-off between sort of local development where if you run everything on your laptop, you get really fast feedback, right? Because you make a code change and you hit reload and then boom, you can see how everything works, right? The problem of course is that if you run 15 services on your laptop, it might melt, right? And then you can also run everything remotely but then you typically access that through some sort of CI CD pipeline. And no matter how fast you make it, it still takes a few minutes to run so you really don't want to take like two minutes for two minutes to wait for, to see if your two-line code change actually works, right? So people do all sorts of different things. Typical design patterns is if you run the system locally, people use Docker compose, which is really nice because it's fast, it's lightweight. Most people already have Docker running. The trade-off is Docker compose isn't exactly like Kubernetes so you create this environmental difference between dev and production. So people then look at MiniCube. The problem with MiniCube is it requires a VM and especially if you're using JVM inside the VM, your laptop melts pretty quickly. So that's sort of the trade-off. You can run your business logic locally and then all your databases and all this sort of stuff in the cloud. And some people do this. It's a little bit more work around setup but it's sort of more resource efficient. Another sort of, the next model is you cannot run a single service locally and then use network proxies with something like telepresence to actually remotely proxy all this stuff into the cloud. And so you might have like your 15 services running in the cloud and one service running locally. So that kind of means you can't operate disconnected but it actually gives you that fast feedback and you don't worry so much about resource consumption. And probably where most people start is you start with just all remote development. You have Kubernetes with a CICD pipeline and that actually works. I mean, because you generally need a CICD pipeline anyway. So this is probably the least amount of effort to actually set up. It's just that you, from a developer productivity standpoint, you know, you make a code change, you have to wait for the pipeline to run before you actually can test it out. So you have a development environment. You have a service template. You need to now deploy your code to a development cluster. So this is actually, it's kind of complicated but it's eminently scriptable, right? But I'll just sort of walk through the mechanics of this. So you want to build a container image that contains your code and all the dependencies and this is pretty straightforward with Docker. So you just call Docker build, right? Then you need to tag the image. So that way you can actually roll back to this image. So you always have a semantic way to say this is the version that I really want which you can do with Docker tag and if you're clever you can actually do Docker build dash T and pass the tag in. You push the image to a container registry once it's actually built, which frequently lives in the cloud and this again is Docker push and you can see these are all sort of one line commands and this is my point around how Kubernetes, Docker and Envoy actually give you most of what you need and you just need to write a script around some of this stuff, right? You need to update your Kubernetes manifest so it references the image. So this is a little bit trickier because your Kubernetes manifest is a YAML file so you actually need some sort of templating system. Nothing super sophisticated, you can even use said or some sort of regular expression to actually deploy this and then you need to tell the cluster to run this Kubernetes manifest and what's gonna happen, you run this thing called kook how to apply and what it does is it actually downloads the image, it looks at all the configuration information in your Kubernetes manifest and it says, oh, I need to run five instances of this particular service and it will actually do the right thing and then you just need to repeat this for all your dependencies because if you have a service that depends on three other services, you need to run this sort of in some sort of recursive kind of way. So it's a lot of steps but you can see that mechanically it's straightforward and for you to be self-sufficient, you wanna put this in some sort of script so that you don't need to necessarily remember all the details of this and if you've built a CI CD pipeline to do this, the CI CD pipelines more or less do the same thing. So you've written all this code, you've tested it, it runs in your dev cluster, you wanna run it in production, right? What are the differences? Well, so more or less the same thing, you do the same thing mechanically so this is where script is helpful because you wanna actually do things the same way. One nuance is that when you get to update the Kubernetes manifest, you might wanna put in some more production configuration information so Kubernetes lets you specify, for example, the RAM limit and how much memory you allocate to your particular container and you might have a lower threshold for your dev cluster than for production, right? So you might wanna update that and I'll show you a little bit more about why this is important. You're also wanna start thinking about deploying Envoy as part of your production infrastructure so that you can do something like canary routing and how many people are familiar with canary routing, canaries? Okay, so for the folks who aren't familiar, there's a whole bunch of different terms for this and there are different flavors of this but what I mean by canaries is basically routing some percentage of your production traffic to the new version as opposed to doing this huge cutover where you basically route, you go from 100% to version one to 100% to version 1.1, you do 99% to version one and 1% to version 1.1 so that if 1.1 is going haywire, it's not affecting your entire site, right? So you wanna actually, and to do that properly, you really need a layer seven router like Envoy. The other thing is you wanna make sure you keep the same environment for same environment and deployment process for Dev and production to eliminate differences because you don't want the, it works for me, conversation. So then you apply the Kubernetes manifest to the cluster and then you pre for all the dependencies including Envoy and Envoy actually deploys the same way as any other service. And then finally, you wanna analyze metrics. So you really wanna deploy each service behind Envoy because it actually provides, it automatically starts measuring basic layer seven metrics like throughput, latency and availability. So you don't need to figure out how to instrument your existing service for this and then you can add service specific metrics as appropriate and you shove this all into Prometheus or Datadog or I think there's like five companies out there that all do this, right? So talk to any of them. So I have no idea which is better. And so what happens is you end up with this, this sort of architecture for distributed development. You basically have your monolith. It talks to an Envoy which is a layer seven proxy which provides the canary routing. It might do authentication because if you're doing an API service and it's independent you want it to authenticate requests only for your monolith as opposed to random people. It does, it maps incoming requests to your particular service and it provides layer seven observability and seven people might note that this looks suspiciously like an API gateway and yes, it is actually an API gateway of sorts. It does API gateway type functionality but you basically want a sophisticated API gateway that supports these general things. So once you actually have a single team that actually adopted this you wanna start thinking about how you get other teams adopting this and the key thing is you wanna make sure that because each team might be actually at a different point in their development process you wanna support a model where teams can simultaneously be one team might be developing and prototyping and another team might be actually running in production. So you wanna be able to support any of those particular use cases for a given team. So the fourth thing we just talked about hopefully to summarize is to maximize productivity you wanna optimize that loop and that loop is making your workflow from source to getting feedback I'm not saying into production I'm saying literally to getting feedback whether it's from your end users or seeing if your code runs or whatever you wanna really optimize that and you wanna optimize that by making people as self-sufficient as possible. Okay, so safety. So this is, I actually had this before the previous talk but I think it's sort of a good segue because everyone makes mistakes, everyone makes oops and so I'm gonna, if the wifi holds up I will do a little demo to show safety because I think it's easier to show than to actually talk too much about. So what I'm gonna show is actually a sample multi-service application running Kubernetes in this case I'm using Kubernetes in Google Cloud and I have Envoy as my layer seven router it's gonna send metrics into Prometheus it's gonna talk to this stub authentication service to do some hokey basic authentication and it's got two microservices, the task service which talks to MongoDB instance and a search service. So you can see like there's like four or five things that actually need to be deployed. So let's see. So and you can see like in the director and I have, this is on GitHub so if people wanna try it afterwards. So what I'm gonna, oh actually so what I'm gonna do is is actually, let's see if this works. Okay, so I have a Kubernetes cluster the only thing that's running in it is the database which is MongoDB because MongoDB it turns out doesn't actually respond nicely to me taking it up and turning it down. So I'm cheating a little bit here I'm actually gonna deploy all these services we have the script that we wrote called Forge but it basically does all the things we just talked about it does the Docker build for everything it does the recursion and you can see it's actually run everything. So if I do get services you can see all the stuff is running or it's spun up and all these pods are running and what I can do is I can actually open a remote shell to the cluster which will take a minute because it's gonna spin up a VPN connection and what I can do once this actually spins up so what this is actually doing is it's creating a local proxy from my laptop to the Kubernetes cluster so that because some of my services aren't actually exposed to the internet so if I curl to the search service you can see that it curls the search service if I curl to the task service it actually operates. So those are services running in the cluster and this is my very clever segue because it takes GKE a few minutes to actually give me an external IP so you can see, oh so I have an IP for Prometheus and I'm still waiting for that for my API gateway we'll see if it actually comes up. Okay, here we go, so what I will do is so if I curl to the gateway URL slash tasks and I enter in, see I have some hokey authentication you can see if this actually, demo gods smile at me okay so there you go. So it just, so I've just, what I've just done is I've deployed a bunch of services and I've actually exposed the task service and the search service live to the internet through this API gateway and this API gateway is actually Envoy, right? So what I'm gonna do is actually I'm gonna actually generate a little bit of load and you can see this is my load script and all it does is it, it's very dumb. I'm gonna just run this in a loop and the way I'm gonna demonstrate safety is I'm gonna actually show the source code for the tasks application here and in here you can see that I have cunningly put a sleep so it affects performance so I'm gonna implement a massive performance optimization by changing this and deleting the sleep so this app's gonna run much faster now and however since I'm not sure if anything's gonna actually break I'm actually going to actually do a canary deploy of this and this is why canaries are actually so valuable, right? So I'm gonna actually deploy this thing and you'll see that this little, well if you can see there's a new service called tasks canary and that's running and what I can do is I can actually, let's see where's my Chrome here. I can actually go to, all right so in Prometheus, I'm just using Prometheus here and I don't really know the query language for Prometheus so I'm gonna execute this query and this is actually taking latency and if I zoom and I hit to say five minutes so first of all this is, it's a little hard to see with all the scrolling but basically it's automatically collecting metrics from the API gateway that's sitting in front of the services and notice that as a developer I didn't actually do anything this is just a standard Python web app but it's automatically collecting metrics and let's see if I can, here we go and what you'll see actually which is more interesting it's a little hard, can I zoom? So what you'll see is you'll see that I actually have at the very top these three graphs they represent P90, P95, P99, latencies and the very bottom is you can see at the top that was my original version which was exhibiting 600 millisecond latencies more or less and then at the very bottom is my canary which is exhibiting much lower latencies and so the reason why you do canary is you can actually deploy, route, in this case we're routing 10% of our traffic to the canary and we're actually looking at the graphs and we're saying, okay, nothing's breaking, nothing's crashing, actually performance, throughput, latency, whatever is actually improved and then you can actually do the cut over, right? And so this is sort of, and this is just done with just Envoy, Kubernetes and Docker and a few scripts to kind of pull this all together but this is sort of the power of actually building a developer first workflow because if you think about it well I'm not really a developer, I only kind of program but if you sort of pretend that I'm a developer I actually haven't actually put in any sort of metrics infrastructure, I haven't actually, I actually haven't put in the API gateway, I'm just deploying it just like I would any other service and I've basically been able to cargo colts all this stuff from a Git repository. So, so basically I've actually, well this actually, so I go back and forth so sometimes I have the sleep already in there and sometimes it's coming out so I didn't actually, I didn't push a buggy update, I pushed enhanced update into production but the idea is sometimes you have that trepidation when you push this button and you're not actually sure if it's gonna work with Canary workflows, you actually have this assurance of safety such that if it doesn't actually work you're not screwing everyone, you're only screwing a few people and that actually feels slightly better, right? This is not healthcare so you're not gonna, in healthcare you probably don't wanna kill anyone but this time this is a little different. So essentially what I'm hopefully, you're taking away from this talk is that having an efficient development workflow which lets teams be safely self-sufficient is able, lets you actually move really fast and actually not break things and that by being able to create this distributed workflow you can have all your teams actually rapidly building different services very quickly using pretty vanilla off the shelf stuff. So anyway, so that's the end. I'm actually not sure about time but I wrote an article that actually walks through this entire Canary workflow which has a link to the Git repository if you wanna play with it yourself. We make open source tools for Kubernetes which I sort of demoed but not really because that's not really what I was trying to do if you go to our website. We're also looking for a C++ engineer to work on Envoy, it's all open source so if anyone knows of C++ engineer who wants to work on network proxies, we're hiring. Anyway, thank you, so have you taken any questions? I've listened to the presentation and I'm still scratching my head trying to understand what Kubernetes is what it does for me, is it an orchestration tool or is it, it's just not clear to me. Okay, sorry, so I probably didn't do a good enough job of explaining, so the simplest way to think about or the way I think about Kubernetes is you have some service or a set of services that you put in Docker containers and you wanna run them and Kubernetes actually runs all your containers. So in the demo that I showed where I'm sending a request to the API gateway and that API gateway is then sending that request to my web application, all that stuff is actually running in Kubernetes, right? So you could think of it as an alternative to having a bunch of EC2 instances floating around. I'm happy if anyone else wants to try explaining. So what Joe said was it's a better way to run virtualization akin to vSphere, so more efficient way to run the data center. Just out of curiosity, you were running that on hosted Kubernetes in Google? Yes, yep, all on GKE. So that canary deploy, I totally get how it works for maybe a web worker or something that's stateless. What if you wanted to deploy a data service so you wanted to update Postgres, which is a good integration or something? So there's a couple ways to handle that. It's a great question. The way that we actually are working on, we're actually working on Envoy upstream, so this is one of the things that we want the C++ engineer to actually support to work on, is Envoy has a feature called shadowing, and what that lets you do is actually take incoming production traffic and duplicate it in real time. So then what happens is you basically deploy your stateful service, you implement shadowing, and you actually can then route one copy of data into real production, one copy into your test systems, and you can actually do even cooler things where you actually can, the way you really want to do shadowing is you want to actually split it three ways. You're incoming traffic three ways, and you have two versions of your production service side by side, and you diff those two, the requests, and then you also then diff your test service versus your production service. And the reason why you do this is you want to eliminate stochastic noise, and so then you actually can actually prove in a fairly automated way whether or not your new service, new version of your service actually seems to be pretty close to that of your existing service. But you're right, so the Canary Eye show where it's just rerouting 10% of your traffic or whatever wouldn't work very well for stateful services. I have a question more about your company DataWire doing all this really cool open source stuff, as do you guys provide any hosted products as well? We don't provide a host of products. We're basically all about just making the developer's life using Kubernetes easier, that's it. I mean, we have a hosted service, but it's really for, called Kubernetes, which lets you do on-demand ephemeral Kubernetes clusters so you can run regression tests in your CI system. But that's, but I'd say that's, but it's not a hosted offering. It's just basically, we actually just wrote it for ourselves. It's not something we charge for anything. It's just this, we run regression tests on telepresence and we use Travis and in order to run your regressions, you need a Kubernetes cluster and it turns out it takes like three minutes to spin up a cluster in Google and then it doesn't go away. So then we just wrote our own sort of orchestration for that. Quick question with the mic. With Canary workflows and specifically at an organization that hasn't really used Canary workflows that much and we would have to take different amounts of effort to use them in different applications. For example, using them to test our database infrastructure would be difficult or not reasonable. Where like the low hanging fruits, which types of applications really benefit from that? Like holistically, quick rollback or applications that fail frequently. Yeah, I mean, I think the easiest is basically more of your stateless services because you don't need to implement as much infrastructure. I think the biggest bang for the buck would be services that you update frequently that you're worried about impacting and users for. So any services where you're like, oh, I'm very nervous about actually updating this. That's actually a good candidate. There's this e-commerce company that we worked with and one of their big metrics is conversion. So the percentage of people who, when they put something in your shopping cart, what percentage of those people actually pay you money in the end? Go all the way to check out. And so when you update the shopping cart, you don't really want conversion to change. You only want it to go up because you lose all this money if it doesn't. And so that's a perfect candidate for Canary because you basically have this enhancement to shopping cart. You roll it out, you canary, because it's so sensitive to your overall business. And there was a really good talk, I can't remember who, earlier who talked about sort of debugging and he talked about how important is the business metrics, right? And I think you really want to think about your business metrics, things that affect your business metrics or the things that I would start with Canary. If there isn't too many other questions, I was hoping you could go into a little more depth on Envoy and what it features, I'm not familiar with. Yeah, so Envoy is, so the way I think about Envoy is it gives you all the basic functionality you would see in an NGINX or an HA proxy around routing. It adds on global rate limiting, circuit breaking, all these sort of resilience to advance load balancing where it can different load balancing algorithms. So that's sort of a category of stuff. It offers sort of native HTTP2 and GRPC protocol support. That's sort of the second thing. The third thing is, as you saw, it provides very rich observability through StatsD, so it collects all this observability data automatically and pipes it out through StatsD interface. And I think the most interesting piece is traditionally when you look at all these sort of web servers, they are managed through a static configuration file, nginx.com or haproxy.com or whatever. And that works reasonably well when you have like a few, few sort of a few proxies that you're floating around, but at Lyft they have thousands of proxies, and you wanna basically start orchestrating them together. And so there's a set of dynamic APIs so that Envoy proxies know to actually continuously pull for new changes to say your endpoints, so they can actually do dynamic routing. And so there's a whole bunch of, there's probably I think six or seven APIs that Envoy natively supports designed so that when you deploy groups or clusters of Envoy's, you can actually have more centralized management of the servers. And if you're an nginx shop, nginx just rolled out, just announced the nginx application platform, which includes nginx plus, which has some of the stuff they're sort of catching up maybe, but it's all sort of proprietary and Envoy is 100% open source, so. No, I'm just gonna, I'm just gonna help you have a graceful exit, but no, it's all right, you can gracefully exit yourself, I'm 100% positive. But thank you, this was, I learned a lot, so I appreciate it. All right, thanks everyone. Thank you.