 Hi, what I want to talk to you about today is kind of a journey of an existing company into doing this, what is this cloud native thing? Apologize a little bit for the use of buzzwords ahead of time, but you know, you got to get that talk accepted. So just a little bit about me. I'm currently principal engineer at Mailchimp. I like to say that means I come up with all the bad ideas, so there are only good ones left for the other engineers. A little bit more about me. I've been a Kubernetes user since way before 1.0 came out, so I was there through some very dark times, and if you tried to maintain API compatibility, just a little bit about what I do at work and kind of at home as well. I want to try to somewhat hurry through this just in case we have any questions at the end. So let's go a little bit about Mailchimp, and is Mailchimp, and I have some stuff from marketing that I promised I would read, so bear with me for a second. Mailchimp is the world's most powerful marketing automation platform. Over 60 million businesses and individuals from community organizers to Fortune 100 companies stretched Mailchimp to connect the right people and the right message at the right time. Man said that we're basically sass to be the marketing department, if you will, for smaller to medium-sized businesses. Our average customer is a company that is probably selling goods or services on the internet, but they want to have email engagement, place ads, and that type of stuff. Mailchimp can kind of act like that, give them analytics, who saw what, who bought what, that type of thing. So that's what we do. Most people know us for email. It's right there in the name. To give you an idea of what we do, we send just over a billion emails a day, and for around the holiday season, if you can imagine, we actually do a little bit more than that. We usually have an annual report, so I'll wait until that comes out before saying the crazy numbers of emails that we sent in the last few weeks. We're hiring, I think, almost everyone in here is, but we have three physical offices, and we also have some jobs that are available for remote people. So if this interests you, or you think I'm doing everything horribly and want to come tell me how to do it the correct way, by all means, please come along. Before I get started in here, I want to say, do a special shout out to Team Pendo. What's Pendo? You can read it down there. As we're coming up with a name for the team and the tooling, we specifically didn't want anything nautical-related because that's been a little overplayed in this space. So if you want to look to see what that is. And this is members of the team that are current members, our executive sponsorship, and some of the folks who were on the team earlier and have kind of rotated off. So I want to say a special thank you to them. Anything that is good and correct about this project and presentation, you say thanks to them, anything that is bad, wrong, and you hate it, you can blame me for that. Just a word of warning, and I think most people here are going to do that. Anything I tell you is very specific to the context that we have. Your context may or may not be the same. So don't wonder exactly why we did some things and why we didn't do others. I'm happy to try it about that offline, but a lot of times it was because there's these years of context that I, particularly a company that's been around for a while where, you know, kind of doing everything what you hear at a keynote speech or a speech like this is not going to be a good fit for everyone. So I just wanted to get that out there. Talk a little bit about what the kind of current-ish Melchimp architecture is. And this is going to go over a lot of details, or some of my co-workers in the audience that are going to shake their head and say that's not the way we do things. But it's more to give you an idea of what we're doing. And as you know, because the real world is never as clean as it looks on slides, but happy to, you know, trade horror stories afterwards if you want. Melchimp, the kind of public-facing web app, which if you're a customer of Melchimp, you come, sign in, you want to create an email campaign, schedule some ads or something like that, is a typical LAMP application. It's a monolith. It's a very large PHP application. We have your normal things. It's Apache, ModPHP, we use memcache for caching. We do MySQL and fellow repairs, you know, so somewhat simplified diagram. But, you know, if you kind of think what was state-of-the-art or might have even been old, you know, 10-plus years ago, because that's, you know, when Melchimp, at least, the current kind of implementation of is about that old. So, like I say, that is a legacy type application, but kind of a very typical LAMP thing. Yes, you can have a successful money-making business on PHP, you know, regardless of what others will tell you. So, a little bit more behind the curtain. It's really not as simple as that showed before. If you want to look on the left side of this, it's more of a logical diagram. You're kind of going to recognize the blocks there. But as things grew over time, there were things that didn't quite fit into the monolith, and so we have these, we call them services, but don't think about them services like you're hearing in the buzzwords and everything today. A lot of these are kind of offline process and stuff. Don't necessarily fit into the user-facing direct flow or production traffic, but they are business-critical applications, and they kind of live off to the side. And so, we kind of an interesting thing that we have this monolith, but in reality we have these supporting services around it. And that's not actually true either, is we actually have 17 instances of the monolith back whenever MailChimp was growing like crazy. The easy thing to do, rather, and the most prudent thing at the time, rather than kind of figuring out how are we going to support twice as many customers on this platform, is like, what if you just installed another instance of the platform? And so we did that multiple times. It's actually somewhat interesting the way that we do that. Won't spend a lot of time on that. One of the unfortunate truths about that, since this is all hardware, as you can imagine, not all the hardware was bought at once, so all of it is subtly different. And also these shards, as we call them, each one has been a little bit of an opportunity to do an experiment, like, oh, what if we did database fell over this way? What if we tweaked this this way? What if we did that? So while the exact same code is running across all these shards, it may not be the exact same configuration, so when we say we have 17 copies, just being honest, it's kind of really like 17 apps that happen to have the same source code. So having talked a little bit about the kind of infrastructure, architecture talked a little bit about how does code get from a developer all the way through to production. This is primarily talking about that big lamp monolith that we're talking about, but most of these supporting services are similar, at least have a common building block, so kind of a normal person who's deploying code is going to follow the same flow regardless of the application. And once again, this is going to gloss over some details. And because the real world is not as clean as it looks on slides. So here we have our developer who's going to work locally using Vagrant. We have some custom Vagrant boxes that we build, which kind of have the supporting infrastructure components in it, MySQL database, Memcache, PHP, Apache, et cetera. Due to normal thing that you create a PR, it gets pushed up. We have Gingas us to in tests. We do code reviews. So this is very typical kind of in an enterprise type environment. We have this thing that is aptly named Deployer, which it deploys code. And really, it's just a wrapper around fabric. So underneath that, it's just doing rsync. And so we rsync every merge to master once this past test, et cetera, gets rsync to every application server in each one of those shards. So if you think, what's the most simple way that I could get some things deployed, it's pretty straightforward. And it works really well. But we do actually have said that every merge to master actually goes to production. So we are doing actually continuous delivery. And that's relatively recent in the last year or so. So what are some engineering expectations? Like I'm just telling you all this to give you an idea of the context in which we were kind of bringing what is cloud native into such an environment is that the developer, since we're developing in PHP, expect if I edit a file, I can immediately see the results in an API call, a browser, or something so that their iteration time on changes that they make locally in their local development environment is very, very fast. They expect it to be basically instantaneous. And that some subset of the tests can be ran locally. But that gives them just for basic things like planting, et cetera, just so that they don't have to push it all the way through the build process to see, oh, I didn't follow the style guide in that function or something like that. And kind of more from the operations and infrastructure side, master is always running in production. So that means that master needs to always work. And all the shards are running the exact same PHP code. And one of the ways that we're able to do that and push very often, multiple times a day, is we heavily use feature flags. And so we dart launch a lot of code and then use that to slowly turn it on for users across shards or something like that. So that's kind of where we're at, kind of the context coming in to do these things. So if we have that, and it's working really well, why would we want to change what we're doing? One of the things is in a lot of this, if you've talked to other people, it says, oh, I had a big monolith, and I wanted to break it into services. A lot of these are going to be the exact same reason, is we have multiple development teams all working in the monolith, and so you can kind of imagine if you have two features that are not even related, but they're both sometimes working on the same shared code, so you have those things. And the monolith is a pretty large and very old code base. And so you get this thing where sometimes to do a quick experiment is a lot of work, both on the development side and on the operation side, to be able to get something out quickly. So that's one of the big things we want to do. What's the time that we can go from a product idea to having it out there for users in a quicker amount of time. And sometimes those users are going to be those external users I was talking about, but some of them are also going to be like, what is an internal tool or something like that, because we try to as much as possible maintain parity as far as deployments, workflows, et cetera. I said earlier that we're actually running completely, at least this application, runs completely on bare metal inside of data centers, actually multiple data centers. And so we have the problem that on a normal day, the hardware is just very underutilized. But on a kind of busy day, we need more hardware, if you can imagine. So we're always kind of fighting like, well, how do we do scaling, because scaling when you're just running completely on bare metal is interesting. And we've actually started somewhat looking at VMs. But in real, that's just kind of a small evolutionary step. And then you have to run management stuff around that, et cetera. And so we're thinking about that, but we're also thinking about some of these other things, like what, instead of doing just an evolutionary change, what if we did a larger change? And we also have the thing that, like I said, our operating systems, all the way up through our PHP packages, the versions of those, the version of Memcache, et cetera that we're using, is deployed via Puppet, kind of normal way. But then we're actually deploying our application code using this other method, the deployer there. And so you get interesting things where, what if I need to upgrade this version of this PHP module, but I have this code that needs to use it. And so you get this interesting thing, like how do I deploy those in log stuff? And if you're doing it kind of managing those with two different things, or even if you're managing them with the same tool, whenever you're in that type of environment using kind of now what's viewed as kind of traditional or legacy, depending on how you want to call it, configuration management schemes, is against kind of difficult. And we have interesting use cases where deployments get interesting things where they're not fully atomic and such a thing. And it gets particularly interesting whenever you need to do multiple things within a single deployment because you're changing multiple things. And so that's just from a purely practical thing, is being able to do that. But also, and obviously in some like Kubernetes, it's a little bit easier to canary deploys, blue-green, that type of thing. And another reason is because these shards that we were talking about earlier with hardware and those type of things, is that you're somewhat fixed like this shard belongs to that hardware. And so as we're thinking through what is our kind of DR or business continuity planning around that, it makes sense if I can abstract that application away from the hardware it's running on, then I can move that application around much more easily. Like can I pick up that version, that shard, if you will, and move it to a different data center? Right now that would be very hard. And it's not just because I need to get the data over there, but I also need to get that exact same configuration going. And then also another thing is we're doing that, like how can I do standardized metrics, that type of thing? OK, so how do I get there? So we've done all this. This is the context we have. This is why we want to change. So what did we do? And this is still in progress, but how did it actually turn out in the real world? Because I'm guessing that's what most people want to hear about. So the phase one we had, since like say MailChimp is primarily on on-premise and data center thing, is we had to deploy Kubernetes clusters on bare metal into multiple data centers. So some things that we needed to do that as you're deploying Kubernetes, you'll hear all these things, is we decided that we're just going to treat Kubernetes like it's just another app that we do. So we're going to actually manage Kubernetes, the components of it, with Puppet. That's what we have. We have combined decades of experience with Puppet on the operation side of the house. So we're just going to treat it like that. We treat, like say, et-cd like just another database with backups, et cetera. Another thing that we do is we're using a lot of the existing tools and knowledge we have. We're able to leverage that. And really, from just a purely practical standpoint, probably, just to be honest, a lot of the kind of interesting new ways of deploying Kubernetes are not quite ready. Like if you played with some of those tools, a lot of them are still like pre-release, don't use these. They only support single masters or something like that. And so then we do the normal thing that we actually have some nodes that run in the control plane components. That's kind of one Puppet class. But we also run workloads on those as well. And we actually run multiple, kind of, they're fully independent clusters. We're not using federation or anything like that. I'll talk a little bit about that later. Our networking scheme, if you, kind of, what we're coming into this part of that context as well, is we heavily utilize CDNs for DDoS, caching if it's applicable, et cetera. Our low balancers for that public traffic live in a DMZ, our app servers live in some VLAN, and like our database servers live in another VLAN, and we're using network ACLs to, kind of, control access between those different layers. And if you were to walk into one of our colo locations, you'd see switches, routers, that type of thing, and you'd recognize the logos on them as the kind of typical things that you're going to run in more of an enterprise. So this is a little bit like, say, simplified, but of those who are kind of used to doing, like, three tiered applications in a more traditional environment that should look familiar. So what we decided to do, much as we did, was deploying Kubernetes, the components, and everything, we're going to treat Kubernetes as just another app. And so what we do is, in this case, these nodes here, like a lot like those application servers over there, so we're just treating Kubernetes nodes as application servers. And like I said, we did this to keep it simple. We didn't want to introduce this kind of interesting idea of what is this container networking, and then if you really get down to it really, you just need the pods to be able to talk to one another. And we already had this idea of front end and back end networks for various other applications. So what we did is that back end network is just what we are using for the pod network. And we just do use host routes between them. And if you can imagine that, so you can go into one of our nodes, you can type IP route, and you're just going to see all the routing between there. No overlay network is pretty straightforward for someone who's familiar with Linux routing, that they could just go in there and see it. We're using Flannel to actually manage the host routes. But in reality, we could have used a shell script to do it. Flannel is just so much convenient because if something changes, it picks it up right away. So pretty simple, particularly if you're running on prem. Authentication, this is one of those things where, also as we're bringing Kubernetes into an existing environment, Kubernetes supports using an external authentication. And one of the ways it does it is with OIDC. We use Google Apps for authentication for various things, and Kubernetes supports that enable it. Oh, we'll use that. But if you happen to know that at least Google's version of OIDC does not support group information, at least not directly, and we really want to be able to have that so that it fits into our normal scheme that these people are allowed to do this, service accounts are allowed to do that. And so what we did was extract that code, copy that code out, and made our own version of an OIDC authenticator. And Kubernetes has a scheme where the API server can ask via a webhook, this user gave me these credentials in an OIDC or just have a token. Is this a valid user? And is this user who they say they are? And so that's just code that's lifted straight out of Kubernetes Core, and we're doing that. And then we're using another authoritative source to get the group information and handing that over. This has been something that a lot of people seem to struggle with in the community. And we kind of did the simplest thing that would work for us to get that going. And what we have is kind of one of those groups that are coming from that store, mapped directly to a cluster role, or a namespace role if you're familiar with those for the RBAC inside of Kubernetes. And our audit logs for Kube API server, so that we can see who did what, flow through that same system as are all of our other logs. So that, like I say, we're just kind of integrating Kubernetes into our existing things. Speaking of integration, this is somewhat of a bad metaphor that's Freddie, our mascot. So the big Freddie is our existing infrastructure and the little one that he's integrating in this, sorry. Yeah. So one of the big things about bringing Kubernetes in there is we wanted to really treat it as just another application. How can we do this change but still support it so that people are comfortable supporting it like they do other things? No one likes to get page at three in the morning with an unfamiliar system. So at least with this, I say, oh, I know how to use IP route because I've been doing that my whole career-type thing. We need to integrate with other systems. Like I say, we're log shipping. We're currently using Fluent D and I'll reserve my comments, but we're using FileBeat to actually ship operating system logs because we use a rather hefty log process and pipeline. You can imagine how many logs that we produce in there. And so we have a very mature pipeline there. And so we're actually looking at, now that FileBeat has official Kubernetes support, to be able to add the Kubernetes metadata to the logs that are flowing through it is switching to that and just deploying that via Puppet because that's what the team that kind of owns and maintains our log process and pipeline is familiar with. So that's another thing where we're like, oh, how can we treat this thing in Kubernetes just like we do other things. We currently, and this is where I'm going to go also over a lot, to be honest, use Xabix. And then for Kubernetes, what we're doing is using Xabix for kind of like operating system down level monitoring, saying like, oh, I threw a hard drive or CPU or I got memory error. So we're still using our same tooling there. But we're using Prometheus and Alert Manager for kind of like Kubernetes and up, including the applications. There's experiments that have been going on since before I got to MailChimp that were actually investigating using that combination of Prometheus and Alert Manager to even kind of go further down the stack. So another thing that we needed to figure out is we have, you know, secrets. We have passwords for database credentials. So we started looking at it and everyone's like, oh, you need to use this product. You need to use that. And there's vault. There's this and that. And what we do for Puppet is once again, it's one of those things that is just very straightforward. What works very well is there's this thing called eYAML, which is based on this public private key encryption for secrets so that you can actually check them into a Git repo if you want to. And, you know, because you have to have the private half of that to be able to decrypt it. And it was somewhat fortuitous that Bitnami has this project called sealed secrets, which does that but for Kubernetes. So you have this controller that's running inside of your cluster. It knows about the private key and you can lock it down so that, you know, only it can access that private key. But you can use the public half of that key to encrypt secrets and put them in there and it will handle, you know, decrypting them, et cetera. So that concept, while not the same as Actooling, was already familiar to the engineers on the team. And we've actually had to make, most of these projects had to make upstream PR. So it's kind of one of those, to be honest, like everything that can work that great out of the box. And so we've had to do some things or it didn't work great for our context. It might have worked great for someone else's. So Ingress, what good is it if you've got this nice big Kubernetes cluster and I can't access apps in it, is what we do is we run HA proxy, which we had been running once again for years. When we run HA proxy, it's just a DOM layer four proxy. And so it's just a bit pump. But we already had supporting for like fell over, getting metrics out of that. So we're just kind of treating that like it was running in one of the other environments. But we run traffic, which is another open source project, which is as an Ingress controller. And we just run that on the node port service. And so HA proxy is low balancing across all of the nodes within the cluster. And then those get routed to traffic. Traffic is just another deployment. We're not doing anything weird. Like we're running as a demon set on each node or anything. So pretty straightforward. If we do it the way the packets go, yeah, sometimes you take the long way around with your packet, but it's very easy to understand. And at this stage of our deployment, we're much worried about is it correct and easy to understand rather than as a 100% performant. We're willing to make some of those trade-offs. We terminate TLS at traffic. And one reason we went with this two levels so that we could do that. And so that our Ingress controller would have access to the full Kubernetes API because one of the things we do is we have low balancers that are internal only and applications that need to be accessed from the public internet, or at least by our CDNs. And so we just use label selectors on the Ingresses if you're familiar with those. And so I just tagged this as a DMZ app and I tagged this as an internal only app. And so we're able to use multiple low balancers through a setter by running that inside of there, multiple asserts, so forth. And kind of at the kind of edge, then we're just using good OHA proxy to do that. Right. So now that we have that, how do we get applications out there? I can tell you about a lot of things that didn't work. And some of these might work for you. We had originally started playing with Helm and we kind of read the Helm documentation and went all in with all these nested charts and everything and that felt horribly for us. And we did some other things, but now what we've actually come back to, we've actually come back to Helm, but we're actually, each application and the same source repository as its application code, they actually have a Helm chart in there as well. So I'll go over a little bit of that. And so our actual usage of Helm is really kind of using it just for templated Kubernetes manifest. Okay, so what are we doing on it? Space with. For now, for deployments and Kubernetes, this, the general shape of this is the same as what you saw before, is a developer works in their local development environment. I'll speak a little bit about that later. Does a PR test run by Jenkins and Jenkins whenever those pass, rather than calling out to this deploy thing, we wrote this command line tool called Pando, which is really just a wrap around Docker build and some of the Helm commands. So once your build is finished, we pushed to Artifactory, which is our functions as our private Docker registry. And then it makes Helm calls to the various services. And I said before, we're not doing any federation. And so you can explicitly target one or more clusters. So if you have an application, these run multiple clusters, you just say explicitly, I want to run multiple clusters. Again, we're a lot more over of like being explicit rather than being magical. Okay, doing that, what's the developer experience? And the majority of time whenever we're talking about the developer experience, where they spend most of their day is writing code on their local box. So we were using Vagrant before and so we're using MiniCube for things that are going to Kubernetes. One of the interesting parts about doing that was how do I get that instant feedback that our developers were used to? And this is a little gross and I'll admit that is we actually have a flag locally, if you're deploying locally that will enable NFS using just the Kubernetes persistent volume that uses NFS that mounts back to the user's workstation. Since all this is running locally and so they can change a file, it reload or whatever and see their change in PHP so they have parity where they're existing. Expectations of development, like I said, it is a little gross because there's NFS involved. But it works that way and since it's just a flag in the deployment, we're still using the same deployment tools that we're using remotely. We tried some of the things if you've seen, for example, the way that Draft works that it will automatically rebuild a container whenever you change files. We tried some systems like that but they did not work so well. It was one of those they worked 90% of the time. You had it up in three seconds but about 10% of the time it took a minute. So the 99th percentile on that was very bad. What we do is like say the chart is embedded locally or embedded into the same application repository. Most developers do not care what is the deployment so we are developing some boilerplate ones to be able to say like, hey, I have a generic PHP application, just let me do that, copy it in. That way we don't have any kind of interesting dependencies between things. That does mean that whenever we decide we need to make a change to a chart or whatever we do have to go and open PRs on a bunch of repositories but once again, much more explicit versus magical. And we actually use the same tooling locally. I was saying that Pando command line tool also is used locally of when they're doing deployments and locally we also run an instance of Prometheus since it's pretty lightweight whenever you don't keep any metrics around and that way they can iterate on their application code if you need to iterate on some deployment stuff but also like some metrics so that I can set some alerting threshold or something. They can do all that locally which is kind of nice. So I call this 3.5 because a lot of this was all happening at the same time like how do we deploy real applications? We've talked about this and if you know, it's not great to just show toy applications. So the first application we chose was this very simple application we have. It takes snapshots of web pages for people. It turned out that application was not so simple and it turned into a complete rewrite and we were developing the tools at the same time. Turned into a giant bike shed and we still have not finished it. So what we really launched first, through a prior effort that log process and pipeline I was telling you about earlier, this very mature, it was already containerized and it was running on a different kind of bare Docker host and we had a team there that were very patiently and hey, we wanna trot this Kubernetes thing. We'll be your test subjects. So they were extremely patient but also extremely excited about doing this and this was real load, like business critical but not customer facing. So if we kind of broke it, it would be okay if we broke it for even an hour or so because users could still come to the site, just logs would kind of pile up. So we did that and right now at least the pipeline, the stateless part of it, if you will, runs a, we'll say 95% in Kubernetes and they're actually kind of for CPU cores own the most CPU cores there. We, MailChimp was using a different chat provider, one that's not so hip anymore but and we were moving to Slack and we had kind of a drop dead line on this and we have many, many chatbots. If you can imagine a company that's 10 years old or older and all the little chatbots we have and they're written in all sorts of languages, Python, PHP, Java, probably some other odd ball ones but we need to move all those over very, very quickly and a lot of those are maintained by different teams, individual development teams, et cetera. So what we did and we didn't do this for all of them but this was a very good opportunity, like as we're migrating those because we needed to run a lot of those in parallel because they need to work on the old chat system and the new one as we were transitioning we actually deployed the new version of those that were talking to Slack and the Kubernetes and they gave us an opportunity to work with a lot of different languages and frameworks and development teams and kind of figure out some of these tooling things that we were talking about earlier. So that was somewhat a nice thing about that was since it did have a deadline on it we couldn't spend too much time bike shedding like I was telling you about that other application. Tell you about something that happened kind of late breaking, this was last week. Someone said, hey, my deployment doesn't work. I get this error and then I look and it turned out that in that particular cluster we have just over 1300 CPU cores and they were all requested by application. Those know in Kubernetes if you do resource requesting say, hey, I need four CPUs or I need half a CPU or something. Say we're a little overzealous in what people were requesting for CPU cores and it was one of those whenever we were building this since like say we're kind of in this we're production but still kind of early production we don't need to worry about capacity planning. We have 1300 cores in that cluster. It'll be fine and it was not fine. So that's one of those things that we probably the way that we actually wound up solving it wasn't by adding a bunch more cores. Like I said, this is on-prem. So we actually went through and looked at what are applications actually using? What is our kind of like daily peak of CPU that they're using per application? Because what we had kind of done is did a couple of applications earlier and then everyone had just kind of copied that. And so we were able to shrink kind of the footprint there significantly. So that's one of those things that in hindsight probably it's obvious now that should have been one of the first things that we did but we didn't then you can learn from my mistakes that it actually does matter. Okay. So now that we're here and we're not done this is kind of still some preliminary work. Like what are some future things that we're gonna do? Is one of the things, everything that we mentioned here is things that could go down and the business would keep functioning. So it was a good way to test the application. It's out on Kubernetes without having to do a huge amount of work. So what we're actually starting to do at the beginning of the year what we actually just planned doing updates on one of our business critical applications. It's not that monolith I showed earlier but this is the application if it's down we our customers can't give us money so it's a little important but we're gonna do that that kind of update of that 100% on Kubernetes from day one. And so that's one way to force us to figure out any other issues is because this app is down or if this app is incorrect then ultimately we don't get paid. So that's one way to make sure that you do something correctly. And then that's really because it is architecturally a lot like the monolith. That's a good thing for us to start thinking about if we get through that then we can move that big monolith which is the real customer facing part of that application of MailChimp over. And another thing is since I showed earlier some of those supporting services we're gonna look at actually doing a service mesh in that and use an envoy, et cetera. She just told me my time is up. GRP servers and PHP and I don't have time for questions I don't think. Sorry. But maybe we do. Any questions? That's an interesting question. Right. I think it's one of those yes but that's probably just being realistic that's not gonna happen in the next year maybe not next to if ever. I'm just being realistic with you because really if you wanna think about the MTAs for us if you worked in Mail is there's a lot of interesting things particularly around IP addresses, et cetera that we would have to solve and really our kind of pain points are more in on that application side. So yeah. Yeah, saw on a road map but may or may not happen. All right, I think we're oh one more and then I'm getting kicked out. Yeah, right now for us I think Istio is a little too magical whether it injecting IP tables, rules, et cetera but we're just beyond proof of concept with just doing some things with Envoy. There's been a couple of projects that are came out recently. There are a lot more interesting and I think a little bit more our fit because Istio is kind of like all the way over here and we need like here. So all right. One more and then I'm really gonna get kicked out. We want to treat those as separate concerns I think that will happen because I don't think we want one to be blocking the other but obviously it makes it easier if you're already on Kubernetes to break it up because you have somewhere to deploy that and then if it doesn't go well you can kind of undeploy it for lack of a better word. But yeah, that is part of the plan and really a lot of it is as we need to go in and like revamp various parts of the app is to do that as a service. All right. And now we're really done. Sorry, I went over. Sorry.