 So, before we get started, I'm just curious how many of you out there have migrated your clusters or systems to Kubernetes or Docker? Anybody? No? Cool. So, basically, in this talk, I'm going to talk through what we used to do, why we decided to move to Docker, why we moved to Kubernetes, the problems we faced, and sort of the system that we have now, which works across multiple different cloud providers, and we can install it on on-premises as well. But first, a little bit about me. So, I'm Patrick McGuigan. I work at Algorithmia. We run a mixed hardware cluster, primarily on AWS with CPUs and GPUs. On a daily basis, we create and destroy some 10,000s of containers. I don't know the exact number. And I also ran our migration to Kubernetes in late of 2016. What we do is we're basically a serverless AI cloud. The idea is that we have a common API for algorithms, functions, machine learning models to run as microservices. Users can create algorithms in seven or eight different languages. Whenever they push to a Git repo, it automatically creates a version API endpoint, which then anybody can call. It's set up as a marketplace, so users can publish publicly if they desire to. You can set a royalty fee, or you can make it private. You can share it within an organization. There's a whole lot of different options that happen there. The key is that we will scale our cloud, both the number of replicas of a particular algorithm that exists and the number of servers that we have running at any given point in time based on demand as it happens. So for example, we have this one algorithm for doing filtering, like you might see in Snapchat or anything like that. It has a whole bunch of different possible filters. So you take an input picture, you give it a path to save it, and one of the filter names, and you get this nice little stylized picture of my dog. Thousands of algorithms out there, there's 50,000 different users. Anyone can create anything. Some things are utilities or sentiment analysis, NLP. It kind of goes all over the place. But we do support all the major deep learning frameworks, TensorFlow, MXNet, Cafe, Torch, I'm sure I'm forgetting some, but the point is we're trying to do anything. So with that in mind, I'd like to first start by saying what we used to do, and then why we moved to Docker and Kubernetes, getting into the problems, and then how we generalize for multiple cloud providers. And then I'll finish up with the things I'm sort of excited about with Kubernetes. I'm sure you've heard a lot so far this week on where things are going, and I'll talk a little bit about what I'm personally excited about. So this may look familiar to you. It's a pretty traditional web application diagram. We have DNS that has entries for the website and for the API servers. Those will point to load balancers. We're using AWS, so that's ELB. Behind the load balancers, you have a bunch of virtual machines that are running servers. For example, a web server may be on port 9,000 and an API server on port 9,100. When an API request comes in, it goes to the load balancer, round robins to one of the virtual machines that does some authentication, and we have our own scheduler to figure out where we actually want to run that work. It's pretty standard, but just in case some of you haven't experienced this before, here's how a deployment might work. So first of all, we have to use right a whole bunch of our own small bash or Python scripts to do each step of the deployment, which was managed by a third party called distelly. And the way that would typically work is you have our three servers. We pick one of them to deregister from the load balancer. The load balancer will stop sending new connections to that server. We wait for work to drain off of that machine. Whenever it's done servicing all active requests, it gets registered fully. You can then update the application binary and then reattach to the load balancer and start responding to health checks. After 30 seconds or whatever a configured interval is of healthy things, then the load balancer will start sending work to it again. This is pretty standard. This is what people have been doing. There's some variation of this for 10, 15 years and any sort of web serving architecture. But it has a whole bunch of limitations to it. In particular, you can really only run one copy of a service per machine. And the big reason for that is if you have a server running on port 9000, there's only one port 9000. And if the load balancer is configured to ping port 9000, well, you're kind of out of luck if you want to run multiple copies. You can do things like run HA proxy to balance between ports or something along those lines. But it starts to get a little bit more messy and it's usually just easier to create more machines running one copy of the service. Additionally, it can be really difficult to add services. You have to install libraries on the host. You may need to configure particular directories. We have to write the deployment pipeline. We have to figure out what's going to monitor it. If the process crashes, how does it come back up? How do we know that it crashed? There's a whole host of problems there. On top of that, like I said, we have a public marketplace. But the customers came to us who a public marketplace doesn't always work for them. They may have data privacy requirements such as a HIPAA requirement or maybe they're in EU and there's data privacy laws where you can't send data across or outside of the EU. There's also bandwidth or latency requirements. Users who are sort of in the big data space, they have all of their data in a certain data center. Why should they send all of that from, say, Europe to our data center in US East 1 just to do some processing? Really, they'd like it to be within their own data center. So with those motivations, we felt like we can't just handle or can't only have a public marketplace, but wanted to have an enterprise offering where we would install our system in whatever cloud, whatever region on premise that users wanted. But before we can do that, there's a whole bunch of barriers. We have to figure out how you're actually going to deliver your applications. You have to figure out how to work on different operating systems based on the customer's requirements. Like I said, we have third party software called Distelli for doing our deployment management. We would also use some things like Algolia to power search on our website. There's a high proof of concept and the list kind of goes on. But what I'd like to talk about first is why Docker. So Docker gives us the first two of these barriers where we want to talk about the operating system and how to actually bundle the application and deliver it to people. So the benefits of containerizing is that in general, you're no longer tied to the host libraries or operating system that the small asterisk will come back to. It gives you an easy way to distribute and upgrade the application images. You can simply take the Docker image, you can push it to a registry, you can dump it to a tar file, you could send it via email, whatever you'd like to do, there's multiple ways to distribute it. It also gives a unified way to view logs and kill and manage your applications. So for example, if you're running HA proxy and some shell scripts and a web server using Apache and a different web server in Python or whatever else, they might all write logs to random spots on the operating system. With Docker, single Docker logs, you can view application logs for anything. It also makes it somewhat easier to run multiple copies on the same machine, but not fully. So I dockerized our system or completely dockerized our system in August of 2016, roughly. Prior to that, we had used Docker for a certain components, but it wasn't a fully fledged entire ecosystem. There's a couple caveats. As I said, you aren't really completely free from knowing what operating system you're running on fully. For us, when we're using GPU devices and other drivers, those actually need to be installed on the host operating system itself. It's not something that you can only have existing in the container, because you have kernel modules and other things that you're dependent on. And the kernel is not a part of the container, it exists outside of the container. Additionally, and this is a little more convoluted, but if you have a container and you want to spawn more containers, you would give that container access to the Docker Unix socket, which is from the Docker daemon. It's var run docker.soc. Since this is a file on the file system, it has a user ID and a group ID. And the user who's running your application inside of the container has to either match the user ID or the group ID for this file or won't have permissions to write to it. Well, it turns out that in Ubuntu or CentOS or a couple other different operating systems, when you install Docker, they get a different group ID. And that means that your container, your application is running with the group ID of say 999, that means that it can only actually access the Docker socket on other hosts that have Docker group 999. If you are fully using Kubernetes, you don't really have to worry about that, because usually you're not interacting with Docker directly. You're gonna interact with the Kubernetes API server or something like that. And when you're using the Kubernetes API server, there's like role-based access controls and tokens that will handle all the access that you need. But if there's some sort of middle ground where you are using Docker but not fully using Kubernetes or some other orchestrator, this might be a problem if you need to target different operating systems. We worked around that by sort of customizing the base image that we're going to use, even if we're using Ubuntu or CentOS or anything. We installed Docker and have a specific group that we use for all of our AMIs that we share with customers. There's a couple other issues and it's something where Docker people like to think that it's very easy, but in some ways it can be complicated and have sort of unknown performance issues. When you go to figure out what storage driver you want to use with Docker, you kind of get this whole little matrix here. If you're using Ubuntu or Debutin or whatever, there's just this whole sort of list and you're like, okay, well, which one do I choose? Turns out the recommended driver has changed multiple times over the last three years that we've used Docker from device mapper to AUFS and now it's at overlay two and that's something that the community sort of realized this one has performance issues or that one has performance issues and it kind of keeps evolving. For us, we ended up using AUFS. We might be changing that, but there are some tricky bits with a particular storage driver that they may not explain to you very clearly in some of the documentation. For example, with AUFS, it does require an extra kernel package to be installed and this is documented, but we were automatically upgrading our kernel versions for security reasons or whatever else and then when we rebooted a host, well, our process, we actually forgot to update the Linux extras package. So you reboot the host, the kernel's new, we're missing this extra kernel package and then the Docker demo to set their crash looping. You can easily be fixed by being a little more intelligent on how you're upgrading your packages and making sure you upgrade everything you really need to, but it's something that sort of caught us off guard when a month or so after we're using Docker in our system suddenly had these weird edge cases we were hitting. Secondly, with AWS specifically, EBS volume performance is really bad until the volume is fully warmed up. Depending on how big your volumes are, this can take several hours to read all the data. You don't actually have to write anything, you just have to read and scan the entire volume. In our case, it was taking 45 minutes to an hour and a half to actually fully read the volume and when you're trying to be an auto-scaling system bringing up and down lots of hosts, an hour and a half, you can't wait. That's not reactive in the way we want to be. Because of that issue, we found that with very right heavy workloads, like when we're spawning lots of containers and we allow to turn in our system, the Docker daemon would end up taking 45 seconds to a minute to create a container or do anything. And that's something that typically takes 100 milliseconds or less, something like that. And all of that time was just file system sync times were way too high. So one way to work around that is if you mount empty EBS volumes, they don't need to actually be fetched or pre-warmed or anything like that. So you can mount empty EBS volumes and do a bunch of work on those and they'll function way better than if you have something that needs to be pre-warmed. This only really matters for situations where you are doing lots of auto-scaling. If you just bring up a host and you expect it to be around for a long time, you might not actually need to worry about this very much. But we'll have questions at the end. Sorry, thank you. But overall, we're able to move everything to Docker, works great, but it doesn't do everything for us. So the next question is kind of why do we move on to Kubernetes? So Docker, we kind of abstracted around our operating system and we have a nice way to bundle our applications. But there's still a whole bunch of other problems. So Kubernetes is going to help us get rid of more of our third party software by being an orchestrator, becomes very easy to plug in new applications that we might need into the platform. And also, we want to get better server utilization. Before we had Kubernetes, we needed about 12 machines in order to run our base system and that's for highly high availability and the fact that you can really run one copy of a service per machine and there's this whole orchestrating problem there. So how does Kubernetes get us that? Since this is the end of the week of Kubernetes talk, I don't think I'm going to run through all of this. But the most important things I found are services which give you ability to do load balancing, service discovery, DNS entries, deployment slash replica sets. They allow you to have end copies of a thing running. You can do rolling updates. It's all native. You can configure how the update's going to go, say, do one at a time, two at a time, wait a minute in between instances. It's highly configurable, lots of things you can do there. And thirdly, the CNI plugin, which is the networking plugin, which allows you to get the actual giving IP addresses to every container in the infrastructure. And it also establishes routing between all the containers and allows you to create network policy so you can say, I want my API servers to talk to Redis, but I don't want the web servers to talk to Redis or whatever situations you may have. You can lock down the way this works. And then on top of that, there's also service mesh, which you can do even more fine grain firewalling. But we'll come back to that in a little bit. An interesting question, I think, is what don't we use Kubernetes for? It's a highly powerful tool. We could probably do everything we wanted to with sort of native Kubernetes ideas and operators and stuff like that. But we kind of chose not to, in part because we're a small team, it's a huge project, there's a lot of things going on. And we didn't want to throw a ton of variability into everything else that we were already doing. But on top of that, we had spent a lot of time optimizing our system for the way that we were scheduling algorithm containers. We would look at things like whether or not images were there, which Kubernetes would also do. But we have higher level data about what will this algorithm need in terms of the data files that it's using? And what are the likely files it might be using based on characteristics we've seen in the past? Things like machine utilization is something Kubernetes is also very like it will natively track for you. But pending requests and queue sizes is something a little bit more abstract it might not know about unless we added that on top. We also have some special security concerns that go as part of our scheduling. We will proxy requests for users, maybe put them on an isolated network if we don't trust it or the user wants it to be on an isolated network. And all of this we were still getting within single digit millisecond performance. It's one of those things where it's like, well, we spent all this time optimizing and investing in this code base. We didn't feel there was a need to move all of that inside of to be a little bit more like Kubernetes native instead of running in our application. Something we may change in the future, especially as we want to extend those capabilities further. But for now, it's something you should consider. Do we need to move every single aspect of our system to be Kubernetes native or can we leave some of it within the application? Also, GPU and device management, Kubernetes will allow you to expose the devices to a particular container or particular pod. You can say I need one, two, four, whatever it may be. But there's a lot of metrics that you can't get out of a GPU. So in particular, GPUs will track memory usage, et cetera, by the process ID. So when you're looking at a GPU, if you're just exporting a process ID, one, two, three is using this much memory, that doesn't tell you anything. Well, you need to know what was that process? Who was it? What's the meta information about it, which would be the user or the algorithm, that kind of information or application knows? So we had to find our own way to expose that information that wasn't sort of natural with CAdvisor and other metrics that are already generated for you. Secondly, with ingress controllers, ingress controllers are great, but they have some certain limitations and different ingress controllers have different limitations in terms of the amounts that you can do with SSL termination and things like that. And for a lot of our enterprise customers, they would have particular certificate authorities, certificates, different, they might need to validate X509 certificates from the clients who are coming in, and all of these configuration options didn't exist in any current ingress format. So rather than go through and figure out how to get all these annotations and other things, we found a way to template our own sort of hop proxy config and just kind of rolled with that. It's very close, it's almost like the ingress controller spec, and so I'd like to get it back in there, but it's something that we kind of have to work with the other projects, and it's hard to figure out whether or not they would accept our changes. But overall, we completed a migration in production in January of this year. Everything was running on Kubernetes fully. We were able to drop the minimum number of servers we needed from that 12 to six, and the six is because there's three Kubernetes masters for a highly available setup, and we'd have three Kubernetes nodes for highly available Redis, API servers, all those other things that we run in our system. We also only need a single load balancer now, and that's part of that ingress mindset where you have a single Nginx, hop proxy, whatever sitting in front of all of your services, and then it routes based off of what the incoming address was. Now we're able to replace our third party search with using elastic search. It's very easy to get elastic search configured and deployed within Kubernetes, and Kubernetes, again, we have native deployments and rolling upgrades, so we could get rid of distally entirely. Overall, it's very easy to add services to the stack as well, such as Prometheus and Grafana, if you hadn't heard about them the rest of this week. There are amazing tools for graphing, monitoring, etc., and it only took about a day for us to integrate Prometheus with our stack because there's so many deployment templates already out there. It was just easy to take this, plug in the little bits that we needed specific. After that, it's probably spent about a week or two weeks adding metrics everywhere I could and creating graphs and templates because it's amazing how much detail you can get about your cluster from these two tools. So what went wrong in this process? There's a lot that can go wrong if you have an application that currently exists not in the context of Kubernetes and you want to migrate it piecewise. It's always possible to duplicate your entire stack inside of Kubernetes and sort of just point your DNS to the Kubernetes one, but that may or may not work depending on your situation. And you might not even be able to migrate everything to Kubernetes if, say, there's some other team you depend on and they aren't willing to do so, something like that. So the projects are moving fast, like really fast. You might have seen that from this whole week, all the features people have talked about in the last year, the things that are upcoming. But they aren't always compatible with each other. So for example, Docker 1.13 came out earlier in this year. And at that time, Kubernetes 1.5 was out. However, it was not compatible. And also Docker 1.13 wasn't targeted for Kubernetes 1.6. And it wasn't until 1.7 that it was targeted and things were fixed. And it resulted in Docker was making some changes to the way that they did IP tables rules, so they added a default deny or default drop policy or something to the top of your chain. And that broke everything. So people try and upgrade and oof, didn't work. Secondly, there's a ton of different versions that are in development and used at the same time. Just in November, I think there was 1.612, 1.710, 1.83, and 1.90, Alpha all released in Kubernetes, in addition to other versions. So if you're going to be taking time to migrate your service, what kind of version are you going to target? It's really a project management decision, but it does become difficult when the docs are continually going to be getting updated and you can't find the information you need. Or you try and go with the bleeding edge and then you end up hitting bugs and you kind of get stuck in your migration process. It's just something worth considering and probably looking at what the chart is and estimate how much time you might need and then try and target something appropriately. There's also a lot of new concepts which can break sort of the way your deployments may work. So readiness checks with a deployment are very similar to a health check from a load balancer, but there's lots of things in the application and pod lifecycle that are different. That you might have to change the way that your application gets stopped. You have to change a rolling update strategy. You may be used to doing certain things at initialization, which now you have to create another container to be in a knit container. And none of this is particularly difficult, but it is something where every service you migrate, you will have to look at all of these things and figure out what do I need, what don't I need, possibly creating five or six containers if you have a lot of initialization that needs to get done. But the real interesting thing is the technical challenges. So as we said, the CNI plug-in will generate IP addresses for every single pod in your cluster. But those are not routable from things not within the Kubernetes cluster. And this can become really awkward if your services are trying to talk to each other via IP address. For example, we have API servers and we have Redis. And Redis needs to talk to itself to replicate data. And if the master goes down, one of the replicas needs to take over. So if you migrate Redis before migrating to API servers, well, your API servers won't be able to talk to Redis unless you put some sort of proxy or something in front of it that they can address. And then that would be able to address the things inside of Redis. Something you just need to understand your own application, what's talking to what before you can really migrate anything safely. Secondly, classic load balancers are not really meant to address containers. Again, the IP address isn't real. So what may happen is, is that you're exposing your containers or your pods on, say, port 9000 and the traffic is getting proxied to that container from the host. But it's just as likely that the traffic is getting forwarded to a container on a different host because that's the way that Kube proxy works. It will forward to something that is active across the fleet. And then what happens if host B goes down or deployment comes along and updates that pod or whatever? Well, if that connection that was going to host B gets reused for some reason, it's possible that ELB will mistakenly think host A is down. Because from ELB's point of view, it's talking to host A. But in reality, the track is getting forwarded. This can happen with any sort of HTTP client and any other application you're using as well. The main reason this comes about is because of connection pooling. So the connection from ELB that was going to host B, if ELB were to create a new connection, it would Kube proxy would only forward it to an active pod. You don't have to worry about that. The problem is that these connections may or may not get reused based on your application config. And if you're not careful about setting headers such as their connection close, you may end up stuck with this problem. Additionally, ELB allows you to do a TCP check, which would not use any pooling ever. So it depends on what sort of situation you have. Secondly, when you have a CNI plugin, you have a whole bunch of hosts and they get networked together somehow. For example, in this diagram, we use the WeaveNet plugin which works great. It has lots of configuration options, but there was one we weren't aware of until it sort of bit us. Since we downscale our cluster, it's possible you can end up with disconnected nodes. So say like we have this diagram here and host B and E we turn off because we don't have traffic or something like that. Well now you can see your cluster is divided into two different groups and they can't talk to each other across that divide. So that's something where you can actually configure the number of connections a given host is going to have. And you can easily prevent, if I expected downscale and nodes, make at least n plus one connections. Simple heuristic, but it works, it gets the job done. But you do need to be aware of your sort of topology there. For us, we highly monitor the number of connections that each host has, how many are failed, how many are whatever, and set alerts on that. All very easy using Prometheus and Grafana. And you could consider your network topology while scaling. These nodes will not refresh their node period list. So if a node goes down, it may not actually try to find a new node. You could make all the nodes aware of that. We thought that was a little too complicated and haven't had a need based on our heuristics and monitoring yet. So how do we go from here to multiple clouds? There's still a lot of moving pieces of infrastructure that you need to create, VPCs, networks, virtual machines, et cetera. And we also have to get it to work on a whole bunch of different cloud providers. So the first step is that, oh, sorry. But there's a bunch of considerations. With your application, you might want to think of first. Vendors should like to try and lock you into certain to using them. For example, AWS has services like DynamoDB, Kinesis, ECS, all great services. However, they don't all necessarily have an analog in every cloud provider. And if you want to target an on-premises environment, certainly there's no DynamoDB on-premise. Additionally, you can use Kubernetes to replace a lot of things you might otherwise use infrastructure for. For example, you can use a service to replace having physical load balancers, as we've discussed before. But you still have to figure out how to provision. And this is where, I believe this was mentioned in the last talk as well, where Hashicorp's Terraform is great. It allows you to create .tf files. You can commit infrastructure as code. You can version it. You can plug in a whole bunch of different variables for, say, region and AWS account or whatever else. And then we can bring up a cluster with those parameters. But once you have a whole bunch of .tf files for AWS, you can't actually just change those to the Azure equivalents. It's not just to find and replace virtual machine with the virtual machine from Azure. And there's a number of reasons for that. I'll try and go quickly through these. So AWS has an internal DNS to your VPC, but other cloud providers do not. So you might have to reconsider your architectures a little bit there. Again, I would rely on Kubernetes to do services to use DNS and not actually have to use a DNS within the cluster. But you can use a Kubernetes-level DNS. And that solves the problem for you right there. In Rackspace, if you have a public and a private IP address, those come as two different network cards. And there's this issue with Weave and the Kubla where it would end up sending traffic to the wrong network card if it's basically if you're trying to send traffic within the cluster, but a route didn't exist, then it would actually send it through the public network interface instead of the private network interface. And there are ways to work around that. It's on that issue there, how to work around it. Last, API versions are never simple. Rackspace is not exactly the same as OpenStack. Within a different OpenStack clusters, you can have multiple versions of authentication even. And the types of APIs that are available may or may not change between different customers who have an OpenStack cluster. So it's one of those things like you can discover all the services available within OpenStack for a particular environment. But you sort of end up with this giant matrix of what's there and what's not there. And you really just want this to be simple as possible. And it will be a little complicated for OpenStack. Load balancers, if you're doing a lot of copy and paste between Terraform files, you might see AWS, HTTP, load balancer, cool copy, that's OpenStack. But it turns out that with AWS, if you have HTTPS, that means terminate SSL. But in OpenStack, HTTPS means do not terminate SSL. So there's just these really small things that can get lost in this mix of so much going on that you end up not noticing very fine details like that. Additionally, there are differences where Azure, you can only bind to a single internal load balancer for a virtual machine. And again, it's because it's meant to be a classical one application per virtual machine. AWS, Azure, and Google Compute, I believe they all have concepts of like an API gateway or something along those lines, which is an application load balancer, I think is what it's called in AWS, which can route a little more natively to containers instead of virtual machines. It's something that, again, doesn't exist in every environment. Certainly won't exist in an on-premise environment, but if you only want to target a particular cloud provider, you could use those types of load balancers instead. But with Azure, traffic, if it goes from, say like you have two containers, a web and an API server on the same machine, and you want to talk to, the web wants to talk to the API server through a load balancer from Azure, it actually can't do that. Traffic can't go from a machine to a load balancer back to the same machine in Azure, which is just a part of their design because it's not meant for this sort of environment. Again, you can use Kubernetes services and you can just avoid load balancers entirely, avoid DNS entirely, and that's sort of the lesson is that Kubernetes will allow you to become more cloud agnostic just by having all these primitives that exist everywhere. So at the end of all of that, we have a whole bunch of different Terraform files with some parameters, and then a whole bunch of Kubernetes and files for deployments and stuff, and we can go from absolutely nothing to bringing up our entire system, running GPU algorithms and all of that in less than an hour, which I think is pretty good when it takes 25 minutes to provision an RDS instance in AWS. So not much we can do about that one. So looking forward, there's a lot of performance improvements coming up. Just last year, the GPU, exposing GPUs to a container was just added to Kubernetes, I believe, in one four, I think, and since then it's become easier to expose multiple devices, but then you have AWS coming out with FPGA instances, and I do think that FPGAs might be useful for some machine learning algorithms, maybe they could be faster, but I have no idea how to get Kubernetes to run and expose an FPGA, I don't know, but it'd be really interesting to see what can happen in the future there. Next there's sort of, when we're building a whole bunch of images and shipping them around our cluster a lot, we found that the biggest bottleneck ends up being just how big some of the images are and being able to compress them better would be an ideal situation, though I'm not sure this is likely to change. Multi-Cloud is sort of becoming a little more of a native thing within Kubernetes, so what we have is a system we can deploy inside of a particular data center, but you can imagine it would be ideal to be able to run across multiple data centers, which has problems of security concerns and data traveling over the internet, you can also get a lot of latency and it gets confusing and hard to track all of that, and that's where things like with a service mesh would enable you to track better how your services are talking to each other across all of these regions. Now there's the concept of a federated ubernetis, where if you're trying to run, it's sort of the difference between running Kubernetes that has masters in one data center and all the nodes are part of one cluster versus do you have Kubernetes running in five different clusters in some way to federate access to each one of those in a common format. So I think that's about all I have right now, so you can contact me for any other questions at atrickadalgorithmia.com. If you're interested in infrastructure, how artificial intelligence, machine learning, serverless computing, Kubernetes, how all that sort of matches together, we're hiring for pretty much everything and we have a coupon code. We give people free credits every month, but there's also a free coupon code, so. Thanks, so questions and thank you, Ed. It is kind of scary, so there are a couple ways to expose it and that's something where when you're using Kubernetes it doesn't matter anymore anyway. But it's one of those things like as a temporary step, you can also expose it via TCP and then you want to enable TLS and then you have self-signed certificates and it's a little bit annoying to deal with. One of those things where when we are controlling the stuff that's running, you've mounted the Docker socket to only a single thing and again it only has limited permissions and you can't do anything. We don't run any privileged containers or anything like that. But again, it's something you want to get away from and use Kubernetes and then you can have role-based access control and allow certain users to create certain API or only access, certain things, only be able to create, destroy certain resources. It's an AWS thing, it's in there documentation. It's basically if you have an EBS volume that has data on it, the way that the volumes are stored is that they're backed up in S3. So when you create a new volume and you attach it to an instance from a snapshot that volume gets attached immediately but it doesn't actually fetch all the data. So if you have a whole bunch of blocks, basically every time you access a block for the first time it has to get downloaded from S3 which can be slow. And if you have say a 250 gigabyte drive then it's gotta download all of that data from S3 even if there's nothing there. If it's an empty block, it doesn't know it's an empty block until it fetches it. Yeah.