 Welcome, everyone. Thank you for coming. This session is using Kubernetes and OpenStack for CPU and GPU intensive workloads. Really, what the session is designed to do is to explore where those make sense, when they make sense, and then how you ultimately be able to do something like that using things like OpenStack and Kubernetes to drive and where those workloads are most appropriate. I'm Marco Cepi. I work at Canonical. I lead up and am interested in things like Cloud, Cloud-native architectures and really pushing the next levels of computing, getting every ounce and drop out of my CPU, my memory, my disks, and my GPUs to make sure that the smallest footprint possible can get the most amount of performance. Because of that, we actually tiptoed to different territories. What I want to do is I want to actually just briefly recap what we're talking about when we talk about Cloud. Because at the end of the day, there's lots of different things that help define Cloud. But at its core, it's essential. The way I view Cloud is you have effectively the commoditization of consumption of three things. Compute, as in I've got VMs that give me instances that allow me to have CPU time. I've got network, which allows me to get network connectivity between more than one instances in my compute. I've got storage, a way to persist in accents, and utilize storage in an on-demand fashion. At its core, that's what Cloud is. Now, of course, Cloud as a scope goes way beyond that. We've seen this in the Upstack Projects. There's lots of other things that are adding on-demand service value additions, whether that's things like DNS, or load balancers, or databases. But at its core, all of these things are to provide commoditized accessibility to these things, which is traditionally a lot more harder to get access to. So why I want to talk about Cloud at its core is because really the thing we're going to be focusing on, we're talking about CPU and GPU intensive workloads, is the compute layer. The compute layer really boils down to those two things. At its layer of compute, you have access to either CPUs or you have access to either GPUs, or both rather, I guess, it's not mutually exclusive. But the idea is that at its foundation, a Cloud platform should provide you the means to abstract and access these things. So what's great about OpenStack is that it does that for you today. Right now, if you have an OpenStack running, running in one of the more recent versions of OpenStack, you can actually go ahead and define a way to pass not just CPUs to your virtual machines, but also pass in PCI devices directly and access GPU to hardware. That's in your Nova compute units. So I want to focus on GPUs. I think everyone here understands the core functions of what CPUs provide you when you're running workloads. It's the way to actually do the logical calculations of the tasks at hand, whether it's processing traffic from web server, managing with record lookups in a database. That is effectively CPUs. But what I'm talking about is why GPUs, why GPUs are becoming such a hot topic today, where they came from, why people are using them in computing and outside of just in your laptop video card as an example. So what we've really got is the rise of a general purpose GPU, something that's more than just a graphics processor unit. It's actually something that allows you to do computation cycles in a much different fashion at a much higher rate. So this is effectively what you find in your computer today. It's a CPU, it has a bunch of cores, there's a caching layer, and then on the motherboard you have a system memory. What's interesting about this model is it's designed in a way that's to process certain instruction sets really well. So if we're dealing with code that branches a lot at a lower level, it has a lot of taxing computations to take care of, the parallelization of multiple core CPUs is designed to tackle that. But it's not always the most efficient way to handle different types of workloads outside of just deep nested tree branching and code executions. That's what GPUs come into. GPUs started as a way to give you graphics and display. I mean, back in the days of Amiga, you have a simple way to actually have a dedicated board handling the graphics instructions and doing a refresh of lines on screen. What that came into is as GPUs and displays and monitors became more dense, more highly available, cheaper to consume, we actually saw GPU vendors building discrete cards that can handle parallelization of tasks in a way that gives you from 0 to a line refresh much faster. To achieve that, they actually started packing more dense cores, using more expensive but higher throughput memories and reorganizing the architecture of the boards in a way that is actually super conducive to doing parallelization of very shallow branching but high data throughput. So being able to process a deterministic route really quickly without doing too much instruction sets. At the end of day, what this gets you is from my laptop, from your gaming device, the ability to push a lot of pixels at the same time. So that's what a high bandwidth GPU gives you today, that's what this market pushed into. What was realized is that it's actually more than just graphics that actually gives you the ability to utilize these things. So there's other data models that can harness GPUs in a way that would provide them with the parallelization and the density to execute these things faster than on a traditional CPU architecture. So what we're getting into is what actually works well for GPU processing, what would you actually want to use GPUs to process instead of CPUs? So we look at things like instructionalization parallelism, that's your traditional workloads today. Things like databases, web servers, caching softwares, network functions, things that have been written in instruction sets that work against CPUs and that could potentially utilize one or more core, but have very complex branching instruction sets. We do a lot of complicated routines to produce the results. What we're seeing data parallelism come into place, where you can actually utilize more cores to produce and render faster output, is in the areas of rendering, graphics cards for rendering data to screen, rendering actual frames for movies and things of that nature, and also things like transcoding, where you're actually dealing with, again, that high velocity of data, that high volume of data to produce a result, where splitting those out into hundreds of thousands of jobs on a single card, is much faster than having it split between one or two or three cores in a CPU. Because of that, what actually happens is in some arenas, this is mostly in the areas of machine learning, blockchain management, in memory, in process deep learning applications. They have the same problem exists, very shallow code branching that you have in things like rendering and transcoding, but can actually benefit from a lot of data parallelization, where you can computate a bunch of things very shallowly, very quickly, by using the high bandwidth data model. So that's where we're seeing these applications that are springing up, things like TensorFlow, which is Google's machine learning library, Ethereum, which is a blockchain-backed cryptocurrency, blockchain itself as an item, Apache Spark, which does in process computations for big data sources, and many, many more applications are applying to find where we have data models that are much more denser, with very little actual computation driving to be run against them. Because of this, it does very well to that model. So I want to walk through an example here. Most of these new breeds, these breeds up here at the top, TensorFlow, Ethereum, et cetera, because they are built to be stateless in that they're given an input, they produce a computation, they produce a result, and store that somewhere else. A lot of these new products will be picked up and used are using things like Docker and Ubuntu to develop and actually build a way to deliver these things at scale. So not only can they utilize multiple cores, thousands of cores on GPU, but also be able to utilize multiple hosts running multiple GPUs to increase the parallelism of tasks. So because they're using things like Docker and Ubuntu, it makes an interesting intersection of how do you actually start utilizing these in a team? A lot of people here today will either have an open stack or a building open stack or building a cloud that people can access. If your cloud doesn't provide GPU resources, what oftentimes happens is these teams won't be able to actually execute effectively their tasks for data processing, and that could be numbers of things. With it's training deep learning models, using machine learning to do predictive analysis, inline processing of data streams, all these things that lend themselves very well GPU workloads will be hit in orders of magnitude and performance when done just on CPU resources alone. So this is where Kubernetes comes into place. So open stack today can absolutely give you instances with CPU, with memory, with networking, with all the things you need in order to run general workloads and even specialized workloads, including GPU deployments. But if you're starting to combine your actual physical Nova machines with a bunch of Docker containers running on them, odds are you're gonna run into a management problem. Because things like VMs from an open stack cloud, and Docker containers are so different in not just how they're produced, but also how they're managed and how they're deployed, it lends itself to a new breed of systems to manage these things. And in fact, they become very complimentary. So Kubernetes itself is really just a management platform for Docker and Docker style containers. And what it's giving you is the ability to actually say, here's how I, using this new format of delivery mechanisms, this is how I consume the general compute, networking and storage, all of these components, but at that level, at that Docker style container level. So it becomes very complimentary in that, in a typical data model that you're running these new breeds of applications, these new data science type things, these new high performance throughputs, almost all of them are stateless by nature because state itself becomes a bottleneck for computation. If you have to continuously flush the disk, you're not using the latest NMV SSD disks or on memory chips, it actually becomes the bottleneck compared to your CPU processing power. So because of this, what you normally see is there's a set of persistent storage that's modeled outside of this workflow. And in fact, what you find is data scientists who usually lever just the stateless portions of it inside these Docker containers. And what you have is a pipeline of delivery saying I have an ingestion process that stores in a database, on a file system, to a disk, to an HDFS volume, to a number of different places. And this is something that's persistent. This is something that has to actually exist and persist state. This is where you'd wanna run VMs. You don't normally wanna run a virtual machine, sorry, a database service in a Docker container because if that container goes away and it's not set up properly and it's not morphed and configured, you'll lose all that data. And that's not something you normally wanna do when you're talking about computations. When it comes to running and executing multiple cycles of data training or analysis of data streams, those lend themselves very well to being stateless. They boot up, they're given an input, that data's processed, it's exported somewhere else and that process shuts down. It's done, it's done its thing. There's no need for it to persist state which is why they're typically modeled at things like Docker containers and Docker style containers. So because of that, this becomes a very complimentary thing where normally what you'll see is teams that are doing these things will run in a Kubernetes on top of their OpenStack or potentially alongside their OpenStack where they're using OpenStack and traditional VMs and the things that we've come to learn and appreciate and that commoditize resource pool to spin up and store all their persistent data, all their persistent management but then using Kubernetes to get that same style, that same flow, that same feel before that Docker container process. Instead of having to go into a VM somewhere, install all the tool chains that you need, then go and launch those containers manually and then go manage those processes. How do I scale them up and such? Things that OpenStack is done for us, for bare metal and for hardware is done at that same level whether it's on an OpenStack or next to an OpenStack or even below an OpenStack. That's what Kubernetes is looking to achieve and that's where these two intersections become really interesting. So I wanna show a few examples of what an output workflow would look like for these things. And then I wanna show how you can actually get started today if you're using Ubuntu and you wanna deploy, if you have data scientists that are itching to run these processes and to really start maxing out things like GPUs and CPUs, how simple it would be for people to get started doing so. So this is what we're looking at for this demonstration. We've got a data ingest pipeline. This is actually just grabbing a huge pool of images and putting it on an NFS file storage. Nothing particularly fancy or necessarily stressful. And then we're gonna be using TensorFlow to help train a model that we've predefined and then evaluate if this pool of images actually matches a set of criteria that we've defined in this model. So is this a cat or is this a dog which is the kind of typical data evaluation for a TensorFlow workflow. And so what we've done ahead of time is we've said here is a group of pictures of dogs and here's a group of picture of cats. And we've trained this model and we have a huge pipeline of hundreds of thousands of images that are sitting in this NFS storage. And we're gonna use TensorFlow in Docker containers running on top of Kubernetes using CUDA and NVIDIA GPUs to go and process the cycle through does this actually match a dog or cat and give me an output. And then finally serve those results. So the first thing we've done is we've deployed Kubernetes. And once Kubernetes is deployed you're given a dashboard like this. And this is actually very similar to what you find in Horizon. It's a dashboard of all the things you have running inside of your Kubernetes. In this case you can see we're running a set of workloads. Gives us the idea of how much CPU usage is being moved across the cluster, how much memory is being utilized. And all these are running on top of a GPU enabled set of servers. So this is what our TensorFlow job looks like once deployed. And TensorFlow itself has a dashboard for its own application. Much like many applications today there's some way to log in and view what's happening there. So once we're training this model what you see is something like this which gives us an output of here is all the data we've crunched. Here is the outliers. Here is the evaluation testing. Here are the peaks and the wanes. And then furthermore from there here's the idea of actually a standardized curve of how accurate our estimations have been. Where using a data training model we go and back and assert is this a cat, is this a dog based on a set of images we've already had. So how accurate are we actually translating these things? And we can see here that after a few moments of time we actually get pretty close to a 90% certainty that what we've guessed is actually accurate. So I'm gonna switch out of this and actually show what this looks like to get something like this running. And while I'm doing so does anyone have any questions so far? Yeah, absolutely. Let me get rid of this. Cool, that worked. Okay, so on this example here since I don't have any GPUs at my house that I could use for these things I actually spun this up against an Amazon cloud using their P2 instance types. So it's K80 NVIDIA GPUs. I've had other people in other sections. A few folks over at NASA are using this same methodology to do different computations for machine learning statistics on rocket analysis. And a few other folks are using this same methodology for doing rendering large scale rendering of 3D models. So it's applied at varying levels. NASA's doing this on bare metal as NASA scientists. I've did this in a public cloud and then the GPU rendering farm is actually running a hybrid of both OpenStack and Kubernetes for their different persistent volumes. So let's go ahead and see how we booted this up. So the first thing it is I've already configured a cloud endpoint. Again, I don't have an OpenStack readily available for me to show this on OpenStack but this was applied in the same way. Now, what I've done is I've installed a tool ahead of time called Chandra. So Chandra is just a way to get a really good idea of a starting point for a model for software. In this case, for building things like OpenStack clouds, for building things like Kubernetes clusters, et cetera. So I'm just gonna go ahead and run Chandra up. Yes, I have a small screen. And what it does is it gives you a list of things that you can actually go and deploy whether you wanna deploy big data workloads, persistent big data workloads, like HDFS that have two clusters. If you wanna be deploying things like Kubernetes clusters, OpenStack, other interesting pieces, we're gonna go with Kubernetes and a Kubernetes core deployment. And so, from here I can choose where I wanna deploy this actually. So there's a set of endpoints. If I had an OpenStack cloud, you see OpenStack, my OpenStack cloud listed in here. There's a different set of exit points for this to be deployed onto. And what ConDrup is gonna be doing for us is actually take a set of starting examples. So I have an example topology that I wanna start with. One master control plain service, several workers on a GPU. And I can then use this to manipulate that, scale my cluster over time and manage it. For the sake of this, I'm gonna go ahead and use a pre-existing cluster I've turned up. And we're gonna do this on Amazon, US West. And so what ConDrup's gonna do is gonna say, hey, here's a list of the applications I can deploy. So this is just a standard Kubernetes topology. You've got etcd, which is its data backing store. This is very similar to how you would leverage my SQL or another database store inside of OpenStack for coordination of services amongst other. A Kubernetes master component. This is the API server. This is the identity manager. This is the controller manager. The schedule manager. And then a worker which is actually the effectively the comparable NOVA for Docker containers effectively where you actually run your workloads. And then we're using Flannel for an overlay network, but this could be potentially something else. I'm gonna keep the default topology because it's probably the easiest. I'm just gonna go ahead and deploy. So this is gonna go ahead and run through deployments. In a few moments, this will come back and actually execute this on my cloud. So this could have been me booting up and OpenStack cloud using PCI pass-through profiles to give me actual access to GPU hardware. Or in this case, I'm using Amazon to give me P2 instances to actually have access to GPUs on demand from a public cloud. What's great about these things is because we're using things like the standard upstream Kubernetes mechanisms, it actually gives us a chance to really do some interesting things with Kubernetes where we can use our local, either bare metal or our local OpenStack to deploy a cluster. And as a team or resource starts to grow and they no longer can be tenable on that cluster, they can actually use and burst out into some other public clouds, whether that's Google's hosted GKE or whether that's Amazon, another cluster deployed in Amazon itself. By the end of all this run, what you're presented with is kind of an actual view of what's been deployed. So this will take a few minutes to run and it's not interesting to see how it's deployed, but you'll see how you can actually start utilizing it. So I have deployed this cluster just a little bit ago. I would say probably about 10 minutes ago. So let's see what we have here. Yeah, so 15 minutes ago I deployed this cluster. This is that same cluster, it's running on US East One. And it's got a GPU enabled worker on it. So what we can do now is using this by having this setup run, all the things that are required to actually operate and manage a Kubernetes for GPU workloads. That's enabling flags, making sure drivers are installed. All of those operational components that would normally take days, maybe even weeks to go and actually get assessed out are all managed by using these tools, ConDrup and such. And what it's doing is actually giving us operational insights and expertise to manage that Kubernetes cluster and all the underlying pieces below it. So let's just give you an idea. We'll see that we'll run the NVIDIA SMI command which shows us what's actually on that box. So on this worker we happen to have the NVIDIA Tesla K80. No processes are currently running and it's driver version 375.51. So what it's done is automatically detected the hardware running on there. It's deployed all the drivers required and configured Kubernetes in a way that knows that it can now schedule GPU workloads on that box. So what I'm going to do is just run this same example. Oh, one second. And while I wait for this example to start spinning up, does anyone have any questions for me so far? Yeah. Yeah, that's probably one of the downsides on GPUs that are being built today because they have such a higher throughput, magnitudes more than what you find on system memory drives. They're very small in size themselves. I think the biggest cards now are maybe 16 gigs at most. What we normally see is we can actually, well, we do see that pop up from time to time and it's just a matter of waiting for card developers to get to a point where the L2 caches and the size of memory on card aren't necessarily gonna be a bottleneck going forward. So if you're running really memory, really high intensive GPU workloads, it's typically better to spread them across more than one card or a couple cores actually channeled appropriately than having a cluster of jobs split between two different core caches where you start to hit an overhead of actually jumping between system memory and GPU cache. So we don't see it as much anymore. Most of the modern cards today have quite a bigger set of cache than what was on generation previous or even the previous generation before that. Even the Tesla K80 card is probably a bit older than what you find today in most modern NVIDIA drivers cards. But yeah, that's a good question. I think you'll see in the next coming year a lot of that bottleneck will go away for sure. Especially as more card developers develop this capabilities much like how NVIDIA's cards have here. Any other questions while I get this last piece set up for us? Right, so we're gonna go ahead and deploy the TensorFlow example. And so I'm gonna walk through just real briefly. This is gonna be a lot more lower level Kubernetes bits. So I'm just gonna highlight briefly what we're doing and how this all works. So we're gonna just run a real quick manifest. Manifests are a lot like the definition of how to actually execute your workload onto Kubernetes. So what this is giving you is, I'm gonna make this a little bigger for everyone. What this is giving you is just we're gonna run a simple container to show that we have the NVIDIA devices being passed through appropriately. So we're just gonna create a quick job. It's gonna be called NVIDIA SMI. That's gonna run this command using the NVIDIA CUDA image. So these lines here are kind of important. The image we use, this is the Docker container we're running. In this case, it'd be NVIDIA CUDA, but this would be potentially Google slash TensorFlow or your application that your data scientists have worked on and have packaged up and deployed. And then we're gonna make sure a couple other things happen. We're gonna make sure that we utilize the NVIDIA GPU resources in Kubernetes so it knows that we need to have it on a machine that has GPUs enabled. And then we're gonna make sure that a couple paths get passed through. So it has access to the drivers on the machine. And that's it. So this is about a 20 line file that gets us a running job to be able to test and validate these things. And this is executed. So I went ahead and applied that YAML file to our Kubernetes cluster. Now if I do a CUBE, Cuddle, we see that we had an NVIDIA SMI job run. We had one job run, zero successful, 11 seconds. So it must still be running. So it's still running in a few seconds. This will come up after it puts the image, downloads it and runs the task at hand. And this is effectively a stateless machine. This has got no state that's being preserved. This is much in the way you'd run your existing applications, your existing workflows that your data scientists are using for their images. Yes. So Docker Compose is, it's definitely comparable to what we're doing here. Docker Compose gets you kind of a running set of bundle of Docker containers. What Kubernetes is doing is composing that in its own way that gives you additional flexibility on top of that. So they definitely are, they're definitely an overlap, but Docker Compose is a much more lower level mechanism. That's if you have a machine, you SSH to it, you wanna get some stuff running immediately, install Docker IO, go run Docker Compose and against the YAML file and get things running. What you get by using something like a coordinator like Kubernetes is the repeatability and flexibility of those jobs being run again. Yeah. Right, so it is similar, it's not gonna be the same thing by any stretch of the imagination. So while this is wrapping up, we'll look at this one more time just to kind of walk through. So this is a very Kubernetes specific file format. It runs on Kubernetes only effectively, but is achieving the same results as a Docker Compose where we're defining a set of primitives. This is a job, it's not a pod, it's not a service, it's not a node, it's a job, it's gonna boot up, run its thing and stop. Whereas we could say I wanna run my TensorFlow dashboard, which is a service, it consistently runs and needs to make sure I have a minimum number of these things running and that would define a separate set of schema results for those things. And then we define the spec for this job. So we're gonna be applying this label, we're gonna be using this container, we're gonna run this command in that container. These are the resources Kubernetes needs to make sure are present for execution, these are mounts we're gonna apply to it. So while it does achieve the same thing, there's definitely a lot more in this spec and it's not something that you can just take from one and push it the other. All right, well it's been two minutes, I'm not quite sure why this job isn't finishing up, I'll have to go poke at this in a minute. But the important thing to note here and we'll see how this second install is running, moving along progressing. So the important thing to note is when it comes to things like GPU workloads, a lot of new GPU workloads that are being created are being created using Docker containers, being developed on Ubuntu by data scientists and it lends itself to a new set of platform primitives. Data scientists aren't necessarily operations people, but I'm sure there's some overlap, but generally speaking they're more focused on getting things running and getting data back than they are necessarily focused on making sure there's 100% uptime on this machine running here. So what Kubernetes, the platform lends itself to, very complementary to that open stack primitive is, operators need to make sure that your persistent data is there. Your databases are still running, your file stores are still there, you have data integrity and you have uptime. Data scientists are typically just saying I'm gonna have a set of processes that I wanna run and execute them quickly. Kubernetes on open stack or alongside open stack gives them that same primitive while also leveraging their format that they're currently developing in. So with that, I'm just gonna leave a few notes and then I'll open it up for questions. All right, does anyone have any final questions? Yes, the front here first. So that's, no, it's a good question. So the question being what would you recommend as a pattern for having access to GPUs? Would you recommend running them alongside an open stack, utilizing the hardware directly or leveraging them through an open stack where the tenants inside of an open stack instance have the pass-through device and it's inside of there? I think that's really gonna be dependent on the cloud and the team utilizing Kubernetes. We've seen both patterns. I think both work really well and it depends entirely upon what fits best into your org's needs. If it's easier just to say, here you've got a class of instances that have PCI pass-through devices and each team can just go and say, hey, spin up a Kubernetes as large as you want it within your tenant, you can utilize them, you have it there. It makes it easier for your operations team. It's one less thing that they worry about. There are some instances where having the full breadth of metal underneath the Kubernetes cluster is more important. At which side they work really well as well next to each other where you're sharing maybe your networking control plane and managing them as entities alongside. But there isn't really one recommendation over another. I think you'd see them being used in both models for certain. Yep, absolutely. So if you wanted to utilize this Kubernetes to run COLA on top of, that most certainly is a workflow you can push through it, absolutely. And having GPUs exposed up through that way as well, completely possible. Yes, I think it depends larger than the domain of the business. We're seeing a lot of data scientists uptake on GP GPUs just simply because of it is multiple magnitudes more for quicker to process a lot of their tasks than it is on CPU-based workloads. So in certain domains, that's definitely exploding exponentially. You're not gonna see people clamoring to have their database services necessarily run against or more traditional database services run against those things. But for tasks that lend themselves very well to high parallelization of processing of data, you'll see GP GPUs being uptake quite quickly. I think because we're seeing a lot more availability of GP GPUs on clouds, it's much more accessible to developers. So I know a few developers that just kind of hack in their spare times and they'll spin up a GP GPU instance for a couple hours, which is maybe three bucks and do some interesting data processing because they're interested in learning how they can expedite tasks or playing with machine learning or deep learning models and kind of planning for the next breed, which is building data models in clouds and delivering them to edge devices like your car or your thermostat or something else as well. So there are definitely an uptake in people because of the availability of GP GPUs in clouds where normally it's very expensive investment plus a lot of power draw, plus a lot of heating and cooling bills when you get a good size cluster running at your home. So I definitely think you see an uptake in that for sure. Yes? Yeah, so this is the, is there a way to do something more than just pass through of a device? There's a project in OpenStack that's trying to tackle this, Cyborg, which is trying to find a way that using the OpenStack APIs provide a, not a pass through directly, but actually a chunk of GPU or a chunk of networking and stuff. So there's a project tackling that now if you're interested. I'm not sure where the progress is on it, but we've been kind of following loosely this, the progress of there. So it's very interesting that in a perfect world you'd have a slice of CUDA cores available for instance, which is much more economical than an entire instance sitting on a board, which can be quite costly in the size of the amount of cores you've just lost. If you're not utilizing them to their full potential. So yeah, I would check out Cyborg for sure and join up the development meetings there and kind of follow along. Yes, so I found a new friend. So yeah, I'm very interested to see how that grows into the project to make that a little bit more accessible. So cool, thanks for joining. So for the most part, and James Page, correct me if I'm wrong, but for the most part, once you whitelist the PCI device, the schedule will just find a matching Nova host that has that whitelisted device on it. So you won't be utilizing. Generally speaking, you don't want to necessarily have heterogeneous hardware in a zone, but that happens a lot and it's fine. So I'm pretty sure OpenStack handles that pretty well today. We don't do much additional munging there. It's just whatever happens in upstream. Right, so since you're using PCI pass-through, it's not necessarily the host that needs that driver. It's the tenant, yeah. Sure, it could be absolutely better. I hear there's a project that might help with that. But yeah, we just use whatever mechanisms exist today in the upstream OpenStack repose. Yeah, normally speaking, when we have customers that are deploying GPUs into their hypervisors, they're setting aside a region to handle that and a lot of that hardware is as close to possible the same that way they can say here's a region with availability zones with GPU processing available. Yeah, any other questions? I think I'm running up at a time, but I have a few more, yes? No, no, it's a good question. So how does Gigi Charms relate to Docker in general? What's? Yeah, Gigi Charms. Yeah. So what Gigi Charms are is a way to encapsulate operation of software. So how do I deploy? How do I upgrade? How do I scale? How do I configure and reconfigure Kubernetes as an example? So what we've done is we've worked with the upstream projects. So the Charms for Kubernetes live in the Kubernetes tree. They worked on there and upstream and we tackled how do you manage life-cyclist code? And so Charms themselves absolutely could launch Docker containers, but it's a bit on a different plane of operation. So if your lower-level control plane services can run in Docker containers, that's something you could do. We have examples where there are telco network function virtualization vendors that have a lot of their stuff Dockerized and non-Dockerized, so they'll write Charms that do Docker launch over here and Docker run over here and Docker compose here and then here's how I do the other additional bits. So there's definitely a place for it, but it doesn't necessarily fit directly into the model here. It's generally easier to give your developers, your data scientist access to a platform where they can just press a button or run a single command to get workloads spun up and spun down, instead of having to write a bunch of operations code, because they're generally speaking not the same people that are operating your persistent data source, your under clouds and stuff. That's very true about the stack. It's really important about just both the stacks that can run themselves and the stack that can be upgraded every day, right? You know when binary matches the whole thing that is and you're gonna keep it up-to-date, operate it cheaply, that matters, and the affinity is to do that with our students. If you choose to remain in that speed, whether I can find it or not. Yes. I'm very impressed. I'm, yeah. So Lexi does absolutely, I'm sure. So Lexi speeds every two points. Yeah, absolutely. So if you wanna run a couple Lexi machines past the PCI device too, you could use that as a target as well, cool. I think we're out of time. Thank you all for joining me. Now if you have any questions, I'll be hanging out for just a little bit longer, but enjoy the rest of your day, enjoy your lunch, and then we'll be back here in about 45 minutes.