 Good morning everybody. Thanks for showing up today. This is Computational Fluid Dynamics, Kubernetes, Kubeflow Open Phone. Before we get started, I just wanted to thank two folks. I don't think either of them are in the room today. Eduardo Orango and although Coquincador, they were super helpful when I was going through this sort of process of figuring all this stuff out. Eduardo's actually not at Red Hat anymore. He moved over to Nvidia. So either way, still helpful or thankful for his help. This is not a super serious presentation. It's Friday. We've been here all week. It's, you know, all tired. It's going to be fun. I promise. I hope if you want serious, there's a blog post that I wrote that details all the gory GitHub repos and all this other stuff. This presentation is on the SCED website. You don't have to worry. We can go back to this, whatever. So if you don't get a picture of this, it's fine. And like I said, this is going to be fun. Full of bad jokes, terrible memes, all that kind of stuff. So I'm getting at least one round of applause. So we're good. Okay. Here we go. This presentation is built like building blocks, right? We're just going to try to put things together, get deeper, deeper, deeper as we go to explain how I actually was able to achieve this because in reality it was mostly just kind of an experiment. Before we get started, just a couple of questions. If you're a Kubernetes user, raise your hand real quick. Everybody. Okay. That's not totally a surprise. Who's doing HPC not necessarily on Kubernetes, just anything that would qualify as HPC, only a few. Anybody doing MPI stuff? About the same number. And then lastly, open foam existing? One. Okay. Cool. How did we get here? So me, I've been at Red Hat for 14 and a half years. I'm not a software engineer. I'm not a Kubernetes Kubeflow open foam developer. I'm not an aerodynamicist, but I do have a race car and it does have a wing on it. So I ended up getting this email from an account team out of APAC and they were like, oh, this automaker is doing open foam in a container. They want to do it in VMs. And I was like, can we do this on Kubernetes? And I remember it. I have these friends who do aerodynamic stuff and they offered me some of their models to play with and I was like, cool. Okay. Let's do an experiment. Let me see if I can run their fluid dynamics wing job in a Kubernetes environment. So review open foam and MPI. What is MPI? It does parallel or distributed computing. So we have lots of things that we do then lots of little things in parallel. Open foam is for computational fluid dynamics. Air is a fluid. Water is a fluid. It can analyze how it flows around objects. So we have this big job. We want to break it into little jobs and that's where MPI comes. Well, but not really, right? MPI doesn't do the breaking up. It just does the processing. Open foam is a tool. Well, it's actually lots of tools to help do fluid dynamic stuff. Water is a fluid. So we take our big fluid dynamics thing and we break it up into little pieces with open foam and then we use MPI to do all the things. Those things are actually really like individual servers for most folks. They're used to HPC and clusters and etc. And it does this over SSH, right? So MPI finds all these systems with SSH, goes into them, does the things. We need some kind of shared storage because all the little pieces need to find the work that they have to do, so to speak. So we got to figure that out as well. Okay. It's not really that simple, but for the sake of simplicity, we're going to draw this picture. Open foam, MPI, lots of stuff, RWX. Cool. Clusters and Kubernetes. So how many of you have something that looks like this? Right? You got a rack of servers, those stuff. Okay, cool. So 2014, this cool Kubernetes thing happened and then people started trying to do this, right? We want to run all the workloads in Kubernetes. So this is a workload. Can we run it in Kubernetes? Let's see what we can do. Starting to draw parallels, right? So open foam, MPI, fluid dynamics job, we've got Kubernetes, Kubernetes runs pods. If you remember the previous picture, we actually have a bunch of little things that we're doing, fluid dynamics. That's a lot of pods. We have SSH. We have RWX storage. We have Kubernetes. We have RWX storage. We'll figure out the rest. I'm not really sure, but we'll get there, right? So let's go deeper. Cubeflow. How many of you know what Cubeflow is? Whoever heard of it? Oh, wow, lots of folks. Okay, cool. So Cubeflow project dedicated to making deployments of machine learning. Well, we're not doing machine learning. We're doing fluid dynamics, right? But it has this operator called the MPI operator. Now it says it's for all reduced distributed training. It's just MPI. Like we're doing an MPI thing. Let's do an MPI thing for one thing. So let's go deeper. Operators, how many of you are familiar with Kubernetes operators? Most everybody in the room. Okay. Operators are this, it's too complicated. So operators are just a pod, right? There is a controller pod that looks at instances of a custom resource, custom resource definition extends the Kubernetes API. And then it does stuff. In this case, we're talking about the MPI operator. The stuff that it does is MPI stuff. So we've got MPI operator. We have this MPI job. I'm running a wing CFD job. It's going to happen in a lot of pods. Okay. I don't know. So lots of pods, Kubernetes, RWX, all the things. I used OpenShift. That happens to be the Red Hat Kubernetes. We have a storage product. Well, it's moving to IBM, but we have a storage product open data foundation that gives me RWX. And then we had to do stuff in the middle with Cubeflow. So let's go deeper again. Wing CFD, MPI operator. When you use the MPI operator and you instantiate an MPI job object, it fires up a launcher pod. That launcher pod then runs MPI into all the job pods that do the work. They talk to storage if you have storage defined. And all this happens on top of your Kubernetes environment. Okay. It still does this over SSH though. That's gross. And I had to do some nasty tweaks that are in the blog post to make it work on OpenShift, but whatever. We've got it working. It's fine. Okay. Here we go. So what is the open foam process? So the first step is this meshing process. You take a 3D object, you break it up into like a meshy thing. Okay. Cool. Then you decompose this into lots of small jobs. That was the thing we talked about early on. Then you do the problem solving and figure out what the fluid job is. And then you put it all back together. Relatively trivial. So I'm really good at this. And fortunately, the open foam project provides a bunch of examples for how to do stuff. And I also have these guys to give me things to do. I actually helped Moreland build their open foam cluster like they're in Atlanta. I went over there and had lunch one day. It was fun. Okay. So Dan break is the example job that's provided by the open foam project. It's a very simple water flowing over a thing. Let's try to run that job and see if we can make it work. So what do we actually have to put in here? This is where the rubber finally meets the road. Trigger warning. There's actually code. It's barely code. So this is the Dan break example. So you do some stuff. You break it up. You run the parallel job. That's the meat. And then you put it back together. So how do I get that script into this launcher pod, right? Well, fortunately, I actually know a thing at least maybe about Kubernetes. We have this thing in Kubernetes called a config map. It makes it easy to inject files into the containers. Well, that's just a script. Why can't I just inject that as a file into the container and then make the MPI job run the whole script as the entry point to the work? So that's what I did. I put it in a config map that I defined against the Dan break job that went into the launcher pod that launched the wing CFD pods that talked to the storage on the container platform. Okay, cool. And then yes, it did work. This is what it looks like. It was doing things. The first one is the Dan break job. The second one is motorbike, which is another example within. This is the open shift user interface that just shows like, hey, they're using CPUs. There's pods stuff. That's the log output from open foam for the one person who's using open foam. They've seen that a lot. They spend a lot of time looking at that text. This is the more land job. This is not really a picture that I generated. That's when they sent me, but that's what I did. I ran this job and it worked. Well, sort of. Remember this picture? I don't have one of these. I got this. I wish that was me. That's not my laptop. It's not my background. It's a cool place to work every day, I guess, or just really humid and mosquitoes. Anyway, so I didn't have one of these. I did have one of these. Really, I had that. Unfortunately, I didn't spend a lot, but I did have this. Again, I just kind of was messing around. I used M5A 4x large, which are not compute optimized instances. I was like, all right, well, let me do two CPUs for pods. I'll just do a boatload of pods because more is always better, right? 512 CPUs, it took five hours, and more land was like, man, that's really slow. And so the reality is you have to know what you're doing, right? Tuning, tuning, tuning, tuning is important. I'm not really sure. I could have used the compute optimized instance type. I was probably, I probably should use bigger pods that are using the whole server instead of lots of tiny ones. Maybe I was over slicing the job. When you have the surface and you break it up into lots of little pieces, you end up with overlap, and then you're basically just reprocessing the same things over and over that get edited out at the end when you put it all back together. Or just know what you're doing, right? Actually, being an aerodynamicist know what any of these things mean. Where can we go from here, right? So PMIX is a new thing around open MPI that gets rid of SSH. There's an actual server daemon kind of deal. It can run on Kubernetes. So it would be great if we didn't have to legitimately run an SSH daemon in all the pods and then SSH via MPI from the launcher into, this is just gross. Of course, more tuning. NVIDIA is working on doing GPU enablement for this particular project for open foam so that you can use accelerated processing via GPUs. But ultimately, what was the point of this whole exercise? It really was an experiment. Can I even do this? So the reality is that I wasn't trying to test if AWS was a good place to do this because the reality is AWS really isn't a good place to do HPC. Your data is probably not there. So moving all your data into there, that's going to be ludicrously expensive. And it's just expensive to do. Your instance types might not be available if you need a boatload of them. There's all kinds of problems with this. Most of you, if you're doing HPC, you probably have clusters on site. That big picture of all this, like you can do this. It actually did work. We were able to run open foam and more generally, whatever MPI type job you have, we were able to do it. One of the important things to think about, though, is that for those of you that are raised your hand that you are doing HPC, Kubernetes doesn't think like most HPC schedulers. It is a scheduler. I would like you to run my job. It will do it. It has no concept of priority, preemption, queuing, like that kind of stuff, like traditional slurmy, whatever your HPC tool of choice is. It's not doing that today. But there's a whole army of extensions and schedulers and other things that are coming about that are doing those things. Red Hat is investigating how we can do some of them. We've actually got the product manager for that in the room right now. But ultimately, the purpose of the job, the purpose of the experiment was like, can we do this? Yes, we could. Okay, great. That's it. That was really fast. Wow. I was like barely 12 minutes. So, yeah. Happy to answer questions. I realized, again, this was supposed to be just kind of a fun, happy one. But yeah. That's all I got for you. Questions? Okay, we got a few. Hold on. We got a microphone. Great talk. So, where are you going to go from here now? Like, is this project done? Was this just kind of like? Yeah. So, it's good. Thanks for asking that. I'm talking to the NVIDIA folks about it. They're interested in figuring out how we can do that. I've done NVIDIA GPU things on AWS before on OpenShift and Kubernetes. We know it works. So it's just a matter of how do we do that with OpenFoam. The other thing is, where's the script? So, like, these steps, I was thinking about it yesterday, actually. It doesn't really make sense to do all these steps in the launcher pod. Maybe it would make sense to, like, do the preprocessing and decomposition in a tecton pipeline and then take that and then attach that as a stored object into the launcher pod and then just run the MPI thing. And then when that job finishes, do another tecton pipeline to run reconstructs. Like, does that make sense? I don't know, maybe. It would probably work. But, like, how much effort would it be to actually automate it? Would I need to write a tecton task to launch the MPI job and then, like, another... So, I don't know. It's an experiment that I'd be interested in doing. But that's kind of the, like, where would it go from here? And then I'm trying to figure out some performance-y stuff. Yeah, here. So, re-jiggering some of this, make it look more like Morelin's environment and see if I can, you know, get close to the performance or whatever. We do have some clusters and labs at Red Hat that I could run some of these things in. And then just figure out, you know, like, is there more storage optimization that I need to do or those kinds of things? So, that's kind of where does it go. You know, again, it was really, it was a thaw-dish experiment. Like, can we run these types of jobs? Grav, like I mentioned, is the PM that's handling a lot of the HPC stuff. Just doing more HPC things like this, not necessarily more open-foam, but just other HPC things and see how do they work and just lots of blogs and other things. It's a long answer. I, sure. Go ahead. Yeah, I noticed under the MPI run command, you didn't have the allow run as root flag. Just curious, like, did you have any issues with setting it up to, like, run as a user instead of run as a user? Yeah, so where was it? Privilege. So many images. Lots of memes. Nope, that wasn't it. It was... I don't remember. Anyway, we'll, we'll, we'll say here. So the, the, most of the challenges I had were because of SSH and because of the open-foam container. And so the open-foam container expects to run as a very specific user and trying to run SSH as non-root is like a monstrous pain in the butt. And so for the sake of just, like, get it done, I just kind of blew the permissions wide open and tried to do it however I could. Red Hat has done a lot of work around randomizing user IDs and other security enhancements in OpenShift to, to reduce the blast radius of, like, misbehaved workloads, so to speak, and be more secure. And so we have some weird tricks that we've done in our builder images to, like, do things to Etsy password as the container comes up so that things think they're running as the right user. Anyway, I just, I didn't want to spend the time to go through it. So TLDR and your question is, yeah, I just blew the permissions wide open to make it run, but you could probably spend a little bit more time to make it run not like so, so badly. You wanted to see one of the, this one? Thank you. You're welcome. No question about this slide, just wanted to see it. Yeah. Okay, cool. So are you or Red Hat doing more on Kubernetes and HPC or was this just kind of a one-off experiment? For me it was a one-off experiment because of that email that came in, like, you know, they wanted to do it in VMs and I said, well, you probably don't need to do it in VMs. You could probably just do it directly with Kubernetes. You know, Red Hat is not going to suddenly become an open-foam company, right? Like, that wasn't the purpose of this experiment at all. It was really mostly, like, what does it look like to use the MPI operator? Red Hat is invested in the Kubeflow project community for other reasons around AIML. So there happened to be people that I could talk to easily to fuss around with this thing. But, yeah, in terms of, like, any kind of strategic roadmap-y thing, like, it's not this specifically. It's more generally, like, we are exploring how to do HPC in a Kubernetes world and what we could maybe productize, you know, or just what we can support, who we can partner with, you know, and so on and so forth. Of course, we're doing a lot of great work with IBM in this arena, but there's a lot of other companies and projects that are out there that sort of play in this space as well. Okay, and I was going to do a follow-up. We got time. Okay. When you were running in AWS, were you using their MPI drivers? No. So, open-foam and open MPI and Kubeflow don't require any kind of special driver-y bits. It was really just, I did a vanilla open-gift installation on AWS. You probably could have also used EKS on AWS. I used the Open Data Foundation because I needed RWX. You probably could have used EFS. You know, you just need some central RWX storage that everybody can read and write to. And the only reason I needed RWX was because the launcher did the initial writing and decomposition of the job and then wrote that out to the shared storage, and then all the workers needed to access it and then write their stuff back. So like, if I had done it in a pipeline-y thing, I might have actually been able to get away with some other way to do it, but again, I just didn't explore that deeply. But yeah, there was nothing special about the way I was using AWS or what I was doing on AWS. If I had had a large cluster of hardware at my disposal internally easily, I mean, we have a lab like my group. I just didn't feel like figuring out how to use it. This was easier. We have a system. I can order a cluster. It's like... Anyway, yes, sir. Question for context. So what is Morland's workflow right now then if they're not using something like this? What are they doing to get there? He has a rack of old Dell 1RUs. I think he's got like six or seven of them or whatever. That he was using again. And it was mostly like, I wasn't trying to get them to go Kubernetes or anything like that. It was more like, well, I know somebody that uses this tool, so I can at least poke them if I run into a jam. But yeah, their existing workload today is just 100% bare metal, no cube, no containers. It's just like a blunt do install directly on the servers and then open foam installed in the OS. Well, no, I mean, it's using MPI. So he's got a head server. I mean, it looks a lot like that in a way. Like there's a head server where he runs the preprocessing stuff. And then that's done on a NFS share. The head server is also the NFS server. And then they do MPI run, which then goes out to the other more powerful servers. And then they are pulling from the NFS share and doing the actual processing. And then the reconstruct happens at the end. And then it spits out the cool image. Hi, great talk. I was curious with regard to, and I'm not super familiar with open foam. Do you have to, how do you handle the scheduling of all the, yeah, so all the, all the parts on the, on the right hand side, do all of these need to be scheduled in at like one go? Essentially what I'm trying to say is, is this a gang scheduling problem that we are trying to solve here? Yeah. So I'm going to partially answer your question because I don't fully know. So, so take what I'm saying with a grain of salt. The MPI operator in the actual job definition, and I don't have the full job definition, you, you write a pod spec effectively for the launcher and a pod spec effectively for the workers. And you tell it how many workers you want when you run the job or sorry, when you instantiate the job instance of the CR, the operator creates the launcher pod and creates all of the worker pods. And it pretty much expects that they all can run at that time. And I think if it can't run all of the workers, it just sits there and waits. So it doesn't like run one of them ahead of time and wait for it to finish. And then, oh, now I can fit another one and run that on the back. So I guess it is a gang operation. Like it wants it all to happen at once. I don't know if that's a limitation on the MPI operator. Like if, if there's a way to tell it, like it's okay to run something ahead of time. I just never looked into it any deeper. And when you ran this on a cloud provider AWS in this case, did you have to like pre scale the compute to be able to handle all those parts and then scale it down? So I, what I wanted to do was I actually wanted to do this as a cluster auto scaling job. And so the way open shift is built, and you can do this with, with most of the Kubernetes, um, there's a way to make open shift auto scale itself based on workload demand. And so what I was hoping to do was have like nowhere near enough actual nodes and then run this job. And then when the cluster auto scale are detected that there were pods waiting, it would start jacking up the number of nodes to actually run enough stuff, but I just, I didn't, I didn't get there. Um, so you wouldn't, you wouldn't necessarily have to have that ahead of time. The only caveat I guess is I don't know what the timeout is, if any, on the MPI job operator. And so if it took like a really long time to get enough nodes to then schedule the pods, I don't know if the job would have already failed because it was like, Oh, I didn't get to do it. Nevermind. I give up. Like I'm not sure what the timeout kind of deal is. Yeah, that makes sense. My next question is on, actually you mentioned that multiple pods were running on like one bare metal instance, one VM. Is that accurate? Did you see resource contention when multiple pods were running on the same VM? Yeah. Um, no, I didn't see resource contention because the pod, when you write the pod spec, you specify like a, uh, a request and a limit to do the best, sorry, guaranteed quality of service tier. So for those of you who've ever played with the quality of service tiers and Kubernetes, like you've got requests and you've got limits. If they're the same, it says like, make sure I absolutely have that much to run or don't run me. Um, and then underneath the covers, you know, the Linux kernel and C groups and the scheduler and all that other stuff, make sure that there's no contention. If you wrote the spec for these badly and didn't have a, um, a request and a limit to put you in the quality, uh, the guaranteed quality service tier and you ended up in either best effort or like, uh, whatever the other one is, like you could totally end up with resource contention where it just would throw two pods on the thing and they're both trying to use all the CPU and they both, one of them starves the other one out. So like you definitely want to put limits and requests to make sure you're in the guaranteed QoS tier. Um, otherwise you'll end up with contention. Yeah. But that's like normal. That's not specific to the HPC stuff. That's like generic Kubernetes stuff. Yeah. Just wanted to, uh, confirm. So when it came to the cube flow, it was, uh, providing your MPI operator. So then all the, uh, communication from the launcher pod to the compute pods, uh, was just being solely managed through whatever cube flow provided you. Correct. Uh, at that point it's all, uh, where's the SSH picture? So it's, it's all still SSH between the, um, the launcher and the workers. So the operator ensures that the launcher pod is created and the worker pods are created. Once all of the pods are up and running, the job or the script will run inside the launcher and that MPI run command MPI itself will then look at the host file to find, well, there, it'll look at the MPI host file to find all the targets, but those targets are all just pod names. And so then literally like MPI fires off a bunch of SSH connections into all the workers and then executes whatever. Yeah. So that MPI run MP four interphone. So everything after the four is what gets run, um, effectively on each of the workers. Um, so, so the, all the cube flow, all the MPI operator part of cube flow is doing is watching for those. Um, that's not bright enough. It just washes for these and then creates the pods and then executes the command. It's not managing any of the like communication stuff. That all happens effectively over the Kubernetes service layer. Well, over the Kubernetes software to find network, I don't think it's actually even using services because it's talking directly to each pod by host name. I think that's where I was curious where the obfuscation was happening or the, the, yeah, of the traffic. It sounds like, you know, cube, cube flow is providing you the MPI. So if you have questions regarding MPI, you got to look at cube flow source code and then regarding, uh, uh, like in the, the communication outside of that SSH or, or otherwise, um, you're looking more back to Kubernetes itself. Uh, yeah, sort of, I think, again, most of the communication is coming from what, what is happening in the MPI run commands from a like network or storage communication perspective, the, the SSH connections from the launcher to the workers is just over the local SDN, whatever network you've configured for your Kubernetes, it doesn't have to be a software to find network. Some people who do it on bare metal, they don't use an SDN. Um, and then of course you've got the storage communication, which in this case is, is network communication, um, between the workers and the, the storage that contains the little job bits and pieces. Um, from a tuning perspective, like open from is pretty good about the storagey bit because basically all the workers, like they, they load it up into memory once and then they do their thing and then they spit it out. They're not constantly talking to the storage, but if you have a different type of MPI workload that is constantly rewriting to storage, then of course that storage network is going to be important to ensure that it's got, you know, high performance, um, uh, characteristics or that the disks have high performance or both. And then I haven't had a chance to look at your blog posts, which I look forward to, but, um, when it came to open foam, I take it, you probably just pulled, uh, like a Docker file from somewhere or pulled the container. Yeah. I think I had to rebuild their image to make like a mine. Oh, I had to rebuild their image to change the SSH configuration to make some of that stuff work because the, the open, they provide a container for you to do stuff in the open foam project, but the container that they provide is like super hardlock to one specific UID and like behaves weirdly if you don't use that. And so I had to like mess with the bash RC to make it load the right path, these stuff to find all the tools and everything. It's a pretty minimal change, but basically, yes, I'm just using their, their container image and all anything I had to do from a container image perspective, it's all linked in the blog post and it's in a repo. I think there's a couple repos that have the different bits and pieces. Okay. Thank you. I have not, I know we have customers that do, do some of those more advanced network storage things. I don't even know that I have access to infinity. Somebody I read had somewhere in a lab has infinite band that I could try. But the reality is that like that's kind of invisible to the environment. It would be more of a like the beauty of Kubernetes into a certain extent storage and the kernel and all the other things is like that stuff shouldn't kind of matter. Like maybe I would have to define a PVC that exists as a, as a, as an end point on that infinite band storage, right? I don't know. Is infinite band like ice cozy-ish or is it more of a fiber channely or none of the above all of the above. Anyway, TLDR. Did you hear any of that? So you may have to attach an infinite band devices directly into the pods, which is, which is something that you can do. There's ways to attach devices into pods. It would probably make the, the wording of the, the job and like the pod definitions for the word launcher and the other things. It would make that kind of start to get hairy because you got to like be adding storage definitions to everything. I had to do that anyway to make these connect to the RWX. But if you also had to add devices to the pods, like it would, it would just get trickier. Yeah. And I don't know if open foams commit like tools speak infinite band. Yeah. All Kubernetes is doing is, is effectively, if you think about the, the path of how pods get launched, like Kubernetes tells a kubelet on a node to run a pod and then under the covers cryo and run C and all these other things are being instructed on how to bring up that container. And in that process that includes like, you need to attach this stuff from the host into this container at this point. So in the case of infinite band device, you know, it would, like if you think about a pod man or a Docker run command, it would be like, you know, attach slash dev slash whatever into the container at slash dev slash whatever. So there's no, there's no knowledge required of Kubernetes of infinite band, unless you're talking about like some CSI infinite band driver thing, but that would be baked into the CSI driver for infinite band. Anyway, so back to the original question. I did not try infinite band. I might be able to find some somewhere, but I don't know that, that open foam would know what to do with it. Even, even if I did, it does. Okay. Well then there you go. So you alluded to a couple of race conditions and the MPI bootstrapping process. Can you speak a little bit about those and whether you address those? No. So you said it was 512 ranks. Is that correct? What did I do? So I did 256 pods with two CPUs per pod. So that ended up being a total of 512 CPUs. And I think it ended up being, I don't know how many that ends up being per node. I'll make up a number. It was like, presumably it was one rank per CPU. One what? One rank per CPU. One rank? MPI rank. Oh yeah. Yeah. Yeah. Yeah. I mean the MPI run command was pretty vanilla. And the host file that gets generated is also super vanilla. Yeah. So there's no, there's no fancy MPI stuff going on. The reason I ask is that we ran into bootstrapping problems with about 80 ranks or so. So we'd see race conditions where, whereby the workers would report ready. Would, sorry. The workers, the MPI launcher would start before all the workers were ready sometimes, which we caught a crash in the bootstrap. And then the other thing that would happen sometimes is the workers would start and would report ready before SSH was ready, which then would cause the MPI bootstrap to fail again. Okay. Yeah. Yeah. Yeah. So I definitely did see this problem, especially when you're dealing with large numbers of pods, the launcher pod will fail like a bunch of times because nothing else is ready yet. So MPI comes up and it's tries to do its thing. And it's like, I couldn't find half of these machines. I don't know what to do. And it crash loops and then kind of backs off. So I don't know that they could probably do things to make it smarter. One, number two, I'm not sure if you can define liveness and readiness probes in the template spec for the, for the job. I mean, it should support the full pod spec, but the challenge is like you'd have to write a really nasty liveness readiness probe for the worker to like SSH into itself. You can, you can do a post dark hook where you just like check to make sure the ports, port 22 is open. For example, yeah, I mean, you could do, you could do things. It's just, I never, I never got to that point. Fortunately, the, the pre-processing, and this is why doing it is a tecton pipeline could be beneficial because all the pre-processing steps would get done. And then the only thing that's failing is the MPI run. If you look at this, like blog mesh set fields decompose, they're happy to run multiple times while it fails because ultimately it does the first three things, then runs MPI run, and that's when it fails. So if you have other steps that come before the actual MPI that are not, I'll use the word idempotent, which is probably the wrong word to use here, but we'll, we'll go with it for now. Like if you have things that would blow up if they got run twice, this is going to be bad, bad news bears, right? Okay, we're getting a, we're getting a hook. Thanks very much. If you have other questions, I'm happy to answer them outside. Again, the presentation is uploaded to the portal. You can find me pretty much on lots of places on the internet. I appreciate it. Have a great trip home, travel home safe.