 Okay, it's recording. So welcome everyone, we'll have a restart of our research and user group finally after quite a long stop. So just the topic today will be HPC and Slam integration and Guard will talk to us a lot about it. So I'll pass it to him quickly but just a few items as I see quite a lot of people joining. If you can add your name to the attendees here in the Google doc that I put in the chat, we'll go through introductions for those that are joining for the first time the group just to know each other. One thing I will just mention before is that we're looking for a new co-lead for the group. Jamie moved to other roles so if you would like to help out it's mostly around keeping a healthy backlog of the topics to be presented and then reaching out to speakers. That's pretty much it. So if you would like to help I'm glad to onboard and eventually I would be glad also to delegate to others to take over. So we can start with introductions. I see Dennis, Priyanka you are joining for the first time as well. That's awesome. Now I'm scared because I actually have a slide with Priyanka face. Oh don't worry at all. Nice to be here folks. Thanks for letting me hang out. Awesome. Then we have Dennis first time. Yes, hello. It's the sound working probably, hopefully. Yeah, you're from where? From Finland, from Alt University or my employer. Doing my master's thesis studies working with HBC Sloan integration. Amazing. Thank you. We have Nikos, you presented but yeah as an attendee do you want to say something? You have your sound. I see his microphone is green but we don't hear him. Like we can come back to him and then Raj. Let's test. We can hear you. Perfect. I had to configure my microphone. Yeah, so this is my second time joining. Well, technically it's the first time I joined was about two years ago when I presented Containers Assets, my project in the CMCF to the user group. And yeah, I will concern alongside Ulfric Argo. Awesome. And Raj will check his microphone. I don't see him in the list. Can come back to it often. Anyone else first time? Davide, you want to introduce yourself? Hi, Ruin. Davide Pastorino from Italy here. We are here with two of my colleagues because we were curious about the Kubernetes and Slurm integration talk. As we are a company that specializes in HPC based in Italy and I am a solution architect. So I'm very interested in integration with Kubernetes as well. Awesome. Pierce, I see a few people I think first time as well. Take the chance. He has no microphone but he's okay. But Pierce is from the SKA office and he was also in our ad user booth at the last CookCon. He was stuffing the booth as well. And I think, did I miss someone? Vitsa? Yeah, Vitsa Albus also the first time here working for SIRV. That's the Dutch NRIN in the multicloud group. So we are developing, yeah, Kubernetes clusters and we are looking how to integrate it with HPC facilities. So that's the reason to be here. Awesome. You're at the right place. All right, Diego. We have two deals with Diego Bakken. Hello, I'm Diego Bakken. I'm working with David and with System and I'm a solution architect in HPC. And I'm here to listen to talk about the Kubernetes and Slurm integration. Awesome. And Matteo? Hello, everybody. I'm also here to listen to the talk. I'm a colleague with David and Diego and with System. It's a company based in Italy. Amazing. Thank you. Awesome. Thank you, everyone, for joining today. We'll have a few of these sessions coming. So this will be pretty interesting. I will stop sharing and I guess Eduardo, you can start again for giving us this introduction. Anytime. So thank you, Ricardo, for inviting me. This is kind of like a post-CubeCon presentation. I had the opportunity to share this idea with Ricardo. Someone that is here, I see on that a participant, but it's not on the Google Doc, is Andrzej Beltre. So I have been working with Andrzej. He works at Sandia National Labs in North America and the US. And we have been brainstorming this idea on how can we better integrate that old-school HPC systems and the new cloud-native. So I call this presentation Cloud-native Slurm. And it's our point of view, the way that we see Slurm integrating into cloud-native ecosystems, different from current implementations that are out there. This is kind of like my take on it. Who am I? I have quite a bit of HPC experience. I started doing containers for HPC at Scylas for those who know singularity containers. So I was kind of like the initial contributors there. Then I moved it to OpenShift, trying to get OpenShift to work for HPC environments. And now I work at NVIDIA doing similar what I did at OpenShift, but now for focused on accelerators. So I'm now focused on helping people to get their GPUs working in Kubernetes and not just working, but at the scheduling level, right? Like how to optimize resource consumption for me. So that's kind of like a bit of background of who am I and why I'm very passionate about this topic. So I will carry an agenda on what efforts do we see in the Kubernetes ecosystem? What is Slurm? What is KCP? And then the final idea. And I will open the room for some Q&A. So, Kubernetes HPC efforts, right? Sometimes it's feel like this, right? Like, Kubernetes is awesome, but we are somehow trying to use it for things that it was not built to, right? So there's a lot happening to make Kubernetes work for AIML, for these big distributed workloads. So then we get into this ecosystem. And this is like the perfect way to talk about this, right? Like, we have right now for those who are signing to the mailing list, we have two new working groups that are visit, that is the working group serving and the working group accelerator management. And to me, this is all part of the same, right? Like, there is a growing need to make Client Native better for HPC in all sorts of ways. AIML, that is like the busing word right now, but like we have here Ricardo that he's from CERN. So like we want to use Kubernetes for scientific real purposes, right? Like curing cancer or looking at the stars or many other things we are getting to do with Kubernetes, but we are realizing that your net was not designed for that. And that we need to to your net is to get it to do what we want. But to me, it's surprising, like why putting so much effort into twisting your net is to do something. If there are other tools that already do it. So it's because your net is provides that developer friendliness, right? So that's where we are here. And we want to move everything into a cloud native way of doing things. There are some good things that have come out of these efforts like this working like this research usage group, or like the working group batch, new tools like Q, like the job set patterns like the cellular plugin that allows you to kind of like inter interject into cure net is way of thinking projects like the flux framework operator. But all of them fall short on what a real HPC system is today, right? So if we compare Q to what a slurm is or to a like Torque or ACC Condor, those tools are being tested and proven for like these high scale systems on where cure net is still growing together. Another thing that I want to point out is that Q the job set, they are pre scheduling tools, right? Like Q and the job set help you to optimize on a pre scheduling decision. Once Q releases the job, right? For those like I'm not going to get into much of how Q works. But basically Q poses a job. And then it on poses the job on some things have been checked out. And then it handles the job to the cure net is scattered, right? And what is once the job is on cure net is scattered hands, it's like we cannot do much on it. There is a scheduler plugin. But the thing that I don't like about the scheduler plugin for cure net is is that do you end up in a situation on where you have to synchronize to a scheduler's and to a states of the word, right? You have to be moving information from pure net is to whatever you have in your escalator plugin. So your scheduler or decision management is aware of how the system is going, like if which nodes are super utilized, which nodes are not that much utilized. So the scheduling decision is optimized, right? So like, that's why I don't like the scheduler plugin, because you land into these synchronizing to a scheduler's and to a state managers at the same time, and that can get into a lot of errors, because I've been there. And that also can get you in a not optimal situation when you want to do high performance scheduling, right? As a community, we also have that was not good. We also have the MPI operator that is a great tool that it's was developed for Qflow. And now it's kind of like being proposed as a standalone tool, because it is very important for AI ML training, mostly when we are doing this distributed training, or for any scientific tools I have seen, even people trying to do like bioinformatics in on top of Kubernetes using the MPI operator. So a quick introduction to Slurm and now, Hey, Priyanka. So sadly, the YouTube recording of Priyanka saying that she loved Slurm was cut. So I kind of like hack it here. But it's really amazing when when Priyanka goes and mentions that Slurm plus Kubernetes is like the drive of doing things, not like one on top of each other or like those weird mixes that we have seen some projects out there. But trying to get them to live side by side is a good approach, right? And this just happened two weeks ago during a cube con keynote, right? Like something that the HPC ecosystem was not even aware of, right? Like, I was chatting with someone from skmd, a Slurm company. I was like, Hey, guys, your logo was at the keynote, and they were not even aware of that, right? Like, this is fresh. So we as a cloud native community, we are realizing that we need this and we need a community solution on how can we use a Slurm when we need it. And for the things that Slurm has been designed to do for over 20 years, and then fall back to Kubernetes for everything else. But it's not just like falling back and killing a Slurm and turning it on and off every time we need it. But a way that we can use a Slurm as another tool, right? Like for those that have seen the cloud native landscape is a growing sheet that at some point, I think they are going to start selling the actual bed sheets for everyone, because it's so huge. But what is a Slurm for those that know the cure net is architecture, if you really look this picture like from really, really, really far away, it's kind of the same, right? Like is the control plane is the nodes and is the API server kind of, right? So we have the Slurm D in this picture, the yellow part will be to the cure net is people will be kind of like the API server, right? Like is the most important thing where all all the other components kind of like reached back to. And we have the Slurm DVD that is the database that to cure net is ecosystem people will be easy, right? Like this would be kind of like the easy where all the state is managed and is stored there. And we have the the demons and we have the nodes and we have like all sorts of CLI tools for interacting with a Slurm, depending on the things that you are doing, right? Like a Slurm has this very optimized way of like, if you're running just a batch job, or you're just doing like a s run, or if you're doing a MPI job, or you're just queuing or you're checking how the queue is going, they have different CLI tool for it on where in the cure net is ecosystem, we have Qtl for kind of like everything, right? We have this one tool to interact with everything. And it's one of the things that we like from cure net is like this single API to manage them all. But why is Slurm is so awesome as Slurm supports PMIS as Slurm supports MPI on this largest HPC cluster like these things that we read on the news that they win awards and that it requires like a full city electricity for a day, just to keep these systems running versus on cure net is we have never tried kind of like a system that big, right? Slurm supports containers pretty well, right? So you can run from singularity and other HPC based containers. It also supports regular Docker and Podman, what we end the cloud native ecosystem are used to, right? What got me into this idea is learn recently when I said recently like in the last couple of years, not like so fresh, they introduce it as learn rest API. So as learn rest API is a open API compatible 3.0 compatible is a stateless. So it's just like basically, you do a request and it throws you like what happened to the request, but it's not like storing a cache or saving anything that is happening during the API calls. It has two modes in it. Basically, you can talk to the run as learn rest binary and get it to do things. Or you can do it as a unique socket or a TCP that is constantly listening from from calls on a specific port that this is the part where I want to focus on. That's why it's highlighted here on the security side. It supports JSON web tokens for notifications. And you can have like multiple users which two communities will be kind of like map it to what we have as service accounts, right? So this is where we are already starting to see that we can have some mapping between what is learn can provide the rest API. That is what we like at the cloud native ecosystem by like doing everything via APIs. And how can we from a cloud native ecosystem that we like API, we can start calling as learn using the API, right? And we have this really nice tool that we as a community have been building that is called KCP. A project that was born pretty much during the pandemic because it was born like during the 2020-2021. And it's a cure net is like contraplane without notes or pods, right? So it's a very interesting picture. This was one of the talks that they gave in the last KubeCon on where KCP is basically a way to have a uniform API to communicate with any platform, right? Like you have a platform that you want to expose via an API and an API that we all know that is the cure net is API using kind and using Jamel because we all love to have our laptop full with Jamel. So KCP help us do that. Like it help us put a API from like from facing to any platform you want, right? But how does that look like, right? It's basically a very stripped down version of cure net is is a contraplane with API server, control manager, because the control manager is pretty much the secret source or the magic behind cure net is what we all love to build operators and controllers and this reconcile loop is what allow us to do that. It has it see and we can then build custom controllers just utilizing this like this is a high level view of what KCP is cure net is without pods without containers without notes. Once you deploy KCP in your laptop, if you try to do like cube CTL, it gives you a cube config. By the way, once you deploy KCP, it gives you back a cube config at regular cube config. But if you do cube CTL get notes, it will fail. Like it will actually throw an error. If you do QCTL get pods, it will throw you an error. But it actually knows about secrets or back like anything like role, role binding, all of that is still there. It has namespaces and it has the most magical thing, custom resource definitions on which we can build controllers and we can then get crazy doing whatever we want. Now, how do we get to cloud net tips learn? So after I show it Homer in the first slides is like after breaking everything trying to make cure net is be an HPC platform. Why not using these tools that we as a community in the cloud native have built to make a slurm interact with cure net is right. So now we are basically putting a translator next to a slurm so a slurm can sit down at the same table with cure net is and they can share information, but they sit on different sides of the table. Like they are different clusters. They don't have to manage or synchronize a state of the cluster because they are their own clusters. And then you can run a slurm optimized for what it is optimized. And then you can have cure net excited using tools like that's why I put here infrastructure. There are multiple tools out there from in the HPC side of the house that we have werewolf or we have what used to be bright computing that's now inside of NVIDIA or many other tools that they can help you pre provision and out very quickly to a cloud provisioner like you can just move AWS Google or Azure machine to be one type of node to another type of node, right? So you can have two clusters and just like change nodes. So make your cure net is closer bigger or make your slurm cluster bigger, but then just have a single API like a single control plane on which you deploy jobs and you can have a smart like a smart router here in the middle that says if this job is let's say like an MPI job, just send it to a slurm. And the good thing is that you can send the entire Jamel as we right now know for the MPI operator that we have already defined MPI job when using the MPI operator, you could just throw the entire Jamel to the slurm KCP cluster and we will have a controller that will translate that into a regular slurm job using the slurm API, right? Like there is no need for extra complicated things or like cure net is managing pods of slurm and trying to synchronize the state of the word, but it's a cure, it's a standalone slurm cluster that understands cure net is Jamel, right? So it's a uniform API because we can have a single control plane. It will enhance developer interaction because we will just do Jamel as we like to do Jamel. It is multi-Q compatible. Right now the Q project is exploring this multi-Q option on where you have a small cluster as we see here on the slides and Q is basically deploying jobs to all the clusters and then on the other side the slurm KCP will get the job and if the job is compatible it will run it and it will communicate back to Q that the job was accepted, right? So that's how multi-Q works. It will query each cluster that it has on this list asking like can you run this job? Can you run this job? And we will tell multi-Q like yeah, I can run the job because a slurm will now understand Jamel, right? And it's cloud and on-prem compatible. We can still have nodes and jobs concept, right? Because a slurm on its API it provides a way to query a slurm about the state of the nodes. So we can communicate that back to a central control as or another talk that was really nice not to like self-promote myself but we as a community are promoting a cluster inventory API that will live in this high-level control plane and then we could communicate to this cluster inventory API how my slurm cluster is going, how many nodes do I have, how many nodes are healthy, the CPU utilization, all of that in a way that other Kubernetes components understand. So we can just translate from the cluster out and we don't need to be complicating things synchronizing the state of the word between a slurm scheduler and Kubernetes scheduler. How it would look like internally once we communicate with KCP that is going to be the the front-facing control plane KCP will have a controller running that will translate all the Jamel into a slurm API. Once everything is translated into a slurm API it becomes a regular slurm workflow on where a slurm is going to communicate with a slurm internal command controllers and it will deploy the jobs to the compute nodes. So it will basically look kind of as we know Kubernetes architecture but is now a slurm and we can communicate with a slurm via Jam. The roadmap for this project the proof of concept I think is the same with public repo I think we will be open sourcing it in a couple of months since my friend Angel that I think is still here is from a national laboratory we have to request some permissions and things like that because I think San Diego is like a weapons lab or something like that politics that I don't want to get everyone into it but we are requesting for all the required permissions and things like that to open source it and we will be open sourcing the proof of concept really soon so everyone can run it can run a demo and we plan to have a paper submission for the super computing conference in KubeCon North America in November so that's kind of like now my idea on how this is going and now I open two questions. Awesome thanks Eduardo. I think there will be quite a few questions feel free to drop them in the chat or raise your hands and I will take notes in the minutes as well. Anyone want to start? I will start by apologizing with Priyanka I had you in the slides. Oh don't apologize I was honored I was not expecting that. Any question to start? Otherwise I can ask one and then we hi Giuseppe. Yes hello everybody I'm Giuseppe from SURF. SURF is the Dutch National kind of IT research center and I'm the program manager for for cloud and edge technology now after this introduction we are trying to solve a problem basically our problem is how do we fractionalize our supercomputer how Kubernetes can help doing that so one of the one of the the angle that I would try to understand in this project is what problems are we trying to to solve because I might see the problems for our research community and I would like to to understand when you approach this project what was the problem that you wanted to solve maybe it's a bit of a generic question but I want to see if it's aligned with our strategy and agenda. Yeah I can start with that again I think I'm going to waste his name so Angel I see him here in the list he's in a laboratory where like a 100,000 nodes is just like the beginning of a research so Kubernetes is still not made for that and they want to run these huge shampy jobs or these huge deployments and they want to also use these new technologies like PMIX that is still not supported by the MPI operator and that kind of like force them to still use Slearn but all their developers and the new researchers they want to use Kubernetes and they are also running Kubernetes internally for some like CICD testing and like minor things right but the new researchers and new developers they want to use Kubernetes and they know how to use Kubernetes but then they face that their cluster where they are going to run their actual research is this big all thing not saying that Slammer is all is that that to the new researchers the new people it looks like all right so what if we can have the best of both worlds rather what if we can give this new wave of researchers new wave of developers a front-facing interface that they know that they are used to managing the Jamel and then we the upside of the house we manage things right like we have these interfaces where we translate Jamel into a Slearn native objects or a Slearn API right so the the need that got us to this idea is that we have these clusters that have been running a Slearn for years and they are not going away and they cannot be reprobation into Kubernetes clusters in the next 10 years or so right like this is these are like long running projects so it's like how can we interact with that Slearn cluster because we know we cannot reprobition it but how can we give it like a fresh look with a cloud native tool yeah thanks thanks for the answer I just want to add one last thing and then I'll shut up it's it's great because I think we are we are aligned also from our perspective as a search but I have another another follow-up question one of the I am a Kubernetes guy by you know by you know the uh background but we always have lots of cultural problems when we discuss Slearn and HPC in the community of HPC is very let's say conservative and from our point of view it looks like we are trying to explain why things could be better with the Kubernetes architecture but I don't know for you guys but for us it's kind of going to help heal cultural bubble and if you have any any any tip or any point of view on that we will we can gather our forces together you know to be more in change yes yeah I think I can start but Ricardo or Priyanka can't follow here I used to be a hard on like Kubernetes everything person for quite a few years but and even that I come from like a old school HPC right but now I understand that we have tools for everything and that's why I said that Homer is light at the beginning where Homer is trying to use a drill for hammering and now we like Slearn is there Slearn is perfect it has been going for 20 years it runs on like top supercomputers why we want to twist your net is to do what Slearn already does we like at the cloud native because it's same we are already used to having this huge landscape of like there are thousands of projects which you just choose what like you have a need and you go and pick from the landscape and you use a tool to solve your need right like we don't twist the existing solution to solve a problem we just go and choose a tool why don't we just add Slearn to this big cloud native landscape Slearn logo should be there right so instead of trying to twist many tools and cure net is to do what Slearn is doing is just like get Slearn into the cloud native landscape big sheet and then away for cure net is and all the other tools to interact with Slearn then Slearn becomes just another tool on on the ecosystem right like it's just a tool so that that's my point of view with this project right like I want to get Slearn to be in the cloud native landscape and to be seen as just another tool right like Slearn means a tool because inside Slearn you also use other tools like MPI MPMIX they are not part of Slearn they are separate projects that you then use so it's like that's my point of view now yeah Priyanka do you want to add something yes I'll just I'll say that cultural challenges are very real the good news though is that we have all been battling them in different areas for so many years now when Kubernetes was open sourced in 2014 the whole question was oh developers are separate from operations right and the whole shift to the dev ops mindset or the SRE mindset where engineers were also in operations took a while and a lot of folks on the on both sides whether it was the sysadmin or the business logic developers were resistant so I would guess that the kind of resistance you're facing to set is with your researchers is similar to that and at least based on past experience what I can say is that as more and more new researchers developers start going into this field and as Eduardo was saying they are more Kubernetes native and they want to use it the slurm folks will slowly see that oh this expands my influence this expands my impact and there will always be a small contingent that will never want to be at all involved and that's fine that's their choice to make and you cannot convert everyone but it will be a slow and steady trickle I would say so my advice is just keep at it keep presenting stories of success of how new use cases have been developed by using these technologies and everybody likes to have success and they will follow along and be patient because it'll take a while thank you thank you it's nice to not be alone in this adventure thank you yeah I don't think you're definitely not you're definitely not alone we have a similar experience internally we have new workloads that got used to having Argo workflows and using all the tooling on top of Kubernetes that makes things much easier GitOps kind of deployments and now we face the need to access a lot of the resources that everyone wants like GPUs that are backed by supercomputers and slurm clusters and we have to find a way to bridge between the two internally we have the same issue Dennis you have your hands go ahead yes this is also quite relevant to my work so at Alt University I'm working closely with the Finnish Center for Scientific Computing so CSC specifically on the Ruby supercomputer on how to get kind of the Kubernetes integration going there because we're we're facing a really interesting challenge is that most of the the code that is being executed on Lumi right now is very GPU heavy so all of the GPU time is basically utilized and we are mostly idling the very powerful CPUs that that machine also has and people are really interested in finding solutions on how to utilize those CPU side resources while the GPUs are doing are busy doing simulation work or LLM so or whatever you have and for the CPU side people really want to do dynamic bursty workloads and maybe something really complex like running a Kafka and Spark or something like that in parallel and for this purpose we have this similar conflict that that the researchers are really interested in in I brought up Kubernetes as sort of a way to deploy these systems very easily however the actually getting Kubernetes and this ecosystem running on that supercomputer I've gone back and forth with CSE quite a lot and there's this resistance in the mindset and they are stuck with their with their software visions and and what they can support on those nodes so having something like KCP or interlink which will be the topic of our future talk in this in this use group as well will be sort of a key enabler of seeing that Kubernetes success so my my goal is essentially to provide an intermediate a kind of bridge to bridge this ASPC world and the Kubernetes space before we can eventually then maybe just run playing Kubernetes on the on the HPC nodes all together and skip all of the intermediate parts but something kind of we need an intermediate layer to to like Priyanka mentioned to show this success and be able to evaluate it in the current context and this is kind of a really important focus of my work at the moment excellent this is a very good discussion any other question I see a couple on the on the chat but feel free to raise your hand if you want to ask them directly if not I can ask are you by Brian do you want to go ahead you have your hand hand raised or Sanjay yeah okay can you hear me yeah I can yeah okay sorry I just unmute so yeah very good talk so I hear some keyword here it's that we try to to introduce Kubernetes on HPC or supercomputer or HPC data center or on-premise but actually I come from a the Kubernetes world and we try to benefit from all this we try to find the HPC capability within the Kubernetes actually we are using manage the Kubernetes like AKS, GKE so for me there are two major issues today one of these we try to utilize the unlimited cloud resource so which means that we rely on the cluster auto scheduling feature so which means we need the computing and we can quickly provisioning the environment and deploy the image and run our computing workload there and when we finish we just decommission all the environment so I'm not sure for this by introduce slurm integration will that help on this direction or because this is really our bottom neck today like how fast we can having a 500 node HPC cluster being be ready on this on this cloud I'm not sure if this is for this on-demanding or auto-skidding direction where we talk about really for a traditional supercomputer or the the data center environment Eduardo do you want to answer? Yeah so yeah if if you're a cloud native person because that's is like this slurm conversation is mostly for as others have jumped to say like they are from a university or a research environment where they have this on-prem not a cloud on-prem cluster as like thousands of nodes and it has like a predetermined software installation and it has to be running for the next 10 years because it's under contract and things like that so it's like it's really hard to move from a slurm to another thing right so that's that's why people like Denny's or I forgot the first question name we want to have a way to interact with a slurm from cloud native tools right like from Kubernetes or other cloud native tools but if you are just on on the clouds and you have you want to have a 500 nodes slurm system I think you can get those from Google and Azure I seen some products from Google and Azure where you can get like as slurm clusters on demand and I mean if at some point you want to have a Kubernetes cluster and as a slurm cluster all in the cloud running and you want to move information to one another you are going to face the situation we are facing like in today's presentation that is how how I make these two ecosystems interact right so that's how I see you being able to utilize as slurm right like going to this cloud hosted slurm yeah yes thanks for the explanation I think the so when we I think in the last five year when the CPU time we we maybe we can do this on demand cloud but when the GPU come into this AI journey the GPU is really a limited resource probably this will do more help for the even via on cloud we cannot really really get GPU sometime really just on demand we need to reserve the GPU for like one year or three year contract to benefit I think for that situation the cloud native more turn into a similar to this supercomputer or on-premise situation so I see the value for this project can extend it to the cloud environment yeah thank you awesome so I see a couple of hands raised Sanjay is the next yeah Sanjay yeah hey can you hear me yes oh all right nice presentation Edward so couple of questions one is uh when you talk about moving nodes between clusters how do you reason about which nodes to move and like if you have to do like things like you have to pre-empt some workloads or things like that how do you reason about those things and the other thing is that typically slurm clusters they have a way of operating where you you have multiple users they get allocations and let's say node hours or things like that and the scheduler then tries to do fair share scheduling and to manage you know the resources allocated to these users and so these are typically set up for certain windows how how would that have like get managed if you suddenly take away some resources from the slurm cluster to the Kubernetes cluster so it seems like there needs to be a little bit of you know either some resource management layer to this well as slurm itself is a resource manager I I see team on the room team could you help me with the second one I think you are the best person to answer this one Tim is saying that his microphone is not working but maybe the the messenger ah it does go ahead go ahead go go go sorry I was trying to fix the mic problem so what was the question again oh I'm sorry I was referring to Tim Winkberg but oh okay yeah yeah the other team in the room but yeah actually so I see a couple of a slurm developers here in the in in the room but for the first question is ah how do you reason about oh no microphones ah we need to invest in microphones around here ah how do we reason about the resource management between the two clusters so that's why ah on the last slide I had a picture on like a central ah control plan rally multi-cluster ecosystems usually what you have is like a cluster that you call like a management cluster and it's a cluster with very few nodes compute nodes is a cluster just made for running controllers where you reason about resources right so ah we are discussing and it was discussed during this cube con a couple of APIs to be able to have a cluster inventory of all the clusters that you have in your pool and then how to make decisions out of these inventories right so that's how we will be done right like this is an ongoing effort and ah we are working on that as as a community right like there is a couple of caps open about how to do cluster inventory and we are working towards that the cluster inventory kept just got merged and we are working for ah the alpha to get into the next urinary release so that's that's how we are trying to solve that problem on how do you reason on resource management management between multiple clusters then how do you take a note from a running a slurm cluster and add it to a cure net is cluster how is lorm is going to ah react to that ah that I think that's a follow-up question more suited for like like a it's lorm ah admins that to myself yeah maybe maybe just a to clarify the integration that Eduardo is describing talks to the slurm API so there there shouldn't be a need to reallocate resources right if you have your slurm cluster it stays there and you just submit things via Kubernetes but you don't have to reallocate notes was that correct Eduardo yeah no ah as I showed in in one of the lines show that you can use tools like a werewolf or ah ah bright computing where you can read quickly reprobition a note like change the OS and things like that right so like in an scenario if you have a bare metal you don't have any access to the cloud right like sometimes like universities ah you have a fixed number of notes sometimes you want to reprobition these notes from being Kubernetes to being a slurm very quickly like in less than a minute you can do there are tools to do that so ah ideally you can like have as elastic Kubernetes as lorm situation where you have fixed one hundred notes but you can at some point have a bigger Kubernetes partition or then have a bigger is learn partition in this fixed one hundred note situation okay thank you so we can move to the next one which I believe is Nikos yeah hello can you hear me again yep go ahead yeah okay so this may be a bit on the other side a bit more provocative but so from my perspective I am on the complete opposite end of most people here we are mostly working on HPC's on the computer center side I used to work for CMS which is the virtual organization that actually submits jobs to the to the HPC's to supercomputers so for CMS we're using it's the condor and then from it's the condor we had gap to do a translation to slurm and now this adds another layer to from slurm to go to Kubernetes so my question is now we're three layers deep has there been any proposal or hasn't has people looked into submitting jobs directly to Kubernetes or at least having an entry point like native to Kubernetes that is can coexist with a slurm cluster without going through this many translation layers yeah I can I can reply quickly to that one so there are a couple of projects there's a batch working group I think Victoria is here so if she can jump in she's the best one to talk about this there are two working groups here by the way right yeah no but I mean I mean for the native Kubernetes which was the question there's this batch working group that has been working on this project Q and extensions to jobs and job sets that aim at exactly that making Kubernetes native as a scheduler to to this kind of workloads and then all the other tools can rely on these primitives is Victoria still here Victoria is here yeah I am here but I don't know anything more about it okay okay all right Alex do you want to add something yeah maybe just a PSA on those two working groups they were mentioned at the beginning of Eduardo's talks but there seem to be a lot of people here who are very interested in this topic and I lead one of those conversations the batch working group for the CNCF where this kind of conversation is exactly the conversation we have every time so if you're interested in this kind of conversation Eduardo it might be great to have you come and talk in that group as well but yeah just to let people know there are more conversations than just this one going on about this very thing um please come out I'll hit a a link down in the in the chat if anyone's interested um the one that I run is separate from the one that Victoria is involved with more which is as people have been saying just directly related to Q and all that kind of stuff I had a question about slunk and if anyone has seen it or heard from it recently and how that compares and my understanding is that has a bit more of the addressing some of the questions in the chat about balancing slurm resources and kubernetes resources within the same cluster whereas I think here you've described more just sort of static clusters one is slurm one is kubernetes stuff I think slunk is a bit more integrated do you have any feelings on that yeah that's why I made a lot of point on that slide on where you run into synchronization problems right like two three years ago I was part of a project where we were trying to use the a scheduler plugin to change the regular scheduling for kubernetes to flux that is a a lever more base scheduler that is kind of like similar to slurm and the thing is it works when you are talking like a hundred nodes or down when you start having like a thousand nodes or more than a thousand nodes then you end up in this problem where you have this baby scheduler living inside kubernetes and kubernetes has to be telling this baby scheduler all the time how all the nodes are going right so that's why I was I'm making a point where the synchronization starts to get very complex right like you need to be secure you need to have a controller that is basically getting the information from all the nodes in kubernetes and the kubernetes as it is right now is not the best hpc node based demon for decision making like it's learn right like it is it won't feed us learn with the right information to make optimize scheduling decisions so projects like as long I don't agree with them on where like yeah they will work on as small clusters but again when you start talking about these big clusters like thousands of nodes you will run into a state synchronization issues because at some point is learn that is running on pods it's not going to know the state of the word as kubernetes knows it right so then you are going to deploy a job as learn will think that the left side of the cluster is busy and the right side is is is available but it I just asked just too complicated of a question I suppose I I'm sorry it's all my fault yeah I had mic issues earlier and it was the platform so maybe something's going on in the background talking about optimized scheduling but like high performance computing you will start running into situations right so that's why I feel like as long should be that's the way I said at the beginning this is my take on this situation as long should live standalone install it as it is recommended and we just interact with it right like we just interact with this learn and we can then interact with this learn and make it be elastic with these tools as I've been mentioning but not having as learned to like having any scheduler leaving inside another scheduler is a situation you don't want to leave in okay so out of interest of time because we only have like one minute and a half left apologies I think that will thank Eduardo a lot for the presentation and also everyone for attending this was a very very well attended and active discussion so clearly we have to continue we have another meeting on the same topic scheduled for two weeks from now Diego I see you're here so you're all good to present in two weeks yep yep so Diego also gave a presentation at Cook on two weeks ago and we can hear about what he's been working on with interlink what we could also do if you want because everyone is here if you want to present on topic on any subject reach out and we can schedule we probably would be useful also to get an update on Q and what they've been working on around jobs and job sets at some point so I'll bring that one up and put it in the schedule at some point all sounds good it looks like the Slurm folks are working on something similar to sunk if I'm reading the chat correctly so just wanted to suggest maybe if they ever want to present Nathan you're here is Nathan's mic not working you hear me yeah yeah we can we might just I don't have we don't have anything planned right now so I'll get back to you awesome yeah happy to follow follow it up as well and other than that I think thank you so much everyone for attending thanks Priyanka for for attending as well and yeah talking two weeks and let's follow it up on on Slack as well thank you thank you thanks everyone thank you Ricardo for hosting thank you bye everyone bye bye