 Alright, so we are recording now. Hello everyone. Welcome to another meeting of our CNCF End User Research Group. Today we'll have a nice presentation from Diego about using the interlink setup to schedule pods on Slurm. He will give us all the details about that. We will just start with a quick round of presentations. So if you have, if you're in first for the first time with us, please, we'll give you the time to present yourself. So I see that one out of first time. Do you want to say a couple of words? Yeah, thank you for having me. You can call me Norvo. It's quite different. And I'm a master's student in Grenoble, and I'm writing my thesis at Eviden, producing knowledge at us on the topic of integration in Kube flow with HPC. So I came here to like to get some ideas to meet with people. Raising. Raising, welcome. Thank you. And we have Alan from SkidMD as well. Yeah, hello. It's the first time I just started at SkidMD about a week ago. So thanks for having me. Awesome. Thank you. Thank you for joining. And welcome as well. So I will wait, Diego, is it the first time? Second time, okay. I don't know if you did the round last time. I was checking the names. You want to present yourself as well, Diego, just in case? Oh, wait, I need to give you a voice. Let me fix that. Diego, do you want to say a couple of words? Can you hear me? Yes. Okay. Hello, it's Diego Bakin. It's the first second time for me that I participate to this talk. I participate to me with other two colleagues of the system to the last talk. Okay. We are very interested in this topic. Thank you. Amazing. Welcome. And is there anyone that we don't have in the list yet that is here for the first time? You come forward. Me, I'm here for fifth time. Jose. Jose. Okay, cool. Yeah, feel free to add your name in the agenda, but go ahead. Okay. Okay, I'm Jose from Spain. I was working at a company called HPC Now. And I'm here because in the past months, we are researching a little bit about batch scheduling against Kubernetes. And I have a small experience playing with CUBE and some of our components. And I researched a little bit the topic on different projects, etc. So I'm here because I'm quite curious about what you have to Awesome. Welcome. And we have two more. We have Camilla. Can you hear us? Then we can go to Rohini. Always first question. Can you hear me okay? Yes. Yeah. Okay. Okay. Rohini. Yeah, technically my first time as a format here, but maybe some of you may have seen me at the rug booths at the last CUBECon. I recently was working for the SK Observatory now working on the Swiss part of the SK project and rationing with an HPC facility here in Switzerland at CS. And so we have a good amount of interest in this area that sort of aligns with the SK Observatory's interest as well. Easy. Welcome. You know each other. Yeah, welcome. And I see Pablo. That's a familiar name. Pablo. Hey, hey, Ricardo. Are you? Yeah, my first time here today, although I know some of you already. I'm also I'm working with Rohini actually on the SK project. Maybe. Well, welcome. I think we are okay for the routing of introductions. If we miss someone, just feel free to speak up. And before we pass to Diego, I was just going to say we have already one more session scheduled in this area for the 15th of May, checking the agenda around volcano and patch scheduling with volcano. We will schedule another one. We're still preparing around Karmada for sort of multi-scale multi cluster support. And we have an empty slot for May 1st. So either we take Karmada there, but it's probably better to present after volcano or we might have an empty slot. So if you have ideas, please reach out. It could be interesting also to have an update on Q. So we could we could ask them to give a quick update on May 1st. If you have other ideas, just just propose them. And with this, I will stop sharing. And I will pass the word to Diego. We'll talk to us about interlink. Thank you very much, Diego, again, for presenting and go ahead. Thank you for hearing me. So yeah, okay, the idea with this presentation is to propose a bit of a summary of what can be achieved with interlink and why we are evaluating and developing such a tool that I'm presenting just in a few minutes. Just one disclaimer here. I'm being focusing on HPC, how to deploy on HPC centers, let's say, but I have some detail at the end where this can be extended also to any remote resource. So this is the reason for this title referring to remote resources. But okay, let's dive in for the moment. Starting from where we in this development and investigation, you can see I have to thank a bunch of people first. And also notice that it's not really just a single institute working on this. It's kind of a teamwork made possible through the project that I'm presenting very briefly here in the first slide. So starting from where I'm employed, actually, I'm a technologist at INFN and ICSC, meaning that there are two kind of domain of science that we need to serve. The first one is INFN, that is the National Institute for Nuclear Physics in Italy. And you can see that we have different kind of area inside this domain. But every single one of them need to access resources of some kind. These resources are not only distributed all over Italy, as you can see in this small plot at the bottom, but they are also different in how they are managed. So there are cloud resources, there are on-prem resources, there are also batch resources that are not Solorum, for example. So they are HTCondor and used for more classical, let's say, distributed computing at LHC, for example. And also we have a supercomputer in Italy, and we will have more kind of HPC centers. And this is pushing the challenge to the next level. So how we can manage to have a common interface for people developing their stuff and doing science, and not only physics, but science, meaning we have here for the ICSC, that is the National Center of Supercomputing, a lot of different chemical, geophysics, quantum computing, whatever, needs to share the common infrastructure. So we need kind of an abstraction that allows us to say, okay, this is our entry point, and all the other magic appears, it's transparent to you. You don't need to care about how your payload is distributed over this infrastructure. Yeah, this is the beginning of the challenge, and what really put a boost in this direction was the participation to the inter-team project, that is an European project that aimed to create a digital twin engine. Digital twin engine meaning a platform that's as to serve as a distributed computational hub for creating and developing digital twins. So in different areas, that's starting from physics, for example, replicating a twin, a digital twin of a detector, for example, but also to replicate model for fluid disaster or whatever kind and so on and so forth. So multi-disciplinary area and that needs to access HPC centers that are also, fortunately, part of this consortium, part of this activity. And this is really the key for what we achieved so far. All these infrastructures that I'm mentioning, so starting from the grid one, so the batch on HD Condor, but also passing to this supercomputer area, often enough, produce this kind of problem for us. So produce a restricted area where we cannot envision to deploy anything that is similar to a real Kubernetes node. And we need Kubernetes for a simple fact, as you may imagine. The frameworks that are being developed for the future of the each one of the science domain that I presented briefly before needs, it's compatible with Kubernetes APIs. So we really strive to have an abstraction that starts below the Kubernetes API layer. So the idea essentially is to serve this black box that does the following. So we started with these three main scenario in mind. So starting from the first one, we envision, for example, machine learning data science pipeline in general, arriving to the point where you interact with Kubernetes API. And then this black box should be able to afloat what you request from that framework into something running on the GPU node, for example, on the HPC center. And then return it back the results that you need. Same for serverless or event-driven computing, if you want. I know new data are available. I want to trigger a pipeline that goes into specialized hardware or into specialized node for a new training of my model, for example. And the most challenging one is also the one that for seeing a backward connectivity from the node. It's a bit tricky. We have approval concepts for this and I show you BAs. But the idea is that, again, I have a Jupyter hub, for example, or IML flow hub, whatever, that delegate the payload that in this case is a simple Jupyter hub instance, for example, and it distributes this into a remote node and connect back to the hub for the user to access them through the web interface. These are the three scenarios that we have in mind. And all in all, what Renga about in the past, where we were thinking about this, is that somehow we have this kind of abstraction already in place, thanks to the virtual cubelet components. And so the idea was, okay, let's give a first try to this kind of approach, where I have my Kubernetes cluster with a virtual node that behaves like a normal node, but is configured to submit, for example, in the case of Slurm, a Slurm job with a container run inside an obtainer or whatever container runtime is disposed of at the remote site. Very often we never experience with two, three centers, obtainer or X singularity, let's say, is the runtime that is most used in the HPC side. But okay, so I offload in this way the payload to the node with this kind of magic, how good is it? So we gave it a try to all this flow, where this HPC GPO node was actually a real euro HPC center, that is Vega, I'm talking about it in a second. And the idea, it was okay, let's give it a try to what is already in place, and it worked quite well. So we were able to do all the chain with some caveat and with some tweaks, allowing us to do the first, very first proof of concept saying, okay, everything is doable with some caveats. So the first blocking item for us was that really we have an heterogeneous set of providers and also when you pass from one HPC center to the other not every Slurm is equal to the other one. So you have somehow to configure and let the provider be in charge of managing how the offloading of the container works. So, but to do that with a full vitro-cubular implementation, they should be aware of Kubernetes internals. And this is a showstopper in many cases. So we ended up with three main requirements for our approach and for our work on top of vitro-cubulat. So creating an abstraction that allow for providers to easily extend the vitro-cubulat machinery, allowing also a central maintenance of all the Kubernetes related stuff. So we maintain all that inside a core component and we envision plugins that as just to describe how the container life cycle is managed. And it will be in charge of the providers. So with this in mind, with this structure in mind, we can summarize this as providing tools to streamline the process of having a vitro-cubulat. It's not that it's not possible to do it right now, is that our concern is about letting the providers do what they want in terms of managing container at their sites. So in a certain sense, we want to lower the barrier for new providers to enter and say, okay, you can use microservices through a Kubernetes API. And so to achieve this really from 10 kilometers far, we develop a vitro-cubulat, a particular provider, if you want to call it like that, of a vitro-cubulat that is split into components. The first one is the Kubernetes core, so where you maintain all the state machine for reports, logs, requests, delete, apply, and so on and so forth. And then this part will talk through a standard, or let's say, a fixed API spec to a remote counterpart. That is the one that will be called plug-in or sidecar with interchangeable terms in the next of the presentation. So yeah, this remote part should be in charge of managing the container lifecycle. All the Kubernetes part is separated from that. So that is the idea, but before diving into the components and what they does and take a look quickly to what we achieve, this is not a silver bullet, of course. We are interested for our use cases in very particular payloads that can avoid to rely on intra-cluster network communication. So for instance, you should not assume that you can access a Kubernetes service that is internal to the Kubernetes cluster on the remote payload. And also the only volumes that are supported are config map secrets and empty beer for your pods. You cannot share PVC from any other kind. That's the limitation and here of the schema this work. On the last side, there is the characters if you want to play the game and on the right side there is the technical implementation. So let's start from introducing the components. The first one is the fact to the receiver, right? So the one that says to Kubernetes, I am a node. Give me work to do. I'll take care of that. But at the same time, whenever you receive a request, it contacts its friends interlink API server that you can consider like the man in the middle because it's responsible to expose the standard APIs that I anticipated before. And then it contacts the plugin that has been installed by the provider and where all the logic for some meetings, loom jobs, for example, is concentrated. And that is really the discovery. So the one responsible to execute your payloads. And if you go on the right side of the slide, you see that essentially you have the Kubernetes API server and all the scheduler and whatever, all the components of the Kubernetes part are talking to this virtual node, just like any other virtual kiblet that is out there. The only thing that changes is that the virtual kiblet itself is speaking a particular language, if you want, that can be plugged to different kind of plugins. So you can see here the virtual kiblet authenticate through a token with the API server and then the plugin does the jobs. We have three implementations in the moment. As I mentioned before, with a local Docker component, you have a single machine you want to run that into a Docker. It's more for demo, but also for real cases where you have a big machine. We have Zlure in this box here, and that is the one we are going to look at. And then we have HTCondo for normal grid system. You have here the schema. I'll let this more for you as a reference to understand the flow. But again, I don't want to repeat myself. You start with a node that is designated to, with a pod that is this designated to a particular node, to a virtual node, and then from there it proceeds in a chain. So the virtual kiblet contacts an interlink and say, okay, I need to run this payload do whatever it takes. And depending on the plugin that is installed, the interlink part does these different things. Where did we test this implementation? So we decided to start from volunteer sites and we were lucky enough to have, as a first volunteer site, the Vega HPC Center. It is one of the biggest Euro HPC Center. And we were able to play with quite a success at that stage. So it was really the fact that they enabled us to convince other sites, other people that this could actually work. And related to the discussion of the last time, the last meeting, for us, this model worked very well. We have a champion, if you want, where we demonstrate things that works. And then, for example, Euro HPC Center that is also participating to intertwin. Come and say, okay, it's quite interesting. We have our way to manage container on our center. Can we plug our work onto this? So we can plug, for example, Unicore, that is their framework, into a plugin serving as a backend for interlink. And this was the second step, second fundamental step, that demonstrated not only that the workflow can work for us, but also that the principle of easily extensible is a valid argument in this case. And yeah, so now, unless there are questions, the plan is really going through demo is a big word. I mean, showing directly what it looks like, some meeting pods on this kind of infrastructure, and the notebook part that I mentioned before is working. So yeah, I don't know if anyone has any pressing question that they would like to raise. I'm afraid to come forward. I have just a quick question. What kind of, what provider are you using from Virtual Cubelet? Yeah, the interlink is, if you want the provider, the provider, so the implementation of the Virtual Cubelet is done by us, and it leverages the interfaces provided by the Virtual Cubelet software. So was this, sorry, I saw KSOC also, was that kind of an inspiration for the interlink provider or? Yeah, sure, yeah. We took that from the architecture perspective, and then we also, of course, look at around how people manage Virtual Cubelet and yeah, but the class itself is fully Virtual Cubelet-based. It's nothing fancy, let's say. All right, thank you. All right, so let's start from what it looks like for the user in case of interactive analysis. Well, I'm sure most of you knows the interface for GPTRAP. What we do here is selecting the image that is actually in the Vega URH PC center, and the user can select, okay, give me a slot with CPU or GPU and in Vega, and here I'm into the login node, into the edge node where we have interlink configured, and you can see that I have nothing running here at the moment, but as soon as I'm going to create a pod that is pointing to that particular node with this flag, you should be able to see, yeah, now it started, you should be able to see that a job, a slurm job is arriving, and it will actually do a singularity exec. Let me show you. Yeah, now it takes too much time. Let's see the job running, the notebook is up and running again, so I'm inside a node in Vega with a job on slurm running a Jupyter lab, and I have NVIDIA, as am I, and I have a GPU available to do all my development stuff. What looks like into, if I go into the node, into the Kubernetes cluster, it's actually, oops, it's actually the following, are you still with me? Okay, yes, it's not my connection, fortunately, so I can take a look at this and say get pod jup, you see here that my notebook is essentially a normal pod that's a solid one constraint, actually, and the constraint is that I'm asking for node selector, so I'm asking to go directly into the Vega node, so you have to be explicit, of course, but if you select the node that is called Vega-NewViewToQueueBlood, you will be spawned into a slurm queue with a singularity container running on top of it, and this works in this case with a trick, the job is connecting back through a tunered connection, but in case your payload is not that complicated, you have, for instance, the possibility to submit any payload that is going to download that data, collect information, train on that, and bring back your results, so I started with the worst case to show you the whole chain, but again, you can be as fancy as you want unless you require pbc that are not supported, you can bring with you config map, secrets, and empty-dir, but the connection to the internal or the remote Kubernetes cluster is not allowed from the inside of the HPC, you can think of VPN or whatever, but it's not the target of this at the moment, at least, so I don't know if you want to lead what comes next, otherwise I'm just pointing you to the, I'm just showing you the documentation, and oops, so in documentation we have two scenarios that you can try on your own, the one is that is local, you have both Kubernetes cluster and a Docker compose on your laptop, and you can check how the components are talking to each other, and the other one is actually what we use to deploy all the machinery, so we use, you have a step-by-step demo where, for example, for the authentication between the Vista kubelet and the remote counterpart, we use the OAuth chain, but in particular in this case is GitHub, but we have also instruction for everything that is well too compatible, and then you install through the client and through some configuration, both the components, the one that goes into the Kubernetes cluster and the one that is responsible to run Slurm job, for example, remotely, and so yeah, it takes literally five minutes to go through this, but the nice thing that I want to stress is that we also, since the specifications are documented through Open API, you can implement whatever plugin you want with quite some ease, and we also have example for this, if you want to extend, and we have people coming and say, okay, I want to extend and they get started pretty easily with this kind of approach, and yeah, just keep in mind that we are at the early stage of this project, so the API, it's something that we want to fix quite soon, but we are not there, so at the same time, there is a good time if you want to come on board to lead what the API will look like in the future, but if you want to rely for production ready stuff, it is not at this stage. All right, there are questions, otherwise I go back to the presentation. I see Pablo has a raise hand, go ahead Pablo. Yeah, thanks. First of all, thanks for the very interesting presentation. I had two questions actually, so the first one is, I'm not sure I understood, you mentioned the limitations, is that you only get like a config map, empty there, etc. So what are your, what is the approach here, if you spawn a notebook that needs access to some, you know, data that is somewhere else, and then I have a question that I can maybe ask. Yeah, the first one is, there was partially a lie inside that, because you can still access common and scratch area, not scratch, common area that you can decide to mount at singularity time, so for example you have your scratch area or your data area inside HPC, you can bring data there through whatever machinery, and then you can still access that via a pod annotation, you can say, okay, mount this area because I'm allowed to. Okay, because you're inside the HPC and you can just mount it, okay, I see. But for instance, at PVCs, is that a design limitation or is it just the fact that it's not implemented? Well, on one side it's linked to the fact that we have no internal access to the network of the cluster. Okay, yeah, I see. That is limiting a lot what we can do, what we can do there, and I suppose it will be many other issues that we need to take care in order to allow for that. Okay, thanks. The second question might be a bit brief, hopefully. I was just wondering, so I'm not an expert on the internals of the notebook, but it was quite interesting that you showed an interactive session. I was just wondering how the connection is established since obviously you don't know which node it's going to end up running at, which worker node, so I was wondering how it works behind the scenes, how you can establish that interactive connection. Yeah, with again, vision from a bit far away, we are instrumenting the Jupyter Lab image with an initial script where you connect through with your Jupyter Hub token back in a SSH session with the Jupyter Hub, so you open a tunnel basically to the Jupyter Hub and then you start communicating with the proxy on the cluster, the Jupyter Hub proxy on the cluster through this tunnel. So yeah, not every image can work on that, but it's really a script that you have to put in an initialization of the docker to adapt any Jupyter Lab image to work in this way. Okay, thank you very much. Thank you. Awesome, I have see Rohini, go ahead. Yeah, thanks, Diego. That was really a good presentation, an overview. I had a question. Yeah, no problem. Sorry, just to clarify, I have just a couple of slides to conclude, but yeah, go ahead. I had a question about the O-off chain in the sense that is it something that's implied here that the IDP that you've configured your Jupyter Hub with, there is an implicit level of trust with the HPC facility that the JHub tokens will be trusted or is that established? No, the only trust that the only trust that comes from this implementation is at the moment you can have finer grain, for example, in the Julia implementation we have the grain of authorization that is on the pod level, so the pod has to present an annotation with a user access token to be trusted and to be executed on the slur node, but generally speaking, the general implementation relies on the fact that the Kubernetes cluster that is talking to the interlink API is trusted by the HPC, so the HPC trusts the interlink server as a group account if you want to to put it in this way, so the administrator of the Edge machine trusts whatever comes from the Kubernetes cluster, so there is trust between the cluster administrator and the provider administrator. If this is not enough, again you can ask at the plugin level for whatever additional information you need to allow the user to submit to the queue, for example, if you want to allow only certain people, you can plug your mechanism for say, okay, if this pod arrives with valid credentials, it's allowed to run. Okay, yeah, that makes sense, and I guess that's true independent of what the entry point is, so here in the demo the entry point was jupiter hub for the pods, but even if it is coming from some other data processing way, the fact that the trust is between the interlink server and the HPC means that, okay, yeah, okay, thanks. Awesome, Dennis, do you want to go next? We'll give you the chance to finish after Diego. Yeah, yeah, no problem. All right, thank you, Diego, for the great presentation. Just one quick question, this is maybe a little bit use case specific, but do you support having multiple containers in a single pod when deployed on the HPC infrastructure, and what will interlink do with networking and shared memory in that case, because we have a use case where we would really benefit from deploying essentially two separate components at the same time in a single pod, sharing resources? Yeah, so in that case, again, it's not really interlink internal because it was the information of multi-container pods is transformed into into what is managed at the plugin level. So currently there's Lurem plugin that we use at Vega, for example, that's the following. Allocate a Lurem job with the sum of the requests for the container and then kickstart two singularity processes, one for each container in the single job. This grants you at least to land on the same note and do some stuff. It's not the cleanest possible way that you want to work with. Again, if you're interested in doing better or less some use cases, you're more than welcome to join the efforts. All right, thank you. I think you can go ahead, Diego, and wrap it up. All right, yeah, so I wanted to share also that we were a bit worried about the scale, as always, but it ended up in being a nice experience here all in all. You can see here a plot where there are 1,000 pods deployed. It's not enough, but it's not too less, I suppose, resulting in 10k cores consumed by, in this case, we leverage some example workflows from CMS collaboration. All of them, all the 1,000 pods, were managed by a Kubernetes cluster that is really a common one, if a common one exists at all. But yeah, there were no dedicated hardware or special stuff that we did to reach super high performance from the control plane side, let's say. That was a good news for us. Also, this is to mention that we are developing and we developed this tool also with this in mind. We are trying our best not to create a lot of requests to the, for example, plugins, so in the case of Slur, a lot of SQ command, but rather trying to catch whatever possible and be on the lazier part more than on the aggressive part in terms of synchronizing the statuses. So this will allow, for sure, to scale as an important role on this achievement, if you want. So all in all, we have this deal in mind. We keep and provide maintenance as a community for the core part of the virtual kubelets, so communication, authentication, authorization model, and the API that goes through the communication, the remote part to the entry part of the edge node, the caching of the state and whatever. And then we hope for interesting party to give a stab to providing the implementation that they want to in their use cases. And yeah, talking which we have experience also outside HPCs that are interested in this, so for instance your services for which will be out for you to give kubernetes access to them, but with this trick you can create your plugin and say, look, with this you can make a call to a kubernetes cluster, to a kubernetes counterplane, and that your things deployed on my SAS, for example. So that's one model that is taking a bit of attraction, and that is bringing us to thinking, okay, how we wish it would evolve from here, if other parties are interested in maintaining and collaborating. And also a paper is in the writing and for reference the sign and the implementation and very first experience that we have so far. Yeah, I think we touched most of this point, let me stress just again, we are not using all these lume for implementation, we have a lot of other use cases that are running on the heterogeneous set of resources that we have, INFN and in Italy in general, we are also working on that level. If you are interested in making questions or whatever, you can join the slide channel for that we have for the community and for the project inside Intertween. And last but not least, the link to the documentation is here, but it's also in the description of the event so you can go whatever route you prefer. And yeah, that's all. Thank you very much, David. That was awesome. I was going to ask you if you can share also in the agenda of the group the links both to the presentation and to the to the documentation. I think that would be very useful for people checking it later. We have time for more questions, so I see Kevin. Yeah, it was just an interesting project. I was curious if you all have considered adding this as a provider to the VirtuaCubelet website to kind of bring more awareness to how you can use VirtuaCubelet? Yeah, that's a good point. We haven't reached yet the level which, okay, we knocked at the door and say we want to be published there, but yeah, I think it's worth appearing there to be to be aware of this possibility. Yeah, I mean, if you look at a lot of the providers, part of it is I think a health of the project, because when I look at VirtuaCubelet, a lot of those providers are pretty old, so it makes it seem like that project is not really continuing. So I think like at least having it showing in the open source community that you are using this kind of shows that it's still actively, people are still actively using it. Yeah, good point. Even if like it's, you know, you quote-unquote think it's rough. I was actually thinking about this a year ago, and I wasn't aware of this project at the time, so it's just like having that information, because when I looked at VirtuaCubelet, I saw there was a CRI provider, and I was like oh, if I want slurm and CRI, I'll have to implement my own, and I stopped. No. And it's a similar story I heard also from other people, so yeah, okay. Yeah, I think that's a good point. That's a good way for people to find the project. Thank you. Other questions? All right, well, people think I'll ask one. What are the resource requirements on the interlink cluster in the origin cluster? Is it just like the HCD resources, or is there anything else? You mean from the control plane perspective? Yeah, I can say that for the scale test again, it's not a lot, but it's something. Really, we had ATCD running on not even, I think, not even SSSD, let's say, so on whatever kind, really normal control plane reached this kind of performances, because again, it's lighter for what we with Sozofa is from the intensity of the work is really less intensive than having real nodes and real pod. But yeah, we need to measure this, but so far, if you want to reach at least 1000 pod, you don't need any particular requirements. This is assuming, I didn't get it completely, if the interlink metadata on the virtual kublet node is that stored in HCD, or do you have your own storage? So the state, all the information are stored inside the pod resources, so eventually are in the ATCD of the kubernetes cluster, so it's nothing dedicated. What you have inside instead is the cache that is done on the remote part, so the interlink part does inside a cache of the status is allowing for not going each time the Kubernetes cluster requests a status to go ask the SLURM. If it's running for a certain period of time, I cache that information and then eventually I update the cache later. So just to avoid bombarding the SLURM services with too many requests. Okay, thank you. And I had one more, I don't see any other questions. So there was a question already about the network requirements, but I had another one which is at the scheduling time, because I understand this is at the pod level. So for example, if you want to do some sort of, say, distributed training for ML and you want to collocate pods, this would mean, like, collocation is just basically putting the same node name in the other annotation, which is the virtual kublet, and then you just assume you're running on the same HPC farm, and somehow you'll have, like, fast interconnects, all these things. Is this correct? Yeah, there is something that is interesting, but we didn't have practical use case to test with, that is the MPI configuration. So in principle, you can pass to the annotation to the pods whatever option you can pass manually from the HPC login node to the slur submission command. So in principle, you should be able to configure your job, your different pods to leverage MPI of the sites of all the machinery for that kind, but we haven't yet have a use case that is willing to test that. So it's just a wide guess. All right, and I'll ask the last one and then I'll pass to the others, but if I would use a library for, say, machine learning or anything else that does distributed pods and training, how does this translate to, have you tried this, like, for example, doing TensorFlow, distributed and then seeing what happens? Yeah, with the other try the distributed part with TensorFlow, for instance, different thing is if you want to have, if you have some service that you want to set up, for instance, I'm thinking of MF flow tracking server, you can expose that and in case of Vega, you can contact that for tracking your machine learning performances, but yeah, doing actually the real multi node, if you want, training is not something that we test, but if you are willing to, with some use cases, that would be good. But for example, the virtual kubelet, how many resources does it advertise to the scheduler? Is it like unlimited and you assume there's unlimited quota on the back? So the node itself can be either unlimited if you want to allow for any kind of slur submission to the virtual node, but otherwise you can instantiate a flat static resources when you install the virtual kubelet. So you can make that node appear like allowing for 10k pods and for 20 terabytes of RAM and so on and so forth. So typically you would have like one virtual kubelet per queue to have control like this? Yeah, it's up to the use case and the topology you want to achieve, but yes, if you want the easiest case is, okay, I'll let slur limit my usage so the virtual node is unlimited. Okay, yep, like this. Other questions? I'm watching for raised hands. Follow up on that one. So go ahead. If you are submitting a pod and you hit slurm, I guess would the pod show up as just running at the kubernetes level and then in slurm it would actually be in queue? No, no, no. The status is one of the things that is managed centrally by the virtual kubelet and it relies on the status retrieved from slurm. So until the pod is not running, yeah, the container is not running inside the slurm job, the pod status remains pending. If for any reason the submission failed, you have provider failed or something like that and yeah, but you do not see running if the job is not running. Okay, thank you. Very, very nice. So we are getting to four minutes from the end, so we have time for one more question if someone has one. Otherwise, thank you so much, Diego. This was really good. I suggest if people have additional questions so we can also follow it up in the Slack channel for the research user group. I know Diego is also there, so feel free to to ask questions there, follow up on the discussion. So I don't know if anyone has any other thing to raise. Otherwise, our next schedule meeting is currently May the second one in May, so the first one we don't have a speaker yet, but it's very likely that we'll find one. So yeah, see you in two weeks and thanks again for participating. Bye-bye. Thank you very much. Bye-bye. Thank you.