 Yeah. But yeah, were there any talks about, you know, the batch HPS also in the normal, like in the rest of the days or was it only in the co-locate? Because we might have, yeah, we might have went to those. There was quite a few that we attended, given, yeah, our interest. But I wasn't sure if you listed those because I kind of was trying to look into my schedule and see what was, what did I pinned exactly? But I can't find my schedule. Right. So yeah, I know that during the event there was quite a few, like a few related to GPO, like the guy from Google was saying some stuff to optimize Google. I think there was another from Vulkan or above Vulkan. Don't buy, I can't remember the vendor now. There was few others, but like me, for example, I focus more on Kubernetes itself, not necessarily only on the batch. So one of my observation and main team in KubeCon for the remaining days was EBPF. So as a way of improving one performance, getting more observability and more control on your network layer in Kubernetes. So there was a, we spoke to like few vendors on the booth, like Calico and Celium, both very interesting. That's something we're going to probably try soon or soonish in GR as well to play a bit more with EBPF. Other interesting, quite a bit around automation and practices, like people, you can see clearly that like ArgoCD is getting more mature. So I went to a talk where the, how it's called, let's say vendors, but the team responsible for owning ArgoCD or talking how to extend it, what they plan, the rope up and stuff like that. That was quite cool. Yeah, I actually mentioned the batch in HPC Day, but there was also GitOpsCon as a co-located before the conference, and then a ton of talks about GitOps as well during the conference, both for Argo and Flux. Yeah, but maybe we can, like if you find the links in the sketch, maybe drop them in the chat. I just keep recording so we should be able to then find the chat and collect them again. Again, it was as well like, it was my first KubeCon and I was surprised by a number of people attending. There was like 7,000 or so people on the site, and sometimes it was quite challenging to get into the room if you're not early enough into the room, especially if the particular talk was like super interesting and getting a lot of traction. It was like, if you are not 15 minutes earlier in the room, you're not getting in. Luckily, there was a way of watching them virtually, so that was quite cool. There is one talk I haven't watched even yet where I think it was Mercedes-Benz explaining how they migrate 700 or 7,000 cluster from Terraform and other infrastructure to using cluster API. So that's quite a few lessons there. And I think that was another observation, like Kubernetes become more and more a framework to doing stuff like that. There is quite a few talks around cross-plane, so cross-plane as a way of managing infrastructure other than just pure Kubernetes stuff using Kubernetes and Kubernetes reconciliation loop to enforce the state and stuff like that. That was quite cool as well. Few post-mortems as well, so few talks about post-mortems, so if you're running Kubernetes for a scale for people, so that was as well quite interesting seeing how I think it was Datadoc, so Datadoc running into the issue and spending like few months investigating stuff on their Amazon infrastructure. Yeah, is that the DNS one? Yes, that was a good talk. And I'm pasting the links here in the chat as you speak. Cool. Yeah, we have some joining us. A small screen. Yeah, three of us at the same time. Cool. Yeah, so there was as well quite a lot of talks like casual talks happening during the lunch to various people and then again to vendors. So one of the vendor we're kind of using is the one behind OPA. I don't know whether any of you is using OPA, OPA is a way of defining a policy. So before going to KubeCon we faced a few issues where we tried to restrict some stuff, so we started discussing with them and so on. So that was again quite a nice thing, side thing in KubeCon. Nice. All right. Anything else? Okay, so I tried to collect a few talks and put it in the chat, but feel free to add if I missed some. Yeah, I think I actually been to the one, the improving GPU utilizations. Yes, I was there, I think, was it from Google? Yes, that was quite interesting. But it kind of was a let down that they didn't actually talk about the implementation details, but it was quite interesting about how the theoretical side of things. We also try and grab something else out because I need to find my SCAT from Kenya. There was one, I feel one of the attendants was, the one that I attended that was quite interesting was network hours scheduling, although it did look like something that would require quite a lot of time before it can be implemented on production cluster. Let me see if I can find it. It's a good start. Oh, yeah, we just, so this one was quite interesting. But you're just not listening to them. Yeah, we're talking to them. Yeah, the another interesting one was fmr containers. So I'm not sure whether you are aware about fmr containers. That's a new thing in KubeCon at is 123 and I think it's going to be even more mature in the future. The idea is when you need to troubleshoot a problem with your port or with your application inside the container, rather than using kubectl exec, which many people do, you'd run the command kubectl debug. The difference is exec move you kind of like almost as a site into the container, your troubleshooting. But with debug, it's been additional site container. So we can define that you have completely different set of tools in that site containers. And by being a site container, they share the same namespace, Linux namespaces, not Kubernetes namespaces. So by doing that, you have the access to the same network namespace and so on. And you can use ns enter, so namespace enter to even get access to pit namespace and a few other Linux namespaces. And that was quite cool because it allows you to have a very minimalistic image. You can use probably distro less and start building more tools and buy that into site containers. And the site container once you finish executing your kubectl debug command is gone. So you don't need to worry about having the site card constantly running. So we actually do use that term. It became beta in 123, but it was alpha before, so you couldn't enable it in the clusters. And we were enabling it. And what we do is we keep one image that has all the debugging tools we need for networking, everything, file systems, whatever. And we use that image to debug and attach using informal containers when we debug stuff. I put the links there as well. Through my sketch. Oh, another one interesting, kind of interesting, depending on what you do is about kubectl. So kubectl, again, you use Kubernetes to manage your VMs simply. But they added quite a few additional features like live migration and so on. So that's another product which is becoming more and more mature. And in the future, probably it could be a way to, rather than directly using, I don't know, open stack to spin your VM, you can use kubectl to control and do VM migration and other stuff. Yeah. So kubectl, that is a framework again. Yeah, there was a few talks as well about horizontal port autoscaler using KEDA as well for that. Yeah, the talk wasn't bad. I think there was a project called Pixi, or I think Pixi. Pixi had to capture the traffic, right? The being able to actually reproduce it. Yeah, that's a good point. There was an interesting talk about, you can remember the title, I will find, about different approach to doing a load test or in general a test, because let's say you run a web application on your Kubernetes. It's a web service. So you, to deploy a new version of that web service, you can do a few approaches. One you have, let's say staging cluster or something like that. You deploy there and maybe test somehow, like write some integration tests. Another one is having like canary approach. So like blue-green, where you redirect a bit of traffic to your new version. That's what people quite often do with services mesh like Linkerd and stuff like that. But the talk was that the problem with that approach is you're not necessarily has consistent inputs going to that web service. Let's say if you're doing the change in middle of night, you don't have the same traffic or the same number of users. So you don't really know that whether the way you're promoting your new application is working at all or not. So the idea is, I think it's still using eBPF as well to capture the traffic. So in other eBPF related is to kind of tap, like start tapping the traffic and record that traffic. Once you have the traffic kind of recorded, you can reapply that later on anytime and it's going to have the same volume and so on. So you can deploy your web service in without impacting a user. You run recorded traffic and if the behavior is not different than before you're changed, then probably you are good and you can try A, B release on that point or canary release. That was quite a good talk. I will try to find out the name of the talk. I just pasted it there. It should be the last link in that. Yeah, the one, the reproducing issue in your CI pipeline. That's the one. Yeah, I still have the schedule in my head somehow. I didn't watch them but I remember the title. Yeah, it was very good to be honest. Yeah, it was a, according to team of eBPF this year. It's quite interesting. I remember hearing about eBPF quite a few times in the past but this one really was, yeah, you can see already that the tooling is getting more mature or at least more known around the community. In fact, there was other talk. I think this is also quite relevant to the scheduling part which is about bandwidth management using eBPF again. Let me just paste a link here. Yeah, it's definitely the the the conference of the eBPF. Yeah, I think it was something that we could see coming already for a while now. But yeah, this one was definitely the confirmation that's going to get bigger from now on. In fact, the one from bad management was quite interesting because it was starting to, yeah, they were adding this physical possibility of adding into your basically deployment. The also the resources around how much bandwidth you want to allocate to a specific well pod and that's quite interesting as that it's only possible through the way eBPF works and how you can get those kind of informations out of the kernel. Yeah, I mean, again, very cool. I'll paste the link there. Right now I do have, oh yeah, there was, sorry, I was looking at the information here. There was something that we're talking about where they were also mentioning how with this new approach you could also have higher speed of communications. Let me just look at this. The scalability limits of token bucket filter by the bandwidth plugin, at least departure time. Yeah, combined with eBPF. Yeah, something about being quite cool, both in bandwidth management and getting more speed out of what's available. So what looking at with eBPF is with Cillium to do sort of like cluster mesh or not only like a service mesh but really multiple clusters to be meshed together at the pod level even and you can easily do load balancing across clusters without having to rely on services, which for the batch use case is actually quite interesting because we don't want to have the like we don't really care about the service abstraction. We just care about the workloads and that this is something we started prototyping which is to mesh multiple clusters and be able to schedule across them from a single plane basically. So that's interesting because when I chatted with them about scheduling across multiple clusters I thought that their response was oh no this is really only meshing the networks together so that pods can speak to other pods in other networks in other clusters but the scheduling of them you'll still need to do somewhere else. Right. Okay. Yeah, but it allows you to like even if you have if you want to distribute the workloads across clusters you can rely on having like some services running internally in one cluster without having to replicate them everywhere for example and you could have this workload clusters that are really disposable while you have the service clusters in the same mesh or the service component clusters in the same mesh. So we've been playing with this also but actually with some tricks you can actually schedule across clusters as well. This is something we've been playing with but maybe for another time. Yeah I'd love to hear that and also whether you're seeing reasonable performance for you know often you want jobs to be co-located so that the network speed is fast enough for them to talk to each other but yeah anyway that would be interesting to talk about maybe next time. The big thing we've seen is that you still need like node to node connectivity like layer 3 connectivity between all nodes across all clusters which yeah it kind of makes sense but it's not like a gateway or anything you need like full mesh between nodes as well. I think something like that was mentioned in the data doc talk about DNS where they try that they are using the celium exactly to do a pod to pod routing possible across multiple cluster. I'm not sure whether across multiple different cloud providers maybe they achieve even that. But there you need some sort of like VPN connectivity I guess because you need to expose all nodes to all nodes so this is this is our dream which is to burst using a mesh like this but it's actually trickier than it could be I guess. If you look at other things for service connectivity they use gateways here it's really like a full mesh between all nodes at least my understanding up to now but it is promising sounds amazing but it's actually something maybe maybe we should bring them to present celium and eBPF to the group. That would be cool yeah. We're getting Liz to come to CERN in two weeks so maybe she can also do a talk the same talk at group. Let's put it for the list here. They're definitely bringing a lot of more interested parties into this research group. You mentioned the gateway I think the another talk was about gateway API so it is going from beta to GA. It was quite good talk as well. I can take the link out actually because I'm having my schedule I think. I'll put here for I'll put it for the next one the topic for celium and eBPF but maybe we could then see. I'm just placing the gateway API link as well. It's nice yeah. The other stuff I had here in the summary is just I saw that references to batch workloads not only in the talks but also in the keynotes so in the TOC update it was mentioned that there was the new group formed as part of the tag runtime and then also in the Kubernetes updates the batch working group in six scheduling and then like the keynotes also from CERN we mentioned the use cases and there were other mentions. You mean the one that you gave? Yeah. Yeah so bonus you're like oh the one from CERN I don't I don't know who those people are. No yeah yeah yeah so no but definitely it wasn't a coincidence like I thought. Yeah yeah yeah and also in the other ones like this has been appearing a bit everywhere but I think there was it was clear that from the references constantly in different keynotes slowly building momentum and when we see the other activities as well and then yeah so and then there was one session dedicated to the Kubernetes working group batch that will also be the video uploaded so Aldo gave an overview of the work that has been going on already and and the plans and there was not a lot of different people speaking but there were I talked to a few and it seemed like there were both developers and also end users interested in using these tools so that that was quite nice and just really really quickly so they summarize the motivation I think we all know about it here but they also mentioned that their goal is to it's three main tasks one is to update the job API to allow new types of workloads that are not just the typical batch job as defined by Kubernetes up to now then things like queuing and advanced scheduling and then I think the interesting part that there was a nice talk in the co-located event about was the optimized scheduling on the node itself to make sure that the nomad apologies are set properly so that you get full performance and you're not losing like 20 or 30 percent of your capacity because of it so I think it was a nice I think Alex you were there as well right yep I was there very jet lagged but yes I was there it was good I would just reiterate the the number the amount of batch scheduling related talks that the batch day and Aldo talk and then I was on the panel a day later and then you spoke in the keynote and it was we weren't quite at ebpf status but batch was was rising in the ranks of conversation it's good no and I'll pitch one more talk which was from some other CERN colleagues and they they gave a talk later I think Thursday I don't think the video is uploaded yet but basically what they've done like we have this large grid computing environment and they've been playing with getting Kubernetes being a grid site and it doesn't matter if it's on-premise on a public cloud whatever that presentation where they showed that they could scale a single Kubernetes cluster to a hundred thousand cores in the Google Cloud in this case quite easily and fast and then even scratch it when they don't need it yet and they justified that this is like an out-of-the-box solution to integrate new resources into our grid infrastructure and also the ability to to request resources that we don't have is our GPUs and their dream is to have like a home chart that does help install grid site and and you just add it to the infrastructure so they gave they gave some some summaries here of their what they've been doing integrating heterogeneous like ARM and GPUs and then they actually built an analysis facility on top of this so they have the Kubernetes layer as kind of the base layer to add the resources but then they add like Jupyter Hub and they add the ability to deploy like task clusters dynamically for different users so people can do their analysis using Jupyter Hub and then scale out using tasks to a very large number of resources so I don't think the video is uploaded yet but for sure it will be Nathan so I from from the link in I will find the link in the agenda for you and then there should be like a little with the video some reason my computer is blocking a bit oh Apple's yeah yeah yeah yeah Apple's the link in a bit so I think it's it's an interesting talk because it's a real use case and pretty large of doing both batch and and kind of more interactive analysis is is Panda batch processing yet another batch scheduler thing that people have written no Panda is like it's a specific scheduler for Atlas so they they have their own workflow manager on top and that's where all the work so Panda is their thing yeah yeah I just wonder whether they is it an open source thing or it is yeah yeah because yeah I can't find anything I can't find it. Panda batch processing does not yield a usable Google address say that again Panda batch processing does not yield a usable Google address so far yeah so I'll give you I'll give you one where where the actual documentation is so it is a generic tool but it's very much it's used by other experiments as well but it was developed within Atlas I pasted the link there cool thanks I wanted to find the link to the talk let's see here's the link to the to this one and yeah the video should appear there I think they they are done with all the collocated events and they started uploading the main conference videos as well there's some sort of delay where videos are available like if you have the virtual access you can go to the virtual platform and watch the videos right now otherwise they will get to YouTube at some point as well and I think that's it that's all I have yeah so yeah I think that the expectations were met for for the conference they were and they they mentioned that 65 new attendees in a coupon before so that's pretty impressive that's pretty awesome I was really disappointed it's a miss out actually but um definitely gonna be there and uh well gonna try to be there in Detroit should be good I really felt like it was three years worth of budget all spent into one coupon because of the pandemic I mean quite quite a lot of things going on I had to say it was a good one to be here to attend to tend to yeah and definitely you can see the difference of having in-person conferences and being able to like just bump into people and discuss quickly it's very different yeah definitely I remember I attended the virtual one the previous year but then yeah you could see you could definitely feel there was like yeah it just felt so less so to speak um this one I think I really enjoyed the part the the part that was not in the virtual one last year which was the sponsorship booth basically you could just go around and and find people and just talk to them which was something of course it was difficult you can't do virtually not in the way you can do it in in person and it was quite massive because there was like two pavilions full of sponsor showcases it was in between was killing me to be honest three days after I was I just couldn't move anymore I don't know how he had the strength to also cycle in the weekend but then yeah me I was just barely hanging yeah so I think that's that's what I had but one thing that I wanted to ask as well because there's not a lot of time between now and October basically so if we organize like a new batch and HPC co-located I think it would be nice because it would help keep the momentum but we need to be really proactive to reaching out to people to do submissions to make sure we have enough content there were a couple of talks that were quite good that we didn't select for this one but maybe we need to make sure we advertise this as much as possible both in the like new world but also in like there are some interests like Nathan is here there was some interest in like involving more things like more established components like slurm in the HPC environment and and try to kind of to the bridge between the two and see what's the way forward I'm sure Nathan will have a lot of opinions about this so that would be pretty awesome also to have that discussion I think his microphone never works though I think Ricardo A reached out and suggested that we submit something around Armada we'd be happy to do something of course I also wondered in that batch day do you know how pretty bass ended up on batch day I can't seem like it was it was a weird one to include especially if we had other good which one sorry it was a whole talk on pretty bass during batch day which seemed I like pretty bass it was Travis Adair and you know the people who did Horovod and Lidwig AI and it was more ML ML yeah so I think it was more to get a yeah I will have to go back to the notes but I think it was because they had like this idea of a note less Kubernetes that is quite interesting and also because they had different use case with that was the reasoning yeah I think because if you look at the schedule there's there's quite a lot of components or it's not really vendor but like component based talks but not so much end user talks and I think for the next one it would be really interesting to have those but we need to reach out to I see yeah but I mean if we could get end users of any of the schedules that spoke last time that might be that might be very interesting yeah and the other question will be depending on how many submissions there are if we make it still half day or a full day mm-hmm I'm going to submit something definitely I know that deadlines approaching but let's start it at least yeah hope you're amazing your sandbox or you submitted to sandbox as well yeah yeah that's one another thing we probably need to do definitely need to do is work out the next set of agendas for this I think we've run out now tends to work quite well I think send them up front yeah yeah we could use like five minutes for that yeah I don't know if we'll be able to do it now but we could think about it but yeah we've got as you say it's only really four months until next Q-coin anyway the CFP is this Friday you know right I know I know I'll start it I'm actually I'm traveling to some relatives tonight and I plan to just sit up late and just type up my submission because for the batch HPC if we organize again the colo it will be late at the CFP but for the main event it was already this one I'm just gonna get it in it's good I think yeah I guess like if everyone has we can probably drop the topic backlog we have there apart from the gem session I guess and maybe maybe if people can add there what they would like to hear about so we just talked about celium and evpf we had other other things mentioned there right so we could get the atlas people also to present because it's like a use case would that be okay as well yeah absolutely we mentioned the gateway avi did we ever get a presentation on that probably not right I've missed a couple of sessions but I don't remember one is that is that so germane to this group I mean it's it's interesting it's good I don't know do people suffer that in in this world of research I think the most the main thing would be how to use that for bursting and multi cloud things if if at all possible interesting okay I can see that a little bit it just seems like it's so much more directly useful for if I have a product and I need different different htp end points to go to different places okay yeah but they have this concept because there's the gateway api and then there's this multi cluster service api that basically allows you to make a service external and then define on the other cluster that it should consume a remote service and then they kind of linked to each other things like okay what else did we have here cluster api was that one as well yeah yeah we were very interested in that I think we never had that talk about it as well yeah crossplane we did get something about numa as well that would be pretty cool maybe yeah probably not for a whole session but I noticed the other day that proposal finally got merged the ones about six years old around username spacing you see that I did not see that in cover names yeah yeah that and it's coming up in the next uh community's release right oh no I mean it hasn't nothing's been implemented I think this is just like getting the like proposal merged like oh no I thought there was intent to get it in soon oh maybe I don't know but yeah that was a at least a big step forward having it looks like an agreement on to do it on doing it yeah no that's a super exciting one um maybe the excitement in my voice is not quite uh relaying that excitement but yes super exciting but where did you see that actually because I'm um hold on I've got it here someone sent it to me yeah half alpha release target 125 okay I see no I think that's the one that's it yeah yeah I don't think there's okay yeah targeting 125 that that's pretty good also okay yeah the uh enhancement got merged basically after about six years which is good I guess there was quite a lot of to figure out on the kernel on friend yeah yeah no it's not a small one I just saw but I imagined it would never happen given that it'd been there all right that sounds amazing actually I think Jonathan just put uh username and rootless stuff that would be pretty nice and uh yeah I think we should get a talk from Nathan with a microphone as well and dedicated session that would be good we'll just sign language we can add those uh Nathan would that be okay to give a like you just mentioned also that you have some reports from sites on what they want and what they report it would be interesting to to hear about that as well would that be fine so yes it doesn't matter it's a research group we'll have to be accurate and these are netizens we also had right yeah we did we did get a talk from about rootless uh quite a while ago right it wasn't specifically about user netizens but uh yeah I can't remember guy's name you'll be on it was it was a key hero oh yeah it's always a key hero on these things so yeah for sure it was him but I he gave a talk about rootless but it was more like all the issues of like network nobody fs and uh when you do user space stuff but maybe like a more focused talk on user netizens would be cool as well yeah yeah so I'll just drop the link here because uh all this rootless stuff is being tracked by Giuseppe and like here as well that's a good link to have I think we have maybe maybe take these topics put them in slots and then we we hunt for speakers yeah sounds good sensible Nathan do you have a preference on when to do this like before summer after summer okay so we can schedule the other ones already and then leave us space awesome okay there's plenty of talks to go and watch now should we put all these links in the agenda I won't be able to do it now but maybe maybe later we and then we link it from the slack channel so that other people can go and check them as well yeah yeah we can get get back and get these coming yeah yeah from the recording the text is also appearing after I hope I think we can also export it I think yeah in the recording we can then find it I think that's it do we have anything else Jamie no I was just trying to see if I could export the text easily but yeah that's fine we'll grab it later uh no nothing else from me I mean I'll I haven't got a huge amount to contribute this time unfortunately because I wasn't there um no it's good to see uh people go a lot out there anyway what one thing is that we do I forgot that because we didn't do this this time but remember we have the possibility of doing talk in the maintain a track as well about the group uh and this last time it actually was quite nice we got a few people interested in the group as well so we can we can consider for Detroit to also have a slot for the group yeah I'm always up for that I think that's good it's just a nice refresher we need to submit it when the maintain a track uh people will come uh we need to submit it there yeah just tap me up I'll uh I'm working it with you okay cool all right good I guess that's it see you in two weeks yeah thank you all yeah yeah so yeah thanks all right thank you thank you bye