 update. So thank you and I'll see you at lunch. Hi, hello everyone. My name is David. And I'm Neil. We'll be talking about what the hypercell SIG has been doing lately. I've given various versions of this talk over the years, so hopefully this is not too boring if you've been one of the previous ones. But hey, we have a logo now. We do have a logo, yes. These are slightly less boring than they usually are. So here's the agenda for today. We will do a quick recap on what the SIG is. We will talk about deliverables and what recent work we've been doing, and we'll close with a few notes about what's coming down the pipe. Do you want to do the intro, Neil? Yeah, sure. So the SENTOS hyperscale SIG is primarily focused around the SENTOS stream. Our goal is to enable people to collaborate to bring what they're doing to support large-scale deployments of SENTOS throughout to the community and to be able to help everyone kind of benefit from those kinds of things. And key to that is being able to have this community-centric cross-company, cross-stakeholder collaboration on packages and testing, because in any real organization that's going to run any kind of Linux at scale, whether it's for desktop, servers, cloud, whatever, you're going to make your own packages, and you're probably going to be either replacing some stuff or shipping some stuff or updating some stuff or whatever. And in a lot of places, they wind up just doing it internally and then never sharing it with anyone. We want to bring that more into the open and to help people build a good community of practice around this kind of stuff and maybe even upstream some stuff to make it better for all the other people. And open to anybody who's really wanting to do this kind of stuff, whether it's desktop, server, cloud, you know, weird thingamajig here or something, yeah, we're open to it. Yeah, and to be clear, even if this is primarily targeting large-scale environments, you don't have to work at a company to contribute to the SIG. It is really open to anyone, so if you find this kind of work interesting, or if you think what you would like to do would be a good fit for the SIG, by all means feel free to reach out and get involved. So the SIG was established in January 2021. We started with six founding members from various companies. We've now grown to 30 members, which has been quite fun. And to be clear, this isn't necessarily 30 members that are actively doing things all the time, because people have various interests. Some people will join just to work on a specific thing, and they will keep working on that. But generally speaking, I would say we have a core group of people that is fairly active, and then as a wider group of people that kind of come and go or will join for specific activities. We hang out on IRC on CentOS Hyperscale. The room is also raised on Matrix. You're welcome to join at any time. Most of us are in the US, so you might get interactive replies if you hit us up on US time. But people generally keep an eye on the channel. There are formal meetings every two weeks also on IRC on the CentOS meetings channel, which you're also welcome to join if you would like. And for the last year or so, we've also done monthly video hangouts. This was something that started during the pandemic and it proved very nice both to actually have some kind of face-to-face connection, especially when we couldn't meet each other at conferences. But it's also useful, and this ended up being a mix of social time and occasionally working through problems or just shitposting. It's fun. Lots of that. It's an open Zoom meeting, so it's anybody's welcome to join. We've also started doing actual in-person meet-ups now that we can travel again. We did our first meet-up in Boston at DEF CONFUS last year, and we did our second meet-up at Connect earlier. It fuzzled them. I wanted to do a meet-up at this event, but time. We couldn't get it scheduled in time, but hopefully we will do another one this later in the year at some point. Somewhere, maybe. We will see. So this is really just more detail about the things that David and I just said. You want to check out. We try to document all the things that we're doing, and we have a back catalog of talks and things like that, as well as our track or ticket tracker that shows what kind of stuff we're working on, what we're thinking about, things of that nature. Feel free to check all of them out. That's more detail. We also publish activity reports every three months, I believe, whenever Sean emails me that I'm due for an activity report. Which he did just before this event started. We are due for another one that will come out after this conference at some points, but you can find the previous ones linked there. They're all published on the Santos blog as well. Okay, let's talk about scope. Generally speaking, the C is focused on, as I said, enabling large-scale deployments. This happens on various stages and various layers. But the main things that we try to do is, first of all, faster moving package backwards. So there are sets of packages that we would like to be able to maintain at a faster pace compared to what's in stock, sent to stream. And because these are packages that are part of core, sent to stream, they aren't the kind of thing we could maintain in a Pell. So it makes sense to do this kind of work in a SIG. One example here is SystemD that we will talk about in a little bit. The other large buckets of changes we maintain in the SIG is policy and configuration alternative and changes. So packages that are shipping the distribution that have opinionated default settings that don't necessarily match what would be a good fit for these kind of environments. So we ship packages that have alternative settings here, but with the idea that they are still dropping compatible with the stock distribution one. They just provide extended abilities and extended opportunities. The SIG is also a good space for doing large-scale testing of features. Generally speaking, in some cases you are able to test changes just by touching one package, but there are some things that involve the distribution overall. So it is useful to have a place where this can be applied and it can be tested in a production environment. And we will have an example of this later as well. Finally, we produce kernel build as part of the SIG and we also produce live DVD images. Okay, let's talk about package backwards. Packaged backwards are delivered in the hyperscale repo that we maintain on a stock sent to stream system. You can DNF install, send us release hyperscale that will give you access to the repo. I just said these are meant as dropping replacement for stock sent to packages. So if you're running a stock system and you install this, it should behave the same as your previous stock system. And if it doesn't, that's a bug and you should let us know. These packages are built against a Pell and they require a Pell because, frankly, a modern system isn't terribly useful without a Pell. We only target X86 and ARM because those are the only architectures we can actually test on, but if somebody is really passionate about, I would say, PowerS390, you are welcome to join us and maintain them. There is a long list of packages I'm not going to put here because it would be very boring. But those are the last ones that, from looking at the build tag, came up recently. In general, these tend to be either packages where the version is too old to be useful for a specific use case. So it's a back pro from Fedora or a specific patch that is missing that will take a while to get upstreamed into stream properly. So we maintain it in the SIG until it's updated in stream. And same for the updates, by the way. Oftentimes, these end up pre-based in stream and we can drop them. All these packages that we follow the upstream very closely and want to maintain them. Two major items that we are working on lately are Open SSH and Kimu. Open SSH is the event primarily by Meta, where we have an internal build on Open SSH with a fairly extensive patch set that we'd be slowly working with the Open SSH upstream community to get open sourced and ideally merged into Open SSH proper. So we are going to try and get this maintained as part of Hyperscale. There are builds of these already out, but they are not in the repo because, frankly, I don't trust them to be usable. But if you want to play with them, the sources are out there and the patch set is there. Likewise, for Kimu, we have a few members in the SIG that are interested in having an up-to-date Kimu. Ideally, an up-to-date Kimu in a Pell because Kimu would actually be a good fit for a Pell. But also, I think there's room for doing SIG work here. So we're hopeful that we'll be able to provide Kimu and potentially an extended virtualization stack as part of the SIG in the future. Right. So as part of what we do, we try to make sure that whatever packages we're either backporting, modifying, forking, whatever, that we can keep track of what's going on. So if you're involved in Fedora, you're probably familiar with release monitoring and the automated notifications for when new versions of software is out. So we implemented a variation of this that actually takes two feet. It takes release monitoring for upstream stuff, but it also takes information from CentOS Stream Core to see whether, you know, when stuff gets updated there so that if we have a package that is actually forked from CentOS Stream, we can do rebases and stuff like that as well as being able to track the things that we use, we track from mainline from Fedora. For example, like as we're going to mention later with SystemD, where we track further along, we can make sure that we're caught up with that and keeping up to date. We don't yet do the rebuild stuff. That's something that we're kind of trying to figure out how to do. It's a complicated mess of interactions with all the infrastructure to do it properly. But it's a goal that we want to have because we want to reduce, because like with the package that we have and a bunch of different things, a lot of it is very mechanical after we get the initial package made. And so we'd like to be able to reduce the busy work and make it easier for us to do more high value stuff. Yeah. And if you're in a state that maintains packages, you can run the same tooling if you would like. It's published on that repo. It's relatively straightforward to deploy an open shift. Or really any other deploy environment where you can deploy containers. So let's talk about SystemD. As I mentioned, we have a branch of SystemD that we track as part of Hyperscale. The release version currently is 252. I think we had 253 in progress, but we'll probably just do 254 at this point because that just came out upstream. We have builds for StreamMate and Stream9. This generally tracks the latest ish SystemD and it's built with the defaults of the latest ish SystemD in Fedora. Among other things, this means that it ships to SIGURB 2 by default. And I expect at some point you will actually drop support for SIGURB 1 when that ends up happening upstream in SystemD. It also ships with a plethora of extensive demons that SystemD provides that aren't shipped in center stream by default. So it ships with UMD, it ships to NetworkD, we're solved the UMD. If you're running the stock kernel from stream, you will need to run it with PSI, which I believe is not enabled by default in 8, and you have to boot with PSI equal 1 or something like that. I think we have PSI 9 enabled by default. In 9 until there, and UMD is actually provided in CentOS StreamCore now for 9 as well. It's just not installed by default. And I don't think we have the weights in the stock one. We do have the weights now. We added that upstream. So only NetworkD and ResolveD is also upstream, also turned off by default. But NetworkD is the specific one we add on top. And then there's a bunch of stuff from Meta and the SystemD community backported down into to make it better for our use cases. Yeah. So an example recently was the Gironl stuff, for example, where there was a lot of work coming from Meta, where folks were working on improving the Gironl upstream, and these were changes that we were able to test in production by leveraging the SIG. And then we were able to get them landed into SystemD upstream, and then they became available as release builds. This build also supports SELinux. Although, as far as I know, none of us actually run SELinux in production that are running this environment. Yes, well, you do, that's right. If you're passionate about SELinux, we could really use more people to keep an eye on the SELinux stuff here, because it's hard. But generally, these are just backports of the SELinux policy on to from Fedora in a module just to make it work. And if you're interested about features in SystemD specifically, you can reference just the news file from upstream, which is very comprehensive. The way we actually do is that we have a repo that tracks the actual SystemD upstream repo that we use to stage patches. That's the one that we make releases from. This usually tracks SystemD stable, sorry, not SystemD. We then have another repository for Rallenge that we used to have this on Pegger, but we moved it to GitLab so we could leverage the GitLab pipelines. CI, these are both scripts that developers used to do the general work, but it's also where the CI is maintained. The way the CI works is that we do daily rebuilds of our packages against the latest SystemD Git head. So the idea is that every day we get signal on whether, has there been any change in SystemD upstream that would break on stream in our environment so we can do something about it. Sometimes it's a bug on our side, sometimes it's something that needs to be fixed in SystemD, sometimes a little bit of work. This has been very useful at Meta internally specifically. Oh, I work for Meta, by the way, in case it wasn't clear. But it's also useful in general. Neil does not work for Meta. Neil works for Reddit. But this has been useful in general. Long term, we would actually like to have a bit more of extensive testing here so we could do things like, say, spin up a VM or cloud instance and run a battery of test, potentially run the test suite. We're not quite there yet. How to work with SystemD is documented fairly well so you can reference the contributors guide if you're interested in doing work there. Do you want to do this one? Yeah, sure. Some time it goes, lovely folks from Intel came by and said, hey, we want to try to do cool things with our CPUs and there's nowhere to do it. Yeah, we were okay with doing it. Let's go for it. We worked with Intel to create a space for them that included things like a Xelib Backport with some enhancements that they've been working on, some Xelib C stuff. They're ABI and API compatible. Last I heard, actually, that some of that's going to move to the Isosig and some of that is going to move into Centaur Stream Core finally. Some of that's going to be completely thrown out and replaced with a Fedora change at some point because there are some things that are targeted that we could go into Fedora to then bring into the next rel or something like that. There's some stuff going on there. Yeah, you can reference the blog post for more information. This is actually a pretty good example, I think, of the kind of change that can be staged in the SIG and then can provide a place to prove that it's actually useful and valuable and then it can get upstream all in all the proper places and long-term we can sunset the changes in the SIG itself. It's also one of those examples of where something can start out in our SIG and then fork out into their own SIG if it makes sense and they have some kind of sustaining energy for it. Oh yeah, as I mentioned, we target eight and nine concurrently. Most of the new development, I would say, happens on nine and then is back ported on eight. I think most of our production environments these days are on nine. I expect at some point we will sunset, well, we will definitely sunset eight by the EOL, but I expect as we get closer to the EOL work that goes on eight will start dwindling quite a bit. As part of the SIG, we actually did quite a few contributions to String itself that I'm not going to read off all of these and this is not a comprehensive list, but these are some examples of things that we worked on in String itself. This generally led to either contributions on GitLab to center String proper or working with the developers in String to lend the corresponding change in the distribution. I did not list all of their e-bases because that would be far too boring. That would be a very long list. It's far too boring. On the testing side, as I mentioned before, the SIG also provides an avenue for doing large scale testing. The example we have here is the copy and write change we've been working for quite a few years now. The idea behind this change that you can read all of it on the Fedora Wiki is that it's that change to the packaging stack, so RPM, DNF, and that ecosystem to leverage ButterFS and leverage copy and write in a way that can make packaging installs more efficient. This isn't just patching RPM. It is a fairly extensive change that involves multiple components, so it's kind of difficult to test by just installing a few packages. So what we did in the SIG is that we provide a repo called Experimental that has all of the packages needed for this change that you can install the release package there and then do DNF upgrade and that will give you the whole stack and it will let you leverage this feature. This is what we run in production and meta, by the way, because we actually run the stuff and it works fairly well, but it both provides a way for people to play with it and test it, and it's also a nice, it's a nice playing ground that allows the change to evolve and there's been quite a lot of discussion upstairs in RPM on the best way to integrate this change and it has, the form of it has changed quite a bit and I've linked the various MRs because I, the various PRs, so I think it's a fairly interesting thing to look at if you're into the history of the change, but having it out there made it very easy to find people to hear some, it's a place where this actually works that you can test, that you can play with. Yeah, and so for the kernel, we, on top of the stuff that we're doing with the user space stuff, I, in the SIG, actually make a kernel that enables other features. Primarily we do, right now I think a simple DRM enabled across the board to, for basic graphics, and we have Butterfest support. We build for CentOS Stream 9 and 8, based on the CentOS Stream 9 kernel tree, and it's available in the experimental repo and it's really like a compliment to the other stuff that we're doing in the user space. And as part of doing this particular work, we actually did a bunch of contribution work to CentOS Stream 9 upstream, and I wound up being the guinea pig for figuring out how the hell to handle upstream contributions in the CentOS Stream 9 kernel, as, and part of that I chronicled my experiences and kind of wrote for our SIG, a cheat sheet, of like how to correctly contribute fixes to the CentOS Stream 9 kernel, and in addition to that work, the SIG helps with Kmod Butterfest for the Kmod SIG to make sure that it stays working and that they can ship it for, for Rela as a separate kernel module. This is a user space. I can do this one. On the user space side, there's a few changes that take care of the user space side for Butterfest, so we have a backward of Butterfest Progs because StockRail doesn't ship Butterfest, so they don't ship the user space for it either, so CentOS Stream doesn't either. So we ship Butterfest Progs and we ship ComSides. We also ship the storage stack and install it in the distribution where Butterfest support is restored, so this can be used for building, building anaconda effectively and building images with Butterfest. We also ship other kernel user space, so it's still, for example, there's a few other things I think in the same vein. That list is not comprehensive. Finally, we ship a modified build of Kpatch. Kpatch is shipped in the distribution, but the version that is shipped in Rela and CentOS Stream doesn't include the actually useful bit of Kpatch, which is the part of making kernel patches because I believe it's meant for you get kernel patches from Radot and you can apply them. We ship Kpatch with the build side, so you can make your own kernel patches if you want, and we also ship support for Clang, PGO optimization, which is something that Meta has been working on for the past couple of years, because we use Clang extensively for kernel builds internally. I am not going to talk about PGO because it's a complicated topic, but if you're interested in this, there was a long talk at Lumber's last year, I believe, that covered this, so you might find that interesting. In addition to packages, we also have container images. Container images are great because they give you a way to quickly play and test the system. These are built from scratch on OpenShift. You can find the repo there. These aren't based on existing string container images. They are bare images built from nothing. The reason for this is because the CentOS string container images were originally based on UBI and that made it very complicated to layer changes on top. These are hosted on Quay, and you can use them with that Polymer one-liner if you want to play with it. These are minimal container images, so they're fairly suitable for doing work on top if you would like to do so. I like using them for CICD stuff because Apple is also pre-enabled and activated in it. Yep. On top of this, with the container stuff, along with the kernel and the RPM Cal, that builds up to the live media spins that we've been working on. Currently, we have two CentOS string mate spins, like an Omen KDE Plasma, that were built with kickstarts using live CD tools. These include basically their whole suite of stuff except for the Intel repo, and allows us to give people a complete experience of what our entire stack looks like and how it performs. You can download it and check it out. We've been, for the past, what year and a half or so, been plugging away at getting CentOS string 9 stuff done using Kiwi, using Kiwi descriptions, getting a whole bunch of infrastructural work done to support it within CBS, because for CentOS string 8, I was doing it on my laptop, and I was uploading everything by hand, and it sucked, and it was very manual, and sometimes I screwed up, and that was not good. For CentOS string 9, we're trying to automate and simplify the process so anyone can do this anywhere. Because of the work that we're doing for this, we actually, we helped essentially spin up the alternative image sig, and we've been using our expertise to help them get started to start providing images as well. Yeah, the idea is for, eventually, all of the image building work to converge within the alt images sig, so that becomes kind of a service that is available for other sigs that would like to build images, so we don't have to reinvent the wheel every time. Coming up in the future, as I mentioned, there is more work to do around image builds. There's a work we talked about, Kimu. We have a long-standing goal of putting together a way to do transactional updates using butterfs. We would also like to have cloud images at some point for hyperscale. These are both things that we would like to have in the future, but aren't quite there yet. Here's a few links that you can reference. I will upload these and attach them to SCAD, so you can get a copy of the slides afterwards, so you don't have to remember this. And that's all we had. Do we have time for questions? Okay. Are there any questions? So this is not a question for Neil. It's a question for you. You mentioned that you contribute stuff to stream, and there's this issue that was even mentioned today during the keynote that Red Hat only wants stuff that is useful for rail. Does this create a problem in practice for you? Yes. This has been a long-standing source of tension, I would say. I think, in general, things are trending in the right direction and are getting better, but I think there's a lot of variance depending on who maintains specific components within stream and within rail on whether your contribution will be accepted or will be timely or not. So there's areas of the distribution where it's very easy to get changes, where it's very easy to have a conversation on, hey, this component should really be very based, and we can get that done quickly. There's other areas where this can take a while, it can take several conversations, it can take getting somebody offline on a meeting to actually discuss it, and sometimes it's not a good fit. And sometimes you do end up having to do this downstream because for whatever reason it just doesn't work well. So I think that's something that we are trying to figure out overall. I would say that compared to trying to do this back in the day when we were with Santos Linux 7, this is far better than it ever used to be. And just the fact of having everything in the Git Lab repo where you can submit MRs directly there, you don't have to rely on attaching patches to bugzilla and hoping for somebody to see them, that's already been a big improvement. But yes, I agree that this is something that we generally, we overall need to do better at. Any other questions?