 Let's see. Oh, wow. Yeah, that works And this is how I can be evil with a lot of care patience and and having a good sense of smell Your band from reality Wow No, I have talked to the the pop cosmic people. I'm gonna meet up with them I think at Linux Fest Northwest actually because Carl Rochelle is gonna be there and Talking about cosmic alpha, but yeah Yeah, no, they're not gonna be here yes, but No, they already were committed for Linux Fest Northwest, so they probably just didn't want to do too Well because it's earlier than then I look Linux Fest Northwest. They probably aren't ready so I think I'll Give a minute or just get started or whatever. I mean Okay, okay, okay. Yeah so Hello folks I'm here to talk to you about making links kernel builds and kind of a little bit of the Life of a distro links kernel maintainer My name is deal gampa. I Kind of nowadays call myself an open source advocate You might have seen in previous times and eons gone by I would also call myself a professional technologist I left it off mostly because I didn't want the bullets to get too tiny As if any of you were at the previous talk that I did with Davida Kovalka about Fedora saw he remix, you know, he kind of joked that every time I do a presentation another bullet point grows on this list I am trying But I am the owner and principal of velocity limitless which includes which is a place that you know does engineering around this kind of open-source stuff and you know as part of doing velocity limitless stuff I'm a contributor to Fedora Santas open Susan magia and through those I have a litany of Roles within all of these projects that I'm not going to read off to you But you can feel free to look at it and gaze into all the things that I have done there I contributed to a lot into the base system stack notably with package management and So system software delivery with RPM DNF Kiwi Koji OBS similar things I'm also a KD contributor. I'm a member of KD EV and most recently of the XORG foundation. Yay and Another thing that is I'm a co-host of the pseudo show podcast. Check it out. It's fun Well, there'll be hopefully some new episodes coming out real soon Where we talk about the business of open source and the intersection of all the things related to open source in in wide commercial and Personal use So we're actually here to talk about the Linux kernel not about me. So we'll talk about the Linux kernel The Linux kernel is special and maybe also is not special Kind of depends on how you look at it. So When you look at the how the Linux kernel is simple, maybe it isn't so special It's written primarily in C in assembly has no external runtime dependencies Which you could argue we make it special, but you know in different ways. It's not special All the subsystems and drivers are included in the kernel and it is substantially well documented Although if you compare that to the variety of open source projects that are out there being substantially well documented is Special in a good way But it's also complex, you know, so they've just added rust to the next set of languages that are in the mix For better or worse depending on how you look at it, but it does add some complexity to the stack It builds user space components that are tightly coupled into the kernel With the kernel that's built. So for example, you have things like LibBPF and Perf and things like that that are actually not for kernel space But for user space, but they have to be tightly matched to the kernel to to be actually useful Now in recent years, there's been this tug of war between whether they want it to be tightly coupled or not We've seen this go back and forth I think now we're at the tug of war where they're now tightly coupled again Probably in a couple of years. We'll see them pull back to be a little bit more lax. I Just assume they're tightly coupled now because that's just easier for your own sanity and I recommend all of you do the same All subsystems and drivers are included in the kernel notice I listed this in both categories and the reason for that is very simple Because it's all in the kernel you have to actually to some degree understand every subsystem and every driver as part of Maintaining it and supporting it That means that you're looking at when having to deal with a concern around a particular driver You have to dig into it and you have to kind of understand what the domain of that driver is What it's trying to do and how you're going to deal with a problem related to it Whereas if you look at other operating system environments where the drivers are decoupled from the the main kernel code that's not your problem and All of it's your problem. And so that's from that perspective. I consider it complex The built tooling is sufficiently complicated and necessitated its own bespoke programs. So this is fun when you look at how The Linux kernel is actually built by itself in theory. It's simple. It's just a bunch of make files No in practice. It is a bunch of make files that can also call back into themselves It's recursive make it has a bunch of other fun stuff like that it has variables that go upwards and downwards and sideways and the documentation about what those variables do and what they affect and how they how they do things to the kernel build is weak and So because it's so complex you often there's a lot of scaffolding and extra tooling either within the Linux kernel code base To use it to work with it or sometimes distributions do their own tooling on top and we'll talk about that in a little bit But you you tend to see specialness all over the place because the links kernel build system which is I believe formally called K build is is Completely within its own realm It's not in the realm of crazy like something like say basil, but it is up there in the realm of I Don't know if I am happy that I have knowledge of how this works So When you're building the links kernel for yourself There are a few reasonable choices and a whole lot of unreasonable ones So I'm going to cover the reasonable choices and I'll talk about why they're reasonable So you can seed the build from your running kernel Using make local mod config or it's variant if you want everything built in and you want a 800 megabyte kernel image local yes config Don't recommend that it as it turns out side another all yes config mod yes config local Yes config produces images that are so big that It can't be loaded by any bootloader and it can't be loaded by firmware so Don't Seeding a build from an existing kernel old def config so if you've done a kernel build before maybe you're upgrading to a new kernel Old def config is useful because you have maybe the config that you used the last time you put it in there and you run make old def config and Anything any gaps that have been discovered as part of trying to run the build again It will fill in default values and hopefully you get a usable kernel Making the config the kernel with all the things as I said all mod config this is Well what it says on the tin it'll turn everything on but if anything that can be turned into a module will be built as a module And everything else will be turned on as yes These very simple three modes are probably where the vast majority of people who want to Experiment with building the kernel are probably gonna start from and these are all reasonably good choices because no one has any clue what all the things do there are hundreds of knobs in the build for the kernel and Not all of them are really all that well described and you're not necessarily going to know what the consequences of them are So it's a really good idea to kind of think about Taking something that you know that works working from it and then tweaking it over to try to get a better understanding of things And then I list the slightly less reasonable choice of doing it interactively and selecting everything individually So there's a number of different ways to do this mock ex make x config gives you this nice pretty cute based front end UI That gives you a little help thingies and descriptions if they're filled out in the metadata Or you could do make menu config which gives you an endcurses type UI and Or you could apparently there's a GTK version of this that's not by default and I don't know why it's there But sure make g config and that'll give you a GTK UI for no reason at all But it's there But the idea is that you can go through this individually and you can look at them and make choices This is great I think if you're trying to build really small kernels if you're trying to build something for an embedded platform where it's super easy to Decide what things to turn on often and the reason why it's that way is that oftentimes? There's a super switch of some kind that you can say Platform foo like platform raspberry pi and that just turns on all the right drivers and all the right things for that and Turns off everything else for you But if you are outside of that and maybe dealing with more generic things like x86 PCs, this is going to be an adventure Because as it turns out computers are complicated and there's a lot of stuff in there that you don't know about So I wouldn't recommend it, but if you are one of those, you know Control want all control over all the things. It's a great way to kind of learn of where all the things actually are And maybe if you want to do it in the VM, which actually I recommend doing doing make xconfig for building a kernel for a VM Environment, it's a well-defined platform. You actually there's actually documentation about what your hypervisor Actually provides us hardware. It's a great way to kind of learn how all the pieces Lock in together if you want to like get a better understanding of how the kernel piece goes together Oftentimes I do stuff like that where I use Make xconfig to just you know say platform VMware, and then I have my VMware hypervisor knobs and then I can use things like old defconfig or Local yes config to kind of see how the different permutations of things work And that's really useful and helpful to try to understand what the impacts are of choosing the different modes So for example the difference between having things as modules versus having things all built-in Or if you're one of those people who's like I don't want to have an initRD or an initRMFS or whatever you want to call first-stage boot if you want to get rid of an early boot stage While in theory you don't need to get rid of kernel modules in practice you do if they actually involve early boot like setting up desk controllers or graphics because Nothing will show up and then you can't mount your file systems, and that's not great, but you know Those are all the kinds of fun experiments you can do as an individual for that But keep in mind building a kernel for yourself is still a lot of work This is a cross-section of a subset of a kernel that that exists that You know I've used on on one of the environments that I have built out and you can see that it's a lot of knobs and While I know what most of them are at least a couple of them. I don't know like Obviously config bug on data corruption sounds real bad, and I would really like that, but I actually don't know what that does and Then config BT. I mean I think that means Bluetooth and some of the other things make me think that's Bluetooth But it could be bit torrent. I guess I Mean it's not but you know those are the kinds of things like a lot of these are very terse Names and there are no descriptions in these in these the config files that You use for the inputs of things like make old def config and whatever so you kind of need to be able to have Lookup powers by being able to read and sort through the make files and understand where they're coming from And it's really helpful to do a couple of times doing it interactively to kind of see what the hierarchy is And then you can understand from mapping that to these config files But again, it's still a lot of work to do this and again. We're not even talking about changing anything We're just talking about building and again This is it's a simple case that we're talking about right now building at once when you scale this out and do this over time with lots of other things You will find yourself doing the process Sometimes completely from scratch because a bunch of things have changed and there's no reasonable way To infer from what you previously had before to what you need to have again a couple of times What'll have some time one thing I saw was In one kernel upgrade, I think it was like a few years ago I went from a kernel that had an option for a driver that was one name and then the next release it changed to another name But if you did old def config the old name went away And the new one doesn't have any reference to the old one But the new one has a default of no so you just suddenly don't have it and you don't know that you don't have it because you Don't know what you're what you're missing Until you tried and then you suddenly realize hey, why isn't my keyboard working? Oh, it's because the USB controller driver got renamed for some reason and the default for it is no and not yes so that kind of thing can happen and that's annoying and This kind of is this is the the crux of like even if you're just doing simple like we're gonna build and keep upgrading and then Build again is difficult because the kernel doesn't have stability for anything inside of the kernel And that even includes the build tooling the build tooling is is intentionally not stable They don't provide a contract for how that works other than the make commands will do as they are written out on the text literally So old def config will do exactly what it says, but that doesn't mean it'll do what you think it what you want it to do So those are the kinds of things that can happen in yes And the kernel can also is the kernel build is also capable of attempting to wrap everything up into a package and install it properly There's you know make RP been an RPM package or build dev or snap or whatever Yeah, and it doesn't work because a lot of times the those things don't necessarily match the distribution that it's being targeted for So yes, it's fine to try it and see what happens But if it turns out that it doesn't actually integrate with your system You're gonna probably want to take a look at what's going on in that packaging and adjust it accordingly I know that I think in the last two or three kernel releases The acai Linux people actually made fixes to the RPM spec file included in the kernel tree because Doing development without being able to turn into a package and install it and being able to roll back and forth is super painful and so Having a simple spec file and having a simple way to produce a package that you can install is very valuable The Debian package I think got revised like I think a few releases ago like last year two years ago or something like that But a lot of the packaging that's inside of the kernel source tree is not exactly in the most shiniest of states But it's also an easy place to contribute if you want to like if you find some things not working and you can make a fix They'll take it. It's it's always useful to have those working, but It doesn't mean that it's going to work just out of the box is basically what I'm trying to get at So the Linux kernel for distributions has to take a little bit of a different tack because they have to care about A fair a fair number of things on top of the stuff that you have to care about if you're just doing it for yourself You know one of the bigger things that you know, I alluded to when I said that hey when you upgrade these kernels that you wind up having a problem Knowing whether you're up whether your drivers are all enabled when they're supposed to be is tracking build options based on features architectures and distribution needs I like Almost every distribution has some kind of script that actually inspects all of the make file trees and all the config states To check for deltas and changes to be able to detect things like config option renames dropouts adds things like that and To try to be able to create some kind of inference of what new switch mode needs to be done This stuff is not very good, but it is a starting point and it at least makes people aware that something has changed In a lot of respects, that's pretty much enough to make sure people find out and fix it Another aspect is executing kernel builds in an isolated and reproducible way and that's actually like kind of a cornerstone of Linux distributions Like most software Nobody assumes in the build scripts that you're doing it clean and in fact a lot of the commands inside the Linux kernel Build scripts some of the commands actually depend on a quote-unquote unclean system For example all the local foo config modes depend on pulling the config from your running system to be able to create a new kernel build and That is very useful if you're trying to test patches or whatever But it is very annoying if you actually want it to do an Independent build based on an independent config that doesn't have a relation to your what you're running on your system So a lot of times I mean it's not that there aren't modes that do absolutely do this like the def config modes and things like that but Generating the configs and then making sure that they are actually used is Extra stuff that a distribution does to make sure that their tooling does the right thing because otherwise you get weird surprises and like hey a flight you set because it was There and now it's not but it doesn't throw errors when you pass an option that doesn't exist Because all it is is a make variable so Hurray Stuff like that is actually something that they check But and of course the integrating with the distribution machinery I mentioned this earlier the upstream spec files or Debian packages or snaps or whatever don't necessarily integrate with the distribution machinery Whether you have grub whether you're using some kind of hooks with system D Or you're using an init ramifest like drake it or whatever like those pieces all have to be plugged in somehow and Unfortunately, we don't actually have a standardized path for handling this stuff So we can't just have it in the upstream kernel code Every distribution has got to do it a little special And the last bit is something that you don't really see a lot in community distributions But you see it more in the enterprise Linux distributions like rel and slas you see Backporting features and fixes to align with the user space needs and the requirements because Again remember what I said about how the kernel includes user space components and things like that that are Kind of tightly locked with what you have in the in the in the kernel itself At least in enterprise Linux distributions like rel and slea rel being read out in price linux slea being susan linux enterprise Those two what they typically do is they break out the user space components and they build them a separate packages and Those get updated Against the will of the kernel so they get updated out of sync and that means that the kernel needs to have Things back ported to make that user space component work as expected So things like that can happen and then there's of course other pure user space things that actually technically require You know syscall interfaces or things like that that exists in the kernel that maybe came from a newer one But need to be brought back because xyz application uses a bc feature, which oh well You have to deal with it, but these result in distributions implementing tooling on top of the Linux kernel build process to enhance it for these needs So in particular with red hat distributions All the kernel Stuff Goes into a central repository called the always ready kernel part of the continuous kernel integration project Otherwise known as CKI. Yes, I didn't choose the names. Don't don't look at me I don't I don't know they they could have better names, but this is what it is. I live with it So do you But there's always ready kernel stuff that is mostly around the idea that at any point in time The tree can be branched or forked to create a rel kernel that actually works because it continuously tests both the rel and fedora Config's inside of there or in theory any other distribution variant fig And this is necessary because particularly with red hat enterprise Linux They fork the kernel at a particular point in time freeze the version freeze the interfaces and then back port the hell out of things from the future back into it and that is There are there are many talks and papers about how that process is done I'm not going to get into it, but it is an important factor to consider about the design choices of this because If they didn't do it this way You're talking about a kernel package srpm that has probably somewhere close to half a million patches by the end of a Rel life cycle that is a lot and I promise you no build system can handle that Well, maybe not half a million, but a lot it is it is many tens of thousands and maybe hundreds of thousands I Think at one point I saw one that was like 89,000 patches when it was all blown out based from the The tag that it was branched from and pulled all the way up and it's like, oh Yeah, I see why you don't maintain these as patch files on top. That's a lot And in arc the way that it works is there's a redhead folder that's mangled into the source tree to Or merge into the source tree that Includes all the the distribution setup scripts and then it's hooked back into the parent make file So they modify the parent make file at the top of the tree to call into the red hat folder to do all the things a Notable feature of this particular setup is that all the configs that you set if you notice if you remember from a few slides back I said showed you this list of config options that are being set in the arc setup Each config is actually a small file and what it does is it goes through and evaluates each of the config options and Generates the kernel build system compatible config file after verifying that all the options exist And because it does it that way with the per file option file thing It is very easy to track these things From commit to commit to understand what's what's been changing and what's going on So coming down from arc into talking about the fedora saw he remix So the fedora saw he remix is a derivative of fedora linux optimized for Apple Silicon as I gave the talk yesterday with a david a Kavalka About the fedora saw he remix. There's a lot that goes on in there This is just purely the kernel part of this in the kernel part It comprises of three major changes a large patch set from the a saw he Linux repo targeting a major kernel version So for example 6x6 The configuration changes to turn it on so everything is pretty additive It's designed to be additive so that you can just Switch the knobs on and get that done and then enabling a 16k page table Page size flavor for arm 64 This arc model helps by allowing us to reliably maintain it in a way that it minimizes split brain So for those who don't know what split brain is it's the concept of which something lands and then something has changed at Different intervals and they don't line up and so something breaks because they're not coordinated correctly And so you get weird at what you get weird states in the end This is one of the reasons why if you look at CICD stuff Nowadays your CI definitions are inside your code repo because when you make code changes you want to change the CI at the same time In the era before this where this where the CI stuff was In a separate repo or stored in the application or somewhere else Somebody would change the code But the CI wasn't changed to go with it and so all the tests fail and then it can't land because the tests haven't changed and no one likes that and This is the same kind of approach if you're changing config and you're changing modes and you want to have all that all tested You want it all to happen at the same time That that's one of the advantages of this particular approach But the challenges in building this Do not start at the beginning they come from moving to newer kernel versions So getting actually started and getting a kernel out the first time. It's easy It's you bring it up you pull the patches in and you turn on the features and you're done, right? Well, the next time you have to do this then you got to Pull all those changes up and you got to reapply them on the new version you have to make sure that they work that they that the code changes underneath that haven't broken something and They inevitably do because Linux kernel maintains no stability in its own code There's a document inside of the kernel source tree that explains that you don't actually want this because the trade-off for this is When you have that kind of stability you can't really do major architectural fixes Very quickly because you have this you know backwards compatibility and forwards compatibility concerns and they just don't have any of those and This is a very common thing to happen and this is one of the reasons why if you look at Linux distributions that operate in a way that they engage with The community they tend to say we want you to have these changes upstream We want we will only backport things that are in the mainline tree because ultimately they don't want to carry it forever and You know with Fedora saw he remix I have to carry the patches basically forever until they land and it's been two years now and The patch stack hasn't actually gotten smaller And It the and one of the consequences of this that kind of sucks is I can't actually move to new kernels as quickly as Fedora does Fedora's mainline kernel basically has almost no patches on it actually at this point I think it's at no patches because because we got rid of the last of those when we got the secure boot Lockdown feature merged in like 2017 2018 or something like that And so the Fedora kernel typically doesn't have any patches other than maybe Select backports when the bug has been discovered. That's already queued for stable and we'll go away eventually But for Fedora saw your mix we don't get to benefit from that And so what ends up happening is that sometimes it takes so long that we wind up skipping kernel versions because it took So long to actually get things working again. So to give this a concrete example Fedora saw he remixes currently on the 6 6 kernel series. It has taken so long that we are skipping 6 7 and going straight to 6 8 because well 6 weeks have passed since the last time we did it and a new kernel has come out Well 8 weeks actually I think that one had an RC 8 but point being month and a half has passed new kernel Everything we did is just now pointless. So we got to go do it again In reality, we didn't actually spend enough time for this to be a waste But in there have been cases where we've actually gotten just to it at the time that the new kernels coming out And now we have to start the treadmill all over again And we have to decide whether or not it was worth actually releasing that that version or not and sometimes the answer is no It's it's not worth it because it also takes time to test because you have a lot of code that is Somewhat peer-reviewed some what not most of it is peer-reviewed because the asahi linux project actually requires peer-review for even patches landing downstream, but It's not integrated in the upstream stuff So when the upstream stuff changes then that means all the code patches have to be rewritten and it's a whole new test cycle It's a whole lot of extra work Sometimes we observe very strange things like Wi-Fi suddenly not working or Bluetooth crashing When trying to use it or reboots the reboot signal not actually being respected from it because Something just changed about how the watchdog is working or how the mailbox for the mailbox subsystem that actually handles power signals is Done or whatever like there's a lot of those things or the uapi changes for graphics and suddenly none of the drivers work again And so that has to be fixed stuff like that happens So that's kind of the door saw he now we go to sent us hyperscale, which is the other kernel I maintain So sent us hyperscale is built on sent off stream the immediate upstream for red hat enterprise Linux This means the default for sent off stream is actually the rel kernel or the kernel that is for rel it calls itself the sent off stream nine kernel Whatever phrase pick what you want, but the rel kernelist drives from a specific linux kernel version and Features and fixes are back ported in such a way that the behavioral and api stability is somewhat preserved so they try very hard to Successfully lie about what kernel version they are So if you look inside of a rel of the rel nine kernel as of I think we're on what nine three If you look inside the source tree it still says it's 514 But if you look inside for example fs xfs you're going to be weirdly surprised that it looks like 6 3 So you you get to be surprised that it's like Some of these things are so far up some of these are kind of behind Some of them are in the middle and a lot of them are broken And that that's just kind of the consequence of This particular style of maintenance Um Building a variant with different features became very difficult initially the centos hyperscale Spin used the rel kernel with different config mods and tiny changes that we back ported or contributed to the rel kernel Over time it became more difficult to do that to the point that I couldn't release kernel builds anymore that worked and so I got kind of personally fed up about it and switched over to Building from the fedora kernel so Centos hyperscale still builds on centos stream, but that doesn't mean that we have to use the rel kernel So we don't anymore and so this is now branched directly from the arc repository just like the fedora kernel is And now we go down to just doing config knobs And with no significant patches and no non-merged patches I can make new releases Really quickly it is very easy for me to pull them off and it's really just a function of can I spend like the I think it's like two hours of doing the build and test before uploading it and having the build system do it Which takes another two hours. So it's basically half a day for for doing a for doing a kernel release And uh, yeah once that's done We move on with our lives and and I can do other things or around this But the nice thing about this moving to this process is that I can just get them out the door and Because I'm mostly just doing config changes and any other non config changes I can contribute to either upstream arc or further upstream the Linux kernel depending on what is appropriate uh I am not maintaining long-term deltas and I don't hit the downsides that occur right now with the fedora sahi remix kernel Where I really have to work very hard to make sure that things are actually working Before I can release it because it probably isn't Out of the gate sometimes. I'm lucky it works just straight away Those are good days But so what about other Linux distributions we talked about sentos and and and fedora a sahi Other distributions have their own approaches to this so the susa distributions user repository that's confusingly called kernel source Um And that is actually not the source code of the kernel That is the source code of the patches and scripts used to generate the packaging files to then build the kernel In the build system. So this they they as part of their setup. They export another repository called kernel That actually includes a fully constructed git tree Of all the things And the the the kernel source export goes to the build service The opens is a build service And the prepared source tree just goes to another repo on github so people can look at it Debian uses a packaging repository that contains a make file that dynamically constructs the build flags and options Based on the vendor target architecture and kernel image Variants to build that package the sources have to be separately downloaded and the source trees have to be merged before running the build um, so debian does what in some circles they call split packaging whereas Um, uh red hat and ubuntu use do what's known as merge source or source git packaging um I source git is a confusing term. So I just merge source is much easier and explains what it is um Ubuntu uses a variant of the debian style But they merge the sources with it for the similar reason that arc does so ubuntu basically takes debian's kernel Packaging as it's upstream pulls it down into its own kernel source tree And then starts cherry picking back porting or adding custom features into their kernel They call this um, if you look in their git history in there They call it ubuntu sauce And you can always find them if you type in the words ubuntu sauce into a git log search And you'll get to see all the sauce that they add um Each ubuntu version has its own repository named after its code name for example the upcoming ubuntu 24 04 lts has the noble repository because it is a noble something or other um Arch magia and open mandrieva all have simpler packaging schemes that are much more similar to how you would do it as an individual So unified simple config file that is just updated alongside it It gets you know injected into the make file build system and it gets picked up and it gets run So this is the closest to how an individual would do it and it's actually pretty similar To how the packaging works With the for example the spec file that is included inside the linux kernel tree if you were to actually run it instead of doing it from the make file So when you're talking about this the point I want to make is that building the linux kernel for distributions is really about the long haul You want to be thinking about How you are going to keep up with it over the long period of time so that means a lot of times You see packages and this is not just the linux kernel This is pretty much everything if you look at any linux distribution The packaging is optimized around okay, you have a lot of upfront effort And then for the longer term you have this payoff of not actually having to do a lot of work to keep things up um If you wind up being in a situation where you have to do a lot of work every time then Something has gone horribly wrong because the idea is that it's not supposed to work that way so But but in spite of the differences as I outlined with You know fedora sahi sent us hyperscale Even susan red hand and all these other just and and ubuntu and debian and all these different ways that they they do the packaging These characteristics are all there right this idea of being able to trace the changes of being able to Verify things before running a very expensive kernel build to make sure that the options are all set Everybody has these scripts these little apparatuses that they build around it To make sure that they're able to build it properly and all of them don't have documentation for this um, so that that's a common thread as well, so Just like any other open source project or most open source projects What the distro does to actually build their kernel is often not actually documented and you kind of just have to poke at it and figure it out But these maintenance strategies are all optimized around what they're trying to do So you see ubuntu and and and arc optimized around the fact that Their kernels are going to collect a lot of non Of a lot of back ports and non upstream changes over time And so they do they have full forks of the repos And You know debian and arch and maoshia and open mendriva They don't do any of that. They mostly take the upstream. Maybe have a few patches here and there And the lifetimes are shorted out that it's not a big deal And so they don't fork the git repos because there's no point. Well, there's no need. There might be a point. Who knows So that's it from for like my prepared bits of this, um Questions from anyone Okay, uh, well, okay, let's see vigen I am here you go I am not eating anything Have you Is it working Cool, uh, have you uh contributed to the upstream kbuild project? It seems like uh, If the rpm build isn't working there might be an opportunity there to make things better. Um I've thought about it I've thought about contributing to so the question was asked because i'm not actually sure if the microphone is actually working Questions asked that have I thought about contributing to the upstream kbuild project? um And the answer to that because that's what the kbuild project is actually what owns The build mechanisms for all this stuff, whether it's building an rpm or building a dev or a snap or whatever um I have um, I don't use that mechanism enough myself to really have to deal with it. Um, but uh, it I would if I'd hit a problem and I knew how to fix it I probably would send a patch anyway I mean, I think I have a grand total of maybe one patch with my name on it in the linux kernel and I have A few maybe like five or ten patches which have my name listed as Either a co-author or reported by or some other fancy tag or act Well, I have lots more that are act by and reviewed by because I've been doing Reviewing of kernel patches a lot lately because of the asahi stuff Everyone says they want more reviewers so I I try we'll see how that goes um So, yeah, I mean, I'm not I'm not close to the idea. I certainly could it's just I haven't had a reason to myself Uh, any other questions? Yeah, try tapping it to see if it actually works. Yep, that works Uh, which distributions do you think have the best developer experience for a local developer wanting a quick build test cycle? Mmm. Good question Uh, I would probably say That the one that has the best workflow it It's a toss-up. So arc has a good workflow if you are primarily cherry picking fix a cherry Committing new code On top and then being able to just hit the run the go button to run it And you didn't need to do any config knobs or whatever the config knobs are actually the real messy part Because in arc the structure for the config knobs is that they're actually divided up by targets So you have common which is the base one that applies to all distributions that are used in there then you have Um You have a rel folder That's for the rel target and then you have a fedora folder for the fedora targets And then there's the secret override folder which applies to all targets And no one knows that the override folder exists But if you create the override folder and put a thing in there Then it works on all targets and you can test to see how it works across the different combinations Um Though the the fact that that is quote unquote secret is actually the biggest problem I have with it It's not that it's secret is that it's just again not documented and because They keep changing how this works It has like As some of the other members in the hyperscale sig are aware like during the time frame that i've been maintaining the hyperscale kernel I basically had to rewrite my own scripts for maintaining the kernel multiple times because they kept changing all the commands And all the flags and the methods and invoking them so If they decided to stabilize and like not change it every like few months Um It would probably be an okay experience because then at least at that point somebody could write it down And it would actually work as it said on the dock um If by taking arc out of the picture the other one that's kind of I think an okay experience for it is Um, I would say the open mendriva build because the open mendriva Stuff is interesting in that they've hacked their rpm to be able to do cross architecture builds So they have a way to cross compile any and all packages on Through it with automatically handling triggering emulation for the check section while not doing it for everything else and other fun stuff like that Because bernard rosenkranzer who is the lead of open mendriva Mostly works on embedded platforms and he used to work for lanaro for a really long time So there's a lot of in of that kind of influence In the way that open mendriva actually works so And the the spec file is easier to understand that helps too So when you're working from just like maybe just applying a patch or two and just appending it into the spec file and then Setting a knob by appending it to their to their config. That's really easy to do But otherwise and so I would consider it a toss-up it's based on which what you actually want to do But like the susan one and the debian ones really suck Because the susan one is a bunch of Yolo scripts that you kind of have to Run in a specific order and it doesn't tell you what the order is and so you can get some very interesting results The debian one sucks because it is actually really really hard to understand how the pearl make generates the config file There's a double debian folder So they have a debian and a debian dot master which then calls into itself that generates the files and Puts it back in and rewrites the build as it's running so that it re executes itself and does things That's terrifying to reason and scares the crap out of me. I don't like it So Yeah, so that's kind of how and arch is of course dead simple. They just put the bloody config file and just run it so it's but like Adding patch files and making sure that they actually work and then tracking what the build does and Being able to do it in a way that's isolated reproducible is a pain in the butt on arch So I give it points down for that because you'd get unexpected side effects from your host environment Like make package by default doesn't give you a charoot. It just does it on your system This is how all the aor helpers work. This is how all that other stuff works So you just get into no fun land when you're trying to figure out what's happening. So um For a lot of packages not even just not just the linux kernel I consider kind of the arch stuff to be kind of the the bottom tier here for that They traded away a lot of complexity and in a consequence also traded away a lot of reliability which I'm not a super big fan of Thanks. Yeah I'd like to ask like what's dev config actually good for if not the distribution using it I'm sorry. What I didn't quite get so like just make dev config, right? Yeah, but like now the distributions are actually using it Because the linux kernel is not made for distributions. It's made for the developers Right. So the whole make build process all the things that are actually in there They're not made for distributions. They're not made for users. They're made for the linux kernel developers This is also the reason why most of the apparatuses that linux distros develop never make it into the kernel itself like There's no technical reason that the config file needs to be one ginormous dot config file rather than a bunch of Is this like actually just being ashy? Let's just not there's no technical reason why it can't be done in such a way that Um, you could have the each config setting be its own file and have drop-ins and stuff like that Um, other than they don't care and don't want and don't need it And they don't care to need it because for them they are just maintaining this one big file and they just Twiddle a knob and call it a day or they use the ik config their in kernel config to source it and then rebuild their System with new code to see how it works and for their workflows. It's not important But for distro workflows where they're never doing host equals target It matters a lot more and so this stuff change is is has to be managed differently Yeah, my My question has to do with and I definitely sympathize with the statement you had in your slides Where like every change in the kernel needs to be easily tracked testable revertible And you know moving towards minimizing the amount of effort needed So being like somewhat newer to doing like kernel work maintaining like different potential like kernel forks I guess just based on like your your presentation and in your experience What would be like your suggestion For trying to maintain So many forks and different kernels like is it the arc Like are you like I know you mentioned the arc model a lot So I just want to make sure I don't get it like twisted like Are you saying that's one potential good model to explore? What would your advice be if you're largely code oriented around how you're doing stuff? The arc model is great because the arc model implies that you're a developer and that you are actually writing code into your tree And then you want to build on it iteratively and test it and whatever so The only reason the arc model works is because it's tightly coupled with something called cki, which I mentioned earlier Cki is a Is a project to do continuous kernel integration What that means is that every commit merge Change that happens It runs through a battery of tests it kicks off other tests integration tests functional tests unit tests the whole works And try us to gate every change on those before they can land In practice we can't actually do that because the upstream linux kernel doesn't do gating And so there's no actual way to prevent bad code from landing But what we can do is know when it's good or bad And then be able to work backwards without having to use things like get bisect Which actually is hard to use when you have linux when you have a code project That prefers to use merge commits that have non-linear merging And so non-linear merging means that The histories are all wonky and bisect gets very difficult And so being able to know when a merge lands when you have the good and the bad state means that you can Narrow down things like your bisection and figure out what actually happened and things like that They're useful tools in the tool chest that upstream linux is not Currently at a point where they all agree that they want But when you have that then what the arc model has is very useful and powerful When you don't have that if you're primarily relying on just building from the mainline tree and maybe having A branch with like maybe 10 or 20 or 30 patches The arc model might be a little bit overkill But it also depends on your focus if you're largely a developer type and not really used to the The sysadmin type model of thinking about the process rather than the outcome Developers tend to think about the outcome Administrators tend to think about the process And if you don't want to really think about the process the then The regular other models that other distributions do don't really fit So that's really where it kind of lines it lines down on My personal preference again as someone who's done both sysop and developer stuff is I actually would prefer Both of these things to be split out I actually would prefer that things that are packaging or distribution integration To be distinctly separate from the peer code parts Because you can't satisfy everyone with the integration of the two and also you get a lot of Unclean merges and things like that. This is mostly related to my opinion of how these things should be integrated But from a developer point of view the arc model is great from a sysop or an administrative point of view You probably would like the suza version better because It very cleanly separates how you're dealing with the individual pieces and putting it all together Thank you Any other questions? Okay, then I guess That's it Hey, and I even made it to time Mostly one two three a b c d e f g h i j k l m n o p q r s t u v I'm just gonna go ahead and get started Hello everyone. Thanks for coming to the last talk in the kernel track I am also one of the track runners, but I am also the closing talk in this track and My name is Anita Zhang. I am technically a software engineering manager at meta Officially I am engineer d manager d that is my title in workplace And I support what I call the linux umbrella family of teams But today I'm just going to talk about one of them linux user space at meta Um, so if you saw this talk at all systems go or you saw a recording of it This is pretty much the same talk Maybe with a little more details here and there and some updates Because it's been like six months since that talk I'm gonna talk about what is linux user space at meta Talk about some of the things we've done in system d in the past couple of years And then move on to some of the linux distribution related stuff Before closing with some of the things that we're doing in bpf Oh, if you have any questions During the talk I'd prefer if you wait till the end to ask them Okay, so what is linux user space at meta? The whole philosophy behind the linux related teams at meta is that we are upstream first We have some hyperscalar problems that I think other folks are also interested in finding solutions to And also other companies also have solutions so by or Problems so by coming together we can kind of work on a general solution In like a well established project that we can all benefit from So the linux teams at meta specifically i'm referring to the kernel teams we have a team of A mix of maintainers and contributors to the linux kernel As well as the former operating systems team, which i'll talk about in a bit Yeah, so a brief history of how the linux user space part got started You know the operating systems team at meta their charter was contributing upstream and building community They focused primarily on chef which we used to Update all of our fleet of bare metal hosts, and they also supported centOS, which is the linux distribution that we use in production They also supported rollouts of system d and they also did upstream packaging, which is bringing in the latest packages from fedora and related places and some other stuff Some right before i guess around the time the pandemic started There was already some reorgs going in place But the responsibilities of the team kind of split out into smaller teams with a narrow focus So a lot of the production support moved to a different internal team And the open source parts that were left eventually became what we now know as the linux user space team Which is what i'm going to talk about today So today, we basically still work on linux distributions and upstream packaging Primarily in fedora and centOS though. We also have some contributions in Debian as well and arch We still do a lot of community buildings such as these we come to events and meet with other users of linux and things like that As well as maintainers We still contribute heavily to system d We employ a few maintainers of of system d at meta though really only like One or two are active at the moment um, and we make pretty We're starting to make more contributions to Um, bpf as let the application space We already have a kernel team that contributes heavily to bpf And i guess it's not really a meta talk unless we talk about All of the machines that we support So, um, our teams is mainly focused on the bare metal fleet. So that's the stuff running directly on the hosts and hardware And of course we have millions of those and for the most part it is a largely homogenous fleet of Mostly for compute though increasingly As you know like with ai we have more gpu related hosts as well For the most part teams still maintain their own collection of bare metal hosts We also have a huge pool of shared hosts that Most engineers will be running their services in containers and those end up on the shared pool On the bare metal fleet at this point. We're over 98 upgraded from sento s stream 8 to sento s stream 9 Um system d in particular. We actually deploy as it's released next week. We're going to start rolling system b 255 We're currently on 253. We're just skipping 254 because of the We kind of missed that window um during the holidays If you're interested in system b releases in particular, uh two years ago don and i did a talk as sento s dojo About uh how we release system d in hyperscale So moving into some of the stuff we've done with system d Um, the first thing i'm going to talk about is system d umd It is a user space out of memory killer that uses um c groups to pressure stall information or psi And it's a drug to make hill decisions This started off as a facebook incubator project called umdi The system d umdi part of it is just kind of taking the policy and ideas behind umdi and making it work in system d Um, if you're interested in more of the technical details I did a talk on this three years ago at plumbers Uh, the graphic you don't have to like pay too much attention to it It exists mainly so I can talk a little bit about How system d umdi is different than like the kernel um killer for example um, so kind of the magic sauce between system behind system d umdi is psi So pressure stall information is a way to kind of quantify the amount of time that tasks are stalled Unlike the kernel um killer which focuses mainly on the amount of memory used and killing like the largest consumer Or like how fast it was consuming memory and things like that The thing about psi is that it only works on c groups So I do get a question a lot about whether umdi can support per process killing and I think just with the way psi works That that's not really uh something that we can do uh in the confines of system d umdi So um in fedora 34. We actually enable system d umdi to be the default uh user space um killer It actually took multiple versions of fedora before we kind of reached uh configuration that Wasn't killing everyone's desktop environment constantly. Sorry But out of this we did get some other benefits in system d All of the libraries and stuff that I used to develop system d umdi Now exist as various knobs in system d so in later versions You can actually configure and monitor psi more directly with system d Without having to use system d umdi And in terms of where system d umdi is at meta We actually rolled it out alongside the original facebook umdi It's taken over a lot of the uh policy and killing work that fb umdi did before We mainly keep fb umdi around for um experiments So uh we have a tool called senpai that does automated uh memory sizing tool automated memory sizing the thing with um Memory use is that it's hard to track how much memory and application is actually using Like it can allocate a lot of memory but not actually use all of it And so senpai is a way to kind of keep poking a process until We figure out what the working set is Um and the other thing we're still experimenting with is Profiling before we do an umdi kill. I think that's still Requiring tweaks to figure out The next thing I'm going to talk about is system d network d Um, so system network d it's a daemon that's shipped with system d. Um, you know, it's been in there for quite a while already Um, and exists to manage your network configurations As far as I know not a lot of distributions enable system d network d by default or like they're not using it but The idea is that you can basically manage your um network interfaces with system d style configurations So up until about 2022 meta was still using network scripts To manage their network interfaces But network scripts are actually dedicated in rel 8 Uh, thanks to our friends at redhead. We were able to keep using it until Up until the end of sent to a stream 9 base 8 basically But for sent to a stream 9 we decided to go all in on a more modern network manager Um, and in this case, uh, we had chosen to go with system d network d In order to drive home some of the benefits that we're getting with system d network d versus network scripts At least to configure the basic network, um interface it's On the network script side. It was uh A root file and i of c of g file for network d. That's a dot network config Um, if we needed to configure a virtual interface like a bridge or a tunnel You know for network d. That's just adding a dot net of configuration Um network scripts is kind of limited in the number of um, or like the types of virtual interfaces you can make but You just do that in the same like i of c of g file Um dot link files are way for you to kind of, um, configure the, uh, nick Um settings, um, which in network scripts They we don't have that directly But we were making a bunch of east tool calls and cheff I wanted to kind of drive home that we had a lot of east tool calls kind of scattered throughout cheff and so being able to You know, um centralize it all in a dot link file was nice And then there was a bunch of other stuff like sys cuddles that we were also configuring out of band We had to implement our own custom health checks for network scripts And also logging that all that stuff we kind of get for free with network d And I guess it's just nice that we don't have to rely on a bash scripts to configure our network anymore Everyone likes that right? Um, so system me network d at meta It is rolled out as a part of sent to a stream nine So the 98 plus percent of our fleet The production bare metal fleet that has sent to a stream nine is solely system me network d Um, it took a long time to figure out how to do this Safely and verify that you know our network wasn't just gonna randomly go down Um, and we've debugged a bunch of things along the way as well I think the nice thing about like a big scary migration is that everyone just wants to come together and make sure it's done safely And it was also an opportunity for us to make everything more, um maintenance friendly because you know as with a long running configuration You just kind of pile things on over time So this was a chance for us to clean that up Um, and of course we had a lot of support from the other system d maintainers Uh, you watch not bay in particular. He works on network d and he uh, whenever we had like issues He helped uh implement missing features or like give us ideas to Fix things So system d journal d. It's probably the most simple daemon in system d. It just collects and stores your logging data It has a lot of metadata in it um The driving question at meta as to why we ended up working on journal d was could we stop using our syslog So at meta we still forward our logs directly from journal d to our syslog for storage because There are a bunch of issues related to like reliability and size of the journal files that we were trying to resolve Um don talked about some of this work at plumbers in 2022, but basically it covers um Journal compact mode. This is something he did to lessen the size of journal files. Um, which is nice because it means we write less He also made various fixes related to um butterf s encryption so that our journal files are much smaller Um, if you saw a sweet taste talk yesterday, you'll know that our fleet is all on butterf s for the most part Um, and he also made a bunch of fixes related to like journal file corruptions and things like that Moving on to the centus hyper scale sig Um, so if you don't know what the hyper scale sig is it's a special interest group Formed in 2021 and the focus is on making centus stream Um, you know better for like large scale infrastructures um What we do there um You know we can test alternative policies to the centus default ones Um, we release the latest system d based on fordora rawhide and centus hyper scale Um, so that is also the same version of system d that we deploy to the meta fleet Um as part of the hyper scale sig there is also a repository called hyper scale intel with optimized packages um, and we're also using it to Uh deploy rpm copy on right in the meta fleet And of course, there's also like a bunch of infrastructure and automation work to make this easy to Kind of install and download You actually there are actually some people who are running sent to us hyper scale in their You know personal laptops, and I think daviday actually runs it on his router or something So moving on to um the fordora asahi remix Um daviday and neil actually did a talk about this yesterday, but I'll briefly go over some of the work there um, so asahi linux is Basically the goal is to bring you a polished linux experience on apple silicon max And fordora asahi remix is the official flagship linux distribution for asahi linux um, and we're interested in this because it gives you Basically a high performance reliable and pretty readily available a art 64 platform We brought together like we went and like you know got together with all of the community members It's really nice to see a bunch of motivated people do something really cool with technology And for us, uh, we actually have a few kernel developers who are using it as like their daily drivers um, and yosef actually did a blog post about how he's using a soya linux for butter fs fs tests So now i'm going to go over um some of the work we did with frame pointers. Um, so first off, what are frame pointers? Assuming you have like some knowledge of registers in a processor And also how like function stacks work Basically the frame pointer is a register that you can use to point to the start of a function stack Um, so that makes it really easy to kind of jump up to the stack and jump Basically unwind and figure out how your program executed This drawing in particular I took from the bpf book from brun and greg But the thing about frame pointers is that it's optional right if you don't include If you don't compile or you don't make use of the frame pointer You can use that register for something else and you save a few instructions um When it comes to like saving the address of the frame pointer and things like that So actually um since 2004 Frame pointers were not compiled in by default and this was across many different distributions So like I said, you got some performance wins, especially when In the days of 32 bit systems you only had Six ish registers and they were only 32 bit in size Nowadays you have like 16 and they're all 64 bit So uh, that is less of a problem Um, and like I said, if you don't compile in the frame pointer, it actually makes it really hard to stack unwind stacks Which you kind of need for debugging profiling tracing Um, and brun and greg actually last night he posted a really good blog post about Um, the history of frame pointers and where they are now so, um frame pointers enabled by default in fordora As part of fordora 38 We uh, we have met a few of us got together and decided to Kind of make frame pointers turn them back on by default And this required a lot of benchmarking and exploration because I think a lot of people are bitten by the performance of frame pointers in the past And so we had to do a lot of work to make sure that those regressions would not happen Or at least they would be an acceptable amount of regression so, um After all that the steering committee decided to conditionally enable it for fordora 38 and see where it went from there Um, some architectures already had frame pointers enabled by default Um, a few architectures we omitted because we didn't have them available for testing Um, and then python 311 in particular had opted out because they saw some pretty huge regressions But in 312, um, they actually recommend enabling frame pointers. Um, so there were probably some potential fixes and things there Um for fordora 40 the decision was re-evaluated and um actually frame pointers are now here to stay in fordora hopefully forever surely after the fordora Default or the fordora change Um was accepted and things like that Ubuntu 24.04 LTS also enabled frame pointers by default Um, and recently arclinix is also enabling frame pointers by default Um So what are one of the benefits have we gained? Um since fordora 38 at least So christian the maintainer of sysproff. He actually did He actually rewrote or redesigned sysproff in fordora 39 And as part of that he did a couple of case studies. Um, he actually found some By using frame pointers and you know his new shiny profiling tool. He was able to fix several Performance issues and fordora Some of which were at least one of them was in system at umd because I stamped his pr, but the Yeah, it was a good blog post on case studies of how you can use frame pointers to actually improve performance So some of the new newer things were working out with bpf shiny Um, so, uh, I'm gonna talk about the first two and a little more death But the last three we actually had Some talks about it already. So yesterday dan already talked about schedx, which is a way to Write linux kernel schedulers with bpf And then the last one bpfilter um quintin had a talk last night and it's a way to speed up IP tables by converting some of the rules to bpf instead bpf trace I'm gonna glaze over a bit. Um, we only recently started making more contributions again, but You know probably next year someone will have like more stuff to say about the cool stuff we're doing bpf trace Okay, so system d bpf d This is a new daemon in system d that is going to be monitoring and auditing bpf programs This is currently still in review, but I think we have a couple more ideas for detecting Issues of loading bpf programs and things like that This came well, this was originally tied to the work With bpf token, which i'm going to talk about next, but um, I think the issue that People keep talking about is like It's not always easy to get observability about bpf And so that's kind of what system d bpf d is doing now And then there's bpf token. So this was merged Either end of last year earlier this year But the the work on bpf token spun out of the need to use bpf inside username spaces So i'm sure if you know, maybe a bunch of you have containers and you're like, why can't I use bpf inside my container? Well, uh, it's because of the the privilege issue So with bpf token, you can actually delegate bpf permissions from a privileged process to An unprivileged process and that will actually allow you to run bpf programs inside containers and username spaces And we will actually be working on adding this functionality to user space soon Ideally in system d, but it kind of depends on where the design goes Which is kind of in discussion right now So that's all I had for today Linux user space wouldn't be where it is without the people that make it happen And these are some of the people who are on the links user space team any questions you can ask now or You can contact me get my Handles there um I've advocated turning off our syslog, but I hear dark comments from people that system d journal d As you kind of mentioned in passing is subject to corruption and you end up with a binary file that you can't read How how great a danger is that? There are some narrow cases where So to be clear, I believe these are resolved already, but um in the past we've run into cases where like if you Immediately shut down or power off without cleanly righty finishing right into the journal and we lose some logs but some of the Some of the fixes in the past year should resolve that like we're able to skip over corrupt entries and things like that As for Recent corruption issues. We haven't really seen as many now So it's hard to I guess it's hard to quantify the real risk Now that we don't really see them. I appreciate all of the upstream focus for things you do with system d and Coro at her centos If you were Restarting today without any centos or system d or anything in place. Would you still make the same decisions? With those technologies and and what you have built so far and in the second question if from what I understand most of your Services that get deployed on this fleet are containerized And I don't remember if that's still Tupperware or what that orchestration layer is but is What Would you still be going with that or is it tightly integrated into other things you do that is requiring from the bottom up system d sort of management? So the first question was if we were to start fresh would we still kind of make the same decisions as we do now Really hard to say I guess I I don't think there would Either if we started fresh, I guess Assuming like we would still need to have Tupperware SMC proxy and all of our other internal infrastructure It's hard to imagine how We would reconfigure everything or like we just plop it down I don't know. I can't give you that answer. I I do know some of this the decisions We made to work on a few of our open source projects they There were maybe some of them were related to Whether we were legally allowed to use some of the other projects and things like that So no, there would probably be some differences, but as for what it would look like I I have no clue Lawyer driven infrastructure is a real thing. I understand Oh I guess if you wanted me to answer the second one. Yes, sorry that was second So we are still using Tupperware. You know, like I said, it's hard to start over and kind of rip everything out Still Tupperware still have write our own containerization system As for what's changed inside the container we used to run busybox. That's now system D and The it took quite a while, but we did end up upgrading the container OS to also sent to a stream 8 and they are also working on their upgrade to send to a stream 9 now and I guess we're moving towards a world where we are able to run More than one service inside the same container We're kind of moving a little for some jobs that need like Multiservice workloads and stuff like that. I have so many more questions with nested On the system D umd could you comment more on how Responsive it is compared to the kernel only is it Able to capture processes that are gradually increasing in memory and stuff like that Sorry, I missed a little bit of the middle which component of system D. Are you talking about umd system D Yeah, uh, and what is it more responsive than the kernel umd? In terms of process that grows slowly grow in memory and other So the question so the question was is system D umd more responsive than the kernel umkiller I would say not necessarily we do have like intervals of polling Which I guess are not necessarily faster than what the kernel can do, but it's it's very different because You know the kernel umkiller has a bunch of stats that it's looking at And umd has like a completely different set of stats that looking at so they're they respond to umkills in different ways Interested in this idea of uh system D BP FD. It certainly seems like Using the service infrastructure that system D has to conditionally load BP F programs depending on the state of the system or associate them with targets could be Really powerful. Um, I don't know for example if system D has a language to express Start this service if another service has failed. I mean, that's the type of Condition where you might want to load additional testing programs I'm just curious what the If that's the sort of use case maybe that people are interested in For system D BFD, or is it mainly loading profiling programs at boot or or what? um, so system D BP FD Well, so the original idea is like Kind of spun out of the needs for like BPF program management But that's not what the system D BP FD is going to do It's currently only focused on like auditing and logging BPF programs In terms of like Doing operations on failure system D should already have that already And you are able to Connect different BPF programs to system D services as well So, um, you know that that specific case you described it can be workable in the current system D model already Hi, uh, you mentioned most of your machines are homogeneous and you mentioned GPUs has been something that is changing this I was wondering what are the challenges that you're seeing and what are solutions that you're coming up with managing large weights of GPUs Especially if you have multiple GPUs on the same hardware. Oh my gosh. I have the foresight to make some bonus slides Um, so I didn't want to you know kind of Ruin the potential for any questions like this So I actually got a similar question at all systems go when I first gave this talk Um, and it was it's related like any interesting stuff that came out of system D with regards to like the GPU fleet and things like that I think uh You know, I maybe I naively thought that you know, it's user space. It's not going to change that much But there are a lot of calls. Um, especially in the system network D related to You know talking to the kernel and stuff and one of the things we did run into is like Hey, there's a really there's a 25 second timeout for net link calls and system network D and in the GPUs You know Some there's like multiple network interfaces and some of them could take more than 25 seconds So we would see kind of interface configuration failures there So I guess it's just like thing At least the parts that talk to kernel, which are also closely intertwined with hardware Some of the assumptions that we had before are starting to break So this is kind of a more general question about system D. Um, I've noticed that It seems to have a design to thought philosophy of trying to make things easier to configure like In terms of security, there's sandboxing. There's secomp. There's some network management One of the notable things that can be really difficult to configure that Seems to be kind of left as an exercise to the reader is mandatory access control. Do you have any comments there? Like sc linux or app armor or smac or any of those Oh Have any comments on that? Okay. I think I I used to work on security it can get kind of gnarly in terms of coming up with the one configuration that works for everyone And I think while system D is kind of making some strides at least in like the secure boot areas Yeah, I don't think we really really touch the other parts. Okay, just curious if that was an explosive decision or That you were aware of or not