 Hi everyone, my name is Kelsey. This is my colleague, Nishila. We are software engineers over at Microsoft. We work on the Linux kernel team over there. So today we're wanting to help give an introduction to the Linux kernel, a little bit of a start of what it is, how you can get into maybe start customizing it and start contributing upstream. That's something that you're interested or even just interested in knowing how that works. The first thing to get into, of course, is what is the Linux kernel? It is the core component of the Linux operating system. All the operating systems are going to have kernels in it. It manages the hardware and software interactions, enables things like multitasking, resource management, input-output request, handling memory. A lot of workings go into what the kernel does. It's of course a very important part of every operating system. There's a lot to it. It can generally be very intimidating to get into. Lots of information. There is so much, it is something that you're never going to know all of it. That's a little bit exciting. It can also be a little intimidating, of course. So Linux kernel was created by Linux Torvalds back in 91. It has been growing a lot ever since then. Now with thousands of contributors around the world, lots of companies have Linux kernel teams, which is awesome. We all get to contribute towards a, it's what's called the upstream kernel. We'll get more into kind of that. And a lot of companies will have their own customized kernels as well. But it all is based on what we're going to refer to as the upstream, is this what everybody, most people collaborate towards, and then we bring it down and customize it from there. So we will dig into that more. So on this first part, we're going to get a little bit into the subsystem structure. initial is going to cover that. Just keep in mind, this is very high level as an introduction. We want to be able to give concepts, things that you'll be able to research later. There's a lot of information that we're going to be covering here too. Of course, it's not going to be a test. You don't have to remember it all. And we're going to provide resources at the end. So it gives you an opportunity to be able to dig into all of this later on your own. And of course, you're always able to check the resources, reach out to us if you have questions. And there's a huge community out there to help as well. So, bye. Thank you. So I'm going to be talking a little bit about the various subsystems in the Linux kernel. Mostly just high level overview of what each subsystem does. So like you can see in the picture, there can be multiple applications or multiple processes that are running on top of the kernel. And one of the first things that I want to talk about is the process subsystem, which basically manages processes life cycle. So it includes what happens when a process is created and how the processes are scheduled and how a process is terminated. So those things are basically what the process subsystem manages. And there's also the scheduling algorithms around when a process needs to be scheduled on a CPU. So process subsystem is kind of the main subsystem that manages anything around the process life cycle. And the second thing I want to talk about is the memory subsystem. So this is the subsystem that comes into picture when memory needs to be allocated. So this is basically physical memory and also virtual memory, which is basically the abstraction for different processes about the underlying resource that the hardware has, which is basically the RAM. So the next subsystem is the file system. So the VFS is the virtual file system, which kind of has a common, provides a common set of APIs for read, write, and other assist calls, which kind of trap into different file systems based on the underlying physical storage device. So VFS is like an abstraction before the calls go down to the file system specific calls. So the next thing is the networking stack. So the networking stack is basically what's responsible for how a packet is processed from user space all the way down to the kernel. So the kernel has basically a bunch of drivers which enable you to talk to the underlying network interface card. So the entire TCP IP stack, which is responsible for what happens when data is passed through a socket at the application layer, all the way down to the big card is kind of handled by the kernel itself. And the next thing is device drivers. So this is basically the subsystem that handles different kind of hardware that gets plugged into the computer, like the keyboard, the mouse, what happens when you have a display board attached to your computer. So device driver subsystem is kind of what manages how different devices are handled in the kernel. And the next thing is architecture specific logic. So this is basically processor specific logic that is present in the kernel tree, which is responsible for supporting different processes. So everything else that I spoke about earlier is processor independent and the architecture specific subsystem is basically what manages based on what kind of CPU the hardware is made of. So can we go to the next slide? Yeah, so this is kind of the structure of the kernel tree and I basically had this to kind of map what directories map to what subsystems. So there's the MM directory which is the memory managed subsystem and then there's the net directory which is the networking subsystem. And then there's the FS directory for the file system or the VFS subsystem. So that's kind of how the kernel is split. So this is actually the entire tree and Kelsey is gonna talk about where to extract this tree from and how to build a kernel and what's the process that needs to be followed in terms of getting the kernel and building it and compiling it. All right, so this page here is the linear kernel archives. So this is going to point you to what I was referring to as the upstream kernel. This is the one that a lot of us all contribute to. This is kind of the kernel rule them all like the upstream, the primary one and a lot of the distributions out there like if you're thinking you've gone to so forth there they're gonna be pulling from this tree and then generally adding customizations on top of that. So if you want the original source this is where you're gonna go to get it. Sometimes people prefer of course getting one stream you can go over to Ubuntu and they provide their kernels as well and you can download those. Even like, you know, at Microsoft I work on the Windows for sub system for Linux has its own kernel. You can even, you can get that we have our own GitHub page and it has its own customizations and this is where I pull updates from as well and we try to stay as close to the upstream as possible. So that is generally a lot of times in a lot of companies that's the goal is we wanna contribute to this kernel and then we're gonna be pulling that into our products and we wanna try to keep us true to upstream as possible. So this is where you go, go to kernels.org. You would be able to, as you see there's several options of how you can get different kernels. There's a long list here, mainline, stable, long-term you see bunch of fun numbers we'll get a little bit more into kind of the cycle of how these are released and I'll get into what each one is a little bit more later but just know if you wanna get started and just do a Git clone get one of these kernels to get started and save this website and this is where you're gonna go to retrieve that. Again, also anything we reference I have on a resource page and we do did upload the slides onto Sketch so you can access those, you wanna reference it later. So once you get your kernel a lot of times what you wanna do is play with it, modify it, have fun with it, break it, do the fun things. Super easy way to get into it is one thing I try to tell people is you don't have to have a huge understanding of C or Rust now or even how the kernel works to start playing with your kernel if you want to get some kind of benefit or customize your kernel really you just maybe know a little bit of Git is going to be your base that's gonna be wanting to get your tree downloaded and then you can get started and learn as you go so I know a lot of people are very intimidated getting into kernel work and once you get it started you just make little steps, right? It doesn't have to be intimidating you don't have to know all the things that's very okay so where I recommend anyone who's getting started to start playing with your kernel and customizing it is the config so there's a very large configuration file what this is gonna be is you see a little list of things you can do with that so you can enable or disable features optimize the performance a lot of things are gonna be like support for new hardware so a lot of people will customize their kernel to be lightweight as possible by disabling all this hardware support that a lot of people have but they don't have that hardware so that's a great way like maybe you want to have a quicker boot time so you can do that and then there's a ton of different debugging options maybe there's something you're trying to debug you're managing something you're trying to debug it like a file system maybe you want the file system debugging options and turn on this is where you're gonna go to turn that on so this is a great place to get started all these little make things all are gonna result to the same thing so it's kind of a pick your own adventure what are you comfortable with do you wanna change your configs through a graphical interface or a text interface this one that's pictured is make menu config this is the one I tend to go to it's pretty user friendly which is great the only one I don't really recommend for beginners is configuring the file manually some configs depend on other configs you change one config that something's dependent on you might break things so it's a fun way to play around or see what breaks things but I always recommend what these top ones are a little bit safer and you can always do a it's called a gift diff diff the files and sometimes it's fun to see like change the config diff old config in your new one and see what all changed so if you wanna just get started this is where I recommend to start playing with your stuff then obviously the next part is modifying code so you can either add your own code you can maybe get code from somebody else super common way something maybe somebody wants a feature or a fix that isn't a kernel you don't have maybe you are running an old version but you're happy with your old version and somebody released a version that you know or a feature that maybe not released even you can add that you can add your own and that's kind of things like doing like a security fix that's huge that's important that's actually really common for like whether it's us maintaining kernels like even like at your favorite districts a lot of what they do is going to be finding fixes that might not be they're gonna be on like a certain kernel version and pulling in updates but there might be an important fix that they need to it's called back porting where they're gonna go grab it from a different a future kernel and bring it in cause it's not always an option to be on the latest and greatest we need stability but sometimes we need to pull in some fixes so these are all things that you're able to do on your own so after you get used to the config files getting in and just messing with the code sometimes it can be fun delete code see what happens or add things and just kind of see how it alters or even like doing hardware control so there is lots of stuff you can do which is yeah it can be a lot of fun so the building or kernel I was a little skeptical to even play it in here but it feels important of course that is going to be an important step but it is we're gonna go through this a little quickly the reason being is depending on the distribution you're running these steps might vary a little bit it's gonna be a generous same as you can see here kind of says like on Ubuntu, on Red Hat based system, on SUSE so we're gonna go on the general basics just note depending on your distro you can go in reference there's gonna be generally the standard commands you're gonna run through it might vary a little bit so you'll wanna look up your distro and find but we're gonna go through the general steps to show you so first step is make sure your setup has the packages it needs to do the build so make sure your packages setup and by the way I actually I wanted to show an example of the documentation out there there is a lot of documentation out there sometimes it's just finding it this is from copied directly from a website called kernel newbies this is a common website to give out to anybody that's getting into kernel development to find a very wide array of instructions so this is a great website I also have it linked at the end and this is where I pulled the instructions from this reference point so step one make sure your system setup with the packages and dependencies it needs then you're gonna build the kernel it is as simple as you're just gonna be in a kernel tree and you're gonna run make you see a little flag here a dash j you can speed it up by adding more processors to it it does take a while it can take a while unless you have a lot of resources to give to it so do be prepared for that there's a lot that's building once it builds you make a little change it'll be a little quicker or remember that unless you clean it all out and make it start over again so it's not that bad instruction wise to actually get this build as you see here there's a little link that says read some comics I couldn't leave it out because it is one of I think the one of the better XACZ comics so how to put that in there but it does it can take a while so be prepared for that do new research go have fun do something I don't recommend just watching that and see what it's doing unless you are really into that once you have it built once you want to install it again this is kind of as simple as just running another make command what you see here is it's going to install there's a bunch of modules so there's going to be like different kinds of module supports wide variety what they do so going to do like a modules install and then and then the install so it's actually going to install the kernel to your system some distros be warning there are different instructions for this if they use like a type of packaging like Ubuntu uses like Debian packaging and sometimes they're going to be like oh put it into a Debian package and then you're going to install it that way so be aware there might be different instructions and it is kind of a sometimes choose your own venture find what's going to work for your distribution your setup and yeah your setup and then you're going to also need to update your grub so the grub is going to be for booting on this one it's going to be so letting your computer know this is installed this is going to be an option for booting anytime you restart you're going to update your grub and then from here you just restart your computer and it's going to boot with the latest kernel now I'm going to give a giant warning here always keep your old safe kernels at least one installed if your kernel doesn't work your system will not boot that is so always keep a safe kernel if you want to be extra safe especially where you're starting spin up a VM to get started that way you know if you're just lost you don't know what to do you know it's as much as starting up another virtual machine for you that's always a super safe way to to do this but keep a kernel a safe kernel as a backup always that you can default to because if you start customizing as a kernel at some point you're going to you're going to build one that's not going to boot for you all right so once you have your new kernel up and running next step you're going to want to do is know how to test and debug your kernel maybe you're either wanting to find a regression or you're running into an issue and you don't know where to check initial is going to take over this part so when it comes to testing the kernel it pretty much comes down to what is the change that we made and when I say what's the change it's not necessarily a single line or a hundred lines but it basically depends on the nature of the change right so let's say if you're changing the data type of a variable from un32 to un64 that's something that can be quickly verified by assigning values to the variable add a quick print k kind of look at the dmessage log so dmessage is basically the log that's that captures everything that's coming out of the kernel so print k messages are visible in the dmessage log so a quick way to check something small is add print ks and look at the dmessage log so when it comes to bigger changes like adding a new feature and let's say you add a new .c file and you want to test it through the automated framework there are a couple of frameworks that are available one is the LTP that's the Linux test project so the source code for that is available on github it's not a part of the kernel but it's basically a test suite that kind of tests the stability and reliability of the kernel itself so it takes about three to four hours to actually run the entire test suite so let's say for example somebody's changing the memory allocator for example right like somebody's changing a config flag associated with how body allocator works so that kind of change is scary and that needs to go through the entire test suite where everything else is allocating memory and you want to see how that changes impacting the rest of the kernel right so that's LTP and the other automated testing framework that we have is the case of tests so this one takes slightly shorter amount of time it takes about 20 to 30 minutes to run the tests and it's a part of the kernel although it is like a user space program meaning all the C code and the binaries are running as user space process so this is primarily used to test things that interact between the user space and the kernel let's say for example syscall so if you're adding a new syscall then definitely adding a new case of test is a good idea so that's the main tools that we have to test the kernel and other than that there are multiple tools for analyzing the code changes that we're making while in the development process so there's like the bunch of static analysis tools there's a script called check batch which is one of the basic tools that all of us use to kind of check the code style and fix things before sending it for review upstream and there's also bunch of dynamic tools like KMM check KMM leak and kernel address sanitizer which kind of helps understand if there are memory leaks and if there's use after free errors so it's a mix of testing tools that you can kind of plug and play based on the nature of the change itself I'm also going to be talking a little bit about debugging the kernel so other than printk and demessage there's basically a bunch of config flags available in the kernel they kind of start with config underscore debug and based on the subsystem that you're testing there's more configs that you can enable and rebuild the kernel to kind of see extra information to understand what's going on in the system so definitely play around with the additional debugging configs that are available and the other thing is function trace that's the F trace so basically it's a part of the debug FS file system where you could actually set a filter on the function that you want to trace and once that function is hit in the execution of the program you should be able to see a graph of who's calling that function so F trace is kind of useful for a lot of other things as well so that's something that's very handy to use one of my favorite is the dump stack function it basically prints the stack trace in the code so usually when there's a kernel panic or a kernel loops there's a nice stack trace that kind of tells us who's calling which function and what's the faulty address but dump stack is kind of handy when you don't want the kernel to crash but you still want to know who's calling who and that's like a lazy way of figuring out then navigating different files so on the right hand side is a picture of how dump stack works so basically there's a file that is being written into in the proc file system so we are echoing a number three into a proc file and it goes through this is called API so this is called write gets translated into the vfs layer and the vfs write further gets translated into the module specific writes this call so yeah that's one of the handy functions and other than that there's also object dump and there's also added to line so these are tools that kind of operate on addresses so basically once you have an address you can actually figure out which line in the kernel and which file in the kernel is causing the fault itself so both of them need VM Linux which is basically the statically linked kernel which has all the object files together so once you have VM Linux generated you can use object dump and added to line to kind of translate the faulty address that you have and just by running a single command you can figure out which is the line that's causing the crash so gdb as usual um just running gdb on VM Linux itself lets you get into the debugger and you can do a ton of things with gdb and there's also something something called list star programming which will let you inspect the kernel data structures and what the fields are so that's where gdb becomes helpful there's a lot more things to kernel debugging than just gdb so I have a little link there to kind of summarize everything that we have that could be useful so what happens after testing and debugging that's when we're kind of confident that whatever change that we made is upstream ready or we think about sending the patches upstream right so Kelsey is going to talk about what that process looks like so I'll hand it over to her so through kind of reference a couple times is sending stuff upstream it's important so even if you're messing around and you do a little change you want to see it maybe it's useful to other people you can send it upstream if it's a fix definitely like send it out send it upstream that's a huge part of the beauty about of course open source right is all kind of helping to a better product that's going to benefit everybody right so of course we want to encourage fix something and that is you can find it anywhere else send it somewhere so a huge part of where we're going to kind of cover is a little bit of the that process what what does it look like what does the release cycle look like upstream and then also ways that you can contribute and the resources to be able to do that so to get to kind of get started a little bit of the development process to see an outline of what it looks like is there is of course multiple steps like let's say if you you have a change you made a change you built a patch you're you tested it you're confident in it at this point you're the developer now Michelle had mentioned all these different subsystems that are in the kernel and different subsystems have maintainers so there's going to be a maintainer so think like memory management networks PCI so let's say you made a PCI change you're going to want to you're going to want to send that to the PCI maintainer and that maintainer is going to review it it's going to go into huge part we haven't covered yet is mailing lists my latest kernel is widely all the communications don't know mailing lists so you're going to be sending it to a mailing list and we will we will cover a little bit more in the mailing list here so what you do is you send it the patch to the maintainer it's going to be via this mailing list maintainer wants to say the community signs off on it maintainer says it's good they're going to send it what's called Linux Next this branch is like kind of think of it the experiment branch in a sense this is where we send things to get where people can access it test it it's a little risky if you're going to be trying to use this on your like dev machine or something but all the latest things that are going to be going into like a next kernel release generally we're going to go to Linux Nest for next for testing so that's why it kind of branches off there once there's confidence in it it's going to get sent into mainline so this is actually the linus history it is maintainers send it then to mainline this is going to be the primary the next like big release is where this goes and then once it mainline is released it's considered a stable kernel so that's where we go through lots of testing it's going to be stable and from we'll just jump to this next slide is there is a new stable kernel release about every was it like nine to ten weeks depending on how many bugs are found how much fixes have to go into it so you can kind of see like if say if we just had a new release today there's going to be a two week merge window and this is where all the maintainers going to send their patches that are ready into up for mainline to get merged this is going to be for that next release that's going to happen in that two to two to three months period so once everything is determined the merge window is closed that we go into what is kind of the testing period so these are patches that are kernels are called like they're pre-patch so a period you see pre-patch you actually I've always called them release candidates if you look at the kernel name it's like a dash rc with a number and these are sent it's always announced on a mailing list everything's done through mailing lists and it will it's going to be put out there for people to test and if a regression or a bug is found something is found that is wrong with it you report that if you are going to fix it or you fix it and then a new release candidate is put out until there's confidence in it and then that's when it's going to become a new the new stable release generally I believe there's about I think the goal is like generally around there's average like eight pre-releases before like within each of these cycles that can greatly vary of course because you never know how many fixes if we're going to find a new problem that's going to need to be fixed there are also release candidates for like once a kernel is released and it's stable there's no new features that go to that it is only bug fixes so they'll still get updates but it is only bug fixes nothing new will go to that like new feature wise but those still anything that's stable like right now I think the latest release was the six dot three that is only getting fixes now so there's still a once it gets a fix there's still a release candidate that gets sent out and we'll actually touch on that because it's a great way to contribute back upstream is to test those but those are actually a 48 hour window of where you have to test those a lot of times there's only one release candidate gets sent sometimes you see a go into that two to three if a if a problem is found all right so about in that like 10 week period we get a new stable kernel it kind of replaces that the old one but about once a year is there you're going to get a what's considered a long-term support kernel they're maintained for about two years so instead of getting replaced once a new release is there they're they're they're about two years they can't get extended that's not a hard set thing these are generally the ones like a lot of distributions will go with to be because it's going to be a little bit more reliable because it is getting maintained for longer so a lot of distros are going to be picking like what's called an LTS kernel because that is going to be maintained for a long period of time it's easy for you to pick up and continue to maintain and develop and have that consistency so as like I reference you know it's easy for me to reference WSL the windows the subsystem for Linux just because since I work on that like we always stick with the LTS so currently we were on 515 and we just jumped to 6.1 and then you know come next year when the next LTS we'll jump to that and that's a pretty common thing for like you'll see your a lot of your distros like Yvonne too and so forth to do as well so the mailing list I was talking about there are lots of mailing lists all the communication is done through mailing list there are things like IRC channels still or other communication forms but you're going to be mostly doing mailing lists I listed some of them here a lot of subsystems have a mailing list so it's kind of learning which mailing list do you want to participate on where are you going to go if you have a like a KVM fix you're going to want to send it to like the KVM and to that mailing list for the people to review if you see here in bold I have this one like Linux kernel one that is the primary mailing list that is like it's very active mailing list like you can subscribe to any of these if you're interested I only recommend subscribing to that one if you are ready to filter those messages somewhere else there are like hundreds a day it will flood your mailbox so very interesting there are archives so you don't have to subscribe to these to see them this website here if you go to this you can go through and view any of these mailing lists there's mirrors you can go through and see they're very updated active so that's a great way to do it without getting your mailbox over over flooded and you can see there's sometimes there's even like Linux newbie ones like sometimes there there is like a see yeah Linux C programming and you can go in there and sometimes it doesn't always have to be like patches and so forth sometimes there's there are mailing lists for conversations some are more active than others of course as you can see like number of subscribers there's quite the variety and if you do go to this website you'll see there is a ton more mailing lists so you kind of go to the ones you're interested in don't subscribe to all of them you'll get very overwhelmed maybe just like pick a couple and start with that so a huge part one of the biggest questions I think I get asked by anyone who's interested in checking things out is where do they get started other than of course what is like the downline so for like how do they actually contribute especially if they've been customizing a kernel and they want to like maybe get into a kernel job like how do they get experience or anything like that and there's a couple different ways you can do that one of them is testing the release candidates that I mentioned earlier so there is if you go on to the you can check like the Linux kernel mailing list or the stable mailing list and if you look I did a screenshot of one of them and what I always do is I just like I have a little check for this review I'll look for review or you can do it always starts off with this message with this is the start of the stable review cycle you can search that and that way you can find it these one this one specifically is for a kernel that is already stable so you can see it says yeah it gives you about a time if you check when it was sent to the timeline it's about two days and all you need to do is you download the kernel there's a good get source for it just download it build it and boot it and then you can do more tests for it any information is good information so sometimes I might only have time to literally build and boot the kernel and then I will check initially I brought up dmessage that's a very good log to check any for any errors and I keep a little log and I'll dip it and I'll see if anything new pops up or any critical errors and then you can just report that and that's a great way to just start giving back because you literally just need to build and boot built the kernel it's a great way to just get started find a good way to start contributing on the mailing list and also break that maybe that intimidation barrier of just sending those emails this is also super useful because there's a lot of hardware out there and it there's a lot of things and if you find it doesn't boot or you find a message you want to report that before this gets released that way if there's a commit that we can that can get dropped and then we released that is great because it's much easier to fix it in this stage then after it's already released then of course the next way that you can start contributing is it's always good to report bugs if you don't know how to fix something reporting things is that is super important do keep in mind if try to report it to the right place there are kind of I think three general places there's upstream there's if you're distro maybe you're running a distro kernel it might be distro specific if you're unsure you can always report it to both somebody's going to say this isn't relevant to us and can close it sometimes it's just good to let people be aware can also be like a testing issue like maybe if you're those are always opportunities like the test suite that initial I was talking about those are open source projects too that you can also contribute to or sometimes we'll find like maybe we'll think there's a regression but really it's just a test that needs to get updated and then going to the next is the fixing bugs if you found a bug or if you don't know where to look you can also go to this is bugzilla this is a common place to report bugs for upstream you see it down at the bottom there's like a little like bugs reported in last 24 hours last seven days you always look for bugs to fix so if you have no idea you just want to do something search search on here see if there's something that you feel confident starting to dig into you can also search the curl tree for to-dos and you can find a lot of fun ways to participate in there mailing list you can find work to do sometimes somebody's maybe sending something and it no longer applies to a tree and they need help they might not have time you can always get in and help people in the mailing list those are just can be a little more difficult to find unless you're already active on the mailing list and you're paying attention to them so these are generally a little bit more beginner friendly ways to find work yeah so yeah this is this is probably a good starting point I'm only going to touch on this part a little bit is if you're going to send something upstream there is there is some guidelines you want to follow to get your patch acknowledged to be successful in contributing so get is used everywhere so it is important to have a bit of understanding on how get you can use get to send to create and send your format and send your patches you always want to be detailed in your descriptions what are you doing what are you trying to accomplish basically think if somebody's good if you're going to review a patch what would you want to know imagine you're sending it to somebody who doesn't understand somebody doesn't want to take time to try to understand what your code is doing do a good description is always helpful and then especially there's a error later on and your patches somehow related they're going to see what that change why that change was so details are very important also doing small changes you don't want to do a massive say if you're adding like an entire driver you don't want to just put it all in one commit because it's really hard to review it's hard to debug if anything is wrong so try to break things into logical changes picking the right people to review your patch you don't want to send a networking text to the PCI right like unless they're relevant of course so there's actually a script you can use to help and there is documentation that is going to help guide you through this so these are just kind of key points I point out but all of this is I listed like this documentation here and this submitting patches here it's going to go over all of these points and it's going to be a good guide like as you're doing that just do it's almost like a walk through right and then that way you can make sure that you miss all the important parts and your patches taken seriously if you miss something it's okay somebody will probably respond to the mailing list and mean like hey you or your sign-off is missing you're going to we're going to need your sign-off on this or really can you add more description or maybe doing a suggestion on your code don't take offense if this happens if you on the mailing list everybody is having critiques or suggestions done that is very normal is that's part of the working together part I know that can be super intimidating of course when people are reviewing the code but it's all part of it and that's the great thing is like we're all working towards having better code and better contributions and helping so it's going to be okay if you end up missing a step there's lots of steps I've found a lot of people on the mailing list are very understanding to that and a lot of people just want to help so it is okay to forget things or to miss something we're all we're all learning so yeah just this last part here is documentation there is a widespread of documentation which is beautiful from everything we are covering to so you can just see a list here it is the development process of code of conduct if you want to learn about subsystem there is so much documentation so the beautiful thing is is this website linked here you can go there and find this plus so much more a lot of this documentation is also going to be in your kernel tree so if you clone any kernel tree down there's going to be you're going to find a documentation directory that has all of this for easy reference for you just keep in mind as you update your kernel the documentation does get updated as well so there might be a little difference the website is keeps up to date with one of the latest releases so there might be it could always be good but sometimes do keep in mind if there may be was a big change in documentation update on the latest kernel but you are running on an older kernel there might be a difference so try to always maybe reference the kernel that you are that you're working on but there's a lot of these are going to always stay relevant like submitting patches the essential guide so there is so much out there to help with this lots of documentation to reference and definitely lots of people in community to help so here are the resources lots of it that we've referenced the only thing that I don't think we talked about is going to be at the bottom here is this extra training this is this is done by the linux foundation beginner's guide to the linux kernel development this is wonderful I've gone through this I've said this is almost the first thing I send people and they do just like where do I start this is a beautiful guide it is free so anyone can do it and it's going to help you it's going to go through and as long as it hasn't changed since I've done it it's going to help you write a patch make a change walk you through building a kernel and also talking about the a lot of what we just covered is going to be taking you more in a hands-on journey through that so if you're interested in doing that highly highly recommend taking this training it is a wonderful wonderful resource I think that's it oh that was the last slide we did have a Q&A slide I think we opt to just get rid of it and sit on here so yeah that is that is it so does anybody have any questions for us how do you find a lookup maintainers of specific ports like ARM or RISC-5 or any any architecture is it in the same maintainer list yeah like oh like who to contact or like who to think should that do you want to take that one or do you yeah there was actually a script inside the kernel tree if you run that it kind of maps who maintains what subsystem so that's kind of the first thing that we usually look up to figure out whom do we send the patch to instead of just spamming it to a wider audience so that's what I'm aware of yeah yeah that's really that's generally definitely kept up to date and your first it's a wonderful little script and you just kind of signify like the directory and it's going to let you know kind of the people that you'll want to reach out to or who who touches that so yeah does that answer okay perfect all right any others is uml viable way to test the Linux kernel is what is it uml user mode linux user mode linux so it's it's been a while since I've looked at it I haven't done any kernel development but yeah it essentially allows you to run Linux kernel and and user mode oh okay yeah um I you know I'm user space sorry yeah no I think you're catching me and um want to either like a misunderstanding or maybe a um thing that I need more knowledge on as well but I mean really you said for testing right yeah yeah I mean it shouldn't be I imagine I would have think I'm gonna get help here that'd be wonderful so the question was if user mode Linux is useful for testing oh yeah uh there's yeah there's the k-unit framework that is the framework for unit testing inside the kernel Linux and he uses um uml as a platform so we can run tests without need to reboot or virtual machine it's very easy to very fast to test with uml I appreciate you thank you all right any any other questions that's it okay thank you thank you everyone appreciate you join us