 Before we actually get started, a question. Who here has their background set up? Like, who here has followed the instructions in the wiki and they have a virtual machine running for the demo? Oh, a few people. Congratulations, you all passed. The rest of you, how many of you have Mac.RM1s or two? Okay, first of all, how dare you come to a Linux conference. But the second thing, the setup will not work for you. Because Virtual Vox has decided not to support M1 and M2. But for you, we have a solution which also can apply to everybody else. We have some Linux instances running. So, I strongly recommend that you set up your background set up. It's pretty easy. You sell to applications, you run it, you download an image, you're good to go. But if you don't want to go through that hustle and you want to make me spend money, but it's not that much money, Linux is a good sponsor. Not sponsored, if you know. You can go to that URL or that QR code. And if we did this correctly, it will take you to this. Hopefully yours will not be in Spanish. So, these are Linux instances. You pick one, put your name or your next handle or signature or anything next to the one. And that's yours for the entire workshop. I use a Sage like that, that's the password. It machines, I will kill it as soon as we finish the tutorial, so you cannot mind Bitcoin between, you can mind Bitcoin between now and the end of the conference. So, I'm gonna come back to the mic in about like four to five minutes and ask the same question again. And if 80% of the hands are up, we go start it. All right, the question was can I put the URL up? And the follow-up question was can I make it bigger? There you go. But again, please do the background, it's better for you. If anyone needs help, you can raise your hand and one of us will come over and attempt to help you. Attempt to be in the keyword. Yeah, no guarantees. For people who just arrived, we are doing setup right now, or you are doing setup, we're just watching you do setup. If you are able to, if you're running Linux, or are able to use the vagrant instructions to do the workshop locally, just go straight to the GitHub and go to the workshop, read me and follow those instructions. If you're on Mac, how dare you? Just scan the QR code and reserve one of the Linode instances so you can do the workshop remotely. And raise your hand if you need help and we'll come over. So, but before you raise your hand for help, who here has their setup ready by now? Either SSH into a host. Okay, perfect. We're gonna give a few more minutes and then if you don't have your setup, that's on you. Okay, so we are gonna get started. Okay, so welcome to this workshop. Do you want to introduce the workshop? Okay, so in case you didn't know, this is system D101. This is not a rehash of the workshop that we did last year, even though a lot of it is similar. But we decided to split the workshop of last year and added more content, so we split it into two sections, system D basics and system D advanced, right? So today's basics. Hopefully you will all be here tomorrow for system D advance as traditional. There will be snacks in the middle of this workshop, so if you stay, you can get to eat some delicious food. Yeah, and those are the ground rules. My name is Alvaro, this is Anita. We both work at Meta, production engineer, software engineering manager, but this talks has nothing to do with Meta. There's nothing in it that I would say it's particular to Meta, and we did it that like on purpose. So this is all information that you can take home and use it on your own machines. So before we had started, who here is kind of familiar with system D? Okay, that's great. There you go. So this is probably, there's gonna be some sections of system D that it's gonna be a surprise for all, for some of you, but it's good to have people who know about it. Well, I guess we're in a Linux conference, like everybody should kind of be familiar with it. I'm assuming that this part is not recorded, so who here is leaning positive towards system D in their general sentiment? Okay, good. Who here is leaning negative towards system D in the general sentiment? Okay, perfect, and I assume that everybody is in the middle. The thing starts so far, so it's good enough for me, right? All right, cool. So let's get started. We're gonna start with a little bit of a small slide. There's no need to need it, and there's no need to actually follow the slides. After this slide, we're gonna do the we're gonna start the workshop, okay? Do you wanna start this way? Oh gosh, what is a service manager? Well, I mean system D, the service manager, yes. Oh gosh, boo. Should we get, we can do that in it. Yeah, we'll call A.V. back later. In the meantime, I will try to use speaker. The speaker seems to not be working. State, right, not working. Yeah, okay. I'm just gonna keep going. So you'll hear us say service manager a lot. Basically, we're just referring to something that manages your services, or in this case, just the processes on your hosts. It allows you to find some operations for your services, and also provide some guarantees about life cycle in theory. In the good old days, we had these init scripts, and that was our idea of service manager. And then you end up with a bunch of shell scripts that look a bit like this. You have to kind of define all of the verbiage for how you want your service to actually operate. And there's like a bunch of boilerplate that you copy and paste from your previous shell script because who knows how that worked before. The problem with this is it's not really clear how you start and stop a method. It's not exactly the same between each script. So for example, in this Cassandra example, are there like two start commands? I don't know. Yeah, for instance, if you were to ask on the previous command, what is the user that start Cassandra, it would not be clear unless you're actually familiar with all of those commands, right? So you now not only have to be an expert on the application that you're deploying, you also have to be an expert on batch and how to set up your startup scripts. So the slides, no, and these are gonna be very short. But after this, we're gonna go to the GitHub. And there is a GitHub where we do the tutorial. Sorry, the question was, is there a website where we can follow the slides? Go ahead. And maybe even more importantly, there is no way to figure out whether your service was actually running and getting kind of observability or logs out of it. At least not in a standardized way. So we kind of named those problems, but additionally, boot and shutdown was slow. Like I said, getting logs was not standardized, so debugging was kind of hard. And at the time, Upstart and SystemD were emerging technologies. And in the end, SystemD won. So that's what the majority of Linux distribution these days are now using to manage their services. Yeah, so this is just ceiling into the territory of the workshop, but the main difference between, and you're gonna see it when we start, the main difference between SystemD and Unity is that in your batch script that you were doing, you tell the Unity what to do and how to do it. While in SystemD, you tell just what to do. SystemD will figure out how to do it, what is the best way to represent the things that you want. Okay, so now the workshop, if you wanna follow along, is on this URL. I'm gonna give you all a second to go into the URL and then we'll get started. We can get the URL back up. So the way this workshop is written is that you can actually do most of this on your own time if you just, if you fall behind or you're not able to get your setup or anything. You can watch us do it and then you can also take this workshop home and like do it and follow the instructions. If during the workshop you need help or you get stuck somewhere, just raise your hand. One of us who's not speaking might come over, might, if you have a question though, just to distinguish between whether you need help or not, please raise two hands, just so we know to call on you and answer your question. Okay, so the, if all of you heard the background, the way that you start this thing is you do background up and this will pull the latest background that we have built that I built yesterday. Hopefully you already did go ahead. The question was kind of doing more and the answer is yes. Is that better? Perfect. Okay, cool, so there's three. Okay, we still have some Linux instances available and if they run out, we can just provision some on the fly. So feel free to use them. Okay, cool, so once you do background up, background SSH should take you inside the machine and this is a, hello, what we, this is a virtual machine running on your host. It's run Fedora 39, if I'm not mistaken. It has the latest and brightest of system D and has a lot of code examples already made for you. So we start with the, so what you do on GitHub, you navigate to workshop, getting started and then the read me and this is exactly what we went through. So we can get started with system D. Okay, that sounds like I can maybe make this work. Okay, cool, so what we are gonna go with, what we're gonna start is basically the most simple part of system D which is creating system units. All right, so what I ask people to do in this one is that in your terminal, in your, you can go to ETC system, you can do LS, so you can do LS and then do a follicle ETC system D systems and then you're gonna see that right there, there is a bunch of files that has extensions. Some of these extensions are target, some extensions are service, there's gonna be timers and other things. Those are the ones that we're gonna go and check out between today and tomorrow. So the first things that we probably wanna do is that we want to create a file that is called My First Service. So you can use PIM if you're comfortable with it and then just copy the content of this into the file. You can use the Nano, I think we have Nano on maybe Emax. So this should be the simplest service unit that you can create. It's still a little bit more complicated than the simplest simplest, but it gives you like a good understanding of actually how you talk to system D. So here you see you have a declarative document instead of having a shell script that define how to start your application or where to start your pit files and all the things. You have a declarative file that tells you the description, what is your file, right? What is the service that you're creating? It has an application that you use to run, has a policy for how the service is gonna, sorry, it has a policy for how the service is gonna restart and has an install section that we're gonna definitely look at it later, but it basically, if you remember the Google run levels, those are your install. So if somebody wanna install everything that has to do with users, all they need to do is start multi-user target and this unit becomes part of that, right? So there's no longer they need to put this into a particular folder, you can just describe what you're doing. This, though, assume that there is a script here that is called service whoamai.py and it's executable, right? So let's see if this is true and you can go just copy and paste it and execute it. And this very little clown script, like everything that I write, is just display a bunch of information about the running process, right? It has the infamous Python syntax warning. It shows you who started the unit, the scope, whatever that means. Ask what is the main bit, whatever that means. And then it just print a bunch of environmental variable that as you can see, those are, as Unix always do, inherent from my environment, right? So if I type M, you see that those environmental variables seems pretty similar. By the way, if you are using background or the keynote and then you're in a Tmax shell, which you probably should, if you press Ctrl K, you can clean your buffer. So it's a good thing to have image clean. Okay, so now we have the service. So we have tell system D what it has to do, which is how to start the unit. Now we need to tell it to start it, right? So and the way that it works is that you use system CTL, you do start, and then you specify the name of the thing that you just created. My first service of service and you started and very anti-climatic, nothing really happens, just returns. So yeah, what happened? We can go and change this and ask for status. And I know that today we take this for granted, but the fact that we have so much information about how a service is running for free by just saying this is the script that you used to, like this is what I want to start, it is pretty amazing. So you get a bunch of information here. First of all, it tells you where the service is defined. Very useful. It tells you what is the status. It's running, it start running 18 seconds ago. It tells you a main feed. Okay, so this is interesting. So now you have a unit and there's something in that unit that is the root of that unit, that is the main feed. It tells you how many tasks it has. Task is basically how much forks are running. So when you start an application and then you wanna like to allow to do something, you need to fork an exec. Each fork counts as a task. It tells you how much CPU is using for whatever that means. And it has this beautiful, beautiful thing over here that it tells you C group. So before we kind of dive into this, and we have a tutorial for C groups here, how many people here are familiar with C groups? Okay, nice. This is great because like people know system D, but now they, so they might not learn much, but now they are gonna learn a lot about C groups. So before we actually talk about what our C groups are, just imagine that C group is a control group and it's a way for you to impose restrictions on a service. But system D does something very interesting with C groups. It is for each service that it starts, it put all of those, all of the things that are on the service on a C group per service, right? So this one has a C group that is called myfirstservice.service, right? So it basically, when you say I wanna start my service, it creates a C group and then everything that starts goes in there. So now system D can keep track in a better way of what things started with the service, right? So in the good old days, when you start Apache, who I'm all, and you wanna like stop Apache, what you do is that you kill the main pit, right? You have to figure out because Apache would write it like a pit file. But you never really knew what did Apache started, right? So like, sometimes you chill out and then you double forks or you escape the parent. So there's no way of keeping track of that. And then you stop Apache, but then there's a bunch of loggers running around. With system D that doesn't happen because system D can keep track of all the services because it keeps tracks of the C group. And when you say stop the unit, it will stop the C group. So yeah, so let's do that. I believe that's the 10 things that is, let's stop the service. Oh, by the way, sorry. And here in this, you can also see there is a little bit of the output of the unit, which is also kind of useful. Oh, sorry, and the unit just sleep infinity. I will probably show you what the unit does, but later. So okay, let's, we can stop the unit. So to stop the unit, we do something very simple with if you do start for start, you do stop for stop. That makes perfect sense. So now let's go back and do status. And now you see that it provided you all the information that you need, right? So it looks correctly that system D was the one to stop the unit where the unit was. What is the status? So it gives you all this information that you may want to have from the unit. Now, if you want this to actually start the service when the system start, you need to enable the unit. And if we go here and we do system CTL enable, you're gonna see that a system D create a sim link for the service units that you just wrote, and it put it there into multi-user target. Not to spoil the future, but what is gonna happen eventually when you start your machine, system D will start the first target that start all the targets. And one of those targets is multi-user. And when system D start multi-user, it will start all the units that were installed by multi-user. I will do that in parallel as much as it can. So it makes boot fast. So so far, everybody who's following along or there's somebody who is behind it, or are following along. Perfect. Success, we did it. Okay, cool. So let's move on to modifying a unit, right? Again, this is the system D one-on-one. So we have to go through all the basics so we can do better. All right, so let's do something very interesting. We can take the unit that we need and we just modify that we just created over here. In case if you're typing along and you don't know how to do this, you can do control R to the reverse search. And then you find BIM, which was the command that I use. Presenter, it takes you there. And then we're gonna add here the user, nobody, right? So in Linux, there's basically only two users that are present in every single host. So the first one is root, the second one is nobody. That is kind of like the only one that systems tends to agree on and then everything else can even have like different UIDs. So we are gonna start, so we're gonna modify this unit at the user, nobody. And now by looking at the unit, you can actually see what user it's gonna run the service. That's what we wanna do, right? All right, cool. So let's start the unit, okay? I'm gonna take some time. I'm gonna start it. And then a system D did start the unit. Like if I do status, you're gonna see that it start the unit, but it actually complain about something, right? So it says like the unit file or drop-ins have changed on disk, but system D will not pick up those changes unless you explicitly tell it to pick those changes, right? This is mostly because you care about atomic changes. Like sometimes you need to modify more than one unit at the same time. So if you modify one and those changes get picked up immediately, like catastrophic things can happen, right? So system D allows you to like reload everything when you start. So the way that you tell system D to reload stuff, oh, first of all, let's stop. And now you do this little command over here. Systemctl.demon reload. You do that and it holds. What basically that does, system D will go and read every single unit. And now you can start and nobody's complaining. Now the beauty of this is that, remember this old status? For some reason I thought the user would have been the status, but yeah, but it, but it, yeah. So, and it shows that the user that I was using is nobody. Okay. I'm gonna do, there you go. So let's take a look at this script. Service, who am I? The best way you can do this is you just cut the script. It's very short, but also you can see it because it should be on your bin folder. Service, who am I? Yes, sorry. Those are hands for questions. Okay. It's do everything at a change. It's atomic. Sorry. Oh, yes. Thank you very much. Yes. So the question was does system D, when you do the more reload, does it reload everything? Or you can specify certain things. It does everything at the same time. That was my understanding about a year or a year and a half ago. I don't know if that have changed. But yeah, it's atomic. I used to remember that this is like an event loop, right? So if, like, it actually has to like reload everything. So it has to stop. So this process tends to be very heavy. You don't wanna do that like every single time. Cool. Any other questions that people may have right now? Oh, this doesn't look good at all. Sorry. Does this one looks better? I think it does. Okay. So let's see a little bit about the script that we are running. We're gonna abuse Python a lot here. We do some coding in shell and in C to show you later, but in general we try to do most of our stuff in Python. So the script is pretty simple. It basically just figured out stuff about the unit. Then it prints this beautiful ASCII art. It tells you what is the user. It tells you what the way it was running. It tells you something, something, what is the service. And then it prints all the environmental variables. If there is no sleep, environmental variable, sorry, as long as there's no sleep environmental variable, what it does, it goes and exec into sleep. So whatever, if you saw here into the terminal when I did status, you see that the thing that is running there, it's sleep infinity and not the actual exec unit, right? So it keeps track exactly what it is. It kind of also tells you like the POD. So, and question because I didn't realize, do you guys see the, is okay, perfect. That seems to be better. I don't know if it's the best, but it's the best I can do. I go, so, oh yeah, okay, cool. So yeah, so we already went for what the script does. So let's do some journal CTL and let's learn a new command, right? So you can paste that journal CTL command. And this is actually quite interesting, right? So journal CTL, it's one thing that comes out of the box with system D. We have a full session on it. But for now, what it does is that everything that goes into STD out or STD here of your unit gets logs into the unit, right? So you get log for free. No need to start like skimming into bar log, myService.log, even though you can still do that. But the point is that system D journal, sorry, journal D will log your things, right? So now you can actually see the entire output. Remember the status has a little output at the end. Now this is like the entire output. And you can see that it shows like the informations that we were looking. Yeah, no need to know everything, but in general this just tells you show me the last 30 lines. No host name tells me don't show me the host name because that's boring. And then that's you tells you show me the unit. If you were to remove, let's do this just because of fun. If you were doing journal CTL.f as follow, as you would tell, and don't put any unit in it, you are gonna see that it's basically just showing you everything that happened on the system, right? That's not part of this tutorial, but I thought it was a fun thing to know. Okay, cool. So override units, all right, cool. So let's do some override units. So there's a few ways in which you can evade a system unit. It's a text file, so you can just beam into it and then, great, you're good. But maybe you want to do something that is called an override. And override is pretty useful, right? So imagine that you are a package maintainer and then you are creating your service. And your server, like, I don't know, you decided you are NGINX, right? So when you install the NGINX RPM, it comes with a system unit, which is the thing that the vendor tells you or tells how their system should start. You don't want to modify that unit, right? Because, like, if, first of all, if you want to change the user, that system, it starts the service. And you don't want to modify that unit because if that's managed by an RPM, when you reinstall, either the RPM will decide not to upgrade that version or worse, it will override it. RPM will not override it, but in general, like, every package manager can do that. So the way the system dissolves this problem is by allowing you to have override configs. So these are configs that you put next to your configs that allows you to extend it. Or, what is the opposite of extend it? Reduce, I don't know. Like, if you don't like the something start as a user and you don't like it and you want to remove that option, you can do it also with override. Hey, so let's do this. So the, okay, so the way that you would do this is that we're not gonna do it like this, but if you definitely just want to do this by hand, you create, remember your unit was called MyService.Service. So if you create a directory there called the name of your unit, dot t, and then add any conf inside of it, this is called an override unit, right? So, and whatever you put there, system is gonna load the unit and then load your conf and whatever the conf does, it's gonna win. But instead of doing that, we can actually do the thing that is called systemctl edit. Since systemd already knows where the files are and where it has to put the files, like, sure, it must be nice to help people out. So if you do systemctl edit, you get this beautiful thing, right? And it tells you, hey, I am doing exactly what you were planning to do by hand. I am editing a EDC systemd system, MyService.Service.T, override.conf, right? And it kind of shows you the previous unit. So this still exists, this is the other unit, whatever you put here, it is what it is. So, what are we doing? Oh, yes, we are adding an environmental variable. So we don't have to put everything again. We don't have to specify everything, we're just gonna say for service, the environment, add special m equal equals activated. I think we can just save it and then it tells you successfully install. So, oh, sorry, there was a question, Anita. You probably know this answer better than me and I don't. When you do demon reload, systemd reloads all the units, right? But can you reload just the unit that you modified? No, you, just the content itself, you can't do individual unit reloads. Yeah, I did remember that, yes. So, cool, I didn't lie to you, that's good. All right, so, that's the one. All right, so let's do, so we modified the unit, we added stuff, so let's restart the unit, perfect. By the way, when we did systemd still edit, systemd also take care of the heavy lifting of not only finding where the unit is, writing the override, saving it, it also reload the demon, so you don't need to do demon reload if you do it this way. So, I restarted and then we're gonna do this journal CTL thing over here. So, I know we're dumping a lot of syntax to you, but N, like we say, is 15 lines, the unit, and we're doing G for grep. So, we're grep-ing this, a content of this line, right? So, if I do that, it kind of shows me that the unit with that POD has the special M activated, which is the thing that we added. So, great, it works. All right, so, cool, so let's do, let's modify the unit again, and now you can see that we have the previous unit. Let's add this memory max over here, that is 10 megabytes, right? So, we're just setting one of many systemd options to control how much memory the service can use. We save it and then we can do the following, and this is why I wanted to do this, I remember now. Now, you can see that there is an override, right? And then, that the memory here is set directly to max, right? So, I want to explain, like, why do I think this example is useful? Remember that we changed the user, and we needed, or we added a manager variable, and we needed to restart the unit for that change to pick up, right? But now, I changed another setting, which was the memory, and I didn't need to restart the unit for it to pick up. So, anybody here can say, why do you think this happens, besides making life inconsistent? Yes, go ahead, anybody, just shout it. Oh yeah, but we did also this with an override file, right? So, the thing is that even it has to be something on the option, you wanted to say something. Oh, you go, go ahead. Sorry, I didn't hear you. Do you see that a running process needs more memory? You can give it more memory without needing to restart it? Yes, so it is that. I will repeat in case that isn't sound. She said that if the process needs more memory, you can just give it more memory, you don't need to restart it, and that is true, right? So, like, systemD will try to do its best not to restart service, not to need that. And in order for the restrict memory, what you do is that you modify the C group, which is something that you can do live, right? While adding an environmental variable, you can only do it at the startup of the application, right? So, those are kind of things, right? So, in a way, this does obstructs you a lot of trying to understand what's going on under the hood, but the point is that if you do it with a bit, systemD will take care of doing most of the heavy listing for you. Okay, so that was it. And yeah, and this is, we hit, go ahead, two hands, good. The question was, can I talk about some cases for the systemD overrides? So, the one that I give about NGINX, it's kind of pretty common in terms of vendors, right? NGINX is kind of the default application that my brain goes, but in general, like if you're a vendor and you want to do applications, that's what you do. The other thing is that if you're a system administrator, one of the things that you tend to do, you also add certain overrides, so that every application that runs on your host like has a firewall, which is an option that you can add, right? Or you have a, or you run something before every unit starts, there's an option that is called pre-start or pre-exec-start or something like that, that is something that start before your application starts, right? So you can add something to log, I don't know, right? So those are the kind of things when you want to do modifications on the unit without actually touching the unit. Do you have another example of that, Anita? Isn't the one that comes home with Fedora? Yes. All right, so Fedora comes with an override. What does that override does? The one that changes the abort and timeout behavior on units, that's supposed to speed up startup. I mostly use overrides to override vendor-shipped units, like if they ship, if they have something for SSHD to service and I don't like a certain behavior, I would use an override for that. It just really allows you a way to further tweak what's already on the system. Yeah, the Fedora one that you were saying is that like the Fedora, like the Fedora pack? Yeah, Fedora also use ships overrides for their units as well, or like certain packages units. Yes, okay, cool. So the rest of this part of the tutorial are us just going through a bunch of systemd commands. It should go pretty fast, and then we're gonna take our first break because it's unfair to keep three people like locked up for three hours. And then you get to hear Anita instead of me, so that's a plus. All right, cool, so let's, okay, so this is quick and dirty walkthroughs to some systemd commands. We already went to status and then the unit name. You already saw that. What you didn't saw is that you can do status and you don't need to put a unit name, right? So if you just do status, just like that, it shows you a kind of a list of the system from the point of view of systemd, right? So basically you're asking for the status of P1. So, sorry, not the status of P1, that's not what it is, but it kind of give you like, oh, so these are all the things that are running, all the processes, kind of like give you like a nice overview. This is a virtual machine, it has very little going on. So that's cool. Now, a lot of the time you do PS outs and then you see all the processes, you don't know what's the service of a unit. Like if I do PS outs F, you're gonna see that list. And yeah, there's a slip's finity, but I have no idea what's the unit that that is. So I cannot ask for the status of that unit, but what I can do is I can ask status for 5423, which is a magic number. No, I'm lying. It's a PYD of the unit, right? So if you just know, sorry, of the process, if you just know the PYD, system D will, since it's keeping track of everything that is under the hood, you can use system D status by the PYD and that's all good. If you do system CTL without the status, you get a horrible view, but it does tells you like all the units that were executed in the service. Or in the machine. So it has a lot of information and if one of those is read, it means that it failed. Nothing failed here. Okay, so that's status. Very nice, very useful. Now, assuming now that you do know your service, like this one, and you want to actually see the content of your service, you can do system CTL cut on the name of the service. Again, since systems D knows exactly where everything is, it can nicely assemble the list of files for you and display it. Very useful. And it displays exactly what we say it should display. Now, if you don't want it to cut, if you want it to show, whatever that means, nah, we know what it is. There, it will kind of tells you all the properties that the unit has set. All right, so these are all the options. Go ahead. Yes, yes, yes, yes. So you definitely can do, so the thing is system D keeps the status of the entire, so the question was if we can use this to pick what you can put in override and the answer is yes. System D keep a list of all the things that are running. And sorry, and for each service, it has a full list of properties. So if you set ones, the other ones are the default value, right? So now this will show you all everything, so you can override whatever you want. With that said, if I'm not mistaken, this one is using the divas interface. So we're gonna go that. So this doesn't really map one to one to the actual setting. Like for instance, here it says restart you, I'm not sure. Yeah, so it says restart you seconds. All right, let me go to the terminal. Restart max you second, that's a microseconds, I believe. It's, yes, it's microseconds. The unit, the settings is restart max seconds, right? So you don't use the use seconds there or runtime max or time or something like that. You can check the docs, but that's nano seconds, nano, nano, okay, sure, right? Max is hard, right? Oh yes, no, no, no, perfect. But yeah, that you can do. You can also expect just a few settings there. So you don't need to watch everything. You can just specify that to be and it will tell you what it is. And here's what I was telling you, right? So you have exec start and it kind of show you the way that system sees this option, right? Like with all the different properties. But it's kind of like, if you wanna know the main PID, you can do this main PID, but do a specify the unit. And this is very useful also for Chelscript. So if you specify like that's P main PID, it will tell you main PID equals on the PID. You can also do, add this option over here that is called value. And then it will just give you the value. So you just shove that into an amount of variable and you're good to go. Any question, comments so far? We're, yes, you're good. Chris, interrupting. So the thing with system CTL show is that it only has what system D is stored in memory at this moment. And that's not necessarily the same as what system D is already running. So certain properties, once the unit is running already, you know, that's it. But you can reload and change the unit and system CTL show will show what system D has in memory. And you know, it's not exactly the same. So just keep that in mind when you're looking at show and you're confused like, this says this one setting, but you know, when I'm actually running the service, it says something else. That's why. Yes, question. Yeah, so the question here, it's the like, I want to paraphrase you pick at the word watchdog and ask if you can set the watchdog for service. And the answer is yes, you can. And we actually have a tutorial for that, which comes tomorrow. Yeah, so many things you can do. Go ahead. The question was, when you use system D kill is when the service doesn't respond? So no. So if you do system CTL stop, like if your service is unresponsive, you can still use system CTL stop. It will send a sick term to the service. By default, you can change all of this, but it will send a sick term. And after like a minute, if the unit didn't dive, it will send a sick kill. So the stop is supposed to kill the process at the end of the thing. Very rarely fails. System D kill is when you want to send signal to the process, right? So the same way that when you use the kill command, you say like kill and then you send like a sick hub, right? System CTL kill, you can do that, but you don't need to know the PYD, you just need to know the service name. So you can say kill the service and sick hub to NGINX and then system D will figure it out what's the right way to do this. So with that, we're gonna do system CTL kill. And yeah, it is exactly that, right? So you send the signal, ah yeah, but this is actually quite interesting. So I'm gonna send the signal kill nine, like that's nine, like sick kill, like there's no coming back from that. And then I'm gonna do status, that's not the status that I want, that's the status. Wait, did I send the signal? Oh, oh great, sorry. Yeah, so you can see like the unit's still running, but the startup time is seven seconds ago, right? So what happened there is that I send the sick kill, like system D send the sick kill to the proper main PIV, that thing died, but the system D has the setting that you remember that it's just like restart always, that we put it, so the unit restarted, right? So that's kind of like intended. If I were to do stop and then do status, you see that the unit is stop. Is it stop? Yes, it's stop, all right, there it says, right? So the difference, so the difference was that system CTL kill, after that, the output is not that the unit has to remain stop. While you do system CTL stop, system D will kill the unit, and then after that it will ignore their always restart, because you were explicit, I actually wanted to stop, otherwise you will never be able to stop your unit. But it's very useful. I don't think like kill is such a bad name for this. Yes, it's the unit's kill, but also the unit's kill has a very bad name. Okay, so there is a system CTL this unit, which is the same one that we already, so when you call system CTL with no parameters, it's exactly the same as calling it with no units. So it would list units. So this one, it's exactly the same as this one, and that again shows you like a nice view of everything that has run on the service. While system CTL status shows you everything that is running on the service, right? While the other ones shows you like things like these weird devices that are for mounting your TTWise, that will not appear otherwise. Did you add this or did I add this? I didn't remember this. Did we add this, did you add this? Okay, cool, we'll pass Alvaro. You have list units, so it's an awesome thing. You can use wildcards for searching, you're basically saying like give me all the service, you can do that. It's also very useful. Yeah, it also works with system CTL status, like it's the wildcard, right? So it kind of shows you like everything that has a service. You can be more specific than just service, but that's it. And I think there's other things that are cool, it's called list pass, list socket, list timers, that do all the things. And it's 11.04. I think it's a good time for us to take at least 10 minutes, bathroom break, coffee break. Thank you very much. If somebody have questions? Oh yeah, oh yeah. So is it time for portals? Yeah, and your prize for being, your reward for being here and learning about system D is that we have some portals pastries here. It's time, okay perfect, niche approved. So yeah, so you can come and grab some portals pastries, they're free. You don't have to wait, you just have to eat them. Thank you, by the way. Okay, four minutes and we'll get back to these fun times. Great town, it is, yeah. I was scammed out of coffee, there's no coffee. Okay, one, two, check. Okay, we are ready to get started. So if people can take your seats in a color. Okay, so for the last part of system D 101, we're gonna talk about ephemeral services. So system D provides a way for you to just kind of generate units on the fly, like you don't even have to write a unit file. The only downside with this is that you, it doesn't preserve the unit between like restarts and things like that, it's just stored in memory. So we can give that a try by running this command here, system D run, service who am I pie, it's the same one we've been using before. So now you see that system D created a unit file and it's starting to run it. We can take a look at the contents of this unit by copying the unit name and doing system CTL cat. And you can see that system D stores the unit file in slash run, which is in memory. And it has the contents of what it generated here. So it should look very familiar to you already since you created a basic unit at first. The other thing you can do with system D run is create an interactive shell. So if you do system D run dash dash shell, this will kind of drop you into the shell inside of the system D unit. So you can do system CTL status to get what the pit of this unit is. And you'll see that from within the shell in the system D run unit, it looks kind of like this. You can go ahead and play around with this for a bit, but it's basically just another bash shell within a bash shell. And then when you're done, you just go ahead and exit and you'll be back on the host or the virtual machine wherever you are. Any questions about that? The question was why might you want to do that? Do you mean system D run or system D run shell? Okay, so one of the useful things about system D run shell is that if you're debugging some kind of complex behavior with your unit, you can use it to kind of drop into what the state of system D is like. For example, hmm, you got an example, Alvaro? Yes, so well, you can use this for tutorials and show how things work. But in general, it's kind of like looking at the world from the point of view of the unit, right? So see, later we're gonna see that system D can do things with your file system so the unit may sync that route is read-only, right? So you may not really know that that becomes a problem so you can start a shell with the route as read-only and then do the commands that your application would do and that gives you like a nice feeling of how the unit works. That is kind of very useful. But we use it extensively in this tutorial because we ask you modify this. Now go into a shell and run a few commands and see what effect that has in the world. I'd say like just system D run shell by itself may not be super special but once you start getting into the sandboxing properties and you run another process along with the shell or something, you'll be like, hey, why isn't this running properly compared to outside of the host and it's because of how the system D property is behaving. So it kind of allows you to test some of those in more detail as well. Any other questions? So the question was how does system D know to drop into that particular shell? I mean, it's creating the unit at the same time as it's dropping you into the shell or like it creates it, run execs and then you're like in the shell there. It connects via the pseudo terminal there. Does that make sense? Yes. This becomes more clear maybe in the second example. So we're gonna start using system D run shell more extensively to run an interactive process. So the system D run shell is pretty much the same as adding this dash dash PTY argument. So you can do more fancy things like I was saying by copying this command and we add dynamic user true, which is one of the properties that allows you to run the process as a random user. So let's go ahead and do that now. So let's go ahead and do that now. Sorry. So we do that. You'll see now that the service who am I, Python script that we ran before is now running as a kind of random user that system D has generated. Later in the workshop we'll talk more about how dynamic user works, but just know that it's one of the ways you can use it to run your process as an arbitrary user. For the very first basics of system D. Are there any questions about this part before we move to the next part of the workshop? So now we're gonna start talking about other system D unit types. So far we've pretty much only talked about service units because those are the ones that allow you to actually run commands and execute processes. The system D has a whole suite of other unit types that allow you to do other things. So for example, timers. If you are familiar with the cron tab, it allows you to kind of periodically run an executable or something on a schedule. And so if you recall the cron format, if you run something every 10th minute, you can do it like this and then you add the script or whatever at the end of the line. To do that with system D, you use a timer unit. So we can go ahead and create a timer unit with these contents. We're gonna use system CTL edit so that we don't have to worry about reloading the unit later. So the difference between the service is this timer section here. And the property, one of the properties under timers is on calendar and that allows you to kind of insert a cron style format and allow you to run the timer. And if we want this to run at, or like start the timer at boot, we also added this install section and it's wanted by timers.target. Let's go ahead and exit. Because we use the system CTL edit, it already reloaded for us. So that set up the timer section. How do we actually include the script that it's supposed to run? We do that with, by creating a service unit of the same name. And we can copy these contents here. And this is just a generic service unit like what we saw before in the previous tutorial. What this is gonna do is run service who am I, a Python script you're already familiar with and set the environment variable to no sleep equals no. Let's go ahead and save this as well. And now we can actually, we can go ahead and start the timer and watch when it fires. So I'm gonna go ahead and copy this, start the timer and we use watch here to see when the status changes on either the timer or the service unit. So you saw that the timer executed and the service status now spits out the usual logs that you're familiar with. Yes, the other thing we're noting about the timer is the, yes, let's dig into the status of the timer unit a bit more. Delete this watch command. So if you're already familiar with system CTL status, on timers we also get this trigger and triggers behavior. So what that does is it will tell you when the timer will next run and trigger this service. There is a property where you can actually change the service that the timer triggers. You can kind of look that up in the man page, but by default if the service has the same name as the timer it will trigger that service. What else is interesting here? We already know about loaded, it's when system D had loaded this unit into memory. Yes, question. So the question is what's the advantage of using system D timers over the previous cron jobs? So using on calendar we don't get to see a lot like the special sauce with timers. We're gonna go into some more properties later. But I think the primary advantage is that we can actually trigger a whole service file. And with that you can define, you can fine tune the behavior of how the command should run in relation to the timer. Whereas with cron tab it just kind of runs the script and then it exits. So yeah, so that's that one. There's other two things that I think I kind of like mainly superior to cron, but first let's say cron it is very simple and it is very familiar. So you can use that one if you like, right? But yeah, so the first one is the use of time. System D has monotonic, so you can say trigger these five seconds after the previous execution. Or you can do, since boot, do this half an hour after boot, like those kind of things. It doesn't have to be tied to a particular time of day. There's also flexibility in giving you supplies. I don't know if that's the right word, but it is like start this at one plus minus five seconds. So you can distribute a whole set of things in time. So that's the first one that I would say it's pretty useful. And the other one, it is where the strong suit of System D is that you actually get status and locks. And like in cron it's really hard to see if your thing run and what was the status of that run. You need to spec locks. In System D you just say like status of my timer and it will tell you yes, I was here. I did what I was supposed to do and it exit successfully our failure, right? So it cannot give you that. Well now we're gonna see all the things that Alvaro just described. So starting with the monotonic timers, let's just go ahead and edit our first timer and change on calendar to instead use on active 30 seconds. So what on active second does is it basically says let's just run this or execute this timer 30 seconds after we activate the unit. And basically it will never run again. So you can also combine this behavior with, that by itself is probably not super interesting but you can also combine it with like on calendar, on active seconds, on boot and things like that to further change how you wanna start the unit behavior. So let's start the on active. So if you're looking at this watch, you see that there is a countdown till the unit's gonna trigger. Oh my God. Great, it triggered. And in theory if we look at it again it will not show like a next trigger time. Yeah, it elapsed and that was pretty much it. Their system has a bunch of different properties like that. In the next example we have on active seconds, on boot seconds and probably a few more that I'm forgetting but you can always look in the man page for the rest. Okay, so similar to ephemeral service units you can also create ephemeral timers if you just kind of wanna run something on the fly. So if you do system D run on active 30 seconds and just echo this, echo hello from the past, it will automatically create a timer and the corresponding service unit for you. As seen here you see that it's now running this timer unit and also will run this service unit when it fires. And similar to before you can do and you'll see that the timer is already active and it's just waiting to, well it probably already ran because I didn't do the watch. It already created the service and the timer and it was gonna fire it after the 30 seconds. Yes, and thanks to Alvaro we can see that, well it was active 39 seconds ago and it already fired down here. Hello from the past. So the other thing Alvaro was talking about was splay so if you don't want, if you're managing like a fleet of hosts and you don't want every single timer to kind of start at the same time because that would overload whatever service you're trying to make them call or something like that, system D provides a way to automatically randomize when the timer execution actually fires. So to do that you can use a randomized delay seconds. I also like to combine this with fixed random delay because if you have fixed random delay equals no, every time you actually reload the unit, system D will change the amount of randomization that goes into when the timer executes. Yeah, it'll make it random every time so it's not gonna have like a consistent or random delay. So let's look at this unit as an example and its contents. Can someone tell me what it does? Brave Soul, any Brave Souls? Sir over here with two hands. And say that it will run every 30 seconds but with a random give or take of 10 seconds but probably the same random give or take of 10 seconds every time. Yeah, that sounds good enough to me. Congratulations, you win portals. You're also, I actually don't know, I'm actually bad at reading crawl tabs. I believe this is 30 minutes, right? Yes, it is, yes. So this will run the timer every 30 minutes. Sorry, not totally correct, you didn't say seconds. No, these are seconds, right? These are seconds. Okay, nevermind, I'm wrong. It'll run the timer every 30 seconds adding up to 10 seconds of random delay. So extra stuff we won't cover. Like I said, there are a bunch of other properties you can use to set your timer behavior such as when the clock changes, when the time zone changes, or when the system wakes up. Ooh, actually one thing about time zones. This is a big gotcha. When you use on calendar, it uses your local time zone. I know, it's crazy, right? So like when daylight saving comes around and your timer doesn't fire for like the next hour, that's probably why. So on calendar uses the calendar. Yes, yes, I mean it sounds obvious like on calendar but that's a huge difference from the crontad behavior. So you can actually stick a time zone on the on calendar if you like really don't want to or you can use them on atomic timers which are based on your system. All right, things in UTC? UTC zero. Okay, so, yes, monotonic timer. Time is an illusion, it doesn't matter. All right, so we're not gonna go over every single unit but the other interesting one you probably want to know about are path units. So what path units allow you to do is kind of watch a file or a certain path and then run like a service or something when that file changes or it's modified or it's created. So let's go ahead and create this SH config watcher path and I'm gonna copy the contents from our GitHub by clicking the button. So let's look at what we have here on this path unit. Similar to service and timer, it has kind of a path type or header, I don't know what you call this. It has a path change property. So what we're gonna do is watch when ETC SH-SHD config, the contents are changed and saved to disk. And of course we need the corresponding service unit along with it, so we're gonna go ahead and do that. So just copy it from GitHub. So what the service is gonna do is when the path unit fires, we are going to systemctl reload shd.service. I got a good question during the break about systemctl reload. So certain service units define a reload behavior. SHD is one of them. You might wanna do this if instead of restarting your whole service to pick up changes, you just wanna run reload in your code itself. You do something to reload the config or whatever. So let's go ahead and save that. And now we can actually, we'll go ahead and enable the path unit and then start it. Now we can check the status of SHD, just to see where we're at. And you can see right now that SHD is just running normally. But if we, let's say we just modify the ETC SSH. SHD config. This is the file that we're watching on the E-PAT. Let's say that we uncomment these lines here to change the log level and then save it. If we look at the status of SHD again, you'll see that the process actually changed to executing its reload behavior, which is what we actually defined in the steps above to create the path unit and the service unit. And also the other interesting thing is that SystemD tells you when it reloads, so you can see down here that SystemD had reloaded SHD. Any questions about this? And then of course you can use SystemD Run to also create an ephemeral path unit like you did with timers and services. So let's just go ahead and copy this one. Instead of path changed, we're using path exist here. What we're gonna do is when we create slash temp slash ping, we will touch slash temp slash pong. Let's go ahead and run this. And we can check that both ping and pong do not exist right now. So if we go ahead and touch ping, we will see that the unit ran and it created pong. Okay, so I have something insane with the path exist behavior. So you would think that when you use path exist, it only runs the service once, like once it creates the file. But if you read the documentation for path exist really closely, what it does is while the file exists, SystemD will keep trying to activate the unit. So if we look at the service here, the service that it created as part of the run, we can see that it actually hit the start limit because while that file was existing, SystemD kept trying to start the unit. At some point it hits the start timeout limit and then it dies and it stops trying. So yeah, that's something you should notice, I guess. Yeah, sorry, it is very useful. You have that gotcha. The other one is that this is using the, what's the name of the thing where it's, I notify, yes. So as you know, I notify it's a best case effort. So you may lose some events of it. So I would not extremely rely on it, but it is very useful for things like let your users add a file into the directory and then you can grab that file and then shove it into another configuration or something like that. We did this for SSH once in the company where Chef would write things into directory and this will basically create an SSH config and reload SSH when it starts. You can see that that is very dangerous and cloney, so don't do it, but yeah, that's what we did. So that's the end of the other unit types that we're gonna go over today. Are there any questions about that? Nope, all right. I'm gonna break or should we keep going? Okay, all right, so the next module we're gonna go over is dependencies and for some reason Alpharil's really excited about this. So dependencies are, I mean, conceptually they seem kind of like easy to understand but in practice they can get pretty complicated. So the, if your members is via NIT scripts, you run the scripts sequentially based on how you number them and there's not really a way to say that, hey, this script depends on this other script except to order them in a certain way based on the file name. With this system D aims to solve that by creating a way to define your dependencies and when system D will try to like graph all the dependencies together, it tries to parallelize the execution based on how the dependencies are kind of linked together and that's how you can speed up, start up and shut down. So instead of running everything sequentially, it aims to parallelize as much as possible. So what does this look like in practice? So let's start with ordering. System D provides two ways for you to order your units with before and after. So for example, we're gonna be talking about the dinosaur and human service. In the dinosaur service, we are actually going to write before human.service so that when these two units run together, we actually have dinosaur.service running before human.service. So we actually provide these units for you already so I don't have to struggle to copy and paste the contents into the terminal anymore. So we'll be able to just run this and see what happens. The thing, the other notable thing here is that on both of these units, we use remain after exit. I don't think we talked about that yet. What this does is it keeps the state around after the service finishes running so that you can examine it once system D has in memory. This is pretty useful if the service successfully exits and you wanna see what happened. So let's go ahead and copy the start command and to show that dinosaur ran before human, we're gonna check the timestamps for both of these units. So we can do system CTL show dash P. We use the active enter timestamp monotonic property and we will first look at the dinosaur.service timestamp and then we'll look at the one for human service. And you will see if you look really closely at the numbers that the dinosaur service had activated before the human service. So yeah, super simple so far, right? Before and after? Okay, now we get into kind of the bread and butter of like the actual dependencies. So 90% of the time I'd say most people only wanna use requires and wants on their units. System D actually provides many more ways to define dependencies, but those really get into like the nitty gritty of how you wanna start, stop and reload on certain units. But most of the time you just wanna say this unit wants this other unit to start or this unit requires this unit to start. So we're gonna test this out with two services, life.service and water.service. On life.service I write that it requires water.service and we also want it to start after water.service. And water.service we're gonna run bin false so what this does is it causes water.service to fail when it runs. And you'll see why I do this once I start life.service. So because I use requires we would kind of expect that life.service will fail because water.service is going to fail, right? So let's start. This is another gotcha here because when you check the status of life.service and also the status of water.service you'll see that life.service actually ran successfully but water.service failed like I said it would. Why is that? Well it's because requires only refers to the act of starting the unit. The default type for every service unit is simple which means that as soon as system defaults the unit is considered started. Which I think can be confusing because people have very specific ideas about when their process is actually started, right? You would think that when we exact and we reach the certain point in our process this is when we want to start. So to do that we can actually use type equals notify. So if we system CTL edit water.service and we change it to type equals notify because in our command we don't tell system D, sorry taking a step back. What type equals notify does is it expects your service somewhere in your code to tell system D when the unit is started and ready to take, actually active. You do that by sending a command, sending a D bus message ready equals one to system D but because we're not doing that here we will actually fail to start the unit. We have a tutorial on that tomorrow. Chambers play. Now if we actually try to start life.service it fails because the water service failed and it had a required dependency and you can see that system D helpfully tells you that dependency failed for life.service. So in the real world this is useful like when you want to start your web server after you start your databases like this is how you can do that. In general this is how system D starts in general. System D says like start everything on the service that it's designed to start and then it use dependencies to map things out. Now we have a quick exercise for you to do. So keeping type equals notify on water.service I want you to modify water.service to send a ready equals one message and I give you a hint here you can try using system D notify to do that. Just take like two minutes to do that and let me know what you come up with. Okay so who has managed to send the ready message and like gotten the dependencies all fixed up? Does anyone want more time to try and do that or do you want me to do it? Ambivalence I will go ahead and show you how I would do it. So I'm gonna override the previous exact start by just changing by adding a blank exact start here and then I will use system D notify. If you looked at the man page you'll see that you can send a dash dash ready. And that's all you really need to do. So if we start life.service again you'll see that it started successfully even with the type equals notify and also water. If you look down and we deactivate it successfully. And to do that in your code you would use like the SD notify API and things. Oh there's a question. Yes. The question was can you show the solution and the answer was yes. Yeah here's the solution once again. And again we have a whole section of notify in general. There's a lot of cool things you can do with it. So stick around for tomorrow. We talked about requires. Now we're gonna talk about wants. So wants is basically the weaker version of requires because what wants says is that we'll make a best effort attempt to start the service but if we don't do it like no big deal we'll just keep moving along with the original service. So we provided two service units here for you. Dog.service and bone.service. Let's see. Bone.service has the same thing we had in the previous section where we use type equals notify and we run bin false so that it purposely fails to start the unit. And in this case when we actually start dog.service it will run successfully because we don't have a hard requirement on bone.service. So let's just go ahead and do that. And if you check the status of dog.service you'll see that it ran successfully and bone.service had actually failed. So yeah. Any questions about the ordering and basic dependencies? Yes, question. Oh, I guess the question was if you define the unit such that you create like a cycle or a deadlock is there a way to detect that with system D? Yeah, yeah. So like two units like depend on each other and if one does like, yeah. Yes, you, so yeah, I mean you can create cycles on purpose and system D is able to detect and warn you about it. I don't know if we'll get to the verify section of the workshop today, but you can use system we analyze verify to detect cycles in your units. And you'll see that the crazy thing is when system D detects cycles on boot or like at a target it'll actually delete the start of certain units in order to get past the cycle. Any other questions? Yes. Oh, does the unit know if it was started by a once or requires? The answer is yes. There's a lot of conditions in which you can basically say because like one of the things that you can start unit just because it's started by requires or wanted but if you stop the main unit you will not stop the previous one, right? But you can also set system D to say like if a user doesn't start this like stop it when you don't need it. You can also say no one can start this unit except another unit by once or require. So you can also like further. So the answer to the question is yes. And you can take advantage of the situation. I'm gonna talk about the install section of the unit. So you saw in the previous examples that we added install so that we can actually enable a unit and most of the time we do that because we want a unit to start at boot. So typically we add this statement here install wanted by multi user target. For most systems multi user target is the default target that kind of signifies like the end of the user space. Like we booted into the system and this is it. There are actually multiple other targets you can do but let's start by modifying. Yeah there are actually multiple other stages of targets that you can do and let's actually do a quick experiment here where we add the multi user target to the dinosaur and human services that we did before. We're gonna go ahead and copy dinosaur service into slash ETC system system and name it a dot service instead. And then for human service we're gonna call it the service. Okay so here's the disclaimer. This is the part that we haven't tried with lean out. So if you're using lean out and now you're gonna reboot your service, your machine. That's how it works and we don't end this story also earlier. I did actually modify and add the install at this point and the point of rebooting here is that we're gonna show that both of the units did not start at boot. This fast in my vagrant machine I don't know how long it'll take your lean out machine to come back up. I hope you try a restart and it takes like a minute or two so. Did anyone do a reboot on their lean out instance and come back successfully? So if we do system CTL on an beta service you'll see that they did not start as expected because there's no install on those statements. So we can go ahead and add the install now. Saving and we also do the same for, so what I did just now is add the install I want to buy multi-user target part that I had above in the GitHub to both of those units and net, oh, don't forget to system CTL enable which creates the sim links on multi-user target because if you don't do that it is not gonna start at boot and if we actually reboot this time you will see that both of the services are gonna start at boot. Any questions about that or anyone need help? The question was what are the constraints between? Oh, what are the differences between adding the install statement and running system CTL enable? So you can only enable units that have the install section. You'll see that if you try to enable a unit that doesn't it'll give you a long log line or warning about it. Basically what the enable does is set up the sim links so that when system D runs the target it knows that these are all the different dependencies or services and stuff that it has to run. So getting to the next part of this we're gonna start talking about synchronization. So I mentioned target units a few times and you might be wondering like what the heck are targets? So targets is another unit type that doesn't allow you to specify anything about it except for like the default unit properties and they exist to synchronize all the different services and stuff at different stages of boot. You can think about them as like this is fee and it run levels. If you actually look at the man page for boot up so just go to like man boot up and scroll down a bit. You'll see that there's a really interesting chart about how all the different targets get executed as part of a host startup or run. And you'll see that at the very end of it the user space boot targets can be either one of rescue target, multi user target, graphical target or emergency target. Graphical is like when you have GNOME, KDE or some kind of graphical desktop environment. If you're on a server you don't really have that so multi user target is where you end up. And you can actually add your own target if you want. So to see what the default target is you just go run system CTL get default. And you'll see that the default target is multi user target. The way the default target is defined is with sim links as with a lot of the other kind of dependency things. So if you look at the sim link for default.target you'll see that it points to the multi user target unit file. So the next part of the workshop I just show how we're gonna create a custom boot target but in the interest of time I'm gonna skip that part now and just move directly into the showing you some examples of like how you can analyze boot with system analyze critical chain and things like that. So if you run system analyze critical chain what it does is gives you kind of a rough very rough time stamp of when each target ran when you booted your host and the order that they actually ran. So in this case we still had multi user target and these are all the things that had run before it along with the time stamp. And if you like pictures and pre things you can actually use system analyze dot pass it a unit name and it'll generate this crazy graph of all the dependencies on your unit. System analyze dot, I don't know what's our favorite service. So you'll see that this spits out the dot format for the dependencies of system d umdi but this is not really useful by itself we are visual creatures so they say that we can actually use this to turn it into an SVG. It's gonna run that again. And if we have an image viewer we can actually look at this crazy graph. Here's the Linux desktop everybody. So there are actually not that many dependencies associated with system d umdi. You can imagine if you do a more complicated service this kind of chart gets crazy and hard to look at but just even like this it's a lot of the default dependencies and not even the ones we specified specifically. And then the other tool is like system analyze plot which I am not gonna do but you can run it on your own time. It will generate an SVG of the entire boot sequence and it's like a multi-page visual of like how the time that all the units started. Any other questions? Okay, we are gonna take a five minute breaks, bathroom breaks, relax breaks. We are present time we are gonna finish here at one because of respect of people who work here and stuff. So keep it five minutes and then we'll get back. Okay folks, don't forget we still have some port of space trees that you can go and take. Come and take sorry, what is your problem? Okay, I assume we are on the right time. So we are gonna finish this strong. We have about 30 minutes to finish. We're gonna do now the beautiful world of journal CTL, right? So as a little bit of introduction besides of all the controversial part of system D system D also decided to replace like Syslog and other the journal CTL. The value study offers to people who manage system is really incredible, right? So you can still like not like it, that's up to you but it is really good to operate on it. So we're gonna do a tutorial on it where we're gonna give you very basic dive into journal CTL. We have been using it a little bit and then in tomorrows which is the advance we're not gonna teach you advance part of journal CTL but its commands are gonna just show up on the tutorial. So it is nice. So we're not further ado. Anita will start this and then I will finish it. Like Alvarez said, system D comes with a daemon for doing logging and it's called system D journal D. The way you configure the system D journal D daemon is with the journal D.com. It's stored under the usual places slash Etsy slash and D and you can look at some of the commonly used properties along with their defaults in the example that we have in the GitHub here. We're not gonna go into configuring any of them today but I'll point out some of the popular ones such as storage which you can set to volatile which will only store logs in memory persistent which will store logs on disk auto which will try and see which directory exists in order to decide if it wants to be volatile or persistent and then none where like we don't store any logs at all we just forward them to wherever you tell them to forward to. There's also a global rate limit that you can set with the rate limit properties. You can also set per service log rate limits so that if a certain service is too noisy you can tell it to shut up faster. And then you can also set the properties on how big your journal should be on the host. Skipping these, there's also the forwarding behavior like I mentioned, you can forward to syslog, K message, console or to wall and they're all pretty self-explanatory. And you can also set the max log level of the messages that go to the journal using the traditional syslog levels or with the integers. So on a per unit level you pretty much only have like six sysh properties to configure how your unit gets their logs into journal. By default all of standard out goes to the journal. You may not necessarily see all the output from standard out in the journal because of how the log levels are set but that's how it is. And standard arrow by default just inherits based on the behavior of standard out. And like I said, you can also set individual log levels in the service unit and also individual per unit rate limits for your logs. So enough about that, let's look at some logs. So if you just run journal CTL in your workshop setup you'll get basically all the logs from the dawn of time apparently or when you last booted. And yeah, just a way for you to page through the logs and things like that. Pretty straightforward. I usually like to do journal CTL R. What it does is it reverses the log so that it shows the most recent log on top because I don't really care about the logs from like two days ago necessarily. I want the logs from right now. So that's what R does. And you can also use dash dash since, give it some relative time stamps in order to be more specific about which logs you wanna look at. So if we do dash dash since, these are logs since one hour ago. In 101 we actually did the last 30 logs. So I'm not gonna run that again here. And you can also use journal CTL to look at journal file that you kind of imported or moved over from a different host. So we provided a journal file for you to look at just as an example. I'm gonna make sure it still exists. You can pass dash dash file and you can use journal CTL to look at basically arbitrary journal files. It's pretty useful if you need to debug something. A user can just sign you your journal file to look at. Okay, okay, so now we can see a little bit how can you as a developer write to the journal, right? So we already saw that your unit by default is under SDD in, sorry, so there are SDD out and SDD error to the journal. We provide a small unit over here and you can go and instead of start first, let's see what it has, right? So as you can see, it's just saying standard journal, hello, and now we can start. And that will, if I do status, you will probably see that, yes, it is right there, perfect. And now if I were to do journal CTL that you, you will see that it kind of shows everything that goes into that unit. You can also see that it shows other stuff that are like from different PODs, but these are all in the context of the unit. Give me a second. The other thing that we didn't say, but we kind of explicitly have been saying is that when you decided to add a unit in your commands like here, you can do the unit with service or without the word service. So without the dot service, it just appended at the end, right, so it's figured out, right? The other thing that we didn't tell is that we have set up a tab competition. So if I do journal and then press tab, it autofill it, so you don't actually need to type the entire thing. This is, this comes part of system D. When you install system D addons, extension, what's the name of the package that you install to get tab competition? Bash complete, oh, I think it's come with system D. And if you install this competition, it comes, yes. It just comes with system D. Okay, and that's moving on. Michael, so that is one of the ways that you can print to the journal, right? So I also provide, sorry, we also provide here as nice script that you can look at it and this is how you can write to the journal with, so if you look at that file over there, you can see that is importing system D journal. We're gonna talk more about system D next, like tomorrow, but that's a Python interface to interact with system D. And basically we're gonna say journal and send the variable and we're gonna send this message. We're gonna set the priority, the message and the syslog identifier for that one. So if I go here and just execute this script, like remember, I'm not even like, technically you're always in a unit by, you can do that by looking at just, like doing system D status on the double pound and that's your own PWAV. You see that I'm literally on a unit, it's just a scope. But yeah, so if I go and execute this nice command and then do grep for hello, you're gonna see that I had a message there from super unique ID and then maybe that's the syslog identifier and then hello from system D journal printing, right? So if you kind of see, that's the identifier that I have. So that's how you can like write to the journal from anywhere that you want. We already saw that we can do this by unit. What we didn't saw is that you can ask for the output to be verbose. So this is also another interesting thing that the journal has, that you don't see it everywhere else. You, like, let me, don't take, so you see like this output is like text like and the thing that you're used to. There is a bunch of metadata under the hood, right? So when I do verbose, you can actually see a bunch of more variables being set. So you can actually, like since you can read this, you can do stuff based on whatever the message was or the login, you see like there's a syslog identifier there, but it also kind of tells you like what was the command, the host name, the message it says, it kind of gives you like the PYD and the command line that was executed and you have the sysgroup and the unit from which this came, right? So there is a lot of information. So you can look, we already saw that you can look for a unit but you can also probably search by sysgroup or any of these filters, I can go. So yes, that's interesting. Oh yeah, we can also do this, journal CTL feed one and this shows, we are searching, like we saw for filters. So we're using everything that comes from P1. As we know, that's the unit system, system D. You can also do this. So look at all the messages that come from a unit but from the PYD, right? So if I delete the PYD, sorry, this is going to be if I, if you see like there's some methods that come from the thing that was exec but system D itself add logs to that, saying like the things that it was doing. So if I just want to see that because sometimes it is useful, you can do PYD one. This doesn't work, right? It doesn't work, right? All right, cool. Yeah, we have some exercises here that you probably can go and look at them. The first one is view the metadata from that file again and you can see what files there was. We're going to leave like one minute or two for people to do this and then we're going to do it ourselves. Oh, shit. Okay, so as probably you already figured this out, you just need to mix everything and then you can just do that and then it will show you everything that was running. It's ask you, okay, so let's take this one. We can basically take that one and then say unit and perfect, the PYD was 333. Since you make this part, am I right? Yes, I got Anita's approval. I got, so any questions so far? No, okay, great. So let's do the last things, right? General CTL, if you do this Mac, you get the message from the Deepak kernel. You basically get everything that you used to get with that one. General CTL list boots shows you the boots of the machine and finally, if this one is not a good example because it only has one boot but if you do with that three, it will show you the logs for the current boot. That's the journal introduction. Go ahead, you can mix and match. Like for instance, one thing that you can do is that like this one, you see has like file solution but you can also add another unit, right? I don't know if systemd, umd, yeah. So like, and it's fine, I'm just gonna show you the journal, probably umd wasn't running on this one. But example, let's do it here though. There's journal, right? Just press U, you said, okay. Yeah, all right, believe me, it's somewhere there. That's the, yeah, that's the one. Okay, go ahead. So the question was, sometimes you do systemd cut and it looks as echo. Yeah. So there is a bunch of things in play there when you're logging to syslog. The first thing is that systemd cut and some things try to log as the main piece of the process because like it's assuming that systemd cut is not the one that is doing the log. So it provides their information to say like, oh, systemd cut is the one, by the way, this is what we're talking. So yeah, so systemd is, so it's, this is gonna go and show and try to put stuff into the journal, right? But it makes no interest to say like, oh, the systemd cut, the one that is showing. So systemd cut tries to operate as the parent piece. If you are running as a user, like niche, by the way, niche, the, whatever you start doesn't have permission to execute as another thing. But if you're running as root, you can. So systemd cut will ignore itself and then will appear as the other. This is very common when you do systemd notify, which is the thing that we're gonna look tomorrow. But yeah, so there's a lot in place. The other thing that has to be in place is that by the time journald sees the message, the PYD that sends the message has to exist. So there's a race condition there where you cannot have attribution on the logs. It is, it's a problem. Yeah, yeah. So yeah, so what Nish was saying is that he basically tried to do a denial of service on his journal CTL and some of those messages got misattributed. Yes, this is something that happened. And I think we are done for the day. So who here has learned something new today? That is great. We're very grateful. We really appreciate that you guys came here. Thank you very much. And have a great rest of your day for the conference and see you all tomorrow morning at same time, same place. Okay, so we have 15 minutes and we have one more example. Do you guys want to see it or it's time to go home? Okay, let's try. All right, let's, all right. Disclaimer, we did this for Fedora 37, but let's see if it works. Okay, so the example is the following. Go into, so this is the one that we're gonna do. It's the workshop, workshop, analyze system, the analyze engine X. So we're gonna do a thing that is called system, the analyze security. If you type this into your terminal, what this is doing, it's giving you an opinionated, an opinionated by system D of how secure your services are, right? So the definition that system D has for secure has a lot to do on, let me take a step back. So in the old days, you start your service like you start an application and this application or this service that is running started with the same privilege as you and here it's everything that you could do, right? It has the same view of the file system. So if you can write into root, the service can write into root and all of those things, right? Today, this is not necessary, right? So if you're having, if you have an application that all it needs to do is check if you have unfinished downloads. There's no reason why it have to be able to write into Barlin, right? So you can actually hide a lot of the system, right? Maybe your application doesn't need to even see home directories because like, why would you get access to your SSH keys to a random service? So you can actually hide them. So all of the things that system D provides for you, if you don't enable them, they called your system unsecure, right? Again, it's opinionated. Let's pass that, let's get passed that. But you see they have some services here and we have a lot of them and they have some data in them. So we're gonna try to do one with NGINX, right? So let's do system D analyze security NGINX and it's gonna show you all the things that it thinks are broken for NGINX, right? So it shows you all the properties that you can set that would change a system D, the NGINX behavior. And it has something that, oh, this is great. It has private TMP, right? So that means that TMP itself, it's mounted as a RAMFS. So NGINX doesn't have access to your host system D. TMP just has like a shim version of it. So if one applications store stuff there, NGINX cannot see it. That's kind of like the whole idea, right? So before we start, let's try to start system CTL NGINX and you're gonna see that it fails, right? And it kind of tells you, hey, execute this command to see why it failed. And it says, it says very not useful information. It says that it failed it to start because it failed it to open bar log NGINX error log, right? So what we're gonna do is that we're gonna copy the NGINX config, the previous NGINX config, and then we are going to edit that config to do in line seven, line seven, we're gonna change that error log. Sorry, line seven, we're gonna change this to be that. Basically, we're saying instead of like storing logs into a directory because we actually want to like avoid this NGINX to write into stuff, we're gonna say just send it to syslog server, which is just journal D. And then we're gonna do the same thing for the access log that is on line 22. We are also, sorry, we're also going to edit this NGINX unit, and we're gonna change the log directory to, this is another unit, this is another setting that is just gonna set a unit, a directory for the logs. Okay, and now we can start NGINX, finally start, and then we can do a curl to port 8080. And if it returns okay, it's like, we got NGINX to work, that was not part of the tutorial, that was just like cloudiness that it comes with. Okay, cool. So now let's try to improve a little bit of NGINX. So we're gonna edit the NGINX, and now we are gonna add these following settings. These settings are actually kind of cool, what they make, and this should be on for most things. First of all, it makes private devices, that means that everything that is in your bar log, that has to do with physical hardware, sorry, bar log, depth, or depth, that has to do with physical hardware, it's not allowed for the unit to see it. Like for instance, there's no reason why NGINX has to write directly into your CD room, or your hard drive. So we're just gonna say you're not even gonna see NGINX SD1 or NGINX SD2, you don't need that. Private TMP, I already announced this, it basically, for TMP, it mounts a TempFest, that nothing has access to it, and only NGINX has access to it, so there is no cross-contamination. Protect Home makes Home, I think read-only, I don't remember what through this, but there's options where you can make it hide home, the other one is just make it read-only so that nobody can write it. And this one are other options that has to do with kernel, like NGINX is not gonna load a kernel model, so why even, I'll give it the chance that it could do it? So all of this, right? So we save it, we restart it, we check that we can still query, and yeah, we can still query, so that exists. So let's see if we can analyze NGINX again, and we go to the end, and we now see that the overall exposure went up, right? That's not the exposure, but the number went up. Now we can do a couple of the fun ones, right? So we can see if there is a socket, so there is no NGINX socket file, and I really just realized that we haven't actually looked at what socket files are, but we can go and create a socket file, I have to say force. Yes, so with these options, we will create a socket file, and then we can add this content. What this is gonna do, it is very simple. So NGINX start as root, because it needs to open port 80. So instead of doing that, what we're gonna do is that, sorry, NGINX start as root open port 80, and then download itself to whatever user you tell it to run. But we don't need to do that. What we can do is just let systemD start listening to port 80, and then with NGINX start, systemD will just handle that file descriptor to NGINX, so that way NGINX never even have to be root, right? So this is what we're doing right now. We're creating an NGINX socket, we're setting listening string to port 80, we are exiting BIM, and then we are going to edit the NGINX, and we are gonna add the following options. For the unit, we're gonna do what Anita just told us to do, which is set up after on requires, and then we are gonna add the following things to the service, we're gonna say the type is simple, we're gonna say that please don't rely on PWAV files, we're gonna set an environmental variable, this is the one that tells NGINX to read the sockets from file descriptor three and four, and we're gonna say private network, so that NGINX cannot access other things other than this port, right? So NGINX itself cannot open port, so if somebody hacks NGINX and can execute stuff, they cannot start listening to two other that is not 80. So that looks great. We are gonna stop NGINX socket and service just in case, and then remember when Anita was showing you that you can start the timer and then the timer will start the unit or the path, this is the same, right? So we're gonna start the socket, not the service, you can start the service, but we're gonna start the socket. Now, if I do status on the socket, it says, oh great, it is running, right? It's listening, all the information is nice. What if I say the service? The service right now you say that it's loaded, but it's not running. It's gonna run the first time that somebody hits port 80, it's the same as the timer, right? So when timer gets to a time, it execute the service, the socket, when somebody hits the socket, it's gonna start the service. So if I had eyesight, I could have done it, then I could paste this, and then yes. And then as you can see, it took a little bit more the first time, and this is of course because it started the service. You don't have to do it like this, but it's kind of useful because up until you actually require the service, you're not wasting resources on it. Question. Yes, so the question was can system be, or is there a way of preload the service before somebody asks it? And the answer is yes. You just start the service on the socket at the same time. So where's the stop? Yeah, so you can actually start. Oh, yes, so you can start it, and then the status of the service is up. So now my, whatever I do by curl should also be instantaneous, right? But it's still going through the socket, right? And if I understood Benita's tutorial correctly, I should be able to just start the service. And yeah, and by just the sheer magic of dependencies to start the service, system is so that you actually want the socket before starting the service. The beauty of the socket, it is like if you're not latency bound, you are saving a lot of resources by not having something running. Okay, so this was it. Okay, and I want you to see that we're doing all of this because we want to, there you go. We want to minimize that number here. We want, this is the one that we want to improve, right? We're already improving a little bit. So let's do, let's add more stuff. So now that we actually don't need to be root, we can just, now that we don't need to be root, we can just set these variables here, and what they all do is very simple. We're saying, yeah, just start as nginx and group nginx, don't even think about being root. Like we, the only capability that you have is that you can bind to the network. And this here over here says, like systemd will create some directories for you that you can use that you don't need to create. Like bar run nginx, like nginx doesn't need to create this directory. It's already there by systemd. The same with the state, barlib, barcache, and etc, right, for the configuration. So now nginx doesn't need to be able to read or write these, sorry, to write these directories. So to create, okay, we save it. Now we can go and remove the user nginx from the, here there is a line that says that the user's nginx, we can go and remove that line. Yes, that's great. And we can also remove the line pyd, this one over there. We can go and delete the locks directories. We no longer need it. We can stop. We can start the socket. And then we can do analyze. And now we have a happy face because systemd really looks images, right? So the security just went through to forward eight. Yes, that's still the case. And now if you really, really, really want to overkill it, we have like more options that you can go down. Again, we're two minutes. And I think that takes us to 4.0, right? So this is 4.8. If we do this, it will take us for 4.0, but how much secure you wanted to have it, that's up to you. But the point is that you get all this feedback loop of you add stuff, what it's doing. Keep in mind that this is one definition of security. It doesn't really have to be the one. But as long as you can still curl your service, and it, okay, that wasn't working. Let's assume it was. All right, but as soon as you can get your service to run, that's what you want. And on time, so thank you, Aaron. Howdy, can everyone hear me good? Excellent. Hello, hello, hello. Okay, give it a second. Hello, hello. Ooh. Yeah, we can kind of stash in. Oh, wouldn't you care about something here? No, I mean, we've got, yeah, we've got sessions. Yeah, yeah. But that's when you've got sessions that need to stick inside of there. Yeah, we'll all be able to do that. Yeah, we'll all be able to do that. Yeah, we'll be able to do that. Send out the viewers if they want to. Hi, everyone. Thank you all for coming, for coming to scale. I always enjoy being at scale and for coming to this. This is a new kind of format we've been experimenting with to do kind of an overview of CentOS and then a workshop so you can get some real hands-on experience with packaging and doing stuff in CentOS. So we've been wanting to do this. We've been wanting to do a larger presence at scale for a while. So very excited to be here and doing this. So what is CentOS? Now I'll have more slides on what is CentOS. But we have a whole ecosystem. What you might refer to as just CentOS, is CentOS Stream, that's the operating system. That we'll talk about more. And that's kind of in the center of everything. But then we have all of these special interest groups around it. And you can see that I'm gonna get a graphic designer to actually make this look pretty at some point, but I made this so it looks terrible. You can see all of our, well, that's not even all of them. There are more special interest groups than that, but that's some of the more active ones that do various things, build different package sets, do alternative packages sets, other images and stuff around CentOS. Oh hi, my name is Sean. I am the CentOS community architect. And here I'll let Carl introduce himself. How do you all? My name is Carl George. And I'm participating in the CentOS project, but also focus a lot on the Apple project, which we'll talk about on another slide. I think the logo's up there right now. And also I'm not on camera frame, like I was just complaining about. Yeah, Apple, this little logo here at the bottom. It's not part of the CentOS project, technically. It's part of the Fedora project, but very interrelated. And I'll talk about that more on the slide where that's appropriate. But yes, hi, I'm Carl. Yeah, so I will also get a graphic designer to make it. I wanted Apple on there because they're within the ecosystem, but they're not technically a CentOS sig. So let's talk about CentOS Stream. CentOS Stream has a five year lifecycle, a five year and some months lifecycle. There's a new major version every three years. And it is where you get enterprise Linux from. It is where REL is derived from and any REL derivative systems ultimately come from CentOS Stream. It branches, I'm just speaking and not following the slides. It branches off of Fedora. So every three years we take a Fedora release and we make a branch point from that. And this year actually will be one of those branch years. So Fedora 40 will be coming out in a month or so. Like really soon, right? So Fedora 40. Oh, okay. I will hold that thought. Okay, so it's a long lifecycle. We have a five something year lifecycle for CentOS. So there's about a two year overlap between when one version of CentOS, the new one comes out and the old one is still there. It is not a rolling release, even though that word has floated around even on some of our own blogs early on. I think people use the word rolling release to try to indicate it doesn't have minor versions. There's no CentOS 9.1, 9.2. It just has CentOS 9. But that's not what people think of when you say rolling release. When you say rolling release, people think of something like Fedora raw hide or opens is a tumbleweed, right? Where there is no major release. It's just constantly getting updates, right? And that's not what happens in CentOS. In CentOS we have a major release and it just gets updates within that major release which is effectively the same kind of release cadence that you get on Fedora or most other distributions. They have a major release and then they get stable updates onto it. And the updates that come onto CentOS Stream are stable updates. These are generally updates that you've already seen in Fedora, either the update itself or it might be a back port of something that's already in a more current version, right? We maintain sometimes older versions of packages within CentOS Stream so we have to do back ports. But the features and the fixes that you see coming in are things that have already, that you've usually already seen in Fedora and they've passed a bunch of gating checks and everything so they're not experimental new features. They are stable updates. I'll interject there. The way I describe it, I might have put it on the slide before maybe. Yes, that last bullet point. I would call them rail style updates where if you're familiar with how that works we do a thing called in rail called back porting where the version of the software stays the same but a bug fix or a security fix or something will get added as a patch to that older version. So you're like the software might be version 1.0 but then there's a security fix that came out in 1.2 but there's also a breaking change in 1.2. We want the security fix without the breaking change so we take just the changes we need for the security fix and back port them basically making a soft fork of the software. If the software ever got rebased like it does in Fedora raw hide you could drop those patches but it typically, I don't wanna say typically there are some packages that get rebased in CentOS Stream. I know that the Rust stack for example moves very fast and that gets rebased in every real minor version so that'll get rebased in CentOS Stream first a few months ahead of time but for most packages there's a strong preference to back porting things and keeping the same version for stability versus rebasing to a new version. And the kernel is like we stay on I think we're on the 514 kernel. So we have 514 but it's like revision number 200 or something like that's how many back ports get put onto it. Yeah, let's do, gonna run the mic to them. Test, test, that's sensitive. Yeah, my question was you don't update like the package version, right? But the only way you'd know the back port was done is to either look at the change log or to look at like the sources for the patch in like from the spec file list, right? Like looking at the spec file or? Yeah, I can answer a little bit of that. Yes, that's sensitive, I'm just really loud. Definitely the change log. Also, if you stay for the whole session we're doing an RPM packaging workshop where I'm gonna get into that in depth detail. It's the field called the release. If you've ever seen the string for the package it's like 1.0-5 that what I usually tell people is the first part before the hyphen is the upstream version and the other number is the package version. So like the revision of the package. So the first one would be dash one and then you need to add a patch and then it's dash two and then you realize you forgot to ship a file that's supposed to be shipped with the software then you do that and dash three but it's all still version 1.0 of the upstream software is where you're starting from. I think it's a little bit misleading when like you mentioned in the kernel where you're on like dash 307 and because there's so many patches added the upstream version is less and less relevant when you patch to that degree but in a long life cycle distribution like CentOS stream or rel that it's unavoidable because you need to keep a lot of those versions the same for a long period of time. That covered? Definitely jump in with questions and I'll come out to you with the mic. Is the order Fedora first then CentOS then Red Hat? Next two slides. Do we have that on the, that one shows. Okay, a couple slides ahead. Put a pin in that and yes. I have a sticker actually I have here. Talk about it. I'll go ahead and talk about since we had a question. We don't, it doesn't really matter what order we talk about these in. So this right here, can y'all see that laser pointer on the screen at all? Good. So historic, a lot of people come up and ask I thought CentOS came after rel and I thought it was a rebuild of rel and traditionally that was true. In the past we had the Fedora project a version of Fedora would get taken basically taken internally private inside Red Hat and worked on for a year plus and then all of a sudden kind of throw it over the wall open source that many engineers aren't a fan of because it's, you're technically going from an open source project to close development and then dump it over the wall open source. Like people that care about open source know that that's flawed and don't like that but that's how rel development worked in the past. It was all private development and then thrown over the wall. Ta-da, it's open source but you can't really do anything with it other than use it. The CentOS project will come along and use those sources to create a what we would call a clone or rebuild of rel and I'm gonna go back a slide to talk about that one but that's not how it works anymore. The new model that is a lot more sustainable and more interesting for people that care about open source is that we have Fedora and then it branches into CentOS. There's a period where it's like the early development and you probably don't wanna start using it day to day yet and then at some point we actually announced it as here announcing CentOS string 10 and then about six months after that that's when you see rel 10 which is, yeah, so here's how it worked in nine a little bit more granular view. Fedora Rawhide is the true rolling release in this ecosystem and the dashed lines are gonna be like development so you'll have, not right now but Fedora 34 branched off from Rawhide got a little bit more polish and then released as Fedora 34 maintained for a set amount of time. At the around the same time we branched off CentOS 9 development worked on that for a little while getting more polish and then at a certain point rail nine beta branched off of that and then we did the CentOS stream nine announcement and then a little while after that rail 9.0, 9.1 and so on branched off from CentOS stream nine development or actual CentOS stream nine. Things change a lot between here and here we're getting things stable for the whole life cycle which the rail engineers controlling those packages they have to think about it not just what we're putting in the next version of CentOS but what is the business gonna have to support for 12, 14 years depending on the extended stuff that they sell on that side of the house. And that's what it's gonna look like for 10 and this is, we are actually right about here right now we're in the last bit of the Fedora 40 development thing we're about to break off the sync we've already got the first initial composes to make sure everything works for CentOS stream 10 but they're nowhere near the point where you should start using them. Once, once Fedora 40 is released we're gonna interrupt that sync we're just using that to bootstrap things and then independent development will happen for 10 things like deciding exactly what version of the kernel they're gonna use and exactly what version of OpenSSL is gonna be used things like that that all is gonna happen right around in here but once we actually announce it as it's here it's ready, you know no one's ever gonna say it's ready but we'll announce it and that'll be a good time to get on board and start using it and that's around the time of the five and a half years we mentioned. So going back to where we were before the great questions, keep them coming. Before I go on, was there any more questions about that kind of life cycle interaction in the branching? I know it's a lot of information dense can CentOS stream be used in a production environment? Totally. Absolutely. Many of us do use it that way. There's been a lot of, I'll just call it a lot of fun around that. A lot of people try to say like no you can't use it now it's too unstable, it's this totally different thing but the reality is if you actually look at it the difference between RHEL and CentOS at any point is like 10% like it's 90 to 95% the same software versions may have a few extra patches things that are as you can see in the diagram we're a little bit further back in time here but what happens here in CentOS is what's being planned to go into RHEL 9.2. So it's not just experimental we have that on the not slide here. Yeah it's not bleeding edge, it's not experimental it's RHEL style updates, it's things that the business people want to put into the next RHEL minor version and breaking compatibility would literally break contracts with customers so that's not what's gonna happen not that mistakes can't ever happen they certainly can but it's not just throw this against the wall and see what sticks there's QA and stuff happens before it goes into the CentOS release so I think it's great talking about using it in production you get bug fixes before RHEL gets them and a lot of security fixes as well so I think it works a lot better in that use case. One thing I will point out to people is that the traditional CentOS users that would use CentOS in like the first five years of the lifecycle that care about those new features coming in the minor versions are a lot better they like CentOS Stream the new model a lot better than the people that would use CentOS in the years five through 10 where they want it to not change at all there's no more minor versions those people and they want the 10 year lifecycle as well which CentOS doesn't have anymore so those kind of people there's gonna be a preference to something like RHEL or something that tries to copy RHEL but CentOS for people that were using it in the first five years it's still a great operating system still the same major version very similar software and works most of the same way it's very functionally it's basically the same it's just the development works a little bit different speaking of that development do you want to talk about contributions? Yeah sure and I'll add to that I think different people have different production environments right so for some people like the those minor release branches are very important and for some people they're not like if you're the kind of person who would who would have gone straight from nine four to nine five as soon as it comes out then CentOS Stream is you know perfectly fine and if you're the kind of person that would redo your system after five years which a lot of people do but some people want these 10 year life cycles minor stable minor branches or whatever ooh fancy so it just some of it depends on what your needs are in a production environment okay so CentOS Stream I think we kind of actually address a lot of this a little bit we've hit around it danced around it but some of the issues that we had with CentOS as a rebuild project as a clone is that you couldn't actually do anything right it was people like to say it was bug for bug compatible with RHEL and so if you were to file a bug against CentOS the rebuild saying you know this package is broken or my application doesn't work in this environment or something if you could reproduce that bug in RHEL well then it's not a bug in CentOS right you need to go talk to Red Hat and get it fixed in RHEL first so with CentOS Stream we're actually able to fix bugs in CentOS as they're happening and in fact that's how RHEL bugs are fixed like if you file if you're a RHEL customer and you have an issue you probably have an account wrap or whatever I don't work on the RHEL side of the house so but you would talk to them but it's all gonna go through CentOS Stream me neither y'all aren't talking to sales people yeah I'm not a RHEL sales people I'm a CentOS promoter so and as someone and I felt that pain a lot in the project in the early pre-stream model and everything I felt that pain I would file a bug with the project and then run headlong into like yes we see the problem we reproduced it on RHEL so we're gonna close this bug because our goal is to be bug for bug compatible that is a frustrating experience from the outside from the contributor, the reporter, whatever it's also a frustrating experience for the maintainers on the inside which later I became for a while for a few years I was working directly on CentOS at work and it's just a frustrating thing all around where you're just okay you can use it 100% as is you can't really you can do your own changes to it but you know you can't get anything into the distribution for the greater good for everyone else which that's kind of we kind of we fixed the glitch in a way now moving upstream of RHEL CentOS can accept those contributions the RHEL maintainers now maintain their packages in CentOS also and to fix it in the next version of RHEL the next minor version they work in CentOS do that everyone using CentOS gets that change first there and then it shows up for RHEL customers later RHEL maintainers also have the option to do stuff in older minor releases but there's a lot more red tape for that and that's frankly it's off top for this but it is sort of tangentially related Oh, I have this as a sticker sheet I'll put it out with the cookies later I'll pass them out Oh, go back real quick? Yep No, that's a clicker for the size of it Oh One more Yeah, not just the fixing the bugs ourselves you can fix the bugs you can actually do things if you could report a bug and also say I found this flaw I identified the issue upstream here's the commit that fixes it and here's a pull request to add it as a patch to the package and you can do all of that end to end and the RHEL maintainer can just come along and say yep, this all checks out and we should fix this we should fix this in RHEL and in CentOS, merge and then they do their work to get it in the product optionally into the older minor versions also but CentOS maintainers who are now RHEL maintainers are unable to actually merge those contributions from the wider community rather than just closing down stuff as you can't change it because we have to stay the same I want to pass these up Cool So that's CentOS Stream which is, as I said kind of the core operating system that everything else is built around but I want to talk more about kind of the rest of the project because CentOS Stream is a good operating system it has a lot of good uses but I think some of the more exciting stuff happen with what we do elsewhere within the project and so first of all we have actually a board and the board is to set strategic direction for the CentOS project it is not supposed to be a technical governing board except it keeps getting kind of bogged down in some technical things because we don't have other ways to deal with them which I'll address in a second So it's 10 members they are kind of self-appointed whenever there's an opening we have an open nomination process from the community and then the board votes on who will fill that we actually just went through this process we had a board member step down and we just went through the process of appointing a new one so board members are able to serve for as long as they're able to serve people come and go and get busy with other things in life so all of the board members except one serve as individuals regardless of who they're employed by so if you're on the board as a Red Hat employee you happen to be a Red Hat employee but you are supposed to be on there speaking for yourself and not speaking for your employer and we have people on the board as well who are not Red Hat employees the one however is Brian Exelbeard who is the Red Hat liaison and his job is actually to act as the voice of Red Hat to the project so any of the other Red Hat employees on the board are not speaking for Red Hat it's the voice of Red Hat Brian with his glorious beard speaks for Red Hat but he doesn't have a magic veto power the whole board operates on a consensus model so I mentioned that the board keeps getting bogged down and like these technical issues and so a proposal that's going to go through soon and it's been through a few iterations here is a SIG council and this is basically a way for all of our special interest groups to speak to infra to speak to the stream engineering team and to have their voice heard and to talk amongst each other is to say oh we've been having this problem in our SIG or this problem over here and to work together to solve some of their common issues and this would be made up of a chair from each of the SIGs as well as people from stream engineering and rail engineering and they'd have regular meetings to kind of work through everything so as I said it's an open issue but I want to bring it up because it's I think actually a really important development in the project giving us more of a technical board allowing the board of directors to be a non-technical strategic thing hey Carl do you want to talk about Apple Apple is not part of CentOS but I'm going to let Carl talk anyway sure so I think I mentioned this earlier on in the in the intro about working on Apple Apple I'm not saying Apple a lot of people think that with my hiccups and I'm saying Apple like the hardware like the iPhone but no Apple or Apple extra it stands for extra packages for enterprise Linux it is part of the Fedora project not technically part of the CentOS project but all related and friends and directly works with CentOS and I'll talk about that in a minute well right here on the last bullet Apple basically is Fedora packages that have been built to target CentOS Stream and REL which is a little easier to explain with this little diagram I made these are little boxes to represent packages it is not drawn to scale there's way more than like 100 packages in Fedora like I think like 60,000 last time I looked at it but around I think like 6,000 of them or so well basically the slide earlier where I talked about branching off from Fedora that's not all of Fedora it's a subset of packages basically the things that Red Hat wants to put into REL and support for the business for customers that subset goes into first CentOS and then into REL everything else in Fedora is eligible to be built in the Apple repo and it basically just shows up like another Fedora release if you're a Fedora package maintainer I see a couple of them in here that I know you have your package and you have like the Fedora hide branch at the front and then Fedora 40 is the one we're working on right now that's its own branch there's a Fedora 39 branch and so on Apple just shows up for the maintainer as another branch of Fedora even though it's not but it is so that's how that works it's a way for you to maintain a package in Fedora and then say I use this on Fedora on my desktop but I also want to use it on my servers so I'll request an Apple branch build it ship it there and then it's built to be compatible with REL and CentOS there's actually a really cool talk I would, I'm not gonna take up too much time in this because this presentation is not about Apple specifically but the CentOS Connect we did at Fosdum a month ago I have a talk that's on the YouTube channel about that where we're talking about the changes we're gonna make for Apple 10 and if you've used Apple before or heard of it or interested in using any Fedora packages on REL or CentOS I would encourage you to go look at that we're gonna do some interesting things because of a few different technical challenges we created a thing called Apple Next to work with CentOS Dream but we're gonna flip that model on its head in Apple 10 and Apple is going to immediately target CentOS 10 and then we're gonna have minor version Apple branches to target the REL releases so that way CentOS is a first class citizen for Apple packages rather than an optional afterthought that you might have to build for so we're gonna integrate it deeper and make it work a lot better I think but yeah, look on the CentOS channel on YouTube for that would an Apple 10 roadmap or some title like that? Yeah, I'm like that. Apple 10 overview, that's the title so. Is it meant to be a supplement to RPM and YUM or just specialized for packages? So Apple is RPM packages, 100% supplement, I would still say it is a supplement because they're designed by in the name extra packages, they're additional RPM packages for REL and CentOS things that aren't in the base distribution the first example that comes to mind my mind isn't a valid example anymore but in the older days, I think REL 7 the only web server in the main distribution was Apache, the Apache HTTBD server Nginx was not in the base distribution and you could get Nginx from Apple that changed at eight they added Nginx into the main distribution so it's not a good example anymore KDE is a good example GNOME is the only desktop offered in the base distribution but the Fedora KDE SIG also maintains KDE in Apple for REL and CentOS users that's a great example, old Nome Yes, does that answer your question? Apple will take on the status of primary and the RPM and YUM will go away or are all three gonna be active? Not as long as I'm working on it I am an RPM packaging nerd I like RPM Apple has always been about RPM packages a lot would have to change for Apple to not be additional RPM packages like so much that I don't think that's even feasible I know that there's a lot of move towards what people call like the atomic stuff from Fedora, the immutability but those are still built out of RPMs and then there's RPM layering and other things like that RPMs not going anywhere and Apple only has RPM packages Any other questions? Damn you For the RPM packages that you're talking about are you also going to include modules in Apple for 9 and 10 or is that not a I already know the answer to this Jim Ringer So the question was about modules If you might have seen it before in Fedora modularity You came in at just the right time, Neil Oh, I know Modularity Oh my God, we're talking about that? Yeah, so it was Jim that asked the question so he's trolling us lightly So there was another technology it was still based around RPM the idea was that you could have parallel versions of RPMs in the same repository and through a DNF command configure which set of those RPMs you were going to look at kind of like namespaces on a repository The idea, I love the idea the implementation left a lot to be desired It didn't work very well We actually did have an Apple 8 modular repo for a while and but because of infrastructure problems we had within and other things and low usage we eventually decided to just shut it down and whatever, turn it off and so we don't have that anymore we didn't do it for nine ever and we don't have any plans to do it in 10 and sneak peak while rail 8 has a lot of modules rail 9 has fewer and rail 10 won't have any so that is going away that's an old technology don't really need to know that much about it other than people that have been inside the project for a while you can mention it to them to make them cringe Any other questions related to Apple? All right All right, so I mentioned special interest groups and I'm just going to run through a few of them I'm not going to give a status report on all of them They are all supposed to give a quarterly update we do a monthly-ish newsletter so every month is roughly a third of the SIGs giving a quarterly update so you can just watch the CentOS blog for those and you'll see what all the SIGs are doing but I'm just going to just a few of the more active SIGs there is a new integration SIG so I mentioned everything everything that we do all the updates that go into CentOS are tested they pass a bunch of gating checks and that ensures that CentOS Stream itself is never broken the operating system but you don't run an operating system just to run an operating system you run an operating system to run your applications and so, yeah, right and so the integration SIG has been working on doing further tests to test the integration of the operating system with various applications to make sure to make sure other people's stuff is not breaking with any updates even if the updates are non-breaking sometimes maybe, you know your application is depending on some obscure feature I'm pretty sure there's a next KCD about this so if you have critical applications yeah, if you have critical applications running on CentOS Stream you can work with the integration SIG to ensure that things keep working for you the way I'll describe that is rather than coming along after the fact and saying hey, you broke my thing because you changed, you know this maintainer changed a thing that they thought didn't affect anything in CentOS and RHEL and it technically didn't but they were building on it in a way that caused them to be affected you can actually build in your own tests into that test suite and say that, hey don't release an update for this package again unless my use case works so it can no longer just be you're getting it as is you can actually have a little bit of ownership and make it your operating system too and say, the way this works here is important to me please don't release stuff that breaks that and I've got just some of the recent accomplishments I'm not going to do a readout of these but I asked some of the SIGs to provide some of this stuff hey, let's talk about the high for scale SIG hey, Neal oh no, okay Michelle that's right yeah that's okay yeah, it's okay so how do I put this thing on? you want the handle? yeah, I want the handle I don't want to figure out how to put this on all right, so this is where I need to stand? yes okay I hope so too so the hyperscale SIG did not expect that we do stuff around I'm one of the leads for the CentOS hyperscale SIG we do stuff around enabling CentOS stream deployments and large scale things like whether it's huge fleets of desktops and laptops or even larger fleets of servers and cloudy things and containers and all the whole things as far as that we've done a whole lot of things in both CentOS Stream itself contributing to the operating system again, as Carl mentioned earlier it's about making it your own operating system not just by adding tests to qualify that your stuff works the way it's expected but also to change the stuff that's in CentOS Stream to benefit you and everyone else so we've done some stuff here if you're familiar in the Linux space about how Linux audio is actually shot up to be the best in the world that's pipe wire and wire plumber and we did that along with making it pro audio work out of the box in CentOS Stream 9 and of course, RHEL 9 and with SIG content we do things like provide back ports of various packages including notably system D we provide Fedora based kernel to add additional features that people wanna do in the desktop and servers, container images, expanded virtualization and cloud images and all this other stuff all in service of being able to support our goal of providing people a high quality experience to being able to use CentOS how they need to be able to use it or want to be able to use it but I would say so thanks, I love talking about the hyperscale SIG because it's basically entirely non-Red Hat yes, well the one person okay, but they're just like okay, we could take CentOS Stream as a base and make something else that we want because sometimes there's stuff that maybe we don't wanna put in CentOS Stream but you could make a very different system in hyperscale SIG and then some stems sometimes some of those things do end up in CentOS Stream, which is nice there's a Kmod SIG which packages a bunch of kernel modules again for stuff that maybe we don't wanna put in CentOS Stream for support reasons or whatever but they package up a bunch of kernel modules for things like different file systems or supporting various pieces of maybe obscure hardware but it's not obscure if you need it right so they build a bunch of kernel modules and they build them for both CentOS Stream and they build it for REL and those REL ones should be okay for any of the kind of REL derivative systems out there the alternative images SIG is another one of our new ones so we have certain images that we produce operating system images you can download and install the alternative images SIG has sprung up to kind of create other images you might wanna download so right now what they've made are live images they have a live image for Nome and then one for am I handing it back to oh we're taking a picture Neil is also involved in the alternative images SIG so but there's a Nome and a K2E live image so if you wanna run a live image there are images for that we'll have words afterwards but they're working on a lot of other images too so you could imagine say images around that already have certain other special interest group content enabled right off the bat so you don't have to kinda do that like the hyperscale stuff so you can get maybe a hyperscale image instead of doing CentOS and then adding hyperscale I see a few new faces did anyone not get the sticker sheet that would like one I'll come bring you one okay yeah we'll have them by the cookies at the end did Jim give it to Jim? I did I chased him down to give him one okay good all right almost done there's an ISA SIG this is also a fairly new one and this SIG is investigating new CPU features so the x86-64 architecture gets new instructions and stuff from time to time all of the CentOS stream packages right now are built on the V2 baseline and the ISA SIG is currently investigating basically building packages with a V3 baseline which allows it to use additional instructions in the x86-64 instruction set which potentially can mean packages that perform better and part of what they're doing is like okay let's build these packages with this different baseline and then run it through tests to see is there a better performance and is it worth switching to that by default which could maybe leave off some older systems sneak preview I haven't seen anything publicly posted about those results yet but talking to other people I've heard chatter that at this point using V3 as the baseline for 10 is probably gonna happen like absolute like a really big blocker that's like this particular customer is completely broken with that and needs us to be on V2 it's probably gonna be V3 for 10 sneak preview don't tell your friends are we recorded? we have an automotive SIG this is very cool so one of the things Red Hat is working on is an in-vehicle operating system and they do all of their work upstream in CentOS and so we have what's called auto SD which is a, it's basically a distribution of CentOS Stream but tuned for these automotive applications and that is the work that would feed into Red Hat's vehicle operating system but it's something that you can play around with too just in the community space and try to make changes too so oh, they gave me a diagram over there accomplishments I'm not gonna do a read out if you can see it. There is a cloud SIG they're involved with building both the stuff in the open stack and the open shift ecosystems so the RDO distribution of open stack gets built within there the OKD upstream of open shift which is a Kubernetes distribution gets built there and they also do, they build ASCAS which is core OS is a Fedora image based system but ASCAS is basically core OS but built from CentOS Stream. So one little parallel there I wanna point out for the automotive SIG what they all tell you a lot of times is that auto SD, CentOS Stream automotive edition or whatever you wanna call it is to CentOS as Red Hat in vehicle operating system is to rail so there's a parallel though there and that is similar to, we don't have the diagram here but that ASCAS thing, Stream core OS is to CentOS Stream as Red Hat core OS is to rail so it's the different variants of it essentially. Minor detail. And then this is the last one these are, I put them all on one slide because these are three like non-technical SIGs all the other ones I told you about were like doing testing and packaging and building images and these are three SIGs that do non-technical work which is a really great way to get involved if you're not a packager but you like to use CentOS and you wanna get involved you can get involved in one of these. So we have the promo SIG which puts on events Karl mentioned we had CentOS Connect at POSDOM last month or plans like our presence here at scale or managed social media an artwork SIG that does artwork unsurprisingly. It's funny you said for people that aren't packagers because I think that's what everyone's here for today for the packaging workshop. Yeah, well you can become a packager or not. And then Doc SIG which is oddly enough newly formed we had a Docs not a SIG for a very long time and then it finally got turned into a SIG and there's a lot of work happening there. One of the cool things is actually all of the rail documentation is being developed in the open upstream and we're able to use it for CentOS we just have to do a little bit of work adapting it so all of the kind of like the user guides and stuff for administrators we can adapt to CentOS but we have to kind of do the work in the community to do the adapting of that. And we're also working on improving the contributors guide which is not great to be honest and there's various other things on here and we have recently had archived the wiki because it had a lot of issues but there's still some good content on there we wanna pull out. So those three SIGs it's a good place to get involved if you want. Then are there any questions about that or anything on here or anything sent to us? Can you maybe pull back the curtain a bit on kind of how I guess the people aspect of it like are there people that work on Fedora and Red Hat Enterprise for example or are they once they take their curated packages they don't necessarily communicate back to Fedora team or how does that work? So it's on one of the slides if you go back. Yeah. The answer is it's complicated. I'm trying to figure out how to say it in a sentence. It's a little complicated but let me find the one I'm thinking of. No. Here we go. So I mentioned at various points talking about this that the with the changes to CentOS instead of just a handful of people rebuilding rail and calling it as is and can't fix or change anything. Now we've blown the project up with literally thousands of more maintainers because every rail maintainer is now a CentOS maintainer. So there's 100% overlap there. Rail maintainers own their package in CentOS. Not every time, not 100% but there's a lot of overlap with Fedora as well. So you have a package maintainer who is maintaining it in Fedora and let me go back to the... They're maintaining it across the board from Fedora into CentOS and rail. Not all the time, sometimes it's different maintainers but there's a lot of it the case where they have the package, they understand it and they're like, all right, this change in the software is only appropriate for Fedora or this change is appropriate for Fedora and then we're gonna put it in the next rail major which means the next CentOS major or even talking to discussions about putting the change into older rail minor releases and how they weigh that with different red tape internally and other stuff like that but they have a good overview of the whole ecosystem in relationship on how that works. And not only that, a lot of these people got their jobs at Red Hat because they're involved in the upstream software project. So we're not just talking about packages that know how the stuff works in Fedora, CentOS and rail. They also know interact with the upstream project. Sometimes they're in leadership positions in the upstream projects. Sometimes they started the upstream project. So they understand the software there as subject matter expert as you can get and understand the whole life cycle of it and even understand like we can't make this change in the upstream project because of the downstream effects it's gonna have in all these other places I manage it. It's not 100% with that. There are some things where there's disconnect and those are always areas that when we find it we're like, hey, why are you not involved in your upstream yet and try to work to correct that. At the very least get them involved in the Fedora packaging of it. But you know, Fedora does make different technical decisions than CentOS. So CentOS is, people will sometimes say like it, Fedora doesn't have an LTS release, right? Sometimes people will refer to CentOS as the Fedora LTS. And I guess it's about as close as Fedora has an LTS but it does have different technical decisions. It's not, we use, we'll use Fedora 40 as a branch point for CentOS Stream 10, but it's not a copy of Fedora 40. Some good, the most popular example people will talk about is that Fedora defaults to ButterFS as a file system. And that's a decision that, there were Red Hat people that were not fans of that but Fedora's engineering steering committee voted on that and said, yes, this is the way we're gonna go for this. So Fedora makes independent decisions and Red Hat can make separate decisions. Decisions though made in CentOS though, directly reflects what's going on in the next minor version of RHEL. So those will be things that Red Hat approves of. You're not gonna see ButterFS in CentOS until Red Hat's like, yes, ButterFS is going in RHEL. There's less disconnect there but a lot more freedom for Fedora to do things. But you can get ButterFS in CentOS thanks to our special interest groups. Two of them actually. The Kmod SIG has a ButterFS kernel module, the package you can install. And there's also the Hyperscale kernel from the Hyperscale SIG that has ButterFS enabled. So multiple options to do that. More questions. Don't feel like you're holding us up between everyone in cookies. I'm kidding. For the alternative images SIG, do you guys produce other image types besides ISOs like container images or QCAU2 or, and I guess like how do you build all those different images if you do support them? So we could. No one's asked. So if you're interested in images that aren't live ISOs, come and talk to us in the alternative images SIG and we can figure that out. At the core of it, the image build tool that we use currently for our images in the alternative images SIG is a tool called Kiwi. And Kiwi is a very versatile tool that was originally made by the folks in the OpenSUSA project to produce images of all kinds, whether it's containers or virtual machines or cloud images or disk images that auto expand when you flash them onto real computers or live ISOs. Right now we're using them to create live ISOs but if people show up and are interested in helping to make more stuff, by all means we could. Fedora as related to this based on the stuff that's been going on in, I know someone asked about the relationship between Fedora and CentOS in this regard. Well, hyperscale SIG and alternative images SIG doing the work that they did to enable creating what I will call Red Hat Family Distribution Images. That work fed into being able to do the change in Fedora 40, which is that they now use Kiwi to build their cloud and container images. So there's a symbiotic relationship, a very positive cycle of contributions that benefit both groups in a very obvious way. Any other questions here in the back while I'm here with the mic? Not a long walk. Just came back here to rally it up and get questions. All right, then in that case, we will do a packaging workshop but first we'll take a break for about 10 minutes or so. We can chat, socialize and we have cookies and lemonade in the back. So enjoy the cookies. Don't break for too long, people will disappear. Yeah. Not do the workshop. But yes, go enjoy the cookies and juice and stick around to learn how to make RPM packages. All right, y'all ready to make some RPMs? All right, I'm excited. We didn't lose that many people, so. A few people came in and out, it's fine. It's not gonna interest everyone. I know some people mainly interested in the CentOS project overview, that's fine. Some people like packaging and scratches an itch in their brain, being able to deliver software in the native system format. I'm one of those people. So I like teaching this workshop. Yeah, Michelle and Neil, they're the same way. It's not done appeal to everyone but I'm glad that everyone is here wanting to try it out and see if it's something that does interest them. This is gonna be our RPM packaging workshop. Introduce myself at the start but just again for anyone that came in later. My name is Carl George. I actually work directly on Apple that we talked about for Red Hat. Red Hat pays me to work on Apple, not individual packages. The individual packages in Apple are all kind of unsupported from Red Hat but more just making sure that Apple is healthy overall. That's my contact info. I'll put it up on the last slide too if you want it. This is gonna be a hands-on lab. So I hope everyone brought laptops. Go ahead and open up that laptop and type in bit.ly slash hello RPM and that's gonna take you to a hands-on lab that we will run through for this. So the way this workshop is gonna work is that we'll have some lecture and then some hands-on and then some more lecture and then some more hands-on. I've tried to split up the lecture so I'm not talking just at you entirely. Wake you up with some hands-on parts but some of the lecture does go a little bit longer and the last workshops are all workshops where I'll just, me and Neil's gonna help me. We're gonna float around if you have a question whether it's about the packaging stuff and the technical stuff in the lab or just the lab platform if it's not working correctly for you. Just raise a hand, we'll come over and help you out. I don't have a whiteboard anywhere to write down that URL for latecomers, do I? It's fine. If anyone comes in late, I'll just go back to this slide. So everyone present so far has typed in this short URL, right? Everyone that's going to. If you just wanna watch and listen, that's fine too. But the hands-on part is the best part of this lab. But first you gotta listen to me drone on about, geek out about stuff that I like. So what is RPM? It is originally RPM stood for Red Hat Package Manager. They have since changed that to just not be an acronym and be RPM itself. Somebody told me that Scale did the same thing. Originally Scale was Southern California Linux Expo, but to be more inclusive to non-Linux things, they're trying to unacronym that and just call it the Scale Conference. So fun fact for the day. So it's a recursive acronym action. Neil, correct me. It's not just RPM, it's RPM package manager like GNU's not UNIX. So recursive acronyms, recursive acronyms bug me, but anyways. So, but that's just the name. It is the package format for Fedora Linux, CentOS Stream that we've been talking about all day or in the session and Red Hat Enterprise Linux and many others. It's consumed by package managers on your system like DNF. Whenever I first started using Linux, one of the things that interested me the most was this whole idea of packages that rather than just going to download ISI or EXE files off the internet and install them on Windows, having a central repository where you can just say, I know the name of the software I need, such and such, install such and such, that was just magic to me. I love that hook me in, kept me using Linux and eventually I was able to build it into my job doing package management things on Linux. So I mentioned it's scratching an itch for me. I like that part of it. I'm assuming everyone here has at least used Linux before. So I think in the session we described it as, basic command line familiarity is the prerequisite. So you've probably installed a package from the command line before. Why would you want to package with RPM? I know we had a question about other ways of like Apple using RPMs versus other type of packaging technologies. You have all kinds of talk about things like flat pack and snap and other different methods for getting software in a system. RPM is a lot older, but the nice way to describe that is it's a lot more mature. It lets you easily install, remove or upgrade software like any package manager would. It's got commands where you can query your install packages and also verify the files from that packages are in the right place, have the right check sum, right size, all of those things. Each package is not just an archive of files. It is that, but it also has metadata inside in with that archive to describe the package properties. And more importantly, the relationships with other packages, each one of those packages doesn't exist in a silo by itself. It has its dependencies. It has things that it might conflict with. All those relationships, we'll get into those on a future slide. RPM also lets you digitally sign packages to verify authenticity and you can distribute those packages in the DNF repositories like I talked about, formerly known as YUM repositories, still technically defined in etsyumrepos.d, but that's... You call them YUM repositories. No, I'm gonna call them YUM repos because they are. But anyways, one of my favorite parts about RPM versus other system level package managers is that there's a concept of pristine sources. Nothing, I don't have anything against Debian. I think they're a fine project. They do a lot of really good things, but one thing that bugs me about the Debian package format is that they have their system basically intersperses the packaging source with the upstream source in a Debian folder and they manage them together and when a new upstream version of the software comes out, they have to rebase their changes in the Debian folder onto the new upstream software. And some people like that approach. I think it's bonkers and it doesn't work in my brain at all. I like the idea of separation of you've got the upstream source and the package source. And in the RPM docs, or at least in the old, there's a really old book online called Maximum RPM that it describes them as pristine sources and that's a pretty good term for it. It makes it where when a new version of the upstream comes out, you don't have to do get repository management to rebase commits and packages and downstream changes. You just bring in the new upstream tar ball and then try to build it. And if it works correctly, then you're good. It might not build correctly and you might have to make changes to adjust for the new version. But most of the time you won't, you can just bring in the new upstream tar ball as is without picking through the changes in it and build it with your packaging source and most of the time get something out the other end that's usable right away. If anyone, I'm gonna go semi-quick through this. If anyone has a question, go ahead and shoot up your hand and ask. And I'll come bring out one of the mics or see if Sean will do it for me. In fact, let me hand you that. Alrighty, so what actually is the RPM package itself? I mentioned before that it is an archive. It's got the files that are gonna go on your system when you install it, but also metadata about those files. There are two main types of RPMs. You have the slightly misleadingly named binary RPMs that contain the files that are gonna be installed on your target system. The naming around there is kind of the idea that you'd have like a compiled binary, but it's not exclusively compiled binaries. You can have binary RPMs that are what we call noarch where they are architecture independent. They're not a compiled binary. Maybe it's just Fedora wallpapers or a bash script. And it can be, because it's architecture independent, it's designated as noarch and can be installed on any architecture, ARM, X86, 64, whatever. But the correct terminology in the documentation is binary RPMs, even if it doesn't have any binaries. Weird artifact of time, don't worry too much about it. The other type of version, type of RPM is a source RPM. Instead of this files that are gonna be installed on your system, a source RPM has the software source code, that upstream tar ball, like a tar GZ file, and also instructions for building that source code into a binary RPM. It could be as simple as putting that bash script in user bin, whatever, or it might be having to run compiler commands or all kinds of other more complicated steps. So I mentioned the recipe for building the package, or the instructions for building the package. That's gonna be what we call the RPM spec file. It has a preamble section. It defines all of that metadata, and we're gonna go into each of those one by one. And it also has what's called the body. It'll have various sections that'll define stages of the build process. This is also where we have the change log and other things like that. That's also part of the metadata. In that spec file, you can have conditions that give you the flexibility to use the same spec file for say Fedora and Apple, like we talked about, it's just another branch of Fedora or multiple versions of Fedora or multiple versions of Apple or even different operating systems. You could have one spec file that works and builds a correct package on both. These conditions are in there. We'll go into the format of those next or soon. Oh, you can also say, that flexibility lets you do things like only apply this patch when building for rail eight, not rail nine and up, or only on AR-64, skip the test suite, run the test suite on every other architecture because it's broken on that one. Gives you flexibility for tasks like that. So inside the spec file, there is a thing we call RPM macros. That's gonna be a variable that lets you do text substitution. The syntax, it's somewhat similar, if you know bash variables, but instead of a dollar sign, a percent sign right up there. And it has both with and without those curly brackets and it actually is fairly similar to how that works in bash itself if you're familiar with it. You can use the macro without the curly brackets, but if you have anything else on that line that might get confused with the macro name, you need to delineate it and say, the macro name stops here with these curly brackets and then there's some other text. That's the real purpose for that. Both styles can accept parameters that'll influence the output because not all of them are just a straight, this macro has this exact string. Sometimes it can be Lua language is built into the RPM package manager, so it is embedded. So you can actually do Lua things inside a macro that says, with this input, do these outputs and these other things. You can define those. I talked about having them in the spec file, but you can also define those on your system itself. The main locations, it's similar to some other software like, I think SystemD made this a similar structure where you've got something under userlib that is like the system vendor level macros, and then you can have something under Etsy, which is like site local changes for macros that you wanna make, and then you also have a user level one in .rpmacros. You can set up that macro if you put a question mark there at the front of the name inside curly brackets, that'll make it conditional to only expand when it's defined. So if you have the disk macro defined, it'll put in that text there, and if you don't have it defined, it'll not do any string at all. Without the question mark, you would just get the literal characters, percent, curly bracket, DIST, curly bracket. No, it puts the name of the thing exactly how you have it. Maybe that's a very new RPM thing. No, usually you get errors because like the percents. Right, I know what you're talking about. Yes, so like if you have an unexpanded macro inside your build section, it's feeding that into a bash interpreter and says that, what, no foreground control. I think percent symbol and bash does some stuff with foreground, background commands. So you get that error, but what it's literally doing in the spec file is just putting in that exact text of an unexpanded macro, and you don't want that. So if you want it to be optional, you use the question mark. Another conditional form is not just inserting the value of the macro, but to insert some other text. So you could say like, if the rail macro is defined, insert the text dash dash disable this feature. Like think like a compiler flag. You're passing that into the compiler. And so you can say only insert this text if this macro is defined. Ignoring what the value of the macro actually is. Like in this case, rail could be defined as nine, but if rail is defined at all, we're gonna put in this text in that spot. You can also explore those outside of the build process. We'll do, I think that's in one of the labs where you run RPM dash dash eval or dash capital E, and then the exact macro name. And that'll just evaluate that one macro by itself. You can also pass, for macros that take arguments, you can pass them along to it. That's why I have the, you don't actually need the quotes there for like a simple one like this, but if you're passing other arguments, you do need to put them inside the quotes so that bash doesn't interpret them separately. RPM dash dash show RC. That one is going to print out all of the defined macros that your system knows about. Not any in a spec file itself, but all the ones that I talked about in those locations right there, it'll print out all of those together. So you could actually see that, oh, I've got this thing defined in my home directory and this one's defined system level and so on and so forth. So some of the really common macros that we'll have is for file system paths. So like underscore bender is going to evaluate out to user bin, datadir goes out to user share, sysconfdir expands to slash ETC, which I always hate that one because what it evaluates to is shorter than the number of characters of the macro itself, but that's my OCD acting up, so. Yes, but still Etsy with the macro around it is still longer than just putting slash Etsy. But anyways, I digress. You can also do operating system properties. Like the rail maker I talked about, that's going to be defined to the major version. Fun fact, since this is a CentOS classroom thing, rail is defined on CentOS also. So on CentOS Stream 9, the rail maker was also defined as nine because they're really dang close to each other and I keep telling people of that, not everyone believes me. Yeah, but either way, nine, so you can have, if you use this conditional, that is saying that whenever it's major version rail nine, do this thing or insert this text. So the disk maker that I've been mentioning in a few places, that is a special one. If you haven't talked about version in a release yet, that's coming up, but this lets you insert the string .el9 and that makes sense later, but the short version is that when you're building, using the same spec file to build for say, rail eight and rail nine, the resulting package in the release field is gonna end with .el9 or .el8 to indicate which thing it was built against. El9 is literally, it's a little misleading, it evaluates to one, but think of that in the like, bash one true type sense, like if one, do this thing. More common macros are build process helpers. If you've been doing software on Linux for a while, you might have found a project that wasn't in your repos and you wanted to do a custom compilation of it. You might be familiar with the traditional dance of .slash configure, make and make install. Real classic way to design around and make file. Those will usually take extra arguments to help shape where it is. Like if you just run those without any arguments, you're gonna usually end up with stuff in user local, but that's not what we want for an RPM package. So we'll have, we have these macros like configure and make build that actually pass in all of the predefined arguments, distribution wide to install things to the right locations like user bin instead of user local bin and GCC flags and other things like that. It's better to use those macros because you wanna use the standardized distribution options for all of that compilation and installation and not just whatever you come up with on the fly that may not account for everything that's necessary. There's also the auto setup that's, that is gonna be related to before you get started doing the build process and we'll get into the sections of that process later. The first thing you have to do is extract that tar GZ file from the upstream source code and maybe optionally apply patches that you need onto that source code. That's what the auto setup maker is gonna do for you. Some more ones, what I affectionately refer to as the legacy Python helpers, they still work, they're still good, they're not that bad, but PY3 build and PY3 install, that'll correspond to Python 3 setup PY build with some extra flags and install the same way. If you've done by hand installation of Python software, you might recognize that pattern. We have more modern ones now. They're in Fedora and also in CentOS 9 and Rail 9. They're related to the, if you're familiar with Python projects moving from setup PY files to pyproject.toml files, it's tied in with that. The pyproject macros can also find and interpret the setup PY stuff. So in Fedora it's better to always use those pyproject things. They work great in Rail 9. They don't have all of the features in Rail 8. There's some ongoing work to get those all back ported and working, but there's also a thing in there called dynamic build requires that is not gonna fully work on eight. Frustrating Neil, very much. Other patterns you might be familiar with from other software. We have macros for CMake, for Mason. These are called, I think these are referred to as build systems, which is not the same thing as like a package build system like Koji and Fedora. Overloading of terms. It's frustrating and difficult. But yes, they're, build framework. I just decreed that's the new term. But no, they describe themselves as build systems, I think. So yeah. There's also test suite helpers. There's gonna be part of the spec file where you can actually run the upstream test suite. They're unit tests. Hmm? Which one? These are just a few examples. No hecklers. Go. That's my job. So you've got some common ones. The PyTest maker will run the PyTest command with packaging specific options, similar with CTest and Mason test. So good. I get to take a break from talking at y'all and let y'all actually get to do some hands-on stuff and wake up. Right after this is the start of the lab. But what you're gonna do in there, there's a tool, a command called RPM Dev Setup Tree. It's in a package called RPM Dev Tools. And that is going to set up several directories that the RPM tool chain expects to exist. In your home directory, you'll have RPM build and then build RPM sources specs and source RPMs. And during the build, during the future labs, you'll see how each of those gets used. And that is the wonderful hands-on lab we're about to start. So that URL, did anyone need to see that URL again? Did it come in later? No? All right. If you're just shy, just ask me and I'll come by and tell you that you are all butt. Go ahead and load that up, click the start button and follow the on-screen instructions. Unless you're an overachiever and already clicked next and started without me. Some people, there's always someone that does it. Like, I'm starting the first lab and they're like, all right, I finished all four labs and I'm like, you're supposed to wait for the, the lecture goes with it and teaches you as you go. But anyways, if you have a question, whether it's the lab platform itself or the instructions in the lab, this one's really straightforward. I think it's just two commands. But after that, if you have any questions in any of the lab, just raise your hand. I'll come around. It's not gonna be on mic. We're not shaming you in front of the class. No dumb questions, as they say. And I need to make note of the time because this one, we'll give this lab about five, this lab's quick, maybe about five minutes in about five minutes. If anyone's still working on it, just raise your hand and I'll wait a few more minutes. But this one should be pretty quick for you. Just an FYI, Neil here is going to be my sort of co-presenter helper. He's also gonna float around and be available for any questions people have. Anything not working correctly. So feel free to ask him or me questions. I think a few more people are working. So it's been six minutes, but we'll give it a couple more minutes just to make sure we don't leave anyone behind. Another FYI, this lab actually is staying online, I don't wanna say permanently, nothing's permanent, but it's not getting turned off at the end of this workshop. So if you wanted to take this and do it again or share it with a friend, you can. I think the lecture that goes with it is really valuable, but I'm trying to make it where the lab itself has enough instructions for someone to just go in and do the lab with those instructions there. So like, for example, like a friend of mine didn't bring his machine. So if he's gonna later on go to that same URL and do the lab, but just listen to the lecture right now. So that's also an option in case anyone didn't bring a laptop. I was hoping we had notes in the program about bringing your own machine. We didn't get any hardware provided, but I don't see it listed there. So either messed up or didn't print it in the right field or whatever. Quick show of hands, no shame if you're still working on it, but I just wanna know if I should go ahead or wait a few more minutes. Is it, is anyone still working on this lab? Got one, two, okay, three. All right, yeah. We'll wait a few more, let people get all to the same page and then before we move on. Just do the RPM Dev setup tree stuff, hit the next, don't go on to the next part because we have lecture that goes on. Stop being an overachiever, Sean. Only click next once. Oh, so you're caught up, cool. Okay. Like the one time I did this lab at Summit and during the first lab, somebody just had to tell me like, all right, I finished all like four or five labs and I'm like, cool guy, good for you. Your first place, move on. Awesome, I think everyone that raised their hand said they're good now. Keep getting blinded by the projector. So, let's do a little more lecture. I gotta bore you with my voice a little longer. So the lab we just did, that gives you the basic tooling that we're gonna use. But now let's talk about the spec file format itself. So the first section is called the preamble. This is gonna do with the build process at all. This is all about the metadata for the package and that includes like package relationships to other packages and we'll get into those. But for just the plain metadata fields, the name one, somewhat obvious, it is the name of the package. Usually you're gonna pick that name based on what the software is called upstream. It doesn't have to match the upstream. So you could do it differently. Simple example, the upstream might always capitalize their project name but in the package you wanna make it lowercase. Yeah, there's lots of ways to go about it. I think the Fedora packaging guidelines have a preference to lowercase everything unless the upstream project specifically asked to keep a certain capitalization, which is fine. But there's lots of flexibility there but yeah, it probably will match something upstream somewhat just so people know what the package is but what it does have to match is the file name of the spec file. So if you're packaging foobar, your spec file needs to be foobar.spec. The version of the software, that is going to be, I think I touched on this in one of the Q and A questions earlier but the version that's gonna correspond to the software's version upstream, what they released the version as. The release is basically another version but that is the version of the packaging you're doing and you use that separate version to distinguish between different builds of the same upstream software version. And I'll show that in some of these examples but these three fields together, name, version and release, they form around like the Fedora project and other places that deal with a lot of RPMs, you'll hear the term NVR thrown out a lot and that literally is the name, version and release jam packed together and that forms a really useful identifier like if you're explaining in a bug report which what version of the package you have installed typically you'll use that NVR full identifier not just the version, the upstream version by itself. Here's some examples. GNU AUK package is named GAUK and then we've got a hyphen, the upstream version 421 and then another hyphen, a 4.EL8, that.EL8, that is the disk maker we talked about earlier expanded to that so when you built, you could build the same spec file and you'd get all of that .EL8 and then all of that .EL9, two separate packages built for different like GCC lib and other things like that, not GCC lib. Glib 2 is what I was thinking of, different libraries, they'll build for the different, that indicates the target you've built for. You can't just do the split on the hyphen though because package name, it's also valid to have a hyphen in the package name. I'm getting ahead of myself here but you can have a package name vert what with a hyphen in the middle and then your version and your release field. The way I'll do it if I'm doing something custom Python for like parsing this is Python has an R split, split from the right and then two fields, so from the right to the first hyphen is the release and then from the right to the next hyphen is the version, everything else is the name and the one in the middle that's just showing, what's that? Yeah, pythonsstring.rsplit, Rsplit 2. Anyways, and then the middle example, I just had that there to show some different versioning. Some projects use like this, I think it's called like octets, like something, dot something, dot something. There's a terminology called semantic versioning that that looks like and I would not ever trust that an upstream is doing it just because it looks like it. Semantic versioning is a really good system that a lot of project use something that looks like that superficially that doesn't work like semantic versioning rules at all. You might native, yeah, you might understand, you might look at that and say like, oh, I understand that there's the major version and then like the feature version and then like the bug fix version and that is what some projects do. That's close to like semantic versioning. It's a whole schema, I guess, or like standard and it's great. I love it when projects actually follow it. A lot of them don't. For example, Python, love me some Python, but Python, the first two digits are basically the major version, like 3.12 or 3.13, right? First two octets, they're not digits because that's something to point out. Version numbers aren't really numbers, they're strings. I had this conversation at a previous employer with someone that really bugged about the fact that like something.10 evaluated differently than something.2 and I'm like, well, because it's a string of one zero not a decimal of dot one zero. And so, yeah. And because upstreams do all kinds of different versioning like you might have something like the TZ data project that releases 2024A and that's not alpha, that's just A and then B and then C and then D on and on. Yeah. You might also have alpha and beta identifiers inside the version stream. There's all kinds of ways upstreams can do this. RPM has pretty good logic on around evaluating that. It has to understand that so it understands what RPM's an upgrade for which other RPM. But there are still things that can throw it's sorting out of order and we'll talk about that with this wonderful thing called epoch or epoch. This is a third version field and what it's used for. So, huh, the fourth? The name's not evaluated for the version comparisons. This is a third one. So you've got the version and release that I talked about like the upstream software version and the package version sort of the release that you're doing of it. But let's say you have an upstream project that decides that we were using versioning of 2024.01. But we don't like that anymore and we want to switch to proper semantic versioning. Hooray, everyone loves this. And now we're going to be versioning 1.0.0 for our next release. But RPM's evaluation is going to look at those octet by octet or section by section and say that 2024 is bigger than one. Therefore, this one on the left is a higher version and 1.0.0 is not going to show as an upgrade when you do like a DNF update or DNF upgrade on your system. Epoch is a bigger hammer. It is a super version in the spec file where you can say I don't care what the default sorting is, this thing with an epoch is bigger than this other thing without an epoch or with a higher epoch. No epoch is basically an implied zero epoch. Normally you wouldn't have to, you don't have to put epoch zero in your spec files. You normally shouldn't have to touch this. It is few and far between where this is actually needed. Some people make mistakes and I've seen it before. Some packages you'll see in Fedora and CentOS and RHEL have a really high epoch, like 70. And there's no way that an upstream project has changed their versioning in a non-sorting way 70 times. It's just not. But there were, I forget what, I think Pearl has some like 40 or 50 epoch. But there's historical times where someone mistook the role of epoch in release and they were bumping the epoch every time they made a packaging change, which technically works, but it's like a misuse of resources. Yeah, stuff like that. So usually it's a mistake when it gets to a really crazy high versioning. More likely you'll have just a no epoch at all or an epoch of one because one time the project made a non-default sorting version path. It is also possible that you might need to use, I had to use an epoch the other day. I switched the package, Python doc op, it's basically dead upstream, hasn't been updated in like six or seven years. No responses from the maintainer, it happens. Sometimes people, let's say the lottery factor. People talk about the bus factor, which does actually happen, sadly. My manager tells me use the lottery factor because that's a lot more positive. Yes, but they haven't touched the project in a while. That's fine, somebody made a fork and somebody asked, hey, for this package, can you switch it to the maintained fork? It's a lot of people involved, standardizing around it. It's great. And I was like, just makes perfect sense. I've done this in packages before, switching to a maintained fork, but I didn't realize that with that switch, they didn't keep the Python metadata name the same. They actually switched it from doc op to doc op ng, corresponding to what's on PyPy, the Python package index. And so that change caused some unintended side effects with the upgrade. And so I was able to go from, I think it was from version six. I upgraded it to version nine in the fork. And then I realized this is actually broken and not compatible software. I need to undo it. So it wasn't an upstream mistake or change. It was my mistake. And I was like, I need to actually force the upgrade path to go from six to nine, back to six. And I did that with an epoch. So now epoch one version six is higher than version nine without an epoch. That's what it's for. It's just a really big hammer when you, that you should save for last resort, break glass type scenario. We won't have to use that in this lab and you almost should never have to, probably not have to use it, but it's good to know that it exists if you need it. Yes. So also in the spec file preamble, we're gonna have these other fields. The summary is exactly what it sounds like, short one line thing. You'll get, during the lab, you install the package called RPM Lint. We'll use that later in future labs, but it'll complain if your summary's longer than I think 80 characters. So you wanna keep that fairly short. A lot of upstream projects will have like a short description and a long description. The summary would correspond to the short description from upstream a lot of times. But this is up to you. It's just how you're defining it for the package metadata, how you wanna describe it, how you wanted to show up in like DNS search or DNF info. The license, that is gonna be an identifier for the license of the software, like MIT or Apache 2.0, things like that. Small note on the license thing, Fedora is in the middle of a project to standardize, we had our own system for a long time because there was no standard for short identifiers. There is now the Linux foundations back to thing called SPDX. And Fedora is standardizing on using SPDX identifiers across all of our packages. Still work in progress, there's a lot of packages. RPM Lint in CentOS 9 and RHEL 9 doesn't understand those identifiers yet. So in the lab, for the sake of not getting an extra linter error, we're using the old identifier. But if you start getting involved in Fedora packaging, know that those identifiers are technically wrong now and against the guidelines. Coincidence. Neil said that MIT is right on both because the legacy identifier and the SPDX identifier MIT is litter just MIT. Yes. Unless you're using one of the weird MIT variants that used to be called MIT that is not MIT by itself anymore. Anyways, that's a tangent. You've got packages like geeking out on stuff. But anyways, so summary license URL. That's just what it sounds like. It's a URL where you get more info about the software, either a project's homepage website or the GitHub repo or GitLab repo, whatever you wanna call it. Just where you would go to get more info. The homepage of the software. BuildArch, that is, I talked earlier about the binary RPM name being a little misleading. BuildArch, normally you don't have to define it. It's gonna be defining your packages you build based on what architecture you're building the packages, the system you're using to build them. But if you wanna build a NoArch package that is valid on anything like, say, wallpapers or bash scripts that are architecture independent, you can manually set BuildArch, NoArch and define it that way and then you'll get a package that isn't designated as specific to particular architectures. That would matter like, let's say you're packaging up a bash script and you're just putting it in a user bin and then the resulting NoArch package you wanna be able to install on both your ARCH64 systems and your XA664 systems. If you skip defining this, you get a package that only works on one of the other arbitrarily set in the metadata even though the actual contents would work on both. So this is the way you make it. You actually identify to the package itself that you can work on multiple architectures. You don't have any architecture specific files. More preamble sections. The source, this is gonna be a file name or a URL to a file that you need to build the package. You'll, this is where you'll, the first one you'll see usually is the source code archive link like a GitHub download URL or like if the upstream project has their own file server they've published their archives, their tar balls under, you could use that. You could also use it for things like a default configuration file that maybe it's just locally with your spec file. Maybe it's at a URL somewhere online. It can be either one. Then you also have patch tags. These are similar to source. It's either a file name that you have right there alongside your spec file or a URL to a patch file that you're gonna apply to the source code to make small changes. You can use both of these tags multiple times and optionally suffix them with digits. In Fedora, so the suffix with the digit is actually mandatory on rail seven and rail eight, I believe. Rail nine it's not, you can use unversioned. It's a little, I think it's a little cleaner to use the unversioned one and I would only mess with the version source and patch if I had to specifically do something with one of the sources or one of the patches. If you've just got a source tar ball in three patches, you don't have to number that because that can all be applied automatically. If you want one of your patches to only be applied like on ART 64, then you might wanna number them so you could say only apply this specific patch number. Things like that. So now getting into the other, more important type of metadata I think, your package relationships. So the first one is build requires. These are gonna be packages you need in order to build your package. Also known as build time dependencies. As opposed to requires, that's gonna be packages that you need to install this package or runtime requires. Good difference there. And it's important that you, those are separate fields. Some people say like, oh, well anything that is required, I'm gonna have it build time. And that's not true. You can have runtime only dependencies that aren't necessary to build your software in any way. So you have control there and you, it's a little duplication if you have a long list of dependencies but it's not that bad. It's better to have it a little bit more explicit I think. Recommends is a slightly newer one. I believe it works in CentOS 8 and 9 but not seven. Of course you remember the exact version that's introduced in. But yes, recommends it's called, what's called a week dependency. It functions like or requires like a runtime dependency and it'll be installed by default. But the difference is that you could actually remove it and not remove the package that had a week dependency on it, making it an optional dependency. You could also configure your system to skip the optional, skip the recommends and optional dependencies anytime. Supplements is really cool. That is what I call a reverse recommends. There's no requires equivalent to it but basically with the supplements, the example I like to give is think about, think you have a piece of software and then you have plugins for that software and those plugins change over time. You package new ones, add them, you might remove some. If you use recommends for it, the main software to pull in all the plugins, you'd have to enumerate all of the plugins in the main software spec file and then if any of them change, update that list of recommends in the main spec file. Alternatively, you can use supplements and have every spec file supplement the main package and then if you remove or add a new package, you don't have to update the main package at all because it's working in the opposite direction. It's saying I'm an optional dependency for this thing and it just gets pulled in and recognized. Conflicts, that means packages that can't be installed at the same time. It's better to avoid this one if you can but sometimes it is unavoidable kind of like the epoch thing. Usually where this will come in and then to play is file path conflicts. You might have two completely unrelated software programs that both install, they both install user bin foo or user bin convert I think is one that we have to deal with that this software has it but this other software completely unrelated also has it and you neither upstream wants to rename their program but there's a namespace collision so the only thing we can do, if you don't do anything, DNF will let you try to install them. It'll start that transaction and after it's downloaded the RPMs and tries to install them, it'll abort and say I actually can't install these because they both have a file at this spot and that's not gonna work. You can tell DNF ahead of time that these two packages can't be installed at the same time with that conflicts directive so it will actually tell you before it downloads those RPMs. It'll look at the metadata, say nope, these two conflict and that breaks the transaction you can't install them at the same time. Maybe more common example than just an accidental name collision would be Forks of Software. You may have one software and then another somebody else forked the software, they changed the name of the project but they still have the same command and then maybe it was a contentious forking, maybe not and neither projects like nope, we're owning the name or even we have to have this file path because the way the software is written it expects this in every artifact of this other thing. Different real world examples I'm bringing up but in the end you can make those two packages conflict with each other so that way DNF understands that they can't be installed at the same time. Obsolete's, that is a relationship where you're telling the system that one package is going to replace another package. That might be used for if you're renaming a package like let's say, I know this isn't how it worked but let's say you had a WGIT package and you wanted to replace it with the WGIT 2 package. I know Neil's gonna want to jump in because that's not how the WGIT 2 project worked, that's not what it was, but anyways, if you wanted to replace one with the other you could use the Obsolete's in the second to say this is going to replace the first package. Sure, eventually, eventually it will. Okay, so yeah, so Neil brings up the fact that the DNF project itself is undergoing some changes where DNF itself is version four. They've been working on a DNF, a new major version five, but the way they're doing it is basically putting five in the project name and in the binary name and they're gonna have Simlinks, user-bind DNF is gonna be a Simlink to user-bind DNF, DNF five. But it is also at the same time replacing an older project called MicroDNF that was used in containers. It was basically a DNF with no Python dependencies that could only install packages. Not, I don't even think remove them or do other more advanced stuff but super lightweight, great for installing stuff in containers. DNF five is part of the DNF five rewrite is to remove the Python dependency, the rewrite mostly in C++. There's still some Python stuff, but the core executable is going to be slimmed down and replace the functionality of MicroDNF entirely. So they're having DNF five, Obsolete MicroDNF. That's a good real-world example of it. Provides is another relationship. This one's cool. It lets you allow other packages to refer to your package by another name. And that's kind of a vague way to describe it but the real-world example to make it concrete is you could have a package that needs a web server installed but you don't care which web server, you just need a web server. You want a web server to be installed for this software. You could have web servers in Fedora and CentOS actually all provide the name web server. So the HTTBD package, the Apache web server provides web server, just the name web server. EngineX provides web server. Catty provides web server. So you can have another package, require web server and have that dependency satisfied by any of the other web servers. Doesn't matter. If you don't already have it installed, it's, I don't remember how exactly it picks one. I think it might pick the one with the highest version. Okay, yeah. It lexigraphically does alphabetically on the name first. At one point I thought it picked the one with the highest version but that's not correct. Neither does picking one by first alphabetical either, but. A, A, A, A web server. Yes, but it allows you to install the web server first or in the same transaction explicitly name which web server you want to install and then that package will provide the web server name that satisfies the dependency of the other thing. So that's where provides come in. It also comes, I'm not hitting everyone. I've never, legitimately never heard of that. So I'm up here learning things as I'm giving it. Apparently there's one called build conflicts and I'm gonna have to look into that later because it's not a RPM 101 level thing. The thing you gave as an example for requires for conflicts at runtime where you have like multiple providers the same thing but you know that one is not good can also apply at build time. Because like for example, a good example of this is the user bin convert as you started and stopped with yeah is owned by image magic, graphics magic and like four other different things but like they change only in our package upstream they still do user bin convert. So yeah, no, you get to have fun with that but if you wound up having a situation where there's multiple theoretically identical providers that are actually not, you need a way to tell it not to pick that and so build conflicts is the way you do that. So the same way you tell it to not pick something at runtime you can also tell it to not do something at build time. I would just do an exact build requires. Yes, more advanced things. And this is not gonna be conclusive all of the possible tags. If you are interested in those rpm.org has the full docs for every possible thing. This is entry level 101. So no more obscure new things for me. Don't make the presenter look bad. Sorry, this might be an obscure one as well but in Santos they have to care about multi-lib conflicts as well like two packages like 64 bit and a little bit and you try to update one without the other and then you get weird conflicts. So I don't know a lot about multi-lib. I'm sure I could hand the mic to Neil and he could talk for 30 minutes about it but we're not gonna do that. Yes, multi-lib is a thing where specifically for library so that way you can have both like a 32 bit and 64 bit version of the library installed at the same time. I think for the most part those get the conflicts worked out automatically because there's user lib versus user lib 64. So I think most of those are just resolved by being in a different parent directory. Yes, data files in common could cause a problem there and I don't know how that's resolved in multi-lib actually. Let's talk about that more afterwards. That's good question beyond the scope of this workshop. Yes, let's just get rid of 32 bit. That solves the problem. Alrighty, so more fields in the preamble. We've got the description. I mentioned before that some projects will have like a short and long description upstream. This is where you'd put the long description. It can span multiple lines. It can be multiple paragraphs if you want. They don't really need to be that long. Really the target for this is when someone does, if you do just like DNF search on the command line, you'll get that short summary listed next to the package name. If you do DNF info on the package name, you'll get the summary and the longer description. Some software doesn't really describe their software that much and so you just use the summary with the period at the end for the description. Or make up your own. A lot of times I'll try to get like the first paragraph of the upstream read me or maybe massage it down to the first few lines, something like that. But it's whatever you want it to be to describe the package in DNF info output basically. And then a little bit more advanced thing that you can do is what's called sub packages. You might have seen this before if you've seen, I'm trying to think of a simple library. Let's say you've got libpcap and then you also have a libpcap devil package. Those are, it's the same software but you're splitting up the development files into one package and the main library files into the other package. Some distributions keep those all in one package and get bigger packages because of that. But in RPM, it's standard practice to split those things up for the use case because everyone that needs the library present doesn't need all of the development files present. So you would have a devil sub package where you place those different things and you control that by adding a package and description section in that preamble. In each of those, the package one starts a new preamble and then the description just goes along with it. So you could say these are the development files that go along with this and it has this other runtime requirement you have to have installed. The name there, one interesting note, that's not gonna be the full package name but right there in that format, if you just put a name there, that'll be appended to the main package name where you have like foo and foo-devel. You could also do an arbitrary name completely separate from the main package name if you add the dash in flag in the middle there. So you could say like you have package foo with a package bar as a sub package and it's not immediately apparent that they're related because it's not a suffix. So you usually wouldn't wanna do that. You'd have your sub packages named with just the suffix there. That's a little bit more advanced. I don't remember if we have that in the lab or not. Oh, we do I think for the Python lab. No, actually I'll take that back. No, the Python lab doesn't have that. So you won't see that in the lab but it is good to know about. And if you've seen those like development packages or separate packages like that, that's where that comes from. That's how you define that. So the spec file body. This is where we really get in the meat of the build process. The prep section, that's gonna be where you unpack the source code archive, the tar-gz file. You need those files extracted and then you would also apply any patch files to those. Like let's say you have a patch that changes, fixes a bug from upstream or adds a fix for security problem. Anything like that, you're gonna be applying that during the prep section. And we have that helpful auto setup maker I mentioned earlier will apply all of your patches for you without you having to manually go through with the patch command. It's really useful. The next section is build and it's exactly what it sounds like. That's what you're gonna run to build the software. Those helper makers I mentioned earlier like make build and make install, those would correspond to the build and the next one, the install section. Whereas build, that's where you're compiling the software usually. Install is where you're going to place those built artifacts into a directory tree that, what we call the build root, but it's going to be simulating the file system structure on the target system. So inside your build root, there'll be a subdirectory, you know, user bin, user lib, all those things and you'll place those files where they need to go in that subdirectory structure and then that becomes top level structure in the target system when you install it. And then finally, there's a check section. That's not exactly building the software, but that's where you can run any kind of commands to test the software. Upstream unit tests work really well here. You might want to do like a functional test where like you run the command and check for some kind of output. Really anything you want to test the software functionality. Two more, I think we're gonna get to the lab soon. The file section. Files and change log work differently than those other sections. The prep through check section, those were basically bash scripts. These ones, files and change logs, these are text and in this particular format. The files one, that is a list of the files that are gonna be installed on the target system. Like user bin foo and then user shared doc foo, whatever files you're gonna have on the target system, you just list those, the full path to those right there. The change log, that's in a specific change log format. You'll see it in the lab. That's where you're gonna record changes that have happened to the package between the different versions and the different releases. You don't have to be overly verbose there. It could be as simple as updated to the new version. I've seen some packages where they try to jam in the entire upstream change log in there and that's not what that's for. Fedora package guidelines actively discouraged against doing that. I might actually forbid it now that I remember it. Yeah, it should be fairly, fairly short. You could link to an upstream change log, things like that, but it's more if someone runs RPM-Q-dash-change log on their system, what do you want them to see? Do you want them to see 1,000 lines of upstream commit messages? Probably not. You want them to see like, hey, there's this new upstream version and I added this sub package or I patched out this problematic sub-commander, plug-in, whatever. Things that would be relevant to the end user immediately. Sure. Sure. Where's my micro-runner? I've got a lab helper and a micro-runner. For the files macro, is there a way to automate a long list of files that need to be installed? Let's say your make outputs 100 files or something like that. Yes. Yes, there are lots of ways to do that. So I know for example, in the file section you could do, we haven't talked about this slide yet, skip ahead, so in the files, at the file, the name, the start of it, you could do percent files dash f and then a file name, a local file name that's there in the build process. If your build process can write out those list of files, I know system D makes really heavy use of this in their package. That's another one, yeah. So system D does that. So during the build process, system D writes out that this component of system D has all of these files in these locations and you can save that output into a text file just during the build process, not part of the package installation. And in the files section, just say files dash f, this temporary file during the build and it injects everything of that file contents into that. There's other macros, another way, instead of listing every file exactly, we'll get into that, yeah. There's a recursive way in there, I'll talk about that in the next slide, but there's another question, but there's several different ways to go about it, yeah. No one more. Yeah, for the back porting, is it recommended for tracking purposes? Because in Git, you can cherry pick commits and it'll tell you this commit was cherry pick from this upstream commit. Do you recommend in the change log to do something similar? Like, for back porting a CVE, maybe you'll put like the CVE number, but like, would you ever put like a commit to say like this was the upstream that I'm back porting? Yeah, it's a little bit of dealer's choice. There's definitely best practices around it, like for example, your back porting a patch that fixes a specific CVE, you would wanna put that CVE identifier. People look for CVE identifiers and change logs. That's a great example of something you would do. I wouldn't just copy the upstream commit message in entirety and put that. Even if it's one line, that's probably two verbose. What I personally, what I'll typically do if I'm pulling a patch from upstream is I will write add patch to change or fix this thing, you know, back port it from upstream or something along those lines or even better rather than just say added patch for this thing and then a link to like a Red Hat Bugzilla number with more information. So it's really good for putting URLs like the bug number. It's really good for that. I've seen some packages where they will reference like an upstream pull request number and say like, this is a back port of this upstream pull request that fixes this thing. Maybe they also have a CVE identifier, maybe not. But yeah, it's dealer's choice. It's whatever you wanna put there. I would say best practice is the shorter the better. You want it to be very terse and just a quick something you can look out on the command line and understand. No, no, no, the patch content itself doesn't go in the change log. The patch itself is in that patch file, the one we referenced. Oh yeah, okay. Yeah, there we go. You would have the patch in a file like if you did get format patch, that gives you a patch file in like manless format. You save that as a patch file and during prep it applies that to the source code and in the change log you just describe what you did in as high level term as possible. Like I'm back porting this patch for this thing. So some real world advice here for what I usually do is if I'm back porting a fix or adding a patch or in a feature or security bug or whatever, I will take the commit back as a patch, put it in as a patch line and I'll put a comment line right above and save from and either put a URL or the commit ish or the reference pull request or whatever because that detail is useful there when you're looking at the details. In the change log I'll just reference as Carl said the very shortest form possible to explain it because the way you got to think about change logs is that it's some dude who's possibly looking from an SSH terminal remotely on possibly his phone and needs to be able to read it and from that perspective you should make it the shortest, tercest while still being understandable that you can possibly do because that dude is not going to suffer through reading a whole bloody git log on his phone. The level of a verbosity is something to think about there. Another follow-up point of what he was saying, referencing like an upstream pull request or commit, in Fedora policy, Fedora and Red Hat in general across the ecosystem of distros we have, we have a policy of upstream first and what that means is that we don't wanna have changes that are just special to us. We work upstream first. I mentioned that a lot of the rail maintainers are also involved all the way through CentOS, Fedora and the upstream projects. They're doing stuff in the upstream, they're getting those changes there and everything else is like soft forks of management of what combination of these upstream commits we're gonna do. It's really what it comes down to. But whenever you're doing a back ported patch, you, in the Fedora packaging guidelines, in line with that upstream first policy, if you're adding a patch in Fedora, there's a should requirement. There's an RFC, I forget the number, but should versus must terminology where you should, unless you have a really good reason, have a reference to where that back ported patch came from. And if it's something you came up with yourself or got from somewhere random, you need to work to get it in the upstream project because it's very annoying to have a huge long list of patches. Every patch you're maintaining in your package is making your soft fork, a little bit more of a soft fork, a further divergence and that can be very difficult to manage. The much more responsible way to handle it is to get that upstream, I maybe even get it from upstream in the first place. So that way the next version of the software that comes out has that patch built in and you just drop the patch file. That lets you stay at zero or one or like single digit patches and drop them as time goes on. But then for some longer life distros like CentOS itself, you're gonna have the opposite of that where I do have to keep system D on the same version or the kernel on the same version and you end up with hundreds or even thousands of patches. But those are also, those are going upstream first. They're not just secret sauce, red hat things or CentOS things in that case. They are things that are from the upstream project. There's not value in keeping those two CentOS or RHEL specifically or even Fedora. Fedora has the same, Fedora can back port stuff sometimes. You might see it where new version of the software, it has the fix that we want security or otherwise, but because of the way that change stuff, maybe config file change or a library change, it's only appropriate to put that into Fedora raw hide because Fedora releases also have a updates policy where you're not supposed to make big changes. It's just shorter life cycle. The 13 months versus like five or 10 years. So older Fedora releases might get a back ported security fix that get dropped in the next Fedora release because they're included in the next version. All right, so that answer the question mostly. We'll keep going. Where were we? The file section, I believe. Yeah, so different adders, adders last point, but yes. In files, each of those lines, you can proceed it by a different attribute. The durr, percent dir, that'll just say own this directory that I'm giving you, but not its contents, non recursively. If you just listed a directory, by default, it's going to include the directory and everything underneath that directory. Think like user include foobar. If you just put that in there, it's gonna include that directory and every file underneath there, like the PC file and other stuff in there. But if you did the durr on user include foobar, you won't get any of those files underneath the directory just the directory itself. And you may think, well, there's files in there that I did in the build process. I want those pulled in the package somehow. Where that really comes in handy is where the package splitting thing, where you want some files in this sub package and other files in this other package, durr gives you a lot of flexibility so you can say the top level package owns all of the directories and these sub packages own the files in those directories and they may be different sub packages owning different files there. So it gives you a lot more granular control. Config lets you mark something as a configuration file. And it has a really weird kind of dumb default and I don't know the historical reason why, but if you just put config in front of the file, it marks it as a config file, but whenever you update the software, if the config file changed in the output of like the build system, it's gonna replace all the local configuration when you install the package, which is almost never what you want unless the config file format has completely changed and the local config file is basically worthless. When it does that, it will save the old config file that you're placing as a .rpm save file. So it's not completely obliterated, but it's moved out of the way and might surprise people. Almost always you'll want to use the next one, config with the parameter of no replace. That means, and you've probably, if you've ever seen RPM new files on RPM based systems you're managing, that's where that comes from. There's a new config file thing. Could be as simple as they changed a comment at the top or they added new default directives, new sections, things like that. But that will create the config file in with a .rpm new suffix, which is a signal to like the system maintainer that, hey, you've got a new config file and it's up to you to merge in the changes, but your existing one is probably still compatible with the software and things will keep working. But if I forget the exact, I think RPM comp, there's a tool that lets you find all of your RPM new files and RPM save files on your system and go through and merge them and manage them or you just do it by hand. But yeah, config no replace is almost always what you're gonna wanna use. Nobody likes their config file getting clobbered on purpose. Adder is another thing you can prefix the line with. That lets you set the non-default permissions or ownership of things. All right, some attributes will take, will accept relative pass, which can be, which will copy the file into the build route itself. Most of them you're expecting to already be in the build route from your install section and you're just listing where the file is. But license and doc, you can give it a relative path to like the license file in the tar ball and that macro will copy it into user share licenses or user share doc and also mark it appropriately as a license file or a documentation file. For a long time, we only had the doc macro and the difference between these is that you can skip documentation files. You can't skip license files because most licenses have a clause that you have to distribute the license with the software. So that's what the difference, the key difference is there. So creating spec files. You may want to create these from scratch. You could also copy a similar spec file and adjust as needed. That's a real common. Some text editors have automatic templates like for example, if you use VIM and you just type VIM foobar.spec it'll inject a basic skeleton of an RPM spec file just automatically it understands the template. Other text editors have similar mechanisms. The tool you're gonna use, actually you don't use this one in the lab, we use another one, but there's a tool called RPM Dev Newspec from that RPM Dev Tools package from the first lab. That'll create a spec file from a template also and just output it. Similar story with change log entries. I mentioned that's the description between each version and release of the package as it changes over time. You could write them by hand. You could copy the previous change log entry and then adjust it. You could use text editor plugins to inject it. But then there's a tool you will use this one in the lab called RPM Dev Bumpspec. What that's gonna do is it'll either add the change log entry with whatever comment you pass that command. And it'll also at the same time adjust the release and maybe even the version if you need to along with what changes you're passing into that command. And you'll see that in, I thought the lab was the next slide. It's gonna transition right into. So, but there's also about building RPMs. We're gonna build them with the RPM build command. That expects the directory structure we made in the first lab, the RPM Dev setup tree. We've got various build modes. BS is going to build the source RPM from spec file and sources. BB is gonna build your binary RPM from spec file and sources. BA is gonna build both. And then there's also another mode called rebuild that rebuilds an RPM from a source RPM. Quality checking RPMs. During the first lab you install the RPM Lint command. That is a linter that will check spec file, source RPMs and RPMs. It'll identify real common packaging errors. It's not mandatory to resolve all of those errors and warnings, but it's good to try. But it does flag real common mistakes. But just to understand that it's not always gonna be possible. During the lab, some of the first examples don't have a man page with the program that you're gonna package. And you'll get a warning in RPM Lint that says warning no man page. Don't freak out, that's normal. It just happens sometimes. But that is a good indicator. If you're packaging some real software on the internet and it tells you, hey, no man page, you might think, does the software have a man page? And should I be including it? They do, let me add that. Or I checked and they don't. I'm gonna file an issue and ask them to write a man page or write the man page myself and contribute it upstream. So along with the RPM Lint, you can also use the RPM command itself for querying packages. Most of the time when you're doing packaging work, you're not gonna have these packages installed directly on your system. It's real common for package maintainers to be targeting a different operating system than what they're building on. And we'll talk about that in the last lab with a real cool tool called Mock. But you can use, for an uninstalled package, you can use the RPM command, use the package, dash, package flag, and that'll let you query information. A lot of people might have done like RPM-QI on an installed package before to show you the information about a package you have installed. You can do the exact same thing on a local package file that hasn't been installed by using the dash, dash, package flag. Some other flags that go along real good with that for inspection, dash, dash, info, list, your requires, provides. Those are gonna be dumping out the relevant type of metadata that you're asking for on the command line there. So let's get into the next lab. More hands-on, so I'm stopped not droning on as much. We're gonna, y'all are gonna package a piece of software called Bellow. That is just a dorky little name for a Hello World program written in Bash. Go ahead and click start on your lab, and I know the people that came in late. If y'all wanna join us, I'll show you the URL, but for over and else, go ahead and get started on the Bellow lab, and then stop whenever you hit the next button after you click next. So people that are just coming in, not to call y'all for being late to class, it's fine. But if you wanna join in on the workshop, this URL here will take you to the hands-on lab if you have a laptop to work on it. It'll be up after the workshop, so if you just need to write it down and do it again later, you can. It'll walk you through all these labs. There's lecture that goes along with it, but there should be enough instructions there along the side that you can get caught up if you want to. And if you have questions, if you're doing the lab and have questions, just raise your hand. Somebody with a mic, don't need a mic, somebody will come around and help you. I wanna let y'all know that we're on a little bit more of a time crunch than I expected originally due to some great questions that I don't mind at all, just over my minimum budget for time. So I'm gonna talk a little bit more about another packaging thing that we need for the next lab, and then open up the next lab so anyone that has already done with the Bellow lab can start on the next lab. If you're still working on Bellow, just kinda listen with one ear. This is a fairly, the next lab's fairly simple, and then get to the next lab when you get to it on your time. If you run out of time doing the lab, you're not really gonna run out of time. The scale folks told me that there's nobody in this room after us so we can actually go over and keep working, or if you just have to get somewhere else and want to write down that URL and finish the lab another time, that's also an option. But keep working on Bellow if you're still on that lab. But I'm gonna tell you about something that you'll need for the next lab, which is installing build requirements. For Bellow, there were none. It's just a bash script. Everything that's in the default environment is all you need to just get that file copied and then you're done. RPM build, it needs all of the build requirements that are listed in the spec file to be installed on your build host. You can do that either manually, just ahead of time, just install those by hand one by one. But if there's a lot of them, it's a lot easier to use a command called DNF build depth. And you will see that directly in the cello lab with an example of the exact usage. So I don't have to walk you through it. So when you finish the Bellow lab, click next and go on to the cello lab, which is appropriately a Hello World program written in C. And actually, there's no more lecture before the third lab, which is also a Hello World program, but written in Python. So just move through those labs as you get time. Do not feel like I'm rushing you. If you have questions, just raise your hand and we'll come by and get you on track. Don't get too bogged down if something's not making sense. And then there'll be one little last piece of, I'll give that a little bit of time to not overload you. There's one little last piece of lecture about a tool called Mock and then a final lab on Mock. We'll save that for like the last 10 minutes. When everyone's head down, working through the labs, going smoothly. Here, I'll give you a couple more minutes, just working freehand. And then I'm gonna give you a quick little slide about what the Mock tool is and then tell you about the last lab and then you can just finish when you finish. So one more slide of lecture to tell you about a really cool thing. So during all these labs, y'all have been using the RPM build command. Now I want you to throw all that away because you're not gonna use that normally. I'm halfway kidding, but there are drawbacks to using RPM build directly. I mentioned that with the DNF buildup thing, your build requirements have to be installed on your host machine, on your build host. It's also really easy that if you have a build requirement that is already installed, to forget to list it as a build requirement. Like if you have, like if your software needs GCC to build the package and you have GCC installed for some other reason and you build your package and you don't list build requires GCC, somebody else that tries to build it might fail to build because you didn't list it as a build requires. So it's entirely too easy to get in that situation whenever using RPM build directly. You can also only build RPMs targeting the same operating system and operating system version as your build host. So your build host there in that lab, it is a CentOS Stream 9 system. And so you really can only build RPMs that officially target that. You might be able to make an RPM that accidentally installs on another version or on another distro like Fedora, but it's not guaranteed and you'll probably run into library stuff eventually that for not targeting correctly. But there is a solution for that. And it's called Mock. And that lets you build RPMs in isolated charoots. It still uses RPM build internally in its guts, but it gives you the benefit that your build requirements are not installed on your host, they're installed inside that charoot and then discarded when you're done with the build. That'll help you identify any missing build requirements because in that charoot, it is a very stripped down minimal, just the bare minimum structure for getting a build done. And it'll tell you like, I also need these development libraries or this compiler installed to complete the build. Even more importantly than that, that's a great benefit on its own, but it also lets you build RPMs that target different operating systems or operating system versions as the build host. Limited, right? Like there's some limit of support for doing different architectures, but you have to have cross-compile support, I think. Okay, so that's gotten better then. There are also ways you can use Mock to do architecture, different architectures, but that's, yeah, you probably won't be doing that. Yeah. Those charoots that it makes are also automatically created and removed and managed so you don't have to really think about it. It's just, you might end up with a fairly large varlib Mock directory over time that sometimes you might want to purge manually. Mock is really widely used in our EGIS system. It's the Koji build system that Fedora uses. Uses Mock inside of it. The copper build system in Fedora also uses Mock, so they have a standard back-end, essentially. The Fedora Fed package tooling, if you get into being a Fedora packageer, it also integrates really well with Mock. And that's what you're gonna learn about. In the next lab, you're gonna build the Pelo package again but you're gonna use Mock to do it and instead of building it for EL9, the same as your build host, you're actually build it to target EL8 instead and then inspect the results. Enterprise Linux, and the answer is yes. Sent to a stream is the reference implementation of Enterprise Linux. Owen, we're close to the end of our official time. Keep working on the lab if you need to. I'll still be here for, I'll basically be here till they kick us out. But one other thing I wanna bring up is if this was interesting to you and you enjoyed these labs, then I would encourage you to consider becoming a Fedora slash Apple packageer. This URL here, bit.ly slash Fedora packageer, that'll take you to a page in the Fedora documentation about becoming a packageer. Coincidentally, that is great background for getting involved contributing to CentOS as well because it's a lot of overlapping these ecosystems, these communities and the tooling, the tooling's mostly the same across and it's all relevant and connected. So, if this, like me, if it scratches an inch in your brain where this was interesting and exciting to you, check this out. And I hope to see you around Fedora and Apple and CentOS spaces. That's basically the end of the stuff, but keep working on the labs. I don't know if Sean wants to say last words for the overall thing. No, not last words. I just wanted to, like if you have, you take this back and you then have more questions and you wanna find Carl, there is a CentOS booth, exhibit hall opens tomorrow afternoon, I think or whatever, so there's a CentOS booth. We'll be there. Yeah, there's also a Fedora booth that you can go visit and people there know stuff too. So, both a Fedora and a CentOS booth and a Red Hat booth apparently. You don't have to get out of here. You don't have to go home, but you have to get the hell out of here. No, I'm just kidding. I'm gonna be here till they kick us out. If you wanna keep doing the labs, feel free. We ran a little over on some of the lecture, my fault. So, we will be available and or write down those URLs. It was bit.ly, hello RPM and Fedora Packager were the two URLs to get to the hands-on lab and to become a Packager documentation. You want those again? Yeah, I'll be here. If you have more questions on the lab, just raise your hand.