 Welcome back and good morning to Boston and to the open user developer room I'm glad you're here. It's kind of a early bird session. So we have some some nice things for you If you feel are one of those vouchers, we will send you a susie Linux 10 one box When they are out and secondly for everybody. We have t-shirts there in those cartons. So just take one I'm Henne Who's a package at susie linux? I will give you an I think a detailed presentation about packages and susie linux and After this presentation Michael will give you an even more detailed inside About the about prostituting prostitutes distributions with the open build service and show you day And So as Michael I already said my name is Hendrik Funguson. I work in the development department Thank you all for coming down here this early. I really appreciate it Expected to have like three people here. Now we are at East Planet Okay, my history as soon as quite long. I don't suit it in early 2000 I started in the installation support helping to the linux customers Then summer around 2003. I took over everything development related in the installation support means I was responsible for buck reporting and for The beta test of the installation support team and stuff like that and since little over the year I'm in the development part now as a package so today I would like to Tell you about packaging Let me first start start with the purpose of this talk In the end, I would like you all to understand What is a distribution exactly? How do we do that and why packaging is the core part of Making this beautiful This talk is will not be another I can't make your package in one hour in the talk. So I will not show you spec files and Discuss some macros or whatever. I will give you a high-level overview first of what packaging is And then in the end, I will tell you what you need to know to become a package So let's start at the beginning Everything starts with sources So the usual output of an open source code track is something like this You've got the project source calls. You've got the built tools for source code like configure or make files and stuff like that and Sometimes we even have documentation the usual workflow to get such a project running is That you prepare the source code for the build That usually involves something like running a configure script or sometimes even you have to edit files manually then you have to check for the Dependencies of the project. Do you have all the libraries that it needs to have all the other project that it needs? Once you finish that you will build the executables with the tools and After that if that succeeded you install the executables and the documentation So now you can ask why is that not enough? Why do we need packages? And that's the simple reason for that the usual Linux system desktop system consists of 150,000 files For instance alone the root directory on a 10 of 1 has 18 sub-directories and they all have Another 18 sub-directories and you get files and files and files So you get the idea that we are dealing with a large number of files the usual software project installs somewhere around 150 to 200 files, so especially with Installing or de-installing software you're shoveling around files In a large number and it's not only that This normal way of getting open source software to run involves No checks on previous existing files or DD installation sometimes doesn't Delete files that are there you need to make sure that Software project A doesn't overwrite files from software project B sometimes you even have to do stuff manually like creating another user or Creating some settings in some files from another software project. So We need to manage all these files somehow And how we need to how do you manage files? Simple you just put them in a box That's nothing else that you would do when you're moving or whatever so To know what inboxes in these boxes are without unpacking them You usually put a label on them So this Would exactly scale for one to ten boxes and you lose track of what's in there and where they are Stuff like that. So you better start doing something like this You better start writing your package manifest So in my package books and magazines there this and this and this and this and this and in my In the other package books and magazines, too, there's this and this and this and this and That is exactly what our PM does Has files in some archive It has meter data that is delayed and the description and everything and the RPM Package manager keeps tracks of all the installed packages and that would be the manifest So we have source code Then we need to package them somehow to keep the files manageable So what else do we need need for distribution? The usual look on a distribution is something like that You say D this But that would be Like you say the house is like something like this Well, we are now that a house is not only roof, but you need walls You need a foundation and you need soil property in a nice piece of land to build it So a distribution is not this The distribution is more like this You have you start with open source software packed with a software Towers then you package them then you define patterns patterns are nothing else than saying I want to serve webpages to HTTP server and The pattern would be all packages that you need for that and Then in the end you put all patterns together and you call it distribution some words about the When the source code comes from This chart shows the distribution of source code in our current build system. You have this large green chunk that is Software comes from the open source Then you have this smaller red chunk. There is software that we develop in house like the just installation and then you have this little blue chunk which is So party software that we put on a distribution that is not But by some part of the well So if you look at this cake you could immediately ask what does the well do Yeah So if you look at that you could ask what what the heck does no well do and I Can tell you this is not the truth and there are two reasons for that first Novel is nothing that stands aside of the open source community. We are just a big part of it. I Mean we employ like more than 50 Lead developers and project and projects along this room. I see at least one two three There's Brisbane for I mean we employ a large chunk of the open source community that means the red chunk of source code that mobile touches got a good amount bigger and The second reason is about true But nobody is perfect We need to make sure with every software Package that it builds correctly that it integrates with other software packages and Sometimes even that runs correct That means our packages at novel must Assure that these three things are there that means they have to touch Sometimes the source code that comes from the open source community Some numbers to back that already is claim up at the moment. We have you know a build base somewhere 7,000 to 100 software projects and On top of that we have like 3100 patches That makes an average of two point three patches per software project My script to figure out how big the Patches are didn't succeed So I guess You said two to three patches for every software project, but the numbers don't match Yeah, but if every software project has two to three patches and it must be Two to three patches If you have 7200 software projects and let's make it two patches then you would be having 14,000 patches The numbers don't match. Yeah, could be Okay Thanks But I mean you get the idea we have for each software project we have at least one patch So the reality is something more like this novel touches a big big chunk of the source code that is in our distribution so That's just some words of what novel really does So this is how we do this usually to take open source of where we package them We define loose patterns, and then the end call it a distribution Some words about our development We have this software screen code stream That is called factory. So each time the package or updates this package or makes Or security things or whatever it goes straight into factory Then from time to time We kind of freeze that and allow me to get patches and security updates in that factory We don't allow version updates anymore or big feature patches or stuff like that That's usually when we start with a distribution There after some time if we get lucky we get it We make Installable media out of that factory stream that first they're called previews then you get dealers and our cities and In the end you get a gold master Once you have a gold master You copy the state of the factory stream to another location and just call To so this is how we Do distributions Then you have to call code streams one that is factory that goes on forever and The other one is this will be looks version that goes on for two years and we see Sony Security updates or major bug fixes. So now we know what we're dealing with in general source codes packages distribution So I would like to start to get into some details, but like I said before I Can't make you a package in one hour. So I won't try I will show you know what you need to know about Topics that touches that you have to touch with a package first again sources You have to be able to work with us That includes something like you should usually should know the programming language that your package That is a nice nice nice Because If there is something wrong, you know how to fix it and you can look at the source code and you just make a patch That's the fastest way to fix something if you don't know the programming languages of the package your package Then you depend on the maintainers upstream on the developers that developed this project to Get these infixes and that can take quite a long time So you need to know you should know the programming language that you package is Another thing is source code revision systems like The usual task for packages is to extract some patch patch from it from a revision or you Can even have a project that has no stable release yet, and it's just developed in a revision system So you have to check it out Most times you only need the only part where the read part of these tools So checking out packages checking out pilots or reading diff files CVS and subversion are nicely documented. There's a CVS book from O'Reilly You have lots of reference cards in the Internet and for us VM or software and it's the same there exists a nice book Explains all the concepts and how you deal with software Another thing you should know Social books Make I would guess that eight percent of all software projects use new make So you need to know How make files work? Or you can all make or you can give it some Things to do process in the make file and stuff like that you do make is heavily documented in the make manual on So what else do we need? We need to know Not only about build systems, but we need to know what tools that make build systems portable That's something like configure or auto tools configure is just a script that checks your current your current system on Stuff that it's needed for the package and the auto tools are one level above and Make configure scripts portable and give configure scripts input make files to process and then After you did all that you end up usually end up with make files That match your system Every tell you that's a fairly Complex topic. There are lots and lots and lots of layers of interactions in your own tools It's not easy To understand if you look at it on the foster level But the package I usually doesn't need to know how the all tools function in detail What you need to know is What is going wrong when some if something goes wrong and then you can specifically look that topic up? configure scripts Sometimes are just some hacky bash scripts. There's no standard on how to write Configure scripts the that means there's no single point of documentation about it More and more projects use the auto tools to generate that conflict configure script So you could say that if you want if you look at the auto tools, you're also getting Ideal what config is this might look like the auto tools are also documented in the book and a colleague of us that I'm going to make a nice presentation about All these tools beginning from make to configure scripts to be our tools how that all works together So if you want to have a general overview of how that stuff works, you just go to this user here. There's an automatic presentation, so What else do we need? as I said We have patches RPM has the concept of pristine sources that means you never touch the source power What you do is your unpaget Make the changes and then you create patch file with the diff and patch tools So it's a very crucial path to package the needs The different fields are also explained in the manual on the new org page So if you're creating patches for a small software project you might end up with 5 10 If you package a large software project, it can be that you end up with 20 30 patches and sometimes there has to be applied in a Secrets or they depend on each other you patch something then you patch something on top of that So if you package a large software project And you end up with a lot of patches that that that have interdependencies Then you might start looking at some tools that can handle patch management. One of these tools is Crilled that was written by also by a developer Based on some journal hackers hatches with And that food is a tool that can just handle these interdependencies. You can apply patches in a certain sequence. You can Unapply them you can modify them apply them again stuff like that. So If you have a large software project you should Looking to put that was just source code That's just one part of packaging What you also need to Be able to handle this the RPM package manager But it's fairly straightforward RPM consists just of one binary that is there for installation de-installation querying verifying blah blah blah blah all the user stuff Another binary that is RPM built that takes some control files their codes back files Processes them and does all these source code stuff that we talked about before so it will Package the source code Table the fly patches and stuff like that duty finding meter data in the spec files and stuff like that that RPM is fairly good Documented a good starting point is RPM.org. There's a book there. It's called maximum RPM It's a little bit outdated, but For a general overview. It's very nice fedora has some RPM guys that seems to be based on that maximum RPM that is updated and very very detailed strikes the way how to Install packages the install packages, but also has a large large topic on RPM building Pascal made some nice presentation about RPM building that is a little bit more detailed Somewhere around And there even some training on who's that who That teacher Handle RPM so in spec files you can Define all kinds of things we exclusive have our own Microfile that defines stuff like how to install Info pages or how to Re-style servers if you update a package or how to stop the service if you remove the package Stuff like that These are explained in the season package conventions if I'm not for novel.com so You need to be able to handle source code and you need to be able to handle RPM Now let's say you have your first Project in source code form you wrote a spec file and you want to build So you would take our handbook and Just run up here and build on the spec file, but that would mean That you need to have all the stuff the Project the building depends on in your currently running system You really don't want to do that you want to have a running system and we can work with and that is Running very well, so once you start using our preamble just on your system You end up with a large and huge system with all the development files and everything and Once you do that you also Will end up with a system where you have no control over what is installed or what is not installed and stuff like that So you might want to look into so called build tools until yesterday you have the two options in open susan, that's why to preamble it and build Why to preamble with some shell script build around the yes package manager and Builds is a stripped-down version of the script that we use inside the lab to build packages Most work fairly similar. They take this open-suser or susan installation source Pass the spec file look into the spec file what packages are needed to build that package grab the packages and install them in a separate directory on your installation then you use some tool like change route to go to that separate installation and Call up here and go in there so What these tools does do is? That I keep your a they keep your system clean of all the development files and be you build this fairly reproducible that means because the Packages that are needed to build the software project are given in the spec file You always know what ends up in your build system because they use installation sources You also don't have to care about Security updates for instance if you have a if you have a library and we have have made a security update for it you just build against the newest package and That's a good way to keep builds reproducible and resistant clean. I explained why to preamble with some detail on this in the susan build tutorial on the thickie and build is Documented in some cool solutions feature on novel.com. These two are obsolete by yesterday since we started the build service and Michael would tell me more about it in there after my talk So I won't go into detail about that now and then After you are able to handle source code and after you are able to handle packages Then the hard part starts That's maintenance a Good package or follows the project that he packed this He looks for bug fixes. He looks for security fixes. He looks for new versions and stuff like that and That's the task where most packages fail. You find Lots and lots of RPMs in the internet. The hard part is to find some nicely Maintained repository so you can get for Project food one lot. Oh, you find probably 20 packages But as soon as one of one comes out You find you the only two or three maintained packages where the update happens and that Is really the hardest part that's where most pictures fail So if you want to be become a package You should make yourself familiar with source codes with rpm and then you Should know that you probably will invest a lot of time in keeping packs updated and run What I think you need to Be able to do to become happy Are there any Questions about that for the same piece of software exists on the internet So don't you think it's a waste that there's already like two or three different Packages for the same software the same version which basically means that two or three people are doing the same work Sure, there's ways, but you can't stop people from doing these things What I guess because most times it is because they don't know about the other factors Or because they all started the same time Sure Sure, but that's what we will tackle with the That's why Michael you're talking about cross distribution building. So you will you have one Source one spec file and you get packages for many many distributions But I don't think you will ever be able to stop people making packages You just can try to actually Wait You mentioned the build script that knows what packages to install to the change for the environment How do I tell? You specify in the spec file which Builds Builds Each single page Michael would tell in the Talk about how we handle that there's a macro in spec files that call this is called build request And I can roll the parser from that from the script. So it will pass the The macro build request and install all the packages and the dependency they have into Yes I need I Know there's There's a way to figure that out Pass the little files and you make the build fail As soon as the development package doesn't require the other development So you make what that's usually For each package that's worth of two minutes. I mean you just pass the two files or Look at the dependencies and then you say the build failed because your development package doesn't require all the other development package it needs To be functional So that's for each package. I mean if I have hundred packages, that's Maybe half a day of work Yes Sorry They figured out a way to figure out the Relationship between different development packages Sorry, I'm not really sure as to what the question is because if if a development package is required and the package is done properly Then that should in any case bring down its own dependencies Yeah, you only need to do kind of develop I mean if you only need to do your own in-court tracing if the original package isn't done properly so Fairly new problem development packages have had the day because they they don't Bring any libraries that have dependencies because it's just the library of this The development part of the library of this the develop package and they don't bring any requires on All matter who tries on others. So you have to specify the menu. You have to say Mike, you know Gdk to devil depends on Gdk to devil you have to specify that man, but we just I mean we tell a lot of things with the open through the build service Because we will make practice fail on stuff like this So it packed up would never get a binary RPM if he of Some devil packages if he doesn't care for dependencies on Basically the idea is that's also a large benefit of using Y2 p.m. Build or Is that when you start building a change route from scratch Which is what those tools are doing if you didn't specify some package in the build requires It's going to fail because it's not there So the most important thing is that the build phase if you have to rewrite the spec file and build again So what that's not really issue. It would be problematic if you Working on your life system and you have package packages installed for example devil packages that are not mentioned in the build requires And it just works The config is good will work because they're there, but you didn't specify them in the build request That's an issue the build breaks. So what you know, there's a problem. So you can fix it That's not really an issue. So this is also a huge advantage from Building in a change route that is installed for scratch So some figures could also do a lot of automatic probing. It's simply At a lot of features because they find the libraries that I start Once the output of the country's But no What about libraries or header files that are not part of the distribution yet? So I need to do two cycles first the libraries that are required to the package. How can I integrate? What was the only service you just create a project And you put both things in there You figure out a way which one depends on which You might have to add additional installation source Yeah, because Too much meat at the stock packages won't be used in the build service Because everyone knows what kind of so you might have to add back More And