 I would like to start my presentation with a short live demo. I'm not sure if it will work because I only have 20 minutes time, so in the past it took around 15-20 minutes, so I'm not sure if it will finish in the end. So I have a repository on GitHub created and it's a simple hello world program, so I will just change it to hello distro deaf room and commit it. So in the end I can show you that we have a new Linux distribution created with my package installed already. So I'm from OpenSUSE. I work at SUSE in the build solutions team, so we work mostly on the open build service. And some of you may notice that we brought our own beer to our booth, so you can bite at our booth. Cheers! Anybody thirsty? So I wanted to start my presentation with some comparison to brewing beer, so why it's important to have quality assurance? I tried it this morning, so anyway, so what can possibly grow wrong? So if you work for a brewery and you're in charge of the quality of your beer and you don't have automatic testing, then you have to taste it yourself and you're constantly drunk probably. So if your beer is not good, so you get bad publicity, so nobody will buy your beer, nobody wants to drink it. And because it's food, so government is involved, so they will probably give you some fine and eventually they will close your brewery, which means they will for sure close it if the beer is so bad that people get sick from drinking it. So what happens in the end? Sales decrease, costs increase and you're broke. So how is this related to software? So in software it's also important to have high quality and how do we measure quality in software? So one measurement is that the software should meet all your requirements, which also means it should be free of bugs, so software failures. And how do we make sure that we don't introduce bugs? So I would like to decide, Martin Fowler, maybe some of you know him, he published some famous books. So continuous integration doesn't get rid of bugs, but it does make them dramatically easier to find and remove. So and this is my topic of my talk, continuous integration with an open-build service. So what I will talk about, this is basically the workflow I want to show you. So in the right corner you see maybe some of you coding and pushing your code constantly to the upstream repository. It could be GitHub or some other favorite source control management system. So from the repository, we want to get the code into the build service, which automatically builds binaries for us. So for instance, RPM packages, WM packages, which you can install then on your favorite distributions. And so the next step, what we would like to do is build all of these packages, a complete Linux image. So for instance an ISO or raw image. So and with my raw image, I would like to be able, every time I build a new package and get a new distribution, I would like to be able to see if this image still works. So that all the packages are installed, it boots correctly, the services are running, and get some feedback in the end. So yes, it works or no, it doesn't work anymore. So how can the build service and the tools that we use, for instance, we use OpenQA to test the distributions. So the full stack, how can help it? The build service is, so why do you need the build service? So you as a software developer shouldn't care about distributing your software to different distributions to different architectures. You should just care about your source code and then push it to the build service and get a new package for instance for OpenSUSE, for OpenSUSE LEAP or Tumbleweed or Fedora. So this is the only thing what you should care about. So the build service helps here. So you just push your code and your spec files or your package description to the build service and it automatically creates all the specified binaries for you. So you get out different architectures, you get out different distributions. And furthermore, if some of your dependencies change, it will rebuild all your packages. So if one package relies on one other and this gets rebuilt, it will rebuild the whole chain. Furthermore, you can in the build service collaborate. So imagine like GitHub. So if you have a bug in a package, you can fork it, fix it and submit it back. So like you would do it with code on GitHub. So that is one of our main features. So the first step is how do you get your code into the repository? You all work in free software development, so I will leave out this step. You all know how to get code into your GitHub or Git repository. The next step is how do we get the code from the repository to the build service? So in the build service, everything is... So the main structure is you have a project and inside your project you have different packages. So for instance, here you have an applications project with a sub-project popular and then you would have a gym or a Firefox package. So this is basically the Hello World project I showed you in the beginning. And so how do we get it now into the build service? So the main feature of what we use here is services. So services is like you know, you can trigger a service to modify, to change, to update your source files. And we have for instance a task service which you can trigger and it will automatically download your source files from GitHub, repacks it and commits it to the build service so that you get automatically a new RPM package build. So there is another service, for instance a set version which will automatically update your package description with the correct version. So how do you trigger? In GitHub we have a really nice feature. And so there is a GitHub hook already built in in GitHub. So you just have to specify your token of the project and specify there's already a default instance of the build service specified but you can enter some other. So if you want to host your own build service instance you can enter this one. So that means if you set up this with every commit you do to your GitHub or some other SCM management system you get the source code committed to the build service which will automatically trigger a rebuild of your package. So we have the first two steps. The third step is actually automatically built in the build service so I don't need to talk much about this one because as soon as you change your sources it will trigger the rebuild anyway. But how do we get out of the packages now a complete Linux distribution so like an ISO image or raw image? So in OpenSUSE we use Kiwi which is a Perl command line tool to automatically build a fully complete Linux distribution. So we use it at SUSE to build all of our SUSE Linux enterprise and also in OpenSUSE to build all our OpenSUSE releases. So we have two versions now. We have the old version which is in Perl written and a newer version which is now in Python rewritten which we use from the next OpenSUSE release. So how does it work? Basically you have your package sources which are in the build service anyway so all your packages. And then you have an image description which will contain all the repositories or the packages you want to install. All users for instance you can specify config files to override you can specify shell scripts to be executed while building the image. So this is basically an image description so the most important file is config.xml which is a configuration file which contains all the stuff I already mentioned then you have images and config shell script which are two shell scripts which get executed and then you have a root directory which just gets copied over your root directory and you can use it to override files. But do you need to write all of this from scratch? No. So what we notice is that many people use SUSE Studio which is from SUSE an online tool to create image description and images. So many people configured their image in SUSE Studio exported all the files and imported it into the build service. So we wanted to get rid of this step so we included now the base image description so on the main page you just go to new image and then we have some pre-selection of base images we selected and then you get the pre-configured image description. So that is what we have now is step 1, 2, 3 and 4. So the last step is now how can we boot the image, test it and check if it still does what it should do. So there is OpenQA. Some of you may went to the talk this morning from Richard so we already showed some stuff about OpenQA which is our distribution test system which we use for Tumbleweed for SUSE Linux Enterprise so every Tumbleweed release what we do before we release gets tested with OpenQA. And OpenQA is basically you have a web front end which has a database and then you have several workers which run the OS auto installer and which run different tests. So this is several back ends for instance we mostly use QEMO to boot the images and run the tests but you can also have Unreal hardware via IPMI or with a libware. So the web front end looks like this that was yesterday evening so I tried some stuff everything failed. So I hope that my other build will now finish. So how do the tests look like? So you have it's basically written in pearl so you can write in the test everything what you can do in pearl and you have some macros like for instance the asset screen here and it specifies a needle which is boot loader and the timer is 30 seconds. So the boot loader is a needle and it's called because finding the needle in the haystack. And when you run the first time the test OpenQA will stop in the web UI and will ask you I don't know this needle do you want to create it and then you go to the interactive mode and it shows you the output of the screen and then you can with a selector you can select what do you want to see what do you want to check on the screen so it does basically fuzzy image combination and checks if there's some major changes so here you see the boot menu in grub and I wanted to check that there is the first boot option Hello Foster still there so I created here the needle and then it runs through so that was basically about the main workflow what we have for the continuous integration so what is next in the in build service we prepare actually a new release which we will release around February or mid-February so which will basically include the images feature what I already showed you so that you can create some base images which you just need to adapt a little bit and the next feature is the multiple package so that means that you can inside a package you can build several binaries out of it so you don't need it was already possible before but you needed to create some links between packages and this is now built in we use it for instance for our ARM images so we have a base image description and then we apply several patches to the different boards so we have one package container but inside we build many different images for the different boards like raspberry pi or banana pi so I hope that my build is done yet so that is my build from yesterday evening but I can show you this one so here you see the test result and you see all the needles I created so as you see here I checked for the hello4stem so that is my hello world program and here you see my bootloader which checks here 100% match and it also creates you for instance a video so you see here all the steps that got executed from my test so if something goes wrong and you want to compare you just go to the page watch the video and see what went wrong and also for instance a needle doesn't match it compares it shows you the new image and says what is different and then do you want to update the needle or do you want to have a different needle or is it really a failure so that you can report it so we have a booth here which is in this building downstairs on the right you can see my green arrow in the right corner so if you have questions just drop by and we can show you also some more examples how it works how we use it for open susie to test it we can also show you the build service and otherwise I'm basically done 5 minutes? so yeah if you want me to write me an email or you can find me on github or twitter do you have questions about the source service does it support other conversion control systems than github and github? it supports subversion for instance americoreal and the source service the code is on github so if you want to have support for another system just send a pull request and we will include it same question as far as I know the build service usually takes a tarball from somewhere and you need to maintain the spec file within your package is it possible to use the source service to download the spec file from a remote repository? I haven't tried this but I know that for instance some colleagues do maintain their kivi descriptions in github but I don't know if you can pull it completely so the spec file too how does it generate a version number for the package that comes from source control? you can specify it so there is you saw the xml file so that was just the base option but you can specify more options so you can for instance set in the unix timestamp or the git version or you just hard code it and say it should be 2.3 and then increment so there are options to specify the versions so you can also with kivi so the question was if it's possible to also build with kivi other distributions than openSUSE yes you can also you can also you can also you can also you can also you can also you can also you can also you can also yes you can also build redhead you can build fedora so but that are the only two I know so far not with the database ah no ah sorry is there any control when your service is pulling a new code yeah it's specifically it's a REST API so you send and say trigger service to run it and what we did in github we include it so you can send pull requests to github to include a service so they have also hook so every time when you commit to the master or some branch it can send a REST call an api call and which triggers the service run yeah that's where my question comes from maybe you don't want to build package every time you push something to the master yeah so maybe you want to wait until a certain point where you build a new release for instance and you want to and so in the github service you can also specify the target so you can say only a master or only on this branch or only when you tag some if you create a tag in the in the github branch that you can specify it in github when you want to trigger are there plans to expand on other platforms like dbn package why so dbn is already supported in the build service but not in the kivi so you can build the packages but you can build the image