 Okay, hi, just to talk us about Jenkins, there be a net which should become Jenkins, there be an org one day and also should be more team maintained than it's currently is. I first spend some time explaining what the current status is. There's any question, please just ask them directly. So I do some things in Debian and basically I show this to show that I'm very busy. I start also do this PU parts thing which you might know which test the installation of packages upgrade and removal together with Andreas Beckmann. And in 2012 I started to set up this Jenkins instance which is currently still running and I'm also involved in reproducible builds which is largely using this Jenkins. And for that I'm funded by the Linux Foundation. So I'm busy and so I basically merge patches for Jenkins just looking whether it's safe and doesn't destroy anything else then I will merge it if it concerns somebody's other work. There's at the moment over 1,500 jobs running on this Jenkins and it's quite a mess. There are Python scripts, there's shell script, there's Perl, there's Ruby. And so whatever if you want to clean up things I will review it more thoroughly but if you just want to invent the next new island for your test that's fine with me. I really don't care as long as it doesn't break the system. But then I do care and I really want to maintain this Jenkins as a team because I think first it's better quality and also to get load off from me. And there is a team of several people but it's still mostly my setup. There's even Jenkins, Debbie and Orc since the year I think the machine exists. And this in the end I will present my very simple plan how to get there but there could be other ways. So the only other person really knowing this Jenkins setup a lot is Mathieu. He has all the excesses I have. He can deploy jobs, he can change configuration, do everything but he's also very busy and so we need more people working on this part. In Debbie and there's Valarie and others who maintain this reproducible Debbie and Mattie and me as well. Helmut Kroner is maintaining a loan reboot strap, Samuel Thibault, Hurt and Accessible Jobs, Stephen Semberlain test stuff with K3BSD, Phil Hens LVC which is Graphic Installation and Thomas Nitekki is also gave some Java support. In total we have 36 committers, there's also some from Arch Linux, Open Souza, Meet, Coreboot, Geeks, 3BSD and NetBSD so it's not only Debbie anymore. So who are you? Who uses Jenkins, Debbie and Ed regularly, watches, like looks at result pages, nobody is using this Jenkins at all, yay, that's bad. Who wants to contribute to jobs, okay, a few more people and who thinks Jenkins is currently too noisy, probably no one if you're not looking at the results. There are some IRC channels as well. So to me Jenkins is mostly a cron scheduler with a web UI and notifications. Jenkins has lots of plugins to do whatever pipelines and job control and stuff and I mostly use it to schedule jobs and then have a web UI where you can see how a job has behaved over time or other stuff. I also consider Jenkins as not secure. I don't trust any result which is built on the machine. So by design I just do QA stuff there and the other use case could be presenting results so I trigger a job on a secure machine from this Jenkins. The other machine would build it and then Jenkins would present the results. But I don't think we should build anything trustworthy on a machine with Jenkins because of the Java libraries and there's too much non-free stuff and nobody understands Jenkins. So Debian has some more QA efforts which you probably know there's Lincian which works on source packages, PU parts which test the installation, there's CI, Debian, Net4Auto tests. Some people do periodic archive rebuilds, usually only on AMD64. So there's many things which are done QA-wise which are not on Jenkins and it's probably good that they are not on Jenkins. I don't want to move everything there. So as I said I started with Jenkins in 2012 and I was freelancing at profit breaks at that time. They are cloud providers so I asked them for some few resources and that has grown over time. This is a job number and it's even 1,500 jobs now and this was November 2016 so the graph is here now. Today it's at profit breaks at 17 machines and two data centers with a total 170 cores and half a terabyte RAM, 5 terabyte storage and no static IP addresses. And as it's cloud provider, DCD is the software to manage this and so you have hard work access via IPMI like thing and Matia and me have access to it. Which we could other people if there were more. And then additionally to these 13 or 17 systems at profit breaks, there's other systems running at vagrant place, testing ARMHF and there's also 8 ARM64 machines at code thing and there's now 3, now there's 6 new ARM64 boards in Zurich from Axel Beckert which are not production so there's almost 50 hosts which compromise this Jenkins setup and now these 50 hosts 42 are for reproducible builds, testing and the other is for other Jenkins jobs. So everything which is running is in this one Git repository in the Debian QA group. If you check it out, read the install and read me document and job config and the bin directory are the most important things to look at it. There are many things to improve like we could configure the jobs with Ansible, we could build with S build or Jenkins, Debian Glue, could have better name spaces. There are many things to improve in this code base. Please do, I'd be happy for improvements. And there's views in Jenkins. So when you look at Jenkins, Debian Net you don't have 1500 jobs but you see 20 or 25 views where jobs are grouped. And so far this was done manually and this can be done with Jenkins job builder as well but somebody has to do it. So far they are all configured manually. Benefits for Debian, the first there's notification about failing jobs or unstable jobs. Failing jobs is obvious. Unstable jobs are jobs where there's some warning in the job so you can use regular expressions to say this warning should make a job unstable. And one other problem there is there's lots of notifications about problems but hardly anyone files bugs from it. So sometimes I file bugs but I don't always file bugs because it finds too many issues and then there's sometimes I see bugs being filed where I know the problem since two weeks or something. So if more people would go through the results and file bugs that would also help. In reproducible it works nicely. There we have filed over 2000 fails to build from source bugs so far but there's more bugs to be found. So benefits for you. I was hoping that there are more people in the audience who know the jobs, the existing ones so nobody will be annoyed if you don't look. First we have some muscular various QA related jobs like we have a job to find often packages where there's no often bugged file. We run the d-package trigger cycle thing. Depth helper, Depstons, Linzian and PU parts are built from Git on every commit in these. It's now Jesse Buster and Unstable and Strat. We find multi arch version. Packages. So the other side of the question is do you see Jenkins Debian Net as the place to go to do builds on commit for all packages in Debian? Well, maybe. I don't know. Resource wise it's a question but we can add a few more. There's some people using Travis. My point of view is if we need more resources we can get them. I think it's not a problem to get computing power for Debian. Do you know if it has been discussed in the discussion about replacing alias with something else or to plug this? I don't know but I think most alias discussions will happen at the alias sprint after Depcon. And I'll be there so maybe that happens. And it's all there's a packages YAML in this job config for the configuration of those jobs. So you could just add another one there. Yeah, but I mean if the Perl team wants to use this it needs something more efficient than one line per package. Yeah. If the Perl team wants to use that then we can come up with a way. The Haskell team is also using this to calculate the installability that's one group. And they have one notification which are the IRC and mail notification I guess are clear there on the channel of your mail. There's another form of notification which are these icons. So this is for reproducible and every icon is a job on different running on different hosts. And the result is immediately shown there. So this is an unstable job there and there are these blue ones are currently running jobs. It's just the image you can load and display on your wiki pages or whatever the Haskell team is using that to show whether their jobs succeed or not. Then there are 450 CH root installation tests which do up install some meta package. Or they also do up install some meta package and then upgrade to the next distro. That's why the numbers down there these numbers don't add up. They're more than 453 because they're also counted installation in Jesse and then upgrade to stretch or whatever. The Jesse stuff is still tested monthly because it also triggers the upgrade to stretch. The stretch stuff is tested weekly Buster every other day and sit daily. And that fine stuff with packages are uninstallable and sit usually or Buster. Then there's a heck which I wrote which is GI installation test which test Debbie installer and text and graphical mode creates videos and screenshots. And we have tests for Debbie in plain Debbie in and rescue mode also rescue mode and all the different glues. So it's not only testing in Latin but also Cyrillic and eight Indian variants and Chinese and what not. It's true what it is. It only tests Jesse stretch and sit. I didn't bother to set it up for Buster. Also test not only Linux but K3BSD and Hurt. So K3BSD and Hurt have been installable in the last year. I've seen installation succeed where the browser pops up at the end. And most of this shall be deprecated and replaced by something which Phil is working on which I explain in a little bit. But these are the screenshots. So it really it has the installation then boots the lock the installed system and starts the web browser to see if that works. That could be extended to also test the office or whatever. But this is a gross hack. So Phil works on this AVZ test which is it. Do you have physical machines to run those tests? No. So everything is in a virtualization? Yeah. KVM on KVM. Is it currently possible to run tests on physical machines for someone that needs it? Like for example if you need to test virtualization, nested virtualization is not efficient. It is possible but I don't have access to these physical machines at the moment. But if you would have the machines and want to test something we could do that. So these lip word Cucumber based tests come from tails. I took them from tails three years ago because they really have used nicely those Cucumber as a testing framework. So you define the setup. I have a computer with not clear with human understandable English language that are the jobs definition. So you define I have a computer with two gigabytes of rum and I install Debian and then there's some logic what action it means. That's really nice to do and it's now usable for Debian and now the tails people are also pushing their stuff back to us because they want Debian to be tested better to base tails on Debian. So this is now a full cycle of code reuse. And I hope it will be ready finally this year that it can be used. It also produces video. It doesn't produce these images yet. And then there are more Debian installer related jobs like all packages building UDEPs. They are also built on every commit in the get master branch. Plus the manual in all languages is built on get triggers and Phil and we have plans also to build proposed branches. So if you have a buck you create a proposed buck one, two, three, four, five branch and then packages from this branch are built all together from if there are several reports with this branch and the UDI bill images done. Phil has this half working. And for Debian Edu there's also these GI jobs with test installation and we have package tests and they are especially useful for Debian Edu doc because the manual is then automatically visible even without the upload and people can review and check things. But it's also useful for the other packages where mostly syntax errors are detected really quickly. Yeah, then there's Helmhut's work with reboot strapping. So that's cross the bootstrapping Debian from scratch on the combination of all these CPU platforms and Lipsy platforms. I think it's over 90 jobs now and Helmhut finds lots of things. And it's basically his research project and I show it here as an example. Even that is not really in Debian. You can, that's still useful for from a QA point of view. And if you have other interesting things to do why not run it? Yeah, and for reproducible builds there's 483 jobs now. So one third about of the Jenkins jobs are for that. There were 150 jobs more building packages which now were replaced by a small system D service written in Shell because it just kicks off builds. But that's not, that even that setup is not only about Debian anymore there's also 30 or 40 jobs for other things. But let's talk about Jenkins, Debian org only today. You deal with Jenkins what's the role, what's the respective role of Jenkins and the system D services? The system D service is just trickering worker scripts which then build and there are so many, like there's 150 of them and they constantly run they only take three minutes or so there's 20,000 job runs every day and they just clutter the UI because nobody looks at the jobs. In other things it's useful to have jobs but here there's so many builds happening it's just noise nobody looks at them. But then that's a limitation of Jenkins that applies to many things Debian doesn't really scale to 25,000 packages so whatever you want to do I wouldn't say that Jenkins doesn't scale to 24,000 jobs it gets on the limit but it's more the number of concurrence jobs running like I spoke with the Jenkins developers at Fostem and they were surprised about the setup and they told me it's one of the biggest setup they know and that really surprised me so about the migration there is Jerea is the Debian hostname it's ready it's set up I'm not sure whether every DD can log in or whether it's a group try it out and there's a C name pointing to Jenkins Debian org and this is set up by DSA and at least the current Jenkins team members have access maybe everyone in a group and so my idea is just to create a small script which just runs Jenkins job builder and as a first start it deploys self-yaml which has five jobs which basically test whether Jenkins is operational and once this is done extend the script to do the next YAML file which could be the pu parts or something small and do this till all are moved with the one assumption that Jenkins Debian net will stay and still run the job but it will just be another build node so we have profit picks build one two three blah blah blah and that will just become build node zero and will run all the jobs which are currently running on Jenkins itself which has the nice benefit that DSA doesn't need to install any software on Jerea because everything will be running on the build nodes and this is basically my migration plan it's really we have all these scripts in YAML and just move them to the other stuff and for some time until the I expect that to take a month or two we have two Jenkins instances where some jobs are running on one thing and on the other but as we only use Jenkins for some QA efforts which is not read by other things I think so so the YAML file would be the one that is input to Jenkins job builder right have you considered not using Jenkins job builder because in my experience it's quite limited like you mentioned the fact that until recently he didn't deal with the views and it's actually quite easy to just generate the XML config files that Jenkins is expecting it would be stupid to start with YAML convert to XML with the first we move from one descriptive format to another one which yeah yes you're right and to make things worse we also have Python code to generate the YAML which exactly you don't have anything to post-process the XML at some point but I still didn't consider this because I didn't want to write XML by hand I prefer writing YAML and I agree all this transformation are silly but maybe I can help with that because I did that a day job I wouldn't mind I think it's motivated by the fact that XML jobs for Jenkins are particularly verbose and annoying to write but I mean they are always the same so you can have a basic template and probably you only use five different fields that you usually generate with Jenkins job builder and there are many things that you can do in the XML you cannot do with Jenkins job builder like filtering the combinations if you use matrix jobs you cannot easily respect to stuff you can definitely do everything when you write the XML by hand or with another tool chain to write XML of course and XSL and there are ways to contribute as I said please file bugs about stuff Jenkins find or the noise it makes if it's something too noisy please send patches with ever modifications as git preferred give me a git repo to pull from and I think we have resources to run anything because if there's something useful we find resources and nobody has a test setup except for there's one test setup of the Jenkins instance but we are not really using it we just mostly deploy stuff and if it works then fine if not we just fix it up we are mostly reachable via the Debian QA IRC channel can file bugs directly against the Jenkins Debian or Psoido package and there's a dedicated mailing list also as well as the QA list one question one of the things I used to have with Jenkins is the Jenkins IRC boat telling me whether my git patch produces a working or non-working packages if you can build or not can I have that service with Jenkins.Debian.org for what packages? okay let's say if I want to integrate all the modules of the Python team inside Jenkins is it possible then to have a dedicated IRC boat for just this set of modules? it's not it's we use the KGB bot from KGB client and that will join your channel and then give the notifications in your channel my concern was to have not the result of other jobs no no you won't get them you will only get the notifications notifications of the jobs that concern you I'm done with my presentation just have the thanks to Profitbricks and Coating who sponsor Hardware and Debian Linux Foundation for my time and open for questions so one thing I don't know about Jenkins I will just ask is how do you do the mapping between specific jobs and specific workers to make sure that this job runs on stretch walker for example we configure it in the job definition so far we've configured it that all job types run on the same nodes most of the jobs still run on the Jenkins which I plan to just build zero while most of the other nodes are only used for reproducible and and so can some jobs currently break the worker nodes or is that considered not a great practice like if you run something that needs to run as root for some reason so my question is motivated by what I mentioned I'd like to test Vagrant cloud images and that needs to run as root so if you want to run Vagrant or QEMU QEMU in the setup so far we just run QEMU then as would run it as root as needed because also these nodes are easy to reinstall and if your test isn't malicious actively then it's very unlikely to compromise it there and even if it does it's just QA results I don't try to prevent again malicious contributors except you and the code and it's quite difficult to get the hold of this or start contributing so if you want to do something and have looked at the skid repository please just do ask me whatever question you have whether you think it's silly or stupid or simple it's probably way simpler to ask me or Mattia then just try to find it out yourself for hours I'm happy about any question did you already think about what will Jenkins.debian.org becomes after we move away from alias to whatever is going to happen what's your view on that I know it's maybe early to say about it my view is that alias will stay it might have a different name and different software but there will still be a repository I get repository server with users whether it's alias or githlap or foobah or maybe I didn't get on the survey Alexander did he showed that everyone was interested in having ACI plug to it so that was what I was having in mind I'd be happy to extend this I'd be happy also to have it separated because having it separated gives the advantage to be able to do this cleanly as one service where this is a bit of a zoo in the zoo there are different parts the reproducible part is nicely contained in a good area in the zoo so there could be a CI part in the zoo as well but could also be somewhere else CI.debian.net is something totally different and I'm happy that it exists and I'm happy that it's different if there are no more questions I'll show that here the main way to look at results currently is through the Jenkins web interface and the main way to look at what's up to look at the build results or do you do that currently looking so Jenkins.debian.net web interface directly that is I look at the main notifications and they are there for debi and edu they are sent to the team list so that's possible as well I know the Haskell group only looks at this output basically so there's different ways so this output is generated using the Jenkins West API or something like that yeah there's WebNetwork no I don't have network there's the Haskell wiki page they're still not developing for stretch okay, thank you