 Thank you, yeah, as mentioned, I'm Jan Klimko, for those who don't know me, maybe the most of you. I'm a software engineer, made my master degree in engineering and work as software developer at the software country in Munich called Ginoa, maybe some of you heard of. I'm addicted to OpenBSD and I use it since 3.9 and contributing a couple of changes a year since 5.0. And in this talk, I will show you the infrastructure behind bloom.ginoa.de. How many of you know about this website and the regression tests, I saw it, okay. A few, if you're not. It's simply, it gets some attention on the internet, especially on Twitter and in the OpenBSD core developer community. It's simply just this website. It's made of my colleague and he used this infrastructure I showed you today. And I will have some words on it. This website shows two things, the run of the regression tests and the results. You can see here, tests. You see these tables with the different regression tests. And it's a parse of fail. And we had another and we have these performance tests and I've lost a few words for these. Okay, this is too large-scaled. And there are some performance graphs we do regularly. So let's begin. First of all, the performance tests just to show you how they are made of. We have a testing lab in our company, just a server room with lots of hardware and we're using for the performance tests our own product hardware. We make firewalling products based on OpenBSD for application of a gate varies and layer three, four routers. And we use our greater powerful machines to make this regression test. And it's pretty simple. We have, we use two or several nodes based on Kino Linux because of the better performance of these machines as a source and a sync system. And we interconnect them with a 10 gigabit link to our product hardware where we don't boot our product system which is a highly modified OpenBSD. We just, for these tests, we use just the snapshots every night, install an AMD64 system and make an IPerf TCP throughput tests with routing and with relay D and we have an IPsec test where we have two machines and we measure how much the throughput is through both machines if we have a midline with IPsec. And it's just from several Linux nodes the IPerf combined and TCP and in UDP and you see the graphs and the website how they are changing over time from night to night from snapshot maybe some days missing in the graphs then something bad happened, patch day or something and then something bad happens this is possible. So don't rely on the dates in the graphs. There could be some missings. So now to the main part. The infrastructure which is used for the regression test I started two years ago but not to make regression tests. My colleague Alexander uses for the regression test but I actually had in mind to make diff testing easier. What happened is in 2011, I started developing a little test or a little test on little diffs for OpenBSD and send it to the list and one reply I got was, oh no, this won't compile or this has these problems or you send a part and you get back you're missing some dependency and this is because most developers use their own notebook to develop the changes and to test them and actually or if you make actually when you make parts and you're missing a dependency you don't get it because it's in some new system because it's your daily business system you're using and you can't wipe everything from it and make a clean test and so I started about how to make a simple infrastructure that I have a machine by side I can use to wipe and install and then make my test on it so you can make clean tests for one diff you make clean tests on one system and know if a dependency is missing or if something else goes wrong or if you have the problem you really fell into some bigger problems maybe fell into the DDB and have to debug it and it's pretty annoying if you do this with your own notebook where you have to read two mails and you have to read the source code in parallel or surfing and something like this so this is pretty critical and not everything could be done in a original machine and then I start thinking about how we can make it something better or how can I build a system for myself to better and cleaner testing of my own diffs and fortunately we have a nice lab in our company which is just for customers not for customers, for employees to testing with open source technology it's called the OpenBSD lab in our house because mainly developers are there who testing some OpenBSD stuff within it but there's also other stuff there's some OpenBSD forks of some employees for us who just playing around with the source code and making something different and bring something out and what I've done is I took a machine and I make some automatic installation and then I saw oh, there's a way how since there's a way where I can get it automatically and from over the time of the last two years I developed them for me a pretty good working system to automatically install a new machine and then Alexander came to me and said hey, I would like to run the regression tests on it because he don't have hardware and then I have these slightly working infrastructure and he was the first customer of this new system and tested heavily and I show you these infrastructure the main infrastructure are these yellow computers down there these are pretty old machines these are machines I think in every IT company there are some old machines laying on the ground in the corner and nobody cares of and actually this is machines you can get without any question and start playing with it and so we integrated into this infrastructure first I was just one, just for me and then Alexander came to me with a special requirements and one of his requirements was I want to make a path and do tests so he need four machines with three cables on this that's the reason why the first four machines are interconnected with a red cable so that he can make this kind of test the infrastructure is beside of this pretty easy is just every machine with every interface is connected to one gigabit switch and is behind these test master node these test master node is part of the lab network itself and has an outside connection to the internet and it's reachable from the outside I plan to make these computers really isolated the test machines itself are not connected to the internet they're not routed they just have access to our internal open-base demirror which we use also for our internal development in the company because it's much faster to install from this or to install the packages from the server if we would install it from the outside it costs a lot more time that's the reason why I use this and this is interconnected via a real AD running on the test master itself the test master itself has some other infrastructure connected I found a power switch just a power supply which is switchable via a serial connection nowadays we upgraded to a power supply switch which is switchable over a network and the test master is capable to switch every machine on and off so if you're really screwed up if you're on a remote session somewhere else maybe in Paris and you screwed up your machine you can at least you can switch off the power and turn it on again if nothing goes on and more important infrastructure is the serial connection in the test master there are some multiple serial PCI cards in it and we have an every test system has these serial connection so if you make some kernel tests or if you try to reproduce some kernel bugs you can after you have reproduced it you can debug this thing in the DDB which is quite annoying if you don't have serial and you've tried to remotely debug something or debug it via VGA and it's why VGA is horrible because you can't really copy some messages who everything have to type it by hand and so it's quite comfortable the regression tests itself don't run on the test master test master is just the basic infrastructure and it's the interface I provide for myself and for Alexander and for another developer who use it and the test itself and the website I showed before run on the Bloom machine which is a bloom.genua.de and this machine is also connected to the outside world and have public access and from this machine on every night this machine use my interface to automatically install the first machine or the first two machines and run the regression tests on it and some regression tests require that it reaches the OpenBSD test machine three and four so that these machines are just alive and the first two get automatically installed with a fresh clean snapshot every night how does the interface look like? The interface I thought which is pretty nice to me at least as you just have SSH so if you have SSH on your computer you could use this kind of infrastructure you just have to mention the computer on which you want to run your tests as a user and then just SSH the machine at the test master system and then the command if you type nothing you directly get to the serial console and you can interactively typing into it and have direct serial access if you want to install the system you just type install or upgrade just upgrade upgrade is a really pretty nice feature if you have some configuration files and try some stuff and have some greater diff and want to upgrade them if you make a clean install so you wipe everything and you lost your configuration so you can keep it but if you really want a clean test then you should make install and the hard disk is wiped during the installation process to don't get in conflict with other users in parallel I have this lock and free command so you can lock your machine that nobody other can install upgrade it or nobody other is able to switch the power off and on or cycle the power thing the power supply and it's pretty, it's the interface and you have nothing to new more than this and the background which I shown the next slide there's some infrastructure I built but just with the simple shell scripts let's look what's happened first on the test master system there is the SSHD configured which knows that the users for every machine for example here the OT1 has a force command forwardings are forbidden and you're allowed to open an SSH tunnel to the machine so that you're not if you're, if you have to type a lot and have a lot of debugging on the machine it's not that annoying because of the slowly serial access you can open multiple SSH sessions when you first connect to the console and take an SSH tunnel with it so you can locally forward to the SSH possible machine and you can fastly typing on it and debug your stuff or testing your stuff so what does happen in the run command? The run script is just simple shell script where I part these commands console on off cycle and the rest and it starts the console program and the serial lines are managed by a conserve I don't know, are you familiar with console management? Conserve is a demon which handles serial consoles of different types over directly over phone line or a serial card in your computer or over USB or over telnet or something else the steam and manage all the stuff and it's had a client which called a console and if you type console and the user you're directly connected you're forcefully connected to the console wire these infrastructure or wire these this demon and if you type the on off switches or the cycle then the shell script is started with just send the package to the or make an HTTP request to the switchable power supply and then it switches the power supply for this machine so it's pretty easy how the conserve is configured to limit the access to the different machines I limit the access of all the OT users so they can just have such a read write permission for its own here for example is the snippet of the configuration you see we have these cards, every card has the same bout rate and the write permission is just for the for the user of this machine and they have directly access to the device and yeah and this limitation is important because if you don't limit the access then the user is able to connect to run serial access and there are some shortcut commands where you can maybe switch the bout rate if you have to or you can send a break message if you want to but you can also switch the console to another serial access line to limit this that nobody can switch that user don't can switch between the lines you have to limit the access and that's it so it's kind of secure to give authorization access it's pretty easy you just have to give me your SSH key or the developer can give me SSH key here's for example all both keys in the authorize key files and in front of it I wrote and a hard coded environment variable test user these scripts which are running and preparing the machine and rebooting and something like this looking at these variable and this variable does the locking mechanism at all so if some user locking his machine then it depends that every locked command is just done by the test user which is named by this variable and in this way you can also have multiple keys like Alexander has from his notebook and from his testing machines they have different keys and so he can use this feature I for myself have two I have multiple machines with multiple SSH keys and you can put the multiple keys in it but you can give it the same username in front so it's no problem to deal with this the machines are pretty easy configured there's just a directory called ENF in every home directory of every machine and it's just contained files named machine, architecture, IP and the MAC address and with this information the install script knows how to configure the HTTP know the DHCPD and how to configure the TFTP daemon to let the machines automatically boot over network and if I want to plug a new machine into the infrastructure I just copy this directory I set up a new user copy this directory edit these files and then I have a new machine with a new MAC address and it just works it's pretty convenient and easy to do this process the only limitations I have is the amount of serial lines on one computer so but at the moment we have enough it's okay so the installation script you can run over SSH it started via the ENFDEAR program it's a pretty nice program from DJB as far as I know and it works pretty easy it's just you run ENFDEAR and after this the first command or the first argument is a folder and it reads a folder with the files which I'll list it here and after this you give another command which is these installs script and then all these files are then in the environment with their first line as content of the variable and so the script knows on which machine, which architecture, which IP address it has to configure and then the automatic boot system runs and make the auto install what does the script also is in beginning it loads the current PXE boot and BSD-RED file from the local mirror it modifies the DHCP-D config based on these files and the environment variable the information from the home directory then reboots or cycles the machine reboot also runs over SSH it's locked in as root user on the machine and run the reboot command or if it doesn't work it just power cycles the machine and after this it looks at the serial output of the machine itself and when it saw the machine run into the gratulation your system is installed message then it turns off the PIX the DHCP-D boot, the network boot and then the machine recycles and you get your prompt the whole process course about seven minutes seven to eight minutes depending a little bit on the machine and if you just type SSH machine at test lab install, this costs you seven minutes and you have a clean installed system with a current snapshot and you can start testing your diffs and to conclude everything what do you get from this infrastructure? You have a clean current snapshot installed into seven minutes and you have the current source code available via NFS, it's automatically mounted the machines are installed with a site tar TGZ file with where some pre-configuration is done you have also on every installation you have the same host SSH keys so that it doesn't annoy you when you connect in over SSH over and over again after installation that you have to remove the old key and accept the new one these are static and these NFS configuration is static and the interface configuration with IP address is static so you just can start testing packages are also able to install from a local mirror in no time, it's just on the local network so it's pretty fast to install the packages and you can start debugging and as I mentioned before you have direct access to the zero console so if you have a kernel diff and it's panics, you can directly debug it and when everything's fucked up you can't plug the power switch remotely and put it in and retry so I think this is for most cases what you need if you make a diff in the current level beside of actual hardware development then maybe you have to put on your hands on but for most other cases if you have this environment you can remotely debug everything and test everything on clean systems at least I want to show some problems and then some future expectations or some future plans I had in my head which I would like to maybe implement problems we now have we have some race conditions so it's not, it works in general but if multiple users would at the same time install several machines it's not race condition free because the PX eBooter runs or it runs the path slash BSD on the machines and try to get them from the TFTP and if you run multiple architectures or different versions of the BSD kernel then you got in trouble because maybe there's an ARM BSD file kernel to load and some other user in a second later overrided with a new installation and it's an I386 or it's an MD64 and so you can get in conflict but if you install it in a reasonable distance and it's okay and if you just have a few users it runs for the moment at the moment there are three users it's me, it's Alexander and it's Patrick there's some ARM stuff and there we got firstly into the problems that are in the TFTP directory there are some kernels laying around which is not for I386 architecture and the machines doesn't boot so this is a problem I'm working on at the moment. My future plans for the system to make it more usable I got into action or in discussions with other OpenSC developers where firstly more architectures so that we can provide these infrastructure for other OpenSC developers to have an easy access for other architectures which not everybody has in this home or there's just a few developers a few enthusiastic developers who have really many machines at home and testing on but most don't and so most don't care off if something break on other architectures and to lower the, what's it called? To lower the amount of effort you have to make to test it on Spark or on Powerpacy or MIPS I would like to provide these infrastructure remotely via SSH for other developers so that they, if they had it diff maybe deeply in the kernel or so and don't know if it works or not they have the ability to easy test it with this infrastructure and this is my main purpose at the moment to provide this kind of service for the core developers and I'm at Spark CC4 we have in our lab thanks to MPI and Chris as OpenBSD they sent me two machines and Powerpacy G5 system which are, which is to prepare to get into the system and I guess I get got from Chris and MIPS 64 machine which also pretty interesting so we have from the alignment perspective and from the strictness and the memory we have a huge good test coverage and good hardware coverage at the moment and they accept us to integrate these systems. Patrick is working on ARM and he have an own ARM machine where he can remotely debug on and developing on which use of maybe just here at the moment. From Patrick I got another requirement he would like to remotely upload binaries at the moment he has a root access on this machine, on the test master machine itself and he'll log in and put his binaries in place in the TFTP directory and boot them manually and manually edit the DHCP config and I think it's more convenient if I find a way to give him access to just upload the file remotely and switch on and off the DHCP config per computer and because in ARM it's at the moment not possible to make a whole automatic installation because Patrick is working on initially get some board, some ARM chips or socks working and there you don't have a convenient auto installation. So if I implement some switches on this level it would work for this work it would help for this work directly on the socks and the last idea which came me for inactive is a mail interface because the main OpenSD workflow is there are people who are sending diffs on the mailing list and a pretty nice interface would be to send mails directly or forward mails from take directly after the test machines and they get automatically installed the diff applied, build the regression test run and if something in the chain failed you get a reply which has the error messages and it says hey see there this doesn't run or if everything run and you can just start testing on the whole diff as it is replied and is it installed and you don't have to fiddling yourself with installing machine and make other stuff and so just type it in and in 10 minutes you have your machine with stiff and you can test it by your own and it should be lower the effort of testing diffs and maybe get some more progress on diff testing but this is somebody for the future which I may make during the next year or so obviously yeah, that's my presentation have you any questions? Yeah, cool stuff you said you do the regression tests every night it means you step up the machine build the IPsec tunnels do the performance tests and then use your lab for other things during the day the web page you said you showed in the beginning that you said you do some regression tests every day yeah with that infrastructure with infrastructure I showed this is just a regression tests it gets every night at two o'clock I think around two o'clock the cron job a cron job running on bloom starts the SH commands to the test master then the machines get automatically installed if it's finished it makes advanced regression tests and they're ready in the morning or so or it's cost several hours all right and look what's going on the configuration for those machines I've written the regression tests in a way that they can be pre-configured so I've added the IP addresses I need so for the path of view or IPsec tests the packets running from OT1 to OT4 for example the spin packet and then it runs back and between OT1 and OT2 it gets encrypted all the configuration I put statically on there and in OTBC there's a special install tg-set it's called site yes and there put the configuration into it yes so the old day they get the same configuration and everything is set up just log into OT1 it's like make regress more or less in the special directory and then just run cool because I was to make this mass penetration cool more questions here thanks for the talk I wanted to ask you we were doing we're also doing a network testing well we're doing the network testing but in similar environments and we wanted to I want to ask if you consider to use IPMI interface instead of the power switch we used the power switch before but now we use IPMI because it eliminates lots of cables and I need to maintain another device basically thanks I got to ask about the IPMI interface to by other developers but to be true the machines we're running they are so old they don't have an IPMI interface that's the reason but for Spark and for newer i386 machines you're right IPMI would be a nice interface at the moment we are running good with this power switches it's okay but yeah but you're right there's a lot of cables any more questions