 So let me welcome you into our tools behind rolling release distro meetup. My name is Christina Straitava. I work at SUSE as a team lead of the packaging team. What is a team that is handling, I would say, thousands of packages in SUSE Linux Enterprise and in our Open SUSE distributions, Open SUSE Deep and Open SUSE Tumbleweed. And Tumbleweed is the rolling distribution we will today talk about. So in this meetup, we would like to discuss tools that we use for building and for testing Tumbleweed. And it's OBS, it's Open Build Service, which is our build tool where we build basically everything. And then it's OpenQA, which is a framework for automated testing and it helps us to provide as stable as possible Tumbleweed snapshots. Because we know that probably we can have like mixed audience here. So we will try to cover both parts. Like if you've never heard about those tools, we will try to give you some short introduction what it is, why it's important and so on. But then we can also discuss your thoughts, your ideas, your questions, your problems. So feel free to ask questions. If you have any, you can do it directly via microphone, just ask for sharing your microphone or you can put your questions to the chat and I will try to monitor the chat and bring questions to our speakers. So when I'm talking about speakers, so we have currently here Martin Puskal who is a QA person of an open source enthusiast. He works also at SUSE. And we have also Chris Divan here who is a free software enthusiast working an OpenQA and they also work for SUSE. We have also a few other people in the audience here who have something in common with OpenQA or OBS or they at least work with that. So I think that Dan Cermak should be here. He works at SUSE as a software engineer in development tools. He's QA enthusiast and links distribution contributor. We have, who else is here? Probably Luboszko Cotsman, he's here. He's release manager for Open SUSE Lee. And so he is also ex-release engineer for HAL 6 and HAL 7. So he has a lot of experience with various ecosystems. I'm not sure if, yeah, also Varen Kashbetsova is here. So she's a QA engineer. So she has a hands-on experience with OpenQA. So she can discuss how it is using OpenQA every day. So regarding the agenda, we split the whole session to three parts. So in the first part, we will start with rolling this through topic. So what it is, if you've never used rolling distribution, so we will try to cover that for you. What are the biggest challenges? What is our workflow and so on? Then we move to OpenQA where Chris will describe what it is, why it's essential for us and what are our development plans for it, for example. And the last topic will be OBS. So we will go back to Martin Puskal there and he will describe how it works and how we built basically everything there. So during the whole session, feel free to ask. And I think that we can start. So I will hand it over to Martin Puskal. So hello, I'm Martin. I'm working as a QA person in SUSE and I'm also a long-term contributor to Open SUSE, not only in the packaging, but also in some other areas like membership committee and so on. So I actually see some of the people here whose membership I approved, so it's nice. And so what we are here to talk about is rolling release distributions. What are rolling release distributions? I would say I would start with a story of my background, how I got into rolling release distributions, how my edition for the rolling release distribution started. Many, many years ago, I was just a junior Linux user and I was using things like Debian and I wanted the latest and coolest stuff and it was not really that compatible with using Debian. So I started to try to build some stuff on my own, but it was also not very convenient to rebuild everything and reinstall everything. And I was hitting more and more obstacles in the old tool chain on the stable release of Debian back then like 15, 20 years ago. So that's how I got into using of the Gen2 Linux, which was pretty popular back then and you build everything on your own and you have the latest, coolest stuff and you could just customize everything, how you liked it and so on, which was pretty cool. And this is something that I kept doing or using for many years until I joined SUSE and I figured out that I don't have to rebuild everything that I can instead just install open SUSE Tumbleweed at the cost of some level of loss of customizability. Obviously, if you are building everything from scratch, you can really fine tune what you are building, how you are building, you can choose the optimization level for your hardware and so on. But on the other hand, it actually saved me a lot of electricity and a lot of time spent building packages. And I figured out that if I don't like something in open SUSE, I can just easily change it to the way that it works for me and hopefully for others as well. So that's how I got to open SUSE Tumbleweed and up to now, I'm still using the rolling release open SUSE Tumbleweed for things like some of my workstations, some of my server-like machines and I'm quite happy with that. It's worked so far pretty fine for me and it's been several years since I'm doing it, like six or seven, actually, I would say. And this brings me to what is the advantage of rolling release? Why would someone be interested in having a rolling release distribution on their workstation Raspberry Pi or server or whatever? That's, as I mentioned very often and nowadays it's, I would say getting more and more pressing that you sometimes really need the latest tool chain to have the latest cool software running. You need latest compilers, you need latest libraries and so on and trying to make things work on stable long-term supported distributions. It can be very tricky. Like I would say that some of the companies that are behind this event and even behind my employment are making a lot of money from supporting old long-term releases. However, it's still, I'm very aware that we are all aware that it can be very tricky to have latest stuff running on old releases and that's why you can, as an alternative you can switch to some rolling release when everything will be latest and coolest. It of course brings some issues on its own. Like some of the stuff is not really following the development. So if there is a new compiler and you will encounter some new more strict checks, you have to deal with it somehow. Well, not you as a user, but usually someone who is behind the distribution. And of course it can happen that some of the software will not really work, but that's again, in the long run, it's an advantage that you can actually report bugs early to the upstream developers and ask them to fix them before the other distributions hit the same issue. Which also brings me to some, like basically a story, a friend of mine is working at a company that is streaming videos, something like Netflix, but not Netflix. And he was dealing with the issue that they were encoding some of the content on some high performance workers and they were running them again on some stable Debian. I'm sorry to Debian users here. I don't want to bash Debian, but it's like a good example of a conservative distribution. And of course they were interested in encoding it with latest FFM pack and they preferably wanted to have the FFM pack as much optimized for the hardware they were using as possible. And it was a pretty tricky with the old Debian. So they ended up having to rebuild GCC to build their customized FFM pack binary for their specific payload. And when we were talking, I was asking why are you not using something that is not like 10 years old? Like, wouldn't it be easier? And I think that they upgraded to later Debian, they still haven't switched to rolling release, but they are on their way to using something like this because for example, in this case, in this scenario when you are anyways doing a lot of development on your own, I think the latest environment and it's quite a big advantage. And you are also not in a position of like a conservative company with a relatively limited manpower that just wants to pay someone for the support and don't deal with it. You have plenty of people in-house that are able to deal with issues, but just don't want to be, you don't want to have your hands tied behind your back by using old stuff. So that would be one example when using of a rolling release can be beneficial not only to the end user enthusiast who wants to use bleeding edge, but who wants to really have latest stuff because they really need it. That being said, what is specific about open to set humble wheat from other rolling releases? Obviously, I would assume that plenty of you know or have some experience with Arch Linux and maybe even with Gen2. And there are other rolling releases. What is specific for open to set is that we are kind of proud of our quality assurance that is done via means of open QA, which will be discussed later by Chris and even it has a special talk even later by Dan who is here in the audience as well. And we are building everything in our own build system open build service which I will also cover a bit later. However, what is important about this is the quality. As I've said, I'm using open to set humble wheat in a lot of spaces. What I can say is that it's happened to me like once a year that I had some issues that I had to manually solve and like search for what to do with them. And I think that in most cases it was like some hardware driver specific things that may be where if I recall the last thing it was something with GPU and the easiest solution was to just roll back to the previous or just boot using the previous kernel. So that's like not like something and most of my colleagues even at work are using open to set humble wheat nowadays and they are able to work like 99 1.9% of the times. If they are delayed they are not delayed because they are using humble wheat but because of something else. So that's just a very brief and generic introduction to what a rolling release is and what is open to set humble wheat. But what I would like to do now is to ask our audience if they want to share something or if they have some comments about rolling releases some experience is positive, negative or if they just have some questions and whatever just please, please speak up. If somebody's I'm not so comfortable speaking up you can use the Q and A we can read it. Or just the chat like Luna did and that's actually four computers. Wow, you have a, you have either a lot of time or you're very good with Arch Linux. I used to run Arch back in the day and boy it broke every few months. So I stopped. But as I also wrote in the chat you tend to learn a lot from Arch Linux. So one thing that's definitely superb about that destroys the documentation. Yeah, Jan has one concern, right? Who wants to respond? This is very common complain. I don't think it's a concern. It's a concern. It doesn't look like that. That's not a concern. The updates are gigantic. I hear it on every meeting guys, right? Yeah, he even confirmed it himself. So I mean, yeah, the updates are gigantic but they never crash. And that's partially also thanks to OpenQA because broken snapshots aren't published. If something breaks, the installer breaks the distro it never reaches you. Which kind of is the point of a gating pipeline? That's why we are not rolling as just pushing the packages as they are uploaded but we are more like releasing snapshots but pretty often. So in the snapshots when they are released they're always tested by OpenQA so they really should work in at least basic scenarios but we can later look at the OpenQA coverage as well. And that's interesting. Sometimes OpenQA release is like a huge, huge updates that are often caused by triggered rebuilds and they are usually triggered sometimes not even automatically but sometimes manually because of huge tool changes like G-Lib C-Onc or compiler that it's considered to be better safe and just rebuild everything. People are sometimes a bit surprised that they need to update 2,000 packages once the snapshot is released but it's usually not a problem unless you have a very bad network connection. So we have one comment from Luna about OpenQA. OpenQA is awesome, even the other registers are using it like Fedora and maybe even Debian. So I think that maybe we can jump into OpenQA part. So if Chris is ready, so we can start. Yeah, so you could probably consider that a bit of a spoiler. I was prepared to mention a few examples of OpenQA which is known and also Debian are some more recent additions to the family but let me start from what is OpenQA? So for those of you who don't know anything about it I'm going to give you a little metaphor to work with. So it is a testing framework, right? But think of a human tester. What a human would do to find out if, for example, if the distro works, if soft ground it works, what would a human do? Human would go and install, for example, using a USB stick or some other means and run the installer and start typing things and selecting things on the screen and going through the installation process until they managed to install the system and then run their own applications. And now imagine OpenQA doing that in much the same way except it's synthesized. So any of the input is, of course, sent through the backends and mouse clicks and such are synthesized as well. And there's, in fact, the video stream that's running that tells the job what it's actually running. So rather than somebody looking at the screen, you can see that, see as in record the video and find out if what's expected is actually going on or maybe something else is happening that you weren't expecting. And I'm going to show you on my screen. Let's see if screen sharing works still. So I think you should be able to see the OpenQA web app. Yep. And I'm not going to go massively into technical detail. I'm going to keep it at the base overview. Of course, if you have particular questions, feel free to ask at any time. And yeah, so the web app is probably one of the main ways that people interact with OpenQA. So what you can see here is an overview of the groups which are certain distress of OpenQA that are set up in this instance. And you can get an overview of how many jobs passed or failed, for example, or you can see in some instance what's still running. Because of course, depending on the amount of hardware, you cannot run everything at the same time. So this is where the scheduling feature of OpenQA comes in as well. And let's maybe take a look at an example build. So you can already see there's some failures. So imagine as a test reviewer, you'd be checking this to find out, the current state of it or all of the test passing or if not what might be the problems that are currently there and need to be verified manually. So for example, this is the job that passed. So this is of course the best case, the jobs run through and you can see what's called the different test modules, such as for example, boot to desktop. This is a very simple example of you booting up the system. So imagine again, from the user perspective, you have your system installed and you're hitting the power button and you're looking at grub for instance, and you can see that it offers you to run up into the tumbleweed. And the test records this screenshot, or more specifically it's called a needle in OpenQA, which tells us that this is working correctly. Because if this weren't showing up, it would suggest something is going wrong. And yeah, maybe to go a bit further, you can see there's a few other tests going on. There's also some command line based tests, but maybe let's have a look at something more graphical. I see there's some questions in the chat. Maybe we should check that out. Yeah, we had one question from sleepwalker, but it's more like general question. So we can go back to it later. All right. Yeah, so I'm going to look for an example of some, maybe some GNOME examples. And yeah, GNOME shell being the default when you install up into the tumbleweed. Here for instance, you can see in the test that this is going through the full installation process as part of the test. So this is also a case that's used a lot. And I'm just going to go by example, the steps to show how it generally works. So you can see the screens you would go through. As a user, you can see the installation process. And at every step, you can verify that the process works as you would expect from what a user would do as well. And you can even see the lovely, oh, this is open to the welcome screen. Meaning of course that you successfully installed the system. However, let's also take a look at something that didn't work well because, you know, this is ultimately where QA comes in. You need to be prepared that something doesn't work. Eventually things fail. So let's take an example. So here you can see that a LibreOffice test failed. And it failed opening a file apparently. So if we're looking at the test, you can see that there's a file picker there. And the test is sending the commands to open a file in LibreOffice. And if we go through it, you can see this is the current state of LibreOffice as it's being used there. I'm just using my key work, by the way, to go through the sections, which is very nice. I think it's quite nice to use with different input methods as well. Yeah, and now what happens? It seems LibreOffice is gone, right? And in this instance, it's a symptom of a failure. And I'd say this is a, I'm not gonna do it specifically why this happens, but this is a common case, something happened and something causes the application to crash. So you would notice in the test that the next screens are not showing up that you would have expected to show up in the test. And yeah, maybe as another example, since OpenQA can also test command line applications, here's an example of something that is not graphical at all. Let's see if, just open some, no. Unfortunately, it's a bit delayed now. Yeah, if there's any questions in the meantime, feel free to ask. Actually, we have one question here in Q&A section. So Jan-Maras is asking, so with that screenshot as something to check against in the test, similar test could probably recognize resolution issues on some non-standard layouts, for example, white screens. Resolution issues. That's actually a rather tough one to crack, unfortunately. So OpenQA is by, so the common way to run OpenQA is in a VM. And it's, as far as I know, it's usually expecting to run in a 1280 times 768 or something, or 1024 resolution. And if you do it something else, it tends to break, which for instance, the cubes OS developers recently ran into. So resolution detection isn't actually, as far as I know, working that well. You could test it, but this is a little bit of a pain point. Yeah, so I'm not aware of specific cases for this sort of scenario. I would be interested though, like if you have a case, something you want to test, I'd be interested to look into how to make that work. Yeah, so this is an example in the meanwhile of a test that is not graphical at all. So while OpenQA is especially good at doing things that require user input, this lovely example of a Rust test. So what this does is actually it provides the running system that say a developer might be using to set up a new project and call commands to install Rust packages using cargo and such. If you don't know anything about Rust, that's not too important here, but this is just an example of something where you have the base system running and you have for instance, proper networks access provided and you're able to run a command line application like you would expect. And yeah, you can see in this instance that these are just the console outputs here. I don't know, well, you'd be able to read it through screen sharing, but you would see that it's actually typing cargo new for instance and showing the output from that. Yeah, another thing I'd like to show is since reviewing failures is quite important. We have some really good features in OpenQA and this has been improved a lot over the last year to investigate failures and it's aptly called the investigation tab. It's unfortunately a little bit slow at the moment. Yeah, I'm gonna tell you anyway some of the basics it does. So you have a few means of for instance, comparing if previous runs failed in similar ways and maybe passed to identify what kind of failure you're looking at or if there's a new failure or maybe there have been some changes to the test setup or for instance, depending on the hardware you run tests on. So OpenQA can run tests on a lot of different types of devices. The very common case is of course QEMU. So there's a lot of virtual machines running. There's also bare metal. So you might have ARM servers or you might have Spark or PowerPC or whatever you need to test in particular. Might even just be a Raspberry Pi. So if you were looking to set up OpenQA, the baseline is actually quite low and it could be as simple as installing OpenQA on your computer that you use for development. And what you need then is just one host which runs the web app, the scheduler and the database which is what you're looking at here and one worker as it's called. So this is the part that runs the actual job and does the heavy lifting and that later on reports back to the web UI what the result of the job run was. Just checking the clock. I guess we have another five minutes for this lot. Yeah, maybe since you mentioned the Raspberry Pi, if you are really quick, you can take a look. There should be bare metal tests running on a real Raspberry Pi somewhere in Guillaume Garde's closet or desk. I don't know where he has his set up currently lying around but on Tumbleweed AR64, if you go to that job group. Yeah, exactly. There should be, I think the juice, I think the juice tests, there are some actual bare metal tests. These ones I guess? Ooh, they're right. Yikes. Oh, maybe let's take a look at a passing one. See this one? Yeah, I mean, it should be visible via the variable so it will not look any different because the actual, the only difference is that it really runs on a real Pi but the connection is still via VNC so there's no actual video recording in there. That's what the cubes guys are currently investigating. So I think if you check the test settings, you should be able to, there should be some general hardware settings or so if those are in there, then, oh yeah, it's a general, that's running on bare metal then. On a Pi 4. Oh, very nice. Yeah, so yeah, okay. Looks totally unimpressive, unfortunately, but this is running on real hardware. So this is, in my opinion, this is one of the big things that OpenQA can really do because it's not running in QEMU, this is running on a real Pi. If you have some incompatible U-boot or whatever you nowadays use to boot something of a Pi, if there's some incompatible change, you're good luck finding that with QEMU. It's possible, but it's ugly. But if you test it on real hardware, if it breaks, it very likely breaks for everyone else as well. So plus one for gating. Okay, so I don't see any questions in QA or in chat. So is there anybody who has some question would like to share or some faults or anything? Because if not, we can move to the next part, which is short introduction and discussion about OBS. I have a question. Can OpenQA test other operating systems, not only in all distributions? Oh, yes. That's a great question. OpenQA can test a lot of stuff. So you can even test Windows on it, if you're so inclined. And there's actually Windows tests that are run in... This is for Lubos. Exactly. We actually do test WSL. And our opens is a leap images in it, and it has to be tested in the Windows 10 VM, or 11 nowadays. So yes, we do that. And it works. Thank you. Let me maybe take this one from the audience, whether OpenQA can test mobile Linux, like the PinePhone, the LibreM5. So yes, in theory, you could test your PinePhone with that. I've looked a teeny tiny bit into that, but it gets a little bit tricky. So one of the things is OpenQA wants to run... First, you want to run inputs into your device. And with a PinePhone, it would be really great if you could actually type on it, which that gets really hairy. So emulating a touchscreen via some, I don't know, mechanical fingers that's going to suck really hard. So that's not going to be too easy. And then you also have with these things a surprising issue. And that is, if you create a busted image and it hangs during booting, you want to be able to kill it. With a PC or a Raspberry Pi, you have a rather simple way, and it's connected and you do that one. This has a battery, so you're going to wait for a few hours. And so you could make it run. And I'd say if there's a distribution that's really committed to producing high quality to continuously test it, you can do it. So you can, for instance, unscrew something like this, connect a USB, connect an SD card multiplexer, boot off the image and solder a few cables in there that you can really unplug the power supply. It's possible, but it's a ton of work. I would have comment on this as well. In most cases, you actually don't need physical hardware. I am aware that as one of our OpenQA developers was actually sort of cheating in some Android game by running Android image in QCOW and just playing the game via means of OpenQA. So as long as you can run whatever you want to test on the physical device in some QCOW, you can test it actually pretty easily, I would say. Or actually cheat in the Android game or whatever you want to do. We have one question in chat here. How does the needle matching work? How precise is it? Can two broken pixels break the whole test? So I'd say the answer is no. Yeah, see, Dan's also commenting on the, yeah, actually, if Santa is here, feel free to chime in. Otherwise, I'll try to answer this. So, we have something. Bad news, Santi. I can't hear you. Not sure about address. No. No, no, it's like super quiet. It's not the volume. It's the probably incorrect device. Just go to sound settings. Just open, you open your mouth and I speak for you. I could be. How about now? No, yes, okay, cool. So long story short. I throw in a needle in case you want the visualization. Amazing. Thank you. So because OpenQA, so it's basically a program that is only taking a look at whatever is on the screen. The backend is using OpenCV. So there is an algorithm that is going to take a look at the image and is doing some normalization operations. Then it takes a look at our reference. So this is basically a JSON file with XY coordinates. And then there can be multiple areas of something that we want to look at, something where we don't want to look at. And then there are multiple operations that can be done over those areas. So what OpenQA is going to do is, it's going to look at the VNC, take a screenshot and run it over save algorithm in the backend. When this is happening, it's taking out the difference. So it's going to look in that new screenshot, the reference and then it says, okay, so here in this very small amount, there is the button and let's say that we want too much hello and everything is black and white. So there, because we have only black and white, you have the problem of what happens if somebody adds one pixel. And this is where it becomes tricky. So we have a much percentage. So when the image is basically kind of normalized, you get a certain value that is then converted to from zero to a hundred percent. And you can define at the moment of the creation of the needle, how big the pixel difference can be from what you get in the screenshot to the reference. And from their own OpenQA will tell you, yeah, this looks like more or less something similar. So let's say somebody decides to change black and white and inserts a gray, then if you have a low tolerance then OpenQA will say, yeah, this looks okay. But if you are having a higher score, so you want to have at least 90% of match, OpenQA will tell you like, these are two different patterns. It happens very often, for instance, that we are having text that is being changed from bold to normal, to not bold, basically. And depending on this matching level, we will have more tests failing than in other cases. But if you- So you want to make OpenQA hate you, then change the default font. Yeah. Or wallpaper as we plan to, right? All that. So we are slowly running out of time. So if you don't mind, I would switch to the last part of this meetup. And that's OBS. So Martin, if you can start just some short intro to OBS and then we can go back to the last part if you can start just some short intro to OBS and then we can have discussion. Thanks. So basically what we are using at SUSE and Open SUSE and some other projects as well is that we are doing is using OpenBuildService as a tool to build our packages and other things, not only packages, but various container images, ISO images for KVM and so on. It's in a very brief description. And let me share you, share screen and show you what actually, what you can build as in, if you are just interested in building packages in OpenBuildService. That's actually a nice example is the OpenBuildService is accessed via either web UI or command line client or API. And the command line itself is somehow built in project in OBS. Project is something that is like, imagine it as a directory containing packages. So project is Open SUSE tools and package is OSC, Open SUSE client. And this project is quite a nice example of what all we can build in OpenBuildService. It has basically enabled every possible Linux distribution as a built target that it can build. So apart from RPM-based distributions, which are built from OSC spec file for, then we have, we are building for Debian-based distribution. We are building for arch package management based distributions and we are also building for some up images and we can also do some flatbacks and so on. And upbacks for Windows as well. Yeah, that's, I did not know that actually. That's how I learned. But let's just quickly scroll because it's the list is too big to discuss everything. Like CentOS 6, well, CentOS 6 is failing. Debian, Debian, Debian 6, well, it's already unresolvable because it's some new stuff, but there is a plenty of distribution built targets here available and plenty of them have a lot of targets enabled. Like for Magia 8, we are building for 32-bit Intel, 64-bit Intel, RV7L and AR64. So a lot of Raspbian as well as a built target. I always learn about existence of absolutely obscure distributions from here because I still don't know some of them. Like Exubuntu, I think it's not the longer relevant, but yeah, but there are like weird strange distribution open Euler 20.03, I have no idea what this is, but I'm not sure if I want to know what it is actually. What we can build in open build service is not only the images and containers, but we are building like ISO images for distribution. Like when open Sousa Tumbleweed rolling release is created, it's also like from packages, the ISO image is created. The ISO image is usually the thing that is passed to OpenQA for testing. So OpenQA usually works with an ISO image and it just goes through some installation or there are some other scenarios, but for the sake of time, let's assume that this is more advanced topic. So yeah, but we can build like ISO images, KVM images, open container images, lots of stuff. We have like JOS, this is like just enough OS image which should be building just the KVM images for various arm flavors. Like yeah, it just creates a raw image which is supposed to be copied to the boot media and so on. So we can do a plenty of things in open build service. Apart from building for various distribution, as you could have already seen, is we can also build for various architectures by default or by the most common use case is that for various architectures, we actually have a physical workers that are doing the building. So like for X86, 64, obviously we have an Intel machine for PowerPC 64 little and then we have a PowerPC 64 little and then for ARM, we have ARM workers. And in most common scenario on these workers is that we are running built inside of the KVM environment. The open build service can also work with system D and spawn. I think that there was some experimental support for Docker. It can also run in change route environment. It can also run built in a KVEMO environment. So you can actually build for other architectures that you physically don't have access to which is pretty good for adding support for a new distribution or new architectures to your distribution. Like we have some experimental building for RISC-5 in open build service as well. We also had some experimental support but it was more like a pet project of one of our colleagues that we were building open to set umbilveed or attempt to build open to set umbilveed for M68K. I hope I'm pronouncing the architecture correctly but this is the old RISC architecture that Mac computers used to run 25 years ago. And it's, the hardware is still manufactured but it's not really powerful. And usually people don't run a generic Linux distribution on it but you can find it. And so this is something that we, all of this can be done in open build service. If you would be interested in deploying open build service, the open build service actually has a fairly, fairly good documentation. It has its own project pages, open build service. And if you want to have just a very trivial setup, you can just download the installer for the appliance. Obviously it will not create you the huge build environment like we have in the build open to set.org but if you want to play with build service, you can do it via this way. We have also extensive documentation for the build service available as well HTML and you can learn how to add support for support build receipts and package formats, what is supported, how to set up various things in the environment and so on. Or if you don't want to install open build service and go through the configuration and setup and importing the packages to bootstrap and build target and so on, you can just simply use the publicly available open to set one like the one I've been showing to you, the build open to set org. Let me share it again. Where everybody is free to create an account and create your own project and set up your own build targets. There is, as you can see in the OSC example, there is a plenty of available targets. If you would want to build for some distribution that is not yet available, but it's some build kind or type that is supported in general by open build service, you can just ask the build service developers or maintainers to just import the distribution for you as well, it's something that can be done as well. And you can just create your own project or ask for a project to be created and you will get something like working directory or workspace where you can play with your software. So we have like here for GNU Health, which is a set of packages for basically monitoring of health things. It's like something that you can see at your doctors when they are entering your diagnosis into computer. We have like special projects here set up that are subprojects that contain all the necessary packages and they are maintained there. Sometimes the upstream developers are using actually public build open source org to verify or to build the packages for them, for the things that they are creating. So if you are interested in any of this, you can just go to buildopensource.org and try it in some environment that you just create for yourself. Let's see, there is like a arch core, but it's import. Let's see if there is some nice cloud development. There is some interesting example, this continue distributions. And basically when the distribution as a build target is imported, it appears as one of the projects in build service. There is not much magic behind it. Some KDE test projects for KDE testing and so on. Martin, can I mention one thing? Yeah, for more minutes, maybe you've said it because I had to reconnect and because of bad connection, but imagine that you just want to rebuild your own tweak of the image. So it's super easy to fork the package or the entire project and you can just do small modification and rebuild the image by yourself, which is pretty cool. Takes just few clicks. Yeah, and there are actually people that are doing this. There are some derivatives of open-source being created in build service. That's right. Yeah, and there is also, there was some person that was creating, it was like Hacker's son's frontiers and he was creating some custom images of open-source at Humbleweed for that contains some stuff that was basically like privacy-oriented and somehow to be, so basically it was for people in the countries that are not so fortunate as our country is, or the country that I am that have to connect to the Wikipedia through Tor and stuff like this, so that can be done there as well. And I think that as a starting point, anybody who wants to do it is just look at the open-source at Humbleweed and start with the key descriptions there and just tweak them according to their needs. And yeah, it's actually, I have to admit that I've never tried it because I never felt the need to do this, but it should not be that difficult. It's really quite straightforward. Good example is you receive some obscure hardware like a new ARM-based laptop that may not work out of the box and you want to play with the latest kernel and do some out of three patches that's exactly perfect for this. Yeah, so basically that's for the basic introduction of the open-source service. There is plenty of things that we haven't covered yet, of course due to time constraints, but if you have some questions, comments, ideas, complaints, feel free to use remaining a couple of minutes and ask them. Q&A has a question from Jan Maraz, but I feel like this was maybe answered. Can you double-check? Yes, yes. So we have only one minute, so maybe we can just, if you have any questions, feel free to join us in a work adventure instance. We will be there and we can answer your questions. Keep in mind that there is a limit of four people, so we have to, maybe we should stand a little bit further from each other and, you know, so we can actually ask. Okay, sure. Okay, so I think that we can move to work adventure, so thank you everybody for sharing your wisdom here and for all questions. And that's all from us, so thank you and have a nice day. Bye. Bye-bye. Thank you, bye-bye. Bye. Okay, thank you. Thank you to all speakers, all the people participating.