 So, welcome to the workshop about the Always Operating System. My name is Petr Fichl and I'm here with Dominic and Miro. We're going to guide you through the workshop. Just a brief overview of the agenda. We would like to explain the vision of the Always Operating System. Then we would like to do a kind of high-level overview of how the things are working and then write down the examples to show you some tangible stuff, how things are working so that you can start playing and experimenting with this stuff if you have a laptop with yourself. But at the start I will give word to Dominic for a short introduction. Yeah, thank you, Petr. So, I'll just stand close to you so we don't lose this. Yeah, Operating System CI means we do CI for an operating system. You can see us in the Fedora space. Fedora CI do things there. There's documentation on the wiki. We try to keep all this public and internally inside Red Hat, we're called Operating System CI OSCI and basically do the same thing. So, the vision for us is with Operating System CI and Always Ready is that we don't want to be in a world where we develop things, then we say, okay, let's stop and take this and try to ship it so we can stabilize it, test it and see how it actually works out and then eventually say, okay, this is a release. What we want to be is that at any given point we can say what we have right now is good. So, instead of writing a new feature and then testing it, we say it's not actually finished until we've tested it and we know it works and we can ship it all the way through. For us in Operating System CI, that means all the way from the upstream through Fedora, that's the visible part. So, we want to make sure that there's a solid connection that when features are developed, you think about the actual end product where it's going to end up and how it's going to be used, which is a good reminder because we want users for our software, right? So, it's not just about the new features but seeing how they're used. So, you can reach us under the CI mailing list in Fedora. We have the wiki and we also hang out in Fedora CI on RC and you can reach out to us personally and you may have seen things like the CI pipeline and we work with Fedora infrastructure, for example, on the Fedora raw hide gating, which will apparently be a thing for real this year. So, fingers crossed on that. But, yeah, for those of you who've done some package testing in Fedora, there was a time, I think it was last year, where we wanted to introduce testing to all the packages and there was some mishaps with the user experience there and so there's some gaps there that we are aware of. But, I think we've gone through and addressed a lot of those, so there will be kind of a reboot on the user experience this year. And, yeah, this workshop is about if you have questions on how to make things work, how to tie in the processes that you have. And, yeah, tie those things together and make sure that we have tests for all the packages. So, that's about all the intro I get. Now we get to the real interesting part. So, at any point, I think this is a little more about being interactive. So, if you have any questions, just feel free to interrupt. Please do not wait for the end. I know that we are at different levels here regarding where we are with the testing. So, we are a couple of people here, so we can also spread out and answer questions directly if you have any. Thank you. Okay, thanks for the introduction, Dominic. Before we do the high-level overview of the staff, because the internet connection here is slow and it would take some time to start preparing a laptop for experimenting, we've prepared some container images for you. So, if you want to experiment, you can just make sure you have Podman or Docker installed on your laptop. And then you can copy the file from the USB key and then use the command here, the Podman and Podman load to load the image and then the final command one to run in the container. At the start, Dominic already mentioned this portal. So, fedoraproject.org slash wikisci is the main portal and from that you will get to everywhere. So, if you want to write this into your browser and on that page you will find all the links, including the link for this workshop. There is a special project at the bottom of the page. So, please bookmark this page and you will also find these instructions for copy paste available on the workshop page. So, now about the vision. Dominic shortly already mentioned what is our goal regarding not like waiting with the testing for a later stage, but maybe to pinpoint once more, Operating Systems CI, what is it about those really very big number of packages which are distributed together and we would like to make sure that everything works together and this is very hard as the number of packages grows. So, our vision of the future is that we would have this always very operating system where packages would be constantly kept in a good shape. There would be extensive test coverage which would be regarding the features and also the integration between the components and thanks to gating no change which would cause some problems or breaking some features shouldn't be allowed to be merged into the code base. And this should finally result in this that we should be able to release new versions much more sooner or even continuously. So, the high-level overview. So, there are three main blocks or pieces of the puzzle which make this working and these pieces consist of more little pieces which implement these different features which are necessary. So, I will start with the process. So, we start with the standard as interface. For all of these I will do just a very brief introduction, very brief description and anytime you are interested in more details or later during the workshop we can look into some of these but we don't want to spend however the time of speaking just but then you can experiment and play. So, the standard as interface specification which describes how the tests are discovered and then executed. Basically it says define some basic terms like what is test, test suite, what is other these relevant terms and what files has to be present where in order to be executed for the CI. Then the gating concept as already mentioned if there is a change which after test appears that it is breaking the package or other packages it should not be allowed to be merged. So, the package is effectively gated before doing some next step in the release pipeline. Notifications we would like to there is a federal notifications service which can be used to configure what notifications you would like to receive. This is something we would like to improve and thanks to also the consistent format of messages it should allow us to nicely configure from the user experience to receive only what you are interested in. Then the tools section. There is standard test roles which is something like an implementation of the Sundar test interface specification. So, it's a bunch of Ansible roles which makes it easier to run the test and provide the output which is expected or which is described by the spec. We have some efforts around metadata to have some efficient way how to store this metadata preferably directly in the very close to the test code directly in the Git repositories. We have flexible metadata format proposed for that which has some nice features. Then there's the pipeline which takes care of like detecting the changes and after it finds out that there is some change it will execute the test and prepare all the stuff but we will speak about that later and the peculiar as for the modern pull request workflow you probably know what is something like GitLab or GitHub. Then the last but crucial part are the tests themselves. What test types do we expect to be used in the Federal CI? We believe that most of the tests should live as close to the code as possible. For example, unit tests, it makes sense to have them directly together with the upstream project directly on GitHub for example but for when you are packaging the component for Fedora you would probably like to make sure the basic functionality is okay and that the component nicely plays with the other components. Probably this basic functionality or integration test would make sense would make best sense here. The test code stored directly in the repositories where the spec files live or dedicated space for the test in the test name space or even on other places and fetch from there. For the test execution we will be speaking and doing some practical steps so I will skip that for now. Okay. We are going to talk about the environment for the test like if it is a closed-time part where you are inside in the free-sense you could I am thinking of a package where I would like to use the nice map to create a device if that is possible what I would have to do to get an environment back and we can judge it. Okay. The pipelines I will be speaking about the CS system that runs the test name. It is hosted in an open-shift instance but it runs a VM inside a container because it is an unprivileged container so it is a VM where you have root and you can do basically anything if that answers your question. It is a VM. It is a 64 VM. Yes. It is a proper VM currently it is only a single host so you cannot easily be able to spin up multiple VMs but I believe we will be able to deliver that in the coming in the coming year. Is that clear? So it is kind of resource-price if it is a system virtualized VM We currently do not have any big performance issues given the fact that we have how many components we are testing so currently it is not a large portion but we can scale it up it is open-shift. It should be as easy as adding new nodes so the scaling up should be easy. Of course I would maybe even just add that we are planning to also support running tests directly on a Docker container, right? If it makes sense because not all components make sense to test in that environment. Just a question because we are running an unprivileged stock container I am wondering what about the infrastructure tomorrow if you want to use a SCI copy? I am focusing on the infrastructure that we use to support this So I want to point out that this is the default implementation so we have the whole concept of those tests and they should work more everything is plugged together and the pipeline that we have you can put those if you don't want to worry about it yourself if you have special needs you can either work together to extend this or you can hook up a system that you already have so you don't have to necessarily put everything in here this is really about the default implementation to make it easy if you don't want to worry about setting up your own infrastructures scaling up, working with open-shift or anything like that this is what you do Okay, let's continue because the time is ticking so we have enabled testing the CL pipeline is extremely able for all available or stable Federa releases and also Rohide Yeah, we are taking tests for the corresponding branch so if you are testing if you have the tests in Federa 29 they will be executed for the 29 build even scratch build so we should be working automatically but you can discover that the source code of the pipeline itself is on GitHub so if you would like to take a look how it works you can take a look there so for the build pipeline this is testing Koji RPM builds including scratch those are the links actually to the pipeline I can click it so currently for the blue RPM builds this is the only way how to discover that the testing has happened so if you build some scratch build and you have the tests this gives you a repository here you can find the results as you can see something is running here and this is the pipeline itself maybe I'll just show the artifacts where you can actually discover the tests how they have been run so for example this is a bus that runs a Bicolib test I'm going back to the presentation Oh, yeah, backlinks I guess so for each federal version there is a separate pipeline can you go back please so there is also a raw height pipeline there also we have poor request workflow so it means you can submit a poor request to the disk git and it will from the PR applied to the head of that branch build a scratch build and it will run the test on it so this is even before you do anything change to the production version it will be on the next slide the Pager integration itself so this is how looks currently the integration with Pager so here is a poor request to the disk component here just for testing and here actually here in the flags the tests that have been run simply because the CI is something not managed by us it actually builds the scratch build but currently we don't use it the CI also builds a scratch build and runs the tests that are there even if you add it with a poor request this test will be run and we are adding a pending status so currently you could see only the result once it completed it could take several times so we have there now a poor request to loopable which is a thing that updates actually the flags according to the messages on federal infra to have the pending state there to like retest the poor request you can just add a command in CI test and if it rerun in case you would need to rerun it for any any reason so there is possibility now for this for the build pipeline to actually get both the updates both the update doesn't get to the testing of stable repos if you add a gating YAML file to the disk git there is an example example here for the file component that is currently enabled for federal 29 this is this needs to be in this form both the update push state because this is like the event that it should like git you can change testing it doesn't need to be it can't be pushed to the testing repos if the test didn't pass and this is the name actually of the build pipeline that we currently use this might change in the future once we go to unified messaging currently you can just copy over this to your federal to your federal disk git branch and it will gate those tests that are there in disk git and here is an example usually if you have both YAML this test gating here is a little too small but usually you see there are no tests required so there is no gating enabled but when you have gating YAML file you can see that actually all tests passed this is a little bit different here discovering which test actually gated the package because in automate tests there are also several other tests being run by federal for a long time RPN Deplin and some others but it is not easy to see which actually gated the package so that needs to be some improvement okay so standard test rules as Petra mentioned is like an implementation of standard test interface this is basically what the pipeline does it executes an Ansible Babel you can basically do anything the important thing is to conform to the standard interface standard I would say so you need to create the test log file at the end and there is current specification says that the Ansible Babel should fail if the test failed we are actually wanting to change the reasons it can't distinguish between the test failure and actually the playbook went wrong even before running tests but currently there is a situation so you can actually run tests directly in Ansible that's things that people already do but you can also use our roles that make it simple to run tests via some of the testing frameworks for example a lot of red head guys use big lib then we have support for avocado test framework there is a basic role which can just run simple shell scripts a lot of people use that and source is a special role which actually doesn't wrap around the test suite but it actually makes you available the source code this is handy if you have the test directly in the source rpm and you want to run them for some reason yeah and we have tests again different test subjects like what the build pipeline is running it's running classic test means that it's passing actually to Ansible the tech classic currently Atomic is currently not used it was for the previous pipeline but currently that is really not used I would say that and for container currently we don't have a running on the container on the containers but it might be we even want to improve this tagging and stuff with a better better way how to specify the metadata for the test and for the testing itself I'm mistaken so here's a main documentation if you want to see these are really nicely structured I think better already mentioned and I don't need to do anything okay perfect thanks so with these links we will achieve most of the if you are new to this concept you would like to try something find really everything which is either on the first link which is the portal or if getting started the quick start guide is a good place to start because it nice to list all the steps which are not necessary for executing a test so let's jump into running tests so if you would like to run a test there is a definition in the standard test saying that you would place a file which is named tests.yaml under the test directory in the disk.git where spec files lie and then executing the test is as simple as calling just Ansible Playbook on this test file and the quick start guide lists all these steps so if you have a we strongly recommend for the experiments using either virtual machine or running the test in a container because the specification requires that the Ansible Playbook is run on the root because for example it prepares the environment or whatever is needed there is dynamic inventory support there so using this you can for example test all the mentioned testing containers or atomic host so that's the limitation if you are afraid what Ansible Playbook would do on your laptop it's definitely recommended to rather run this into a fresh virtual machine or container so there you would just install for example the FED package and you are prepared to do the testing with the FED package clone you can just clone the package package disk.git repository you enter this test directory and you call Ansible Playbook test summer and that's it as simple as that so if you have prepared if you have run the commands to prepare your container from our USB flash keys you can you can now do this do this testing because in the container they are already fetched also the repositories so to give you a couple of examples so there is a basic role which can be used to run a single command as a test so if it makes sense for a component it could be by like packaging and some changes might happen in the package or the environment or in the pipeline which is building the RPM package to just make sure the smoke test passes so you can run like minus minus help very very very simple command so as here we see the sorry there was a wrong button so here you see we use the standard test basic role and the text classic means that this is intended for running for testing a classic system not inside container or in a specific host and we list the tests there is a smoke test directory which should be entered before test execution is the current directory and we run this command this is just a command line tool for gathering or reports for the last year and there is like integrated simple test smoke test minus minus test which does this report from github for the last year and at the end we do just the standard packages is only one simple package and Ansible playbook together with the standard test roles will take care of everything install required packages, prepare the system make sure everything which is needed on the system is installed or if there are some extra packages which would conflict that would be removed and it runs the test so that's the very simple example of a basic law then another example from real life I think a very successful story is from python Milofranczuk started to play with it a couple of months ago and we reached this state where there is a test repository for python and because there are multiple components like multiple versions of python there are also some additional packages which are related to python many modules and such but for this example there is a simple smoke test which is called virtual env and it resides in this shared test repository using this line this repository is fetched and then the smoke test is executed and it's executed for example in this case for two python versions and you could use it for testing various more combinations here if you click on the link in the example so this is the page on ci-examples there you would see that there are multiple components covered it's about 10 or 12 python related components covered by this single test which is quite simple smoke test for the virtual and functionality but it very nicely covers all these components because it's stored in a separate namespace and it's referenced from multiple places so you maintain only one copy of the test then I jump at the beakerlib role so as I mentioned we have lots of tests written in beakerlib in the baseless query team so for the baseless components and we wanted to be able to execute them very easily so this beakerlib test role has been implemented which allows only list tests which will be run and everything else the standard test roles take care of so it checks the required packages it stores the packages and investigates these directories with the beakerlib test and executes them so that's about beakerlib example how it would look like and this is something from real life so this is bash example bash is 105 shells which are the POSIX compliant and again it makes sense to have a test for them shared in a single common repository and use them from that place here again repositories fetching the tests from this shared repo destination folder and here a new thing which was not there last year it's a flexible metadata filter it allows if the number of the tests grow if the number grows then instead of listing very very long list of tests which should be run you can use the metadata which are stored directly with the tests and from there using the FMF tool you would select I would like to run only tier 1 and tier 2 tests which are relevant for the classic for the classic test subject so that would save you save you updating if you add a new test then adding this to the list of tests which would be run to multiple repositories for example and now one more example which is demonstrating the standard test source role so here you say only you apply the source text always means it will be run for all the test subjects and it takes care of extracting the source extracting the test from the package source sources and then the standard is basic role just executes them as you see here you run the tests and you run the smoke so you don't have to copy the tests into this git repository or store them elsewhere as you have them packaged in the source rpn now I'm sure you mentioned about the B Clip if you would be interested in writing some tests for the operating system which mean like integrating multiple components starting services and doing stuff like that so this shell level integration testing library could be of good choice it has some nice functions support churnal and phases to make it very clear where the problem was like in the system setup or the test itself or only cleanup there's something to documentation and if you would like to experiment and create a simple test you just run biker wizard tool which will ask you several questions so the result is a complete test with everything with everything defined so that was a couple of examples and the overview and now we have roughly about 20 minutes to do some experimenting so we are here Petra, Mero, Dominic and also we have here Brno and Andrei who are willing to help you you would like to if you have any questions or start experimenting with that so even you could try to for example take a component and try to do a smoke test for that component which would be like calling a single command and such I would just have a question did anybody know how to use the container image or not there was a link there so in the if you run the container in the root folder there is a test directory and there are actually there were checked out repositories for two components seven at once is not a very good example there so root's home directory so you want to type cd because if you say root directory slash and I was looking around and slash so here are the instructions if you found the link so after preparing the image you can you can run it like this and then as just mentioned you can directly cd to the home directory and test sub directory and there is Python 3.6 tests already already downloaded and then with Ansible playbook you would run them so that's the command so here we have the tests we try let's try the Python 3.6 and we call Ansible playbook Python 3.6 sorry Ansible playbook test.cm there is everything there so there are some tasks which are done for all the roles for example making sure that the required packages are installed on the system then there are some additional steps which make sure for example if the tests have some special required packages for individual tests the roles would make sure that all these packages are installed like here installed a specific package requirements we see the GCC Python 3.6 R-Sync and Python 3.Tox and do some other steps and then finally the environment is ready it would execute the tests so the specification says as Mirro said if the tests pass the Ansible playbook should return 0 as return code with no problem we see here that there were some changes on the system and the final result was okay so 24 tasks were okay and here in the directory we would see directory artifacts and here you see the test log applied by the specification to have a output of the tests and test roles are quite concise here they just do here the result of the testing so I have here twice the result because I have run it already once and in the in the pass a small log you would find a complete log of running the test so here if there is any problem or anything you could have a look here could have a look here for more details so now if we jump if we jump to bash and run Ansible playbook so the page suggests you could run it with a text container so this I will show you how it looks like so the test demo looks like this there are some tests relevant for the classic test subject and there are some different tests for these the difference is in the FMF filter so from the test you will be selected only those which are tagged by container or atomic so if I run the playbook minus minus tags container and I am running in the container so we would get only those tests which are relevant to the container environment to be executed also this was the case of the shared repository so the role will also take care of fetching the test from the test repository and then actually exploring the directory for available tests applying the flexible metadata from our filter selecting appropriate tests and running them so here we see installed specific package requirements some additional dependencies which are missing are changed and installed so these tests are using vClip there are some packages some requirements are already installed in the system any questions maybe if you have to be free here to discuss right? this is quite a thing I just came from a crowd talk anyways how is it to improve the actual testing of the filter it sounds like there is a lot of moving pieces there fan messages going around I don't know I have a whole questions I guess let's say I just want to land a change in some part of this is that I would send up some containers and CentOS TI built from the upstream code do you know what I mean is that both in Flora and CentOS right? also so for the pipeline that is running for the specifications of course that is text for the tools like the metadata format that is also to utility the standard test roles are package in Fedora that has CI that will be tested once it is merged and updated the pipeline itself that coordinates the different pieces is on GitHub can contribute a change there that that will be run for the CI and if that checks out and it is merged from the pipeline it is updated automatically so I guess the sample change would be like common thing is that your log messages and some CI systems will try and use them to extract the most relevant error messages I don't know if the pipeline does that but that is one of the things as a developer it is like how easy is it for me to go in and improve the systems if you are using the standard test roles that would be a forward request against the standard test roles which you can also the standard test roles are easier to test so you can maintain that I don't know if it should be in that I don't know how many I may investigate usability is the main point especially with low velocity but I think that is actually being worked on so once we have that first thing in place it might be easier to iterate on that the puzzle consists of many many little pieces this Wikipedia portal I should list most of them so if you know where to look you can directly so for example I don't know you would find about in the metadata format so you can find some information here and link for the GitHub where to file issues or if you are not sure you just don't know it is possible to file a general issue here in the contact section so there is this Federal CI general project and we gather issues general we track them there and if we know where to redirect we can help you with that so here I just executed just so you know so FMF list when I entered this this is the shortest repository for the shell tests we have there just three simple tests for the truth of concept and you would be able here to select tests based on for which component they are relevant or if they have some tag yes so this is for this part FMF show you all the metadata for a particular test case for this login you see what is the component what the test does how it takes how long it takes to complete and such for the future improvements we have several ideas the test demo does not sound to us like extensible enough we would like to introduce a new format which would allow us to define more in more detail in what point of the release pipeline for example you would like to do some very basic test for the PURQ test itself so you would say there is a PURQ test check the code run the pilot on it and make sure there are no something like that maybe you would like to run a different set of tests for the scratch build which is created from this PURQ test then when the PURQ test is merged and the real build is created and now the question should this be merged into the compose or not then you could define some additional test coverage which would be executed and make sure that some maybe integration tests are run against this package which would make sure that it doesn't break other packages which are allowed to add so this is the very very fresh very fresh thing the CIFMF have a PURQ request just with the initial set of examples how it could look like just to give you a brief idea so for example the last one defined somehow how to discover tests so you could use like list the test or use the FMF you couldn't have some requirements for the for the hardware you can do some extra setup steps on this then you would select which tool should be used for the execution of the test and in each of these steps you would be able to say okay so I'm using just a plain text file for listing tests which should be done I'm using FMF for getting the information or I I stick with the old TCMS where I have like thousands of tests so I will use it there and here we would say for example so for the build we would like to run smoke tests that would be tier 1 and some basic test for that one once we do an update like body update or errata we would run this on all the architectures which are supported we would run additional tiers and we would do a separate test job only with security tests so that they finish as soon as possible so we can make sure there are no security issues in that so make some improvements in this area this is coming from real user stories for example the developers as you have distributed as a bunch of components it makes sense to test them together so there is a real use case when you run some tests only with all these packages together so that this should make it flexible enough to be able to execute those tests go ahead kind of along those lines maybe I just lost track of how the testing process works for Fedora maybe this isn't the right place to do that but for critical path packages and automation is there a way to use this system to say I don't know I've been a job update and I want to make sure that if you still have that kind of activity before that's going to tag it or not is there a way to do that when you run some packages and you run a set of functional tests when you're tired just go not just package specific for the package friends they were not targeting that but basically we would like to also have a composed level test or something so that definitely this should make it possible of course maybe it will be not in this give because this is not completely component centered but that's what we are looking for we would like also to make some consistency by defining how the Fedora how the message format which is shared on the UMB how it should contain the information about the test path or failed also to make this consistent and if possible also make this consistent upstream and downstream so that we could use the similar or same same tooling for processing the messages or emitting the messages this is started for the test progression and the results what we are looking into also for the rating event themselves to have a consistent format downstream upstream for the messaging even for the quality environment so if somebody enters a new build system we would like to standardize the format so it's not a mess like with all the other or the different build systems for the time they were created it's a little bit messy in the message environment so we are trying to standardize that and onboarding maybe later on legacy systems also Petri is showing something when you kick tests a little bit off can you tell it which of these what you just said you can specify which tests should be on when it's verified that's just coming I can tell try these tests kind of the same question if I'm not doing it locally I can do without committing anything I can add the test demo and do a scratch build minus minus SSRPM that would work as well currently the pipeline setup that it built picks up also scratch builds and run the tests according to this also if it is minus minus SSRPM where actually there's nothing really in the written that's fine still so if you create a pull request with this test demo so even if the test if there's nothing in this git and you just create the test the pull request which would contain this test demo like for the test execution the job would be run okay maybe I didn't understand but I would have to commit it to my local review so the question is if I just add the file to my local file system which is of course a good one whatever and kick off scratch build with minus minus SSRPM it would build what's in my local file system without that I have to commit anything can I still kick off these tests from your local currently now there's this that is currently not supported the test needs to be in this git or in a pull request you could test it locally because you just create the file I can still build on the Brutalware and it just creates for my local files and I will use the SRPM that is built on the official build system the question is if it would as well work if that has a test demo no because this is based on the test demo starting the git repository or submitted as a pull request if it's on your local because the tests are not packaged into the it's not packaged into the SRPM in this git it's not packed into the SRPM somebody that is a configure maybe didn't explain that so we are out of time thanks for coming if you have any questions you can reach us thank you