 Yeah, so I will talk about About Jenkins Garrett and other tools we're using to do all their things with commits so First of all, let's start with some code or spaces and tabs Later we might get a commit and of course we push it Earlier, I said, I don't know when Garrett was introduced. I think one or two years ago Maybe two years so we Change that to push to Garrett So what's what's Garrett actually Garrett is? Is for reviewing all For varying a single commit so in difference to get a pull request where you usually review a whole branch or multiple commits This is really usually you review every single commit and say no I'm here. He is missing a tab or whatever command which in my opinion is much more precise because you just look at the Commit and think okay. This commit is fine and on github. It's more likely. Yeah All this 50 commits looks good. Let's merge it Usually you don't find the missing things there and also Garrett can integrate Jenkins or other tests with it. So it really depends how many tests you want to do, but yeah, there are multiple integrations to it So just take a short look how how Garrett looks like so this is a Usually interface you just see all the comments. I don't know how many projects we we now start to use it actually but yeah, it's It's a lot. It's you sometimes a little bit slow because it's full of driver script, but yeah, that's a different discussion But it has also some SSH Yeah, some SSH not API interface and you there are also some other tools where you can interface with Garrett like there's the git review plugin where you can Yeah, which helps downloading or uploading commits to or from Garrett So back to our commit after we pushed it together we we do some tests so in our At the moment, it's just the Jenkins which compiles and it do the unit tests But we might change it to maybe TTCNC or to the Osmo GSM tester Yeah, but we could all edit for now. It's a simple compile and unit tests Then some of us must have to review the commit and say plus two which means there looks good for me and Later somebody have to press the merge button on Garrett because Garrett usually should own the Should own the master branch if you use it Um, so Jenkins, I think everybody knows this driver monster. I don't know how many plugins we are using Maybe 30 or 40 Not sure Let's take a just short look at our Jenkins how this is One of them so we have lots of Test drops some some I actually use some are offline by now And you also get the logs of the test drop and for example, those are the Garrett drops and if the drop successfully One for certain Garrett commit it also You also see plus one for this commit for testing that it that it worked well and you can also It also links The the test drop directly to the commit so you can see it actually So there basically are four most important tests We have also some other support drops and yeah lots of other tests, but I think those are the basic four topics we are we're doing at the moment so Garrett and master are not really different They also just compile and take a look at the unit tests, but we are also have dependency like if if Libos of course changing it triggers multiple drops or And also get get back to it um Yeah, so I don't know how many people really try to configure a single Jenkins drop and try to do it by hand. So let's See that we want to change the build slave test multiple build slaves. So every time you you have to go down and it's really It really takes time where you get to the configuration. So, um We changed like Couple of months. We starting using a different tool which helps us configuring the Jenkins We decided to use the Jenkins drop builder It's basically a Python script that comes from OpenStack. They started they created it and it's doing YAML So the input is usually YAML and at output is the XML which is also Um pushed to the Jenkins via the API. So you can directly configure the Drops there and also we can use heavily templating and Others yeah other common techniques. Um I'm just taking a look into into our repository where we have a Simple drop Like this this is a quite simple drop. It's uh It's just having uh here this is the drop actually just a trigger. So for the osmo gsm tester if we want to um For the osmo gsm tester the Jenkins builds all the binaries And give it as the tarball to the gsm tester and the gsm tester then runs all their tests its nodes and For this task, uh, this is basically just a trigger which triggers other drops so for building the osmo bsc and and um So basically what's important is the node actually here, which is a which builds it and here we also have all the different different Triggers which should trigger. So this is quite a simple one. We have also like files which are hundreds of lines Which are doing the tests. It's a better example like Maybe maybe the runner is a Better example. So um, let's set here. So we have like an get repositories. The other one was just triggered and this one is this running the the Tests on the gsm tester itself. So It's based on a git repository here. We have our or git repository which can be included multiple times in different drops just by using the name and Getting further down here's another git repository and Um, just Here it actually starts That's our template. So it's really good to because we have multiple drops which are quite similar to each other. We are using templating and yeah I think it would be another talk just about the json drop builder. I just want to give you the Basic idea how how it works. It's a little bit different how it templates actually but um And we are starting to using or so ansible for our or builds less. So we Have we have we had the problems that for example, if you want to build osmo com you need 10 15 different Requirements like I don't know what no libt dbi isn't so much used anymore But a lot of of libraries we are using like libtelok and So we started using ansible to set up the build Notes itself, which is quite similar to json drop builder. It's also written in yaml and Yeah, you say police install this or that package and uh, yeah So that's basically I can show a short example of of the ansible So this is a simple task how how we all install the um The packages for via ansible and so we can you can use lists and items to define your packages and So we have basically started to use it. So we have a fast simple way to reproduce our builds less um Yeah, and it's it's basically quite simple yaml I think most people can can actually just read and understand it without knowing the language itself but writing Of course, you must know basically so in this particular example, it's the It's installing all the the items listed here. It's the item here and before we we tell with Those two lines that it should Updates that it should execute up get update if the cache is older than 30,600 seconds And uh, yeah, so the script is basically Yeah, so ansible Doing this for yeah for All of it like here we are um Adding a user and also generating an ssh key and yeah And everything resides in the osmos ci repository both the drink and drop ansi ansible repository So if you want to take a look Okay, this fourth basically the talk Yeah, what I would like to be able to do is To test if a project builds on different Linux distributions and run some some tests, but on particular distribution is how Do you know how to do it with these these tools? And at the moment, we are only testing debian But we also have a open through the build system And there we actually built for okay only you won't do debian on different architectures but We we should We should add also the other Build specs so we at least we know somehow it's It's building or not building on this or that architecture or distribution But that's the moment. We are only doing debian but Basically if you wanted to do that with ansible and Jenkins then you would basically In addition to the Right now we have ansible files that describe which build dependencies to install using apt on a debian machine You would basically do the same for whatever other distribution you want to build on So you basically in ansible in the yaml you would describe well for for building osmo com on fedora You need to install these packages and then we could create an A container or a jail or whatever On one of our build hosts Run ansible against that. So basically this build slave has all the dependencies and then we could have build drops that build on that distribution so You can do it with docker. We even using docker in in some of our builds it It's also possible But but the easiest way would create a build slave and we can add it to to the Jenkins That would be the easiest way how we could integrate it So if you have like a vm with fedora, we could we could add it to the Jenkins and Also enable that the garret and the master is building On on this machine Yeah, so the the architecture of the build hosts now looks like that we have of course the physical machine Which I think always runs some devian, but it doesn't really matter so much and then we have lxc containers Linux containers for the individual build slaves, so we have an lxc container for devian 8 and 1 for devian 9 and as i said you could have one for the fedora or suzer or whatever you had We used to also have some free vst jails But that we deprecated that Because basically we're not aware of anyone using osmo-com on free vst and it created lots of lots of work for us and Without any users. It's not really worth putting effort in it Yeah, but so basically we could have such build slaves and then as as links is pointed out we could execute stuff there From from Jenkins What we do have is that some of the Some of the build drops that we have or the build test drops actually use docker internally To speed up the builds Because they have already some basically some environment that's So I we have some build drops that build natively in the linux chain And then here we have some build drops that basically start a docker container and then build inside the docker container So then we have docker inside lxc in in that setup And the one advantage of that is also that you can have any number of parallel builds in an easy way and So It depends on on the specific build drop and then we have all the ttc and three stuff Which again also runs various different docker containers inside the lxc To run the tests Yeah, and the other approach is To use obs but open source build service is not really for compile testing It's for building packages for the respective distribution So you could abuse that of course, but because if you can build your package Then yes, you've done your make and your packaging and you make check and whatever So, you know that has also run, but that's not really what it's intended for What possible Can you explain the correlation between git branches and the refs in in garret or if there is or How does that work? So actually garret has its own It's or how I say Yeah, yeah, garret doesn't have or it has some understanding of branches, but it's just you Let's say you you You just put another commit on the master. So actually garret Garret has garret usually Only tracks the master And if you have a commit you push it to garret and garret does some let's say Virtual branch for every new commit you push on it. So they Yes, you can chain multiple commits But garret doesn't have a real branch of it garret use actually In garret use their own reference, which is in in every commit you see those Those references You can show it on the in the garret The change ID here. So this is the commit message here and When you do on your local machine to get commit There is a post or pre-commit hook in your own of your on your own Machine which generates those Those commits those change ID. It's completely random and Garret tracks all commits based on this change ID So in the so pushing to garret you should always use that reps for master, right? Yes, but okay, so maybe So maybe it's the topic then that I am wondering about Can you show the list of current open commits On what project any of them, okay, so there's branch. Okay, so it's a topic that's in brackets after the branch So how does that relate to the workflow? So that's not a git tag. It's just a descriptive. Yeah, it's a topic. It's Just a description about it so If if you know how feature branches some people some people use get flow with feature branches or every new feature they create a new branch and This is a little bit similar how how garret works because If you push so if let's say You checked out the git master a week ago you develop and now you push it to to garret usually First you rebase your branch to master and push it to garret And garret sees garret knows. Okay. Those commits are for master. I don't I ignore all old Commits which are already on master and just correct the two new commits. It didn't know before so, okay, so just Like if I wanted to Suggest some code by making a commit, but the intention is not really to say, you know, this is the intention is to to Have this merge to master as such. It's just like a way of communicating Maybe hey, there's a bug. This is here. It is. This is what it is. This is my attempt to fix it But it's not actually me saying I suggest this code gets gets merged. Is this the The right way to do it to push a commit I'm actually there multiple ways to do it my preferred ways. I just put Put a not for master or not for merge in front of it if I want to show it to somebody Because then it also gets tested by Jenkins, but there is also a draft feature Which are not where you can put Your commits, which I even non-public if somebody doesn't know the exact. Well, it doesn't Doesn't get to it. You're not really using any of that So the normal process in osmo commas you just push to a branch Not to ref sets for master. So you push to something like la forge bts, whatever fubar And that's not visible in Jenkins at all because Jenkins is what we use to review stuff that we want to merge as or in in carrot So we we don't want I mean if it's really something that's still in process And maybe I just want to have some other people to be able to check it out and have a look at it Then we really push to branches that are typically prefixed with the developer names Or you have all these daniel slash something or nil slash something or la forge. There's something branches um or p maya branches and Then once It's ready for being merged then we push to ref's heads for master Which makes it appear in carrot and we go through review and merge it to master How has a comment? So sometimes what you can do actually To be sure that it's not let's say Directly merged. I sometimes do it. I just mark it myself like with review minus one Just and I add some comment to the to the comment. So then It's more difficult actually for somebody to merge it because I mean when you see the minus one Then it's I mean you take more attention at the end of the list because yeah sometimes you want to propose something but You are really not sure if that's the correct fix. So you are just really okay That's my proposal but can somebody really check it out. So That's what I do. Usually I add minus one Like myself to my comment and then I put some comment and then maybe I ask an onirc. Can somebody check it? Yeah, this yeah So with minus one is meant on carrot you say Plus two so it's like a vote system where I can vote on on the commits and this is like an example Where where I developed something but without a test and I gave it minus one. So others might just see okay It's not for really it or there are still problems Yeah, like sometimes if if you push something, but then you realize also like oh, actually this is missing Some code or actually I want to add some tests or whatever then myself I add a minus one like hey keep it there because I am actually going to update it at some point. Maybe it takes a few days, but So my experience with with Garrett in another project is that it's If you push something to Garrett it usually doesn't Leave Garrett and go into master unless you you sort of take care of that happening Um, so just just it being in Garrett is is not automatically in my experience Some a guarantee that it will will arrive in master But I think I think if I understood the question the right way, I think it makes sense to push also proposals into Garrett Yeah, of course you can better discuss it, but it depends if you want to discuss it in Garrett or on the mailing list I think that's That's the difference between both If you just push it to a branch you can't discuss this in there You have to discuss it on ISE On the mailing list, but if you push it to Garrett you can discuss this directly there and And can also annotate single lines in my opinion much better than on the mailing list I think it's also good to Because this way you can keep a record of like possible changes because some maybe at some point After a few time you figure out. Oh, actually this patch is really needed and I made it But of course if you push it to a branch sometimes, uh, you just delete it at some point But in this case you can probably go and figure out where it is in Garrett and actually reuse it Okay, any other questions Because I don't know if it was If you push some Commits into a branch and not trying to push it into master it will be test by Jenkins or not No, it won't be tested by Jenkins Jenkins is only testing master and Garrett patch has said so Such a commit you push to Garrett that could be changed though. I mean, it's just a configuration I mean, we if there is a need for it. We could ask Jenkins by configuration to not only build or test master But also some other branches with the certain prefix or something like that. So if that's needed it can be configured Yeah So what I personally would want to see is that we have even for the non Non Garrett projects we have build testing in place that actually passes So even for projects like rt ls dr and so on I think it's really useful to have Testing in place because the Jenkins execution. I mean normally There's a clean environment It's being built and you run make check or something like that to run some unit tests Not sure whether we build with address sanitizer now and w arrow. I think we do So There is advantage of stuff being built and knowing that your master is currently building on on at least those systems that we Primarily support which is Stevian so far and I'd like to Encourage people to work on on adding this even for the non Garrett projects and the non cni projects. So to speak