 All right, welcome to the penultimate talk of the FOSDEM 2020 distribution dev room up next. We have Frederick Rosa Talking about integrating new components on fast and slow-moving distributions Hello everybody. So I'm Frederick Rosa. I'm working at at Susie as one of the Susie Nuke's enterprise release manager for server and desktop and Yeah, I will Walk you through how we handle changing big parts of the distro On the various distro we we take care of but first some words about Susie So we don't do this thing. So this is mostly for French-speaking people But we do this thing Especially if you wear at the booth of the open to the booth But yeah, most seriously Before before we go into more the details and the policies we we we use first a bit of General Vocal vocabulary so that we are all at the same level About distribution style. So Rosalie these days There are three kinds of distribution style What we could we can call rolling release regular release and LTS or enterprise or stable releases so Rolling release. What do we mean by that? It's usually bleeding edge a bit sometime a bit too bleeding Released as soon as possible as fast as possible and As example, I would say open to stumble with of obviously arch Into and probably others Regular release. It's the thing which came first in the Linux world Usually distribution like that are released One time a year sometime two times a year In the very old days some crazy people were doing that every three months I was part of those crazy people The thing is those distro every every for each release they update everything Which can have a lot of and bring a lot of challenges because you have to stabilize everything at once And then you have the LTS or enterprise releases which are Released very less often so your licker dance sometimes even less than that and There the point is to change as little as possible The reason is especially in in enterprise in the enterprise market Customers are extremely frightened as soon as you change one bit somewhere because if they have validated their entire Infrastructure, oh, it's running fine with this version of this distro And then you tell them yes, but you need to upgrade to the next service pack and they are like, oh my god So how much has changed? Can you guarantee that those those packages have not changed etc? So It's really a matter of changing as little as possible But still bringing bug fixes new feature, etc, etc So there I would put open to the lip. I will explain a bit later Ubuntu the LTS flavor and on the enterprise market The Slee family slas and sled rail You name them So now jumping more into the open to the and to the world I Will use a lot of acronyms OBS so who has Have heard about what OBS is Whoo, that's nice So for those who didn't put their hands up OBS and I see you internet OBS is the open-bill service it's a service that Open Sus is provide to everybody to build packages for almost every distribution not just the open to the distribution but it can be for Fedora for Red Hat for Red Sorry for Debian Ubuntu Yeah tons of distribution are supported So that's what what is part of our processes So Slee as I already mentioned Sus Linux enterprise It's a enterprise this or that sooth is is creating With a set several flavor server desktop Open Sus tumbleweed so tumbleweed is rolling release as I said before of Of open Sus I Will maybe sometime talk of also about open Sus a factory The point is that initially open Sus tumbleweed was a rolling release created by Greg Kroepman based on a Regular release of open Sus and then he was changing a lot of things on top of it and after a lot of discussion a lot of processes put in place now The development Distribution which is open Sus factory is then tested and shipped From time to time and this is what we call open Sus tumbleweed more on that later Open Sus a leap it's a stable slash LTS release of open Sus the point is it's a mix of Component coming from the Slee so from from the Sus distribution plus a lot of other packages coming from tumbleweed slash factory the same So it's a mix of something new something Old and something new some people might say it starts to resemble things about wedding, but no Anyway Let's talk about how we handle integration of of changes in in in the open Sus and to the world First in the OBS so in the open-bill service we have the concept of devil project The point is that every source when I say source. I mean a source package Every source package Need to be in a project and a project can aggregate various packages or packages together and We tend to have those Projects so those devils project per topic So you want obviously to group things together Like having all the thing around X11 so the X11 server Plus the libraries in a single project because usually they will have to be built together Same about Katie stuff Katie all the Katie packages would be in a Katie project a Project can be cut in several sub project, but that's mostly the idea same for gnomes et cetera, et cetera and any changes in those Projects are done. I'm sorry Any change in the distro sorry has to be done first in a devil project and then pushed to To the upstream distro. Let's call it this way or the downstream depends to the distribution for integration The push is can be made either by Human being or but how does it work? Don't be frightened by the schema, but You have done some work on any random packages So it has been done in the devil project Now you want to push this work in the distro. Let's say in Tamil with So the thing is you are going to do what we call a semi-tree quest, which is basically the big diff of changes You push that to This distro and then The magic or the pain it depends where you you are you are begin So they are going to be several things done at the same time First this change is going to be put on what we call a staging project this is a snapshot of the distro and We take this snapshot and we just put this change in it and we rebuild The distro based on the build dependencies, et cetera goes by this change So this this is building Then at the end you will get end up with a minis in mini distro The distro we built with this change and then you give that to Open QA, which is another so the open to the project Which how many people know about open QA in the room? Let's say half so for the other half people half of the room Open QA is an automated test testing framework based on Images of recognition among other things which is There to mimic a human testing the distribution in all the application So it's going to start the distribution. So in the virtual environment or sometimes even on real hardware Click everywhere. It's where it's needed Check that. Yes, you get the proper button at the right location Etc. Etc. Etc. Etc. So install a system then run it check that whatever application you expect to run run, etc, etc So we have that on one point on one sign and in parallel we have reviews done on the on the change itself like Review on the code itself was does it add a patch? Is it well-formed? Is it introducing a vulnerability? Is it with a reference to an upstream commit for instance? There is also reviews like is there Is the entire thing Correct in term of legal License so there is a license tag is there something in the in the table, which is under appropriated license, etc And a lot of boats are doing automatic check on that and just with that just with the automatic check we already catch a lot of errors Based on those two if everything looks fine basically you have a change It doesn't break the entire QA thing. So then we commit we accept the change We put it in the distro We rebuild the entire distro And then we have a new ISO that we are going to put again in open QA But this time not with a small Sub subset of the test suit, but the entire test suit When I say you have a one submit request in fact, we never do that with just one change Usually we aggregate change together either because they need to be aggregated like you need to upgrade two things in parallel Or do you upgrade that and you need to patch this or the package and this or the package otherwise it breaks And of course it would mean if we were doing one staging per change We could not keep it keep up with the pace. I Forgot if you have questions during the talk, please ask them don't wait at the end Interrupt me So Another thing that we have so this is more for the through the part of of the thing We have a policy that we call factory first Which has been in place for quite a while 2017 Which is kind of we want to make sure that all the work which is done on the enterprise distribution Which is sometimes a bit on older software has to be always pushed to the Factory or tumbleweed a thing to make sure that we Don't keep things just for the enterprise customer, but it's available to everybody even if an enterprise customer Everything is under Free software license It's not the point of point is to make sure that we always retrofit and push our patches or changes upstream upstream being tumbleweed but then of course being in the upstream project so this is very important because Based on our experience. It was not always of the case in the past and sometimes we had to fix bugs Now when we rebase our code base and take a new snapshot of tumbleweed to create the next generation Of sleep we discover that the bug which has been there for eight years We patched it every two years when we were redoing the branching because the patch was never upstream and The worst part was when you when we have upstream it it was still applying So it would have been done eight years ago and that would have saved work for people Over the years, but anyway, so we have that in place and we have a lot of bots ensuring that this is done properly so now Yes, so comment from the room. It's also a way to make sure that Slee doesn't go in one direction and an open suit is in another direction because in the end Slee is a Downstream of of open suit so if we I mean you you all know you You hack on this through so as soon as somebody tried to do a downstream and doesn't contribute back His changes then they are the one who are going to feel the pain later There are ton of example there Let's talk about noom 334. So Over the previous last month we have been working on on a brain in In the various distro So let's go over that. I Talked about the deeper project on OBS so Usually there is a one legal project which would be in that case go home, but The open-source go home team has decided to use two layers One layer one one project, which is the go home factory Devil project, which is really the devil project Sorry for people on the internet. I hope I didn't kill you here Twice let's avoid the third time so yes, no factory is The project which which contains the last version The latest version of the stable version of GNOME so until September last year it was 332 then 334 was released. So that's where you expect to have the latest version and then No, I'm having a cadence of six months There is another project on top of that which is GNOME next which is obviously by its name It's what's going to happen in no means of future The idea is to have in this project not the Development packages of GNOME from the beginning like when there is a dot one or dot two of the next version because it changes often and sometimes Maybe more in the past Everything would not build Properly etc these days it's improved a lot But at RC at release candidate time of GNOME Having the packages there so that we can start preparing everything in advance And this also allows to these two layer things if we get bugs report on the stable version We still can push those bugs those bug fixes To our users they don't have to they are not bug blocked because there is a development Version in being integrated Yep, and as I said next GNOME next is layered on top of GNOME factory So OBS is smart enough to do links and branches. So It will detect if you do a change in the Stable branch so GNOME factory it will propagate immediately to GNOME next So this way we know we are not going to lose a fix because we fix something and then we Rev up the version of the package and sometimes you end up losing things so Upgrading 3 34 in thermal with So as I said the packages were upgraded to 3 dot 33 dot 90 something during the release and we had also an automated Build of a live image So, you know this kind of ISO In the old days you were burning a CD You were putting that in in a computer and you were booting the CD directly you didn't install to test things These days you can do that with a VM But still the point was let's create an image based on tumbleweed. We just Replace the GNOME on top of it with so GNOME next packages and let's see if it boots and it it works Like application start. I think that already helped a lot to make sure that What it is in GNOME next works good enough But even if it's just a few tests not the entire test suit, but still Um Yes, so that's what I described and then GNOME 3 34 was released point zero was released And then the work was to get this pushed into GNOME factory This is a bit tricky because the open-source GNOME people like to do things the right I have yeah, so what do I mean with that? if In GNOME next basically people can submit things even commit things yet directly we if they are part of the Open-source of no team if they are not that can just throw submit a change Change request is going to be reviewed by somebody else, but Any change in no factory has to be reviewed reviewed by four eyes Which means the person who submitted the the change or the version update It needs to have two other people reviewing what he or she did Because Yeah, it's kind of Ensuring that there is a good quality of the change Sometimes even up to is the change log properly spelled So that's why I'm when I say the open-source GNOME team is Sometimes very picky But that's kind of to ensure that over time Those packages will be easy to maintain but somebody else so Once 3 34 landed in factory Then it was pushed to tumbleweed Then there is another For eyes review Review this is not done by the same people. This is done by the open-source review team So again, they checked things sometimes they notice things that skipped through the open open-source GNOME team But overall usually it's quite smooth. There is a legal review of the changes For GNOME, it's usually not a problem But sometimes you can have one random packages which is pulling One Github repo in a terrible copy of a Github repo where there is a test case Which is under whatever fancy license and then so legal both say I need a lawyer. I cannot pass it We have that and then we have this entire staging approach that I described earlier where we put Gnome in it and then we try does the rest build and Sometimes yes sometimes not really So we have to make sure that ah there is this packages this is all the packages and the third one Which we are fine before we try to upgrade GNOME now that I don't no longer build though. So then we Notify the maintainer of those other packages. Could you have a look because if we accept this staging which we want It's going to break your packages so we do that as a compilation level and then at the QA testing level open QA where it's going to Install the mini distro with the entire GNOME update. So this time install not just a live live image And then run all the possible tests which are written like I don't know starting g edit starting Evolution making sure you can still read your mail whatever and often you discover that Because open QA is based on image recognition You're some pixel have moved a bit Some the theme has changed completely So there is a bit of fuzziness so open QA can be can be set up in a way so that it doesn't expect 100% match But being fuzzy up to 95 or 90 percent. I don't remember. What is the default? Yeah, Richard is telling me 95 is a default. So Sometimes if it's like the font has changed a bit, but not too much So the fuzziness will make sure it's okay But yeah with with 3.34. I think it's it's 3.34 that so the theme was updated to something Distinctly, I mean a human could see there is a difference Then we had to update the needle Yes question. So the question was we have this kind of layer of of Of the on the distro level on the package level do we have this the similar concept on the open QA level So we don't have we don't have it but we have something a bit different is that open QA is not when it expecting to check for a needle so To check for a screen screenshot and finding something which match it doesn't search for just one matching We have a list of matching possible acceptable thing and Also with that we have some tags So for we can say you can accept not any Any of the screenshots we have put aside, but those which were tagged like this is for starting evolution and Then we were like we will be in that at that point in time in the test. We are expecting A needle which is starting evolution check all which are in the in the database. It's a github repo but checks all of those and if one match we are good and We can even refine with something like we want a repo a tag of starting evolution And the distro is tumbleweed or is sleep or is open to the leap because sometimes we have Things which are different but on purpose Yes So the question was do we use text recognition to avoid the screenshot thing? No, I don't think it was ever tested it was yes So answer from the audience from Richard Brown, so it was tested and the current approach is way better Yeah So it's surprisingly good Let's say because what one thing I forgot is on the needle in Part we just don't we don't do full screen Matching we can just highlight. I want to check one part on of the screen like this button or This text and even if it moves a bit in the image in the in the result The matching will be fine. So with that plus the fuzziness It's usually able to to catch a lot of things, but it's it's it can be a pain for instance for localization because we don't test the distro in French Spanish Chinese Japanese Korean because it would mean we have to do really all the needles Okay then Once the staging was accepted There is a open clear running again, but this time with the entire test suit and if it works There is a bot which is going to say yeah Sounds good. There is as much breakage as before. So we didn't break much as before So the quality is the same or improved. It isn't it didn't decrease and the bot will automatically release a build to or Open open system a read user So in this case 334.0 was available less than a month after it was available upstream which is Good and bad at the same time It's good because it's yes. It's less than a month It's bad because we were able in the past to be even better than that Sometimes we were able to release it the day after the release of upstream Or even sometimes the same day because the table already really are ready a bit of a before This time it didn't work well because we are lacking hands to do the packaging So if you are willing to help, please join It's fun So I talked about open system a week. Let's talk about sleep and open to sleep first Yeah, I said about Leap that it's based on on On sleep, but how do we do things? the idea is There is a set of packages which are I mean sleep is a bunch of pack of packages From those packages We create what we can call a share core and then leap is reusing that and Inject all the packages which are coming from tumbleweed which are the latest and greatest. Let's put it this way and we do that for the 15.0 code base. So this is where we branched directly. We did we took a copy of tumbleweed We just kept the packages we wanted for an enterprise distro And for leap the community decided oh we want also those packages because they are not part of the Enterprise distro, but the community wants those packages for instance Susan doesn't ship anymore as supported packages the KD environment. We just ship one desktop which is GNOME The other desktop are available through the community By either in leap or with another mechanism, which is called package up, but I will not go too much into details about that we do that for the major release so 15.0 and then Sleep has a cadence of doing a service pack every year which is And the the idea is a bit a tick-tock approach where there is one service pack where there is going to be mostly stability fixes plus a few Feature and then the next iteration will be let's update a bit more things because we are starting to educate enterprise customers Yes, you can still upgrade the kernel while being Stable distro. It's a bit difficult, but they're starting to get it And we we do for instance also we update the desktop also from time to time so we don't have the pain of Maintaining a very old desktop And we this way customer can enjoy new feature from the desktop upstream And we enjoy less bug fixes to backport also So how we did it firstly 15 sp2 So we have we had GNOME next so on OBS and There which was it was building on top of tumbleweed So we did we added another repo in this project Let's build it on top of sleep 15 sp2 which was and is still under development Yeah, I forgot sleep 15 sp2. The development is Always available on all the OBS so you can see where we are going This is not behind closed doors So then we looked at how how much is it broke because Even if we have those this factory first policy We make sure we make sure to push all our patches to tumbleweed But sometimes we push package patch, but we don't apply them or we don't enable them Because some changes are really for enterprise customer, and you don't want to do that for let's say regular or Yeah, non-enterprise customer where it's like limiting things etc. So we don't want to have a lot of people shouting at us So we discovered this kind of breakage like patch up not applying anymore, so we had to date that Then Because Lee is kind of older than tumbleweed we discovered. Oh This doesn't build because it needs this new package, which is not in sleep This doesn't build because it needs the latest version of Mason So now we have to update Mason in the next service pack and then you discover Oh, but we need to update see make also because the test suit of Mason doesn't pass anymore Because you need another fix in Piki G config Because the one we had was a bit too old and it was it had to bug which only is seen in the test suit of the latest Mason, so this is the kind of you start to put the finger there and it's like oh my god, but it's It's interesting It's challenging that it's quite interesting and you also discover that even if The tumbleweed and the package is done by the open-source team are of high quality Sometime Anybody can miss that. Oh you need another There is a bit requires which is a new version of this package It happened it was already in tumbleweed So nobody noticed that it need to upgrade this in the bid requires but instantly it's not the case So you have a random fader of why it's not building So this kind of thing we we we went through At the same time we also created Staging project same ID we takes Lee Which is developed under development as a base we put them the normally based on top of it We check does it build then? Yes, it builds the let's create a distro of From that and let's run on open QA and fix all the needles and all the test case and all the Changes which were not really expected because you upgrade whatever components then for instance Over two or three release. I don't remember now no shut down GDM When you are logged in so you no longer have GDM running in in your initial graphical console Except that there was some test case in open QA which were kind of oh, let's search for GDM to detect where it's which VT it's using There is no longer GDM running anymore. So you are like I need to have that the test case this kind of thing Yep, so what did I said? As I said We also make sure to push to sleep only not the anti-ognom next project But just the packages which are Which are shipped on sleep because again, we don't ship everything to our enterprise customer And all the other packages they are always available to the enterprise customers through what we call package up Which is kind of a repo where all the packages which are available on tumbleweed More or less on only are available to sleep customer unsupported So they can and we say you can install those packages on your distro But don't call us as I won't be supported But we won't tell you you broke your system because you install those packages. We double check those So to get things into some green state it took roughly a month Then we Okay, this is this looks great. Let's accept it. We accept it and then We started to see all another set of failure because for Speed reason we do things on Intel platform For our staging because already building this and having the test run. It's kind of time consuming So we don't usually do the staging approach on other architecture except that The sleep world is like we support a R64 PPC-64 Ely so power and mainframes. So yes, you can have no running on a mainframe We even discovered that we were the only one for this version of GNOME We were the first one trying to do it because it was not working and it was a bug upstream Despite other distro having it built but apparently nobody was trying ever try to run it Kind of funny it was a bit of a pain to To debug it in the end. It was not even GNOME fault It was a bug in the Mozilla code base because Mozilla doesn't care about mainframe obviously Yeah, so but we we were able to back for the proper fixes and we even had Yeah, we after that I thought fixing fixing that we updated also the rest of the application and Sometimes we forgot things So we we discovered after a while. Oh, let's rebase things and then wait. I think I forgot to update decomp So I had one of the layer which is kind of saving all the the configuration on disk for GNOME I had the version of 3 GNOME 3 26 and Everything was running fine So it's it's Yeah enterprise table so we can say GNOME is enterprise table or at least decomp is enterprise table But it was surprisingly I would have expected to have random bugs here and there no So right now we are running at three thirty four dot two one and between one and two because we have a few packages We did that before the Christmas break, so We will update soon To three thirty four dot three or maybe four if there is a dot four release So we we have GNOME Inslee since So full version so since November then We should have GNOME in leap because after all leap is created on top of Slee so it should be fine. It should be easy One thing I Didn't tell you I mean I kind of tell you told you but is that The difference also between Slee and leap is that on Slee when we do a service pack When we cannot when we can avoid changing a package we don't touch it at all which mean on a service on service pack one Everything which is in touch Between service pack zero and service pack one we just grab the binary package We don't rebuild things because it's kind of this mindset of customer Oh my god, you change packages. You are going to break everything these days with reproducible builds and all of that It's it should be less of a problem But even getting to a state where one other person of our packages reproducible is we are not there yet And even when we'll be there, I hope it will be to get this mindset into people's mind into enterprise people minds into Administrator, etc. It can be very tricky so Slee is kind of let's modify as little as possible But Slee is this layer in this idea of layer of service pack service pack one service pack two is done on top of service pack one, etc Open to the leap. It's not a layered system. It's a regular distro So a regular distro where you rebuild everything when you want to do your new next version So which mean bootstrapping the distro Which mean you discover that even though you have your GNOME packages which are Which we are able to build fine on Slee 15 sp2 everything is nice and shiny Then you try to rebuild The entire distro not just a service pack that you have modified, but it's everything and then you discover that That's not really working because there are all the packages which are either not part of Slee which are just in leap I don't Not kidding me for instance for a while. We didn't have upgraded game. So game was not building anymore We did upgrade it, but there were other Random packages using GTK maybe or whatever which will not build anymore. So we had to update that So it It was Painful Let's be honest to get to a point where everything was revealing fine for for leap When I say painful, I mean it took 45 days. Okay. There is a Christmas break in between but still It to end Nicely enough The staging containing GNOME 3.34 for leap was accepted last week I mean this week just in time for for them. No, I didn't push them Open to the release manager to accept the staging. It just happened that yes, all the roadblocks were Were no longer there. So we were we were good to have it which mean the next Milestone of some in term of my son, which mean kind of my son before the final release the next development might son of Open to sleep 15.2 which would happen. Maybe it happened today. I don't know But it should happen next week. Let's say because it's kind of automated. It will have GNOME 3.34 and I'm already my god I was fast There was not enough question in the audience So, yeah to summarize we have a Great system to build things which is open which is open build service Which you if you want to use it, you don't even have I mean you can use Instance we provide but you can even install it on your system. There are people running their own open QA Open build service instance On their system to create their own packages So we've opened big service we've opened QA again open QA is not just An open Susan and Susan project Fedora is using it I I don't know if all the destroyer using it. Yes And let's use it. So a dbm distro. I mean dbm OS OS 3 base distro uses it. So, yeah, definitely If you it saves times it can be painful to set up I mean to to have something where you are end up to a point where it's passes everything But in the end the the value is tremendous Yeah, so thanks to those tools to our processes to make sure that we force Enterprise this developer to push their changes upstream So we were still able to upgrade GNOME, which is kind of a big stack Just for sleep around 200 packages Which are intermixed. I didn't check for for tumbleweed, but it's probably maybe not a double 300 300 at 400 We were able to do that for tumbleweed in the months for in total for all the destroyer in Four months a bit less than that And one thing is we maintain quality We didn't ship especially for the tumbleweed user They didn't got a breakage due to Upgrade of the desktop Because it was not released until it was passing the QA test the automated QA test and that's what We want. I mean we I think we we probably can everybody in the audience can agree on that We don't want to ship broken stuff to our users So It's nice. I think we can do better I hope we'll do better like doing things more in parallel I mean more people joining initially to do the initial revision of dates as part of the of GNOME next making sure that we do in parallel something we didn't do for time constraints having leap and Sleep work being done in parallel not one after the other But we are we are getting there Any question comments Yes, so yeah, so question Yeah question was Because we import binary RPM binary in sleep if if those binaries depend on Another as a dependency on another Package which is updated Is there breakage? in the sleep world it should not because the reason why we We we say we these we do this kind of binary import is we guarantee to our user ABI compatibility we So far the average compatibility usually is kind of enforced by the SO name and we are we use Policy which is derived from the Debian policy of having a package named by the SO name so when things start to be like The upstream has changed their API so that the idea is different So it's end up with a SO name where the major is bumped We will see that and we will either in that case Say it's it's fine. It will use the old copy the old package But very often in that case we will update This binary package to a more either a more Recent version of the package or we will for this one. We say we want to instantiate it in sp2 and Rebuild it so that it's used on is the latest version of the library Other question. Yes Hello, I'm sorry Sticking to English and question was to mitigate a bit All the difficulties of leap not being able to rebuild because of sleet changing things and not discovering that Oh my god, we are breaking leap should we should did we try to do periodic rebuild of all of this kind of thing? Sorry we don't The reason is also because we have the staging Which are always monitoring what's happening on sleep so we Those staging use a snapshot which is usually refreshed. I think it's on a weekly base so every week on on on leap we will discover The breakage as soon as they appear because the base of of leap is Slee so we will see that There are changes in in sleet and then they we see that it's it's breaking So we don't have the concept of let's rebuild the entire distro and see what's break because The OBS is doing that constantly Monitoring all the bill requires etc. And with this idea of taking a snapshot of A distro and then updating this snapshot from time to time Any other question? Nope. Oh, thank you