 Thank you very much Richard. Great, thank you very much. So, yeah, I'm basically here to convince you all that my... There we go. That my sincere belief that rolling releases are the future of the new distributions. You know, the more I look at this, the more I've been working in this. So my history is I've been an open suzer contributor now for 11 years since the project started. In the last three years I've actually started working for SUSE, testing the enterprise distribution to the regular stuff. So, you know, I've been a regular distribution guy for a really long time. But because of tumbleweed, what we've been doing in tumbleweed for years now, I see this as the trend for all distributions, where things are going. And of course that comes with certain issues and certain problems and that's what I'm going to talk about today. So, really need to start in the beginning with all of this stuff of, you know, how this began and what the heck is the Linux distribution anyway. So, you know, we're in the distro dev room. Most of us know what a regular release is. You know, most Linux distributions are a regular release, releasing every X months or X years on some kind of regular cadence. Generally speaking, once it's released you're freezing the versions or if you are upgrading you're somewhat reluctantly upgrading those versions. You know, we're trying to keep a nice stable distribution for our users. We don't want to constantly be breaking everything all of the time. And then if we do have to fix something, we're generally talking about back porting individual fixes and narrowly fixing the distribution, not wholesale changing package versions. You know, so generally speaking, this is, you know, the Fedora, the Debian, the traditional open Susan model and of course the Ubuntu model. One of the problems with regular releases, okay, how do you then actually do a big step change? How do you develop that? How do you, you know, get ready for the next version? All these distributions pretty much have some kind of deaf branch. You know, tumbleweed, it's like open Susan we used to call that factory. In Fedora, they call it Rohide, Debian Cid. You know, a development branch where developers can just check in their stuff, build, you know, packages get built, maybe daily ISOs or something gets spun out and you have the, you know, the next base for the next major release of your distribution. You know, so it's always released and, you know, nothing's frozen, nothing backforted, just pure code going in, making packages. And without fail, it's normally broken. You know, you're lucky if it gets working in the last couple of weeks before you release, but even then it's normally broken. But that's what as distro guys have to live with. You know, we're dealing with this lovely broken deaf branches all the damn time. But that's kind of a big problem for not just distro developers but any general developer looking to target your platform. I mean, you know, they need to be close to, you know, your latest and greatest, your latest upstream and all of the latest versions of all those upstream packages you have in your deaf branch, but it's completely unstable and unusable. And, you know, we've got work to do, but it's always breaking. So you end up with people coming up with wonderful hacky cludges of, you know, maybe taking the stable base and running some PPA or OBS repo on top of it. But, you know, then they have to rebase all of that. It's always a mess. So, you know, it's a nightmare to develop on. It's a nightmare to develop for distributions in particular. This becomes a big community and societal problem. I mean, if your deaf branch is not stable, how are you going to get contributors to your next release? Well, you're not. They're going to go away and play with something else. You know, you get less bugs being filed. You get less polish on your distribution. And then, even with your regular release cadence, you have a hard time actually getting that release out the door. And I think most distributions that have been around for the last years have had these terrible problems. OpenSuser definitely has, of chugging along with a plan, trying to get a six or an eight month release, you know, release out. And then you're, you know, in the last month or so, and you realize our deaf branch is totally messed up. We're going to have to delay for two months, three months, four months. And, you know, we've done that. Fedora has problems like that as well. It always happens. And a big part of this is the technical debt that comes with that. You know, because you haven't got people using that all the time, all these cludgy little hacks get in there, you're going to have to fix it sooner or later. But your deaf branch is a complete mess. And from a project point of view, that generally ends up being a bit of a decline for that distribution. You know, less people are using you. You're not moving that fast. You're not that interesting for anybody. Your project starts stagnating. Everything starts going downhill. It's also a problem for upstreams. You know, they want their software in the hands of users. You know, that's the point. And having a regular release with all that freezing is way too slow for them. They're, you know, they're not going to be able to deliver their software to eight months or a year later or two years later. Deaf branches don't do that at all. And regular releases don't do it at all. It's a complete mess. There's no way of targeting for upstream developers. Everybody likes to talk about Flatback and Snappy and containerized apps and promise that it's absolutely wonderful and going to solve everything. It solves some of the problems. And I'll be talking about that more in Janssen at one o'clock today. And then last but by no means least, of course, users. They're all enthusiasts. They're here. They're the ones who want to run everything, see what we're doing in a minute. There we go. They don't want to wait. They want to get their latest software as soon as the upstreams published it, you know, new version of no more KDE, they want to be running it right away. But they also want a consistent experience. They don't just want some hack-together little bundle running on top of whatever distro they've chosen. They want that polished. They want it integrated with the entire experience. They want it easy to install and maintain. They want it well-patched. And when you start talking to a lot of, especially long-term Linux users, they'll ask them why they picked their distribution. That kind of somewhat untangible quality of, oh, it just felt right. It was put together properly. So that's why I'm still using it. Is the reason why we end up keeping those users and why we drive more people into our distributions is because we've got those things right. User experience, but more than that. Just building it properly, building it consistently and convincing people that we're doing this the right way for the right reasons with the right things. And they're the ones who are going to be the contributors of tomorrow. But they can't use our dev branches and our regular releases are too slow. So income-rolling releases. Now, everybody hopefully knows what a rolling release is, but generally speaking, a rolling release has no release schedule. Most, if not all, packages are frequently updated. Some distributions like to be partially rolling, so have a frozen bit, but I'll be talking about more on that later. And updates are delivered when they're ready. And of course, when they're ready is a nice, quirky definition. So when is something ever ready? And the main examples that I can think of with this are obviously Arch Linux, and Gentoo, who kind of popularized the whole thing, and I'm here to talk about Dumbledweed. Quite often the feedback I hear about rolling releases though is, it's unstable. It changes too much. I can't trust it. Even if I do try it, it's unreliable. It breaks too often. Or it's just hard to live with. There's too much extra work I have to do to keep up with this rolling thing. And when it comes to the stability side of things, and in this case I'm really talking about stability in how fast is something moving. The, I'm talking about something always changing. That's the point of a rolling release. You want to get that new software in the hands of users, you need to be moving at that pace. You're moving faster and faster. We need to be keeping up with them, or in some cases maybe even going faster than they are. That's the point. But then you've got to deliver it to your users and your developers who need to use the damn thing. So it has to be built and tested, and then integrated together in a consistent way. And this is a problem that I don't think many rolling distributions really get a complete handle on. There's a tab in distribution side to think of, oh, we're just putting together a bunch of packages and then it's a collection. No, we're shipping an operating system that contains a bunch of packages, but we have to get all of that integration right. And that's more true with a rolling release than anything else. And also, that also means finding a way to shield those users, not from breakage, I'll talk about that next, but from those behavioral changes. Upstream has made some crazy decision to change something in their application. What if the user who's got his project to do tomorrow doesn't know how to do that yet? Doesn't know how to use that new feature or deal with those new defaults. So there has to be some way of shielding those users from the unexpected behavioral changes just so they can get on with the work they've got to get on and learn the new feature when they have the time to sit down and learn it. On the reliability side of things, rolling releases have typically get accused of always breaking. It's got thousands of moving parts from thousands of different upstream projects. Getting all that working together, especially when those upstream projects have dependencies somewhere in between and they're moving at different places is a constant challenge. Just like I said, when it comes to the stability side of things, we have to build this together. We have to test it together and we have to integrate it as one consistent distribution. This is not a piece of parts. The rolling release has to be everything done as a consistent distro. In practical terms, that means testing at the submission. So each submission being tested to make sure that isn't going to break everything and then also not forgetting about testing as a whole. It doesn't matter if that one package is fine, if when you put that in it breaks 20 other things. So testing the entire distribution, including all of those changes in one go. And obviously, users shouldn't be shipped something that doesn't work. This isn't a dev branch we're talking about here and really dev branches that break aren't that useful anyway. The thing has to at least work in some basic functional sense. And then I looked at how other distributions deal with testing. Most of us deal with testing. We used to deal with it this way. The concept of passive testing, we put it in some testing branch. Debian testing, whatever. And just leave it there for a couple of weeks and trust that the community will go away and test it. And then if they haven't found any bugs, that's good enough. We ship it. It's madness, but we all do it. But it's absolute madness. And it works better or worse depending on how big your community is. But it's still playing Russian roulette. Okay, you've got a huge pile of testers. Okay, you've got more chambers in the gun, you're slightly less likely to get your head shut off. But if you've got lots of developers, you've got more bullets in the gun. Eventually, someone's going to get shot. So for a rolling release, you need to actively confirm this thing will work. Does it work in isolation? Does it work in a broader context? And does the behavior change in the user, the way a user expects? You need to shield the users from that. You need to at least be able to warn them, okay, this package is going to completely change how these commands work. This UI element is going to change. You need to be aware that this stuff is going to start hitting your users before it hits them. But you need to have all those answers, ideally, before upstream have even finished their release. Or at least as fast as you can afterwards, because you need to be shipping this stuff as fast as you can after the upstream projects have finished what they're doing. And this is really the ultimate challenge of the whole thing. And well, yeah, the solutions are hopefully quite interesting. And then talking about things hard to live with, when you look at rolling releases, Arch have a great philosophy that I really agree with. Do it yourself. It's a learning exercise. For the Arch, it works really, really well. The Gentoo way, they don't have a nice definition, but that's kind of my variant of it. Do it yourself, and then you've got time to read the ArchWiki to figure out what's going on while the whole thing's compiling. And with both of these distributions and a lot of others, if something goes wrong, that's it. You're on your own. Good luck. Get out VM or Emacs. Yeah, you might get lucky to fix it. But who said rolling releases had to be difficult? Those fit for their users fine, but the problems are much broader than that. And I think the solution is actually much, much broader than that. And this is where Tumbleweed came in. Now, I'd love to be to take credit and say it was all my idea, but it wasn't. Tumbleweed initially started in the open SUSE community by Greg Crow Hartman. He was working at SUSE at the time, and what SUSE was doing and open SUSE was doing with our traditional releases was way too boring for him. Upstream kernel hackers, he needs something completely different. And it originally started as effectively an add-on to the original open SUSE distribution. So we started with the base system and did rolling stuff on top. Now, over the last couple of years, it's totally transformed from that model to what I'm going to talk about in a minute, where it's now moving at the pace of contribution. The entire code base will move as fast as our contributors can get those stuff packaged. And we have a process and the tools and the technology to move that incredibly quickly with those upstream projects as long as we have the contributors to do it. It's continuously tested, and when it comes to who we're expecting to use it, well, basically anybody who comes to Faustump. I mean, you are the target audience for Tumbleweed perfectly. As I mentioned with the original Tumbleweed, so this is before 2014, it was a rolling update repository that we added to the existing open SUSE distribution. So you started with the base system. We released that every six months or eight months or a year, depending on how things were going. And then we had a roll on top of that. The focus, obviously, with Greg, he was interested in the kernel, his desktop environment of choice, and the community added other things as well. So it's basically a bit of user space and a bit of kernel space, and most of the stuff in between, we just relied on the stable base to work fine. And if it didn't work fine, then Tumbleweed would quite often just overwrite those packages with whatever it needed in that rolling bit. So, you know, it was a Franken SUSE. It really was built that way by design, partly because it was easy to maintain that way. And then, of course, every eight months, we'd do another open SUSE release and then have to reset the whole thing to zero again and start from scratch, which was a huge problem. So the partially rolling, the Franken SUSE approach, it just did not work. We tried it for several years. The main problems we had were packages breaking sort of over that chasm between the stable base and the rolling part. You know, some dependency requirements or whatever. You know, just weird behavior issues appearing out of nowhere, weird dependency issues, seg faults you couldn't believe. And, of course, you can deal with those issues one by one, and then you ad hoc replace those broken parts so the Tumbleweed rolling stack on top gets a little bit bigger because it's more from your base system. But then your stable base is no longer stable. So you've defeated the point of trying to be partially rolling. And then this reset to zero meant you'd be happily rolling along and then we'd do a new major release and you're back to square one and starting again. Which, yeah, it was just painfully disruptive for users. So often we avoided rolling back package versions, but it was still a complete script because he really had two different communities working on it. The stable guy is working on the stable distro and the Tumbleweed guy is packaging things slightly differently and, you know, targeting their audience slightly differently. And looking at this, the open SUSE team at SUSE spent a really long time analyzing how to actually fix sort of all of these problems properly. I mean, they started first looking at our dev branch and looking at how do we get that more stable. I kind of started looking at it from the other side of how do we get Tumbleweed more stable and we kind of ended up meeting in the middle. And we found what I really think is a universal truth that when you're building a distribution with, you know, several thousand moving parts, if you need to move, if you need to be ready to move any part of that, you know, any package anywhere, you've got to be prepared to move everything. You've got to have the tools, the techniques, the processes that, okay, that new dependency from upstream is going to, you know, require you to, yes, change your init system and your kernel and God knows what else. And you have to be able to be prepared to do that at a moment's notice. And if you're not able to do that, you're not able to do a rolling release consistently. You might be to start for a short period of time. Sooner or later, you will hit a brick wall and stop and have real pain. So to get this solved properly, you need to always be ready to change everything at a moment's notice. Thankfully, in OpenSusa, we had two Aces of us leave. The OpenSusa Build Service, which is what we've been using for years for building all of our distributions. And it really has always embraced and tackled this idea of distribution is a cohesive product. You know, we build all our packages there, but the Build Service is aware of all those dependencies. If it needs to rebuild part of those packages, it does, so it can rebuild part of the dependency tree. It's dependency aware. It's what OpenSusa and Sousa have been using for years for building our distributions. It's also cross-distribution support, which, you know, that kind of challenge of being able to build for other distros as well really taught us a heck of a lot of how to handle dependency trees, because Debian does it way different from RPM and Arch. We even had one guy who passed in Gen2 support and he was pointless but beautiful. And, yeah, it's used by, it's not used by us, you know, on Cloud Linux Foundation, use it for Tizen, VLC, plenty of other people use it too. So we had the tools for building and, you know, really kind of helped with that rolling stuff already, but you've got to test the thing as well. And, yeah, my other job when I'm not doing OpenSusa stuff is working in QA. OpenQA is our really nice testing tool for this. We have a boost-outs test for this weekend. Unlike every other testing tool I've ever found, this tests from a user's perspective. You know, we're not necessarily caring about the code. It's API-neutral. You don't have to teach it what the API needs to do. You tell OpenQA what is the user expected to do with this software. What commands are they going to type? What buttons are they going to click? Because OpenQA is also graphically aware. So we can actually look at the screen. We have OpenCV running on there. It can even actually do screenshot analysis or partial screenshot analysis. So is this UI element on the screen wherever or where it needs to be? And, yeah, therefore you can test the entire distribution end-to-end from the way the users are going to use the damn thing. Without this, we couldn't do Tumbleweed the way we're doing today. And in fact, it's adoption in a serious sense really began with Tumbleweed. And since then, we've now started actually using it for the more stable distribution of OpenSUSA Leap and for the enterprise products at SUSA, which are all testing using the same framework. In terms of what it can test is pretty much anything. You have a nice dashboard with hundreds and hundreds of scenarios. I mean, I got to A to E here on one example. And in fact, it's a really bad screenshot missed out for the columns, but it's also got support for multiple architectures. So here we're testing Intel, but we can also test ARM, S390, Power, 32-bit Intel if you're interested. Mostly based on VMs although we have support now for testing on bare metal as well. So what it normally does, we'll start up with the distribution, fire it up in a VM, start testing it the way a user is going to do it, start the installation, run through the install, installs finished, start up the applications. If you want to feed it a disk image already of an existing installation, then you can do nice fancy stuff, like doing upgrade testing and migrations and all that kind of stuff. Has support for multiple machines, so you can do cluster tests and other weird and wonderful stuff with that. And like I said, it's graphically aware. So when something goes horribly wrong, you can actually see, okay, I was expecting a tumbleweed, a blue screen on the left there, but in fact the system running came up with the wrong graphic. So even those tiny little behavioral changes, which they're not bugs, it's just something that's changed upstream, get caught by OpenQA. If everything else has stayed the same way, if the UI is still working in the same way, OpenQA would just ignore it, it's fine. But if something's practically changed, the buttons moved, the commands changed its function, OpenQA will always find it. So with those tools, we've developed this process for effectively continuous integration for an entire distribution. When you're submitting something to OpenSUSE's tumbleweed, you're submitting it to an OBS project called Factory. That then goes into an automatic review queue, the usual kind of RPM lint stuff of checking basic sanity checks and unit tests and that kind of stuff, nothing too fancy. But for any package that has a potential system one, a package that has a potential system wide-breaking impact, which is pretty much every package, we have a process called staging where OpenQA will, sorry, OBS will spin up a variant of Factory, well, a variant of tumbleweed with everything that was working in the last release plus these changes and nothing else. So basically a what-if distribution, what if we merge this in OpenQA, so an actual proper smoke test, booting this thing up and installing it to make sure the distribution still works and that takes about 20 minutes. It's not a long, complicated process, so something gets checked in but this is before any human being has really actually looked at the code. No one's actually reviewed anything yet, no one's seen it, does it work? Is it worth looking at? If it doesn't, it gets rejected, Dev has to go away and start again. If it passes though, it ends up in the review queue. Then your usual kind of code reviews, spec file review, checking everything works. Then it ends up in Factory where all of this stuff is then put in a single repository. If we have lots of check-ins at the same time, you end up with a big pile of changes and then we test the entire distribution end-to-end in a few hundred different combinations on every architecture that we're interested in. That takes about, depending on the hardware, six to seven hours now. If we've got more hardware, it'd be shorter. So it's all very highly parallelized. That will run through all those tests. The QA is therefore totally automatic. We actually told OpenQA what the last acceptable failures were. Nothing's perfect, so there are sometimes acceptable failures in there. If that new build doesn't have any new failures, it ships, it becomes tumbleweed four in the morning whenever, it doesn't matter. So that whole process is completely pipelined and automatically done. So writing these automated tests for OpenQA totally gate the quality of our distribution. So it's great. If a user reports a bug now saying so and so broke and I was doing this, well just take that, translate it to the OpenQA descriptive language, you're never going to break that again. So every bug report becomes a nice easy test case in theory and you can shield yourself from ever making the same mistake twice. At a relatively light cost because you're already booting up the VM so it doesn't really take that long to pop in, do what the user's doing and done. So functionality checks, all that kind of stuff totally able to be shielded with it. But that also means if you do listen to my talk now and do download tumbleweed and find something you don't like, contributing to OpenQA is the easiest way of fixing that because as soon as you put a test into OpenQA it's never going to happen again. So the main website for OpenQA is Open.QA Documentation is there and we have a nice little bug tracker feature tracker for that. If you want to know more about the whole sort of pipelining and using OBS and OpenQA for continuous integration at 1.30 today which of course clashes with my other talking Janssen but you can pick which one you like more. My colleague Christian is talking here about how you can use this approach in your project. So not necessarily on a, well if you want to on a full distro level but even on just a smaller upstream project kind of thing you can use these tools to build your package in OBS test it in OpenQA follow the same kind of pipeline. Normally when I talk to people about this they say that's really fancy but I don't want to wait for all that testing. I just want it now it's too long. In fact there is one guy working at SUSE who does like rebuild all of factory just so he can avoid OpenQA he is crazy but he does do it. In reality though let's say working with upstream projects especially these really demanding ones like GNOME and KDE the process keeps pace with them upstream GNOME releases their tabls and within 48 hours upstream GNOME is in Open SUSE Tumbleweed faster than Gen2 even and piled that quickly. The process works even with these big upstream projects with hundreds of moving parts. In the case of KDE Plasma 4.9 which came out last week we actually shipped it on the same day that sounds really impressive in reality KDE kind of cheap because they give the tabls out five days before but yeah we shipped on release day it's kind of cool. We've also taken that process so we've got the build service so we can do all this fancy stuff and we actually have derivatives of Tumbleweed that kind of make that constant what if all the damn time because OBS is get aware so we can actually attach it to GNOME Git projects or KDE Git and then build GNOME Next or Open SUSE Krypton which are constantly building those packages as soon as something gets checked in and constantly testing those packages as soon as something gets checked in we get to file all these upstream bugs before anyone else does because we're seeing it in the context of an actual distribution all of the damn time but on the border Tumbleweed side every week now our Tumbleweed release manager Dominic writes a report of what's changed this week so the bots that do all the magic of the pipelining automatically pump out release reports and change logs of everything that changed but that's just ridiculously long so he tries to summarize it to something humans can do and he made the mistake last year of saying it's been a really quiet week this week so then I went through those logs and figured out what does quiet mean this was last year we did three snapshots snapshots are basically Tumbleweed releases so that whole pipelining end-to-end a new build three times so three new sets of ISOs three new repository versions all in there those versions all together contain the 146 updates we're still shipping DVDs ISOs at least so you know bit of shuffling around of packages 15 new packages on there 38 packages out to make room for it and one new kernel that's quiet that was a year ago six months ago this was it five snapshots every single workday we shipped another snapshot twice as many packages twice as many packages on the DVD twice as many we had to take out many kernels it's now six months later the last decent run drew we had 15 days of non-stop tumbleweed releases every day a new snapshot I didn't bother doing a slide for that because I can't be bothered to read that many logs I mean it's just crazy the process really really works and you can do a rolling distribution across the entire code base without any problems it just works from the user side of things though that can then be terrifying all this stuff is changing how did they deal with that OpenQA is testing it for a practical point of view it works but it's still changing the behavior is still going to change luckily with OpenSuser we've got SQL source we use BTRFS by default and we have it by default with a tool called snapper setup so on a standard tumbleweed machine of course you can disable it use whatever file system you want but in a normal configuration every single time you install a package on a tumbleweed machine you have a snapshot before and after so if that package changes something you don't like just roll back to the snapshot before keep on working go back and learn how it works whenever you have some time you're completely immunized from those behavioral changes you're also completely immunized from us screwing up your package accidentally because you can always roll back to how it worked yesterday and even if we completely balls everything up and OpenQA completely screws up and your machine doesn't boot anymore you can still boot to that snapshot from grub so there's no risk as long as you're using BTRFS and snapper and there's very little risk of us with that because we're using it so much it just doesn't break on our systems Austin will be talking about this more this afternoon actually more talking about the next generation of this idea of not just using snapshots as a safety net but actually more of an atomic way of delivering updates so actually doing transactional updates this concept plus plus but still using RPMs so that's tumbleweed what about the dev branch I've mentioned it's got a part in the process but in reality once we started doing this and delivering tumbleweed to users we realized we don't need a dev branch anymore factory no longer can be used by a human being it exists in OBS tucked away and we never publish it we don't have a traditional dev branch anymore it might be broken OpenQA or tell us if it is we won't make it tumbleweed if it worked it's tumbleweed so the old Greg's tumbleweed we basically took the name the old factory we put added testing and then we called this a new tumbleweed so it's a bit of a new distribution and yeah no more dev branches we're rolling on for that and it serves that same purpose while also serving all those other purposes of rolling release does of hitting a bigger user base being more useful dealing with upstream projects etc but what about regular releases I mean you know there's still a use case for that maybe not for the typical foster audience but you know other people you know don't want to necessarily have everything changing all the time well in OpenSUSE we kind of killed that idea off as well partly because of declining interest in the concept but also because of an alternative concept that's way more interesting we now have a distribution called LEAP which starts with the SusanLinux Enterprise Codebase which is now totally in OBS so we build that you know kind of like CentOS I guess but we don't limit it to be a one to one match the community then contributes to that codebase as well and adds whatever packages they want in addition to so you end up with a full blown community distribution or fully tested or fully integrated but with that nice Enterprise codebase underneath it so it's rock rock rock solid and stable and much more conservative with its pace of change so in the past you know everybody knew us as OpenSUSE the one distribution disc project we now have two tumbleweed rolling ahead continuously updated continuously tested like I said perfect for the foster audience and then LEAP with a shared core underneath it sharing it with SLEE and you know much more conservative anybody who doesn't like things changing often who wants to update once a year perfect for them SIS admins your little server in the corner that you don't want to touch that often for patching I mean you can run a rolling release on a server nothing's wrong with that but you know you're going to have to find some nice way of automating those updates you don't have to worry about that with LEAP like I said it's kind of a combination of the OpenSUSE project and the SUSE Enterprise side of things and this is roughly kind of roughly how the whole thing looks so you know there is a shared core of a nice stable base system which is also what the Enterprise side is using there's a small amount of Enterprise packages which just don't make any sense in a community sense or you know covered by licenses we don't want to ship or you know whatever just you know doesn't make sense there and then OpenSUSE has thousands of community packages sitting on top of that shared base and most of those community packages isn't all of them actually originating some form another from Tumbleweed where you know it's a pure community distribution all those packages always moving always rolling on the interesting question I get normally get asked at this point is well what about SLEE because you know everyone knows this model it's like you know Fedora with Red Hat you know you do a Fedora release based on it sometime after you're not doing an OpenSUSE release anymore you can't really base SLEE on something based on SLEE that all gets a little bit circular you know what is SUSE going to do for their next release of the Enterprise product well they're going to base it on Tumbleweed not in the sense that they're going to do a rolling release but the Tumbleweed base system that's constantly rolling is stable enough and usable enough that SUSE Linux will be basing SLEE 13 on Tumbleweed forking from it at some point for their next release for SLEE 13 so not only does the process work for you know other guys across them even enterprises are looking at this and like that's the best way of us developing our next major releases we don't need this old model of a regular release in between and all those hassles with that or the work that comes with that and when you then also tie it with the leap side of things where leap of course is based on the Enterprise service packs and that stuff as well but with community stuff on top you end up with this weird nice complicated sort of infinity loop of open SUSE is simultaneously downstream and upstream of SLEE and you know something that the community does in the last leap version is a candidate for the next service pack and you know it makes things actually way more interesting with that whole working with a corporate partner side of things because they have a lot more options on the table and you know less things get less stagnant which also means you end up with more enterprise developers working on your code base which is good for polish and everything working nice so in review if you're a developer and you want to deal with you know if you're dealing with any upstream project Tumbleweed is a perfect option for you it's going to keep up with those upstream projects more upstream projects than you're interested in you're going to get the latest and greatest of everything it just works and if it doesn't work exactly the way you want it to it's really easy to contribute to you know we've got OBS and we've got open QA so you can make sure we don't break it the same way so yeah and of course you've got snappers so you can just get back on when you're working in the meantime when you haven't got the time to deal with that so it's a perfect platform to work on and work with us on especially if you're an upstream developer as well targeting Tumbleweed seems like a natural choice for quite a few projects you know it's a distribution there we have all these tools you can do stuff with us first add a nice paste, get everything checked with the latest kernel and the latest system D and the latest python or whatever and then worry about how those other slower distributions are going to have to deal with it and if you want to then use our tools like OBS to actually do that building it's perfect because you can actually build originally for the open SUSE and then flick a few switches and build for a different RPM distribution and then add a Debian file and flick a few files and you're building for Debian the build service takes care of all of that so it's a perfect platform for that and like I say containerized applications are cool for those other distributions but there are actually more work for you in the long run that's what I'm talking about later this is actually less work the community takes care of a good chunk of this you just have to worry about your little path on top and if you're a user you manage to get the latest and greatest of everything it just works if it doesn't snappers there to take care of it and it's a perfect place to start contributing because that barrier of entry is so low it's easy to just throw something at the build service and say hey I want to submit this if it doesn't work it would be open QA or kick it out right away it's easy to start contributing with us so it's a perfect platform to start with that you don't just have to take my word for it because we've actually been looking at the statistics of how many people are using tumbleweed since this transition from partially rolling and the dev base the dev branch to just having a pure rolling release and no dev branch and our user numbers are looking quite good if you look purely from a dev branch perspective our dev branch now has 10 times more users than we ever had if you factor in the partially rolling distribution it's growing at 67% a year and really weirdly those weird little peaks around April seem to match up with the Buntu releases and I'm not entirely sure what that means but yeah kind of fun so yeah that's it for me I have a bit of time I think so yeah 10 minutes so 10 minutes for questions yes Land how do you report it back against a rolling release so that you can reproduce it where's the reference that I can give you so you are the same code base to reproduce what I'm trying to report the if very rarely does a problem actually live in just that but if it happens because we have this snapshot model so effectively every single tumbleweed version you're using is a entirely cohesive distro you actually have a version number for your distro in ETC OS release for the date for the day of the snapshot so you can mention that there we can then fish it out of OBS and run that there what we'll normally do is just see does this happen on the latest version because if we fixed it already that makes it really easy to close the bug so while it's rolling you still basically have a distribution release and another release that's part of the whole kind of building it consistently sometimes maintainers of development projects they refuse to enable the OBS tumbleweed build target and they say oh that's already factory enabled but when I contribute packages I'd like to make sure that there are two purposes contribute back but my main purpose is to get the package built for OBS tumbleweed for my running system very soon so what's your perspective should every seven projects enable OBS tumbleweed this snapshot as a build target generally speaking most developed projects should well most developed projects shouldn't need to enable it because they should be submitting everything into tumbleweed already so this shouldn't really be an issue in the sense of OBS but I want to have it but I want to have it now you want to forget the testing and you just want it now usually I contribute when I make sure that it works for me so I'd like to have the package now for this particular tumbleweed snapshot not factory with that diff of potential day in that case persuade them if they get stuck call the board we can help out with that there's no problem from a build side of things yes producing tests passively with the users and now you use OBS in writing test and what the progression of contribution versus what Susie is doing can you repeat the question? we were passively testing in the past we're now actively testing who's writing those tests is it Susie or the community because open QA's adoption really started in the community most of the tests were open Susie tests first and it's only in the last year or two where Susie has been actively contributing to open QA it's a community project first which now has enterprise guys to most of the tests come from that original group original contributors and now the enterprise side of things the process is totally open all the tests are in github even for the enterprise stuff we've left that open as well relatively easy to write a test submitted in generally speaking there's no objection to this but generally speaking we don't get that many contributions individuals outside I guess the kind of tumbleweed release management loop people who are actively sitting there working on the factory process contributing a lot maybe the release manager for tumbleweed they're the ones who see the things break initially before anyone's actually got it because we haven't shipped it yet and they'll just throw together a test and patch it in I'd like to get that a bit more broad that's one of the reasons I'm here is if there's stuff in tumbleweed and you just don't like it and it's missed in the current process that's why I've got those links in the slide so you can broaden that out a little bit because we don't scale but the process does yes then it's still being able to sort all the other more commonly slower distributions if you're a project depending on certain frameworks and if they change you have to come up with lots of quirks to support the latest version of the framework for tumbleweed but still being backwards comparable to the old stuff in those traditional distributions that have a large use of it it's a challenge it's a challenge but then when you also look at the trend even enterprise distributions playing around with the idea of things like modules and certain parts of enterprise distributions moving faster than others it's a problem that everybody's going to have to deal with at some point the reality is we can't the regular model doesn't quite work what I actually think the solution might be is taking this process and finding some way of defining a more moderate cadence tumbleweed runs as fast as we can contribute there's a rolling release model that fits that problem nicer of something that kind of runs a little bit slower to fit but everybody else can keep up with it yes how big is the gap between the traditional releases and the tumbleweed especially when you want to distribute the 42-3 the stable kernel based on the and the tumbleweed just constantly moving so I'm more interested in the next temporaries the gap is large and admittedly does get larger it gets larger the more these service packs are it's a fact of life that stable base isn't going to move that fast however if you look at for example how SUSE has done SLEE 12 compared to SLEE 11 the amount of change in each enterprise service pack has been significantly more and faster than in the traditional sort of SLEE 11 SLEE 10 in an earlier era part of the reason SUSE has been able to do that is because open SUSE is doing what we're doing with leap and tumbleweed so things like jumping a new kernel version in service pack 2 isn't quite so terrifying when tumbleweed already was running it for a year yeah it's so you see this kind of balance of enterprise destroyers catching up because the rolling release is going really fast him first sorry yes so you talked a lot about the advantages of this release model if you have to change one thing or improve one thing in the tumbleweed release model what would it be that's a really good question okay the one thing to really improve right now is actually have a delay between building and actually publishing right now as soon as it's built and tested and everything's fine we're publishing it we need to put a delay in there because now we have all that crazy number of users our mirror infrastructure can't keep up so if you're one of those poor guys who hits that new snapshot first thing before our mirrors have synced yeah it's going to be slow so we've really got to find a way of teaching the build service to okay it's ready push it out to all the mirrors and then flag it up for everybody to use and that's on our list at the moment because it's painful at the moment okay I'd like to add something to this because I run several VMs with tumbleweed and I think one improvement could be to have an easy way to have a caching proxy you know if you run several systems and you have a caching proxy to save your band we actually have instructions on the wiki how to do that already in the mirror go ahead actually we have an asking server you can do all of that another question would be will this new release model with co-op 13 or improve the confusion about because on the open source factory mailing list even others even people more familiar with all this are sometimes confused of what to take from leave and what to take from you know and then they say okay there's a list it's called from Slee yeah pulled from Slee pulled from factory but then you also see oh there's pulled from older leave honestly I'm pretty much confused part of the reason for that confusion this is all a relatively new concept when we first started Leap we purposefully didn't define where this line was between the shared core and the community stuff because we wanted the community to help define that so yeah you know that's the safe bet push the factory push and tumbleweed it'll always work it'll always get to where it needs to go but the release of 42.1 yeah we were purposefully leaving it very open to divide that line around 42.2 we made it way