 Hello everyone So next up we have Richard Brown and he's going to resurrect some dinosaurs for us And let's see what can go wrong with that Thank you. Thank you very much Yeah, so I'm here from the open-suser project to talk about our containerized applications so things like snappy flatback etc Like resurrecting dinosaurs Jurassic Park eating our users how it's all going to be terrible So Yeah, those who can't remember the past I doomed to repeat it. So before I get into the history lesson I'm going to talk a little bit about my history I've been working in in linux now for Let's say 12 years, maybe even a little bit longer mostly contributing to the open-suser project Qa distribution building package building. So I'm a you know, a real distro guy, you know, I'm really interested in Dealing with all of these upstream projects how those how to then package that software and then redistribute it to users You know, traditionally in the form of a distribution, you know from suzer And so and so I have kind of been looking at this problem from from very much that perspective Before that or you know alongside that I have also been, you know working as a system administrator So I've been actually dealing on the you know hard end of things with users in the real world dealing with software and you know Dealing with operating systems other than the wonderful ones that work in open source So starting at the very beginning I never thought I'd be presenting at foster and are having an opening slide of a windows 95 architecture diagram But this is the place to start with really back in windows 95 windows windows 3.1 The traditional dependency problem is best exemplified by dl or hell You know windows 95 windows was trying to You know solve all these wonderful dependency problems and their solution was pretty much Okay, let's have these things called dlls and let's shove all these libraries and dump them in c windows or c windows system And every application will just consume them from there and everything will be wonderful But of course it's windows so it wasn't You know, it's even even down to Absolutely nobody not even Microsoft taking care of any kind of a bi backwards compatibility Every single update either from an application or from the system Goes and touches those dlls And then every application that required on it all end up magically breaking Even if you had people who took care of that and dealt with that and actually had Certain run times and libraries prison, you know, take it, you know well maintained and patched carefully and reservedly You still had nonsense like the global com com class IDs So, you know a single application ends up stealing the resources from another If any programmer wanted to create an instance of that class it just always broke. It was a mess. It was a nightmare And as a developer Working with that stuff In real terms, it was a nightmare to work with, you know, every single application had to be Developed with every single combination of every single set of libraries that every user might have and whichever version combination they might have And then deved that way and tested that way And then of course you need to do an upgrade to your application So your patch you have to then retest everything And then of course your various upstream and other projects are changing their dlls And they're having you have to repatch all of that and then even if you did everything perfectly You still end up crying because it's still ended up breaking And then Windows 2000 came along and Microsoft claimed they fixed it And in many ways they did It came along and they said okay, we realize this is a mess We're gonna stop putting all of our shared libraries in one big dumping ground in the system in the you know The root file system in c windows or c windows system And we've got this wonderful thing called side-by-side assembly Which is basically containerization. I mean obviously with the lack of sandboxing But generally speaking it completely changed how you practically delivered applications on a windows files on a window system Then all the developer had to do was take Their exe and the dlls that they cared about And they deliver them in an in an application folder On the user system and windows would give it a nice separate memory space load up those dlls only in that memory space Private to that application So you didn't have to worry about what the system dlls were going to do or mess around with that If you were relying on the system dlls You could be way more certain that no third party was going to go messing around and breaking them on your user's machines Because windows was now protecting every single system dll with their WSFP or WFP And for those handling the whole transition between 95 and 2000 they even had this wonderful tool dups So you could take a nasty old school application with everything scattered all over the place And run dups against it. It'll figure out where all those dlls hooks are and then you could just bundle all of that into your application folder Migrate everything along and everything was happy and everything was wonderful And you didn't even have to do it necessarily with a A windows 32 application. I mean, you know, they had run times in there four pozzics run times in there for os2 I don't know anyone who ever really used them in anger, but you know, they were there. This is the windows 2000 diagram So the problem was solved right nonsense. This is microsoft. We're talking about It was and is and remains an absolute security nightmare From me, you know, how many exploits are out there that are some random security dlls sitting in some random application folder That's security relevant that everybody is having a wonderful time exploiting because So and so hasn't updated that dll in their application It's constantly constantly happening and it's happening because also there is no centralized way of updating these applications So every single windows application developer then has to worry about, okay, how are we get we've got it? Okay, we've got this security issue. We fix this dll or we fixed our own application How do we then deliver that to our users? Oh, fine We'll just make an update and then you end up with the adobe update and steam and blah and blah and blah One issue that doesn't get talked about a lot, but especially in the open source side of things. I think gets Yeah, at least far more lights to print on it as well is when you're bundling these libraries Do the licenses actually Permit that use I mean, yeah, how many how many gpl cases do we have of some License just bundled in with application because nobody realized actually that can't be thrown away in there And this is something that happened more and more often in the windows world Of course, it wasn't terrible storage vendors love it because This ended up needing to be used more and more because you're putting on a whole bunch of dlls on the system But you know, they're the only people who are happy and you know I guess people like spending money on more hard disks Meanwhile, about the same time in linux land We were looking on smugly Because we thought we'd already fixed this Kind of and we'd fixed it because we delivered our software in distributions And I could talk about all the technical details of which distributions better and blah blah blah But purely from sort of a philosophical and organizational perspective Every distribution dealt with these issues And actually resolved them just by doing the work they were doing Regardless of their technical choices of package manager and build tool and other such things Because Any responsible distribution at least had a care about the security of the software they were delivering to their users They have security teams. They're auditing the packages. They're putting in their distribution They're monitoring cvs for the packages. They're putting in their distribution They're rolling out security updates when it goes wrong In the case of of enterprise or commercially linked distributions You know, you're also likely to have them on the embargoed security lists and therefore getting Information about those cvs before they end up being public So you've got a chance of at least getting an update out on the day of the announcement broader than that just not worrying about the security side of things You've got package maintainers who are, you know, they're out in the community wanting to package their software Keeping it updated putting it in the distribution. So the whole how do we maintain this stuff is partly handled by these Maintainers doing it just because they want to run their software On their distribution of choice And of course, like I said any responsible distribution as well also has a legal team Looking at their packages making sure that the license the licenses of the software that they're delivering Are compatible with each other? So you're not delivering it in a way that is invalid with the the desires or the requirements of the developer in question And just, you know, generally compliant. You're not out, you know, awfully breaking laws by Bundling stuff in that breaks licenses So even though we're kind of doing the windows 3.1 dumping everything in one shared bundle and generally live With shared libraries things aren't too bad and when we're talking about containerization and Yeah containerized tools like flatback and snappy, you know, everyone's talking about let's kill off shared libraries That's just, you know, it's a it's a mess. Um, you know, this space doesn't matter now anyway. Everyone's got bigger disks But when you start looking at the real work of maintaining everything Shared libraries have another benefit, you know, it's not just about the disk space Distributing fewer libraries means you've got less work worrying about that security the security implications You've got one set of libraries to worry about auditing one set of libraries to worry about patching Less manpower to maintain it And it's easy to view that legal compliance, especially when Despite everybody's best efforts, you know, projects love changing their licenses And there is nothing worse than One version of a product being project of on being on one license and then changing to another and figuring out Okay, can we can we ship both are they compatible? You know, we've we've seen that before where projects end up actually, you know We have to move because their license isn't actually compatible with their old one. It's it's you know So shared libraries do solve a lot of those issues by at least giving you one Contain, you know one integrated base once you've done the work to actually put that base together So mission accomplished. Well, no, you know the traditional distribution model also has its flaws And these are the flaws which containerized apps really Sell promises trying to fix You know, improving the compatibility so developers don't have to rewrite their application for all these different chosen bases of the distribution Obviously portability also so you can just do it once and then have it work everywhere And distributions typically like to do that work at a certain cadence And that cadence is unlikely to actually match the application developers cadence. So therefore The issue of pace of chains comes up, you know a new upstream release has happened And no distribution is shipping that new version because you know their release schedule says they'll do it in eight months or whatever But the reason they do that is because it doesn't yeah, they want to make sure it works. So There we are Like I said a little bit like the windows 3.1 situation the compatibility problem The containerized apps are trying to solve Really boils down to these simple points, you know, we we have lots of distributions You know, maybe too many of them With many different libraries different apps and different modifications in there The applications themselves require different libraries And application developers don't care and they don't want to care and they're never going to want to care They just want to build something that works and push it out there Which is fine But most people But we've solved this problem in the in the distribution world already We have maintainers who deal with all that work. It's not upstream packages putting stuff in distributions most of the time It's distribution packages figuring out what crazy stuff upstream have done and then distributing it in the distribution And it works wonderfully when it comes to free and open source license applications obviously Provider applications whole different story On the portability side of things much the same story much the same problem You know, if you're doing all of this work to get the thing working You only want to do it once you don't want to have to repeat this three times four times five times And with a very diverse distribution ecosystem Having a nice application that built ones and runs everywhere definitely has a certain appeal but again Does it really matter if somebody's rebuilding it anyway? And then the pace of change problem Like I say distributions generally have fixed release schedules And those release schedules generally mean the package versions are frozen in there any changes are very reluctant if they are going to be in there So that holds back to new versions new innovations new things Projects have released from users. That's a bad thing Rolling distributions are becoming more and more of a trend In my in the case of open suzer, we're now running, you know, we're now doing tumbleweed as one of our main distributions And this totally solves that problem In fact, in many cases, we're now not only keeping pace with upstream projects In some cases we sometimes outpace it and then have to actually slow things down So we're not shipping their next release before they are Yeah, because it's all in git. So we just tell plug our build service into git and everything's fine and wonderful kind of But that's the distro side of things Containerized applications want to solve these same problems And when I'm talking about this, I'm talking really about app image flat pack and snappy Of course app image started in 2003 flat pack 2013 and snappy So, you know flat pack have been doing this for a very sorry app image have been doing this for a very long time I'm not going to go into the ins and outs of you know, how they all individually work or compare them I'm talking in general terms because these issues are shared across all of them. So, you know, generally speaking They're but they produce a bundle Be it a tarball or some other way of packaging things up Containing an application and all the dependencies all of the libraries that it requires And then they run it in some kind of container or ch root or sandbox And they make huge promises these are all ripped straight from their website straight from their documentation You know all the compatibility problems are solved. You just put everything in there. Everything's going to work fine You know the portability problems are solved. You just put everything there. You can run out on every distribution Everything's absolutely fine and wonderful Pace of change. Well, you're the one doing it. You're doing the the upstream source You then can do your app image or flat back everything's fine and it's just going to work everything's perfect And then you get architecture diagrams like this Or like this one from flat back And Actually, it's a bit too fuzzy. So I'll stick with this one What you end up with these Tool sets actually doing if taking the application The set of dependencies which in the case of snappy and flat pack actually include frameworks or run times where there's effectively pre Pre-curated bundles of these applications all sitting there And then below that there's just the operating system and you don't have to worry about that You know in the case of snappy, it's you know, they're always going to say a bunch of I quite like flat pack because they just say an operating system, which is if you're going to do this That's how it should be The problem is You're making assumptions about what is actually Below that line Soon, you know, you've you've taken the dependency problem of okay, my application needed this library And now it's my application and all of its dependencies need This version of glibc or this kernel or whatever And no one's quite defined where that line is yet And there's no common base every distribution is different Even debian distributions in that family are different rpm distributions in that family are different We package things in different ways because we've solved these problems differently in our environments over the years So sooner or later as you start crossing that line between the containerized application And its run times and whatever it's getting from the base system It's gonna start breaking Like I said app image have been doing this since 2003 And they know this problem. They don't lie about this. They don't do graphs like this Instead in their documentation They outright say that the only way you can build a portable image with app image Is decide which distributions you're targeting And then look at every dependency you need that's not in any of those And put them all in there It'll work, but that's a huge amount of work But that's what you have to do to actually avoid this issue We're seeing this in open suzi with like steam which is doesn't quite fit into this model But you know steam is effectively Delivered on on Linux as a Ubuntu runtime with steam running on top. So, you know kind of following this same model Again that runtime makes assumptions as soon as we change something in our base system That isn't compatible with what they're expecting from a you know their Ubuntu one times expecting it all breaks You need to worry about the dependencies You've just shifted the problem from higher up the stack to lower down the stack So if you really want to make a portable application you have to put in every single dependency you might need If not expect crashes your users will not be able to use your application So what's the point? Well, like I said in the case of flatpack and snappy They're trying to mitigate this problem by doing runtime. So at least you say With flatpack you have like the gt the gnome runtime in there You effectively have all of the libraries all of the dependencies to do a gnome application sitting in that runtime So as a developer you can at least target that runtime know that's going to be there Generally hope that everything further down the stack is sane and okay And just get on and do the work and it does mitigate the problem But at the same time you've generated a whole nother one, which I'll talk about in a minute And the responsibility of that kind of middle distro becomes You know really hard to do their job right They have to then make sure that runtime is keeping up to date with everything that that runtime is secure that runtime is all patched And those answers are something which is still in a very gestative state at the moment I'm not sure about the real solution But looking at it from a distro perspective, you know way back when we had this concept of a linux standard base of you Know a common linux that every developer could just assume is underneath there um Of course, that was a pretty hard and pretty impossible agreement to make But now with these applications with containerization becoming more maybe it's time to revisit that idea Because like I say the dependency problem gets shifted around by these technologies Maybe collectively the distributions should be to get around the table and find out okay, you know, here's the new line Here's where we're going to tell every Container developer, you know, we'll worry about this and we'll do this in a roughly standard kind of way And then work on from there. So, you know, maybe we need a new lsp that actually works So the reality of the big promises of containerized applications is well, they're not going to be compatible You do you know, unless you're going to worry about every single dependency, it's not going to necessarily work They're not going to be portable unless you do every dependency for every distro Which you might possibly want to have a user running it on The pace of change thing that's solved it does solve that real problem But you can't say it's just going to work now You can maybe say it's going to work on certain distributions like app image suggests you should be saying But you can't guarantee it's going to work everywhere And this is just like you had in Windows 2000 We're repeating history with this Because even if you say this is still worth it and still worth going ahead With these technologies and still doing that way We're going to end up with security relevant libraries in all of these containers, you know You're bundling all those dependencies in there to make this work Who's going to be making sure they're valid? Who's going to be making sure they're secure? Who's going to be patching them? When they're in there How are you going to patch them? You know, okay, most you know snappy does have the concept of app stores But how do you guarantee then that everybody with your snap from every app store gets the right version at the right time? I mean that's going to be a bit of a mess You know, how do you how do you handle the package management for flatpack with app image? They don't even have a rough answer for that now as far as I can tell It's you know, just download a new bundle from the website That's you know, they're going to have to have people creating updated apps for this thing Who's doing the legal review for all of these bund all these by these dependencies in these bundles Who's making sure that they all can be distributed at the same time? Especially when you look at certain licenses where The the runtime aspect of the license is fine. I mean, you know, oh, we can plug into anything and everything's fine But we care about how you distribute or redistribute the software That's where our lawyers inside open suzer really spend a lot of time and effort paying attention to you know Making sure that we can legally redistribute this software Who's going to be doing that for your application for these containerized applications? You know, if you've got thousands of people using these these tools to build all these applications Do you need thousands more lawyers? But of course, it's not all bad storage vendors are going to be even happier because you're going to be buying even bigger discs These tools are really powerful They are really cool But with great power comes great responsibilities And I think with with these athletes with these tools like snapback and flappy flappy flat pack and snappy the the only real solution If you're using them is to start thinking like a distribution developer and maintainer You're going to have to deal with the same responsibilities that we deal with in the distribution world You're going to have to consider How you bundle this how you maintain it? What is your update cadence? How can you how do you deal with the update cadence of all your dependencies? Do you therefore end up doing regular releases every few months? Are you you bundle them into your application? You know, where do you bundle them? Where do you draw that line? Which of course is something like I say the the new base might be an answer for You're still going to have to develop and test everything on every possible distro Unless you bundle every dependency that you need from anywhere You're still going to have this problem of making sure that does your bundle work on these distros You're still going to have to if you're if you're doing some kind of regular update cadence You're going to have to worry about all your patches and making sure they all work again And I suspect you're going to end up crying because it's all going to break anyway And you're going to have to worry about the security side of things you're going to have to worry about Auditing all not just your code, but all of the code of every dependency you've chosen to put in your bundle And you're going to be monitored have to monitor the cvs for those bundles. You're going to have to audit those libraries And as you heard in the previous talk You can't necessarily assume sandboxing is going to be enough You know sandboxing is a great safety net that some of these tools do recommend and do use But it should be thought of as a safety net not as a solution to these problems Security is something that should be integrated by design And then you have a safety net to cover your ass when it screws up You're going to have to have a nice fast way of getting that update through and out to users very very quickly And in a relatively efficient way And of course that gets tricky if you've got all these big bundles full of all of these big libraries many many times over And you know, let's just imagine that you know, everybody's got a copy of open as a cell inside there Inside their container How many times you're going to have to download a whole new bundle, you know on the same machine Does a user really want to download 40 gig just because you know, we fixed one issue in one lib That's happened to be in 40 different applications It's going to be painful And yeah Legal side of things We're going to have to you know Anyone using these tools is going to have to review the licenses going to have to make sure that what they're doing Is the both board is work is compatible And it's compliant with the licenses that not just the original software is included But all of the dependencies and libraries are involved I think distributions can and should be part of this solution You know, I've been up here now. I've been ranting about why all this technology is bad I actually like this idea. I like the idea of containerized applications because from a distribution point of view I'm going to have less work You know if application developers are doing all of this stuff and worrying about all of this stuff and distributing it I have less packages to worry about in my distribution. I have less responsibility for those packages I I actually want to encourage this idea because I think it is If done moderately and sensibly A more sustainable model than everything being centralized on a distribution But I just want to make sure my users don't get eaten by broken dangerous apps in the process So like I said, I think distribution should be Coming together thinking about a common base for where that line is between The you know underlying distribution side of things and all of the libraries on top Otherwise the portability promise is never going to be achievable unless that exists I don't really see everybody bundling everything in every application. That will just be crazy And I also think this you know with some of these tools is kind of a Tendency to kind of paint the distributions as a with it. This is the evil that the tool is expecting You know try to get rid of oh, you don't have to worry about sending your package submission anymore You don't have to worry about this crazy view. You don't have to worry about that Distributions have these tools and these processes and these techniques because we've been dealing with these problems in the real world for years So actually looking at our tools looking at how for example the open suzer-build service works looking at how open qa works Has a surprising relevance to these technologies and you don't have to necessarily reinvent every wheel The solution might be a simple case of patching in a new package format into those tools To then handle Yeah, flappy or Snappy or flatback One last final thing for this I'm not sure despite me liking the idea and thinking, you know, I like the idea of less work I'm not necessarily sure it's actually a that a sensible way of going along Because it comes from this concept of like I say traditional distributions are too slow But they're not if it's a rolling release And in the case of open suzer with tumbleweed we have a rolling release now That isn't just some nice fancy dev branch which anybody can go on and play But is actually usable for real work it gets tested and only gets updated when the thing has been tested And not just tested from a crazy code perspective But actually automation tools firing up vms and on real hardware and running the thing like a real user will use it So you know it installs right and you know the applications work, right? And we do all of that at a pace where we can still deliver the upstream projects on top at the same pace Those upstream projects work kde plasma on the same day of release known 48 hours that kind of thing Maybe that's the better model to actually fix this app delivery Maybe rolling releases are the way to actually go with all of this and just you know Reuse that knowledge we already have reuse the tools we already have we use the infrastructure We already have in these communities in these distributions And just angle them way faster And yeah way more We're well an awful lot faster than we've ever done in the past And with that I have time for questions There's one guy here There you go No but they're Keeping the base of the distribution static and Keeping the applications updated only might be a good middle ground for example Let's imagine using Ubuntu with a bunch of ppas To update just the applications and keeping a stable core so that we avoid Big differences and big changes break the users experience I really wish you'd been in my earlier talk in the distro dev room this morning because Yeah, I basically discussed that The problem I see with that idea is You end up with that same problem with the You know that line, you know the line between the fast moving containerized thing Just becomes a line between the fast moving applications and a static distro below And you end up with compatibility and reliability problems with that line In reality, if you want to do a rolling release properly, you have to be prepared to move everything Your application is going to want to change dependencies right down the stack Have the tools to change everything right down the stack and Yeah, you have to find some way of preserving its stability, but you can do that So, um When we're talking about moving the line, why don't you just move the line down And include the distribution with the whole container. So actually is really a virtual machine Well, then it's a virtual machine Yeah, then you end up with the issues of you know nice big applications lots of patching I mean you can But it's not exactly a very efficient way of doing things Yeah, I have Hi rich Hello So, uh In the past we've had several libraries and things like that that end up with the binary till mostly wire related issues Uh, especially With one application using one version not being able to load files or talk over the sockets with the other application This is not something that containers are solving. It's just making it worse. Nope Uh, is it something we can fix? I'm not asking for the fix. I'm asking if we actually can fix it I don't think it's something we can fix and I mean Basically that example you just gave and the example of non open source software Our two examples where I think this technology is awesome and should be used full stop no caveats You know as long as somebody takes care of all the responsibilities You know, that's that's where these actually solve a real problem But using saying this is a universal packaging tool for everything I am not convinced Here I have a question. I don't see where The the rolling release model, uh, how the really rolling release model could, uh Fix the test matrix problem. This is one of the problems we have in distributions is that An application may rely on several libraries with different versions and another one another Application would rely on other versions. You can test everything It increases complexity and that's why this is all put in containers Because you have the control as an application developer on which versions are used So I don't see how the rolling release solves the problem Is it supposed to solve that problem? I don't know I I think rolling releases go some way to solve that problem because they So assuming assuming your rolling release isn't just a collection of packages But is a a cohesive distribution that it's like built with for example, you know Dependency consistency and you know, you make sure it actually works So like with open suzer obs make sure all the dependencies actually work together and the distribution, you know is cohesive and works There's not just random packages all over the place So so in that case your build process is found we'll find out where you have these, you know multiple applications fighting for different libs Um, and then it's a case of either packaging or dealing with those upstreams It's these okay. This might help. This was kind of the example he gave there Of you know, there are there are cases where these tools helped, but I don't think they're universal Yeah, but for example in the GNOME project, we had some problems with changes in the CSS management in different gtk3 versions and so several versions of gtk3 are not Come to be installed in parallel So one application may rely on an older version And they're more recent and another version of gtk and You have to choose which version you will install so If you're really on a distribution model, you only have one of those versions installed and I don't see how it can solve without containers Don't I think the problem needs to be libs library maintainers need to stop actively supporting multiple versions that can't be installed in parallel Shifting the blame slightly. I know Hello On the first slide, I think the first value point you wrote was that with windows 3.1 And the shared library model system all libraries dump sucy windows system One of the major problems with it was and it really was in fact one of the major problems The missing backboard compatibility of the abis But the idea of that concept even back then was That you have backboard compatibility as long as the dll names taste the same So if you just use the newest dll with the same name or in linux You use the newest shared object with the same as own name. It should be abi compatible Yep So um and as you said Things break and we use this container stuff because We have abi breakages Although there should be none So how but how is defining a new standard base helping against the In my opinion real cause of the problem that we have applications relying on Properties of the abi that are not guaranteed and specified or library writers unintentionally Breaking their abis Well the idea my idea for the standard base would be when you had those abi breaking changes You know that would require a new standard base version So at least you're tracking when those changes are you know around It's another dependency problem. I admit it, but you know We we need to spend a bit of time thinking about this and you know Well, or the alternative is every single application developer using these tools has a lot more work to do Um, and I I'm I'm not sure either solution is wonderful But I don't think people really realized how much extra work they're putting on their shoulders just throwing everything into these tools So, uh, do you have already a plan to go forward? Like which of the suggestions that you proposed, uh, you think is the best way forward Like is open suze is like trying to propose a new linux standard base Well, this is the first I've said about it. So every distribution that starts talking to me now, they've seen this presentation I mean, you know, I'm not fussy. I mean, you know, it needs as many from as many places I wanted to start the conversation here first Guys no more questions Thank you. Thank you. Thank you very much