 So hi everyone, this is Peter. My name is Adam. We're from Red Hat and we are working on a project called Fedora modularity By the way, we have stickers for you. So if you come here after the talk you can get one or two Yeah, only 25, but that's fine So I'm not sure who of you was on Langmuir's talk last year here in FOSDEM He was talking about how distributions are great But not really and how we might need to change them to To be ready for the future. So let's start with why they are great because I think that's still true They ship package software, which means it's pretty easy to install. They also ship dependencies and everything else with it So it just works. They integrate it and test it. So nothing breaks really and some important thing it's also patched for security vulnerabilities because For example, if you are a developer and you're developing your application, which is included in the distribution You might not have pretty much time to patch for some security holes. So distributions can do it for you and These tools have a lifespan which basically means they're released at some point Die at some point. This is for example Fedora and if we have if you have a look at a bit more detail We can see that I have for example distribution version one it includes some libraries and software which is Pretty much frozen for the whole lifetime and then I have another distribution or another version and the same story there This is fine, but there are some problems because for example applications are Released at their own pace independent of distributions and They can also come in multiple versions, which means Conflicts right Distributions often include only one version of every application because they got dependencies They can conflict with each other and it will be impossible to install So we need to choose a version, but how to choose the right one? That's a tricky question Because for example, if you are a developer you might Prefer this one fast and latest because that's what developers want But if you're assist admin for example, you're running an application You don't want it to change so much right so you might Prefer the other scenario so I Guess the right answer. We need all of them. We can't just choose one But how to do it? Well, there are several several solutions to this so for example we can have Distribution on a fast track and another one which is a bit slower So for example, this can be Fedora. This can be CentOS or Ubuntu Ubuntu LTS if I'm a bit more crazy, maybe I can have continuous upgrades But this still doesn't allow me to mix and match versions as I want So another solutions are software collections. How many of you heard about software collections? All right, so that's like half so software collections are kind of special Packages which entals of her in a separate space so I can Install like more versions of the software on the system But it's pretty hacky. It doesn't work always So that's a solution, but not ideal Another thing would be Linux containers Who are not who of you know? Linux containers Almost everyone that's great. So yeah, we know it's an isolated space which looks like system and it's for Single application and I can run multiple containers on system, but sometimes If I put stuff in the container, I need to look after for it. I might have problems with updates And still I'm building containers out of a single distribution. So If I install some different version of something this can get conflicted as well so That's we that's why we are working on modularity which kind of combines everything together and Now I have like one minute short animation with subtitles. So I won't be talking. So if you could please read This is a talk. I'm not supposed to be talking here. All right, so that's basically it. So it will look like this We will have something Some something called base runtime is just tiny system with a pretty long life cycle and another software and The software would be like How do you call it in English? Independent sorry independent of the distribution, right? So as I said base on time, this is what Peter is going to talk about Which is a small system Containing software with proven API stability like kernel and glibc and some come some basic tools to make it boot and the software run as a module and Module is similar concept to Android applications or iPhone applications So it's pretty independent on the system and it includes all the dependencies in itself and As I said modules are like applications. So it can be database. It can be Firefox, but it can be also lamp stack or Basically any group of packages and it will looks like this. So I will have packages and some metadata and I can also have packages as an API and as dependencies which The difference is that if I for example have a Firefox module The Firefox package will be API. This is the only package I care for but I can have all of dependencies libraries, etc Which are not guaranteed to be stable, but users won't care about this users care only about the API and Modules are defined as a by module MD file and Basically defines everything which I'll show you right now So this is the module MD. It's kind of cut off on the top, but there's nothing important Important is that we have some names stream version, which is an ID of the of the module Description licensing some references where is the project leaves backtracking some optional metadata and Here is the important part. So I have dependencies there is the base runtime and And To build it they are built requires which is based on time and based on time built I guess Peter will tell you more about it and These are lists of the components So I can include packages in my module and I can include another modules in my module Which might be useful? For example, if I have an application requiring Python, I don't have to list all the Python packages I just place the Python module and In the RBM world If I have a source package, I can build multiple binary packages of it but Oh, sorry So in the RPM world, I have source RPM packages Which are packages containing the source code and if I build it I can build multiple binaries and I don't want to Probably include all the binaries. They can be devil packages and stuff so I can filter them Thanks to this filter here. I can specify the RPMs API and in the future there will be more types of API just then not just RPMs and Also something called install profile, which is interesting Basically explains how to install the module and I can have more ways to installing So for example, if I have web server, I can install it as a production or as a developer With a different configuration So that was module MD and Module MD is modules are built in something called factory 2.0. I Don't have time to talk about factory 2.0, but if you can Google for example or find Talk recordings from Defcon, which was like last last week You can find some interesting stuff about that and Yeah, and when we build them modules are delivered as multiple artifacts which can be RPM repo linux container flatbacks OS 3 ISO, whatever and The distribution when we put it back together will look similar to this picture So I have base runtime the minimal system Peter will be talking about And I have the modules running on the system In this example, I have RPM packages here and container images here and As we can see as I say The module contains packages and all their dependencies, right? so there are Some challenges how to deal with for example conflicting packages in these two scenarios, so That's why we use technology similar to software collections to avoid that or we can use containers and What should I choose RPM so containers? Well, I don't really care because the workflow can be pretty similar So for example This is pretty bad, I will just make the window. Yeah, so In Fedora we use DNF to install software so I can type DNF installed HDTPD, which is a web server. I can Configure it by editing the configuration file. I Can add my index HTML and I can use system control start HDTPD So that's the workflow in current Fedora. This can be the workflow in the module Fedora and I can't say if it's container or RPM packages. It's probably pretty same For both and maybe even a few days later. I can DNF update So the workflow will stay the same Even with modules, which is I guess important So how all of this works? So we focus on the package instead of on the distro version So for example in Fedora, we we store all the packages in this gate, which is distribution gate and Each package has similar has several branches. So for example f24 f25, which is like Fedora 24, Fedora 25, Fedora 26 and We want to change that to Reflect the package. So for example, I can have package and have branches according to the versions. So instead of building Distribution versions from the this gate. So taking like All the branches f24 to build for that 25 and talking all the F25 branches to build for that 20 sorry 25 I can use Modules to build modules Yeah, so I can in the module md. I can specify Packages I want to include in a module and I can just mix and match whatever I want and Then I can install it on base runtime we call it module streams and That's that's like very end of The module we don't call it versions because in some cases it would be true But if I have for example lamp stack, I can't say what this version of a lamp stack because I have a version of database I have a version of PHP So that's why you call it streams. And these are basically just variants of the modules and Before you install you will be able to choose the right version or you can just download ISO With the versions pre-selected so the user experience will be kind of same like with Fedora today and And you can use DNF DNF update will work. It will keep your system up to date So that's basically it about Modular distro and now better will tell you something about the base runtime, which is the minimal system I can give you a microphone I guess I will just hold it. Okay, so as I mentioned The base runtime is a module that sits below all the other modules all the other stacks that we have This part is meant generational cord though So I will provide some background on what generational core is what base runtime really is and the difference between them base runtime Was originally meant to be only the lower layer of the of this diagram It was meant to provide hardware abstraction layer with system tools utilities and short libraries to be provided by other modules Included in a stack that we wanted to call the generational core This initiative is temporarily on hold. However because of the complexity and the introduction of modularity to Fedora, we don't have all the resources and the plans and the And the clear idea how to actually separate it what what components go into which module and so on Also because the name is quite quite long and nobody could pronounce it. Nobody really understood what generational meant in this context We decided to just Broaden the definition of base runtime itself include some of the system tools and libraries Not to do not the share libraries layer. I will get to that But basically when we talk about the base in February 26 We mean base runtime no generational core. We must we may still revisit the idea of the generational core in the future So the implementation don't don't worry about the picture. It's not really important if you can't read it Yeah, it's a module like any other it's defined in a module and the file It lists all the components that we want to build as part of the base runtime module It provides a stable and minimal bootable system on bare metal machines on virtual machines it also defines the Container base image for both system system the end spawn and docker files just docker The components that we include were inspired by the LSB core set POSIX user land and the atomic host Currently we we include 700 binary RPMs defined by roughly 170 source RPMs when installed as a container base image for Docker. It's a 82 packages and Takes roughly 88 members max of this space How we actually implement it is that we have this the module I just described Which is something we something we want to ship, but we also have to build it somehow So we have another module called Temporarily based on time built environment which includes all the build dependencies of the components in based on time It also includes all the build dependencies of those build dependencies and all the entire recursive build dependency chain currently, it's roughly 2800 packages which one build is roughly six six thousands binary RPMs Although we will ship the base base runtime built environment We won't be supporting it in any way and in the future. We hope that Parts of the built environment will be split into into the applications themselves and base runtime will build up and only on itself and The applications that provide the components needed to build it So Okay, yeah, yeah, I will I will try yeah, I know I'm just quite speaker So the challenges that we face when developing based on time is that There's more than one day one way to do it actually so choosing the right content is it's the biggest challenge we decided to to go with the Yeah, well with the LSB core set and and the atomic host set and The posix user and set I mentioned in previously But it's definitely not the not the finalist We may or we may exclude some of those components or or add something else to the to it Keeping it as small as possible is important because we minimize both the memory this space The space footprint as well as the attack surface and of course smaller set means that we can rebuild it more much more faster and We can also test it faster another problem is Building the whole set we started with federal beta federal 25 beta and just hope that to Selected those 3,000 packages We needed for the for the base one time and based on time built environment and hope that they would magically rebuild themselves And it would just work That didn't work as expected we got like 400 fail to build from source issues in the big in the first run That was mostly because parallel was removed just before federal 25 beta was branched Many packages build require parallel, but don't explicitly state the build dependency So they were failing and it was mostly caused by auto tools and auto conf So we fix some of the issues and then we based our sets to use federal 25 for this candidate 3 which and the build dependency failures the build failures dropped to 180 back then So we created a new tracker back and we were reporting all the FTBS issues to the package maintainers It was mostly the The missing build dependencies, but it was also another more poor packaging practices for example and the lack of CI in federal I wasn't helping Another issue was the Unresponsive maintainers for example when we reported those FTB FS issues We gave all the maintainers two weeks to respond. Otherwise, we would just fix it for them In most cases we had to wait the whole two weeks because there was no response whatsoever In other cases The package maintainer had a different opinion how to fix the issue which is not always a problem But it it just slows down the progress Of course the discussion helps if it's if it has potential Another issue is that we are developing base runtime and all the modules in parallel with the traditional release We since we are trying to build it. We haven't actually got past that yet we are working with the frozen package set and Federal is still introducing new changes in raw height that includes gcc7 package conf And basically anything Also when a maintainer fixes an issue for example when with the building the package They often rebase the package which means new dependencies and new build failures caused by the by the rebase That's not always This is a picture with With the build dependency graph of the current base runtime self hosting prototype it includes those 3,000 packages And the build dependencies between them As you can see When trying to minimize the build dependency chain both for runtime or the self hosting prototype because it just takes forever to rebuild And with all those issues, it's it's just a nightmare Many of those dependencies are obsolete And could be removed because they were added like years ago and nobody really cares anymore But finding them that's that's the problem. I Can provide SVG picture later So what we plan to deliver in forever 26 The proof-of-concept base runtime module With the first version of the API Hopefully all the packages in the API will also ship with the devil sub packages so people can actually use our API That's not as easy as it sounds because the devil devil sub packages often They'll require stuff like pearl and we don't want pearl in base runtime, of course We will also ship the build build environment module although unsupported We will ship several Yeah, we will ship system and container Management modules such as DNF, which is not part of base runtime because it moves at a different pace The same for Docker for example We will also ship Yeah, like a proof-of-concept of the selection of the dynamic Dynamic languages modules mostly for Python Again the same situation as with pearl we don't want Python in the in the base runtime But many of our tools including our PM build unfortunately required In the long term we would either ship like a small set of Python standard library including like small small Python interpreters similar to system Python we have now But completely disconnected from the Python 3 federal ships today We will also ship project bolt run which is a funny name for federal 20s 26 server composed entirely out of modules Unfortunately federal 26 won't have any updates So the reason is big that body cannot handle any artifacts besides RPMs. We are working on fixing that though And it will be made entirely by the majority working group members the reason is again that we have no infrastructure for N processes to to process like this for example module submission requests and Changes in the package database 477 we hope that base runtime will be much more stable all the API will be usable The packages that are not meant to be a part of API will be repackaged and shipped differently So that couldn't be accessed by anybody who is not meant to be using them The same for the dynamic languages runtimes We hope to finish these system Python and Python trees plate for instance We will support automated builds and rebuilds. So whenever you push to this get you will you will Both build a component and the whole module and all the modules that depend on it We will ship updates The update for bodies is in plan The modularity infrastructure will be open to public so anybody will be able to submit a module and build it There will be a new new release of Fedora server and more content. It doesn't have a name that you receive and Beyond Fedora cloud Fedora workstation. Hopefully that will include modules like known for it for example Yeah, we we hope that module releases will become our primary deliverable and the traditional release will be still available But will be composed out of modules. So we will pre-select several modules for the users and flatten them out into a traditional release And it will be awesome Yeah, so that's the end we have plenty of time for questions if there any You Right so yeah, the question was if I for example have an application that depends on a library so how do I How do I make it not conflicting with other libraries, right? So as I said, I can maybe Switch back to my slides So You can use containers so the module can be installed as a container and the container will be we'll have solved that already Oh, if you want to install it as an RPM, we might need to repackage The libraries of the dependencies in a way that will be installed in a separate path It's content. It's content as it's a price Yes, yeah, so there's no way to share No, nothing containers containers are meant to contain everything in themselves, so Just to further cover that right so based runtime will have some libraries that are shared amongst other things Yes, well at the beginning it's probably gonna be more stuff than we would like So also, we would like that shared set to be smaller so you can have independent versions Right yeah Yeah, there needs to be optimizations, but yeah, that's I guess more for the future in the RPM worlds Yeah, if we have one library of a certain version I think it would be just once on the system still but yeah in the containers That's no choice basically, but we can still use layering. So for example, I can have the base on time layer there I can have some Libraries layer, etc. And if they're the same for the same container, they will share both storage and runtime resources If I understand correctly the problem you're trying to solve is The problem of installing mutually That's not the only thing another is that yeah, yeah another problem is that we want another problem we want to offer Applications not packages and we want them to offer them an upstream driven life cycles not the distribution driven life cycles So we will we will ship the RPM repository or container Depending as the as upstream releases new versions not as we just decided And what about Different The question is about it Basically the same the same packaging is basically the same version but built against different different versions of dependencies We will build all the variations against all the all the dependencies in the chain So yeah, there will be a lot of artifacts available in the repositories on the service and mirrors, but The install it the system management tools should choose correctly which module variant to install for you Well there is because we track how we build it so we know it Your opinion of which is correct So yeah, maybe I'll try to answer differently each module will contain all the dependencies, but also the build recipe so it will We always build the same time So for example, it will reference particular version of the base runtime It will reference all but basically if there are more versions of recent I really built against both against all of them but in a module I can specify all the packages by branches for example, but When I build them the both system will save the exact commission she's so it will be reproducible at all times and Yeah The module defines all the software it needs You can say Whatever that means exactly right if this isn't really meant to be exposed to end users per se Except that you may have multiple versions of something available to choose from so that doesn't mean that it'll be that you can Arbitrarily create your own modules what I mean as a developer you could but that's not the concept here The concept here is that these things are still all the fun of the server side much like our cams are today But the units of measure are quite a bit bigger for the most part so that The individual things and have individual life cycles that doesn't mean that you can't go in there and twiddle with the packages But you know you're gonna have a different user experience. Yeah. Yeah. I mean the reason why I'm asking those those questions is I mean this very problem. There's been a lot of interest in the last two or three years and most people have Tending towards the Functional declarative approach that's used by Systems such as But you're still adhering to the imperative approach. So obviously you don't think You don't agree with the other That's a big move So that's a that may be the long-term right answer, but that's a that's a gut rewrite And no way to distribute the existing So, you know, even even if we were a hundred percent sure that the next answer was the right answer We can't just turn it off, right? So, you know, I think we're kind of walking towards towards that direction Okay We'll meet you Perfection in the sense like we won't know so we try it. Yeah, right? So let's let's put it together It's not it's disruptive, but it's not train wrecking, right and see how it works I mean, you know, and then iterate rather than make a big bet There are questions Yeah, okay, so question was How do you run containers? Do we run them on Docker? Do we run them on system? The end spawn and the answer is we will be using OCI containers Which will be able to run on Docker run on Run C and for example, if you have a no CI image You can imagine I Had pretty nice example for example You can imagine PDF which could be built by different tools, but you can open it in the browser We can open it in this reader on your phone. So the OCI image will run on multiple Container engines, but I'm not sure which which would be the default We can do both. Yeah Yeah, I have to find it. Oh, there is it Huge Well, as you can see it'll be a lot of building because yeah when building a module we need to build all the packages But I hope there will be some optimization some nice filters to Not have the cover for the screen as that as big as on this picture. Yeah As for the comments were in container run times the container space is still a warfare zone so if we were declarative about any particular tech there we would be wrong, right? So the objective here, I think is that we want to do kind of the OCI model Which is the open standard and then use different technologies to run them depending on scenario Like, you know, there's some of the reasons for the system need container stuff This is because you sometimes want to run a container before something like a Docker game comes up So you need to work well just like everything else We have to be kind of flexible about the container run times and we want to use because we're gonna need different ones with different kinds of Yeah, right. So do we have any more questions? All right, so I guess we can wrap up Thanks for coming