 Okay. So we're recording. For those of you who don't know, I'm Dennis Gommel. I'm the head of the release engineering in Philadelphia. And today's talk I'm just going to go over what you need to do when you want to get something new into Fiddlewater and have it be part of Fiddlewater. So I was saying, where's the new thing? It's not any spin. It's not, you know, new R&D and it's not, there's something that we don't do today. So I was talking about, like, if we wanted to do something like, say, build RubyGems or Enchip or repo RubyGems or build Node.js or, you know, some new container form that comes along, that kind of thing. Something that's completely new, you know, modularity is something that's new that's being worked on at the moment. So talking about that kind of stuff, no, hey, I want to get this new spin in. We have a process for that. It's pretty well done. So what could be anything? What I'm just going to say, it could be Node.js, it could be RubyGems, it could be new containers, it could be... I mean, it could be anyone could come up with a whole new installation, like, you know, like a way to learn anaconda, but do installs via something other than anaconda or something new. The possibilities are endless as far as what is the new thing. So if you have a new thing, what do you do? You want to talk to a lease engineer and you want to talk to a lease engineer, it's really a... I'm going to go over why that is, but that's probably one of the most critical things that you need to do is engage with the lease engineer really early in the process. And the more invasive and the more new and the more different it is, the earlier you want to engage in the lease engineer. If you turn around and you write a whole bunch of stuff and say, here's my thing, use it. And we have zero way to actually integrate it into our lease process. We're going to have problems and we're going to class and it's going to take two or three times longer to get it into Fedora than it would if you engage with us early. You also need to be open, transparent, work with everyone, but just not just yourself. People have a tendency to want to do a thing and be like, hey, look, I made this thing that's awesome and shiny and new and wonderful. And everyone kind of goes, whoa, it doesn't mean that you can't do the work by yourself or do it. You could sit in the office, work for three months and then come out and go, here's the thing, you can do that, but it's better if you do it open. One of our requirements is that we have to have everything open in Fedora. If you want to build it, it has to be done using open tools. Years ago we offered to do three scans of all the code in Fedora and we had to say no because it's a proprietary solution. So anything you do is going to be open and you need to not align others to do the work. So instead of when people come up with new things, they're like, release engineering don't make this happen. And release engineering has very limited resources for doing new things where really busy people. So you can't rely on us to do all of the work for you. So you need to be in the open. You need to work in the open, you need to be trans-bound. Everything has to be open to us. Today it's a little tricky to do. You can have pieces of Fedora, but it's a little tricky to do all of the composed process in Fedora, but we want it to be so that it's completely open and you want to make them still DVD at home. You want to do a composite Fedora? You should be able to. There's no reason why. The main issue at the moment is that we don't test overly well doing it outside of the Fedora environment. And some pieces of it rely heavily on running things on the specific architectures. So we do that via Koji when we kind of walk down. We can do that because it has cool read-write access to mountain Koji. So it makes it a little tricky. Today is an integrative thing. When we build Fedora, we build the composer. We have this integrative process we run. And it outputs a massive, massive number of things. We have something like 20 different ICDs. We have four different pilot images. We have one set of installed DVDs, but we have four installed trees. We have nearly 20,000 source op-ends and something like 65,000 or 70,000 binary op-ends that make up Fedora. It's a massive thing. And if we end up trying to run a process that is on the side and is not integrated into how we make Fedora, that means we then have to run things multiple times. You know, we have to go, we run Fedora and then we're going to run this other thing. And then something else can go on and we're going to run this other thing. It just doesn't scale very well. So the way we scale how we can deliver more in Fedora is we have it tightly coupled and integrated. Ideally, we're looking at some automation so that we can perhaps decouple some of the pieces of Fedora and deliver them separately. We've kind of started that way for two weeks, but it's a two-week atomic compose. It's a composer Fedora. It's the subset of the bits that are inside of Fedora. So, you know, we need to integrate in with the Fedora processes. We ended up with 100 things that we need to run. We don't have the people to do that. We also have PDC in Fedora, which is new in Fedora 24. And it's supposed to be the source of all knowledge in what is intercomposed in a release. You should be able to go in and say, hey, those, you know, what was in Fedora 24 and you get the list of all the ISOs or installers, you know, all the stuff that goes in there. So if we have something not integrated, that makes it more difficult to integrate in PDC. We need to be able to reproduce. And what that means is we need to know, like inputs that went in, we need to know the build environment or compose environment. We need to be able to, you know, reproduce the deliverable at any point in time. So if we say, you know, in the case of install me, we know the environment we've been so nearly made in. We know the audience that went into it. We have to, you know, the logs of everything. So we, the disability is vital for a number of reasons. Main one being that we have an insurance. If we can reproduce something, we have an insurance that we actually know what went into it. We know how it was made and we know, you know, all the pieces about it so that, you know, we can redo it. And that gives us a level, particularly from a security point of view. And if the Fedora is less important to build, so certifications and stuff like that. If you get fixed certification or, you know, the different government certifications to be able to run the operating system in, you know, Department of Defenses and, you know, different government industries, different government departments require, you know, certifications. Fedora is less important. Well, it's much more important. Everything needs to be audible. It kind of ties into reproducibility. We need to know what went into it. What we got out of it. The, you know, process, you know, front to back. So that if somebody comes along and says, hey, I think that your, you know, Fedora installed our DVDs being tainted and it's installing malware. We can go back and look and go, no, we put this in. We went through this. We got this out and it's not tainted. And, you know, ideally we can reproduce it 100% and we can go look and remade it. And this matches that. There's no way that that thing is tainted or, you know, installing malware. So, all the ability. Everything enables accountability. So, that comes back to, you know, apiability, et cetera. We have the ability of, we know where the sources are. You know, we've got, we've got ZCAS, we've got disk care. It ensures that the package maintainer is accountable for what went into it. You know, we end up finding that somebody put pattern and code, or they put, you know, bad software, you know, the upstream got hacked and their table was compromised and maintained and put it in. We can go, okay, you know, Joe put this in on this day. And so with the ability of all of the inputs and all of the outputs, you know, we know who did it. We can, you know, talk to them if somebody does something wrong, you know, rather than try and scratch our heads and figure out who it is, we can very easily go and get that. So, anything that you want to do in Fedora that's new, we need to ensure that we have that full loadability. You know, let's just know what we shipped and how. Because, you know, if we, you know, if someone looked at the updates of the M repo and said, hey, where did this update come from? And we have no idea. That's great points for concern that somebody's done something that they shouldn't have. What we shipped needs to be definable. What that means is, you know, in the Fedora, the compose process, we have comfy files that define what we shipped in the Fedora. In the appian page, we have spec files that define how the spec file happens. In layer images, we have developer files that define, you know, what goes into it, what comes out of it. So everything that we do, we can't rely on some kind of magic to, you know, we do this thing and it goes up in the postdoc and we get this out. We need to know exactly what we're putting in, exactly what we're getting out, what we expect to have, what we expect to ship. It's pretty critical that we have that. You know, Fesco, QA, and Ryan don't need to know what's being done. And that's purely because, you know, if QA doesn't know what they need to test, what, you know, how can we be sure what we're going to ship? If Ryan doesn't know what they need to build and compose, you can't be sure what we're going to ship. You know, Fesco, as the technical allocator, it needs to know what's being done. So, you know, if you want something to be part of Fedora, you can't just go off on the side and say, hey, I made this thing over here and I did some stuff and, you know, I've got this whole new magic, you know, food thing and expect to have it called Fedora because with the Fedora name, at least we hope there is some level of trust that, you know, it's a known quantity, it's verifiable, it's, you know, you're not going to, you know, if it was a freefall, you wouldn't know whether, you know, is this thing good or bad, which is kind of some of the complaints of things like the Dr. Hub or, you know, like in the, someone was talking about, like, in the 50s though, I think they had some issues recently and had issues where, you know, malicious code got in there and, you know, we want, anything we'll be doing in Fedora we want to be able to say in Fedora, like if it says, this is Fedora, that, you know, people are going to trust that it is what we say it is and so it needs to be defined. It needs to be deliverable and by deliverable, it has quite a few things that we need to have happen and so with anything new, there's really no set, there's no set formula that says, you know, for Fedora we want, you know, you have to meet this exact template because, you know, what's deliverable at live CD? It just needs to be out on a mirror somewhere where the data on leaks can be directed. But with something like, say, LX3, deliverable is a very different thing. You need to, I mean, today I'm going to take LX3 for a little bit because it is one of the more recent things it really added and Ian's here who is now responsible for managing it to an extent. So today, LX3 doesn't have a talk to manager. It doesn't know how to use manager. It doesn't talk to manager manager. There's a lot of the newer technologies and one of the issues we're playing in that we need to soak with the light images is how do we redirect users to mirrors to get the content? We've got this mirror network that has, it has shrunk some of the years, but the mirror servers have gotten more powerful. Their bandwidth has gotten bigger. They're able to deliver, you know, more stuff for us. The story has gotten bigger for them, you know. Years ago, I used to say that we wouldn't go and go and tell about Fedora. It's probably been two years since we're under a terabyte of Pub Fedora. And that's just with, you know, I get the moment there's Fedora 22, 23, 24, and raw hide on that mirror. We're using it in Pub Fedora. We're putting some stuff in Pub All. But we're both putting it on Pub Fedora where sitting at about 1.4, 1.5 terabytes of disk. And the massive amount of scale of what we have in Fedora is huge. If we had to serve everything from one point, that point would just be a complete bottleneck and choke and, you know, people wouldn't be able to get Fedora. But in the, in the backs of the LHT guys, it doesn't matter how you're using mirrors. So every single user that is using LHT on a comic as shipped by Fedora is pointed to a single location to download all of their updates on. At the moment, it's not a big deal because there's not a massive amount of users. Likely any logist's deployment is, you know, either building the LHT trees or they're building, you know, they're mirroring the LHT internally and they're pointing all of their clients at their mirror on that as opposed to... But, you know, nothing other than using mirror management, nothing other than redirect downloads to different download servers has the ability to be out of code. So if you wanted to do something like, you know, say we wanted to turn on tomorrow and start working on producing our own Fedora Node.js because it has, you know, all these MDN packages. So the people doing node stuff can work in the way that they used to working, but they can pull stuff from Fedora knowing that it's being added, it's being built in a controlled, audible way. The content that goes in there is verifiable and all of the guarantees that we give as being part of Fedora. You know, if we wanted to do that, we wouldn't want to have a single, you know, to have everything run just with GeneX, which has a issue of not having IPv6, but has the other issue that everyone's pointing from the one location. So we need to have some kind of mirror integration into, you know, the NPM command so that, you know, we can point at our mirror list and, you know, redirect people to different mirrors that are close to them and get hopefully getting faster delivery. If I have a cool new thing, I think might be a great adoption. Yep. Do I need to boil the ocean and sort out how to do mirroring and not be so lucky as to have a problem with our cool new thing? It's so popular that we actually are struggling with a new laboratory. So I think that something new we probably don't have to have mirror integration on day one. You know, the process is an evolving process. But what we need to do is we need to have a plan that this is how it's going to work. You know, have that in mind for yesterday. Right. I mean, like I was saying before, there's nothing that's sitting comfy and there's no set template. It really is specifically depending on what it is you want to live on. You know, if all we're doing is coming up with a new, you know, say container format that can just be point, you know, we create this container, we put it on the mirror. You just, you know, like the user essentially W gets it. Then, you know, we could take advantage of, to an extent, we could take advantage of just using mirror-managed leader acts. Or, you know, in the case of the way it images, we likely need to have some mirror-managed integration. So, you know, how we live this up really depends on what it is, how users interact with it, how users fetch it, you know, like in the case of, like, the live CDs, it doesn't need any mirror-managed, real mirror-managed integration. We don't need meta-links and mirror lists and things like that, because you get it once and, you know, the user's manually getting it. So, you don't need to have, you know, if the download fails, the user can try again and hopefully get a different mirror that has it, or can redirect it somewhere else. But it's a user-driven thing where when you're doing this tooling, particularly the way you want to do, potentially fail over to, you know, multiple mirrors, so that, you know, if I request this thing from mirror A and it doesn't have it, why go to mirror B and mirror C until I get to a mirror that has, you know, the content that I want to have. In the case of models coming out, we're going to have to look at how the mirroring of that's going to work and how the mirror-managed integration and that's going to be, it's not something that I think anyone's looked at yet, but I think how are we going to live at the moment and everyone's focused on building a model as opposed to, you know, how are we going to get those people and how do we, and how do we enable people to sort, you know, the inner space to sort of get that. So, yeah, I mean, it needs to be doable. You kind of rely on a CDN, which a lot of the new technology seems to be wrapped around. Fedora doesn't have a CDN and Fedora doesn't have a budget for a CDN. We have a CDN that has been donated to us, but I've heard from some other open-source communities that that particular CDN, if you send too much traffic to them, they will eject and kick you off. So, we can't really, we can use it for some small things, but we can't. So, yeah, I mean, the critical thing Mary managed to and how are we going to take advantage of that, it's a lot of working with people. As much as the technology side of how you build it, how you ship it, it is important, the political side of working with people to ensure that we, you know, everyone's on board, everyone knows what's needed. It's also, you know, a critical step. So, quick summary, you need to talk, you need to communicate. You need to be open and transparent and you need to work with us. They're all essentially the same thing. And it's probably the most critical thing in getting something new into Fedora. I mean, nothing always start out afraid on the legal list. A couple of weeks ago, if you wanted to have a new trademark to enable people to be able to build things and ship them themselves, and there's a lot of questions around that and I'm not opposed to that, but I want to make sure that if we build that out where we, you know, have an age phase where people can build things from Fedora and ship it, there is not something that we do today that we ensure that the way that they do it is done with all these things in mind so that it becomes successful and people want to, you know, then integrate that into Fedora. We don't get to the case of, I'm going to get on the topic again, where, you know, color in the world there's a thing called IPMR Tree Toolbox which does all sorts of composing and all sorts of pieces and it's a great end-user tool but it's not a great tool to integrate within, you know, the process of how we compose Fedora as a whole and so, you know, it's been very difficult to get atomic into Fedora and that was the main thing that drove me to decide to submit this talk on getting things into Fedora because the only guarantee I think we have at the moment is that there's going to be more new things, you know, late images are coming out. I would like to see us actually do, like Node.js, RubyGem, you know, pipeline type repos of all the different stacks and big part because while I'm a big fan of RBMs and what it gives us, if you do a new install for Fedora today and you want to say, as soon as you've done the install, you go, you know, DNS install, IISSI. The first thing it does is download 42 meg of metadata for the RBMs. It's huge, particularly given that, you know, IISSI is a $200, $300K application and you download a massive amount of data to get this small little thing and, you know, the metadata for the repo, you know, for the Fedora repo is huge and it's getting bigger all the time. If you look at the stats of Fedora, I have to join the network. If you look at the stats of Fedora, the clicking steps on package TV this is the evolution of packages in Fedora. Fedora 1, 2 and up to 6. That is just one of the Fedora actions that was in Fedora Actors. It doesn't include 7 as well as 4 in extra as much and you can see it wasn't really that big it got 4, it got smaller in extra at that point in time but, yeah, at that point we had just under 5,000 packages in Fedora 7. This happened. This is just the number of source packages that make up Fedora and you see the progression because everyone likes putting all the packages in all of the latest releases of Fedora there's really not a big difference between 23, 24, 25 and 26 at this point in time but it's just under 20,000 source packages in Fedora at the moment. So 22 is starting to drop down below that because it's end of life but at the end of its life it has, you know, 18,500, I think source packages were in Fedora 22 at the end of its life. It's very large 17,000 and 25,000 and it frustrates a bit when we fill up a whole bunch of packages that are often for a while, numbers drop down and they start adding things back and it picks back up but Fedora had it spinning in place. I think it's only going to get bigger. So majority I think will help in making like the metadata smaller I would really love to have this not deliver so much as far as updates go so I'd like to get a still point where more high or something like more high is where developers use and how users live in a stable environment where people can be pretty sure that stuff doesn't break. A lot of the guarantees that you can get by using, say, Fedora 24 today. The metadata around it is really big so I'd like to make it smaller but the kind that you system in they like things that are at the end so I'd like to do things in a way where we can have a Node.js off M repo and have a Node.js M repo in Fedora and so the developer can develop in the way they like the transition man can deploy in the way he likes they get in the exact same thing they can build natively Node.js or Ruby.js or Python packages wrap up the end result into the end as well as into the native format enabling people to work in a way that they want to work while getting them the guarantees and assurances of Fedora something being that will take people that want to do it people are able to do it the time to do it. The least engineering historically is being way way on the resource where better than we were but we still don't have the resources to do anything everything we want to do so having the resources and we're going to talk to you we're going to work with you and enable you to do the thing you want to do in a way that when you're done we're going to be able to incorporate and add into Fedora is the end goal so does anyone have any questions it's a kind of kind of possibility of modularity but in the instance of say we've got RubyGems and we make a native RubyGem repo so you can type RubyGem install food and it's going to get food from the Fedora or RubyGem repo and install it. As a developer that makes it easy for you to develop your code and then assuming that we have everything for that you know the developer can do like RubyGem list or whatever I don't actually use RPMs myself but I know that there's a lot of developers like to work in the native format but you know there would be some method to list and say this is what I have in my dev environment they can give that list to the you know the system then and then there'll be a matching RPM repo where the system then can then install the matching RPMs of that. I kind of imagine that what it would actually be is we have you know we have a repo to say Python 2.7 maybe a repo for Python 3.6 you know one for Python 3.4 maybe one for 3.3 and it will be a a repo that's just for the Python release so if we have three versions of Python three versions of Fedora that have Python 3.7 which I think we've got more than that at the moment in Fedora it's either three or four versions of Fedora you can then enable that repo on any of those releases of Fedora and you get the same thing. It's not necessarily tied to the release so much as tied to the upstream version of the maybe component like Python or Ruby. Does that answer your question? This kind of speculation examples is not really anyone I just I think it would be kind of need to have to where we can divide it up and if you kind of use that thing you can enable and get the metadata but you can then perhaps take a bunch of the Python stuff out of an incredibly bad example but the Ruby stuff you can take out of Fedora and make the base Fedora metadata smaller because if you want it you need to enable this Fedora the question just kind of did it for the recording was if the new trademark is for new repository that people can build that stuff and ship it or like a profit type thing it's more for having a way for people to build things like say build I guess I'm going to pick another tree again build another tree and ship it and say it's enabled it's built from components that in Fedora itself but it's not actually a part of it's not like an official Fedora deliverable it's not part of what we define as this is Fedora it's certainly it's built from Fedora components but it's built by somebody else and it's not a official part of Fedora that's all it is I'm going to go ahead and count your presentation on how not you I think you did a little problem getting something into people I hope to run outside and I say this to someone who tries to do it right and I kind of the advantage of doing it I don't switch compared to and I know this is not a question it's a fact but you know that you are you know if you use a factory now you have a big assembly line that can turn this out like massive scale and a lot of people do interesting things about Fedora and we have to jump right from that to integrating on the factory floor which is a good separation I think part of it is we need to work on funding the converse tools and provide a new way to plug in new functionality we kind of have to say new encouraging ideally anything that ideally all the pieces on the component can be run locally on the machine you can do it on yourself but we also want to do it in curvy so that we have the central point where you want to see what is the latest you know what's the latest branch workstation iso or the latest branch you know cloud images or doctor base image or whatever you get a curvy and you can find that for all the different pieces that we do with but we also want to make sure that you at home can you know go off and make and install media to do something custom you want to do do everything that we do to make the product run yeah but I was just adding another great example of this I guess today we're going to go the build tooling that we're going into and the side piece that is the distribution the distribution the build actually should be on the full different design problems it's one of those things where I agree I agree with those three but I think a lot of it is the side effect of going from the from the garage to the manufacturer and then right and I think I think we're going to deal with that with all of these things because it's not even like it's not necessarily like a negative a negative it's not necessarily a negative you know it is we're really not in the business of saying you have to do it in a particular way I mean I push back on some of the knowledge because why do we use Docker which we don't use Docker today and it's not better to use Docker it is that I wasn't able to see the raw information on what went into the Docker containers what went into that environment and so long as we can get we can get that logging and that audibility out of it then I think to put that we need to be sure that we're going to get the audibility and to get those things out of it you make it expensive we did we used a lot of storage so we used a containerized environment a lot and as well understood we get the it's fine I mean if any of you understand Docker then we'll probably understand do you know in terms of product data we do but you know I mean it's much easier to get the audience that went in and as far as we can see on the outside we weren't getting that list of audience that went into the container so we didn't know what went in there but good we're not going to turn around and say you have to use X technology but if you come to us and say we're going to see how it does this and historically with perhaps being a little more stubborn than we could have been in part because 18 months, 2 years ago that went with me and we now have about 8, 9 people working on it but we're still struggling to keep up with everything that would be ours and it's something and it's something something has to the same idea 100,000 of them DNS and RBM probably not as much as we should the pooling and we'll certainly be able to see the company and I thought they had more of the rights than other guys if we talk to them properly at least once or twice a week and see how the different things really depend if we get a buzz and probably not really being a communication it's a little hard because it all is a really big project so there's a lot of different things that's a lot different we are in the last 4 months so we've been trying to engage more and it can always be a good one now we're trying to better we've done better than we probably yeah this old pool was older yeah and the whole part of the stand I could have given the talk in 5 minutes you want to do something tropical to keep going so our time is up so thank you very much for it