 I'm going to do my best to talk loud enough so the laptop can pick me up. My name is Tim, and I work for Red Hat as part of Fedor QA. And one of the things I spend pretty much all my time on is automation. And what I'm planning to talk about today is continuous integration and the glorious future. Generally, to where we're going and how we're going to end up in this glorious future. So what I'm planning to talk about is a bit about defining things. What exactly are we talking about? And then start asking questions that may sound a bit silly. Where are we? Where are we going? How are we getting there? And then come to a conclusion and leave some time at the end for questions. So what exactly are we talking about? So continuous integration, as a term, first showed up in the 90s. And it was, when it first showed up, it was a proposed solution to the problem that they were hoping to solve and have solved in some ways of software projects having development work, development work, development work, and then do all the integration at the end. And for some reason they would hit problems because that was the first time they tried to do integration. So continuous integration was this idea of trying to put everything together at regular points along the way so that you don't end up with those nasty surprises at the end. But it was originally meant to run maybe once a day at most. It wasn't until the late 90s, early 2000s, with extreme programming that continuous integration became more of what we think of today, where it's a suite of automated tests that run at every commit, at every change, on every pull request, running tens of times a day, up to hundreds of times a day, depending on your exact implementation. So from what Wikipedia lists as continuous integration is that there are, it's based on a lot of different principles. Some of these, and these were all meant for a software project, and some of them make sense in the context of what we're trying to do in Fedora as a distribution. But there are some differences, which brings me to the question of how is a Linux distribution different from the types of software projects that continuous integration was designed for? The biggest thing I come down to is control. We as downstream have, depending on the project, different levels of control, but very little control over what upstreams do and when they do them. They can do new releases, certain upstreams can release bug fixes, or they can just do huge new releases and we have to go through, as a maintainer, you might have to go through and pick out the thing that you want to backport to another release. And even beyond what upstream does, and especially in a community distribution, for better or for worse, one of the strengths and perhaps one of the weaknesses is that things happen. No single person has control over when this is going to change, when this is going to change, when this is going to change. Sometimes we try around release time to keep things stable, but those kind of things can't even change within the project. And when Linux distribution for first started, they were meant to be a point of integration, a relatively stable known target amongst the constantly changing set of things that are upstream. And sometimes we need to make small modifications to those upstream sources, either in patches or timing to provide that measure of stability. But depending on the distro, obviously Fedor is not as stable as Centus or REL. The distribution itself can choose how stable it wants to be. So now that I've talked about continuous integration, I want to set a few ground rules for the rest of the talk. What I'm interested in doing is getting into the what and the how. I think that the folks on the ground are generally on the same page, but I've been in plenty of conversations where things get sidetracked into using this buzzword. Or common one is asking me how Taskatron compares to Jenkins, which I understand where the question is coming from, but it's a difficult comparison. It's the wrong question to be asked in my opinion. So I want to get more into what we're doing and how we're going to get there. So I'm going to avoid tool names whenever possible. Obviously, I work on Taskatron. That's my thing. So I have my ideas on what tool would be best to solve this, but my interest is primarily in making Fedor better. And if that ends up not being Taskatron, I will be sad and I don't think that's going to happen. But the primary thing is to make Fedor better. And as much as this may seem like a bait and switch, this is going to be the last time in this talk I use the phrase continuous integration. It is a buzzword. It has baggage. It's connected to it. Everyone comes in thinking this is continuous integration. This is continuous integration. And it seems to get in the way of what are we trying to do? Whether that matches up with what people think of as continuous integration or not is a separate question that I would like to leave for elsewhere. So that being said, having left out the title of this talk as a term within it, where exactly are we? There are a lot of things that we are already doing. Pretty much everything is in Git, spec files, just about everything that can be in Git is in Git. So we have the repositories taken care of. From a package point of view, those are all automated. And when I say that, I mean, you can just tell Koji to go build something and it will go take the source. It will produce an RPM and it's pretty much without human interaction. So it keeps on every check in? I mean, the build, will it keep whatever level of change something commits? No, not at this point. That's still manual. I was thinking more of once you initiate the build. It's not like you have to run through and then compile all the stuff into object files and then run a make file by hand. That's all done in the spec file. But if there's more automation that could be done. Then we have things like Koji, which I hope I pronounced right, that goes into Koji and does Koji scratch builds. Whenever dependencies change. There's things like the new hotness that attempt to build a new version of a package without changing this. Doing nothing but changing the version in the spec file and grabbing the new tar wall. And it's just kind of a question of, okay, if we just change the source, is this package even going to build as an early warning sign? There are things that can be set in Bode for test requirements. And we have package sanity tests that are run on both package builds and updates when those updates change. So even then you can ask the question, I just list off a whole bunch of tools. And how are we doing? In my opinion, I think we're doing pretty good. Are we doing perfect? No. But I do think that definitely making progress and things work. So getting back to the points that Wikipedia listed, the principles upon which continues, I almost said it. The principles upon which the C word shall be based. So code repository. And as I mentioned, that's basically done already. Everyone committing to baseline every day. This is one of those that I don't think is quite as relevant to a Linux distribution. By definition, we're working with releases. The point of that was so you don't have people working on features, working on features, now all of a sudden they have 5,000 lines of code and they're just going to dump in their repository and go home for the weekend. That's kind of how the distribution works in a way. And I don't think there's really, we can get around that. And I do think that raw hide is about as close as we're going to get. And as close as we need to get in this context. Testing in a clone of the production environment. The tests are run in a recent-ish compose. Assuming there are no failures, there is a new image that's built every couple of days. Every album? A couple of days. That is used for testing. It's a custom build. There are certain things that we need to have in there. Certain packages that are used for every single run that we want baked into the image. So it's not doing a DNF install every single time. But it is using the same tools that Rallenge uses. So it is a reasonable approximation of what production would be. So now we've talked about things that are there. Things that are in process. Automating the build. There is still work that can be done here. The best code you can take, you know, command and build, you know, SRPMs and RPMs from in a pretty much automated way. But RPMs aren't the only thing we produce. Automatic composes. Composes that don't require someone sitting at a terminal, a certain single person sitting in front of a terminal, to make sure that everything works is underway. That works in progress. It will be done soon, so we don't have as many single points of failure. Making the build self-testing. The code is done, but we do have a feature that will be ready soon. It will be in production before Fedora 25 is released. So that tests can be run on every build. Sorry. Sorry to question. Let me speak up for the mic. So you mentioned that automatic signing is underway. Do you know who's working on that? Because my current understanding is that that's not assigned anywhere, but it is, in fact, one of the things that blocks some of our efforts toward the C word. I thought it was in progress, but if you're under different information, I suspect yours is more accurate than mine. So I will have to change that as to things that will need to be done. I thought it was in progress. So it sounds like I was mistaken. Any other questions? Well, on the testing side, are we testing in the ultimate way? The user interface or the RPM functionality? I'm just interested how the test cases are built. Because it secures the spectrum of what you have to test to release a stable environment. I'm going to answer your question in two different ways. The first way is what we're going to be putting into production before Fedora 25 is more focused on per package. So things that don't necessarily need a GUI, things that don't necessarily need more than just a couple of packages. Long term, there are things that we have in place already like OpenQA that is designed to test at least installation. And there are talks of using other things. One of the philosophies that we have in Fedora QA is that we aren't dumb enough to think that we know best how to test everything. And we view our jobs as facilitation. It's getting the people who actually know what they're doing, letting them do what they do best. And we're just trying to make them as efficient as they can be. So I'm trying not to stumble into, I said I wasn't going to use tool names. So this is tripping me up a little bit from getting into specifics. But we want to go other places but we're trying to take it one step at a time if that answers the question. Sure, I'm just trying to use kind of a methodology within my world. I can say that I came to hear that I actually work on the same thing, trying to do the test. The tool that doesn't really matter is more. So that was a yes to question that's answered for now? Sorry, I was just making sure. So will the environment test, for example, a system that is starting and selling and working in progress? It should if it doesn't, that's something that needs to be fixed. It will all be in a virtual machine. So things that required our metal are not going to work. I suspect they're, as part of that being in production before Fedora 25 is released, the code's done. Right now we're primarily trying to find guinea pigs so that we can find the issues of, well we thought that was going to work and no it doesn't. So it should. But it still needs to be tested. And if you're willing to be a guinea pig, we're looking for guinea pigs. Answer your question? Any others? Okay. Other things that are in progress? Automatic deployment. And I'm finding a little fat. It's in progress. Depending on which part, how exactly you define automated deployment will depend on what's actively in progress. But in my mind, it's being worked on. There is a goal. It is going to happen. And it is not work. It is not done at the second. But that. I'm sorry. What do you mean by automated deployment? I'm purposely playing a little fast. Basically what I'm interested in from this perspective is that there's no human. Basically it's going to, once criteria are met, it goes into updates testing. Once certain other criteria are met, that's going to happen automatically. And there's a little bit more. Like I said, I'm being a bit fuzzy with it because it can mean a bunch of different things. But that's the primary thing that I'm interested in. There's some longer term things that would be cool. Like one package changes. We automatically rebuild the install ISO if it's part of the payload that is the install ISO. But I think that's further along than we currently are. But the QA and relinch and infrared apps and all those different groups are working together. So when those things are done, they will work together. And I keep hitting the wrong button. All right, so now we've gone from good to, well, we're working on it to, well, yeah. Things that need work. So everyone can see the status of the latest build. The information is there. You have to know where to look. And even if you know where to look, it's a gigantic pain in the butt. So this, yes, technically the data is there, but in my mind until it is easily digestible by someone who's not intimately familiar with where the results are and how they're stored. This is something that needs work. And we do have plans for it, but it hasn't quite started yet. In my mind, the deliverables are in a similar place. They're better than the information because if you know where to look, they're easy to find, they're easy to get. But you have to know where to look. And I think that's another thing that could be improved. And the other one is keeping the build fast. And I'm going to put some qualifiers on this because when you're building packages like OpenOffice, there's a limit to how fast we can make that go. The thing that I'm more interested in is the time to feedback. So say there's a new Anaconda build pushed. And how long does it take to make a compose after that? And how long after that compose has it run through a set of automated tests? And how long after those automated tests go, if there's a bug, are the developers notified? How do we make that process faster? How do we shorten that feedback loop? It is in this context, I think, is more important because unless someone knows something I don't, OpenOffice is going to compile as fast as OpenOffice compiles. So talk a bit about definitions, where we are. Now it's time to talk about where we're going. One of the things I want to emphasize is there is a V in this. The stuff that I'm talking about with the C router, whatever we want to use to describe it, this is not something that just QA can do. This is not something that just Relinch can do. This is not something that just engineering can do. If we're not all working, even just within those groups, without those groups working together, without working with the community, working with all the contributors, this isn't going to work. So I just wanted to emphasize the we. So what is it exactly that we're trying to do? The most distinct way I've been able to put it is reduce the cost of change without compromising on quality. So one of the things I mentioned before, we're shortening the feedback loop for the things that we have control over so that when there's an anaconda change, it's not like a week or two or some of the things in the past, a month or two, before all of a sudden this bug is found and we should have fixed this a month ago and it's that kind of thing. Avoiding these things and trying to get that feedback loop as short as possible so that we can just keep things moving. Another part of this is removing the requirements for human interaction. You then have single points of failure. You can run into issues where, you know, if someone falls asleep, humans generally, most humans have to sleep. And so if you have processes that are waiting on one person to do one thing, you end up with single points of failure and it becomes difficult to keep everything moving and to keep delivering as quickly as we want to. So getting rid of that, unless it's absolutely necessary. And in my mind, one part of this is to let contributors write automated tests. One of the things I'd mentioned before was we view our role in QA and especially the folks who are working on automation as we're enablers. There are like five of us about, seven on a good day. Let's be generous. There are ten of us, which is an overestimate. The ten of us cannot write all the automated tests for all of Fedora. And if it isn't something that's easy to do, contributors aren't going to do it. So contributors need to be able to do it and it needs to be easy and it needs to be as painless as possible. Otherwise, it's plain old not going to happen. So who is defining the processes to allow the developers to do it easily? And for example, okay, the back is fixed. I need a regression test that will be automatically tested, a harder set of guidelines and processes set around that to enable developers to do that easily. Kind of, but I think in the context of Fedora, that can't happen. Who's going to enforce it? Well, that's the question, it should be a... Should it? It's actually an open source for us. I don't see a reason to do that. You're talking about DevOps. If you're a guy that is developing the tools to allow the continuous integration, which is the process and the tools are just to follow the process, then the guidelines should be set by those guys, right? And making others to perform? I see where you're going. I think that in practice, we don't... No one has that kind of power. I think that, again, for better or for worse, Fedora engineering on the engineering side is a lot like herding cats. That those cats are going to go where those cats are going to go and there's a limit to how much you can force them to go into certain places. Maybe someday we can get to that point. But I think it is going to take a long time of steady change if we ever got to that point. So it's not really much in our roadmap. Am I making sense? Yeah. Okay. Did I answer your question? Okay. Just a quick question. Is there any way we can construct it? You talked about writing the rules, making test cases. How do we get them forward? You're getting ahead of me. This is a later slide. Go to his workshop, which is about writing tasks for our automation system. That is going to go into what is available, how to start doing those things, and we'll hopefully start answering some of your questions. And that workshop is Friday at 3.30 or 1.30. Is it when your workshop is? Is it 1.30 or 3.30 on Friday? 1.30. It's on Friday. And it has a tool name that I am not going to... Because I said no tool names. Any other questions? Yeah, 1.30. Okay. And in the end, something that I had mentioned before. I want to see a better fedora. I want to see a faster fedora. And since I don't like buzzwords and I don't like things that can be confused, maybe we can just call it nonstop fedora. And if that catches on, that would be great. If not, then, at least within this talk, you guys will know what I'm saying. I was wondering if that was going to be an issue. It's like, is this close enough? No one's used that for a long time. So whether we call it that outside of this room or not is a little interesting. I'm interested in the result. And I'm repeating myself and I'm purposely repeating myself. So this whole idea of a nonstop fedora shortened feedback loops and just get rid of the required human interaction where it is at all possible. The things that come into mind are composers, auto-signing are the two big ones. So now I've talked about where we are and all of these things we want to do, but there's the more important question of how exactly are we going to get there. So a few concrete things. We will be covered in his workshop later today and we'll be expanded as we go along. We are getting the systems into place and the code is done. We just want to make sure everything is working well enough before we open the floodgates and tell everyone to use it. So to sort of take things a little bit slowly to make sure everything's working. And these tests can be run on any fed message. So even a test result fed message can trigger another test. And getting too complicated, I'm not sure we're ready for, but the basics are there. Automating relinch tasks to the point where they can happen quickly at any time without human intervention. This is not something that's going to happen today. It's gotten a lot better, even just from Fedora 23 to Fedora 24. I think there have been great improvements in this, particularly in the compose area. And there's work ongoing. It's not going to happen overnight, but progress is being made, and I am not the best person to ask about that. Asking one of the folks that is actually working on the relinch automation is a better person to ask. One of the things that is also in progress is migrating disk it to Pajur. The thought here is that once that migration is there, there are pull requests. Once we have pull requests, we can make contributing tests that much easier. So if you're a user of a package and the packager doesn't have the time to write tests for that package, you can, without being a packager, write the tests, do the pull request, and it can end up being used without having to go through too much trouble. And another thing is that we need to do is formalizing criteria so things can go from one to the next. It's all well and great to have all this automation, but unless we sit down and as a project say, in order to move from, say, updates testing to updates, these are the things that need to happen. Even if it's, you know, for this subset, for Python packages, these are the things that have to happen before your update is allowed to go into stable. For known packages, these are the things it has. However, we want to slice that up to actually sit down, talk about it, and formalize it so that it's not a surprise. Again, progress and plans. By the end of Fedora for 25, contributors will be able to write checks and tests for specific packages just by having those checks written in a specific place in the disk and repositories. By Fedora 20s. It's like everyone will be able to add their own private checks for a particular package. For the package, and all they have to do is add the files, and it will be automatically picked up and it will be run on KojiBuild. So that's the emphasis being on there's, all you have to do is put the files there. In that case, there's no registration. There's no nothing. It's just as long as the files are in the place of convention in the disk repository, it will be picked up and run. I don't know, Paul, do you know what the timeline for moving disk it to Pajeras? I think it's in, as far as I know, it's in staging right now. I'm guessing, and I emphasize that I am guessing, I am not the right person to answer this. I would guess definitely by the end of the year, but yeah, Pierre is the right person to ask that question. Any other questions on that? The other is that by Fedora 26, our automation will support jobs triggered from Pajera.io. Where you have the stuff in disk it, well, that's all well and good. But if the upstream project has to write something for a completely different system, why do you have to write the same thing twice? This is stupid. We're looking to expand this later so it's successful, but this will work more like for a single project, what the C word I'm avoiding, people usually think of. So you do a pull request and it will automatically run a certain set of jobs. You change git, it will run a certain set of jobs and report to a known place. And that will be functioned very similar to how Travis CI, or at least from a user interaction point of view. Stuff in relinch, and I'm hand waving here because I'm, yeah? It hadn't occurred to me. I don't see how that couldn't work, but it's not something that we had thought of. I'd be interested in hearing the use cases you have in mind. Any other questions? And, okay, so then plan, and I'm hand waving here because I'm not relinch and they would be better people to ask. And then factor 2.0 is a lot of what the user interaction point of view is. And, okay, so then plan, and I'm hand waving here because I'm not relinch and they would be better people to ask. And then factor 2.0 is a lot of what the, is coming in relinch automation. Also talk on that. Okay, good. Okay, Ralph would be the person to talk to about factor 2.0. And then within apps and engineering, make Bodhi testing requirements more flexible. And I struggled with a way to phrase this because what I'm really getting at is Bodhi has a concept of these things need to happen. Like, for example, right now, depth check doesn't pass. Then the update is not automatically promoted to stable. Someone has to manually request the portion order for it to happen. You know, starting to step that up to start having things for, we recently had a check proposed for all Python packages. So having ways to, instead of having those two checks by themselves, to start building upon that so that it's not quite as simplistic. And there's also some talk about, and this will need a larger discussion about whether we want to have things that can be pushed to stable, assuming they pass a certain battery of automated tests on potentially skipping karma for a specific number of things. But I emphasize that is it, before the flames start coming, that is a discussion that will need to happen. It just because it's been talked about doesn't mean it's actually going to happen. Just a few examples. We're already checking for potentially problematic ABI changes thanks to the lib Abigail folks and their work. This is not something that we wrote. This is something that they wrote. And we are very grateful. This is one of the things that they will be covering in their workshop, which leads me to my next thing. If you want to learn how to write your own checks, show up at their workshop on Friday at 1.30. And basically come to the end, it's like, I want to see this become a reality, whether we get copyright problems, I don't care what we call it. Let's make nonstop fedora a thing. We have the skills. We either have the tools or we're making the tools. And a lot has happened and a lot of progress is being made, a lot of exciting things are going on, but there's still quite a bit work to do. So at the end, let's just make the store more awesome. Any questions? So I think it was still happened that something will be catch for pushing to stable for example. And then we can find that there is some problem, for example, by ABRZ reports. There are many reports for some version. We know this version is wrong. So we're about some magic, some process for removing the package from the stable right away when there is some big amount of, for example, things. I can see how that would, I mean I guess one thing I would say is that as much as I've been working in automation for almost, software test automation for the 10 years now, automation will never catch everything. Testing will never practically catch everything. So yeah, that's always going to happen. As far as the rest of it, I can see how that might happen, but we've got a long ways to go before we get there. I think that as far as the most valuable, in my mind the most valuable things to work on at the moment are still just making the feedback loops shorter and getting the required human interaction out. And once we get to that point, then we can start talking about integration with AVRT or doing those kind of automations. So possible, but I think we're a long ways off. Answer your question? It's a multi-part question. So in general, when you have some software being written, you would have, or in any case become a package in Fedora. The developers of that are going to have their own tests that they have for their own package. Is that in any way included in this testing? Or where do you guys start picking up the test that the package runs, that the package integrates both in the Fedora system, or is there something from the actual, that particular piece of software that they are own tests that are included in your tests? And this is going to sound like I'm dodging the question, but we are purposely not deciding that. Because I don't think that's a line that can be decided globally. I think that it needs to be up to the folks who know the software better than we do. I think that in a lot of cases those unit tests are going to be run during build. I don't really see a point in duplicating what the check phase does in an RPM build in a separate system. I don't think there's a whole lot of value in that, but I recall, I've heard stories of test suites that come with a package that take three hours to run. And it's like you don't really want to put those into check, but they need to be run. So right now they're in check because they don't really have a better place for them. So my answer is it really depends on the package, and that's something we're leaving up to the people who know the software. Because I think it is a per package decision. So in general, for the majority of cases, you can consider the test that you guys are doing to give more or less the integration of where the units are in the package? Yeah, sorry, I thought about that and the words did not come out. I think our emphasis is more on integration. But again, we are providing the tools, and it's kind of one of those where we're kind of waiting to see where people take it. Because there are an overestimation of 10 of us. But the emphasis is integration because I think that's where we're going to get the most value. Until we get into some of the upstream stuff, for the stuff that is Fedora specific, I'm not sure there's a lot of point in doing anything much else than integration or specific package testing. I think those are the two cases this will help with. It seems to make a lot of sense to me as well because in this scope of the actual package, their own unit integration test would mostly cover that unit as a whole package itself. And therefore, to me it makes sense to test the integration of the entire system together. The system itself was designed to not just run tests. The whole point of the system was to, I don't know, is it still a talk? I start using tool names now. The whole point of it was that it's silly for, I mean every part of Fedora is strapped for resources, mostly the human kind. It's silly for multiple groups, every group to be going around writing their own automation. Let's just have something that's as generic as we can make it while still keeping our sanity and still being useful to just basically run the thing on FedMessage. In this system of running global tests, because yesterday on the packaging Python, there was one back where some application was packaged locally and the name space was completely off. So this system is focused on testing particular packages and the integration of one package with Fedora. But on a global scale, thank you, I have the system to... Yeah, that is not where we are right now. We're just trying to take this one step at a time and to go first with the package specific stuff. The system itself, I mean basically you give it, these are the packages I want installed. This is the version I want to start with. Here are some commands go do. And here are my results, go record that. That's essentially what it does. So that could be done now. It's not what we're emphasizing right now. If you have ideas, please come to us. But right now we're pushing hardest on the per package stuff. And once that's done then we can start looking on what's next as far as the global or certain area things that need to be run. Does that answer your question? Any other questions? Alright, well thank you very much.