 Okay, so let's start another session, the title, testing and automation and cooperation by Tim Flink. Hello, I'm Tim. While probably not as scary as Lions, Tigers, Orbears, and especially not as scary as all three of those together, trying to put together test automation projects can end up being quite a bit of fun. So what I'm planning to talk about is to do a bit of an introduction, talk about where we are, where we're going, what this cooperation, collaboration, whatever you wanna call it is hoping, or what we're hoping it's gonna look like, have a conclusion and leave some time at the end for questions. So just as a quick question, how many people in here have worked with general automation systems or test automation systems? So here we've gone around looking for some test automation only to find something that you swear is almost perfect and then you start using it and you realize that there's enough differences that you basically have to rewrite the whole thing and you might as well, or well that you basically might have to rewrite the whole thing, does anyone else run into that? Have you ever been dumb enough to actually go through with it? So talking about some of that, where at least some of us have seen this before, as far as how things can be really, testing tools can be really tightly coupled with the project that they're associated with. So we are, I swear I missed a slide. No, I didn't. All right, what I'm talking about to start with is something that we're trying to start between the OpenStack folks and the Fedora folks. Eventually this may include Ansible and RHEL, but for now we're trying to keep it, well at the very least what I'm going to talk about is OpenStack and Fedora because I know more about Fedora and that's really what we're targeting for the collaboration. So before we talk about the kinds of things that we want and the type of collaboration that can happen, one question is where exactly are we? Within Fedora we have a system in place that can run jobs in response to things like builds, updates, changed updates, composes, which is on the road to some form of CI, but the gating system that we have is, I mean like I said it's easily overridden and voluntary. Basically what happens is if an update fails too many tests it's not pushed to stable automatically but someone can still push it to stable if they should so choose. So it is a very, very mild form of gating. Gating. Looking at how OpenStack does things, they are far closer to more of, I guess we could call a pure CI system than what we have in Fedora if not exactly CI. They have their system managing the patches that come in. So there are reviews and there's one system called Sule that manages the testing of those patches, the speculative merging of those patches and the eventual gating. So that if a patch comes in, if it does not pass tests it is not merged into the repositories until it starts passing tests. And while it's not quite a Linux distro, it's a complicated project and from some of the things I've seen when you can get past everything in one repository, when you start have to worry about the dependencies between NOVA and Keystone and all those different things that OpenStack has, you start approaching some of the complexities that you find in a Linux distribution. So as far as where we want to go in Fedora is we'd like to get to more of that pure, I keep saying CI. I can sit here and talk about this in abstract terms but along the lines of you can't really talk about clustering till you figure out where we are, more of where you're going is also needed. Looking to start with, there's the question of what are the inputs? When you look at a project like OpenStack or most single projects, it's relatively simple because you have pull requests, you have patches, you have some form of code coming in to usually a single point. Now if we're talking about Fedora, what is the input for whatever CI or for whatever testing system that we have? Are we talking about changes that come into, changes that come into Diskit? Are we talking about the builds that come out of Koji? Are we talking about composes that happen within Relinch? And depending on how exactly we look at it, the system itself could change. And I think that what I'm gonna be talking about today is mostly changes coming into Diskit but it's my suspicion that eventually it's going to involve a little bit of all three for the automation that we're gonna end up wanting. So in addition to the input, one of the other things that we need to have if we wanna start talking about having a CI system is gating. I mean, you could have a system that sits there and tests all day but until you actually do something with that and have it be able to stop things that shouldn't be getting in to disrupt your canonical upstream, you're not really doing the CI. And again, looking at OpenStack, they gate everything or while they start gating at least at the change level or the patch level, the things coming into the individual projects. And kind of going on similarly to the inputs, depending on what input we use, generally is gonna decide where we're going to be gating. I mean, technically we could take the changes coming into Diskit as input and then wait until we get Composys and wait until that Composys passes the battery of tests before we allow the Diskit change in. That's possible but it would be insane and I'm not proposing that at all. I think it's more of a gate changes with your input of changes, gate builds with your input of builds. If we're going to gate images, like the process of, okay, the image has been composed and it's candidate, it passes through a bunch of testing and assuming it passes all that testing, it is then released to the mirrors. Something similar with builds. There have been plenty of conversations about trying to gate builds for Rohide and the Rohide branch and the various Fedora releases where you can have the build but until it passes a certain number of tests, it doesn't become either part of the build route. It doesn't become, go from testing to stable, any of those combinations. And then getting back to the question of which are we going to use? For now, we're closest to the input of builds and gating builds. But what I'm wanting to move towards is to start doing it again at the change level so that if there are proposed changes in diskit to the spec file, to bits that are in the package, that are the sources of the package, before those actually land in the repositories, make sure that those are tested and they pass some form of sanity before they're allowed in. And there's a couple of things we need in order to actually start accomplishing this. One which has been talked about a couple of different times is what to do about diskit. In Fedora, my understanding is that we're eventually going to have pull requests for all packages so that the changes are going to, the changes that we're gating are basically pull requests on a git repository. But another place is somewhere to run all of these things. We can sit here and talk all day about, okay, well, go examine the git repository, figure out how many things need to be run, schedule all of these tasks, but we need to have a place for things to run. While I'm talking about the diskit changes as our input today, if I start talking about images, we need to make sure that for the sake of our sanity, all these different things we want to do can generally run on one system so we don't have one entire team maintaining one thing, one entire team maintaining the system that Gates builds and yet another team maintaining the thing that Gates images. So I think at some point it's going to, it's going to need to be flexible. So the short way of looking things is the goal is to test all of the things all the time, but at least we want to be testing as much as we can before it's released. It's going to be a relatively long road before we can get to any of these ideals of wanting to CI the entire distribution, trying to make sure that everything's reactive, that everything is rebuilt when the upstream components change, that all the testing is done on every rebuild before it lands in anything that we'll get to users. But just because it's a hard thing doesn't mean we shouldn't go there. So getting to the part of this about the cooperation and the collaboration. So I can start in with these reasons to actually collaborate. And I know I've heard all of these before and I'd be really surprised if everyone else hasn't heard these before, where the common thing of, well, I don't want to reinvent the wheel. We don't want to reinvent the wheel, which is definitely a goal. Sometimes it still ends up happening, but that's certainly one of the reasons to start looking to collaborate and not necessarily go off on your own. One of the other arguments I have is that most people have far more work to do than they have time for. So if you start collaborating on one, you have two groups working on one thing, hopefully there is overlap of the boring stuff. So that instead of having one person on one team maintaining a runner and one person on the other team maintaining a runner, you can put them together, have two half-time people working on the runner and then they have half their time to go work on things that are new, instead of just maintaining or working at a slower pace. In my mind, one of the other things is not quite as direct, but I look at it as a way to increase awareness and improve your documentation. I mean, you can write awesome documentation. I'm not saying that you shouldn't do that. I certainly wish that documentation for a lot of my projects was better, but that tends to be just one part of it. When you have more people using the same thing, then they write about it more and there are more examples and there are more people seeing it and there are more people writing about it. And I am of the belief that that will lead to more testing in the long run just because you can have more talking about that same system. But, as I mentioned before, who hasn't heard those reasons? I don't think this is news to anyone in here that, oh, wait, collaboration, wait, if we work together, something's easier. If that's news, okay, that's, well, you've learned something today. But getting more into why is collaboration difficult? Like what I said before, everyone has more work than time. If you start adding goals to when you need to generify things, that can add work. And if you're already on tight deadlines and don't have much in the way of resources, it's one of those, well, do I tailor this exactly for what I'm doing or do I spend a little bit more time or maybe a lot more time to make sure that other people can use it? So a lot of times the answer ends up being the former, which I'm not going to belittle anyone for. I've certainly done it myself and it's understandable. But it's one of the reasons why collaboration can fail. In my experience, test automation in particular is difficult to combine across projects. I mean, we've certainly had issues about collaboration and making sure we're using the same tools between Fedora and RHEL, and that's an upstream downstream relationship. If you start looking at Fedora and OpenStack, that's not an upstream downstream relationship. Or other things that we've looked at between Fedora and Suse or Fedora and Ubuntu, and sometimes it works out, but they tend to not be all that similar because the testing itself tends to be closely tied to a project. I mean, if you look at between Fedora and OpenSuse, we use Anaconda. I honestly don't know what they use, but it's not Anaconda. So that testing isn't really portable and that's just one specific example. Once you have these unique inputs and these different test frameworks, you, things just that tend to grow organically tend to grow around the things that are there. And the question remains that even with all of this, can we do better? But going, so we talked about some of the general reasons that things don't work. Some of the things that we're striving for, but looking at this in particular, looking at the difference between what OpenStack is doing and what Fedora is doing, our workflows are not identical. There is quite a bit of overlap and it's not something that's insurmountable, but going into this saying it's like, well, you know, our workflows aren't identical. This may or may not work. And the requirements are definitely different. You have a software project and you have a Linux distribution, they're different. So as soon as you start getting to these differences, in my mind, those are the places where you can run into obstacles that are difficult to get over if they don't become insurmountable. And I'm not worried about it in this particular place, but depending on which project you're working on and which people you're working with, there's always the politics. There are groups of people who have the Not Invented Here Syndrome. There are any of the variants of that and I won't go into that because I'm also relatively certain that people know what I'm talking about. We talk about the risks, we talk about all the things that can go wrong and all these hopes that we've all heard before and probably have seen not quite come true. So are we doomed? No. No, in this particular case, and in a lot of cases, I don't think we're doomed because from a high level and not so high that the comparison is pointless, the goals are similar. You can abstract it to the point of a complicated project with get changes coming in, get changes that need to be tested, get changes that need to be tested for mergeability, merging those tests or merging those changes into Git and gating the flow of a project from that point of view. Yes, there are differences, but if we can focus more on the similarities or while the things that are different are things that can be modularized and abstracted. One of the other things is tool overlap. Both groups are heavily into Python and Python is the primary tool for both OpenStack and for at least the development projects I've interacted with and also what we're doing in Fedora is mostly Python. One of the other things that turned out to be an interesting coincidence is we're both looking to use Ansible for the heavy lifting. The new version of Zool, and to a certain extent, actually the current version of Zool is very heavily dependent on Ansible. They have their Jenkins JobBuilder jobs that are now being translated into directly into Ansible, excuse me, playbooks. And there are a lot of the bits that are in the new Zool v3 system coming up are based on Ansible. One of the things that we're looking to do in Fedora is something similar. Right now we have our Taskatron system, which works great, and I'm not looking to scrap the whole thing, but what we wanna do is start writing things instead of our own Ansible-ish format to actually start taking playbooks as input to instead of maintaining our own runner, start using the stuff that's in Ansible. Instead of writing our own module system, start using the Ansible module system so that we get the modules that are out there for Ansible, which are far more than we're gonna be able to write. We get the documentation for that module system. So it's not, hey, it's like, oh, we want this new functionality, you should write a module for our system. We've kind of half got some docs over there, but ask us if you have any questions versus looking at Ansible's documentation and the stuff for writing modules is really well done because there's more people doing it. But with that intersection, I think that trying to work together becomes that much more likely because we're starting from similar points, we have similar enough workflows, and the things that we're trying to do have a lot of overlap. And it may not look like it, but when I first talked to some of the Zool developers about this, we were both really surprised from an architectural point of view how much similarity there was between the Taskatron system that's in Fedora and the Zool V3 system that they're writing in OpenStack. And we've both focused on different things. In particular, for Taskatron, we've focused very heavily on the local experience and making sure that if someone's writing a test and they can run that test locally on their machine with minimal effort, and by running it on your local machine, it is as identical as we can make it to the thing that's going to end up running in production. And they have some of that for Zool and they're going to have more of it going forward, but they've spent a lot more energy on throughput because they have a lot more jobs going on and making sure that the gating functionality continues to work because if the gating stuff stops working, no one can merge patches and that's probably gonna be noticed. So by looking at this cooperation as the main point is the execution flow and the actual doing of the gating, not necessarily, oh, we're using this test runner, oh, they're using that test framework, that's a software project, that's a Linux distribution because we can abstract away those things without too much trouble. We're not talking about sharing an entire test setup. We're not talking about taking Taskatron and then replacing something in open stack land. We're not talking about taking Zool and replacing all of our testing framework with it. It's focusing on the commonalities and change flow, getting the gating to work and the areas where we can help each other. So assuming that all of this is still a good idea, I think it is, whether I sold everyone else on it is a different question, but what is needed? Kind of going forward, it's just an agreement that we're all going to benefit from it. As I was mentioning before, I'm not expecting and I don't think anyone's expecting things to be dropping compatible. The times that I've gone in and kind of hoped that something that someone else made was going to drop into what I was doing hasn't generally worked out and I think that's gonna happen here. But if we came in and wanted to use Zool or parts of Zool for what we're doing, going in, having agreement that's both or that it's beneficial to both of us and we come in with work or developers to do it, I think that's a good start. Cause as I was saying before, I can sit here all I want and say, oh, I really want this thing to work and I'm gonna pretend that it's gonna do everything I want in exactly the way I want and I won't have to do any work. Good luck on that. And when you start getting into these things where you wanna keep them generic, a lot of times things stay specific is because that's easier. So by going the more generic route, that can mean more work. And while, so someone's going to have to do that work. And I also expect going forward, this is going to be a lot like I described, I generally described Fedora as a duocracy and I've heard other people describe it as well, where you can vote all you want and it may or may not make a difference. If you come in with an implementation that nobody objects to strongly, no one's gonna get in your way and it's probably going to win. So I think that's kind of how this is going to end up working as well, where it's, we could object to anything or have strong suggestions on how something works, but unless we bring the development hours and the coders to do it, probably nothing's gonna happen, but I do think that's pretty much how it should be. One of the things that I've said a couple of different times is that we're gonna need to have modules so that there's enough flexibility for both groups to, so that some of the same things can work for both groups. Some of this work has already started upstream in the Zool V3 and that's really how I see this working out. At the moment, we're really not in the same place and the last time that I was talking with the Zool developers, we've gotten to this point where we're talking about, oh, it's really kind of eerie how similar these things are and yeah, it'd be kind of cool if we could take this and you could take that, but then going through things and hitting the reality of, we're not in the same place today. This is not going to be tomorrow, we just switch over, but by trying to at least get to the same point and looking at it as more of a gradual collaboration, a gradual transition, that's where I'm hoping this will end up working out and where I think it will end up working out. So this is a bit short, but these are the first steps into a brave new world, at least as far as I think. Are we the first people to do collaboration? No, but it's for the background of where Fedor has been with automation. For various reasons, we haven't had many successful collaborations until recently, the first one I can think of that I consider really successful is the OpenQA stuff that has been running recently to do, to replace some of the manual testing of Fedora Composers. So, brave new world for us. So, this isn't, I hope I didn't disappoint anyone, this isn't, hey, we did all this collaboration, this is exactly how we did it and you can follow these steps. It's more of a, this is what we're trying to do, this is what we want to do. If there are other folks who are interested, please help out. You can, stay tuned to see how things turn out or I'd push people to actually submit patches. Patches are definitely accepted, help is definitely wanted. Any questions? The question was, if we're trying to use the Ansible module system, if we'll get to the point of being able to use Ansible Galaxy. My, and someone can correct me if I'm wrong, my understanding of Ansible Galaxy is that's for roles or can you do modules through, I think it's just for roles. I don't know if you can do modules through Ansible Galaxy or not. If you can do modules, I don't see a reason why that couldn't happen. I'd want to have a few restrictions on what runs on the test clients but I don't see why it couldn't happen. Oh, I, okay, so the question was more of, we're gonna be using roles as test cases. I don't see why not. Haven't gotten that far yet but I, I mean it's like I honestly, that's kind of, it has been in the back of my mind because it's like, you know, you have all of this user submitted stuff, why would we not want to use it? That answer your question? Yeah. Okay. Okay, the question was basically how to come contribute if people are interested. I wish I had a better answer for you. There has been some movement in our organization recently so that is a harder question to answer than I would like but we have an email list and that I should have put a link to. qadeveloplist.fadorproject.org or come find one of us in pound fedor qa on free note. Either, yeah, probably find me or Mcrizak or K. Perall just ask, start asking about automation in fedor qa and you'll eventually find one of us. So yeah, I wish I had a better answer for you. And hopefully we'll have, I mean it's like, I hate getting up here and say real soon now we're gonna have this because I've done that before and then it hasn't happened. But I have a hard time saying this not happen at least having better plans, I'd say within a week or two. Any other questions? I'm not sure I understood your question so I'm gonna try and rephrase it and you can tell me whether I got it or not. So my understanding was that you're wondering where we are in this process and where our highest priorities lie as far as whether we're to the point that we have already switched to Zool in our development environments or whether we're still in the design process. Did I get your question correct? So we're probably closer to the design process than anything. I think that again with the things that are changing right now I think we're closer than we were in a way and I think it's going to be a gradual process. I don't think this is going to be, you know, sit down for a week, design everything and make sure it's implemented. I think this is going to be a very iterative process that moves us towards that sort of collaboration. The thing we're waiting on a lot right now is there is a developer summit for the Zool folks in about three weeks and my understanding is their goal is to have a working module system for the most part working by the end of that summit. So I'm not sure if this answers your question because I am dancing around it a little bit. We're closer to the design phase and I think it's going to be an iterative process. Is that enough of an answer or am I dancing too much? The question was a clarification whether I think you're asking whether we're going to be doing much with Zool 2 versus 3. Yeah. Okay, he was asking if the focus of the center of this was on Zool and it is. I have a hard time seeing that not becoming a part of what we're doing whether in whole or in part, at least in Fedora. Any other questions? If the complete development gets blocked because they cannot push patches because the CI is broken by something and nobody is allowed to override it not even the project leader and so there's this vendor of the center fixing the CI so that they can actually do the rest of the work and they can go in and out. So the question was whether we're planning to go with a system that is completely automated and without the ability to override manually. The concern being that if there's no way to override the system that there can be a huge backlog of patches work effectively stops. Am I representing your question correctly? I'm hoping that in the future we're going to have a system that's durable enough that doesn't happen. Personally, I have been, I've spent a long time working on test automation and it fails. I mean, there's going to be failures, there's going to be things that go wrong. I don't know if there's going to be a great way of getting around it with the whole machine comes to a halt assuming we're to the point of relying on it but one part of this is to override test results if needed. So to give you a clearer answer for the short term, no, there's not going to be just automation. There will be ways to override it and I'm hoping that we, those of us working on the automation will be able to gain enough trust at a certain point that those breakages will be limited and people aren't worried so much about it. Does that answer your question? Okay. Any other questions? All right, then we all have another 20 minutes to go take a nap or something before the next session. Thank you very much.