 Okay, I think we can probably go ahead and get started. So my name is Brian Stinson. I work on the Sentos infrastructure team. And I actually work for Red Hat. The Sentos and Fedora infrastructure teams have actually combined and we're working on the same team which provides us some interesting opportunities. And one of the things that I'm gonna talk about today is Fedora CI. So to give you a little bit of background of what I started working on before our teams merged from a company perspective, but also before we started working more closely between the Sentos and Fedora projects themselves as separate communities. I came in to work on a few different things. In the Sentos space we have special interest groups who are building packages on top of Sentos Linux. And so I actually came in to do a little bit of work on that work and also stand up an infrastructure to help us test our things over in the Sentos community. And that's kind of what allowed this project to be so interesting. The Fedora CI initiative in particular is because it was one of the ways that we were able to collaborate both on a team level, but also between two communities that are pretty closely related. Fedora and Sentos definitely have different audiences. We deliver on different life cycles, but the two communities are, we do a lot of the same activities is I guess my point. And Fedora CI was one of the projects that kind of allowed us to get there. So in terms of the sort of what is Fedora CI? Because like many terms in our industry, CI means different things to different people. And so I want to talk about this specifically as a number of different things. So what is Fedora CI? What are we talking about here? It's an objective in the Fedora Council which is just a way to help guide that. The governance in the Fedora project itself, there's a written set of things that the initiative is trying to do. And we run that through kind of a governance process to make sure that we all are on the same page about things. We do have people that work on it. This is both folks who do this as their day job, but then also folks are coming in and out of the communities to help us with some work. I had, it has infrastructure. This is the part that I most care about because that's what I do for my day job. I help manage some of the actual hardware that these things run on specifically for the Fedora project. And there's a process to go through. One of the things that the smart folks in the OSCI and upstream first teams have done is they've actually created a process for you to get your tests out into Fedora land. This is specifically focused on testing packages mostly before they get into a distribution. So that's the link to the objective right there. It's kind of a long URL and we kind of put a little bit of extra in there. Docs.FedoraProject.org is a good place to find most things related to governance and they're doing a lot of things to move documentation about the individual objectives over into one specific site so you can get everywhere. That's the objective. I know Dominic is also working on an updated proposal for the CI objective. So definitely if you're interested in this sort of thing check out that link there. And this is again kind of a weird thing that we went through in the beginning just to kind of give a little bit of history of the project. Like I said, this was one of the first collaborations between the two communities. And so this caused, I saw a little bit of confusion on the part of Fedora maintainers who were seeing topics come through on FedMessage with the Santas CI topics on them. And it's kind of fun to have that conversation over and over again. It's legitimately fun to talk about the ways that our two communities are working together. And this one in particular allows us to sort of decouple our infrastructures. So if you're familiar with how things are kind of organized in the Santas and Fedora projects most all of our build system work in Fedora is concentrated in Phoenix. There's a data center there that holds the build system and most of the operating system before it heads out to the mirrors. That stuff is hosted there. In Santas, most of our community hardware is actually in Raleigh. And so there's kind of a big gap between the two data centers in terms of geographic location, connectivity between the two locations. And so that was kind of a coordination problem on the infrastructure level that we had to deal with which I know where it's so one of the implementation details is a lot of this stuff is triggered by FedMessages coming out of the build system and that caused some interesting things that we had to do in the tooling in order to make sure that all of these processes run when they're supposed to. And I don't see Jeremy or Aurelian here but they're working on a project to make this a little bit better for us too which is pretty great. To talk a little bit about Santas CI and kind of set the background for what we were doing beforehand. We started up a CI service for projects that were really more or less separate from the distribution but they wanted to build and test and deliver their stuff on top of the distribution. And so that's the work that we were doing before we kind of started this partnership with Fedora. And we had a number of different machines that were available to us to do that. We brought some over from just machines that we had and we put it into our budgets and so we had just a little bit of extra capacity there for running test type workloads. And that put us in a great place when a couple of years ago we had the, we kind of got into a room and decided that it was a good idea to expand our test efforts specifically to the package ecosystem. And that required us to think a little bit differently about some of the tooling that we use because in the Santas CI infrastructure beforehand we basically coordinated with individual projects one on one with what is your test suite look like and we'll just give you a job here that you can do all of your magic things and in your own specific way you can have a place there, that sort of thing. But we found that we needed to kind of consolidate the process a little bit just because that makes things easier. And we really needed to create kind of a CI community out of this whole thing. And it's kind of a loose community to be honest. There's not a single group of people who are focused on delivering a vision for specifically CI but there are a bunch of us that are working in the same space and consolidating on infrastructure and working on the same tooling is helpful there. And so for the Fedora CI initiative I made this slide back when I gave a similar talk at Flock and I tried to get half of the teams involved and try and fit them on a slide this is not half of the teams that are involved doing something related to CI but each individual team has a different piece of what's going on and I don't expect you to read that in the back but so we have the Fedora CI, the OSCI team that works on some stuff over here you've got me up there in the CentOS infrastructure team the Fedora infrastructure team cares about build system work and then there's folks elsewhere in the CentOS infrastructure team who care about the packages coming out of CentOS and it's just a lot of folks that are they're doing similar things but they want different things out of the process. And so we kind of had to we had to think about our infrastructure in a way that allowed people to get things done so we started deploying OpenShift a while ago specifically to support the Fedora CI initiative but then we realized that a lot of this tooling is helpful for you remember that old stuff we were doing in CentOS CI with the individual projects and a lot of different styles of testing we found that doing the work to support Fedora CI deployed in OpenShift is actually a great way to start reaching some of those audiences as well and just to give some some stats for the nerd people like me apps.ci.centos.org we started with Origin 3.5 we have upgraded that a few times and we have plans for upgrades later we started with 19 bare metal nodes I think we're up to somewhere around 30 nodes that are involved there and Jenkins so Jenkins was something that we had a little bit of expertise in beforehand and that allowed us to kind of scale out some of those operations by deploying the separate masters that you need for all the individual projects that we support including Fedora CI in OpenShift itself and that's what it looks like that's the this is the CI pipeline project so if you submit a package build to Fedora there's a job somewhere in here that is actually triggered and runs the tests on your packages and there's logs in there I'll talk about that in just a minute the CI pipeline objects and libraries were that was another team who worked on actually standing this up so that you can interface with the standard test interface which I'll talk about in a minute but all of this is some of that glue work that we needed to put in between triggering based off of a message and then running the standard test interface based on what the packager tells you the test should be and Ansible the standard test interfaces is all written in Ansible you can you can go find that out on packager it's it's a pretty great way to support a number of different operations that you might want to do inside your tests one example is we're finding out in the regular CentOS CI place I talked about all of those those different projects you might have a a separate test harness you might have a separate way to run the test harness you might have a few things that you need to install in the system beforehand the standard test interface and some of the roles that have already been implemented are great ways to to actually help you tell the CI system what it needs to do because it's my job as an infrastructure guy to give you some hardware that you can run a workload on it's the job of the CI pipeline to interface with the hardware and the software to kind of provision an environment for you to stage the thing in that you're going to test and then also to run the tests themselves and that coupling between the CI pipeline objects and the standard test interfaces the really key relationship that that kind of makes this happen uh... so the process of doing that it's pretty simple this is kind of an older version but um... yeah it's it's a tiny little bit of ansible that um... that you put in your uh... yaml file that is committed to this this git and then the pipelines uh... actually run you can run these on your own machine um... this'll uh... you know basically install that package for you and then uh... run that little check script there um... but there's no real magic here is what i'm saying there's just a collection of tools that we've we've put together to take the pieces that you have and turn it into an automatic process and this is what the pipeline looks like um... this one is for vim uh... so there's a bunch of different stages that happened here um... the pipeline itself actually uh... kind of composes a uh... uh... uh... a test machine for you installs a package on it and then runs that test uh... thing that you defined there um... to talk a little bit about the uh... some of the mechanics so there's a number of different test context that you can you can run in one of the most interesting problems that we had uh... was it's easy to run the container based workloads and open shift sometimes you really want an actual machine that kind of uh... that behaves like a machine uh... something that you installed on a cloud provider something like that and so the uh... there's one particular part of the pipeline that actually composes a a cloud image and then boots it in inside of open shift to uh... to run those tests for you and that's really helpful because you can classify your tests in certain ways um... and that's all again that's all part of the the standard test uh... interface there what's there we go that's better so what's coming next there's some uh... i think we're in pretty good shape uh... with the infrastructure that we have right now uh... we've we've had a couple you know call it a couple of years of of running these things through the pipeline and i think the the infrastructure and the process on the operation side is actually pretty okay right now we've got some some changes to make if we can uh... deliver some messages a little bit more uh... in in a more uh... easy way that we can make sure that they actually happen but uh... in terms of the operations process it's going to be uh... you know mostly hardware expansion uh... upgrades in testing that uh... that new stuff that comes in via open shift and Jenkins and and things like that adding features to them uh... to the c i pipeline objects themselves uh... but uh... and you know yet took a look yesterday we we ran a hundred nine bills on this is either uh... uh... you know real koji bills or uh... things that came in via pull requests to the disk it branches in fedora we were about a hundred nine of those three c i yesterday uh... and you know for the most part the the processes is pretty well set for on the operation side uh... the ui an onboarding uh... portion is uh... i know there's a uh... a little bit of work left to do but i would encourage you if you uh... are interested in this sort of thing uh... i had a link in here somewhere maybe i didn't there's a uh... but there's a there was a talk here uh... i think it was last night uh... it was a very good start uh... sort of a workshop to actually play with some of the tests and they provide you some uh... some images there that are you can kind of see what someone else has done and you know translate that over into uh... into your tests if you have a package in fedora uh... so that one is really great and there's there was also another talk uh... this week about the uh... i think it's titled something related to the contra in set up which is uh... that's the software that we use to realize the c i pipeline in our open-shift instance and for uh... one thing that that is really uh... kinda nice for uh... for this year going forward uh... as we do have uh... a central place to talk about c i issues uh... you know uh... i'm monitoring this is a uh... you know one of the infrastructure guys who is is paying attention to things uh... i know the folks that are uh... on the c i team are looking at this the folks that are working on the pipeline software itself for looking at this so this is kind of a good clearing house to start with if you and don't necessarily know where you're if you're running into trouble you don't necessarily know uh... with what where the problem is uh... definitely start here because we're we're all monitoring this that's uh... that's a pretty good overview of fedora c i as you know just kind of a high-level and infrastructure related overview i do want to take a few minutes looks like we have a couple of minutes for questions if you have them uh... but that's all i've got for you yeah anyway to trigger test runs uh... without running a coach you build the answer yes of the the question was can you uh... can you trigger a test run without uh... running a coach you build and the answer is yes uh... if you add that the test specification to a poor quest uh... it if it's yeah so if if you already have uh... uh... the test specification committed to your repository it any action that you do you know you you do a poor quest to us a spec file or you commit to the uh... uh... the interface there we actually do run one of the pipelines that does a scratch billed for you and will run tests it's no the pipeline runs a scratch billed for you uh... any yeah i'm not i don't is so if you want to rerun the tests there's a keyword that you can you can use if you've already run the tests on it um... sure what did you have yeah i don't the so either the uh... you can either trigger rerun of the pipeline that that is already existed or you do need to do uh... uh... billed or something like that but uh... it's is it filed in this uh... in this tracker i would i would file an issue in this tracker right here and then uh... that's probably the best place to to figure out the mechanics of that yeah i was just going to mention rerunning tests like that is kind of a generic problem we have so i didn't add on my side of the open curate which those tests and we have the same kind of things sometimes people are like hey this test in open curate fail or i just want to rerun the test of something can i say send out a request somehow to have the test run again in the end it's kind of a no it's kind of the same with taskatron so it's like a generic thing that we we talked about doing a generic solution for but it didn't get very far but i would i would definitely file an issue here and then um... we should figure out the uh... because there's a number of people that would be involved in and doing that but yes uh... so the the question was is there a plan to provide some of this goodness into uh... you know upstream projects in pager dot i o for example and uh... yes very much so uh... you so i mentioned that uh... you know kind of the partnership relationship between sent us the eyes and infrastructure provider for this individual pipeline uh... we do want to be the infrastructure provider for other pipelines as well and so you know putting your tests in the standard test interface makes it really easy for us to stand up a pipeline for you and say just go do it uh... so it's not it it's not necessarily package specific but uh... the other there are things we can do that in that space and sent us the eyes specifically is uh... we're looking to be uh... that sort of provider what else do we have thank you all for coming out is good to see you