 talk on CentOS infrastructure. Thank you. So we just heard a little bit from Fabian about the CentOS infrastructure and how the things are going for basically the things that come out of the Corsig. So CentOS Linux and the stuff that you would actually use every day, you download a CD, install CentOS, and yum update. That's all the infrastructure that Fabian was talking about. What I'm going to talk about a little bit today is some of the other things that we're doing in the infrastructure space and some of the public or some of the services that we provide to people in the CentOS community. But before I do that, it would probably be good to introduce myself. My name is Brian Stinson. I started working full-time for the CentOS project about a year ago. But I've been a long-time CentOS user as a sysadmin. You can find me at bstinsonmhk on Twitter if you want to tweet me or yell at me or whatever. Yeah, go ahead and find me there. So like I said, I want to talk about some of the resources that we provide and the things that you can do with them. So we have the stable base of all of those services that Fabian talked about this morning, the mirrors, all of the bug tracker, the wiki. All of that stuff is actually going really, really well. Some of the new things that we're working on is to provide services for people who are wanting to build things on top of CentOS. And that's where things like the account system that he mentioned. We'll talk a little bit about CBS and CI. But Fabian dove into a lot of the details of the hardware behind it. And I think that was a really good introduction into some of the things that we can actually do on top of that. So what we're going to look at today is a view into a day in the life of a contributor. What you could do on some of the infrastructure things that we provide. So we'll walk through a short demo of how to actually get things built into CBS. And more importantly, one of the real fun things that we're working on right now is our continuous integration setup. And I'll explain a little bit about how those work along the way and maybe lift up the covers a little bit and show you what we're working with. The next thing that I like to show, and that may be a little bit small print for those of you in the back, so I'd apologize for that. But this is actually a graphic that KB used in one of his talks when he was presenting some of his ideas to us for how people could contribute and where. And if you think of the whole thing kind of like a pipeline, what does someone who wants to actually get some software out there, they're working on a project, what do they want to do? And depending on which stage in the process they're in, they have a number of different steps. So they start over here on the left with their development process. They generate a set of sources, whatever that may be. And in the first case, we're gonna go through and talk about CVS. Right now we're talking about RPMs. So how do you get those sources actually from the source code built into RPMs, tested, and then released? And that all sort of feeds into that sort of nebulous user engagement because you don't build RPMs in a vacuum. You actually want people to go out and use the software that you packaged. So if you put yourself in sort of a mindset of someone who's building a software project, just to get a short poll of the room, how many of you have actually packaged something into an RPM before? Okay, great, we have a good set of packagers in here. That's awesome. So you'll kind of have an understanding of how this goes. So how many of you packagers have actually packaged something in Fedora? Okay, that's a good subset. How about for your own company or for your own use, just at home on your own machines? Okay, good. So those of you that have packaged for Fedora, you'll sort of be familiar with a lot of the tools that we use in CentOS. And that's sort of, that's a little bit deliberate. We wanna make it easy for contributors to use the tools that they're used to and to, you know, we do have our own separate ways of doing a few things, but we wanna make it as familiar to you as possible. And so to start off the whole process, one thing that we should mention is the special interest groups that we work with in the CentOS space. And special interest groups are basically just a group of people who are gathered around a common idea. So we have the cloud sig. I think right now that's RDO and a couple of other people in the cloud space. It's separate projects that are all working together. And it's nice because those guys can get together and basically solve the problems that they all have in packaging together. So RDO works on their packages. They have a problem. They can actually go back to their special interest group and they have access to a wide set of packages who are sort of dealing with the same thing. And so once you start grouping people together and getting them into meetings and stuff like that, once we provide those resources for people, we actually have to find a way to distribute that and to control access. And so that's where accounts.centos.org comes in. We just deployed this past year. And right now it's only connected to our CBS setup, but like Fabian said, we are planning on expanding this to other roles in the future. I can show you a little bit of what we have here. So let's go over to this tab. Nope, apologies. Yeah, here we go. So if you've contributed to Fedora before you're familiar with this interface, this is just a list of all the groups that we have. So we've got the atomic sig, the cloud sig, and a couple of other of those. Basically what we're using the accounts.centos.org system to do right now is just to divide up the build permissions in CBS. And we'll sort of go through that here in just a minute. But the process for that, generally what we do is the special interest groups have regular meetings, and that could be every week, every two weeks, it's all dependent on the group itself. But generally if you're interested in one of these spaces, you can find a page on the wiki with all of those meeting times and who to contact and all that stuff. It's best to go out and sort of introduce yourself, say hi, I am somebody working on this project and I'd like to join the sig. And usually what they'll say is they'll bring you to a special interest group meeting and they'll sort of talk you through the process of how it works and then once that happens you can actually get sponsored into the group and get build permissions. So let's talk a little bit about Koji because CBS is just a Koji instance. How many of you familiar with Koji, your packages? Yeah, yeah, you're all familiar with Koji, that's good. So CBS is just a Koji instance that we have that is specifically for stuff coming out of the special interest groups. And we sort of follow a little bit of a pattern because the idea is that we wanna scale this out to a number of sigs and so we kind of want a way to identify things in CBS pretty easily. So if you look at the front page of cbs.centos.org you'll find a lot of things in this scheme where we have the name of the sig, the centos version that they're building for. So five, six, seven, actually I think we only do six or seven at the moment. But then we do a project and a release. Now that may not seem a little bit opaque so maybe an example would help. So think of the storage sig. I'm a developer working on Gluster version 3.6. A lot of things in CBS would be tagged would be identified by this identifier right there. And so if you think of the targets what would that do? You'd have a target that you can send your RPMs into and then you have the release process that you can go through. So it sticks it into a build root, builds the RPM, sticks it into a place, the candidate repo where you can do interesting things with it like send it off to CI and then you can send it through the rest of the release process. And so if you're familiar with Koji that's basically what is going on right here. I just did a quick scratch build into that the storage seven tags just to show you that it works and this is building, so this would be building Python date util on CentOS 7 for the storage sig. And then that is also very, very tiny so I apologize but you can see that we have tasks that go through. This is the cvs.centos.org interface there. And you can see that it finished and you can get all the build logs and stuff. So we actually published these repositories just like any other Koji instance. You can find the repositories for the various tags. So I think this is, yeah, this is a cluster stuff coming out of the storage sig. And you can see that that's the candidate repos. Generally what we do is we recommend sort of a step-by-step process and that's what this is. So you start with the candidate tags, the stuff that you just built in cvs and then you run it through either CI or something like that and then you can promote it to do some developer testing. So you send it out to people that you know and love and love your software and are willing to sort of help you find some bugs. You can actually tag the stuff into the testing tags and you can actually talk to KB to get that stuff promoted on buildlogs.centos.org. That's a place where we publish a whole bunch of repos that and other things that people are going to be hitting quite a bit. So once that's promoted to your developers, you can promote it again to release and that actually goes out to once you're happy with it. It actually goes out and is signed and promoted and delivered through our mirror system. So your special interest group actually has a separate space under our mirrors that you can distribute all of your packages that you decide to release. Some things that we're looking at soon in CBS, we're working on some of the tools. So sent package, if you're familiar with Fed package in Fedora, this would be the analog to that. We're looking to do some things in diskit. So we have, if you go out to the web and type in git.centos.org, right now what that is is that's all of the sources that go into building Sentos 7. We're hoping on down the road to actually deliver some of the special interest group content through there. So you'll check in your spec files and all that stuff and then we can build it in Koji from there. Along with that, the look aside cache, this is where you post up all of your sources, your source tar balls so that we have a copy of it that we can use from CBS. And then we're working on better ways to do inter-sig dependencies. So let's say that the cloud sig wants to use a bunch of stuff from the storage sig. I think that's a pretty, it's pretty fair to say that that's a good dependency relationship there. We don't want the cloud sig to have to build all of those dependencies, all of those things that the storage sig is already doing. We just want them to rely on the stuff that comes out of there. And some of the things that we're gonna have to do in order to make that happen is work on a few things related to testing and stuff like that. So to do just a quick start for how to get started, we provide all the tools that you need to talk to CBS in a package called CentOS Packager. That's shipped in CentOS Extras so no need to enable any other repos or anything like that. You just yum install CentOS Packager. Or if you're on a Fedora system, you can use this coper that we've built and we plan to keep that up to date. To talk about the actual tools themselves, we have user bin CBS. Basically what that is is it's just a Koji client that uses our profile. So we set that up for you, we ship that in CentOS Packager. So you'll always be sure that you're talking to the right instance of Koji. And the other thing is that we ship is a tool called CentOS cert. When you sign up for your account in accounts.centos.org and you apply for a group or you apply for membership in a group, one thing that you'll need to do is we need a way for Koji to identify you as you. And that's what this tool does. It downloads a certificate that you present as a token up to CBS. So once you have your account, all you have to do is just type in CentOS cert and then give it your username and tell it that you want to generate a new certificate, type in your password and you can see that it drops it in your home directory right there. And then you can talk to CBS. So that's what I did right there. I just used my new certificate to contact the Koji hub. So that was sort of a lightning walkthrough of CBS itself. It's, there's quite a bit of activity going in there right now. So if you saw one of those SIGs on the page up there that you're interested in, if you're working in an upstream project that you think would fit somewhere, definitely do reach out to those SIGs and get involved there. Or come find one of us where we're always happy to talk about new groups of people who are working in that space. So next I'd like to talk a little bit about ci.centos.org. Cause this is one of the other really big sort of public infrastructure things that we're sorting to provide to people. And it's something that I've been working on during my day job for the past little bit. Our target audience, you know, if you're a special interest group, you're building packages in CBS, this is sort of something that you get basically for free. We'll give you a space in CI, you can do whatever tests it is that you like, take the packages that you build, stick them in there, do your, I don't know, unit tests, you can test the repos themselves, test the dependencies, really whatever it is that you need to do. But we're also looking at infrastructure projects that wanna verify on Centos. So if you think about what that means, one of the big projects that we're sort of happy to have in ci.centos.org is RDO. What they do is they take all of the packages that they produce out of CBS, and they actually have an entire long test suite of things that they do. They spin up and open stack cloud and do various things to verify that those packages have worked, but also that the set of packages that they have work well together. And the idea is to do this in an environment that looks more or less like what you would get if you were to hand this off to a sysadmin and say, okay, go deploy RDO. So like I said, they actually spin up an open stack cluster and run all these tests. And you might think that that's a little bit daunting or a little bit slow. But it all comes back to the way that we do things in ci. We have bare metal machines that are provisioned more or less on demand with Centos 5, 6, or 7. We have Jenkins to actually orchestrate the jobs. And finally we have a place that you can actually store off all of the stuff that is generated by your test run. So some people actually do nightly builds or something like that in their ci runs and they want a way to actually store off all of those packages, images, whatever it is. So we need a place for them to store the artifacts and their logs and stuff. Now these bare metal machines, I think right now, the cool thing is it's all behind a set of chassis. We have four of them and it's all API driven. So if we want four network interfaces and two disks or something like that, the fabric itself can support that sort of thing. In general, we provision with a single disk on there and then we have the network interfaces that are up and available for people to configure. And that's all done by a tool called Duffy. So each project is assigned its own API key and that API key is used to request those nodes in whichever configurations we support. So if you want CentOS 5, 32 bit, you just ask Duffy for it and it'll give you a machine. Duffy contextualizes that machine and basically takes your SSH key and injects it into the root account. So once this machine is up and running the call returns and says, okay, you have this machine, you actually have full root access on bare metal hardware that you can use to do your tests. And obviously you do other things afterwards. You run your tests, do any sort of verification you want. Then you actually return that node back to us and we'll reinstall it and give it away to someone else freshly installed. So this graphic sort of shows a little bit of about the architecture. If you're coming in from the internet, you can, everything is done on the project side. So if you're a project member or something like that, you can get through our jump host and onto a jink and slave. Now this part is, if you've done continuous integration before, this part is a little bit, I don't know, kind of freaky and a little bit weird to some people because it's not immediately obvious what's happening here. So you actually landed on the slave, that's slave01.ci.centos.org. You have a workspace there but that's not actually where your tests are running. We actually want you to work in a pattern where the slave, so your workspace that you land in, that's the place where you just request those nodes. You request that bare metal hardware and then remotely from your workspace there, talk to those machines and do your tests. And that can happen over SSH or whatever it is that you like. But Duffy itself is just an HTTP API. And I'll give you some links to a couple of clients and some scripts that you can use to actually make things a little bit easier on yourself when you're working with jobs. But again, we have the Jenkins master, that actually talks to your workspace there to trigger out those jobs and deploy those things. And so in the job lifecycle, basically we provision a machine for you, you do your tests, you call an API that says, okay, yep, I've done my tests, I have all of my results, I've copied off everything I need, let's send it back. And then we start the process all over again. We just set up that machine over and over and over and over. And I think Fabian mentioned in his talk last time that we're doing about a hundred deploys a day, something like that. And we're looking to get a little bit faster with that even. So there are a couple of links to some Duffy clients. If you want a simple one that you can just copy and paste into a Jenkins job, look at that top link right there. That's a pretty good quick start to get you going and started with that. And then we actually had some people in the RDO community who are using our infrastructure quite extensively and they wanted a little bit better interface for them to hook into their tools. And so they wrote a Python client library that you can use. And I do know that a couple of other projects are looking into that. But so that provides not only a Python client library but also a command line tool that you can request and replace machines in Duffy. So what's coming up next? One of the features that we're working on in ci.centos.org is to take those machines and actually group them together so that you have the same network across all the four, five, six machines that you have. Right now they all have interfaces on them into our CI network so they can get to the mirrors, they can get to the internet, all of that stuff. But some projects have asked for a separate network that's maybe a little bit isolated so they wanna test out a storage configuration. They have a separate storage network or they have a separate virtual machine network for their OpenStack deploy. That's one thing that we're looking to do to sort of tie all those things down. We're working closely with Jenkins, some people using Jenkins Job Builder because once you have a whole bunch of things to test it gets kind of tedious to go through the Jenkins web interface and create, you know, I wanna test this thing with this thing and this thing with this thing. It gets really, really, really tedious to go through and create all those jobs by hand so Jenkins Job Builder can automate a whole lot of that. We're hoping to spin up an OpenStack instance soon so we have the bare metal that is run on CentOS. You know, you get those machines there but OpenStack would allow people to, number one, consume that service and test against the cloud paradigm there but then also for them to bring their own images. So we have a couple of groups that are looking to verify on CentOS but maybe do a lot of other unit testing on some other operating systems. We wanna make it easy for them to bring along their own images. I don't particularly want to be running a whole bunch of Ubuntu stuff but they're welcome to bring their images that they have and verify those unit tests too because that provides some value to the project. Let me show off Jenkins just a little bit. If I can get over here. Here we go. Yeah, so if you go to ci.centos.org you'll actually see over on the left here we provision the workspace for every project and then you can see the whole set of jobs that we have. So this, these here are some of the core jobs, the people who put out CentOS Linux actually run through functional tests every night and then also on updates. So if there's ever an update that goes through we do some pre-update checks that all run on ci.centos.org and you can see them up there. So let's look at a quick job run here. So I mentioned that I was working on a tool called CentPackage and I want to run my unit tests. That's something I want to verify on CentOS and of course live demos. Here we go. What Jenkins does, when I actually go out and call to Duffy, get a machine and SSH into there I can, all of these logs are actually piped back into the Jenkins interface. So everything that I do is here and you can see that if we were to go back up you could see that this test passed but everything that I did during my whole entire test run is saved off in this interface and consumable by you guys. So if you're interested in CentPackage or you're interested in RDO or you're interested in CentOS 7 itself you can go back and see what tests were run, what tests were, if it's a failed run, what caused the problem, anything like that. And if you see something we're always happy to be notified about that but we get emails and pages and all that stuff. So we're sort of in the beginning process of standing up this service and working with a bunch of those infrastructure projects. So if you are one and you want to sort of partner with us on that verification, do let us know because one thing that we're working with people is one thing we've noticed is that everyone sort of has a different test setup and we're happy to help sort of accommodate that and add features as projects need. So yeah, definitely do come talk to us. Here are some links that talk a little bit about the community build system, about our special interest group process and about CI and a little bit of contact information. Usually we hang out on CentOS Devel for a mailing list or in FreeNode and there's my email address and my Twitter if you're interested in getting involved in this process. So that is all I have for you guys today. Do you have any questions for me? Yep. No, so we're doing a copper because the velocity right now is gonna be pretty quick. The plan is to get it in Fedora proper and get it reviewed once we sort of stabilize a little bit. It's, I would probably call that sort of in beta status right now in terms of the client tools, so. Yeah, I definitely will. Yep, yeah, the disk it. Yeah, so the, yeah, sure. The question is, is there going to be a policy on the disk it branches to sort of help facilitate people working on the same package? And I think that part might be a little bit up in the air. One of the things that we need to work on first is getting the integration done with the account system and the actual policies themselves might come out soon after that. But that's, I think that's something that we should maybe talk about. Okay, yeah, so the question is, is there any sort of bit for bit reproducible builds effort going on? And I don't know of one, basically the, so the distribution itself isn't actually built in CBS. That's still the same process that it's always been. We take the sources and build them. But yeah, I don't know of any sort of effort in the project that's working on that. Other questions? Yeah. Yeah, so let's actually take a look at that. And we'll hope for good conference Wi-Fi. So let's actually take a look at, yeah, we've got plenty of time. So under git.centos.org, you'll actually find this project that says sources for RPMs shipped. And if you click on that, and this is gonna take a little bit because we have a ton of RPMs in there. But this is, so this is the everything that goes into Centos 7. So if you see something that comes out through an update or something like that, that's all gonna be in here. And of course, we're waiting for, yeah, here we go. So let's look at bind. Oops. Sorry, I'm trying to read this on the, here we go. Yep, so if you come to the C7 branch, that's gonna be all of the sources that we have. And if you look, basically what we've done is taken the source RPMs and sort of exploded it into this directory. So you have the spec file under specs, and then this metadata file in there that actually pulls from our look aside that we have set up separately for the sources coming into the distribution. Does that answer your question? Okay, other questions? Yeah. Yes, they do. And that's all set up through, so initially what happens is the SIG is sort of onboarded into the build process. And we have people go through a few builds and publish those to the build logs section. That stuff on build logs is unsigned. So we sort of have people point their developers at it, people who are early adopters, but that's not really for production use. Once the SIG is ready to actually start doing the release process, the full-fledged release process, they work with one of the people in the core SIG to basically say, okay, these are the packages that we're gonna promote to mirror.centos.org, and those get signed. Yeah. Yeah, so the question is the diskit layout that we're showing here is different from the one in Fedora, and the question is, are we going to do that for the special interest groups? And really the answer to that is no. This is the layout that we're working with, and the idea is once we have people actually pushing up to these repositories, they're actually going to have a separate branch for the special interest group. So if someone wanted to come and rebuild bind in a SIG, they would have a specific branch for that, and so we need to sort of maintain this layout for that. Other questions? All right, thank you very much. I have stickers up here if you want community build service stickers. Otherwise find me around the conference. Thanks guys. Yeah, so you can come in for that. Okay. Right here, I have two. Redactions. Okay, I think so, HDMI, okay, maybe. It's the reservation for, I'm making a chart. All the way up here. Yeah, I've got a meeting. I don't know which one. I don't see it. I don't know the signal there, so I mean. So I have a second resolution. I have a second resolution. But I want to say it. Oh, wait. I have a great, great, great resolution. And I just want to give you an answer. So just in case this happens again, so. 800 by 60. 800 by 60 is not a report. There were probably another session chat introducing it, whereas in Iraqi, the first one was for you. that you have.