 Hi guys, welcome to Hacking Koji. So my name is Mike and the reason I'm giving this talk is because I want people to hack on Koji. I want people to do fun things with it. I want people to make it do things that I haven't thought of. So I'm going to talk about a couple different things that you might do that might be considered hacking Koji. There's a number of different places and ways you might do that. A little about me. I've been working at Red Hat since 2001. I started out doing QA for Anaconda and then became a release engineer foolishly and then rewrote the build system foolishly, I guess. And that eventually became Koji. On FAS I'm Mike M, Red Hat I'm Mike M, so there I am. Koji, raise your hand if you've used Koji. Which is everybody in the room, I figured. So you know what Koji is. You may or may not know the other people besides Fedora use Koji. Red Hat uses Koji. Red Hat used Koji before it was Koji. And Sentos uses Koji now not to build the core Sentos yet, but to build the community builds that go on top of Sentos. Linux uses it, Amazon uses it. There's actually a long list, but I didn't want to get too busy with this slide. And I don't even really know how people might be using it, not telling me. So why would you want to hack Koji? And I can think of a lot of reasons and you can probably think of more. Generally it rolls down to you want Koji to do something that it doesn't already do or at least doesn't already do easily. So that might mean you write some custom scripts to run on your client. I do this all the time. I have an entire directory full of one-off scripts that I wrote to make Koji do this one weird thing this one time. And that's that. So maybe you would like to do that too on occasion. So we'll talk about how you might do that. Sometimes you want to do exactly that. Not so much to make Koji do something, but to get Koji to tell you something to figure out what's in the data. There's a lot of data in Koji. It's been running builds for almost ten years now, about nine and change in Fedora. And so, I mean, you could get into the API and really pull some interesting data out of there that the web interface is not going to speak to you for you. But in order to do that, you probably have to dive into the API or at least learn some of the more esoteric CLI commands. Interfacing with other tools and other infrastructure, Fedora has done a lot of this. You guys, we have it talking to the Fed message. We have it talking to other Fedora tools, very important stuff. So some of you may well need to hack on Koji to make it work with the next bit of Fedora infrastructure. Or you may just want to make Koji do something a little differently than it already does. Make it look differently. They have web theming and or add a new feature. So it's a lot of different ways you could approach this. So before we get into what you would do, let's try to look at a big picture of how Koji is structured. I made this diagram a long time ago and then I realized that I don't think it ever, I couldn't find it. Have you guys seen this diagram before? Right. I cannot, I don't understand how it did not propagate with the docs. But at some point it was in a document and I think it just fell out. So I remade it last night, which meant I wasted an hour in an escape. But the key take on here with Koji is that pretty much everything happens in the hub. The hub is the thing in the middle that for the most part governs access to all the data. You don't talk to the database, you don't talk to the file system for the most part. You talk to the hub and it does that story. Now in all the pieces of Koji do that, WebUI is just an application that uses the hub to do things. Command line interface, it just talks to the hub. The builders, they just talk to the hub. A couple of exceptions to blue arrows is a little cheating. A little bit of cheating. That's why the blue arrows are there with the builders and with Kojira. Blue arrows, they have to talk to the file system in order to run create repo. I hadn't found a really good way to have that work without having it mounted read only. And Kojira, I can probably fix that at some point. But right now Kojira actually does the repo deletions itself. But that's about it. That's a little blue arrow is cheating. So most of the places you would hack Koji would be writing your own thing to talk about how to make it do things. Maybe writing some plugins to the hub or the builders. Or maybe writing a new tool that fits in this diagram. I want to talk about a few odd things about the command line. A couple of these are definitely hacking on Koji territory. And one of them is maybe not, but it's a really cool command line. I don't think people know about. And it's nice you're going to hack on Koji to know about it. So first of all, if you don't know all the commands that Koji has, and there are a lot of them, Koji help will list them all. I'm just going to talk about a couple. And this is the one that's not really hacking Koji, but it's a real Swiss Army knife for me. I use it all the time, and I don't think people know about it. Koji lists history. So most every bit of data in Koji, not all of it, is versioned. They all have event IDs, and every time you make a change, like you tag a build, you don't lose the knowledge of that build once you have that tag. It's still there, you know. So Koji can tell you, particularly with things like tag data, all the information about tags that everyone, and you can hit that with list history. This is just one example. I did a list history of the dash-dash package Koji after the past couple of weeks, and you can see all the activity for the Koji package in the past few weeks. Without that dash-dash after argument, you get a very long stream. I don't have time to go into all the different options that this has. I think there's a lot. For the most part, each of these options automatically limits the query to the appropriate data. So if you put dash-dash user, it's only going to query tables that have a user field, etc. Anyway, like I said, not really happy on Koji, but it's a really useful tool. I just want to comment that you have a good path for that. So if you go to list history, dash-dash-dash-dash, it will show you all of it also. Oh, really? Yes. Which I'm a big fan of. I'm not sure if I have that. W-dash-dash and then tap that. I don't have it. Oh, there it is. I was just like, yeah. I don't know why it's like, it's very quick on my laptop. Probably because I've never used it, so it's not cashed. Oh, yeah. Yeah. I'm happy when there are things that have good shell completion and Koji has good shell completion. Well, I should probably make sure that's up to date. Let's see. So another interesting little thing that people don't know about the command line is that you can do direct RPC calls straight from the command line. So this is a simple one. If you wanted to see the raw data for the package Koji, call get package Koji and you see that it doesn't really show you much to name an ID, but it's a picture. So we have the ID number instead of just a name for that package. Let's do a more complicated query. Call this builds and since you're, it's making a, you know, complicated call it shell syntax doesn't really lend itself to data structures. I just fall into accepting Python on the command line, which you have to be clever about coding, but you can do it. So listing all builds, package ID 1181, which is Koji, I put a limit of one just to get the top one, otherwise it'd be. It's a very verbose, but it's a good way to explore the API which is that you may sometimes want to just run these real quick just to see what they do, just to see the raw data because the corresponding command line doesn't quite do what you want to or is hiding some little numeric detail from you. And also the command line will actually list the API for you. I have abbreviated, I just left one entry in there, but if you run Koji, I'll get the whole very long list of RPC calls and that's just one. It's dynamically generated, retrospectively. All right, well, just a few notes about the command line for Koji Hackers. Let's get into rooming. You're probably going to hack on Koji. You're probably going to do it in Python, I hope. Let's talk about other stuff later. Very simple example. Koji provides the Koji library and it makes it pretty easy just to talk to Koji. You need to create a client session object. Give it the URL server you're talking to. I'm using the Fudoro one here, but you need different URLs for different servers. I'm not doing authentication here. I'm not going to talk about that in this talk, but you can see how authentication is done by driving into the Koji client tool. Just a simple get the package so we get the ID and then get all the builds. This is basically what we did in the last example, but we're going to get in Python. That's all I'm actually doing. Python is in the shell. And print it out. Is it possible to put the same thing in C? Yeah, I'll go to other languages. Try to answer, yes, it's Xmar PC. If you can do Xmar PC, then you can just do that. For more examples here, the Koji client, there are lots of examples. There's no magic in most of the Koji tools. They pretty much just talk to the hub and say, do this. If you want to know how the Koji client does something, just go have a look at just a couple of RPC calls. A lot of the client commands are relatively thin wrappers around an RPC call or two. So feeding RPC API is stable? Yeah, that's pretty stable. Until we break it, which we'll have to do at some point because we have not seriously broken the API yet. We've done a lot to it, but we've been very careful about not breaking it. So other languages, it is Xmar PC. We do use a null extension, which is the most standard extension to Xmar PC. And that just is a new data type of null or null or whatever, which is really missing. Especially if you're going to use a language like Python where nuns abound everywhere, it's a lot easier to have a null extension. So we use that. If you're doing this with other languages, the null extension may give you trouble. There's a number of Xmar PC libraries that don't understand it. So make sure you have an Xmar PC library that does. Or if not, be very careful that you don't make any calls that return any nulls. One really nice thing you can do with command line is use a debug Xmar PC option, which will cause it to output the raw debug data for every RPC connection that it makes. Which is really handy if you're really trying to see the guts of how Koji is doing something so you can emulate it in another language. As a relatively psychotic example, this is using the RPC API with Curl. Just put the XML request in a file and pass it to Koji with Curl. And I would have put the output there, but it's, wow, it's really verbose. How do I figure out how to do that? So that will be the pretty output by add debug Xmar PC, XML RPC. So you can see it posting the same thing that we passed in the Curl and getting the exactly XML out. It's not a concise serialization language. Web themes is another way that you might play around Koji and make it do different stuff. If you guys have used Fedora's Koji, that is seen. The Fedora Koji does not look like the default theme. So what you can do with themes are you can override all the static images and the CSS and you can add extra inserts into the navigational art at the top and the footer at the bottom. And with that, you can actually do a lot, especially if you put some JavaScript in that header. And all you have to do is once you put that content under the themes directory, you just set Koji theme equals whatever you called it and whom is going to use that. And you don't have to copy everything. Koji looks into the default static directory first for everything that is looking for and if it doesn't find it, only then does it look in the theme first if one is set and if it doesn't find it, it falls back to the original static. So if you just want to override the CSS, you can make a theme directory with only Koji.CSS in it and that will work just fine. Or if you just want to change the icon at the top right corner, just make sure it's the same size unless you also change the CSS that you can do stuff like that. So this is pretty easy to do. It doesn't require any coding really unless you do JavaScript in your footer. And you guys see, I mean, this is an unthemed Koji as that's on my laptop. Not nearly as pretty as the one you're used to seeing. All right. Hub policy, there's your hand if you're familiar with hub policy-ish. All right. So hub policy is a way that you can configure that hub to control Koji's behavior in a number of different ways. It's a very simplistic configuration language, not vaguely, it looks vaguely shellish, but it's really much dumber than that. But with it, you can do a number of things like controlling who can tag where, set some fairly complex rules for that. So if you want only this small group of people to be able to tag this particular package and these particular tags with matching rules, you can set a policy that does that. You can control who can manipulate package lists. Most of these policies have defaults. The default package list policy is only admins can do it, but you can change that. There's a channel policy that determines what channel tags are going to, which is very useful if you want to make sure that certain tags only land on certain builders. And it might be helpful to see an example. So there's actually, I think the documentation, there's actually a doc on this that goes over the different policies and is that legible yet? Yes. Okay. So I mean, here there is a simple, a very simple tagging policy. Kind of, it's a little bit strict, but the default tagging policy is there's no restrictions apart from the tag permission settings that are really separate from this. But here is a policy that says that admins can tag, and other people can tag in a candidate, and other than that, no, you can't. Which is very strict, but maybe somebody wants that somewhere. Maybe you have a system where your regular users can only put builds into the candidate tags and you have another process that will move them out of candidate tags after some verification. So you could enforce that very strictly if you want to workflow that. There's a slightly more complicated one. So you, this one locks, prevents tagging, Apple, things that were built in the Apple tag anywhere else. So if you want to, say, keep Apple that builds isolated from the rest of your Koji instance, you can lock them into their own tags. Stuff like this. Now, is this hacking? Well, maybe because you could really control fine-grained what Koji is doing here. But also, you can add new tests with hub plugins. So, and when you start doing that, then you can really start to control what happens. So if you don't like the built-in tests, you can add your M. And just as an example of what the door is running with, the door just has a relatively short policy. You can sort of see the format. Each, it's a series of match lines. Each match rule has a name and arguments. And the double amp is and separates them, sort of means and and, but not exactly. And colon, colon, it gets you to the result of its own policy. And it just, first match wins. Again, very simplistic, but you can do a lot with it. Okay. On to hub plugins, since I mentioned them in the last slide. The hub access plugins, the easiest things to do with hub plugins are, these are the things that we have really put in to the code for you to do are adding IPI calls or adding policy tests or hook into the callback system in Koji. And there are a number of examples of hub plugins in the Koji code. There is a more of a hello world version. So this is really just a plugin that hooks into all the callbacks. There's some decorators who are doing that that are defining the plugin module. And all it does is log every callback that it receives. This is pretty much how the message bus plugin works, too, except instead of logging it, it splits it up to message bus. So it will get more work than that, but that's basically it. Message bus works the same way except it's more cut, you know, you've got to manage the connection in your timeouts. Yeah, so this does, it hooks into the callbacks the same way, except it limits it only to the ones that start with posts. And pretty much just takes that data from callback and splits it on the message bus. So writing plugins can be tricky because it delves into the internals of Koji. But the code is not that bad to read, I don't think, just once you figure out where to get started. And again, there are examples. So if you feel the need to write a plugin and there's lots of good reasons to, you know, have some examples, maybe ping us on Koji to battle. It's totally doable and you can do a lot. Builder plugins are maybe the less things that we have specific hooks for plugins to use. The easiest thing to do with build a plugin is add new task handlers, which is granted terribly useful. So the example that we have in the Koji code itself is the run root plugin that was added recently for the compose tools to use. And that's actually a bunch of setup. Koji Human just looks for subclass of base task handler when it loads a plugin. So if you subclass it, then you've just created a task handler. You don't have to decorate or anything. You want to see examples of task handlers just to put Koji in. And I think I've been mostly progressing in levels of complexity through here of things you might do. When you get to the end and you can't make Koji do what you want through any of those, that was what we're talking about. At that point, you've just got to change the code, which you can totally do. It lends in a year. Checking out make changes. To use the make targets I've got here, you'll have to commit locally first because it uses get archive in the make file to do the build. There's a test SRPM for real quick use that uses a directory. I mean, it's good, might as well just commit, right? So just make the SRPM and do what you will. If you make changes to Koji and you feel like you would like to share them with the world, we're out there. As I said, we're out in Europe. We take full requests. If you're working on a really big feature, you might want to talk to us on Koji Develop before you spend a lot of time on it. That would be a little quicker than I expected. Let me see if there are questions. If not, I can... Yeah? I heard that there are ways to set up a running instance for an environment, be it single, all in one VM, or a small set of Docker containers, just like something to scroll the laptop and shovel things over to in the test. All right, so... We... Yeah, what's the name again? Yeah, I said Koji. Koji. So there's two things. Yeah, Koji doesn't work anymore. Yeah, exactly. Koji Dojo. Koji Dojo, which John Casey wrote. Right, it also doesn't work. Okay, well, as of six weeks ago, it would not run on my laptop. You can override, so... I never use Koji Dojo. I just install it on my laptop. Which I think is a valid use case, but I think the install dock is a bit much to clock at base value if you're doing something kind of on the edge, and there's really a plug-in. Yeah, the only hard part about installing really is getting a damn SERC setup, assuming you want to use sort of what you do. Right. It is apparently... I never met anybody who... Yeah, so far I don't know if anybody who's ever accomplished from zero to running Koji without any previous knowledge of Koji or the build system, who's accomplished it in less than a week. So... I've been doing it in three days. There you go, guys. There you go. All right. I didn't pull off much. Well, it depends on your background too, right? If you're familiar with getting post backs up and running, if you're familiar with setting up SERCs, you're going to have a big advantage over those folks. And my question is, if those aren't things... You're not doing a production deployment of this, but you just need to add something to Koji now to plug in or maybe you want to run it so you can test a UI theme you're working on. Right. Is there a quick way for a demo? So the quick way for me is... is just run it. I find people will have success with Koji Dojo. Sorry if it's not working. I'll try again. You're going to end up just a weird case of... So I think... Yeah, so I think what needs to happen is, one, we call you to fix that dock. Because that dock is technically complete, but maybe not. I'm going to just have a quick start section of the play. For putting Koji down, don't do this introduction. Run these small set steps. Ta-da! Yeah, because when you're working on a UI theme, and I just want to have an instance of Koji running, I don't care about certs. I just need to be able to clean UI in VM or laptop. Yeah, if you work with Koji, don't use certs. Use password off. Right. I would never recommend you use password off. Anything remotely close to production for Koji. Because password off in Koji is not good. It was really just a debug step on the way of development. But it's there. That's fair. And normally what I do when I'm setting up first time is I set up the password off first. And I don't worry about off at first, right? So get it running. Make sure that I can hit the hub with no off. Just run a bunch of Koji-no off commands or run Koji commands that don't need off. Like most of the read-only commands just don't even try to authenticate. If you can get that working, then you know you're mostly there and it's just adding off on the end. Which can be a pain, I know. So sorry if that's not a better answer. No, I was just kidding. There was a recommended, you know. I just want to specify something about the issue that I had basically the certs as you are saying is the biggest issue. Then you don't have to make up the issue you are supposed to have. And you will run it from the other direction as well as a bunch of issues you need on things. And then you actually do a quite big beverage issue because you are promising something but somewhere where you are supposed to have beverages. And then it's kind of, you know, that's beyond the problem. Otherwise it's not a big deal. It's just not a few days. And so that's how it begins. But it's just like a few days in the statement, just like a few days to set up a Demi-margin. Ninety-five plus percent of potential developers out of the web are going to be out of the net. Because getting the X, Y, or Z over here that I can be working on, I can just like docker cobalt or vagrant up a development environment. And I'm not saying that those are potentially great solutions for your production deployment. They can whatever people are doing it, but just something that's low, very mentoring to be able to start hacking on things. And that's not the root of my question. Also, I think it's very important. If there was a very state-of-the-art way of doing it that everybody free was, you know, the way you would test environment, you would avoid coming in and finding one way and another way and, you know, or, you know, it's like right now, I don't know what the other works, but, you know, I've tried one type of development way of doing it. And, you know, it's like, you know, having that sort of whatnot in a way I think we all need to make sure people even if there are these ways that we go now, we won't have to really find them. A few more minutes, actually. So... Are there questions or is there anything you'd like me to demo? I'd be happy to. I've got a live page instance sitting right here. We can do whatever we want to it. You can abuse my laptop. Why not JSON? JSON wasn't... Did I ever study it in 2005? Well, there have been thoughts. Yes, there have been thoughts. Last year, I was proposing a number of things I wanted to do for Koji that are very big and invasive, and I definitely want to go to a JSON-based RBC. I'm not sure how fast we could reasonably drop X and RBC realistically for, you know, for, you know, for entrenched Koji deployment set prominent North American Linux vendors. Yeah, I was wondering whether... It's funny when you were just... I shared with you something about Koji through the book. Uh-huh. And you shared with me something about Koji 1.0. Well, this is how to hack on Koji, so... With the idea of working with Koji... With the idea of... So, as far as 2.0 goes, I think I had a bit of a different view on it now, which is that Koji 2.0 obviously has not happened. And the reason it didn't happen partially, besides me being too busy with other stuff, is that it was a lot. A very big, very ambitious, very risky set of changes and so I think the better plan is to really focus on... Instead of trying to put them all in one ball and say, let's make... We have to do all these things before we can call it 2.0. Let's focus on the things we can do individually. Content generators are in. We're working on adding more testing and it makes it easier to development. We're working on... I found three workers underway. Working on reworking the SSL code. So, various bits and pieces that were in 2.0 have landed on our development. But when we call it 2.0 is... We may call it 2.0 sooner rather than later for all the things along the 2.0 list are done and then maybe we'll go 3.0 before long. 4.0 because I think there's just too much... I don't know, emotional impact of that 2.0 number. I think we just need to get over it and move the API on it by bit. Yeah. Ralph, sorry, you had a question before. So, pretty much all of my clients you know, or that I also find that we are trying to push this many tasks and I'm so pretty into coaching. The only problem is that, you know, right now we've got two runners for both the ground and the other projects. And we've already had conversations that maybe we need to create something right, you know, either a plugin or something that can take a comment, which is supplying that data for the task. So, it doesn't have any metadata, it's just the output that you can't believe in all tasks, right? So, my question is, you know, is there a point in some sort of generic task where we could supply, you know, let's say, an app only executable just like Android, where you could have some metadata that we could be in the process and we could actually, you know, like very fast for some part of the tasks based on the, you know, metadata. I remember there were several things like ISOs, thinking of social media stuff, files and other things that I use. And I think that if we would just write a plugin or, you know, something for each of them, we would be much more comfortable doing it. Well, sure. We talked about this before, didn't we? Yeah, yeah. So, not three years. It sounds like what I was talking about when I was adding the yet another image tool support into Koji. And it was basically the same code as the other image tools, except really all that was changing was what command we were running in the truth and what sort of things we were pulling out. So it occurred to me that maybe there was some way to make that generic and have it be almost like an image spec that, you know, when we do our game builds, there's a spec and you build the RPMs in all sorts of different ways and the spec says, this is how you build it, this is what comes out. You can do the same thing for images and then allow images to use different tools and have different outputs specified in the spec. And then we don't have to write new plugins and new features in the Koji every time somebody comes up with a new image tool. They just let them change the spec. I don't know. So another thing is that I know that we are making new assumptions for Koji and I know that you want to build these for Koji and I know that you want to write new set of plugins. So if you want to think about it and probably show none out, I guess like Koji every year, that's something that we can work with. Let's see. Any other questions? I want to make a pitch to the room that helped contribute to the test suite. Yes. We have like 7% coverage right now. We still are just adjusting the hub and we got it up to 25% coverage and then we started with the other person to see a lot of performance and we added all that new code that the coverage percentage went down. And we built about 1% to 2% that helped us to make a lot of progress in time for the Koji test. We'll get a fresh clone and my checkout's gotten weird extra stuff. In my opinion, it's one of the few instances in the way of doing a lot of good things and one of the first 5-3 there's fear that we'll break everything but if we have a test suite, then we can run on high-2, run on high-3, then we're kind of able to work with some stuff. And then things like, we're back here, with 10,000 on Koji, about 5,000 in the local office. 15%? No, no, no. Twice what you said. Yeah. Fast. Like that. Take the money. I have a question. Have you found any other... Anyone else that's interested in the namescaping type of things? No, so this would be the past namescaping. The build namescaping? Yeah, yeah. Yeah, I mean, I'm interested in it. It's... Is that incompatible with doing that working in Koji 1? It's... So... It breaks a deeply held assumption, which is that NBRs are unique. And we assume that NBRs are unique in a lot of different ways, right? So, you run the build and you give it an NBR, you call it build with an NBR, you expect to get one answer back. Not ten. But now, you can get ten with that. FOPAS, tricky. The work that I did a few years ago that never landed did some gymnastics to work around the file system namespace thing. But to do it right, you really want to get away from using the NBR in the build app at all. But, again, change that and people are going to get that. Yeah. You actually have a majority of the team have the same problem. Right now, we are building modules which are, you know, vehicles and vacuums. And each module actually has its own disk. We are supplying it. And we actually build it and we provide it to the builder. It's kind of epic next time. We request for setting the disk type, you know, directly to the market like we do. So we regulate how the work we do in each space as well. And the uniqueness of the NBR is a problem for us. Yeah. So what is, like, what is, like, what is running in the NBR to provide the set of default namespace? That's the implementation that I have does have a default namespace. You kind of have to do it that way. Yeah. And I think people... There's implementation that is not ready to go go in there. There's a... You know, again, if we had a test we would help. So just so many calls change the behavior with it when you add this one simple change. So it's tricky. But we can talk about that. I think I have a minute of time. Yeah. One minute. All right. So no time for a demo. Any last minute questions? All right. Well, thank you guys for coming. Thanks for listening. I hope... Now I'm done with this. My mind is clear and I'm happy to talk about any sort of random coji things you guys want to talk about. So just hit me up in the hallways. You know, I'm probably going to be doing some hacking on a coji while I'm here. So, yeah, just come talk to me. Thanks.