 So I am David Cantrell and we have Tim Flink here who I'm going to ask to come up in in a minute if he wants to We actually have two talks that are related to each other One is today the other one's tomorrow in the morning. I think 10 30 maybe yeah so what we've been working on is The integration of a tool called RPM inspect into the for the Fedora CI pipeline so the talk tomorrow is gonna be a bit more about Fedora CI and What what we're trying to do there? I was gonna give a brief introduction here and then ask Tim to add anything to it if he wants to So yeah Fedora CI RPM inspect and then how you can help if you're interested in helping there's plenty of work to do Okay, so Fedora CI Please go to the talk tomorrow if you're interested in that a Framework for auto automating tests and providing results. That's really generic, but if you went to the Packager workflow talk or earlier by a meal we kind of talked about the experience that a package or goes through and Wanting to know if builds failed and stuff like that. So this is all tied into Getting to a point where builds that we do of packages can be trusted to put into an OS compose So we don't have raw hide and other releases in an unknown state So doing as many tests as we can Before and there's there's a lot of different aspects to what constitutes a test every package is a little different You know when you test a Python module that's going to be different than testing say the bootloader so How do we get there? Well, we have this Fedora CI infrastructure that's going to provide these runners and triggers and things like that our PM inspect is Just one example that I'm going to go into here. So knowing that builds are tested and working Gating a build before we get there now. I know we have this enabled for raw hide We have it. I think keyed through Bodie One thing that I would like to do is some of these tests move them to before that Which we kind of running right now for our PM inspect We have a few more things to tie together there Because I think a lot of these things need to happen before we even get to that point and not for a high Well, yeah True. Yeah What do you want to add anything else to Fedora CI stuff right now? No Okay. All right. So here's the docs link. Oh, and by the way, my slides are I write them also to be like notes and reference for people later So that's why there's like a URL here So that's that's Fedora CI broadly Now RPM inspect Build deviation analysis tool. Okay, that's kind of a mouthful Let's talk about a little bit of history first So red hat developed an internal QE tool called RPM diff Not the RPM diff that you've seen as part of RPM lent. We like to reuse names so RPM diff began life in The internal one let's say 2003 2004 doesn't really matter a long time ago before avert before containers before the cloud We had no clouds at all And it runs as a service and performs tests on builds that we do internally The service aspect here is key there's no way for package maintainer to really interact with RPM diff at all like it's it's automatically scheduled the results are then post-itting you have to Fix it sometimes often not really knowing what it's testing So what kind of test does it do package policy compliance? So making sure you've named something correctly Making sure you have a an approved license on the package all of that metadata in there We have yeah, the legal checks currently be I verification is a big one for the internal tool Enforcing security policies. Do you have any set you ID route executables in the package payload? And are they allowed? So there's these data files at RPM diff will query to see if You know you you haven't allowed one. So there's about 40 tests that RPM diff does and they're all of Varying complexity This last thing here as well comparisons from one build to the next that's where the diff aspect comes in so For rel packages, this is really important for for compatibility reasons we want to avoid any surprise regressions so Files is appearing new files appearing in a subsequent build a new package appearing or going away sometimes executables would grow Substantially like you know you go from one megabyte to 300 megabytes something went wrong with linking So it'll catch things like that and report it Okay, so how does it do this? internally We use the errata tool developer, okay, so at this point the package maintainer has Completed all of their work. They checked in all of their fixes to disk it they've built it in Koji and now they log into this web-based tool to do the paperwork So this is basically like the internal version of Koji if you haven't used it. All right, so the errata tool schedules an RPM diff job Oh, yeah, I meant Bodhi. Sorry. Yeah Thank you Too many names to keep track of okay, so the errata tool schedules an RPM diff job That talks to the RPM diff hub, which is a Python Server running actually on a physical server It verifies that the build exists. It does this by talking to Koji If the build exists it hands it off to a worker a worker is another physical server There's a pool of them and this is running a Python service as well the worker Performs the RPM diff operation. It copies the builds from it and it an NFS share There's no way around that It then runs a pearl script to unpack the RPMs Then it runs a Python script to actually run the checker program Checker program is written in C and it iterates over all of those 40 tests I was talking about and each one of those spawns multiple pearl and Python scripts and it collects the results and Puts it together in XML format and hands it back up to the hub It's not a great architecture, but again given the age and given the time period it was written. It's kind of understandable It's the service site of it the hub and the worker are too closely tied to the part that performs a test so Runs on physical hardware that would be a little easy to create the biggest problem is the requirement of the NFS share Which is kind of hard to unravel right now It's a legacy code base a lot of technical debt. I am not a pearl expert At all so it's really hard to kind of go through that There's too many moving parts you can't run it from the command line So a developer can't use this tool at all and it only gives you XML because at the time it was written that was you know What everyone wanted yeah Okay, so call graph I love this so the When I was talking about how how this runs the C program the RPM diff checker program This is what the call graph looks like if you're familiar at all with this So this is the entry points all the function entry points in the C program The second row there is basically corresponds with One or more of those tests I was talking about and each one of those boxes is going to be spawning multiple External scripts pearl Python sometimes shell So this is just running it on to Z shell builds and I captured this I thought it was interesting Okay, so back to RPM inspect so Going through all of that code. I thought this is going to be very difficult to make it usable in the fedora space so What are the actual goals here so ensure package reliability Developers aware of the build changes we want developers to be able to modify packages and stay in packaging compliance But we also want to provide data in a CI environment to make decisions. So I Wrote this down got to be able to run it locally got to be able to run it in container Need basically do all the things that that RPM diff can't do directly right now that it relies on external tools to do So the service side that hub and worker aspect that's the part that's being broken out and becoming part of fedora CI and Tim has written that now it runs in fedora And we just need to connect where the results go, which is kind of cool Service listens for new builds and launches RPM inspect runs RPM inspect does those tests and then this hasn't been done yet, but It'll be the next step we get those end of results DB and then we can tie it into the rest of the chain there So Policy checks and then differences. So one thing that RPM diff could not do is just run against one build Which I thought would be really useful for developers. Let's say we have a packaging policy change in fedora and You can just you can check your latest build and say well Does it comply against what the policy is? It does require RPM inspect knowing what that is But we have one point to change and then a tool that can run for developers and then we also have the ability to do The differences between two builds So it does look at sub packages right now. I haven't actually added that that's a good idea actually if if a file by the same name Same location some some kind of a qualifier and it could be a little more intelligent right now All it does at the moment is what RPM diff did which was just identify So when it's when it's doing two builds It'll have the before build and the after build and then it goes through all the sub packages And that figures out the peers and then all of the files and those so anything that's In a before build and has no peer in the after build And likewise so that that's that's where that starts, but yeah, it would make sense to Catch it if it moves because that is quite common Yeah, certainly in fedora, so yeah Yeah, that seems totally doable at least in some capacity Policy check so these are things that I do right now again these ideas came from RPM diff So I check the license tag validate that against the approved license database check header metadata Political concerns is is an interesting one. So this is This is from rel red hat being a US company has to be conscious of certain I Don't know embargoes or or political concerns or market concerns. So there is a the political check in there is a white list for a loud files that would otherwise trigger a warning So an example might be The flag of Taiwan being included in the release and we ship it to China China doesn't like the flag of Taiwan for various reasons so This is a thing that red hat has to be concerned about so it you know Might not necessarily apply verbatim to fedora But there could be other things One of the things I have done an RPM inspect is set it up so that we can easily enable and disable tests per packages So with RPM diff I didn't mention but all those 40 tests everything runs for every build You don't get a choice it's all or nothing so this you can just run one test or you can disable one or two So we have that forbidden language again. This one is maybe not a strict in fedora Red Hat has you know, it looks for English Swear words and some other things Use of macro so we can enforce that and it's pretty easy Java bytecode major version check I was just working with the Java Developers, I forget what things they well I know they do the open JDK packaging but some other stuff too and I was asking them the the origin of this test and For rel we ensure that we don't ship any class files that have a Java major version In the class file header that does not match the version of open JDK that we ship I'm thinking okay. Well that that's pretty good. So I added that in there Also, I also found out that Java class files have the same first Same first bytes as a Mac OS 10 executable Which yeah, it's in hex it spells cafe babe So that was kind of interesting Elf object checks, so there's a lot here especially tied to security enforcement Policies around that and then the set you ID set GID things like that So this is this is what's present now I have a long list of things that I want to add most of them ideas from RPM diff yes So I've built this as a program in a library and the intent is we can just add we can keep adding tests in it Some people cringe when I mention this, but it is written in C The primary reason for that is most libraries that I need to talk with to do the work are written in C I am happy to help you. There is a template inspect function and It's it's pretty straightforward So yeah, I mean you basically if you want to implement If you want to implement a test it's really just One new source file and modifying a header and then it's there so yeah All right, so build types regular Koji RPM builds and also RPM inspect supports module builds RPM diff does not and This is built to extend to other things that Koji might output in the future I know it does more than RPM builds, but let's say at next year's flock We all decide to start using Debian packages, you know, we could modify it for that Very easily it'd be trivial Okay support files so one problem that RPM diff had was it hard-coded basically all of the knowledge it was checking for It was in a mix of header files String constants just it was all over the place. So I started splitting that out into a separate data package And a configuration file to provide even more fine-grained control so RPM inspect comf and then the Product specific data is in the RPM inspect data package now the program itself provides a package called RPM inspect data Generic that is a template package and then I have a separate one called RPM inspect data fedora That's the one that you want to install if you run this on your system It's kind of the same idea as Fedora release red hat release generic release that kind of thing Because there is a separate internal package called RPM inspect data red hat and it is Completely and totally different. Let me assure you Okay, so how do we run this first you want to install it I Actually forgot on that first line to install RPM inspect so you get that in the data package. There is a man page. There's help options But the basic syntax is here Previous build new build is optional. You could just specify one build and then here's an example. So verbose output minus K is I keep all of the temporary files Minus capital T license I'm only going to perform the license test on this RPM inspect run If you run RPM inspect minus L, it'll list all the available tests So you could do multiple ones. They are separated by a comma All right, what can I do today? I've kind of been through some of this here By default it'll output to the console just in like Mostly human readable format. It can also do a JSON output Which we're using to integrate into the CI pipelines And I also added a fetch only mode if you want to use it to just download a build by nvr Discuss I was doing that and it was handy So, yeah, it it does a lot right now Just kind of kind of ran through Mostly all of these. Oh, yeah validate desktop files Make sure you have the disk tag now the way this is set up is as packaging policy evolves as our Build process evolves the knowledge that RPM inspect has will have to change So that's why a lot of this lives in that data package So we don't have to go and change code for these things But there may be new tests that we need to add as time goes on And there may be tests that we can remove that you know are no longer valid So one thing I wanted to do was show So Right now this is So I've got a copper repository in Fedora So every time code is checked into the upstream repo, it'll build automatically. We're doing sent to us Seven and then most recent Fedora's the package does exist in rawhide. I have not Done it officially for previous releases if someone wants to volunteer for that I'm happy to accept that offer The best way right now to use it and the command line is just to enable the copper repo And then if you're familiar with Our build system. So this would be this is Z shell Just listing all the recent builds So I go here when I'm testing it and I I find two previous builds so let's Yeah All right, so I've already installed it because I did not want network connectivity to fail and all that kind of stuff But we'll just run through a couple of things here. So your normal output screen here It'll it'll show you I've well I try to show defaults here And then how to override it Let's see Yeah, the dash Dash-list to list tests format type The test list those are probably the most important. Oh, yeah, and for the let's see product release string So product release string is something unique to RPM diff and I didn't want to carry it over To RPM and spec so internally when an RPM diff job is run It has to be told what release of rel to run for so is it going to run for rel 8 or rel 7 and I thought to myself You know, I can probably infer that from the disc tag So what it does right now is it it grabs the disc tag from your before and after build and if they match That's the release. It's going to use if they don't match It'll stop and tell you to tell me what release to use And we're just going to go with disc tag because that's established And is is well documented So you could take like Fedora 31 builds and say do it against Fedora 30 question But it's it stops and tells you why So, yeah, I know it's not yeah, it's not interactive. It's going to exit yeah, so if I Run it at the command line here, let's see if I want to do the license and metadata tests for Whoops, I've already cashed some builds because again, I don't trust the network I'll do license metadata and then Yeah Koji builds my before and My after build now if you I'm doing I'm telling it the exact path to the build Only cuz I want to save time here But you can just specify the nvr. So, you know your your specification can look like this and What rpm inspect will do is ask Koji it'll say is that a build and if it is tell me where to get it And it'll download it So you don't you don't have to do that separately. It just knows and the rpm inspect Confile tells it where to get that info Okay, so this will perform two tests. This is gonna output in what I call the human readable format, but So that was where it stopped I don't know if everyone saw but I Have the trailing slash there. I Could probably fix that I've actually got a long list of things. I'm gonna fix tonight. Yeah But yeah, so you see it grabbed FC 31. They don't match and it doesn't know what to do It does have a verbose mode. All right, so here we go. Oh, by the way, rpm diff for those Unfamiliar with it We would start this job and then walk out there and get a cup of coffee And talk about how the conference is going and then we come back and it might be done That's that's right. Yeah, so so we've got a big speed improvement here so This is the test name So license test and then it's just reporting text here a lot of this I have lifted from the output from rpm diff Please don't take this as being authoritative. I am Welcome to input of what this this stuff should say question Correct Correct So These are all really good points and I think fedora needs a little more flexibility than what rel does so I would say And this is a conversation that Tim and I need to have to make some decisions on Fedora QA needs to have some input as well My personal thought is that for rawhide the comparison is most useful to the last build that passed And For a released version it would be the last one that we shipped as an update just as a starting point I'm in the fedora CI SIG and Want to have the conversation in that sort of context. So if you want to be part of it I encourage you to watch there and yeah Because again, this is I want input from package maintainers. I want this to be a useful Package maintainer tool. I don't want this to be something that you don't get a say in because no one no one likes tools like that No Yeah Yeah, no, I thought about that the main reason is I've got the the peer detection And I have to iterate in a certain way, but I have written it such that this can run Like as many times as you want on the same system rpm diff was very strict in that Exactly one can run at a time It just yeah So I've made sure that that can happen it can run in a container can run inside mock if you don't want to do a container I mean I've tried to make it pretty flexible in that respect So this is the output of the license stuff and then down here had her metadata. So there are some errors here now this Package build host. This is again a test I lifted from rpm diff It checks to make sure that The package build host that gets recorded in the rpm header Internally ends with dot red hat comm so when I created the rpm inspect data package I said, oh, well it makes sense that we probably want to build things on dot fedora project org host Well, you can see this package was built somewhere else. I don't know Hogwarts dot boss red hat calm, but This was not built in Koji Or maybe it was I don't know So but this also I have here suggested remedy and this is another area that I would love comments on from from people for specific tests It tells you, you know What can you do to fix this rpm diff doesn't do that? Let's see All right The entirely possible. Yeah So that's that's one that that may need a little investigation So I wanted to do a Different format here real quick So that's the default output if you're using it as a developer And this is what it would be for that we're using to integrate into the CI pipeline I'm not doing XML because I don't see a need But it's set up such that we can drop in additional output formats if we need them So there we go Alright Okay. Oh, yeah, so the call graph so I explained rpm inspect You saw the one for rpm diff This is what I have right now for rpm inspect And keep in mind that the rpm diff one doesn't download Packages does not directly talk to koji and can't even unpack an rpm Rpm inspect does all that and I try to make code reuse a thing if you can't remember what the The other one looks like there it is and then here's the new one So again trying to make it a little more Maintainable and usable and not have a single choke point for the whole program So Contribution so I have a unit test suite has been started That's kind of the first step an integration test suite is in the work So this is this is one of the huge advantages of rpm diff right now is Despite being a legacy code base largely in maintenance mode for over a decade It has a very extensive integration test suite all of the people that worked on maintenance for it We're very very disciplined in adding in Integration testing time there was a bug so I want to make sure that we have that for rpm inspect just to make sure it's reliable Documenting the details of individual inspection. So what does the license license test do where does it get that data? How can I change that data who owns the data? It's testing against that kind of stuff designing new tests based on the fedora packaging policy or other things So for instance the python test, you know that you were mentioning would be good And then help maintaining the the data package. So this is going to be a moving target just because fedora evolves And I'm not going to know everything about the distribution. So Send pull requests and also Bug fixes and code and just general feedback This is ready to use right now if you maintain packages. I'm interested in hearing feedback like You know, what does it look like? what fails You know, it's not perfect right now But the more people using it using it will help get it to a really good point. I Have a goal of implementing the rpm diff test that makes sense as well as coming up with a process to Take the fedora packaging policy and get that into a test that is easy to maintain through the data package So as the packaging policy evolves, we can update the data package and rpm inspect will just pick up all those changes So we can keep things in compliance Code lives here right now. I do have there's an organization project called rpm inspect that I'm going to move everything to I haven't done that yet I mentioned packages are in rawhide automated builds are in copper and then We've got fedora si on frino poundo sci internally. I'm decantrell Get hub issues is where to tell me about problems and question so actually I made a bunch of notes about that earlier and I do think it would be really useful like it can already do a lot of what fedora review is doing now and the idea of encapsulating the packaging policy in a data file format I Like I think it would be pretty easy and it could potentially replace that fedora review functionality Yeah question So that was a that's a thing from rpm diff. I didn't really explain that so When rpm diff runs internally for a rel build You're going to get a pass or a fail or something called needs inspection If it's on needs inspection or fail you go in and you look at the output of the individual tests and as the package maintainer You can wave that result and explain why the test failed Or if it's legitimate you have to go and fix it and submit it again I don't know if that's really going to be useful data for For fedora, but I it's in there right now and it's in the results It would be very easy to remove or just alter as we see fit for gaining, but that's what that's from the waiver authorization Internally rpm diff has subcategories of tests that the results can only be waived by say the security team or Another team the default is anyone can waive the result. So that's why that was in there So That's a good question. So what we were going to talk about At the end of the week and the packet workshop is incorporating this into the packet workflow So say you have the spec file upstream you get a pull request for a project a build is kicked off We run RPM inspect the results go to results DB and then that's visible through the packet output in some capacity And No, not yet, and I wanted it it is running right now the results just aren't consumable yet But the idea is to have that Completely tied in so the results are visible somewhere so yeah, and and With that we'll be able to make gating decisions. So if there's a lot of failures or Any failures we can say this this package Cannot proceed to the next step This build can't proceed and here's why So that yeah, that's the idea Sorry Right it does not rpm inspect does not perform any build So it just looks at the output of what an rpm build minus BA would be So when I was writing down I was thinking yeah I could do the review and I went and looked at the code briefly and saw that it does that and I thought Well, maybe that's out of scope for rpm inspect, but Yeah, there's some overlap there with policy compliance question Yeah, I think Yeah, I really don't want our payment spec to start doing builds and stuff like that but if yeah, Fedora review can make use of the Policy checks, you know that I think that would be a win so yeah maybe only and and I you had another question and I went around so You can certainly do that. Yes, eventually it'll be automated when you submit a Koji build But yeah, you can do that now if you want to yeah, and actually I'm interested if you do that so like Message me or email me like how that goes because I'm curious What'll happen there? Okay. All right. Yes Define average size Okay, so I would yeah, so the kernel takes a while right now and I'm I'm considering like Maybe I can restrict it like with With mock or with Koji you can Restrict it to a certain architecture and that may be able to allow us to farm things out a little more in parallel Right now there's there's a bug in one test where when you run it against the kernel package It's actually gonna seg fault. I was yeah. Yeah, right. It's really fast. Yeah So the the full suite of like all the tests that are implemented in there right now Which are like 10 or 11? It's less than a minute. So I was talking about that integration test suite and kernel glib-c gcc are once and I'm Running on the side constantly to see what the output is and and where we can do some performance improvements Oh, oh, I could do tech live That's a lot of sub packages Yes What what fedora leases this available for no. Oh, yeah, okay, so So if we're let's say we're doing rawhide's doing builds for fedora 31 right now. That's what the disc tag is so That check is in place to ensure that by default you are comparing builds from the same like release development chain or Collection That again comes from rpm diff design, but there's no restriction to do that you like if you specify a Fedora 30 build as the before one and a fedora 31 as the after build all you have to do is tell rpm inspect Use this as the release string Assume it's fedora 31 or assume it's fedora 30 and that's Literally the only reason it exists in rpm diff, but I think it's a nice default catch to Prevent any accidental comparisons From builds that were done for a previous release. So yeah Any more questions? Yes Yes Yes Yeah, we don't we don't have any of that set up yet. That is Pending yeah, yeah Thank you. I'm sorry. I wasn't too clear on that, but yes implementing it as as a gating test Definitely the main objective here So yeah, I'll be at the packet workshop So if you want more information About rpm inspect or any details. I'm happy to answer those I will be Working to you know help Figure out how we're gonna connect packet with rpm inspect. So that'll be part of the discussion as well Yes, thank you, and I think I finished early All right, well everyone gets an extra five minutes or they can keep asking questions Okay, thank you