 So, hi, my name is Angelja and I'm here to tell you something about FreshMaker, which is new service We are going to deploy or we want to deploy in Fedora. Hopefully soon The goal of this service is rebuilding container of container images and modules and probably some other artifacts in the future like flatback Automatically when some dependency of those artifacts is updated So at the beginning I want to show you the topics I would like to cover in this presentation So we will talk about motivation. I mean why FreshMaker is useful for people in Fedora and Fedora infrastructure Then I want to cover like the way how it works events and policies, container rebuilds and in the end I will talk about the current status of the project and The plans for the future and I will also ask you for for help with the project So let's start with the motivation. I Think everyone here is a package maintainer or Know how the things work like? Yeah, is it true? Okay, but I will briefly describe the current status So if you are a package maintainer and you want to update your package you want to change the spec file Then you commit to the change to this git submit this submit it as a rebuild or as a build in Koji, which is good system and then submit that to body Which is used for testing the package and once the package is tested it can be included in Fedora's library repository and This is actually what package maintainer us, but it's not the last thing which has to be done with the package because we can have for example local images, sorry container images and To get the updated package into the container container image We have to someone ensure that the container image gets rebuilt and this is kind of Responsibility of a container image maintainer because he should track the updates in federal stable and do the rebuild Or he can just blindly reveal his stock his container image like every week Ensure that updates are in There's also another issue with container images and that's layer container image rebuild So container image can depend on another image and once the parent image is rebuilt someone or something needs to rebuild the children Container images and this is also kind of hard to do because there's no official tool or official way how to do that in Fedora Although this is not so big issue for containers in Fedora because there are not so many dependencies between containers But the same issue exists for example for modules because modules depend on each other much more often and in certain times we want to rebuild like module and all the modules depending on it and just all the three whole whole three and This is like also issue because you have to manually track the dependencies between modules There's no official way how to do that for how to do it automatically The same issue as with RPMs in containers exist also for modules So you can have module which has RPM in it and once the RPM or the package in the module gets updated You want to rebuild all the modules containing that RPM to ensure that the module has the latest RPM and Now imagine that in the future we could have like Modules included in container images. So once you've got the package You have to rebuild the module and you have to rebuild all the containers and all the layer containers And that's how things are started complicated or get getting more complex Open Taylor is also working on flat pecs in Fedora. So that's another another like way how we will distribute content and we need to rebuild flat pecs Based on RPM updates, too So this is like current situational problems. We are trying to solve with fresh maker and As I said fresh maker is here to track the dependencies between all those artifacts and Once that artifact change changes, it should depend all the artifacts depending on it To do that it listens on that message bus as I said, it's a service like running on federal infrastructure and it listens for those events from various federal infrastructure services like Koji, Git, module build service and Based on those events it records the rebuilds It also has to drag the dependencies between the artifacts like what depends on what and It uses PDC as a source for this information and some details are also stored in Koji and And as I said when some event Some event happens. It uses those information to rebuild all the depending artifacts on the All the depending artifacts which depend on the artifact for which the given has been like submitted Yeah, it's important to say that fresh maker itself doesn't do the rebuild it always asks some other service to the rebuild like module build service or Koji so this kind of orchestrator for those services So that's like main main way how fresh maker works Are there any questions like for for these these two slides because it's important. Yeah You mean oh, yeah, the question is if fresh maker triggers the chain builds Do you mean like cross artifact types like They might be if the change is like breaking a bi But normally if you update the library, you don't have to rebuild All the modules based on the library But to get the library into the container image you have to rebuild container images to actually get it there So it triggers the rebuild of container images, but it doesn't trigger the rebuild of modules because For library update, which is like shared object. It's not useful Does it answer it or Okay So what would you store in good or in like which get recall stories? Sorry, and so the second point is that you tried dependencies and it's usually So there Yeah, there might be something but I'm not sure right now What is I mean We have some issues here some situations in Docker images where we need to actually find out what's in the But I'm not I'm not sure right now So let's move Yeah, so this is the first part of a fresh maker it does the rebuilds and The second part is about tracking because for example as container container image maintainer Once fresh maker rebuilds your image you want to find you want to be able to find out like why it happened So we have rest API which can be varied and it provides you the information that Particular container images been rebuilt because of update or following list of RPMs But because of like layered images be updated and so on and it works also the other way around like as a package maintainer you want to see What containers or what modules have been rebuilt using your RPM update so you can query the fresh maker API for the nvr package and you will see all the Artifacts which has been revealed which have been rebuld using that nvr Then where you can also track the progress of the rebuild so in case you carry the fresh maker during the Rebuild you will find out that some packages or some sorry some container images are still being revealed some are already done And so I mean you can you can track the progress of read like that There are some plans to create some user interface which Show it in some nice way, but right now we have just the JSON JSON output of fresh maker But it's the useful So as I said, there are multiple types of artifacts and events for artifacts It's not clear. It's like container image module RPM and so on but For events is not so clear which event should trigger the reveal So for example in for container image, you can have three types of events Which can be used to trigger the reveal the first one is when the RPM in When the RPM which is included in container image is built the second one when this RPM is tested or released and Freshmaker as a tool supports all those events, but it depends on your use case Which one should be used? So for example, if you plan to use fresh maker as continuous integration tool you want to build for example container image as soon as RPM is built to be actually able to find out that that RPM The local image so I brought the container image and All the module and so on but if you plan to release I mean build Regal artifacts only to release them You can actually be quite happy with Rebuilding container images just when RPM is tested or it's actually released so the goal of this slide is to actually tell you that Freshmaker supports a lot of events, but it depends on your use case, which one should be used and this use case is defined by a so-called policy and That means that it's a configuration of fresh maker which sets the boundaries to fresh maker and sets like What should be revealed on what event and in what time in the pipeline? Because we cannot rebuild like every every every container image for every module on every event Because that would mean that there will be just a lot of builds and it's not before the infrastructure so this so we had to find some Policies to actually do the useful rebuilds without and using like So the examples of policies The first one is that we can for example combine multiple events to single reveal So if you have container image and there is some RPM getting into federal stable You you that RPM and wait for example for one day or six hours for upcoming RPM updates and then do the single reveal before those RPM and There has to be some decision to actually decide what What should be the time, you know, they should be the delay delay before we start recording and so other situation is that for example for RP and updates which has Some security impacts you want to release. So you want to rebuild the container image as soon as possible to like have the updated security issue there really soon And I as I said There's currently not clear Answer to the question who decides the policy for Fedora because it could be like special interested groups for Fedora so for container images or It can be first go people like that We definitely want to come up with some Possibilities what what is even possible with fresh maker and how it could work like but you know the delays between rebuilds or if RPM security fixie should be handled as soon as possible or we are okay with waiting be pre-built It's just question of a policy and something some are some are to decide it. Yeah Is there a way to trigger a rebuild? It's fun Yeah, we can we can do it isn't implemented right now because we have been We have to be able to control it like manually or even stop rebuilding of something like in case it would case some problems Or questions You mean like in configuration I can show it Or if you don't mind I can show it in the in the end Yeah, because I don't have to find it It's not like something I would have prepared and I'm not sure I have new laptop here, and I'm not sure where my things are So yeah So if we decide to rebuild something what what can be shipped we have two ways how to do that Again, I will talk about container rebuilds, but it's the same for modules so The first way is that We can rebuild container once the RPM is in federal stable So we have to wait for testing for the RPM to pass and then when it is federal stable repository we trigger the rebuild in Koji and We got the RPM. We got the updated container image. This is easy to do filmmaker supports supports that even today but the problem with that is that Container like container image cannot be released together with the RPM because the RPM is already released in the time This like might be okay, but it's not it's not the best thing to do because ideally we would like to have the container image released in the same time as RPM and To do that we have to do the complex complexity of the reveal and That also has to base up to do that because to have to be able to release container image with updated RPM We have to wait for the RPM to be signed and that happens as soon as the RPM is submitted to body for testing and we can then We can then wait for the RPM to be really testing before the before it is getting moved to the stable So we can rebuild container image once the RPM is tested But that wouldn't allow us to test the RPM and container image in the same time because I think it's already tested. It's going to be in the stable. It just matter of the time before body actually matches the update But it gives us some time to actually do the reveal before the RPM is in the stable repository and it's seen on the mirrors and Even better, we can just Rebuild the RPM, sorry, rebuild the container image once the RPM is being submitted to body. So Both container image and RPM can be tested in the same time or more or less the same time because You're able to take some times, but we can just, you know, Bound it together and test it in the same time And then even release them in the same time The problem with that is that In case the RPM fails the tests, we will have a bunch of container images Which are actually broken right now, like they are useless because we cannot ship them because the RPM failed the test and Once once we have the updated RPM With the fixed test, we will do the rebuild again So it's again part of the policy if we are okay with rebuilding container images, which may not be released because We will find out later that the source of those container images are like broken It's it could be resources. Yeah, that's the only that's the only concern I think and I'm not sure how Is to reveal the container image how many you know was the resource impact of that in Koji and If we decide to do that for all the container images and we plan to have more or them of them or We decide to do it for plastics and modules the same way it can be a problem like Yeah There's also the reverse problem of if you're bundling the Updates and our game to succeed with a container test panel. Does that block the movies? Yeah, it's good And actually those policies should be defined by like green wave Which is another dog Madness After mine told you so you can just stay here to find out more information about how the tests will be gating So I will go to another side This is really important to say because the work of fresh maker ends with all the artifacts being revealed It doesn't do the releasing part. So for example, if it if it rebuilds modules it won't It won't submit them as an update to body It really just do the rebuild rebuild thing There are some plans to do another tool which will take output of fresh maker and do the release But for now we plan to just email maintainer of a container image or module Saying him that here are here is the updated container image for you and It contains following our team updates or it has been revealed because of following reasons and it's up to you to actually decide if you want to really submit this and as an update or You want to do some manual changes there or? There's a container image maternity probably know that it's not right. I'm to the update. I don't know but We need to figure out those are where to do that But not necessarily another way, but something else mentioned to that because sending an event to the meteor is one thing But it's not really We need a place where people interesting container maintenance So maybe there's a place for a seat And a dashboard of some sort that I would be a real good thing. Yeah, I think the easy fix would just be to CC the atomic working But Yeah, I think that's a good point Yeah, and that sounds great, I just I figured trying to requesting a web dashboard be a considerable amount of work compared to There are some plans to make like Mike said, yes, okay, Mike. Yeah, I was gonna say So And I wanted to say that that might ask yesterday like for contribution Do some kind of a dashboard or pipeline over you think which would just show What's the status of the rebuilds and not just the rebels but If you do an update of a package How far is that in a pipeline and you can try the progress of that package update? So that's something We would like people to contribute to It was like don't have time for every Freshmaker have a command line It's it's not a command line to it's like service running in federal. Yeah, I mean But is there a command line tool to query the API so I can see the list of rebuilds? I mean, you can use Google Write a serious online tool. Yeah, we could we could have that But right now we have just used an idea Yeah, so after the clock Right now we are discussing the policies of that's question Sorry Add So right now we want to define the policies to make a project to actually make it useful for federal So we are discussing that with people here This is something we need to do before the deployment because we need to like find out what should be the pipeline for the freshmaker to make use for rebuilds without without breaking And in the face in the process of deployment, we would like to focus more on modules because like the problem with updated components in modules and the need to rebuild the module when when the component is updated is kind of Our first use case and after that We would like to focus on containers You can even do it like in the same time but really There's the need to even modules And I think yeah how to get involved So it's useful for us to discuss the policies And even right now when you know what freshmaker can do for you You can find out some more use cases So we would like to be interested or we are interested in those use cases Which we don't cover right now You can contribute the code of course That's a link to freshmaker figure repository That the code can be found And you can you can find us In federal modularity channel on free note or even further on relink Relink I don't know And There's no idea right you can just write me if you have some questions or improvements So that's actually my last slide so if you have further questions So will you be submitting a new change for federal policy with the policies or Or planning to do it? That's a good question. We were just started, you know talking about them And I will have to Ask like people are freshmaker to find out if that's good way how to do that thing Because right now we have like the tool and we have found out that we need to like really define Define how it should behave in federal infra So I think a good place to start would be what is the policy today like how often are things Rebuild and release Because we could just as a as a step one we could just duplicate the like current zero policy process In an automated way, so it's not actually changing and then we can decide if that's sufficient or do you want to do things more frequently or We handle more more use cases. It's we're not trying to I think this tool is trying to like change the world That's a first step But it's just trying to take you're trying to save Adam some time Right now just we just do rebuild every two weeks I would love for it to be more frequent, but right now we just I do a rebuild before It's all by hand. So I have like this really It's really shoddy script that maps out the dependency of parent images the child images and I build them from up in the order Yeah Cool Yeah I mean, I think we should be generating as often as physically possible, but it should be entirely on the Users end when they want to pull it Whatever Well, and every time a rebuild happens it lands in the canada registry Time cadence So we could if we want to be doing rapid rebuilds for the sake of testing whatever Religious Right now it's every two weeks we can do it daily And I don't think I don't think a lot of users would be super excited about every every time they turn their computer on they have to Hold down Anyways, we're probably out of time and I think we