 Okay, welcome to this session. This is Erdio Continuous Packaging Platform Lightning Talk. So my name is Eichel. I'm Erdio Release Wrangler. So I'm here to present us how we package OpenStack Master. So if you don't, if you have never heard about Erdio, Erdio is just the RPM distribution of Danila OpenStack. Also a group of dedicated stackers and many, many stuff. We have 250 packages maintained and growing and it's not counting the dependencies. So let's start by defining what's a package. A package takes upstream sources. It also includes spec file, which are basically packaging recipes. So it details the building steps, the installation and removal steps, define dependencies and include the patches. So as you see, we've taken things from upstream and we have our own things. So the package is a person working on the packaging recipe, the spec file developer. So we spec file are maintained in a version control, what we call diskit. And also we have peer review. We automatically test and validate all contributions and we see later that it uses the same platform as OpenStack and also has merged management. And the thing is packaging OpenStack can be challenging. So we have to follow upstream changes very closely to validate our spec file against them. So OpenStack is very, very active project. We have like 2,000 contributors merging like three, more of about 40,000 commits across 6,600 projects. So it's kind of a very, kind of a heavy project. And the thing is that's very strong constraint for us package because, well, we used to package things but we used to package software after they get released but with this space we compelled to work very closely with master and with that many project, with very few people, it gets really hard. So one of the things that makes it difficult is the strong dependency between packages like Nova relies on Cinder for block storage and stuff like that. So when we have a change in Cinder it may impact Nova and Nova changes may impact for instance 12 or Sahara because they rely on Nova. So it makes things difficult for us. So we don't have just to both packages we also have to integrate their integration together as much as those upstream. We have to do it downstream too. So we bought what we call the RPM factory. So since we are using OpenStack and trying to integrate with upstream we decided to rely on the same tooling as upstream. But one of the first component we had to build was Dolorean. Dolorean is basically tool that tests the packaging recipes against OpenStack upstream master to detect packaging issues. I'll come back into detail later but thing is when we have a change upstream we rebuild immediately the package to see if it bolts. If it doesn't bolts we will fix it. And then after we build it we will run CI to test integration with the other packages. So we can fix things very early. It can also generate the full repository. So we have repository snapshots to test at the moment the whole OpenStack master as package. So this is used for instance in triple OCI and also pack stack CI. And I think that OpenStack Ansible is planning to use it or maybe already are using it. If you want to know more about Dolorean I'm sorry because this is lightning talk. You have a full blog post at this URL. I'll try to post on Twitter the slides. And we also developed a tool called AirDuePackaging which is a client that automate most of the tedious packaging tasks like updating versions in spec file, retrieving the sources and many, many stuff. So it also helps flattening patch changes for you. So we have to, since we are RPM distribution we have to rely on the same tooling but the other RPM distribution like Fedora or REL. So we use Kujie which is a building infrastructure developed by Red Hat for its family of distribution. Kujie is basically a sandbox environment to build things. So that avoids like any side effects during building packages. So we use it so we can ensure that we are building artifacts during master repository but as close as artifacts that will be shipped in released versions of AirDue. And for that we're relying on the instance setup by the CentOS project called Community Build System. AirDue has a particularity to integrate within the CentOS ecosystem. So if you're a CentOS user, AirDue is a perfect fit for you if you want to try OpenStack because it's been natively developed on and test on CentOS. And well, above that we have a software factory. Software factory tried to reproduce as closely as possible the OpenStack infrastructure. So it's a continuous delivery platform that use Garrett for code review. The Zulu and Jenkins for job orchestration. We are looking to transition to Zulu v3 so we can get rid of Jenkins as we do in OpenStack only infra. And well, we also use NotePool to spawn job executor and terminate vame. So basically just native OpenStack stuff. We also have the same smart gating and commit gating policies. And software factory can be deployed in your own infrastructure if you want to develop your own project as a software, it's a software factory as the name says. And all the config is done by code. So very useful and the workflow is quite flexible. So we aggregated all these tooling together so that we have jobs that will continuously build upon and get changes upstream, new packages. And we also have the platform to review packaging changes. We have CI relying on Zulu to test all of this packaging. So it looks kinda looks like that. And as you see, we put the Ansible logo where we use Ansible. Some of you may have heard about the project, the OpenStack project called Weirdo which we developed to run upstream CI testing against our packaging. So you can run OpenStack upstream testing in your local infrastructure if you want to using Weirdo. And then how we do test change happening in Upstream Master. So we have two git to retrieve, the upstream git and the packaging git. And then Dolorean try to both things. If it succeeds, it gets published. If it fails, then we will submit the place or the patch against the packaging repository so that maintainer gets notified that, hey, the package fails to build against Upstream Master, please fix it. And CI is supposed to fail so that we try to, so we can verify that it is not a false positive. We do also the same thing with stable branches. So we try to check stable branches. We are not waiting any release just to test also stable branches despite their very, very little breakage on stable branches due to the excellent upstream CI. And for these two specific patches, we do the same. The specific thing is that, or despite Weirdo doesn't have much patches, we try to manage patches as a chain of open reviews on Garrett. We're not closing them. That allows us to keep the full history of the patches across the versions. The rebasing is easier. If it's straightforward rebasing, you just have to click the button and it gets done for you. And it makes it easy to deal with patches with multiple packages. But that's the way we handle downstream patches. Despite currently, we have one or two downstream patches which are mostly integration ones. So return of experience. AirDew has like almost a thousand commits for the Yo-Kita cycle. Sorry. 86 contributors. And we code 230 bold failures through Doloran. Yeah. And I'm almost finished. And AirDew and Yo-Kita were available 12 hour after upstream release mostly due to publication delays. But that's it. So sorry. I have to make it a bit short but you can find any references you want about Software Factory on those links. Is the reviewer.project.org which is a platform I'm speaking about if you want to check it out. And if you want to discuss with that, just ping number 80 on IRC. I'm on many OpenStack channels in AirDew. So thank you for attending.