 Good morning everyone. Can you hear me? Woohoo. Well, thanks for ditching out on keynotes if you were even there My name is Robin Bergeron. I'm the community architect for Ansible I organized this fine track with a bunch of amazing volunteers who wanted to talk about the cool ansible stuff they're doing so I Wish everybody wasn't in keynotes because they probably want to hear this but oh well anyways These are the fine people who work on a thing called software factory At red hat. This is Maria. This is Alan. This is Heichel and they're going to talk and I'm going to hope that we don't have a stampede in here in about 15 minutes when everybody else comes over Yeah, so good. Thanks for joining us and I'm gonna stick This over there and I guess this is the thing you ask questions into so yeah So I'm here with my co-worker Salon Pevek. He's an engineer manager at red hat He works with the RDO team and also the upstream stable branch for OpenStack and Heichel Gamar and He's an RDO engineer has worked for Fedora with a Fedora and Centos as well community So lots of experience with upstream communities Maria is product manager for the software factory and Lots of experience with product management other companies before joining red hat So let me first introduce what this RDO. This is our use case RPM distribution of OpenStack, but it is also a community of OpenStack engineers Helping you to deploy OpenStack. That's our main mission And in order to do that we are packaging lots of upstream projects. It's over 200 packages and it's growing So what is in the package? So in the package we have RPM where we build from upstream sources and then From the other side we have spec distribution Specification where we define building steps and dependencies and patches in order to get this working We need to have a system which It is basically source code this spec file. So we need to have version control system called this git on a public platform, it's community oriented and It builds on the same tools that upstream has like a great reviews and then on top of each changes We do automated testing and validation so for contribution so that CI systems so We follow all these upstream changes as they happen on the master branch and then we validate each change So that's in order. We can release as close possible to the upstream. So then there is any change It's immediately producing the packaging So just to give you an overview of what the changes would last cycle we had 2000 contributors merging 10,000 of commits in the hundreds of projects Hundreds of commits per day. Just now alone was 1,000 commits and that was just a short cycle. Okata was five months instead of six So as I said spec file needs to be validated after each upstream change If you are waiting for the release, it would be too late So if you wait for the milestone release, that's like already months in the in the cycle And then it would be too much to catch up So you need to test all the dependencies as well So integration testing to happen on each change again because it can have ripple effects So test platform must be able to handle all of that. So how we are doing that Hi cool. Now it's your turn So now let's roll up our sleeves and see how we do an RPM factory So we're building upon some earlier tooling. So the first one is the Lurian. What's the Lurian? The Lurian is a tool that builds up continuously Pay a peg RPM packaging from upstream sources. So it allows us to detect the integration issue very early So it can also generate the food package repository that we use for integration testing And it acts as what we call the master RPM repository and If you would like to know more about the Lurian, which is a general purpose continuous packaging bold tool you can you may want to look at this blog post and then to Automate all the tedious packaging tasks we have we use the radio packaging client Which does things like flattening patches from get also submitting reviews for you and Also bumping version and many other stuffs that are very repetitive So we also reuse the very same tooling that we read at and So community Bay a reddit based distribution uses for building RPM like Koji Koji is like a packaging bold form that builds and store RPM so Koji basically create for you send boxes that you use for modern balding packages and We use an instance provided by Santos project that is called the same community ball ball system that lives in this Sorry URL Then let's talk about How to build RPM factory out of software factory? So software factory is basically a continuous integration slash continuous deployment Platform based on open-stack infrastructure. It uses very same tools Like Garrett for code hosting and reviewing Also, we rely on zoo and currently Jenkins for job orchestration software factory is looking to transition to zoo V3 which allows to use only zoo and ansible to run jobs and ditch Jenkins altogether and Also, it manages project and dependencies when building testing environment, which is provided by zoo and Also, we use another open-stack infrastructure Building blocks that is not pool that will spoon for you job executors and Terminate them for you when your testing jobs are finished And also it provides smart and commit gating using zoo so Which is basically running a set of jobs before merging things and having your CI reporting status Okay, you can see I says, okay, you can merge it see I says no you can't merge this This is broken and stuff like that. Well, if you're familiar with open-stack infrastructure, but pretty much the same workflow and Also, the configuration of software factory is using the everything as code pattern so you don't need to go through web interfaces or anything you can Manage your configuration through software factory as a usual configuration or repository And it also provides a flexible workflow based on reviewing and testing code before prior merging it so thanks to that we aggregated always two links the one from The RPM Word and the one from open-stack info word to build a community platform to build the RDO So we have a software Factory instance that is branded for RDO that also all our packaging repositories and also the very little set of patches we have and Also, it acts every time we have an upstream changes building packages that we test and All the packaging changes are also reviewed through Garrett and the same goes for patch and all is tested through zoo Jenkins not pull n bolt through CBS So the workflow looks kind of like that So we put an Ansible stamp on all the parts using Ansible Which are mostly the CI jobs like the RDO trunk CI jobs The triple O CI jobs, which are also so used upstream in triple O We also have our own packaging CI jobs and also start see very many jobs that I used to test updates So Ansible is pretty used everywhere in our infrastructure. We use a project called weirdo to reuse the same test that are run in upstream gates to test or Aaron to test RDO So you may want to look at this project So let's see if you use cases. So first use case a change happen upstream So we have a dollar and job that is triggered and I would retrieve all The sources from upstream and then the packaging recipes from RDO and we try to build your package out of it well First first is the first case it works perfect Second case it fast what happened when it fails? Okay, we just create a place all the patch on review dot RDO project dot org that will Be open against the proper branch and proper repository just to say hey Maintainers we want to we want to tell you that we can build the master the master branch currently So please fix it and obviously the patch is supposed to fail or otherwise It wouldn't respect the usual green the red green refactor cycles So all the use case we have a change on a stable branch It's this is pretty much the same more flow, but with a slight difference Whenever we have a new change it triggers a scratch ball on CBS a scratch balls If basically this is almost the same thing as a official bolt, but it's kind of not Registered to the database and they are not published it just to see if we can build the package in the actual build environment And it will and if it succeeds it will return a sky score based on that And it will send you a link to the bold lock So the if it fails then the package errors can see why it fails And also the build artifacts are retrieved Into by into the do zoos jobs just to run few tests additional testing But if it's then it is verified by CI Then we we trigger a non-scratch ball if a core maintainers approves it and if it succeeds Then we just publish the package and then we merge and close the review if it doesn't work then we discard the Artifact and then the review gets a minus two score and the patch is not merged obviously So then the user use case is the very rare case when we have this for specific patches Most of the time it happens on master when we have few Issue a few bugs and we want to not to prevent promotion so we can test all the components we try to backports patches or Backports and merge patches temporarily just to move things forward But what we do is that we try we trigger three repository upstream sources obviously The packaging repository and then the patches repository and try to build the package but the thing the nice thing is that we you levering levering Garrett or reviews to Manage a chain of patches or using open reviews We don't merge reviews that allows us to keep a full record of the history of patches For instance, we may have some area destroy specific patches that leaves across multiple release So they get often repaced we want to keep that history somehow. So that's how we do it and The inventing the advantage is that you don't need to be an expert on We are on rebate kids to rebase a patch if it's straightforward You just have to click a button on Garrett and gets rebates for you and also we ensure the quality of patches by testing them by running jobs and This way it makes it easier to deal with patches when you have multiple packages working together For instance, I let's say for a Nova Nova project We may have the Erdio team working on the package We may have some Nova upstream package maintainers coming in to help us maintaining the package We may have random Erdio contributors who say hey, I want to scratch my own age. So I want to fix that package and provide a patch So that's very common Common topic When you do RPM packaging you want to make it easier to collaborate. So now I'm giving giving It back to Maria So we've been talking about our deal But obviously all this tooling that we are presenting here can be used to build any other kind of software The idea to expose our deal here is to show you something that's quickly changing and it has a lot of Commits so for the Okada cycle 919 commits 86 contributors, this is not this is specific for the Erdio project and About 230 builds fail-builds co-option captured by the Lorian and This happens very quickly So obviously if you're using open-stack O1 to start consuming it We urge you to try this these packages are available within 12 hours of upstream release We think that there's better processes and a better way to Bringing newcomers to your community or to your project that you're working on or to your team and the company that you're working on Obviously all these tools are open source and you can use them. We're just giving you use cases of something. That's very Specific to to what we're doing, but you can use that as a baseline to see hey, it works and Obviously we're promoting tools that are part of open-stack infra because that's the way that we've been working And we think this is a really good way to build community So Garrett versus github pull request is also something that we want you just to talk about and then you know Continue to where your community continue working with your teams or also in my case I'm really interested on how I continue to collaborate with partners and customers as we bring in their changes to a particular Release of open-stack and how we maintain them instead of having to backport changes over into Into it so it allows everybody to work on the same platform to see co-transparency everybody Has access to all the changes that have been made you you can declare who can promote changes and who can't and just Reviewing them together It's quicker to unborn newcomers to your project New customers that are coming in in my case partners that want to also work with the customer So it brings that clarity into the work that we're doing together with open-stack and If you want to join us obviously again one more time these are all upstream and open source software factory project IO is the Software factory Version that's targeted to build the actual software factory project But also builds distributed CI which is another tool that we build and we use that red hat with our partners Skydive network and our analyzer tool is built using This same kind of tooling and the same kind of processes and allurean as well And then we have reviewed our rdo project org which is basically software factory rebranded for the use of rdo is open you can go check it out And we are packaging their rdo as well as obstacles which is monitoring tools that we use for open-stack The idea is that you can see a wide variety of projects software that we're building and we're building them using this this way and Then just you know keeping touch with us This is this is where we are and where we're living Do you have any specific questions around the tools that we showed you today or? Any answerable specific questions as you can see answerable was pervasive everywhere here if you use answerable In your organization or with your work We are sure to look at these tools as a way of Building or using tooling and and using these processes to help you build your CICD pipelines and There are no other questions. I'm going to show you a list of other Meetings or presentations that are happening today at Summit if you see one of these that catches your interest You can go check them out. They're in and around here These work are also today So if you like the presentation or want to hear more about it Those are our Way to contact us and like us on the app and thank you for joining us They did not have questions. Do you have any questions? Oh There is sorry Somebody say a question Guess it was clear as my day So great presentation. Sorry Looking through a few things. Are you also looking it looks like there's a good onboarding capability? Are you also looking to potentially extend ansible into more of like a An orchestration tool here as well to do more than what you currently do right now from bringing up your your systems and Make sure it goes up to speed for certain things Um, could you repeat your question? Sorry? I you said using ansible as more of an orchestrator So you're familiar with a an NFVO within NFV, right? It's a larger orchestrator worry, you know on board certain systems you configure systems you yep automate that system Are you using Zool at all? Am I using Zool? No But I'm just looking to see I didn't I saw some capabilities inside of here I think that's something that I that would be the first place to look at What do you guys think and Zool is that yeah? So Zool is just particular for development. It's not for deployment. Definitely, but yeah In this case particular case we are using ansible to drive CI. It's not used as a deployment tool Yeah, definitely can be it's it's not our use case in audio It is used in for example triple low more and more it runs upgrades and stuff like that So it is used in related projects, but just not directly in the our infrastructure Thanks