 All right Good evening and welcome We're going to tell you a little story about CI CD and how it's done In OpenStack and also how you can do it at home with a sort of factory So Fabia and I are going to tell you about some factories a project. We've been working on for about a year at Red Hat We are senior Engineers at Red Hat. The rest of the team is scattered in the Around here, there are a few faces we can introduce you to by the end of the talk So let's start with a quick state of the art of CI and CD how it's done as an industry standard It's a pretty old problem. So there are pretty much tried-and-true methods to do that Typically the workflow you're going to implement if you're doing continuous integration and continuous development is one similar to this one So basically When the developer submits a patch to the code base to the version control system this event is going to be caught by the CI infrastructure It's going to trigger a round of testing and this testing is going to give back Feedback to the developers if the if the feedback is negative Then the developers know that there's a problem with the current version of the code base And they have to fix it as soon as possible if it's positive then it's going to progress further down the integration pipeline until QA tests and eventually is a release for a continuous delivery So this is tried-and-true this works, but this has some problems There are some problems with well with the enterprise of doing a continuous integration First there's a cost to automation automating a build and automating tests is not an easy task Depending on the project and also there's a cost in terms of a dedicated CI infrastructure. Of course you need a CI server to monitor your Contemplation system Also, you will need most of the time a build server something that will build your code You will need test testing infrastructure testing nodes for that Most of the time you will want to do your tests on an architecture that will mimic as much as As accurately as possible your production infrastructure Which might have a cost also you have to maintain all that so it has a cost in a human cost as well And finally and more importantly, there are some flaws in the workflow itself for instance Commits in rapid succession Will make your CI test obsolete as they run because the CI is not running on the latest version of the master meaning that your tests the test results are not really relevant anymore and The biggest problem with with this workflow is that actually you can end up with a broken master temporarily and you might not even knowing know it because between the moment when you submit a New patch to your code base and the moment you get the feedback from your test there might be some time It might take some time to run the test. So in this in this interval of time You don't know for sure the state of your master That means that people would check out your code base at this moment might actually be working on the float version on the broken version and not know it So in a large-scale Projects like opens like open stack this get this this workflow doesn't scale It's really not possible to work like that to give you an idea and to refresh your memory This is these are both ballpark estimations for the project, but basically they remain true even if they are not accurate Open stack represents About 43 projects each of those projects have their own subsets of deliverables and all of those deliverables are Strongly tied together in deployment and in testing. This means that if one of the masters for this project is broken Then all of the other projects are going to suffer about that. They're going to be impacted by that So it's not but it's not possible to have to use a workflow like that that allows for broken master temporarily Also, you have more than 2,000 contributors at least for this last this last release cycle This means that you can't expect all of those contributors first to be able to test in a correct conditions There are patches there. I mean they're not going necessarily to have the infrastructure to test them to test a cloud and also you can't be sure that all of them are going to abide by a Quote quality standard and things like that So you you can't trust a direct commit to your to your code base by your contributors that you don't know and finally The most impressive number concerning open stack is a sheer amount of patches that you get for for really for a development cycle It amounts to about 400 patches a day Each of those patches can trigger up to 25 jobs and some of these jobs can take more than 30 minutes to execute so you really have to have a CI infrastructure that is able to parallelize those tests and also to absorb the sheer amount of of data that it represents So open stack still needs to do CI obviously So they came up with a new way to do it a new workflow that we're going to introduce now So the open stack way is a little bit different from the old way The main difference is that it's realized on the on the scoring system So there's a new element that is introduced which is a code review part. That means that instead of Submitting code directly to the version control system a submitter will first Submit the code to the code review system The code review system will need a score a given score to make the patch progress further down the the integration pipeline this score consists of two elements first Sorry a value given by Feedback from the check tests pipeline which are which are tests that are run as soon as a patch is submitted or modified and There's another part of the score that is given by what we call core developers which are developers that have Special rights on the project. They are usually experienced developers who are able to to make a patch progress and Merge that merge patch in the version control if those two Elements are met that is a positive feedback from the check tests and a positive approval from the core devs And then as a patch is finally tested in the gate tests the series of gate tests which are So that's the kind of tests Against the latest version of the master of the code. This means that you are sure that if other patches Have been integrated In the meantime This patch is not going to break it It's going to be tested Making sure that your tests are never obsolete and after that if you get a positive feedback from the gate tests You finally integrate the patch In order to implement this workflow The open stack infrastructure consists in in this well, it's a simplified view of the infrastructure, but it focuses on the services that actually implement the CI so So not worthy services are Garrett Which is the name of the code review system if you are a contributor to open stack you are very familiar with this one Then as the pipeline as the integration pipeline management is done by Zool and Zool communicates with Jenkins to start jobs Jobs that are going to be run on slave nodes that are managed by node pool No pool is a service that spins up VMs destroys them when when they're not in there anymore so this This this workflow in this infrastructure has some drawbacks because it's it brings a longer workflow and extra workload on the core developers, but So the advantages are really worth worth it because we we address all the problems that we listed before The main advantage is that this time you guarantee that the master version is always okay. Well According to your test of course according to what you test But your master the master version of your code is always okay You never have the possibility to have a broken master also Thanks to node pool since the test Saves are launched only when you need them only on demand your infrastructure your infrastructure costs are reduced and Also Zool by a gating commits ensures that the tests are never obsolete and The best part of it is that everything is freely available and you have also all the recipes to deploy it by That are freely available on the repository of open stack. So that's pretty cool You're probably thinking wow, that's that sounds pretty nice. I'd like to have the same system for me. How can I do that? Well, you can The thing is that the only problem is that the recipes from open stack are very custom tailored to to their own infrastructure, so it's going to take a lot of work to actually adapt it to your own to your own to your own Needs, yes. Thank you And actually it might be even faster to start from scratch So that's that's a bit disappointing And you're probably thinking now. Well, I wish somebody would do it for me and which is good because we did and That's called software factory and Fabien will tell you more about the main features of the after factory So, thank you, Matthew. So So software factory is a all-in-one CICD system that implement a workflow similar to the one used by the open stack community It's used the same tools that are deployed by the infrastructure team of OpenStack and And we are going to talk About it. So we are going to introduce software factory By making an overview of each component included in it We are also going to show you how it is easy to deploy it and finally We are going to show you how we manage component configuration in software factory So here is a global architecture of software factory So all the components you can see in this diagram except the slave not not on the right can be run On the same on the same node. These are services and in fact software factory is a platform That is designed to be run on a single virtual machine So First we have we include five main components in software factory or get it get it to us project and perform code review get web to allow an easy browsing of code repository and Jenkins as a job manager. We also include Two other tools that has been created by the open sub community Zool as a job scheduler to enforce the CICD pipeline and not bull to To start virtual machine as a test environment on an open stack cloud So as you know Get it is a nice tool to us repository and perform code review, but it can be a bit complicated to configure it and And adapt it to the open stack workflow. So that's why in the software factory Get it comes pre-configured With the review label and access control list. So we define by default three review label to receive a scoring from a core reviewer the CI system Zool and Jenkins and Patch submitter. We also define a pre-configured ACL To enforce a workflow a rune the scoring is a workflow similar to the one used by the open-stab community and We also want hook to tie get it with Rennmine. So for instance when a patch submitter create patches and Include a link to a ticket On a commit message Automatically get it will mark a ticket as in progress when a patch is first submitted and Automatically also get it will close a ticket when a patch is merged. So I said before we also install Zool in software factory So Zool come pre-configured in software factory standing between get it and Jenkins And Zool will be responsible of the Etsy state of project hosted in get it We define five pipeline by default and the two of them are worth mentioning for performing continuous integration The first one is a check pipeline that defines a job that must be run When a patch is first submitted or modified and the gate pipeline that defines a job that must be run prefer prior to the merge of approved the patch and It is worth mentioning that Zool is really smart in this pipeline the gate pipeline Because it will always rebate a patch against the master before performing a job and This avoids the phenomenon of broken master Matthew talk about it before and Also gate in the gate pipeline Zool will be smart enough to speculate about Job result in order to run a job in parallel and we also Define by default three pipelines that are more designed for a continuous deployment So first the post pipeline that defines a job that must be run when patch are merged The tag pipeline that defines a job that must be run when the tag is pushed On the project code base and the periodic pipelines that define the jobs that must be run on a daily basis But this kind of job that will run in this pipeline in those pipeline With those pipeline you can do stuff like I want to push the last version of my project on the Production server or I want to push the last version of my project on Remote repository like Piper, but in order to do that this kind of job may need Specific credential to access privileged places. So that's why in software factory we include a specific Jenkins plug-in To allow a user to store credential and reuse credential during jobs about the slave node So this is often difficult to manage slave node to perform CI and sometimes slave node are the bottleneck to perform an efficient CI So in software factory if we define two way to install slave to define slave So the classic way with the Jenkins is to define static slave long-running slave and we also include Not pull so you can configure your slave via not pull in software factory Using not pull you can have a dynamic pool of slave as Virtual machine as slave on an open stack cloud and it is worth mentioning that in software factory We pre-configure not pull for you. So all you have to do is to specify your code provider credential in the software factory configuration file and and that's it and It is also worth mentioning about the benefits of not pull. So first job scalability You can run as many job as you can run virtual machine on your cloud provider Also, not pull is really smart about resource usage So it will only start a virtual machine as slave only when needed when a job need to be launched and Only for the duration of the job So this is really important because it allows you to save a lot cost on your cloud provider and also Not pull will only start a virtual machine the way you define it using the base image you define using the provision script you define so Thanks to that all your job are guaranteed to run on us on the same well-known environment and From our experiences we used to use static slave to test software factory and We had a lot of trouble with them. Sometimes slaves are not consistent and raise a false negative as test result and since we use not pull All our work tests are really Stable, so this is really really great improvements to use not pull and in addition to the core We also include collaborative tools to To help software development teams. So first at our pad Collaborative text editor really useful during during meeting our brain storming session pasty a rapid copy pasting system and Renn mine as an issue tracker and backlog manager and In order to integrate all this component well together Smoothly together. We created additional component like manage SF that are that is a rest API CLI and a dashboard that allow a user of software factory to easily create and delete Project and so far factory to easily manage project memberships and also do things like configure grid replication for its project and it is worth mentioning that manager SF will propagate project operation on Get it and Renn mine transparently for the user We also created seaut That is a SSO system That unifies the authentication across all components on software factory and finally we Integrate the top menu on the software factory UI that allow user of software factory to easily access each component including software factory by one click And you may say, okay, that's fine That maybe this is complicated to install this software factory and I will say no not at all This is really easy because you just have to spawn a virtual machine using the base image image will create for you and Adapt the configuration file to your needs then run the configuration script and there we go You are ready to us project run CI CD jumps allow open stack and It is worth mentioning that we also created a template to is more again this This step and finally about the component included in software factory We create by default a project That is in fact repository is a configure repository and the purpose of this configure repository is to host job configuration test description And So thanks to that user in software factory are able to provide via patches and get it Configure job configuration and test environment configuration for their project on get it and thanks to that we are and And these patches will go through the CI CD workflow of software factory Before being in order to being validated merged and applied inside software factory So now we are going to show you Concrete example of software factory usage and I will give the talk to Matthew Thank you Fabien. So yes, let's see how we can implement the CI workflow inside of software factory Open stack style CI workflow Let's first connect to the dashboard of a sort of factory So as you can see it gives you a general view of what's going on in software factory as Fabien mentioned before There's a top menu that allows you to access all the services that are packaged within software factory You can also see the state of the tests that are currently running in which pipelines and for which projects You have also a list of the projects that you have access to and you can see at a glance The state of the reviews on it if there are any open reviews on it even if there are any open bugs as well So let's create a project All of the things I am showing now can be done from the command line by the way But this is less visual so we took some screenshots from the website But all of this all of this can be done with with a command line interface So creating a new project is a very straightforward There's nothing too magical about it The thing that we should mention is the fact that you can declare a project as being private Meaning that only a few selected people will be able to contribute to it And also you can import an upstream repository to populate your project Now that you have a project you're going to want to add users to it to work on it So pretty straightforward as well You have a list of users that use sort of a factory and you can see here that you can define the roles that they're going to have within the project Since we are implementing an open stack style CI workflow we also mimicked The vocabulary that is used by software by open stack Meaning that the ptl project technical leader is actually an admin for a project Core developers are the ones that are going to be able to approve a patch to make it to merge it eventually in the in the project and Developers are the people who can who are allowed to To commit to submit code to the code base Once you have you have users in order to start working You're going to want to Thank you. You're going to want to set up the test and the testing environment So we're going to start with setting up the slaves to not pull it come It is done in two steps first you have to be to define the image you're going to use for to run your tests So it's pretty easy actually all the configure all the configuration is done in a in a derivate derivation of the yaml language So it's a very human readable. It's really easy to maintain and Basically defining a slave is as easy as that. There are only a few lines You only need to name your your image obviously the slaves you're going to use as a base image. You're going to deploy The private key you're going to use with Jenkins to manage the slave and also if you want you can Terminate finish the installation and the setup of the slave with a custom customized script Once you have defined Your slave this way all that is left to do is to declare it as a label in the configuration So you can actually mention it in the job the job definition later on so this one is going to be called the best interest seven We've done that so now we can define jobs for our project. We're going to do just that Again, it is we use the template language called the Jenkins job builder Which is a derivation of a yaml? And it's very straightforward as well. We just give a name to your job You Specify what you want to run during this job So if you have already automation on your on your project, you will just call the automated script to run the test on it And finally when the job is run Any artifact that this job is going to to generate can be published with the use of other scripts So in this case in this case, we're going to use fetch my artifact and do the shift upload Which are defined but not shown here and also you can see that this job is going to be run on the node We defined before the best interest seven So last step to configure a job is to add it to the pipeline in Zool So again, it's super easy You have the name of the project and the pipelines that are used by this project in that case We have only two pipelines the check and the gate pipeline So as we said the check pipeline is the one that defines the tests that are going to be run as soon as a patch is submitted or Modified and the gate pipeline is the one where the defining the test that are going to be run on the patch When it's about to be marked so you can see that those Pipelines are Independent from each other. You can define anything you want. We are adding it to the two pipelines in that case Cool. So we have a test environment. No, we can start working to start working we have to track our issues and list our User stories for that we use red mine as a issue tracking system and While it's pretty set forward, you know Issue tracking is a pretty much the same everywhere In that case you can see the master backlog of the project of the factory. So how it lists Tasks to do for a sprint So as a developer I'm going to assign myself a task I'm going to start coding on it do a patch and submit it to the code review So here's what it's going to look for a core reviewer. So for example Fabien is going to check out my What I've done So he can see the commit message people who have been working with open stack are familiar with that you can see The files that are impacted also very important You can see the feedback from other developers from other core developers and the scores given by the automated test feedback Then you can see the modifications that are going to be done by the patch You can see that this patch is that it's ninth iteration. It's been modified nine times It's still not enough. There's still something to correct So Fabien is going to comment on on a part that he doesn't like And when he's done is going to give me a score So minus one because he doesn't understand my the magic of my coding So while this happens The rule is a task with paralyzing the jobs and managing all the pipelines. So this this screenshot comes from the open stack instance of Zool And you can see that There are many jobs that are running parallel in many pipelines So you can see the check pipelines that we mentioned the gate pipelines the gate pipeline Sorry and the post pipeline that we mentioned quickly, but it's the one that is that launches jobs Once a patch is merged You can see also very quickly When a patch when a test is going to fail Well a set of tests is going to fail because it's shown in red The green ones are the ones that are working and going going well You can also see the dependencies between the patches. It is very important But this is out of the scope of this presentation And if you want to know more about Zool, which is quite a beast and very interesting to know how it works You I advise you to read Fabien's blog post on the RDO's blog About it's called a deep dive into into Zool and it's really interesting It's got a lot of documentation about the inner workings of Zool So once the tests are run You want to take the output to see well first if it worked and if not where where it didn't work So there are two types of output in software factory There's a classic job output When it's run that is shown in Jenkins. You're probably very familiar with that because Jenkins is pretty much an industry standard But also as we said we can upload artifacts any kinds of artifacts to to an object storage like Swift and And in this example you can see the artifacts that are sent after a test that is run on software factory and What the artifacts that we decide to send Actually the logs of every service that is running on the software factory image So it is a very invaluable tool for debugging because when you have a problem you can just check anything that way Those logs you don't have access to them to Jenkins Because you only have the logs from the job So we've been doing CI so far. It's been pretty good. No, we want to push it a little more and do some continuous delivery While with Zool, it's pretty easy All you have to do is define a new job that will be Uploading the new version of the code to a production server and then run a deploy script The thing is to connect to this server you need to authenticate yourself So it will be pretty much that equates to have your key in the clear in your job So in order not to do that We use the credential binding plug-in in Jenkins as we mentioned before which allows you to to access a secret in a secure way Once your job is defined just like we did the last time You just have to add it to the pipeline post and it's going to be run after a merge So that's it As you can see Doing CI in a better way In the open stack style It's pretty easy with software factory. So we hope you're going to give it a try You can contact us on a free note and the mailing list and of course check it out on on its website So software factory dash project.io and if there are any questions, we're going to take it now Yes Great. Thank you. Great presentation In the initial traditional model you mentioned that takes about 30 minutes or so to run through your patch to Deploy, I wonder if you had kind of similar Data on your software factory model like how long it takes for your patch to go through the process and deploy For the standard model we shown at the beginning I think like there was one slide that said like oh it takes about 30 minutes No, no, no, in fact this is the job that run in in order to test open stacks in open stack so in in Zool in the the instance of Zool deployed by open stack some jobs will will will take more than 30 minutes to run so So that's why we mentioned that just to show that In in so far in an in open stack Some job are really long to run and there is a lot of job job to run So this is a huge amount of work for the CI So this was in order to introduce the future of Zool But maybe that you want to say more about that not really but what I can add is that the combo Zool plus Not pull really it wipes you with parallelizing stuff because you can run as Fabien said before you can run as many jobs in parallel as As VMs that you can run on an open stack cloud. So that's really much your the tenant is the limit and so That's pretty that's pretty cool You won't be able to reduce the time of your test if your tests are pretty long Well, that's your problem. We can do that. We can do anything for that, but you can run them in parallel. So it's you gain some time So how pluggable is the architecture if I want to swap out Garrett for github? And I don't want to run an open stack cloud. I want to run on Amazon so actually All the architecture turn around github so in order to move out github and use github it won't be possible actually and to run software factory on another cloud platform actually it is possible But some component like not pull won't be able to start to spawn virtual machine on Amazon. So For your slave you will be able you need to use an open stack cloud But maybe in the future. I don't know The fact is we use tools Developed by the open stack community for the need of the open stack community So this is difficult to use them on another platform on the other hand. We have In the roadmap we plan to make it make software factory more modular because we don't really like your red mine and we'd like to use to be able to use other ticket ticketing system especially since it's also a Blocker for some for some teams, you know some some teams already have an issue tracking system And they say no we don't want to migrate So we're just going to do our own way even if it's not as good And so we'd like to have the possibility to use any ticketing system and also add other modules to the system so if I have a means to if I'm using Ansible or a puppet or chef to configure my My actual application all my server stack How do I plug that it if I have you know my code I wait means to reproduce my environment and some automated tests Have you got documentation about how to plug all this into? Okay, so all the job you will need to to run in order to reproduce your environment Can be included in the not pool image you will create you need to create so with not pool you can Say I want to use a base image like the Sun 2s a cloud base image and After you can specify a script so you will you have to write a script that will define Okay, I need to install on Siebel I need to in define some stuff in the image and after not pool will take a snapshot of the image He will create based on the base image plus the script you provide and after it will use this image to spawn them and run your job in it, so So I don't know maybe it is it to your question or no not really Okay Yeah, so I really like it I just have a question about the the entire workflow So one of the things I really like about github is the ability to create Issues or tracks and tasks take those tasks create some code then take a Pull request attach it to the task and then have Travis automatically kick in the test and Provide feedback so the all day entire workflow can be seen from the issues And so what I'm wondering is software factory does it once as a Garrett patch was moving through the workflows those workflows fed back into red mine and Given like so you can go to one place. I want You can see the the workflow flow through is that possible. Yes, absolutely if you if you When you have the ID of a ticket to define in in in red mine you can put this ID in the commit message of a patch and The the state of the ticket will react According to the life of the of the patch in get it and also get it will post Information into the ticket as comments. So maybe it mimics a bit the workflow you describe What's negative what's next on the road map is there anything like ELK stack like the CIA in front team has or What are things are coming up for some for a software factory? Um, like Matthew said, this is a customization We will for now we have a static component in the software factory So we want to provide to the users a way to a way to specify. Okay, instead of red mine I want a tiger for instance And For now that's pretty much it's a we really need to have feedback from from user about what they want and And after we can add up our workflow. Well, personally, I'd like to see more Authentication schemes and protocols supported in seos by the way seos season is a standalone project that you can check out as well It's it's helpful to unify your authentication amongst several services and right now It's it's supporting only well only it's supporting a github authentications were oath also launchpad authentication But I think that to get more Entries in the industry world we should support industry standards like Samo and Carbureaux that we are missing right now But I'd like to work on that afterwards And I like to know is software factory the free software Well, why don't you answer that? Because Tristan is actually one member of the team Well, it's a free software. So it's like a community driven. So far the community is pretty small But if there are needs to be addressed like a feature like you mentioned from the open-stack of rights It's something do it and it's worth mentioning that In order to develop software factory we use software factory. So this is a a good point because This is a the workflow. This is a good workflow because it helps us to keep it stable on the master branch Do you have any other question? Okay And so what do I say need to run it how many I need an instance I need one VM I need To deploy software factory. Okay. So to deploy software factory in fact at each release we create a Pre-built image of software factory with all components including it So you just have to upload this image in glance if you use open-stack start a VM based on this image and After you log in the image in the virtual machine using SSH and you start the deployment script and we have same default configuration So you can as soon as the script is finished you can connect on Using your browser on the UI and so far factory. So this is really easy and it take I don't know maybe 15 minutes 15 minutes. So so yeah Yeah Yeah But I think this is enough because all the the load is generally on the slave and you and you may have to Configure additional node. We are not pool. I guess to to manage all those jobs on the main node You don't have so much load up and if there are some tests that is related to hardware like the storage driver Is the node pool can handle this? I'm not really sure. So this is maybe a complex use case We don't use for for our our needs. So I don't really know if this is possible Thank you So all these system they must have some states like Garrett all the Review the codes all the job Jenkins. They stay on that machine Yes, absolutely. Okay. Is there default configuration that we can move this out so that if that machine Fall we don't lose everything Okay, so first yes, there is a backup system backup after system. So So you can configure a regular backup and push on a swift So if you're not crash you can spawn another one just in Restore the configuration and you and it will it will work So that's it. Okay, so thank you everybody