 Good afternoon everyone. This is building the rack stack packaging from upstream open stack I'm Philip Schwartz. I'm a senior developer for the build release deployments team at rack space I'm Kevin Mitchell. I'm also a developer at the build release and deploy systems at rack space All right, so we have a couple objectives for this as a whole We're gonna talk about the evolution of our packaging of open stack inside a rack space the process that we're using for packaging and The future of the actual process This year is a just Overall view of what our team does and the different pieces that go into it from a high-level area in this actual talk We're gonna talk about a couple things. We're gonna talk about our interactions with git in order to build packaging our ply tool that was created by a racker that is designed for doing patch management and Also the packaging system itself What we're not gonna cover in it is any of our deployments and actually are the targets of the deployments We're only talking about the object of what we deploy So let's go ahead and get to the packaging context itself All right, so we started off with creating Debian packages We actually had one repository on our internal github that had the metadata directories for all of the packages that we built and then we just copied out the say the Nova package Plunked in the Debian directory for that particular Debian package built it locally installed it using the Debian package manager Used a local app repository. That's about it Of course, we had some problems with that in that it was rather disruptive. So then we went on to the next page With this we moved to a more monolithic package It was something that contained all of the projects Nova glance at the time quantum and melange All configurations which were puppet based at the time. It uses virtual environments. No site packages at all So it's completely self-contained. They were built all on a single Jenkins slave and Were copied around from system to system. It's a horrible thing to think about Everyone tells you do not move virtual environments around but there is a little bit of benefit to it It's still maintained using local storage We also move to using torrenting to actually deploy the package in a staging method to all of our actual systems along with HTTP for downloads to and We'll move on to our build flow itself Which is kind of interesting when you see most code most times you see people pull down an actual repository They'll make changes to it. They'll use it for deployment or anything. That's necessary We go a little bit beyond that we use the tool ply to merge a local repository that stores nothing But patches to get inside of it as part of that flow We would then branch and tag based off of it build the in the actual virtual environment Run complete unit tests on it and compress the virtual environment and store it off as an artifact to be used later So why did we move virtual environments around simple answer allowed for finite testing? We create the virtual environment Then we take the virtual environment and we run the unit tests for that product So we create a virtual environment for Nova. We create a separate virtual environment for glance that avoids conflicts But then we can take that single virtual environment run the unit tests on it Validate that all the unit tests pass is kind of a last sanity check that yes, everything's working fine And then we either deploy it into production or we employ it deploy it into our Production staging environment or we you know run Various functional and integration tests to validate that yes the unit tests aren't lying this actually works So Creating this monolithic package actually is in pretty interesting process. It's done completely automated via some code It actually grabs from the local storage all of the virtual environment artifacts pulls down a configuration repository It creates an actual versioned and compressed archive that we then move to the storage server it's basically a large tar ball of nothing but virtual environments and It works pretty well at the time This here is a base flow of what we actually do with the actual build. I'm sorry for it being a little blurry It's not on the laptop We start by creating the actual branching of the repository and Brent doing tags and everything is needed We then build the actual virtual environments running full unit tests on them We build the deploy artifact and then we store the actual deploy artifact into the storage server Sorry about that so this is basically just the same thing in text as It was from the scrub before so there are some issues with this current system that we were using and This is where it is. There's two main ones that we've come into all Projects must be upgraded together. You can't upgrade them separately. It means anytime we do an Actual nova deployment. We're doing a neutron deployment at the same time glance is being deployed So we always have to test everything and verify that they're working We also have an issue with the torrenting that we were using where everything was limited per cell due to network restrictions So when we would actually do the staging we would copy the package out to each system Cell by cell which over time could take a long period of time Especially when you see that in rack space. We're currently running upwards of 144 cells in our production environment over six regions So where are we going? Well, we're gonna get rid of this monolithic package We've already started the process. We've pulled neutron out into its separate package And just to be clear that's a virtual environment and configuration, but none of the other systems So it's not like our monster tire ball and we're in the process of doing that for stack tack which is a monitoring tool and Ironic and we intend to do that for glance and then eventually will actually pull nova out separately as Part of this we're also changing how we're storing them and how we're deploying actually staging these packages out We're moving to using Swift to actually store the object and we're using some custom ansible modules to generate Swift temporary URLs shard the actual artifacts across Swift containers and then using those temporary URLs with Ansible to pull the package down to each of our systems removing that entire restriction of one cell at a time Well, this is the basis of the build flow and what it's actually changed not much has changed here But you're able to see that we're now building Separate package separate tar balls for each of the actual projects All right, and what do we use for actually building these packages? So the primary thing we use is a Jenkins server. We're actually in the process of setting up multiple Jenkins servers because well Jenkins doesn't really scale We use static slaves at least originally to do this But we've since moved on to setting up Docker containers So we launch a we launch a Docker container and then we run a tool inside of that Which actually builds the virtual environment tires it up puts it into an artifact store and we do and another one that Downloads that runs the tests another one that downloads that virtual environment sticks it into a monster tar ball and uploads that to an artifact store So the actual tools that we use we use a ply if you're interested in that It's up on pi pi as ply patch and that was written by one of our rackers We use the Python standard virtual M and we use pip and of course we also have an internal build tool that we use That's what 4,000 four and a half thousand lines of code 18,000 lines of unit test we want to make sure that it actually works Yeah, we've had we've had a lot of people and they've seen the code base look at us And they think it's completely crazy with the amount of tests that are there But how we've seen it is this is a tool that we use and we depend on for our production environment to verify that what we Build the virtual environments and these packages and any way we actually do that will always work So we're guaranteeing that when we build one of these packet these actual packages that the virtual environments We'll always be built in the same manner We'll always function the same and that they run exactly the same it the tool is completely Python based again It's just over four and a half thousand lines of code with 18,000 lines of tests to verify and control What it's doing and that's a mixture of unit tests and functional tests and we use for any base configurations for it Yeah, and this tool handles more than just the packaging So we have pull requests internally we maintain neutron Nova, etc. Repos our internal GitHub And then make pull requests to that and this project is also responsible for running the test on that so that means that When we run the tests after generating the package We know it's the exact same tests that were run when the developer uploaded their pull request in the first place So that gives us better confidence that it's working So the way the tool works is it's based around a Jenkins Environment Jenkins workspace, so it just runs as a Jenkins job In fact, it gets the name of the Jenkins job uses the any configuration to figure out what it's supposed to do So the build script for these is dot slash bootstrap. That's it We use get and ply for Maintaining upstream poles, you know pulling stuff down from upstream merging it in with our local patches Putting it back into our internal repository We do get branching and tagging for leases. We use virtual environment creation using virtual infant pip This is an interesting point. We have We have a neutron client right on our internal GitHub and we want to use our version of it rather than the upstream version of it We may have had We may have found some bugs and fixes fix them But haven't had a chance to contribute them upstream or maybe it's something that's specific to our environment so the tool that we have actually parses the requirements files and finds these overrides is what we call them and We'll install the overridden package rather than the upstream pi pi version We use run test that sh to run the unit test. This is an unfortunate limitation Talks right now doesn't allow us to supply our own virtual environment that that work is in progress But it is not yet complete. So we have to use a shell script that lets us say hey use my virtual environment We compress the artifacts using you know tar gz stuff that you would normally use create the packages Create a monolithic package. Now, of course, we're creating the the project level packages So the next step that we're going to from here is we're actually are open sourcing this tool It's a tool. It's gonna be called striker. We're gonna be making it available through a stack forage project There are some changes that are coming down as we're doing it. It's still gonna remain heavily tested It's still gonna be Python based we're moving to a more standard YAML configuration file set up with full scheme of Validation for the actual YAML files. We're moving from using a Jenkins workspace to actually having a strong CLI So even developers can run this on their local system and actually use it Part of this also is going to be moving to using talks as the upstream work is being done to fix it So we can give it an actual virtual environment to work with instead of generating one One of the other things that we're doing right now is with the current tool We have split out configurations each of our project level have level projects have a separate configuration We're wanting to bring that down to where we can have a single single configuration that can be used to define Everything for the actual flow and how it works where we want packages to go how we want the artifacts to actually be built so we can control it and This will be on stack forage. It's something we want to have out there for the community use We know a lot of other companies other teams are solving this exact same problem and doing it We've been doing it now for a little over a year And we want to make sure that everyone else has the opportunity to try to learn from what we've done and possibly benefit from it and one other thing that we want to Accomplish with strikers. We want to enable building packages that are not virtual environments Maybe you want to use a wheelhouse, you know, just a collection of wheels So we want to have the ability to enable that kind of deployment Or at least that kind of package building So a couple things that we want people to take away from our little talking on what's going on is as much as you've Been told moving virtual environments around or bad. It's not as bad as it sounds It gives you some very good benefits Specifically when it comes to testing and knowing what you're going to be using on your system the idea of using a monolithic package which though it worked we ran into large issues where it actually caused problems where it slowed down our actual releases because everything had to be working throughout it and The other thing is that we are making striker available So it is something that other teams can build on and use to package open stack in a meaningful manner for their company itself And at the same we'd like to open the floor and ask if there's any questions anyone has any Questions specific to it Hi, it looks good. We've we've been using anvil in-house which generates distro packages rpms in their case Why don't you generate? distro.deb files or rpms We could we'd looked at it previously. It was actually what was used originally It was all the code was stored in it when we would do local installs We looked at the idea of continuing to do that To generate virtual environments and have them inside of the actual packages and at the time it just didn't seem like the best tool Yes, we could push it out into an app repository pull it down on our systems and use it But then we have the overhead of managing more app repositories as they grow as more packages are in there instead of having just a singular package or a group of packages that are everything that's there ready to go that we can Wget or copy directly instead of having to use system-level installs and there's also the the problem of Orchestration right when we're doing a deploy and we promise we weren't going to talk about deploy So this would be just a surface-level thing when we're doing a deploy We stage all these packages out there then we you know come through and shut down some packages or shut down some services Change the code over bring them back up and proceed on to the the next step of that deploy When we have a package going in We may have the the package manager Helpfully start the service for us and that kind of messes up with our coordination Maybe we want to change that around so and the other thing is noted in that is when you are doing a deployment If you are using the package manager, you have to pull it down at the time of deployment We can't have things out ahead of time and with our larger Environments with the growth that they've had that would actually add a considerable time to when we would have possible downtime for customers while doing those deployments Okay, but the package manager will give you dependency resolutions and conflict management and those things Are there any any benefits that you can see of striker over an approach like anvil that we haven't just covered? The one advantage that we've seen with it and we've even seen it in our own use Yes, you can get does a pendency resolution from the actual package manager. The problem is It's not always determinant how it wants to do things if there's two packages with a similar version number Or it's a letter in a different place in the actual version or something of that nature You're never guaranteed exactly what you're gonna get You're gonna have to always override and make sure that you have a newer version than what you've put into the package repository previously and newer than what might be available on the actual OS So you can't truly control what versions coming down because if there happened to be an update outside of what you're wanting to do The package manager will try to go for that unless you do pinning and pinning in apt and yum is not always pleasant At times to do when it comes to deployments and things of that nature with how we're doing it building the virtual environments controlling them We're able to control every single version of a Python library That's a going into that virtual environment to know that we've tested it and it works and any system we put it on It's gonna function exactly the same Your internal changes are hosted as patches in a repository that you reapply What's the advantage of doing like rather than actually modifying the source and do developers like patching patches? Developers do not like patching patches patching patches at times can be Very annoying sometimes you end up just rewriting the original patch in order to work with it It gives a little some advantages to it Normally when you would have to do stuff you would have your own branches your own work maintained in there And you would then have to do a rebasing and pulling in the actual code in order to Continue a workflow and then at the same time fix issues that came up You still have to fix those same issues that come up when you're using the ply based method But the advantage of it is you can have a repository of patches that are completely separate and smaller Then maintaining a full repository with the code patched in it So you're then able to have it act the tool actually do the patching and just then verify that it got what you want And update as needed, so you don't need to maintain it And why not using why not use an existing tool for patch management such as quilt or others that exist? I was actually going to bring that up the primary reason that we use ply as opposed to quilt is ply actually keeps the Patches in get so we have a get history of all of the changes that have been made to the patches Which is not something that is very easy to do with quilt any other questions? For contact information one more time in case anyone has any questions And you guys get a little bit of time of your life back