 Say when oh when all right. Hi guys. My name is Andy apes. I'm a senior software developer Adele The noise me maker in chief comment if you want clarifications Bring me later. So What this session is about the thing will actually advance about that is Basically around the deployment mechanism So one of the problems that a Little background so I work on a product called crowbar the little crazy bunny at the bottom Which is basically a deployment system for open stack. This there were a couple of sessions There are a few others that talk specifically about crowbar, but for the purpose of Of what we're talking about here. It's basically a deployment system for Open stack and any other large distributed environments one of the Things that's always annoying is that this thing runs around One of the problems with open stack is around testing a large distributed environment. It's it's Not sufficient to do a single node deployment and assume that it's actually gonna work when you deploy it in a real Multi-node bare metal environment and we've seen quite a few bugs in open stack that were revealed only when folks Actually went and deployed on a multi-node network Real honest-to-get environment as opposed to what Dev stack kind of does where you deploy everything on one stack and hope for the best so What does it actually take to deploy a to test open stack for real and what is it that you want to achieve? Well, you want the deployment to be reproducible and predictable. You don't want every time you deploy it for it to be a little different You want it to look like your production environment. You don't want it to be simulated So for example, if you deploy Swift and you lose use loop devices and simulate the presence of multiple disks available Well your deployment is not going to behave the same in the test environment as it's going to be on a real server with 20 disks You want to be able to deploy the code you want to deploy you don't want to be dependent on oh There's a new maintenance release that just came out Friday and it all of a sudden has New bits new open stack bits that's happened last Friday It was a new maintenance release for sx and if you are gonna go deploy a new system that go hits the internet You're gonna get those bits instead of the bits you were expecting because well, there's new better bits So you want to be able to control what it is that you actually want to deploy in your environment either for testing or for production One of the Things about open stack that's actually kind of cool is that you can customize it you can go create your own Scheduler you can go add your own drivers Well, you want to be able to deploy them in your test environment and make sure it works with whatever bits of open stack You deployed so all those things are currently Somewhat less than optimal in the testing and deploying solutions that are out there so that's pretty much the drivers for this session So some of the problems is how many nodes are we deploying a deploying on one or are you actually deploying a real? Production like environment are you? Deploying one static configuration, or I actually trying to modify the configuration it keeps ruining my punch line It was Well, I will speak faster apparently because I need to speak faster Are you actually configure applying all the diverse configurations that opens that gives you and testing them like you will Actually be using in production, or are you just trying the one true and only? Default configuration that whatever deployment system you're using Quick question how many folks actually deploy and operate open stack You've have you ever hit problems of I don't have the right dependency on the right Python module right packages I'm missing this bit and everything comes crashing down in a fiery ball of hell so have you ever tried to deploy in a new system that you've never deployed before open stack on to and Well your connectivity or the repo you were hitting didn't write didn't have the right dependencies And all of a sudden nothing works for you anymore and all your ci infrastructure is coming bloody red These are some of the issues that basically this solution what I'm Get to in a second basically tries to to solve And that does not include me being friends with Powerpoint which is not very happy with me. So some of the other solutions Some of the other solutions that are out there. There's obviously many different solutions for crowbar. They are one of the Most popular one is Dev stack anvil or Dev stack py is not a one and juju is the other not one each one of them tries to It's not a new problem. Every everybody who's been deploying and testing open stack has hit these types of problems I want to go into details into one of them into all of them if you're interested We can chat about them later, but basically there's still Shortcomings and obviously my solution is the best so so one of the first public Testing efforts for open stack was the canonical folks who've been doing a really good job around Again, it's wrong That's what happens when you put engineers to use bar points Sorry about that So the canonical guys have been doing a great job testing their packages They basically go and get the latest code their package and they deploy it in a test environment They test it and they publish test packages One of the question marks which kind of ruined my thing was do you really need to package? So if you deploy your own code your own extensions to open stack your own schedulers your own Patches because got for sake occasionally. There are some problematic Areas in open stack like attaching volumes once in a while Do you really need to go for the whole cycle? You really need to publish packages just for you to be able to test that Those customization and your deployment will work What would be cool is to actually skip that packaging it would be cool for you to be able to say here's my code Put it in your canonical get repository or wherever you use your production code Put all your configs that you're planning on using. I love PowerPoint. Sorry about that Get everything that you actually care about Stage in your test or pre-prod production and see if it's gonna work Right What's missing from the previous? Solutions from the current State of the world It's not that easy to deploy multi-node each one of the solutions from the prior slides require you to go and Pailer the nodes individually if it's death stack in the go need to need to go create local personality Local RC basically go and configure the node give it its own IP address make sure the right It ties to the right interfaces on that particular node It's gonna keep ruining my Same goes with anvil same goes with juju There's a lot of manual tailoring that you need when you actually want to go and deploy a multi-node environment Same goes for multiple configs if you want to tweak the configuration for example switching the networking mode from flat Http to vlan. There's a lot of oh, so now I need to make sure that The base operating system has the right net we can configure it on it I now need to go and tailor the open stack config and make sure it actually is aware of what's Configured on the node and make sure it ties in correctly all the fun stuff that you have when you try to Deploy any kind of more than single node open stack deployment and none of the current Deployment solutions base address to any real degree the concept of a custom code that you can easily pull from your own repo It seems so simple marketing guys do it So what's pull from source all about so starting from? A production employment system that is crowbar that basically knows how to I'll quickly recap it knows how to take a system configure its hardware deploy the operating system on it Configure the base networking for the operating system and tie it into a management network then deploy from packages the Standard bits for open stack that you're interested in Customize the network based on the components on each node So if it's a swift node that might need both a storage network and an admin network and potentially a public-facing network If it's the proxy the swift proxy node or if it's Nova based on the networking mode You're using it will need to create the appropriate networking Do you need to create bridges so the VMs have something to connect to do you need to create appropriate V lines and so And then lay it down open stack and tie it into the various pieces so that's that's a starting point what this feature this set of tweaks to Crowbar achieve is basically let you do all that but all without the packages What's missing some of the things that from crowbar what? And how does it tie into the ecosystem of of open stack so? By the way, feel free to interrupt me or stop if what I'm talking about makes no sense One of the things that packages do for you and do it very well is manage dependencies nothing lives alone, especially not in the world of open source and At last count when I counted packages and Python modules. They were about a couple hundreds that given Nova Node will actually need to pull down and install in order for Nova to actually work What do you do if all you got is the source? Let me see so one of the things that Dev stack which Gates trunk meaning tests every bit that goes into open stack That's very nicely is maintain a list of what prerequisite packages are required at least for a couple of distributions Not for all not for you all the open stack Linux distributions the other thing That's required is what other Python modules do you need on the system? if we want to use Crowbar or any other system to be able to get to a point where we can Trunk I'm sorry gate to trunk on it on tests run using this deployment system We don't want developers to have to go and update things in multiple places, right? If you need to pull in a new dependency, you don't want to have to go and modify things and Your deployment environment you want to be able to just touch the source code Well, so in open stack currently every project maintains its pip requires Because do people know what I talk about I know you guys do show hands Kind of okay, maybe So pip requires is basically at the set of Python modules that are deployed on a node when Before Nova is deployed and before and before you go and run the unit tests on the code to make sure it for it works So if any dependencies are broken if something if a version If you need to update a version of a module that you depend on and it hasn't been updated that Change that commit that changed open stack is not going to be accepted because tests are going to fail until you go and fix the list of dependencies in the project and at that point you will that's basically the source of truth around what I don't pass on modules this particular module requires Okay, I should have prefaced this and say this was meant for slightly more developer-oriented crowd Other things that you might want to affect on your testing environment are how do you want Nova to be configured? It's extremely flexible. How do you manage the configuration for a multi node deployment? In a sane way and the crowbar environment You basically have a nice GUI that you can play with but everything could do in the GUI You can actually describe in a simple JSON file that you can manage via your Favorite CLI's and you can check in into source control So you have a prescribed set of configurations that you can step through and vary as you want So you can make sure that it doesn't work in just one environment, but it works for everything I'm gonna skip the rest of the things because they're probably not that interesting So what's the process? You take all the bits we we talked about right you take The requirements the list of requirements you have you take the open source bed at the open stack beds you Basically go and package them into a self-contained archive again. This is the crowbar approach That gives you the reproducibility and predictability Nothing depends on things from the outside world unless you choose to you have the option of basically archiving everything You install the admin nodes that basically serves as the controller for the your environment test of production and it basically has a Get server based on it at low server It has a PYPI repository mirror that you can basically get everything all the dependencies Python dependencies from and it has an app repo on it that again basically gets lets you install the environment from a known tested to be good self-consistent set of Bits without having to go to the to the webs if you don't want to again This is engineers doing builds Okay, no Trying to make builds work So you first deploy the admin node and get that running and at that point you can deploy any number of notes So to show you that it's not just vein words It's actually That's the wrong window excuse the windows so Realizing how good open stack conferences What a good reputation open stack conferences have around Wi-Fi accessibility. I'm not gonna try to build Do a build right now because it downloads about 200 megs of stuff Rather, this is a build I ran Today's a Friday So one of the Challenges Often is to figure out what do you need where do you get the information of what you need? And how do you package it in an environment that you can then use for your testing? So the log file that we're looking here is basically Let me start with a little earlier this is extremely verbose by the way, this was so you'd believe that and if you are Enthusiasts You're welcome to read it. It'd be nice if I could see what I'm typing Sorry that so basically this is oh, I should mention that our team here actually was very Contributed a bunch of the code most of the code to basically figuring out dependencies and bringing things down So this this is basically a log file of building the ISO that will contain All your dependencies and will basically use be used to install your environment The bunch we're on the screen right now is basically going and fetching the depend the Distribution based dependencies not everything is open source. You don't want necessarily to build everything from scratch So we still rely on packaged bits for things that are outside of open stack things like python setup tools and Random a large collection of bits How large So what's crawled About 3000 random odd packages The other part and again, I Think I'm going to details that is probably not a very interesting to the audience. So The last part that I will show you is that you can basically Go commit a change to your get server or to your whatever update your code pulling the less bits from production go into your Go to your into your test deployment and say basically deploy again from scratch using my latest and greatest bits and end up with a working deployed environment. That's basically That's basically the holy grail. We can have a very short test codes deploy test cycle where You don't have to worry about any externalities that you don't want to choose to care about Let me pause here and see if there are questions or So I Don't know if you're familiar with crowbar at all. I didn't want to go make a discussion about crowbar Because the intent was to keep it a lot more So if I want to deploy a Nova environment Ignoring for a second all the other other pretty prerequisites that Nova requires no requires a database a keystone Maybe you want to install dashboard. We're basically the way crowbar manages this all is by Managing proposals where you basically say here are my nodes And here's the role that I wanted to play in the cluster in the cluster that I'm deploying So for Nova, you would need a controller You'd need a few computes and potentially you want to install a no volume the way you'd manage it in In a test environment or when if you are CLI type of a person you Say for example, I don't know the text visible. Sorry So, you know, it's live because it's obviously barfing left right and center. So I'll skip the CLI for too many things but big Basically You'd basically get a JSON file that lists all the parameters that you want. This basically lists the default one Right, so the same assignment of here are my nodes one of the things that you'd notice. I didn't have to change anything currently in this crowbar deployment. I basically have three nodes if I Would To commands create me a default proposal and apply it I can basically pick off a deployment of Nova on these previous happened to be VMs on my laptop But those could very easily be physical servers. So it's very easy to script The workflow would be where you'd create Either do what I just did where you create the default and then apply it or you can have a few pre-canned configs with the various parameters that you'd want to customize for your environment things like networking mode Does that I'll just use the GUI it's easier to look at Okay, so so basically crowbar Manages a life cycle of nodes nodes will boot up the HCP and sit and boot into a live image where they basically sit and wait to be told what to do and Crowbar basically orchestrates deployments on those nodes by triggering currently only chef Executions on those nodes when something needs to happen on them. So once you put up the node it will sit there until You commit a proposal that gives that machine a personality or purpose to be to do more than just wait there once that happens it will Crowbar will basically lay down the operating system by either kick-starting the seeds depending if it's read head over to and once it Boots up again and calls into crowbar and says hey, I'm here and ready. What do you want me to do? crowbar will configure chef for that node and trigger a chef client execution on that node tying the different nodes together and tying into the specifics of the network so At crowbar will assign at an IP address make sure it's on the right subnets if there are VLANs or bonding or any kind of Networking configuration that needs to occur it will be performed before you try to bring up the the open stack component that you're deploying on that node and So if you need to deploy multiple proposals right multiple components So if we look for a second at the various components of opens tag that you need To have a functional cluster you'd need to deploy my sequel because again, you're trying to test like you'll test in production You don't want to run against sequel like you want to run against a real database server Sorry, you'd want to deploy keystone you'd want potentially to deploy swift if you that's part of your environment or just glance you can basically queue up all those things and tell crowbar here's what I want and as Components come online crowbar is aware of the dependencies between them. So for example, if I want to deploy No, but I can either choose one of the the default of what my sequel cluster or one Operate against what keystone instance do I want to operate what glance cluster? I want to operate on gets pre-populated assuming they were already created. So as you if it's not it says Oh, I got none and it will wait until one becomes available So basically there's in many scenarios. You have this like chicken and egg Polling how do I know things already ready crowbar basically orchestrates the deployment make sure that things get deployed when they're ready Sorry, you'd have to speak up a bit Do you mean security? So the question is around IP sex security associations so at this point we basically make sure that authorized SS age authorized keys are available and But we don't we don't deal with IP sec. We haven't heard customers actually asking for it Do you have other specific mechanisms you're thinking about other than SSH? Yeah, so basically crow will ensure that there's a secure key generator for the root user and it this it's distributed to the nodes So you can actually and install this an authorized key so you can actually communicate between all the nodes Sure, we actually So crowbar can deploy swift it's actually one might have a warm spot in my heart for swift because I wrote that Any particular questions or So I don't want to dwell too much into Swift and I don't know if it's a little out of the scope of here One of the challenges with Swift is that it actually it's very smart about discs You want to be able in order to test Swift you need to have enough devices to make Available to Swift so it can create replicas or multi devices and on multiple nodes Otherwise the whole purpose of Swift is kind of defeated in a testing environment especially The way Dev Stack is set up currently to test Swift is by creating loop devices or basically taking files and pretending there are discs and Making Swift work against them. It causes Let's say like this. You don't want to run performance tests on that type of setup because You kind of defeat the whole purpose of having many discs and many spindles available and getting much better performance You'd want to be able to deploy in real environment with taking advantage of all the discs you have In your environment to get realistic performance numbers So yes, that's actually one of the things we've we've Targeting Krober to do is basically be able to do real performance testing on the actual hardware with Lots and lots of spindles on the same server Sorry, there's a light right right above you, so I didn't see you there So so you're asking a few questions. Let me see if I captured them. So the first question was is that a particular Bar clamp or not or and how does it relate to the shift cookbooks? So there is a new bar clamp that basically will deploy get in a server configuration. There's no real get server Basically, you can denote one of the nodes on in the cloud environment to be your get Repository that all the other nodes pull from So there's one new bar claim to deploy from source basically all the other bar clamps were modified to pull from that To find that get repo in the environment pull from it to the source and deploy it locally on the node so and That basically is modification of the chef chef cookbooks that were already in the other bar clamps. Does that answer your question? Any other questions So apparently I thought this is all on github already But apparently I was a little glitch This will very soon be on github the code is but there were some branching structure issues, so I want I Won't dwell on that I Think that's if there are no other questions This slides with Up-to-date links to where the information is available will actually be uploaded to the slide share for the conference and You're more than welcome to play with it would love to hear feedback That said I've been told that I need to say that we're hiring so if your developers product managers are interested in open stack Stop by the Dell booth and people would love to talk to you That's about it. Thank you much