 All right, welcome. Come on in. There's still a few more seats left. If you haven't grabbed a beer from outside I would suggest it. It's kind of nice So this is hold my beer and watch this upgrading open stack from Havana to Juno. I am Jesse Keating and That's my Twitter handle. I love complicated challenges I've been doing opens up open stack upgrades from around the grizzly time period and And I have a very strong appreciation of dark heavy beer since about the grizzly period I Promise I'm not an alcoholic. I can stop upgrading it opens stack at any time And I do work for a blue box cloud. We're a fantastic company down in Seattle doing private hosted clouds as a Hosted private clouds as a service So let's talk about open stack upgrades. It's totally awesome, right? There'd be some dragons here Or whatever this thing is Those of you that laughing now. I know who the reddit users are But that's okay because I have a plus five sort of ansible to do all the things I need to do Upgrade styles you can do micro upgrades when I was at rack space. That's what we did rack space was trailing master of All the open stack projects Totally awesome And they would do micro upgrades every couple of weeks or every couple of months always an interesting challenge Or you can do macro upgrades where you're doing major release to major release to major release Or you can do something even more awesome where you skip a major release Let's talk orchestration When you're doing upgrades of open stack the eventual consistency model need not apply Open stack needs some very specific Steps to be happening at specific times in order to make an upgrade successful this means that traditional tools like puppet where you make a setting and eventually your fleet is Okay with that setting isn't really going to work out for you So ordered set of actions you need to accomplish But first before we get to some open stack upgrades. Let's talk about the database So at blue box we use percona cluster We have a two node cluster with one arbiter that's sitting somewhere else to do master elections It was based on my SQL 5 5 in our current production model However, my SQL 5 5 can't handle the neutron migrations. So if I want to get to new neutron I have to first get to new my SQL Which is what that point is I'm missing a step. I had a sheet web page here that showed all the steps of the With the web page shows for doing a my percona upgrade. It's very long is very arduous. It's very ugly Wasn't going to work out for us. We need to do something on our own so We Ansible eyes all of it. We took our two db hosts and our arbiter and we did a number of tasks across both of those On the arbiter. We purged the old package and config we fixed the file system permissions because arbiter left some awful stuff around Run the arbiter role as if it was new Stop the database on db hosts Remove the packages put the updated configuration in for the package modify compatibility settings because the package will helpfully Start up as soon as you install it and if it starts up without having the compatibility setting, it'll destroy your database Don't want that to happen. So we get the modified five combat settings in We turn off replication We install the new package which again starts up the database we run its own Upgrade migration play that that percona ships Restore replication so they're able to talk again and Restart the database again and then we repeat that on the other host. We're doing this one house at a time Remove the compatibility settings restart the database again and Then everything is copacetic Well, we we turn that big awful web page into a nice readable Ansible playbook not positive you can read that but it's there so this is Running on the D the database servers We are not tolerant of failure at all and We're running this in serial. So one host at a time We're doing some checking to see if the the mysql version is what we wanted to so we could run this play over and over And over again, and it won't eat our database the second time through We are calling out a different playbook, and that's this playbook that does the things we wanted to do stopping removing installing Compatibility upgrade etc. Etc. And eventually we remove the compatibility settings restart everything and everything is golden All right, so what about rabbit? You know you for upgrading your database You're very good in your open stack. You might want to update your rabbit, too, right? don't Pulling rabbit out out from underneath open stack is a really bad idea particularly in the Havana code base Your services that talk over rabbit will stop talking and not realize that rabbit has gone away And it'll sit there forever and ever and ever and ever and nothing will ever work until you restart all the services But we didn't want to have a bunch of service restarts in the middle of our upgrade We wanted to try and minimize the downtime for our for our customers. So just don't upgrade rabbit It's probably not a good idea All right on the open stack because that's the fun part There's a repeating pattern in upgrading of open stack You put your new code and config in place you stop your old code you migrate your database you start your new code That's you know it repeats service after service after service within open stack And so that's something that we modeled into our our orchestration The order in which you update your open stack services it kind of matters it kind of doesn't matter But it matters if you care about your services staying up and interoperating as long as possible Which we did care somewhat about I Tried a few different orders This order that we're using works for us. It may work for you It does attempt to minimize the disruption of the Cloud as a whole by having each individual service down as short amount of time as possible So all throughout a few services here and there will be restarting as as the upgrade happens But the whole cloud operation as a whole tries to stay stable as much as possible You're gonna want to avoid Interproject version dependencies so introducing a new functionality in say Nova that depends on a new functionality in neutron Don't turn that on in your upgrade because you'll update Nova and it can't do the thing that you want to do because Neutron hasn't been upgraded yet Lay down your new code first get everything working then start doing feature enablement when everything is on its new code So our order is glance then sender then Nova then Neutron then Swift then Keystone and then finally horizon Some shortcuts that worked really really well for us We had already done Neutron. We'd been on Neutron for quite a while So we didn't have to do Nova network to Neutron as part of this upgrade. That's a big shortcut We were we did our ML to migration We went from OBS to ML to as a different Outage window that one the VMs did lose their network for a very short period of time and then it came back But we wanted to do that Ahead of doing the open stack upgrades so that we're only moving three or four different parts not 20 different parts at the same time Linux bridge, that's right And newer kernel we did our kernel upgrade for some of our Customers as part of getting them on to ML to and Linux bridge. So that was already taken care of we don't have to consider that as part Of our open stack upgrade So there's just some of the shortcuts that we utilized so that we could do our upgrade without having customer VMs lose any Sort of connectivity or lose any sort of uptime our strategy. We wanted to reuse our deployment code Having separate code paths for upgrade versus deployment is a great way to have two problems We wanted to delay the restarts of the service until the right time for that service Which if we were We did want to fail immediately we are not tolerant to any sort of failures in the upgrade process So as soon as any failures encountered stop the bus everybody get off and figure out what's going on But in order to do that we want to have Non-destructive reruns, so if I'm going to stop the bus in the middle of an upgrade process I have to be able to restart that upgrade process from the beginning and have it not eat my world halfway through But that meant we had to design each and every one of our tasks to be idempotent We need to be able to run it again, or we need to be able to discover that it had already been run and not run It a second time through All right, let's walk through some of our services Let's talk about glance Glance has no real surprises. It's a pretty easy upgrade In our upgrade playbook all we really do is run the glance role And we're passing in a few extra things into the glance role We're saying that we don't want to or we do want to force the synchronization of our database So every upgrade and we have to synchronize the database We put the code to do database upgrades directly in our deployment roles However, however, we conditionalize them so that they only ever run if we're in an upgrade scenario So all we have to do is run the existing glance deployment role tell it We want to force the synchronization of the database And we don't want to restart the service because of any config changes We'll let the database synchronization code take care of that And this is a little glance a little glance with what the the glance deployment code code looks like So there's a there's a step to lay down the glance configuration files. There is a Explicit call to stop the services before a database migration. So when database changed, which we are forcing it to be changed And for sync is true Synchronize the database just calling glance manage with DB sync And we will restart the services Although this notification to restart will not trigger because we do not want to restart We have an explicit call at the end of the deployment This says at the end of this the service should be running in a regular deployment It'll already restarted because we've flushed all of our notifications And so in a regular deployment, this is a no op and an upgrade. This is where the service gets turned back on Let's talk about sender sender is a little bit more complicated because there are Sender data and sender control it's separated out a little bit more than glance which just has glance and glance registry So we have a couple of different plays going on We have plays that are targeted to where sender is running on our volume hosts in there We're running the sender data deployment role, and we're just not restarting with services But then we were explicitly stopping the sender volume service Then we go through our controller and we run the controller role and just like with glance we're forcing a synchronization of the database because the database lives on the controllers and At the end of this role the the controller services are back up and running and then after that we can start our data services Another bit more complicated thing is we are introducing sender v2 with our upgrade So we wanted to make sure we get that keystone service added to the endpoint So that's a little extra task that goes in our play Our sender back-end depends some of it is Dedicated NAS devices and some of it is dedicated storage on a particular unit of our cloud One of the machines of our cloud And then we are calling a new Role the stop services role. I don't like to repeat my code So even though I'm going to do this for lots and lots of different services I wrote a role that all it does is restart the service that I want to restart so code reuse is awesome Nova the fun one Although it's pretty straightforward Much like Neutron or I'm sorry much like sender There is nova compute versus nova control. So we have to do things in a two-stage approach Run through the nova data role stop the services Run through the controller role start up the services. That's all that's really needed there Neutron Neutron had a little interesting quirk when going from Havana to Anything newer than Havana. You need to stamp the database as part of the upgrade Previous to I believe ice house. There were no Neutron database migrations, but there are now so you have to give it a starting point All that really means is outside of our normal pattern of doing things on the various parts of Neutron There is a point in which we need to first stamp the database with the I'm sorry once we check to make sure we haven't already stamped the database because you don't want your database to go backwards in versions We will stamp the database With the new with the old Havana version as a starting point Then we can run our control role, which will as part of it Will upgrade the database to the Juno version then we can restart all of our services Including all of our agents in life is good This is the only time in which some of the network connectivity might go down. However, it's very short and most Consumers will just see this as a bit of a network blip or a small timeout things come right back on and off you go Swift really easy the Swift guys have gotten their upgrades stuff down pat All we really have to do is just run the roles as if we were installing and off we go Keystone no sweat Just run the roll and off you go But run it in a way that forces the database sync And then horizon man horizon just a web application Why can't everything be this easy? Just run the roll and you're done No databases to worry about no connections to to worry about just go All right, so let's talk about some gotchas that that can occur because it all sounds really easy, but it's not entirely that easy Keystone pki tokens that's a new feature that came in and I believe ice house Thought it was going to be really awesome. You could get rid of a callback to keystone all the time by having the validity of the token embedded in the certificate of the token Turns out it's not actually faster. So you're not saving any time But also it breaks all of your services As soon as you start keystone and keystones configured to do pki tokens All of your other tokens are now invalid including the service tokens that all of your services are trying to use So restart keystone all your stuff stops Not cool Neutron and nova so with I believe ice house of certainly with juno When you're dealing with neutron you can tell nova that You need to wait for neutron to plug the virtual interface and if it doesn't it's fatal This is a version dependency if you turn this on in one side and you haven't upgraded the other you might not be able to finish your instance booting So it breaks the builds until both sides are upgraded. Now. There's a couple different things you could do you can have Intermediate nova config state that does not turn this on in which case It will assume that if the call to neutron was successful that the interface has come up and eventually it will probably come up Or you can just ignore this for the small period of time that both services won't be right and that's what we did Nova nova has a gotcha Deleted instances in the database The data is still there you go to delete an instance and all nova does is just marks part of the table as deleted Does not remove any data whatsoever on a long running cloud. That can be a lot of data The more data you have in the database the longer your migration can be The longer your migrations are The longer the service is out because you can't have services running while you're migrating your database There's also no supported tool to trim that data out of your database so You can either use a tool that you write yourself or collaborate with the open stack operators to find something that's been passed around and might work Or you just incur the downtime and no ahead of time that you're going to have it What we've done is we've taken snapshots of our live customer databases We've ran them on a test machine to see how long the migration would last And in a lot of cases because of the usage of our cloud We've found that the time it takes to purge the database of the deleted data Is roughly about the same amount of time it takes to migrate the database with existing data in it So we're not saving ourselves any time by doing anything other than just migrating it Down the road that may change and the age and the use of your cloud is definitely going to to have impact on this So no ahead of time before you try and do this what your migration is going to be like So a few resources before we open up to questions All of the code that you saw exists on github under the blue box group Ursula project There is it's a big pile of ansible. There's an upgrade.yml in the root level That is the upgrade playbook. It's making calls into our various roles and doing the various things that we need to do to stand up a blue box cloud It's all open source. You're free to use it free to laugh about it You can contact me if you have any questions about it. I live on irc as well in the operators group So now we'll have some questions and while that's going we'll just leave that on the board So if you're going to ask questions, I would ask that you use the mic so that the recording will pick it up Otherwise, I'll repeat it and yeah So my feeling is that everybody wants to ask about the order of the services You started from the glance and I my feelings that everybody wants to ask about this. So I'm the messenger and The surprising fact is actually started from glance and then seeing them. Why is that? So I started with glance because it was a pretty innocuous service There there wasn't really any wild changes in the glance api between It was really none really between Havana and juno. So changing out the the version of glance isn't going to interrupt any other services So that's just what I started with. It was the it was not quite the simplest service, but it's one that tests our model And does it without having any interdependencies? So it doesn't mean that for example From juno to kilo their order might be not necessarily exactly the same It might not be necessarily exactly the same that is just a matter of testing and making sure that It is a matter of testing and it does depend on your configuration as well Which features you turn on how they're configured where they live that sort of thing All right So there was in paris. There was a talk about the nova object versioning Did you think about like a? Rolling upgrade without yeah, so that's something that I spent a lot of time Working on when I was at rack space and at rack space It was super important because a rack space public cloud is a much much bigger thing than a blue box private cloud a rack space public cloud One of their regions can typically have 6,000 compute nodes And the amount of time it takes to ask 6,000 compute nodes to shut themselves down In order for you to start your migration is a considerable amount of time Particularly if you want to shut them down gracefully where you give any long running Action like a migration or a resize a chance to finish up So with with that type of scenario It's important that you be able to upgrade all of your other infrastructure All of your other nova infrastructure infrastructure before you upgrade your compute infrastructure So you by using nova conductor and by using object versioning You can upgrade the nova api the nova scheduler the nova everything including nova conductor You shut all that stuff down. You leave computes running They're no longer able to talk to the database because the conductor is down, but that's fine They're still operating. They're doing any long running tasks. They need to do Migrate your database bring all those services back up and now you have a version mismatch you have new nova control old nova compute the use of the conductor Saves all that because the conductor acts as a translator between those two barriers and acts as a translator to the database So I know that red hat they claim that they support this kind of like a mismatch version just for the upgrade time. Yeah, yeah so Once you're in that scenario you can roll through your computes and slowly restart them as they gracefully shut down And come back up and they'll come up with a new code once everything's on the new code then you can remove any object back rev that you're forcing And you'd be able to turn on any new compute features that depend on nova compute having that new code So in in a rack space model where your compute set is very very very large That makes a lot of sense to focus on that in our blue box clouds where our compute set is much much much smaller the amount of time and effort to Work through the orchestration to have all that set up is lost on just You know going the other route of doing the restarts Directly on it and getting everything up to the same level all at once So by NHS do we know actually which services support this object visioning right now? I believe with kilo. It's just nova just nova I take that with juno. It's just nova with kilo I thought one of the other services was picking up on that model There's a what's that? Yeah, so cinder is one of the other ones that support it And with kilo there's a few other things. They're making upgrades easier. Kilo was one the last Migrations that you have to do that you have to stop the services in order to do the migrations So some of the the later upgrades you'll be able to do a live migration of the database and it will Create all the new tables where the new data will live But the new code will look for both places and slowly migrate them over time So there's a lot of cool things coming with kilo and with liberty beyond that's going to change the upgrade Landscape quite a bit Thanks. All right. Thank you So the size barrier between whether you would worry about doing nova live up or doing nova version mismatches If I had to guess it depends I'd say it highly depends on your orchestration tool if it takes longer than a few minutes to Shut down all of your compute nova compute processes Now shutting down nova compute doesn't shut down the vm's the vm still run All you're doing is shutting down your customer's ability to launch new instances or manipulate existing instances If that if your orchestration takes longer than a few minutes And if your sla is smaller than a few minutes worth of outage for that That's when you want to worry about trying to do The separation of compute upgrade versus control upgrade If your sla is so large that you can incur that much downtime of the api level commands, then I wouldn't worry about it at all I'm assuming you uh, all these deployments are in ha So if like nova is in highly available more like three nodes, you're running If that's the case, uh, when you say you upgraded glance did you go upgrade glance across all the three nodes? so ha on a service on api level service like glance or the nova api it's a great story for uh Protecting against catastrophic failure or one of your nodes completely falls off the internet It does not protect you against upgrades where you need to stop the services from writing to the database while you do a migration So because of where hadana was and because of where juno is I had to do coordinated shutdowns of those services Before migrating the database. So ha didn't buy us anything at that point a couple of small questions. So how big is your private cloud in terms of number of nodes I suppose and then how much prep time did you or a team of people like you put into the prep before you actually Went ahead and did this So our clouds the smallest private cloud you can buy from blue box is three nodes That's control and compute on two of those nodes and compute only on a third So three compute nodes is the smallest you'll get from blue box and then we scale up from there We're certainly not at the 6000 hypervisor level. We're much smaller than that I don't actually know what our largest set is. I'm not sure that I can say what our largest set is, but they're they're all reasonably small sized Our trick was that we have lots of them and so we have to do this over and over and over and over and over and over and over As for prep time It's been it it took a number of development sprints in order to get the upgrade script Playbook the way that we wanted it and proven to work and to write out the method of operation and Understand where it might fall over and how to restart it. So that took A month maybe of of work on that as far as prepping each site That's a much smaller period of time That's really just pulling down the nova database from their latest backup Restoring it into a vm where I can test the stuff Figure out the time length and that gives us a general idea of how long their upgrade process is going to last Then we just communicate with the customer. Hey, we're going to do this You're going to have api outage for x number of minutes at the end. You're going to have all these new features awesome And then would you would you feel confident using these scripts again to go from juno to kilo From juno to kilo. I'd change a few things because you don't necessarily have to do You don't have to do the database migration. You don't have or the the database upgrade There are you don't have to stamp Neutrona's database that said those scripts will detect that scenario and not actually do those because you don't need to Um, what I would do is look at whether or not I need to do a coordinated shutdown of The nova service before I do any database migrations or anything like that So I think I still have to do that to get from juno to kilo So I'd leave it mostly as is it's from kilo to liberty that things start to get a little bit interesting Our application also used the salameter and the zabix So I want to know did you try to apply update these two service? The salameter service and the zabix Um, so we're not implementing salameter in our private clouds So I didn't have to worry about that and what was the other service? Zabix this is also the Performance management. Yeah, many data in the database. So I worry about Yeah, we we don't implement that as well. So that I didn't need to worry about The as far as the open stack services like like salameter I believe it follows the same model. So that's how I'd approach it of coordinated get get coding config in place Shutdown service migrate bring up Uh But I would definitely look at their upgrade notes every release has some pretty good Release notes and part of those release notes are upgrade concerns for each individual service And that's where I got a lot of information from not just from Having done it for multiple years in in the micro versions But also between the major versions figuring out what special things need to be done Which configuration options need to go away or come into play or things like that. Okay. Thank you Could you go into more detail about the operating system upgrades and where those upgrades done to the cloud nodes or the client So that's the dirty little secret. We didn't upgrade the operating system Our operating system were made the same what we've done is we package up all of the open stack services As python virtual environments wrapped into adabian or even to package So all the all the the python things that Are applicator there are open stack applications need live inside that virtual environment There are a few things that we need on the host level operating system But we've either back revved the new versions of say libvert Or by the on libvert or some of the other Base level operating system things we've either back ported them into our own ppa or we're making use of Our operating system is you've been to we're either making use of the you've been to cloud archive Or we've we've built what we need into our own ppa. So we don't we're not tied to the ubuntu's Packages of open stack we do our own and we do our own in a way that we can lay it down Really independent of of old versions so we can lay it down and run everything out of the virtual environment So in the instance that you didn't do this preparation that you did would the script still work I'm sorry be heavily modified What was that in our use case? We're we did not do that We have laid down the operating system put open stack on it And it going to from Havana to Juno. We need to upgrade the operating system Right, so that would the script still work with modification or would it be a lost cost? They Would work with modification you would have to decide when you want to incur that operating system upgrade Uh, my first guess would be doing the operating system upgrade First if you can And get your operating system up to the level that you want and then lay down the new Package bits and get the services started So if you lay down your new operating system your old services are all going to stop Upon reboot you're going to start no new services The vms themselves may not even start up, right? But if you can lay down your new packages in an efficient manner And get the new services started up that's going to be cleaner than trying to do a split thing And trying to get new packages that don't belong on that old operating system in place before you upgrade the operating system So my question was going to be about how much you might share with osad But your description of the way you do your packaging pretty much answered that question Right, so osad is a is a the open stack ansible Deployment osad open sec ansible deploy It's a really neat project that came out of rack space private cloud And they put it up on source forage and then went through and deleted all of the rack space isms from it So it's now a general useful thing that happened fairly well after our ursula project existed And so there is a lot of duplication between the two but also some differences in opinions and how things are done right now We are not targeting using their stuff. However, we are collaborating with them particularly on the upgrade things They have an upgrade script which tries to That does a lot of this stuff. I haven't fully compared the two together But there's a lot of room to collaborate and i'm you know, pretty good friends with the guys that wrote it So it's it's all friendly stuff going on So when moving from havana to juno with some services being in and some being juno Or perhaps some services being half juno and half havana What kind of experiences did you have around rabbit and the rpc calls that were mismatched? And did you take any mitigation steps around that? So the rabbit problem only comes into play if you're running half of your services on one version half your services on the other So rabbit is only intra service communication We didn't do any of that Because we didn't need any of the live upgrade type capability The only way that you could do that really is with Is with nova and that's if you're using conductor and we avoided that whole problem altogether If you avoid that whole problem then all you really have to worry about is the public size the public api In public apis as long as you are leaving your old one in place And only adding the new api endpoint and not deleting your old api endpoint Then your old clients your old services are still able to use that new service and just make the old api calls All right, well, thank you guys for coming. Uh, hopefully you know how to get a hold of me if you have any more questions and Enjoy the show