 Hey everyone, my name is Billy Olson. I'm a software engineer over at Canonical and today we're gonna have a little bit of an introductory talk on upgrading and You know how to plan your upgrade what to think about kind of you know general good guidelines of what you can do When it comes down to doing an upgrade for OpenStack, you know, it's no trivial task So we'll just kind of cover some of the things to think about As you're going through it So I do work for Canonical Obviously, we are the company behind Ubuntu Linux, but this is a general talk So I will say that you know, I am slanted slightly towards Ubuntu And what I talk about and besides the aubergine slides I have included exactly one slide in here, which is really slanted towards Ubuntu, but everything else I'm gonna try and keep generalized for just general good practices and things to think about while upgrading So hey, OpenStack Liberty's out great, you know, there's a lot lots of good stuff in there, right? I mean if you're like me you're out there running my ice house cloud I just enjoying all the stability that I've had on ice house not really wanting to touch it because it's been there for a long time But I'm looking out there and I'm saying Liberty and Juno and Kilo and all the new features that have come out They sounded great, but they were tough to upgrade to right? I mean in Liberty We've got cells v2, you know, it's the right direction. It's gonna be the one way I think that we're gonna really deploy OpenStack in the future and break it down So that's kind of a more scalable micro architecture approach as far as you know Dividing your regions out there for the schedulability and stuff You've got you know improved IPv6 support I think everyone saw that North America ran out of IPv4 addresses officially within the last six weeks or so So IPv6 is gonna be finally here. I mean it's been ten years now, but you know, we're finally getting to IPv6 You've got some improved cloud hybrid cloud identity management, you know Good old guys over at Keystone have been working hard to kind of make sure you can go different places As far as you know managing your cloud your height your cross clouds there Always some improved usability and things just get better every single release and and get here I am still on ice house right or Havana or Grizzly or Essex or wherever I'm at, right? So what we really want to figure out is how to upgrade, right? and My one slide that's a bunch of specific is why now, right? This is something that I want to talk to you about why now as we're going into the metaka cycle Do we need to think about? Why what what level we want to run on? What we see is that most customers out there if you look at the latest OpenStack survey is a lot of customers are running on ice house and from an Ubuntu perspective That's because we're all running on like a a long term support release like trustee and As we keep going through this you see that there's the metaka is going to line up with our next Ubuntu long-term support release, so that'll give us another five years of support and so we really want you to start thinking about moving from trustee and ice house to your metaka and Xenial ZRS type series And you see that one of the things that we do in our open-sex support model and our cloud model is That we've got a metaka release That will that will match in our cloud archives back to the trustee release So you'll get three years of support there and you'll be able to kind of roll it over and have the same support on the Xenial series so it's kind of your transition area period between your long-term support releases there as you're looking at it So that's one of the real reasons why I wanted to push like an upgrade to metaka As we're looking at where our overall long-term support releases for Ubuntu are Okay, enough about Ubuntu and let's think about our upgrade So upgrading is really simple, right? I mean it's three big long steps that are working down into about a thousand others Planning it testing it and finally doing it, right? Planning it takes a while testing it takes a while and doing it takes a little bit of time But nothing compared to actually all the other stuff So when we go to planet, let's think about studying the release notes It's really important. There's a lot of gems in the release notes that I've actually been out there The good doc people over at OpenStack really work hard to actually put out some Nice docs there to help you understand. What are the release notes? What new features are there? What things to look out for? Different things like that and and when you're jumping multiple releases actually pay attention to all the different Release notes in between, you know, if you're going from ice house to metaka or to Liberty Start taking a look at each and every single bit of release notes that are in there even upgrading from a Point release to a point release on the same release, you know, there's there's new bugs in there There's new limitations There's some that are fixed you may want to make sure that you go to release x.1 or you know like the latest kilo before go making the jump to Liberty or maybe the latest Juno and then the kilo and then the Liberty These release notes really help you understand what you're getting out of it and what you want to actually focus on and then of course There's the distro release notes Whether it's a bun to whether it's red hat whether it's your Maranta. So you're that you're actually going on or Sousa or whatever it is each distro is going to work hard to actually package it and put it together each distros set of packages are slightly different than Each at the other ones they may have slightly different dependencies and versions based on which live vert level. They're running or something like that The distros will kind of lay that out and you'd actually focus on what how those release notes actually affect you And also as you go through it the deprecation notices that are in there, you know Maybe with no longer supported where you're currently out on your support levels Just take a look at you know, what's going away what you need to change within your actual clouds And as we go through planning it you need to start thinking about the architectural impact of your actual upgrade so like for instance if you're moving from ice house to say kilo and you want to take advantages of You know like distributed virtual routing or L3ha for your neutron. You need to think about how that actual Architectural change might affect you Moving the workloads away from the quantum or the net the neutron gateways into the actual compute nodes If you're doing the VRR You need to kind of think about what your overall architectural impact is there to your current cloud It may have other ramifications to your overall You know your switch configuration Your actual physical layout of things you just need to actually take a look at it The You've been running a cloud for a while You've been running your cloud for a while and your cloud runs a specific way for you But you've also probably learned a lot about the way that your cloud works Some of the things that are good on your cloud some of the things that are bad in your cloud Probably done some evaluation as far as like what you might want to change when you get to it And I and I address this with the the lessons that are learned from the current architecture that you've actually driven So take a look at it think about what you actually might want to change based on your current architecture But they can choose carefully don't necessarily think that your Upgrade is going to be the right time to actually make all of the architectural changes that you want to make You may want to break those architectural changes out to be something after the fact or maybe before the fact before your upgrade preparing for it The more you change in any type of upgrade cycle the more can happen and you can be in a real pickle Another thing is to evaluate the tenant impact now this sounds kind of crazy, but you know You've been providing services for tenants in your cloud for customers of your cloud And those are probably the most important to you even though you feel like you know your time is valuable and other things like that Really go out and talk to your tenants You know talk to them about what their peak usage times are because that's going to help you understand When and when not you can you do when can you drive your upgrade? When can you not upgrade when would be a good time when would not be a good time? What's critical to them? So you know it may be that they don't find certain parts of your API or your offering is very critical But other parts that you never would have thought about are super critical It may be that the peak usages that they have of manage and building VMs are during the day But all the services that they offer to their clients happen during the night think about like Netflix, right? If you're running your own video streaming service or something like that You're probably gonna get your peak usage from your tenants users At night when they go home and watch primetime television versus during the day when they're setting them building and preparing For their overall nighttime activities there So and if you talk to your tenants and you understand what it is that their use cases are and what their business operations are They can help feed in when their critical times are to you so you can actually develop a real close relationship With your tenant so that the upgrade goes smoothly for everybody And some of you know a lot of your times probably have some automation built around certain things So if you're making certain types of upgrades you may lose an API version somewhere in there It seems like every release it goes out one project is deprecating some API version X, right? So like sender API version 2 is going away sender API version 1 is going away, you know, Nova API version is changing. They're deprecating. It seems like every release someone something's getting deprecated And will be removed in a future thing So I just try and understand what the actual impact is to your clients. They may be aware of it They may not be aware of it But that's part of the discussion you should have And of course you want to plan for downtime So Even when we try really hard not to have downtime We should at least plan for it. Even if you've got it worked out so that you can have a non disruptive upgrade You know, you're not gonna lose you're gonna migrate all your instances off your compute nodes Your networking configuration set up in such a way that you won't actually lose any access to your Compute nodes, that's great still plan for downtime Because stuff happens. You never really know what's gonna happen in your environment. You might have network drops Sometimes the database upgrades don't go as smoothly as you actually want And depending on which model you have the upgrade to your actual database table It could take a long time and depending on your database You can have locks that are held which actually prevent API access or writes your databases So these are things you just kind of need to plan for is as far as how Certain things might actually impact whether your your services fully available up or not And then when it comes to upgrading it's kind of a you know, choose your own adventure Are you gonna go with like a big bang upgrade? Are you gonna just upgrade your entire cloud at once and just say okay? I'm gonna everything is going into kilo tomorrow. Everything's going to Liberty tomorrow We're just gonna do it all at once or you're gonna do a rolling upgrade type model Where you're gonna take one service at a time upgrade it from you know kilo to Liberty take it through the different steps that you actually want to go through or You know one other model that some customers do go through is the whole parallel cloud Where they have critical services, and they really don't want to touch it So they'll just build a cloud next to it and go through it So big bang upgrades as I call is basically just upgrading my entire cloud at once When you're doing a big bang upgrade I'd recommend minimum to know architectural changes in here It doesn't really allow for a lot of them because you're just upgrading everything at once and a lot of times Things can go really fast while doing this because you might have the appropriate gash is set up The appropriate service restart levels and stuff like that everything runs really smooth But that's because you didn't make many changes other than the software packages that are out there That are actually there. You can have minimum to moderate downtime depending on how smoothly things go The failures during this type of upgrade can be a big problem, right? So like if you just said I'm gonna upgrade everything And you think everything's gonna go smoothly and suddenly your nova services went down That can be a huge problem because now you've all your units of nova service are actually You know kind of wonky and unavailable because you decided to go for the whole upgrade all at once And your rollbacks have a bigger scope because you have to probably do more than one unit of Service up rollback at one time So you might have to rollback, you know Three different api nodes back to their previous level versus just having one api node that had to be rolled back Then rolling upgrades. This is probably one of the more popular ones and the easiest ways of going about it You can roll across services as well as the The units of service that you're actually running So this is where you start out with, you know, three three services on the same level And you slowly update one to liberty Once you get that one up, then you go to the second one So then you have two two units up and then finally you move to the third one where everything is running liberty You can do a few architectural changes in this type of model. It's generally not too disruptive to what you're doing You have some minimal downtime If you have to rebuild any of the the networking stuff So if you come from a release with like ice house Or Havana or something like that where you don't have a lot of high availability built into your Networking and you're using the open v-switch There are times where it actually restarts and then rebuilds all your Your ports within an open v-switch which might cause minor interruptions to your network connectivity And there's ways around that some people have like, you know, kind of worked around it And then they kind of made more support for it and like l3d ha and dvr and things like that But you can kind of expect some some things there Failures are not really as impactful impactful during these rolling upgrades Because if you've got you know one service out there that fails and say three, you know I've got three units of it one unit fails on the upgrade. That's okay I can go and take as much time as I need to to actually figure it out Why I still have two other nodes up running in high availability kind of mode And then there's the parallel cloud, right? So there's some downtime It's mostly with the tenant as you talk to the tenant to help them migrate actually over and figure out You know, it's more impactful to them Because you know you talk to your actual tenants and see Okay, you've got new endpoints now. You need to go start using this Some different things this might change there Failures are kind of less impactful to the actual clients because they clients can oftentimes, you know, test it out run new things But it's very dependent on your tenants and stuff like that I do know that there are various opportunities To help you with this but we do see some customers actually going out there and And running this parallel cloud type approach When you're doing that, you know and your parallel cloud if you have to completely upgrade and change your OS from trusty metaka and you want to redo your architecture This is one case of where we actually see it when they want to rethink What is they're actually doing in their cloud architecture deployment? and rollbacks so Rollbacks are kind of tricky right when you go and do upgrades and And everyone thinks forward, you know, they think that they're going to get the rollbacks working quite right But it's actually harder than you think, you know, you need to make sure you have your databases backed up You have your configuration files backed up or Whatever service you're using like if you're using puppet or if you're using juju or or you know chefs recipes Whatever you're using to actually build your configuration make sure you can rebuild it constantly to make sure it's going to work just fine for you um mirrors of packages, so One thing that you need to think about is if you're running, you know code ice house version in Internally you may be running like ice house dot one right And as you get into a failure situation if you didn't mirror your packages properly and you go do a rollback You might be at ice house dot five all of a sudden and the last thing you want to do is When you're recovering you don't want to encounter new bugs because you rolled back to a level that you hadn't been running on before So you really kind of want to lock that down The other thing that mirrors actually help you with is as you've You know, you've been practicing and planning your upgrade for a while So say you're on you're trying to practice and you start practicing to liberty And for the next three months you're practicing and you're practicing and you're practicing and you're running off Say the event to cloud archive And then you know sometime in february we do Liberty dot one update You know and by the time you go to actually do your real deployment We update the packages on you If you didn't have your mirror you would get the latest greatest new packages And it's probably going to work just fine But you might encounter some sort of new bug or some sort of new configuration that you hadn't Encountered or wouldn't have been susceptible if you would have mirrored your packages So you knew you were going to install exactly what you're going to install and One of the things too in the rollbacks is to understand how your automation and everyone's using automation I'm assuming because deploying this stuff by hand sucks But however you're tooling that you're actually using Understand how it works right so If you need to roll back how does your automation actually handle the rollback You don't want to sit there and be fighting with puppet just you know in a scramble like oh my gosh I need my configuration to say x and puppet keeps redeploying and trying to change it to value y right You don't want to do that understand whether you need to kill your puppet Or you know destroy the unit or whatever you need to actually do to not fight your automation How is it that your automation actually needs to be taken care of for the rollbacks? And and that's easier said than done a lot of times packages are really good at rolling forward And automation is really good at thinking about the way to deploy forward But not necessarily backwards even some of the archives that you might have don't don't make it easy for you And validations right so Things look great as you're upgrading and everything may come up just fine You know i've installed i've gone from ice house to liberty and I think everything is great You know all the apis up I can actually log into a node You know one of my virtual machines. That's great. I've got access to the instances things are good right But then I need to go and perform some maintenance on one of my nova compute nodes So I go to migrate an instance off and all of a sudden the migration doesn't work Well if migration is important to me which i'm assuming it is I need to make sure that's built into my validation plan There might be certain parts of the upgrade path that you know change the way that Your tooling works or the way your expectation works on certain things and you need to understand that from trying it out From reading the release notes, but also have a good validation plan in place Like what does it mean to actually have a valid upgrade? It doesn't mean that everything's just up and running It needs to make sure that everything's up and running the way you expect it to and all the things that are critical to you And to your business are actually still functioning the way you expect them to work And you want to exercise as many of these as you can right so If you have a true validation think about the first time you installed it right when you first set up your cloud and you installed it You took a long time to actually validate that your cloud was working the way you want it to work You can't do the same exhaustive set of validation most likely So think about what the critical services to you For your validation so some of the things might be that the api endpoints work the way you want them to Some of them might be that you can actually do the live migration so you can perform maintenance on your compute nodes Different things like that. So just think about what it is to actually mean to validate your cloud Everyone has their own types of requirements on that But you should definitely do it And then You know plan it out think about the actual procedure you're going to do right So the general procedure of upgrading your cloud is Check your cloud health And and that's kind of key everyone's using monitoring software out there right nagios abix whatever you're using Make sure that your cloud is healthy before you do an upgrade The last thing you want to do is do an upgrade have health error reports show up and you have no idea whether it was because The problem existed before or the problem's new since Now you're doing some real weird debugging because you're not sure what the state of it is So really check the cloud health before and understand what the state of it is You know perform your database backup perform your backups for roll up roll back purposes like your configuration management Whatever packages you have that you want to back them up. So if you have a mirror Make sure you create a new archive and a snapshot of your mirror so that no one can change that Then upgrade your services This is the general ordering that's actually provided Both in the open sec documentation and in general Through the grenade upgrade testing kind of goes through this particular order. It doesn't do all of these the grenade But you know, they're working on getting some of the upgrades there And then once you get all your everything upgraded, you know validate it And if everything's valid declare victory, you know, go have a beer Go have a drink go go to whatever it is that you actually do to enjoy the fact that you just went through A nightmare of an upgrade and then start all over So And then, you know after you've planned it out you plan it you plan it you plan it you plan it Test it Test it like you've never tested it before And you can do A lot of testing on it even if you don't have a whole entire environment That's a complete parallel of your current production environment You can find something as close to production as you possibly can You can use your own cloud So most of these services you can actually deploy On to a cloud since you have it running go ahead and create a tenant Deploy your cloud on top of your cloud it's kind of like inception, you know cloud on cloud on cloud But also know where your risks and your gaps are right If you use say a Cisco version of networking But you're running it within the cloud you can't replicate that same set of networking there So you have to understand that that's a risk right when it actually comes down to doing the deployment You're not going to see the same thing Um And that could be true of like say sender if you've got a different back end for your sender back end If you're running like an emcee Back end and you're not running that inside your cloud You need to be aware of that and understand what what what risks there are there if there are any Use the data you can get from your any data and all data you can get from your production environment Just because you can upgrade your database and region one doesn't mean you can update upgrade your database in region two There could be different characters in there. They are introduced weird bugs to the actual database. Um, in fact, I think In vancouver Time Warner actually had a talk about They had a problem where they were upgrading their database just fine and region a And then they went to upgrade their database in region b and it and it broke horribly So that's that's one good example there Um It could take a lot longer Understand what the impacts are of actually rebuilding the neutron ports On a restart and rebuilding some of those for the open v switch So in in you know a test lab when you have 10 ports There's no big deal to actually create it, but if you're building You know 500 ports it may take a significant amount of time to build all the namespaces Appropriately across your your gateway nodes Um, so so be aware of what it is that you actually have and try and get as close as to as to production as you can And ask your tenants to participate It seems kind of weird sometimes to ask your tenants to participate But they're the ones that are impacted more than you are even right their businesses are bad on it Just as your businesses are so um asking to participate help them understand help them feed in when is a good time Help them, you know actually test that their their data is still working afterwards that they're not really getting the impacts that you Would expect them to get I just really asked them to to be part of that um as you're testing Make sure to test the failure scenarios so It's hard enough to just test it that we're doing the upgrade But you need to test all the different types of failure scenarios that you might actually be running into We talk about rollbacks. You prepared for rollbacks, but can you actually do it? Can you do it at different points in your failure scenario? You know, can you can you test the rollback of I've upgraded a different kernel and some different packages? Can I roll back my entire system to the previous level? Or you know, what is it? What is it if it can I roll back half my cloud to a previous version? So if I've got you know, half my services, you know saying back Keystone sender Neutron maybe upgraded to kilo From ice house and then all of a sudden I get A failure on my nova compute nodes or my nova controller nodes Am I going to be able to back level my cloud? Or am I just going to back level that one service? What is my rollback strategy there? Can I do it? um And that's kind of up to you to decide whether to roll back or whether to to stay forward or to just nuke that particular node but Whatever it is that you choose to do like on a rollback strategy there make sure you can actually do it when it comes down to it and then also test validation failures So if you do get it where you upgraded your entire cloud all the way through to You know to the liberty level and all of a sudden you can't validate some of your stuff It doesn't work the way that it's supposed to do. What are you going to do? Do you sit there and debug it? Do you roll back at that point in time? What what is it that you actually do? You need to actually think about What if I actually have a failure after I've upgraded the whole cloud it looks fine But there is a known problem. Is it serious enough to roll back or is it just run with it? test your monitoring so you've You've got your monitoring solution in place, right? Whether it's stack tech whether it's Zavix whatever it is. You need to understand what alerts you actually have set up How is your monitoring going to react to this? In some cases, you've actually moved services from the current nodes that they're running on to other nodes Did you move the monitoring scripts with it? Are you actually still Monitoring and measuring the same things that you're expecting to are there new things you need to actually monitor? any type of new alerts Did you get any false positives? what's going to happen if you you know you need to understand how your monitoring solution is going to work as a whole because You will probably get alerts during the middle of your upgrade and you need to have An understanding of whether it's a real alert or whether it's an expected alert and how to actually handle it and disposition it The last thing is as you've done all this testing raise bugs Raise a lot of bugs find you can find work rounds by if you actually raise a bug with the community There's a lot of great people that actually help you understand. Oh, that's because by you know Function x or bug y and a workaround is to be able to do this And you can either figure out whether you want to just implement the workaround or whether you want to wait for a fix You can get fixes you can make fixes you can share knowledge, which is like oh this particular upgrade path doesn't work Uh, maybe I should do it this way instead And you can really communicate with other people and members in the community to actually help you understand What the best way of actually going about doing this particular upgrade is? And you can help make open stack better And and you know that's really a good thing if if it's better Maybe the next time you actually do an upgrade and you hit a bug you won't hit it that next time, right? Um, and when it comes time to upgrading it you've been planning all this time You've been going for you You know you spent a lot of time at this point in time planning it planning it planning it Um, and so now you should get a pre-flight checklist. This is kind of like your nasa go no go for launch, right? Um, you should be making this during the entire time that you've been testing and planning and figuring out what it is important Um, you know are the right personnel involved? So if you do an upgrade if you've been planning and practicing and practicing and practicing and all of a sudden you know it gets down to game time And the same people that have been practicing it aren't there to actually do it. They're just handing it over that might be a red flag You might not want people doing that um There's there are certain changes in your configuration that will occur obviously Um, and you might want to have that reviewed have someone else take a look at it because you could accidentally Misconfigure it so you've been testing an environment in one environment, which will have different ip ranges It'll have different host names You might have different differences there in your environment that don't translate to your production environment very well That you might actually have to go and make a change for So if you have to do that make sure that someone else is actually Reviewing all the config changes that you actually have to do you don't want to deploy to the wrong network accidentally because Uh, because someone forgot to actually do it actually do it before because someone forgot to review a change Or didn't catch a change that was a typo Are your maintenance notifications enabled are you actually You know alerting your users that it's maintenance time that you're in a maintenance window Are you in a maintenance window those types of things it'll be up to you as far as what your Actual pre-flight checklist is so you need to figure out what is important to you What is it that you actually need to do but have one have it so you can walk through it and understand Am I ready am I ready to actually do it? Then you know you've been practicing for a long time. It's time to go perform the upgrade Execute the plan that you've been doing don't deviate from it Um when when things will happen Unfortunately, you know nothing's perfect. So things will happen that you didn't expect Be prepared for it. Don't panic. You know just relax and be calm Go with the script that you've been practicing all along figure out the exact way that you want to actually handle that Deploy it upgrade it things go wrong. It's okay. You can figure it out And then You know upgrading should theoretically not take too long compared to the rest of this um, so You're at the end of it And you know, this is kind of an agile type way of doing it is you go and you do the reflections What works well what didn't work well start thinking about What made that experience either fantastic or awful? Um, and and what were the real key points about it and feed that back in take time to reflect upon What improvements you can have as a whole? Because you really should be doing it again. One of the things that we do see is that customers will stay out there for You know You'll deploy on ice house You'll stay out there as long as you possibly can until you're absolutely forced to upgrade And one of the things that make upgrades hard at that point in time is because you waited so long Uh, and you're you're crossing so many releases So maybe one of the things that you could feed in would be doing upgrades more regularly build it into your regular operations and in your regular environment Um, especially with the fast movie a fast paced project such as open stack as a as a whole And that's pretty much what I have so happy upgrading out there. I mean it was just a general Talk here. So any questions? Yep, we've got a We'll start here and then you can pass back if the initial deployment is via automated tools Yes, like fuel or juzu Should the upgrade later upgrades? Is it possible to upgrade manually or should one be looking forward to those? Upgrade automated upgrade from those tools Um, so that that really depends. I mean it's really don't fight your automation if you've selected an automation tool So say you've done juju or puppet um You should really try and choose the same set of tools and understand how that set of tools deploys and upgrades I wouldn't advise switching As part of your upgrade strategy and I wouldn't advise doing it manually because like uh, if you want to manually do the upgrade Um, you may manually upgrade to say liberty But the puppet uh manifests that you're deploying is expecting an ice house template, right? So you'd see all sorts of different errors there on your templates that are actually running some of the parameters might not map over properly So I would use your tools to actually do the upgrade there Upgrade cycles of those tools may not coincide with the fast rate product like open stack Right the upgrade cycles of those tools Right may not coincide. So is it recommended to wait? So for like the so say you want to upgrade to a newer version of say Uh, your juju charms or your your your manifests actually deploy it since they they lag behind the actual official release of liberty Or the open stack is what you're asking. Yeah Yeah, um, I would participate be an active participant as they're developing it Help them understand and find the bugs before they fully ship it If you can If you can't then maybe your upgrade cycle takes later. It really depends on what's important to your business and what you're able to do Um, so if liberty's out now and you're the the support for the puppet manifests aren't out until december Um, if you start now working with them Then any bugs you find you have a chance of being fixed before you hit before it comes december Thank you. Yep Can you pass it? Yeah, so it sounds to me like The only real alternatives you have unless you have very few in-house tenants Are rolling upgrades or parallel clouds? Because if your tenant starts upgrading stuff Changing their applications and then you upgrade to liberty and You don't know what the bugs are even right So What's your take on this? How do you? smoothly migrate The tenant applications at the same time as you migrate to a new Open stack version So that's a that's a good question Especially because your tenants will be using the open stack version The api is the different applications built on the open stack versions out there So the biggest thing you can do is talk to your tenants right because your tenants usages and when they're planning on doing the upgrades will be up to them as well And so the best thing you can do is have that open communication that open dialogue with them And so you can help develop a plan that's going to work for your infrastructure for you as well as your tenants because They have their own requirements on that That that's really the best thing I can I can advise typically if you're upgrading the ender cloud that they're actually using You know a lot of the apis are backwards compatible So if you upgraded the under cloud first That would work well for your tenants to be able to do the migration But that that's of a conversation with your with your tenants Have a question. This is a high level upgrade talk I see but in the list of services that you listed out as a sequence on which to actually do the upgrade I did not see rabbit or my sql listed there. Yes. Is there a particular reason you Recommend upgrading them ahead of time Yeah, so I mean I I look at the rabbit and the mysql those are our cornerstone foundations of your overall the cloud deployment Those are things that I looked at outside of the actual open stack services because they're like the glue the nuts and bolts that actually kind of help Provide some of the stuff together Whereas oftentimes when you're upgrading it doesn't necessarily require a newer version Although unless you're using things like the The the heart beating within within rabbit, right? So then I would I would advise upgrading your key components first like your your rabbit and your your mysql And make sure things are running smoothly and then do the upgrade if you can do that so Any other questions? Okay Great. Thanks guys