 OK. Hello, everyone. My name is Miko Lomo-Jvenko. And I'm lead platformer 3 at Adobe at Cloud Division. And we do run OpenStack. So let's talk about upgrades. Because upgrades, like, it's something that a lot of people are afraid of. Because you can't easily switch and do exact and already OpenStack installation. Especially if you're not running multi-issue deployment. And every forum, every OpenStack forum, people are discussing challenges that they faced. So not every upgrade is like the same. They're all different, based on your deployment type, based on your customization, and based on the type of business you're doing. So let's talk why do we need it, and what they aim to solve upgrades. First of all, the key motivating factors for any upgrade should be, let's say, performance, security, and some key features that you miss. And for us, at some point, years ago, the key motivating factor was that release that we used was already not supported by the upstream community. It was end-of-life release. And whenever we need something new, or we had to patch something, so we had to do by our own. Without the help of a community. So another significant improvement was operating system upgrade. So imagine that today you're installing a new operating system and replacing your two years or three years old with a new kernel, new functionality, and, of course, you're upgrading OpenStack. So OpenStack, as well, new release provides you more functionality and more features that you missed. And as well, based on the last two years of experience, it's any upgrade, mostly like any major upgrade, it solves a lot of vulnerability issues. So there are mostly two main strategies to upgrade your OpenStack. One of the popular rolling upgrades. And so it means that you're upgrading your OpenStack environment one by one, step by step. And in place rolling upgrades requires to shut down your services, your OpenStack components, which is not always acceptable for people, while side by side can all be to minimize the downtime, the possible downtime. So from my personal opinion, everyone will find his own pros and cons for any type of upgrade. So it's based on your current situation, on your current specific. But in our case, I do see that rolling upgrades has the fastest way to do any upgrade, because you're basically not relying on anyone else, except on yourself, like OpenStack operators. And there is another good ability that it's not always required before the upgrade to migrate your resources, like VM. If you can afford VM downtime for a couple of minutes, so you can do it without migrating your resources, which is another key issue for some people, because they limited incapacity, or there is no room to do live migration, or no ability to do it. But cons, I mean, that's probably mostly what prevented us to do it this time. It's like service interruption. Imagine a situation when you have like a high traffic. I mean, I'm saying like, gigabits of traffic, like 50 gigabits, and you can't stop it. So you can stop it, but you lose your business. So sometimes, so you need to choose between two e-mails, what you will choose. Choose to lose your business for some period of time, or try to figure out any other solution, any other way to upgrade it. And maybe the most complex upgrade, it's because of DB migrations, and like, yeah. So downtime is key like a blocker for us in this case. And like complexity as well, so components, if you were trying to upgrade something like five years old to something like up-to-date, so you will always have some challenges, because compatibility and complexity, because every opposite deployment is different. Parallel deployment, this way all of you basically do not touch existing OpenStack installation. So basically imagine that you have a data center and the same physical equipment, but what you're trying to do, just basically you're deploying another OpenStack like installation alongside with the old one. And this way of upgrading OpenStack is the simplest one, because you're not scared about like mess up with like DB or any other configuration, because you're existing hardware, it's hosting like your existing workloads and you're not touching it. So this method allows you basically to do like upgrade without service interruption. What does it mean? You're not stopping Keystone. You're not stopping metadata, for example. So basically everything is available for you. You're just trying to like create another release and trying to migrate workloads from an old installation to a new one. Yeah, it's kind of hard, because for this method you need to have like extra hardware, which is not used, that you can leverage for your new OpenStack installation. And still like downtime could be different, because in this case you're basically trying to migrate resources, which is not easily doable like SLI migration. So you're basically stopping something somewhere and then you started in a new place. And probably the biggest disadvantage that you rely on the end users, you need their help to help you to move those workloads. And also it takes too much time. So it kind of releases. So imagine that with your application, when you have a cluster, and you kind of release it for your application, and you want you to do the same for OpenStack, because all your synthetic tests, all your tests that you're doing in your lab, it's not showing you a real situation with real workloads and with real applications. So at some point after validating your OpenStack release somewhere in the lab, you wanted to see how it behaves with the same workloads and product. So you do kind of release. It's pretty much exciting, because you basically run into OpenStack installation, which helps you to measure performance between two. And you decide, is it worth it to upgrade or not. So the simple workflow is just you can deploy all in one node. But the best way is to deploy a controller and compute interconnect networks, because application needs to communicate. Sometimes some type of applications, you need to communication with some dependencies. And you start in VM with production application, and it talks with dependencies or other services. And you see how it behaves. If you satisfy it with the results, you proceed. So you basically can release, can evolve. If you do it right from the beginning, like a separate controller, networking node, and compute, for example, it can evolve into a completely new installation. So you can add HA later. So at some point, if you look like that, one cloud, it looks like one for your customers. But indeed, it's two, three, or more, depending on what you need. So Adobe, at cloud, we choose this way, because at that time, we were not able to afford any downtime. And basically, this year, we did upgrade from Ice House to Mythical Release. The longest time was one month to upgrade all workloads. And there is no impact on your open stack. It's available. Because, for example, imagine a situation when you're trying to do running and upgrade, and you stop in metadata service, and you do orchestration with Puppet, and Puppet relight on your metadata effects. When it's not working, so Puppet just can easily wipe with zero configuration and restart an application. So it's not the best situation ever. And those upgrades, like parallel deployment, installation, all with us, to not stop the traffic, huge amount of traffic. And we didn't do any changes to control plane. For our end customer, it looked like, so we just stopped some amount of applications, not all of them, not all stacks, and migrated one by one. It was hard, because you don't plan for extra infringement in particular DC. You're basically taking one hypervisor, migrating workloads, when it's done, you're reinstalling or plugging it back to the new installation. So it's kind of tricky. But the big bang in this journey, that you can use this technique to kind of release us. You basically can use it for your own evidence and to provide some strong arguments to stakeholders and explain why do you need particular upgrade, what benefits are, what is performance. So you can basically perform on the fly performance of two installations. I guess, that's it. Thanks for dending.