 OK, so I guess we can start now. Well, hello, everyone. It's a real pleasure being here. Thanks for having us today. During the next 30 minutes, we will be discussing upgrades in general and upgrades in OpenStack. I'm Sebastien, and I work for Inovance. Inovance is a multi-cloud provider. So we basically run, design, build cloud platforms. We have several domains of expertise, among others, OpenStack and SEF. At Inovance, I'm mainly involved in the development team, but I also rotate between the operational and pre-sale team. So Frédéric? So I'm Frédéric Lapierre. I'm in charge of the software development team. So that's it. And I'm Média Bakout, I'm an OpenStack developer on a Zilometer. OK. So these are our details. So let's start with the problem that we faced during upgrade procedures. So this is basically what we got when we have new commerce at Inovance. It's always really difficult to make them understand that we have really specific and really strict rules to manage and to operate the platform. So we established some key principles. So we never, ever log into a server. You use a distributed log to log into every server, so you don't even need to SSH into one instance. And if you want to perform any actions, you can just use software like Capistrano or AMP Collective. You don't install any packages on the system, because at some point, if everyone started to do an APTicket install something, well, this brings inconsistency to your platform. So this is something that you definitely want to avoid. You don't edit manually the configuration files just because everything is centralized and everything is managed by Puppet and Git. So if you want to modify a configuration file, you just edit everything on the Puppet master and then you just run AMP Collective for example to upgrade all the nodes. So that's the way to do it. And then Puppet will restart all the daemon afterwards. In order to proceed to the upgrades, you need something really fundamental and like a redundant platform. So if you don't have any redundancy in your platform, you can't, this is a really basic thing, but you can't proceed to your upgrades. So either you can have an active, active setup using load balancer, which is something really common now. You have an HAProxy and then you have API nodes that you adjust load balancer order requests. Or either you have an active, passive setup. It's basically something running pacemaker and cursing. So one node is active at a time and the other one is passive. And if something goes wrong, well, you just fade over to the other one. Databases must be replicated. So you have several options for this. You can use Galera or MongoDB depending on the technology. So for OpenStack databases, you can use Galera and for the MongoDB, you can use like for Cinometer databases. It's highly recommended to use MongoDB. It's really important to have databases replicated because as soon as you proceed to the upgrade, you want the changes reflected on every servers. You want the upgrade to be propagated. So, and you definitely want something to roll back. And in our infrastructure, the way we do it is that we basically ship servers with all the software installed. So this is fairly easy then to proceed to roll back operations. So let's say you just boot a new server and you bootstrap it and you have all the packages installed and then you want to proceed to the upgrade and something went wrong and you can fairly easily roll back to the previous version. So well, even with a good QA system, you program my rise. So that's something that you might consider as well. And this is why we build such facilities and a tool at Inovance just to make update easier. And that's the tool that Fred is going to explain to you. That's our solution and that's the way we do it at Inovance. So it's one of the answers that we give, but it's not like everyone's answer maybe, but this is how we proceed to the upgrades at Inovance. I'll leave the floor to Fred. Okay, thank you. So the important thing, when you want to be able to do rollbacks, is to be able to... You need to stop using Puppet to install packages because if you install packages using Puppet, it's very difficult to go back, to go reverse and to go to the previous point. So the idea is to separate what you do with Puppet and what you do with your installation process. So the idea is to let Puppet do what he is doing well. So what he's doing well is to manage your configuration files, restart your services, and only do this kind of stuff. That's very important because if you let him manage packages, it will be very hard for you to go back, okay? So what we designed, it's a tool called eDeploy. You can find it on GitHub. It's a tool to manage updates and also to manage the installation. I will only talk on this presentation about updates. The idea is to change the abstraction level to stop working at the package level, but to work at the whole system installation level to be able to do some operation at the system level. And so we separate the system we want to install in two sub-trees, in two kind of sub-trees. One is the data. So what is changed on the system? Could be configuration files, could be your database, could be your logs, et cetera. So this part, we will not touch it during the upgrades. And the rest of the system, that means the programs, the libraries, and all the Python codes on OpenStack will be updated during the upgrade process. So you have examples here of what is in Valib, MySQL, Vallog, et cetera, will not be touched during the upgrade processes. So the idea is that you prepare with these tools, eDeploy your tree in advance to prepare your updates. And depending on what operating system you are using, it can be prepared using Debootstrap or Hume, and some kind of CH-Foot magic. So you prepare everything in advance. You can do some computation between your tree, between your upgrades, and to see what has changed, what will need to be restarted. So you can prepare everything. The installation is done in three parts, just to remind a little bit what eDeploy is doing. It does a hardware detection on the system, send the hardware back to the server, installation server, and then the installation server send backs a configuration script that is run, and then the whole tree that was prepared in advance is installed on the system. So that's a very effective way of installing. And for the update, we use the same trees on the server, the installation server, that is used using Ersync, so very efficiently. You copy the trees, your updated trees to your system you want to update. So for example, let's say we have Apache you want to update because there is a security alert or whatever. And only the Apache part of the system will be synchronized on top of the network. And then you have some scripts to stop the Apache service, Ersync the Apache part of the system, and restart the Apache, for example, if you have an Apache update to proceed. So the idea is you can define what you do by software roles, meaning you have, I don't know, your open stack compute roles, your open stack management role, your open stack object server role, all is prepared in advance. So that's very important. And then you associate this role to hardware profiles. That's how the installation is done. What is very important is that you can store these roles for whatever time you want. Say in two years, you want to reproduce something that you have installed in the past. You will be able to reinstall it very easily. No problem to install the same thing because you don't have to manage Linux distribution repositories. You don't have to store everything to be able to reinstall something. You just store your image and you reinstall it whenever you want. And so with this principle in place, you can also go to one version, to another version very easily. And if you have some trouble, you can erce back the oldest version very easily. And so we design also the tool to be very simple to use, but powerful. So it's only concentrating on the update process and installation process. And very effective because it's only managing updates by using ercing. So only the minimal set of files are compiled on top of a network. And so what is very important when you do updates and you manage your upgrades is to be able to test. And with this process, we test everything in a continuous integration way. We use Jenkins to be able to reproduce what is deployed in production. And that's a very important point that we need to emphasis because that's how we can be the most most assure that everything will be okay after upgrades. So the key principle is that we version everything. So we version the whole installation system. We version the Jenkins job. We are using the same process than in the OpenStack continuous integration system, the Jenkins job builder system to be able to version even the Jenkins configuration. We version obviously the puppet modules and manifest. And also the E-deploy system I already discussed about this, but the unseable recipes, that's more for the orchestration of how you update overalls. Maybe we'll discuss a little bit after about this. How you update in which order you update all the components and how you will be able to be successful at the end. We have still a running system while you update. So with all this in place, what you get is a process that is reproducible because you can test it as much as you want. It's completely automated. It's completely automated because of a versioning of the continuous integration process. And obviously at the end it's testable. You need to use the tests that are available in the OpenStack world. Usually you use tempest, you use your unit test, everything to be able to have a completely validated system. Let me finish. Yeah. Now I will talk about the methodology of the upgrade of an OpenStack cluster. To do that, you have to consider what Sebastian has said before, your architecture design to have high reliability. If you need to update to MySQL schema and you need to do backup because in production you always do backup. So you need to have a configuration management tool and an operation tool. At Inovance we use the pipette to only configure to the file of the system and restart the services and not to install the package because it already does it. And we use the zip to orchestrate the upgrade process. I know I will describe how in OpenStack you will do the process. This is an example of the dependency of each OpenStack component in the API point of view because when you do upgrade, perhaps you need to upgrade the API version of one of the components. And the OpenStack library that does the API compatibility is a PITON whatever client, PITON Novaklion, PITON SwiftPliant that allow you to use, for example, the Kiston V1 API or V1 API or the V2 API. And then you can make a dependency graph of which components need to communicate with each other. And when you have this, you can write this in your uncivil recipe. Then you have the big picture of the progress. You need to see how each node need to be upgraded. The common process is for commandments like Nova API, API is to remove the Nova API node from your load balancer, upgrade it with a deploy, start pipette that upgrade the configuration, modify and restart the service. And then you can read the node to your load balancer. Some component doesn't need that. Like Nova scheduler, you can just start to deploy and start pipette and everything works fine. Your scheduler can work with the new version. The same thing for your compute node. This is basically, but sometimes you need to upgrade the schema of the database and you cannot just do that because many components share the database and you need to stop your service, unfortunately, to make the database upgrade. In the last few minutes, some works have been started to have a compatibility layer for the debate to have a component that can use the old version of the schema so you can upgrade all your open file cluster and then upgrade your schema after and have a really short introduction of your services. But it's not yet ready. Perhaps the work will continue. So for now, you need to stop, for example, your Nova services, so you stop your Nova API, you stop your Nova scheduler, Nova compute, everything from Nova and you need to do the previous process but only for one component of Nova. You do the upgrade of Nova API, of Nova scheduler, Nova conductor, et cetera. And when you have a full chain ready, you can reopen your services on Nova, for example, and you can, after, we add other nodes to make your cluster actually have ability like before. This is the way we are done the upgrade to minimize the interrogation of the service for each component. And because everything is written in a simple recipe, we are no surprise. So, someone from the summary. Well, just to summarize everything, what you need to do and always need to do is just follow every best practices that we just put it in place. Architecture matters also. So, depending on the design that you are going to choose, that's my defer because what we exposed there is a load balance, well, highly available setup. So if you have some other architecture design, this might change your upgrade process. Automating everything is something that you definitely want to do. And you always obviously need to test everything. So everything goes through your QA system. So that's basically everything for the upgrades. And we would like to thank you for your kind attention. And if you have any questions, you can also reach us by email. So these are our details. If you have any questions, no questions. Not actually. On your Twitter, you will continue to use Nova Networks and then you can just stop Nova Compute, do the upgrade process and start Nova Compute. You don't kill your Nova Network, you just restart it after the upgrade have been done and it just flushed your API table and restore your API table and you have a really short network cut. And the customer don't see that, so. Thank you very much.