 Hi everyone, I'm Victor Couturier and I work as a Cloud engineer at Societel General, one of the biggest and oldest banking trends, and this time I will share with you how we achieve a single OpenStack deployment in multiple data centers. So to begin, let's focus on the characteristics of our OpenStack deployments. So for the distribution, we are using an open source deployment of Color and Symbol and we are fully powered by Docker containers or Ansible Playbook. So it is fully community edition. Then we are using the Stein version of OpenStack. So we initially installed it in Queens, celebrating Rokies and Stein, and I'd agree we are a little bit late in OpenStack version, but we soon plan to upgrade to Train on Usury. And in terms of access, it is very important for the rest of the presentation that we are not exposing directly the OpenStack API to the customer. We are using an API endpoint who act as a proxy custom-based for Societel General. So it implements the authentication process of Societel General and the logging framework. But it is not much intelligent in this wrapper. It is just a Societel General-branded API to expose neonatively Nova and Neutron API, by example. So not much intelligence, but it is important to say it exists, and it is this API endpoint which are exposed in the cloud portal of Societel General for all the cloud services or even it is used in our own telephone provider or our API or CLI or SDK. Then for storage, we are using CIF for storing our image or our VM. And for volumes, we are using pre-storage as a single backend. And in terms of deployment, we currently have four or five AITs in three regions around the world. And we count around 120 compute nodes. And the biggest region hosted in Paris is about 80 compute nodes. And it is deployed as much as possible in a full CICD through Jenkins. So everything, every configuration, every upgrade is stored in Git then deployed either via Jenkins or either via the administration container. And for networking, we are using provider network using OVS. So no SDN feature here except for security group filtering, but we are not using overlay network. It is only about VLAN. So the repetition of our deployment are two regions in France and the one in Paris we will focus during this presentation come to availability zone in different data centers. And we count around 20,000 VM creation or distribution per month. And we host about around 5,000 instance worldwide. And the target of this cloud is to host cloud-native applications. So we can find some big data, some web application, some grid use case, our Kubernetes cluster itself or load balancing stuff exposed to our customers. That is only for transform and cloud-ready application. And we contribute nearly 4,000 line of codes directly to the community, mainly in Glantz or Nova project, but in a very useful one like Kola, Colantzible, Cilometer, Itz. And we made another talk earlier this morning about that story. But let's focus on the Paris region and its architecture. So, as I said, we really need to expose the cloud services as a region in Paris with two different data centers. So why not create two different OpenStack regions? Because if we made a little pros of cons, the first pros will be it is really easy. You just need to have three controllers on each data center with some compute nodes and you're ready to go. And in terms of configuration, it is only a new configuration in our CI CD because everything is stored in Git. We just need to add a new directory in your Git repository for storing the new region. So it's not a problem of deployment or management. And we have also Keystone federation features available. So we can federate our project between our OpenStack using Keystone federation. But that's for the pros. But in terms of inconvenience, we have a lot too. And that's big because, by example, we have no capability to replicate our image in the region in all the aviability zone. And we also have no possibility to extend a security group between a region. And in our case, it will be between aviability zone because customers want to consume Paris region as a single region. Like in AWS, when you deploy a web application, your security group can be used in various aviability zone in order to be resilient. So it is a strong problematic for customers to have shared security group in the same region. And it will bring a lot of complexity in this IPI endpoint we only use as a proxy because we need to mask the fact that there is two OpenStack behind the single OpenStack endpoint. So it will be a lot of complexity in our Python code and top of OpenStack. And we want to be as light as possible. And another specificity is between our two data centers. We have a latency of less than 5 milliseconds. And we have no extended layout too. So it is a rooted network. But there is no extended layout too between the two aviability zone. So we decided to make only one extended OpenStack region to expose it to customers as one OpenStack region. So now let's focus on this Paris region. So everything begins with a GSLB instances at the top. So it's just a DNS alias who will equally redirect traffic to the local VIP address. And why is there is two local VIP address, one per aviability zone? It is because there is no extended layout too in this environment. So the local VIP address in case of failure of one aviability zone cannot automatically failover to the other data center because there is no extended layout too. So we have two controllers. We keep alive on VRIP protocol who will failover between themselves the local VIP address and the same in the other aviability zone. And every OpenStack access are unloaded by GSLB who will load balance a layer upper the whole process. So it's very convenient to have two controllers per aviability zone because in terms of maintenance window you can shut down any controllers then the local VIP will failover to the other controllers. And it is very easier to rely on VRIP failover rather than the GSLB instance failover itself because it's not a load balancer. And that is why we have two controllers. And in order to guarantee the quorum in the clusters we have a Gallera habitual components on the third side who assure the quorum of the Gallera clusters. So in case of split brain, for example an isolated aviability zone it will stop by itself because the Gallera cluster will not be reachable anymore. So that's why we just need a Gallera arbitrator because it will not handle the SQL request it will just here to provide the quorum to the right aviability zone. The one left. So with that setup we are able to lose totally another aviability zone. And in terms of compute we have two compute aggregates so it's just aggregate host in over terminology with aviability zone. And we have a local safe cluster in Inge data center because nobody wanted them to run half of the data centers and half of the other data centers. So that is actually for the global architecture. But now let's see how we can deploy that because it requires some tweaks in the configuration files and we see that using co-encival it will be very easy to unleash the ensemble power to perform that kind of deployment. So as we are using co-encival, co-encival is just basic ensemble you can use any features available in ensemble by default. So in ensemble you have group vars notion and you can create group vars in co-encival in order to have different configuration for each group so in our case it will be one group per aviability zone. So let's see how it looks like. So you have an ATC Cora directory with a group vars directory which will be natively unloaded by ensemble and we have two group vars one for the first and one for the second aviability zone and you can see by example a different network interfaces because the VLAN are not the same between the data centers so you have one IC in VLAN 200 and one another IC in 400 network interfaces you have also a different VIP address so its local VIP address can be overridden and you have a node config directory because Cora allows you to override part of an OpenStack component config file by example an OVA or Neutron and Conf and everything is located in this node config directory so with that setup it's possible to have a different node config directory or a different NOVA.conf by example per group so in that case the NOVA.conf is different to target either the first safe cluster or the second safe cluster and it is very useful to ensure that every compute node targets its own local safe clusters so here it is only about configuration file and no modification of code on Cible just plain on Cible so it is very easy to store it in Git and to deploy it on demand so after we deploy that we encounter some issue and the first one was about Neutron because by default when you create a provider network you need to specify the network the VLAN ID at the network level and as I said we don't have extended layer to even for customer VM in that environment so for the VM built in the first ability zone we need to create the first network with its associated subnet and the same in the second ability zone so at the end we have two Neutron network objects with two subnet associated and with that setup there is a strong problem in terms of user experience because you can't prevent user to create a VM in the wrong ability zone in the wrong network by example using the network one in az2 and nobody will prevent the user to create that until the very end of the provisioning process when Neutron will fail to allocate the network and your VM will go to your state and it is a very bad user experience because you need to think about what network is available in which availability zone otherwise you will get an error during the provisioning so to fix that we are using a Neutron feature called Neutron Rooted Provider Network available from Neutron and with that setup it will introduce a notion of segments and this segment will be between the network and the subnet because the VLAN notion will be unleashed by the segment and no longer by the network so you will be able to have one logical object network will be available everywhere in all your availability zone and on this network you will have dedicated segments per availability zone and each segment will contain will contain its own subnet and with that setup you will be able to select your network in any availability zone then Neutron will be able to discover which segment and which subnet is present on the host chosen by Innova and this mapping is done by the configuration so the fiznet attribute you can find in your ml2.conf file in OVS so for the bridge mapping and you just need to insert that fiznet on the definition of the segment and everything will be unleashed by Neutron you just have different fiznet per availability zone and everything is doable with Kola too so we fix the Neutron challenge but there is another one about GLANCE and the GLANCE challenge was more complicated because here is an example of a standard GLANCE deployment based on SEF so you have your GLANCE control plane who is configured you can see at the right to target your SEF clusters and when you send images it will be stored in SEF and when an OVA compute is configured to use the SEF clusters too the magic happens there is a famous copy and write and it will start intentenously your VM so that's a very convenient and common use case when using SEF on OpenStack you want this famous copy and write to spin up your VM very fast and it is working when you have only one SEF as I said earlier we have two SEF clusters in our setup so if we are configuring GLANCE we are only able to configure one store so either the first SEF or the second SEF and it will work for the first IZ by example if we configure IZ1 in GLANCE but when we will spin up VM in IZ2 it will look into its local SEF cluster so the second SEF if the image is available this is not the case so we will download the image each time we want to create a VM from GLANCE control plane and it will not be putted in cache not with a SEF backend so if we want to spin up 10 VM in IZ2 we will download the raw image 10 times from the GLANCE control plane so it is a huge issue and from ROCKY GLANCE introduced the multi backend features because we see we can configure multiple backend so we can configure the first SEF on the second SEF cluster and we can choose on which backend we can send the image either the one or either the two but there is still a problem with that approach even if you can configure multiple backend you need to choose only one and once it is uploaded to one backend it is finished the image is available and GLANCE will make nothing more so you can add manually location but you will need to replicate this image manually and GLANCE will not help you about that so that's why and it was released in USURIL last year we needed to add a new feature to GLANCE to give it as the ability to deploy an image in multiple store it is called the multi store import and it was released in USURIL publicly in USURIL and with that feature we added the capability to GLANCE to deploy an image not only in one store but in an array of store and it will sequentially push the image in your stores so with that full request it was made for USURIL we were able to fix the GLANCE challenge too and from there GLANCE team also developed by example the ability to copy an image from a store to another and for Victoria we added in GLANCE the capability to provision, sync provision the image to be even more faster to provision and upload an image but at this time we required a lot of these features so by pushing called Ansible on Ansible to its limits using rooted provider network in Neutron and developing some code in GLANCE that's how we achieved to stretch our OpenSync deployment we know we ran about 3,000 VMs and 80 compute nodes and it seems it runs quite well so yeah if you have any question I think we can answer it live bye