 Hello, my name is Sakari, I'm COO of Codership and I'm so glad that I flew all the way from Finland to OpenStack Summit because I have had very good discussions with the people on our booth and people come by and they ask what has Galera to do with OpenStack and then comes next fellow to our booth and explains to me and us that they have implemented Galera cluster to the Keystone with the NOVA with the other components of OpenStack so it's been a really good summit for us and hear people talking about Galera cluster. I want to let you also hear about that story so that's why I have invited here Mercado Libre which I introduce to you shortly but let me just explain quickly what is Galera cluster. Galera cluster is a multi-master MySQL cluster. You can read and write from any of these nodes and you can add several nodes on the fly and you can have the elasticity in the cloud adding more and more nodes. The replication is synchronous and client can read and write from any of these nodes. So the benefits of using Galera cluster is that there is no single point of failure. There is no data loss, no slave lag, it's transparent to your application and it also works over wide area network and cloud. And now I would like to introduce Mercado Libre, Max and Dario to explain how and why did they implement Galera cluster. So please welcome Max and Dario. Thank you. Thank you. Thank you, Sakari. Okay, hello everyone. My name is Max and this is Dario. We were invited by Codership to show you how and why we decided to use Galera as our DB replication system for our cloud databases in Mercado Libre. If you don't know us, Mercado Libre is one of the biggest e-commerce platform in Latin America. We are present in over 14 countries. We have more than 90 million registered users. We have more than 11,000 virtual machines running on top of 1100 physical hosts. This whole infrastructure is on top of OpenStack. We are using it from the very beginning. We are one of the biggest companies that put it in production. And this makes us an interesting use case for Galera. Okay, so Dario. Okay, so as Max pointed out, we have a big infrastructure in order to meet our requirements in terms of a solid DB backend solution for all of our OpenStack services. We had a few requirements that were actually based on logical and common sense. First, it had to be highly available, of course. Then it had to be resilient. It had to be robust, stable. Also, it had to perform. If you have performance issues in your DB backend, you're going to most likely have a performance degradation on your services. Also, we have an infrastructure that keeps scaling, that keeps growing. So we need to be able to have a solution that scales with us so that it can deal with our growth. And also, it needs to be simple to maintain. If we keep scaling, our solution needs to scale easily. So moving on to the next slide. This was the first approach that we put into production. It was a pretty simple one, but effective at some point. It consisted of two physical MySQL servers on an active, passive layout. Only the master node is the one taking traffic at this point. Both servers were monitored and actually their server resources monitored and managed by Harbit in case of a failure on the active node. And we had the MySQL data tier of the servers was mounted on top of an ISCSI volume, which was provisioned by a storage appliance. Okay, so we had a few drawbacks with this model as long as we kept scaling. First off, being an active, passive model, that means that you cannot scale vertically. You cannot scale horizontally. You can do it only vertically, either by adding more RAM or changing your hardware specs, and that is a problem. Also, DRBD presented us not a bottleneck penalty in terms of right performance, that was not good as so. But the worst of our problems was that if we eventually had an issue of our master server, of our active server, the failover process took too long. And that translates into downtime in not any of our services, of our cloud services. So we weren't able to do anything with our instances. We weren't able to create, delete, reboot, create volume, stop, anything. And also, we have Keystone running on top of this. And if you don't have Keystones, you don't have authorization, authentication of developers can do their work. So it was a no-go. So then we moved to our next approach, Max. So our second approach was the following. At that time, back in 2012, we were researching for new solutions that could provide active master servers that would replicate and be synchronized. So researching, we found Galera. It seemed like a promising project. So we put it in production. The first approach with Galera was in a virtualized environment. We did this the following way. We put several virtual machines in our cloud. We put them behind the load balancer. These virtual machines will store the database files inside their ephemeral disk. And we also had a node of reference, which if you see the code sheet documentation, you will find more information. So this node of reference had an NFS volume attached where we store our databases files. And it didn't receive any traffic. So this solution was what we needed because we could get client transactions, both read and writes. We will balance them across all our active master servers. And they were replicated and synchronized with Galera. But we found out that our approach, this first approach, wasn't good enough. We had an IO bottleneck with the virtual machines because we were using the local virtual disk. And it wasn't good and fast enough. And also we were concerned about having the cloud databases and services itself stored inside the cloud. So if there was a worst case scenario where we needed to boot up the cloud from scratch, we needed to boot first these virtual machines to bring up the database and cluster and then bring the cloud, the rest of the cloud. So it wasn't good enough. So with these problems in mind, the IO problem and the NOVA DBs within NOVA paradox, we moved on to our final approach, which is the one we currently have in production. And it's pretty much the same layout as the one that Max showed you just before, this one. And what we did was to move out of the cloud and replace the instances for physical servers. And after we did that, we noticed a huge increase in performance. And we actually nailed it. This is the scheme, the architecture that we currently have in every data center we have. And our largest cluster is consisting of eight physical servers. And we have at least one of these clusters for geographical region. And it works great for us. So what did we get in terms of advantages from adopting Galera and using this layout, this architecture layout? Well, first, every server is a master itself. That's because Galera is a multi-master cluster in solution. That's great. And also because of that, any server can actively take traffic. And the load balancer makes your life easier. So you can also easily scale horizontally. You can easily add a new node when you need to scale. You add a new node to your Galera cluster. You add it to your load balancer and you're good to go. Being an active only solution, there's no failover involved. So great as well in terms of that. Minimum to no downtime on server failure is you have a robust enough load balancing solution. In the eventual case of a node failure, your load balancer will take it out of rotation and your cluster will still be in good shape. It's really, really easy to implement and manage. It's really easy to add a node to a cluster, to remove it from the cluster. Or even if you have any problems with one of your nodes, it's very easy to rejoin it to a cluster. It's great. And also, which is very, very important for us, it's very easy to monitor and measure using custom metrics. In terms of MySQL, you can use Galera is actually MySQL in the bottom. So you can use your standard monitoring solution and Max. And you can monitor it like a server. As you can see, we have beautiful dashboards. And we can see actually what's going on in our cluster. What's the health? What's the status of it? This is very important for us. Why? Because we're really using it in production and we are using it to store all our cloud database services. Okay? This is why we wanted to show you to you that you can really use it. And what do we have there? We have more than 60 regions and counting of Nova databases. We have Keystone, as Dario said previously. We use Keystone a lot inside our company for internal API authorization and authentication. We also use it widely with our Swift clusters. Okay? So Keystone is key for us. We store it also in Galera. We have also, you know, glance services in their volume databases, Neutron. We also store our database for our custom provisioning API. We have a smaller and simpler, smaller separate cluster for our CIVIX service, which is our monitoring tool. We have around 4K queries per second in Average. We reach peaks of 20K queries per second. And we have no problems. What's good about this solution is that both we are using an open-source software, which is working really great for us. You can also use it on your Spark nodes or any hardware. You have a commodity hardware and it's working really well. So this is everything. We really thank you for coming here and hearing us. And you're welcome to contact us if you need anything or have any questions. And hey, guys, don't be afraid of trying Galera in OpenStack. It worked really well for us and it scales wonderfully. So thank you. Thank you. Thank you, Sakari. Thank you.