 Hi, welcome to Scaling Out eCloud control. We are in a fully deboubled mobile ecosystem. We have taken Android open source project and we have removed all calls and dependencies to Google services. Then we have added MicroG which takes and replaces the presence of Play services and we have added an application store so that people can install the applications they miss from Android and we have also included default set of applications which is customized and integrated. Then we have eCloud which is our cloud services next cloud for storage and data and we have a mail server and a search engine. We also provide an open source and self-host option so that people can install all these in their servers and retain full control of their data. This talk is about eCloud global version one which is the first server that we set up after our crowdfunding campaigns and it was a success going from the MVP to 10,000 accounts and over 7,000 rates of data but it was also done in the same server next cloud and mail and it was proving to be quite a maintenance nightmare. We also have some scale issues in terms of storage because we could only use up to 40 devices of block devices provided by our data center and we had to put them together with rate which every time we run out of space would mean taking it out of the server and preparing that all again. We also had to do some fine-tuning to linux and PHP, my SQL configurations because we were facing slow runs. So we started a project called V2 in which we envision why would be the ideal architecture to serve all our global user base. We thought of a geo-distributed and redundant architecture for all services. We would be backed by a Minayo Optic storage and it would use LDL for single sign-off, gallery cluster for the database and it would be using extra end-to-end encryption and deploy it using Docker swarm or not. But on the way we set up a more realistic target which we call version 1.5 and this would use aproxy to load balance to three different next-log instances. Only four node Minayo cluster and master slave replication instead of a regular cluster. All the communication between the nodes would be protected by way of your VPN. Then came some new requirements because we asked for server-side encryption and next-log 16 to next-log 18 upgrade and we also discovered that it wasn't possible to use the object storage as a main backend when server-side encryption wasn't able. So instead we decided to use SET as shared storage and distributed storage which was not easy to set up and fine-tuned but it sort of went it up, then it's very powerful and it allows a lot of flexibility. Then the problem was that when we were ready to go live we detected that we had a major performance issue with our network storage and it was apparently solved by using only a block device which unfortunately could not be mounted in multiple next-log instances at the same time. So we had to drop that and drop the load balancing and use instead only a primary and a failover instance that we could manually switch to if we have some problem or some schedule maintenance. And we thought this could work because the new servers each of them is more powerful than the old one and we are moving mail out to a different machine and there are high chances it could work. We have like double the run and everything. So let's go live with this and listen up. We made some progress. And we did go live with that on July of this year and it was a very long migration, quite hard but eventually everything was up, everything was set up and working and we waited. And when the traffic came, everything was done again. CPU was reaching 100% every time for load and there was nothing we could do. Every time we tried to bring it up, it was full. We checked every setting, every component we disabled encryption between PHP and Jinx to know of it. And we tried to bring the traffic progressively in to see where it went from because we were seeing huge logs in Jinx, like a big traffic. So we were even blocking self-ending IPs who were kind of under attack. And in the end there was no way to bring it up even for our enemy users. So we have to revert back to the previous machine which also would not work. And what we did was simply resize it because that's the final clouding instances and we had it like four times more powerful and eventually the service was restored. Good thing, all this is at the least we managed to sneak in some of the improvements of the last months of development and also of the specific migration, all these fine tunings that we have been discovering. So the R&D project continued in order to be able to use this infrastructure and new servers that we are preparing. We managed to fix the self-test performance by adding SSDs into the mix by journaling or specific pools for mail. We also did a lot of measuring using different web doc tools comparing the old infrastructure and new one. We also measured the drives, the drive performance of the old OVH blocks and the new self-drives. And eventually we were very confident that the infrastructure was actually better now that the old one. So we set up on a second migration attempt which was this hours and this time it worked and this is how it looks now, hours set up. We have true load balancing. We have three different extra instances. One of them is primary because it has the right and able database. The other two are replicas so that we can take snapshots of them for instance. We have Redis also in one of the secondaries. We have separate mail host which has the SSD pool and we can map any arbitrary directory into a faster storage. So that's very flexible. And what did we learn from here? We learned to be data driven to test every component by itself to also test it all together to see how it performs and to make only one change at a time because every change can potentially impact all the measurements you've been knowing so far. So that's it. Thank you very much and enjoy the conference.