 Good afternoon everyone. Thank you for being here. My name is Alejandro Comisario. I'm the head technical lead from MercadoLibre.com And I'm here with Leandro Reoz, who's also a technical lead from several projects in cloud services at MercadoLibre also. We're really happy to be here and let me take just a moment to walk you through about what is MercadoLibre and what we do. So for those that actually doesn't know MercadoLibre, well MercadoLibre is an Argentinian company. We are the e-commerce leader platform in Latin America. We have presence over 14 countries including Portugal and Europe. We are in the eighth position of the online retailers. 1,900 people work in MercadoLibre and when more than 400 are related with IT. Plus 90 million users are registered in our platform today and since our API went public we are hitting five million's API requests per minute. Regarding bandwidth and throughput, we are handling four gigabits per second in coming to from our users and our more than virtual instances are running in more than 1,300 physical servers today. So after knowing a little bit more about MercadoLibre, I will leave you Leandro to explain you why our previous picture solution didn't scale pretty well. Yeah, actually. You guys might say like the serial guy back there that what a nice deployment, what a nice testing ladder you have there. It's so big, but well actually the eighth world online retailer it's running inside fully on top of OpenStack. Yeah, all our services like queuing as a service, caching as a service, data as a service is running on top of it. And we are running the whole stack. We're using Nova, we're using Swift, we're using Glance, Swift, Cinder, Neutron and all the stuff. Actually, by the end of the year, we are planning to release our first version of our platform as a service that is based on OpenStack Nova plus Docker IO containers. And we're planning actually to use heat and salometers to actually replace our custom API that actually handles today all the auto-scaling stuff. Not bad, isn't it? Yeah, we think that's pretty cool too. But let's talk first about what happened before and what our previous solution, our previous storage solution didn't scale right for us and why we are using Swift actually to replace it. Storing billions and billions of images in NFS volumes actually didn't scale up. And that's mainly because of performance issues, because actually our previous NFS storage platform didn't have the ability actually to add more CPU units or more processing power to a bunch of disks. And that leads to actually a lot of performance issues without extreme operations. So we were able to scale enough to handle all our traffic and all our pictures on the side. DLCDN domain wasn't a secure one. So we were serving HTTP content and secure content over secure pages. Well, you don't need to be one of the anonymous hacking team to know about, I don't know, three types of attacks in one second. That's really funny. So what about the online scaling layer? We have a bunch of proxy that we're serving. Actually, our current application, our picture application that actually acts as an online scaling group. So when a user needs to upload the images or stuff actually has to pass through the proxies to scale them up and to upload it to the site. And actually, that can handle traffic peaks. For example, if in the Mother's Day or on Christmas or something like that, actually all the proxies went down and all the stuff went down and actually you're leading to downtime and an impact of our annual bonus and that's really sad. Actually, another weak point of our previous architecture was that all the metadata of the images were actually stored on database. So for example, when you need to access an un-cache object, for example, in a cache picture, you need actually to hit the database to actually get the storage path and all the metadata to actually grab the image and that didn't scale right for us and it crashes a lot when traffic peaks. So, okay, we were able to handle the traffic peaks, but what about scaling up and out the caching layer we were using in varnish and when we need to add caching layer that we rely on to serve all the traffic, adding more servers were really a mess because the rehashing leads to a lot of back-and-hits and of course, as I told you previously, all the proxy nodes and all the database can handle really traffic peaks. So, basically, we were able to follow business needs and implementing, for example, in another format of images or another resolution and we add another picture to the items were unthinkable at that time. So, picture for you guys, for example, think about all the pictures stored in something like this and you can even see the guy back there administering the whole solution. So, why, which is Swift, why we think Swift to get solved this, this thing's up for us. Actually, because Swift, it's open source, we love open source, we are open source guy, we love to hack, we love to deep dive on the code, we love that and actually, there's no licenses cost-related and actually, you can run it anywhere, even in our fridges. I don't know if you guys know that it's back in South America, our fridges are built on open computer standards. I'm just kidding, I'm talking commodity hardware and the most salient phrase of the last couple of years, there is no vendor locked in. Swift is durable and it's hyper granular. That means that you can handle the availability of your data soon in the smart way. For example, if you need to distribute all your data, you can do it across cages and racks and data centers and all the stuff, so you can actually have that in reality to handle the availability of the data. It's multi-tenant, so basically, you need to publish only one endpoint for every user that want to use the solution actually, so it's really cool and combining that with Keystone is a killing combo actually. And you guys don't need me to tell you that Swift scales. You need just to add more proxies now for processing traffic, for example, or you need to add a bunch of data, not if you want to start concurrency or want to add a storage capability to the solution. To picture this for you guys, all the pictures that were stored on Mercado Libre, when you access the site, all the pictures that you see, all the CSS file and the GS files that actually compose a whole side sheet from the site are loaded from Swift. The whole solution, the whole set is running on Swift. But what about single pointer failure? Well, let's not talk about it, because using a shared nothing approach, actually every piece of the solution is actually and transparently decentralized from each other. That means that every piece into the infrastructure actually knows everything about everyone. That is exactly what allowed us to scale limitless. What about having a raced API? Well, for that is great, because having to talk through the same protocol gives us the ability to speak the same language and make the data flow through the exact same way. Talking about developers, well, developers talk HTTP and rest the whole day. So I bet you can choose whatever language you want and build your Swift clients around it. Yes, it is extremely secure, because by default, Swift actually came integrated with Keystone. So every call into the solution actually authenticated and authorized by the AT&T service. So because we can make sure that multi-tenancy is actually there, amongst other things, you can put also public data through a specific ACLs. And if you want to really scale your throughput and your bandwidth among letting specific cost issues against putting all your data in public clouds, you can deploy Swift in-house and take all the control over it. And of course, there is a huge community behind Swift. You can shoot the IRC channel, the main list, and launch pad, and get for sure the right answer for you. So there was Swift for us, and yeah, there was Swift actually to save the day, right? Yeah, actually to save the day, our jobs and our annual bonus as I told you previously. So as Alex said, you just need hardware, you just need a lot of things, but you just need a bunch of guys, a bunch of, yeah, Hanson Gallagastral Ministry, yeah. So how we deploy Swift? How Swift is our Swift deployment? We are using actually a multi-disk configuration environment. So one disk maps actually to one port and one service, and each drive has its own configuration file. So if you need to tune something up on a particular drive, you can actually do it pretty well, and that's a nice feature. We have about 1.2 parabytes, about 400 existing terabytes need, over three copies, and 256 terabytes need actually available on the platform. It's a hub platform installation, and we have more than 1.4 billion of images already outlawed to the solution, and more than 13 millions of images are allowed to Swift per day. And let's tell me guys that actually all the solution, all the stuff, all the picture from all the items, and as I told you before, is coming out from Swift. Yeah, and also Swift scales linearly. That means that if you really need to burst your throughput and bandwidth, all you have to do is add proxy nodes into the solution, but if you want to scale in storage capacity, all you have to do is add more data nodes little by little so you can assure no disruption on performance. So today actually we are 10x faster than before. Imagine this, serving all the static content today, imagine and multiply it by the amount of projects inside the company, and imagine a couple of web servers with RAM, CPU, and networking attached to it. Well, that amount of hardware is exactly the amount of hardware that we actually saved because we put all this static content into one Swift cluster. And thinking about an open source solution that actually is low cost and allows us to into one single operation, put a single picture and all its version and all its sizes in again just one operation and not having to wait, I don't know, minutes, for example, it was just a dream. But today, thanks to Swift, it is really easy for us to do so and scale out. So having this well-bid and rock-solid solution of Swift into our data centers gives us all the tool to leverage the right solution and make sure that we don't have negative impact to the business, right? Yeah, indeed. It's loud, it sounds like a win-win to me, Alex, if you say that you are 10x faster and we are actually saving a lot of hardware that saves a lot of money to the company that actually maybe you are still scared. You don't know if Swift is strong enough, is mature enough to hold all your company stuff or your company objects. But we suggest that don't be scared because Swift is a really strong platform, yeah? But first, we need to give you a little bit of advice that we actually grabbed from our previous implementations. And our long RSC talks with the Swift people that actually under your C. So, for example, use enterprise-grade drives always for SSD drive, too. You don't want to be calling the DCE all the time if you have remote hand support telling them to change that disk, change that SSD drive, or all the stuff. And use a dedicated high-speed network. We are using low-latency switches because actually we are throughput guys. But if you are, for example, bandwidth guys, you are using a lot of bandwidth to upload large files, for example, you maybe have to add, I don't know, a 10-gig link between the data nodes and the proxy nodes, not only the 10-gigs for the replication, actually the data replication network. And use SSD drive for account and container services. Actually, these services are hitting an SQL like database all the time. So, using SSD drive here is a really, really, really big tip because we tested when we proved that you can actually faster your cluster big time using this type of disk for this type of service. And haste makes ways, guys. That means that actually doing things slowly when you are going to alter your Swift cluster, for example, if you are going to add data nodes or going to add device or going to remove a device, do it slowly. Do it to impact the less on the whole performance stack. And do it like Swift stack, about 5% per time for drive, for example. Our hold and working is actually dominated by fast client. That means that we have lots of throughput and high concurrency. So, imagine that you face with the perfect setup of Swift. Okay. So, what we recommend, at least in our use case that we have, again, lots of throughput, is that you shard against the same account through lots of containers that burst the real speed of Swift. Talking about zoning, using the feature where Swift actually put its copies, preferably against zones, do an initial thoughtful zoning setup, take into account where your data nodes are going to be placed, take into account how your zone configuration is going to be for you to handle failure scenario really well. Talking about recycling, okay, if you have data inside your Swift cluster and you don't need it for, let's say, weeks or month, for example, please delete it. If you delete it, you can free a night node and a night node freed into a disk, into a data node. It's actually a night node that doesn't get caching to RAM. Talking about caching, make sure everything that has to be cached is actually there. Take a tour against the mem caching to your proxy servers and make a little query to see if everything is there, including, for example, Keystone made a data regarding tokens. And check your setup for go services. Again, if you see, for example, like in our use case, you see a specific disk kissing the upper line with a specific graphic or having weird IO picks during specific times in the day, well, you have to really take a cup of a really nice and big coffee and go through your whole sound configuration and your background services configuration to see if everything is not configured just right, okay? So check out the background services, the replication, the auditor services, because believe me, in Swift, everything has an explanation. And last but not least, and since I think I got your attention already, really try to understand what Swift is built for and what it's good for. Try to really understand if it actually fits into your company. Try to understand the technological and economic simplification of deploying Swift into your company. And if it's a really good to go, just deploy it, configure it, monitor it, test it, and well, all you have to do is drink tea the whole day like a sore and relax. So, yeah, thank you so much for being here. We enjoyed a lot of this talk. Hope you enjoyed it too. If you have any doubts or technological in-depth about our configuration and deploy a Swift, you can tackle us as soon as we got off the stage. So, thank you very much. Thank you guys.