 Hello everyone. First of all, thank you for waking up on a very cold and early Sunday morning. Thanks for coming here. I appreciate it. My name is Thiago da Silva. I'm going to be talking about an object storage system called OpenStack Swift. I work for Red Hat. I'm a Swift core reviewer, meaning a group of people that review and maintain the code and merge code. And I've been working on Swift for about four years or so since 2004. I thought I'd start with a brief introduction into object storage in general. Like what drove the need for object storage, talk about some of the use cases and what requirements those use cases lead to. And then dive into Swift, look at the functionality, do some overview, talk about the architecture, and then just answer some questions. So it's going to be pretty intro and brief. So we've all heard a lot about the explosive growth of data that is just being generated. There's been a ton of statistics about, you know, the exponential amount of data that is being generated. And what's interesting about that is that growth, data is being generated a lot, but the rate of that generation is also growing. So it's really an exponential growth. And the second point that's interesting is that what's driving that generation of data growth is mobile applications and web applications. When you talk about data growth, you need to also talk about what types of data is being created. And I try to break that down between two types of data. What I call structured data, and that is data that is organized in a relational form. Think of, you know, really databases, right? It's data that is easy to analyze, easy to query, typically, you know, in a column and rows and in a very tabular form. Very easy to consume and analyze. And we have also what is called unstructured data. So that's data without any structure or any order or schema. Your typical example of that is all the media files that are out there being shared across the web, document files, research files. Think of, you know, from x-rays to, you know, even down to JSON files being shared across the web. What is interesting about unstructured data is that it makes up about 90% of the generated data that is being generated today. When you talk about types of data, you also have different types of storage systems for storing different types of data. We all know very well block and file storage. You have block data that is, you know, divided into chunks of blocks. We know how it's accessed, it's really well understood. File systems, again, very well understood. And you have object storage that is just a logical architecture that is used to manage data as objects or blobs of data. What's interesting about object storage is that when you talk about an object, that object contains the data and the metadata about that data, and it usually contains a unique ID. How you access that data is through a unique ID. So it has a very flat address structure, for example, compared to file storage where you have directories and you have files. Object storage typically is just a very flat, maybe you have buckets or containers to put your objects into, and that's pretty much it. The way you access it is through its unique ID. And the other point that is interesting about object storage is that differently from block and file is how you access it. Typically, you access the object as a whole, so you're either writing a whole blob of data or you're getting a whole block of data. Again, comparing, for example, to file where you might be able to open and append and write parts of it and read just parts of it. Typically with an object storage, you put and you get the whole thing, the whole blob of data. The other distinction that I wanted to make here that I think it's very interesting is that with block and file, you have local access so you can think of your data drive, your local hard drive, and you're accessing it locally. Through, for example, your file system, you have fast file system that are you accessing locally, or you have remote connections to, like your samba share and whatnot. But with object storage, what's distinct about it is it's always remote, right? Typically, you don't access an object storage through a local connection. So what are these cases that are driving the need for object storage? The first and foremost, the typical one that you see are the private clouds, right? So the object storage market was pretty much created by Amazon S3. But today, there's many public and private clouds out there that are running OpenStack Swift. I tried to list just some example of them. These are all companies, you know, OVH is based here in Europe. They've spoken at many OpenStack conferences talking about a very large cluster that they have, running OpenStack Swift and storing a lot of data. What's interesting about this use case is that typically, if you are maintaining that private cloud, you don't know what kind of workload you're dealing with, right? You're just providing a service and your users can be very varied and you don't know. But there are other use cases, too. You have what I called web and mobile applications, for example Wikipedia. It's an example that we always talk about where all the media content in Wikipedia is actually stored on a Swift cluster. Which is interesting to think about that is that I don't think the size of that cluster is too big, but the number of users is just tremendous, right? Turkcell is a red hat customer. It's a public reference that we talk about. They are a telecom company in Turkey and they've talked extensively about some mobile applications that they've built. Something similar to WhatsApp that allows their users to send messages and share photos and share videos. And they also built a pretty massive Swift cluster and supporting, again, millions of users, not only a very large cluster in capacity, but also supporting millions and millions of users. Another use case is data archival. And what I mean by data archival here, it can be either what I like to call active or passive data archival or cold storage, if you think about it. Video companies that are producing or filling on site, they need to upload their footage so that it can be whatever they do, processed production in their production site. So Digital Film Tree, for example, is a company that talked while back about how they were filming on site in Vancouver, for example, and they have their offices in Los Angeles and they were able to use a Swift cluster to upload their data to the cloud and download that data from their Los Angeles offices and speed up how that production of the video happened. Next Cloud also has been talking here this weekend. They offer an application similar to Dropbox where you can just share files across many devices or with other people. So again, all these different applications allows you to put data on a Swift cluster. You can put it into code storage and just back up or you can actively beam sharing content. So based on these use cases, what kinds of requirements does that entail? What can we learn from these use cases? The first one is that you need to provide durability. So from an user perspective who is uploading that data, again, going back to that distinction that I made between block and file with object storage, you're really abstracting the data from where the data is being stored. I don't want to know, I don't care how that data is being stored, but I need to guarantee that it's durable. It needs to, I can't afford to lose this data. So the object storage system needs to take care of that. From a user perspective, I don't know where the data is being stored, what kind of drive, how it's being updated, how refreshed that system is, what kind of backup it does, or if it's using flash or hard drive. I don't know. I'm just putting my data on the cloud and it's to be durable. It also needs to be available, meaning I need to be able to access this data 24-7 if I have to. It cannot be unavailable. That's a huge difference, for example, again, from a lot of times even looking at the SLA for EBS on Amazon or what have you in terms of block storage on public clouds. They provide very low SLA like 99.9 compared to object storage that has many nines. It's always available. It's always being able to connect. Accessibility is really how easy it is to use your API and how accessible it is. That's another requirement that drove the need for object storage in terms of all your web apps and mobile apps and how they're accessing your data. They need to provide a very simple to use API to access that data. Of course, scalability. Nobody starts with 100 petabytes of data. You start small and you need to be able to grow. You need to be able to have that flexibility for your storage system to grow as you grow. You need to be able to grow extensively, but you also need to keep it at low cost. Companies are trying to get away from the storage arrays of old time and they need systems. That's why we have this room. That's why we have SDS. That's why we talk about software defined storage. It needs to be able to be running on low cost hardware. Based on these use cases and requirements, let's dive into OpenStack Swift and just talk a little bit about that. Swift is a highly distributed, eventually consistent, durable object storage system. It allows you to store your data safely and chiply. Talking a little bit about the OpenStack Swift community, it's an open source project. It was one of the founding projects of OpenStack. It was produced by or founded by REC Space. What drove REC Space is that they were a public cloud. They had the need for an object storage system so they implemented Swift and then donated that to OpenStack to found OpenStack. What that means though is that Swift has been in production running a public cloud for about eight years. It's very stable. The API is very stable. The system is very stable. It has a very active community. We have over 700 total contributors. We have about 20 or so active contributors on a monthly basis. One of the things, like I said, I've been working in Swift for about four years. One of the things that I always like to highlight about the community is its focus on automated testing. There's a really huge focus on CI and making sure that the code that we write gets properly tested. We have really a huge focus on how we test every single code that we put in there. As an example, I always thought it was interesting how we merge a recent feature, SimLinks, and I looked at the piece of middleware that we're actually providing the functionality. It was about 300 lines of code and look at the test and it's about 2,000 lines of tests. It's like a big difference. I think it speaks to the focus of the community in terms of making sure that you're writing good code. It doesn't mean that we don't have bugs or anything, but we focus a lot on that. It's a very friendly community. It's been a lot of fun to work on those folks. Quick overview. You access Swift through a RESTful API. It's all through HTTP verbs. You get, you put, you delete, you post. The object key, the way you access a given object is through a URL. So differently from other object storage systems where you typically put your data and the system gives you back a unique key. The user itself is who defines what that key for that object is. So the key is just a URL. You have the, like I talked about, the flat address space before. You can group objects, a collection of objects into buckets or containers. And you can group containers inside a account. And think of account more like a bank account where you put stuff in as opposed to like a user. It's not really, an account is not really a user. It's more like just a group of containers. So it's just a way to organize your cluster across multiple accounts. I mentioned already before it's a highly durable and distributed storage system. You're able to load balance your cluster and you replicate the data many times to maintain data durability. Or you use erasure coding. But the point is that there's no single point of failure. Your system can be depending on how you deploy. You know, you can deploy in such a way that you don't have a single point of failures. A interesting distinction about Swift is that it's an eventual consistent system. So it's designed for high availability and partition tolerance, sacrificing a little bit of consistency. The myth that I like to dispel about eventual consistency, that's the number one thing that I hear a lot, is that a lot of people equate eventual consistency to eventual replication. And that's not true. So eventual consistency does not mean eventual replication. It does not mean that Swift, when you put data into Swift, you store just one replica and Swift will eventually replicate your data. Because if that was the case, then you'd have a window of time there that you might lose your data. Because you might have just one copy and that disk might fail. That's not the case. Swift will always store your data durability with multiple replicas so that when a system goes down or something, your data will not be lost. Shameless plug. There's going to be another talk later today where it's actually going to dive a little bit more into how eventual consistency can actually play a really nice role into the Swift architecture. And that's when we talk about, for example, global replication. So when you have multiple data centers and you need to have a cluster that goes across or expands across multiple data centers across the globe, that's where eventual consistency really shines. Swift has a very rich API exposing a lot of functionality. I list some of them there. I'd like to highlight just a couple of them. Object versioning is when you're able to store or if you have a workload that overwrites objects, you're able to maintain old copies of that object. And object exploration is when you're able to store an object and set an expiration time on it and it will be automatically deleted for you. So the system takes care of that automatic deletion. SLO is talking about or is a feature that provides large object support. So similar to other object storage systems, Swift applies an artificial limitation on how big your object can be. And that limit is typically five gigabytes. And that's really artificial just so that you don't have people trying to upload using HTTP, you know, a terabyte object. It wouldn't be a good idea to do that. So we put the artificial limitation of saying five gigabytes is the maximum size, but using a feature like SLO or static large objects, you can segment your one terabyte object into smaller chunks, upload them separately and then provide a manifest file and Swift will concatenate your objects for you. Temporary URL is another feature where, for example, you upload an object and you want to provide temporary access to somebody else and you don't want to provide them your authentication secrets. You can produce a signed temporary URL that you can hand it to them. They can use to get that data and that URL will expire after some time. Swift also provides support for different types of authentication systems. Temporary URL is one of them, Keystone and OpenStack, FormPost. There's also going to be another talk by Christian today talking a little bit about how to develop your own apps and he'll cover a little bit more about authentication. Let's talk a little bit about Swift architecture. Swift has a very modular architecture that allows for the flexibility when you're scaling it out. Covered here four main components. The proxy is your main gateway to the cluster. So that's how users will send their requests and the requests are received at the proxy and the proxy will then proxy out those requests to the backend storage system. The backend storage system is made up of the account server, the container server and the object server. The account server and the container server. The account is really holding just a list of all the containers for that given account and the container is just holding a list of all the objects for that object for that container. In some matter of data, the account and container server are typically data is typically stored on a database, on disk and it's replicated properly and all that. And the object server itself really becomes kind of the most important part here where that's the service that is storing your data on disk. What's interesting about this modular architecture is that you could have those four services running in one machine or you can have them run in separate machines independently of each other. They are just web servers running or implementing the WISG interface. And the communication between them, so not only the communication between the user and the proxy but the communication between them is also through HTTP. It's giving a little bit more of the flow of the data here. So imagine for example that a user is putting an object, is writing an object. It sends that request to the proxy and what happens is that the proxy will then replicate that data. So we'll write three copies of that data and load balance and distribute it across the cluster in such a way that the algorithm makes sure that you're writing the data across different failure domains. So either the failure domain can be a rack, a node or even a whole data center. So if you have multiple data centers you would make sure they distribute your data across two different data centers or if you have multiple racks in a data center you would put in different data racks. The other distinction that I like about this slide is that you can also break up your cluster between what we call the access tier and the storage tier and it allows you to scale them separately from each other. So the access tier is going to be the proxy and it's going to be very CPU intensive. So if you need to scale out bandwidth you might just be able to add more and more proxies. But if you need to grow capacity you might be able just to grow your storage tier and add more disk and more storage capacity as opposed to having to add more proxy. So that's a really neat thing about this architecture. But you can't talk about Swift architecture without talking about the ring. The ring is really about how Swift is able to determine where the data is stored. So the ring is just a binary file that is using a hashing ring to be able to determine where the data or where an object is stored on your cluster. So that's something that you produce the ring when you're as an operator, when you're deploying your cluster and the way it's used is when a object request comes in or container request comes in. The proxy is going to look up, is going to hash the name of the object that you need ID, the URL and from that hash it can look up where in your cluster the data would be stored. So the location is deterministic by using the ring and that guarantees that no single point of failure. You know you just, every node in your cluster is going to have that ring and you can look up that information there. Another interesting point about the Swift architecture is what we call storage policies. So we have the support to store data across different data, storage rules or replicas we say. So I talked a lot about three replicas to give an example, but you could define really pools of data, kind of break up your cluster into pools of data and you might have data that needs to be stored with three replicas or two replicas or a rigid coding. You can also use this to say, let's say you know I have a very, for legal reasons or whatever this data needs to stay in Europe. You cannot go to the US for example, you cannot be replicated to the US. You could also define a pool that maintains that data just in your cluster or your data center that is in Europe. So using storage policies you can do that. And that's an operator defined role that the operator would define. From a user's perspective, the way you would use storage policies is when you're creating a container you define what storage policy that container is going to have and then every object that you're going to put inside that container is going to use that storage policy. So you could have, you know, a policy of flash drives for example, a very high performance or, you know, what have you that can be defined by the operator. I mentioned the four main components of Swift, but there's also what we like to call the consistency engine that's really background demons that are running in the object stored system themselves and just maintaining that eventual consistency. It's making sure that data healing is happening properly, bit rock detection, the updates when you put an object make sure that the container database or the account database is getting updated, expiring the objects. So I gave an example there with my little diagram when the proxy is trying to write to those three replicas and is not able to write one of them. It will actually write to another place to guarantee that durability. So it always will try to write through replicas even though it wasn't able to write to its primary location and eventually it will heal and move that data back into place. So that's all done by that consistency engine. That's it. Maybe at this point you're thinking, yes, Swift is cool, right? I'd like to contribute. We have a lot of work ahead of us. The work continues. Right now we're focusing a lot on the scalability of Swift. So container sharding is an issue that we have where there are users out there that are trying to put billions of objects inside the same container and that you get what we call deep containers and you need to be able to shard that container. Lots of small files. Again, 16-byte files or objects inside the cluster can become a problem. Data tiering is just being able to move when I talk about policies, being able to move from a three-replica policy, move your data automatically from a three-replicate policy to a regecoded policy, for example. That's another feature that we're working on. We also have a project called Humiburg that is implementing or reporting parts of the system to Golang and that's for performance reasons. So there's really a ton of work to do there. And that's it. You can chat with us if you want to or you can contribute. You can find a code on GitHub or you can find us on FreeNode if you have questions if you want to talk about. And that's it. Any questions? Any questions? Yes. What is it? The priority. So what is the priority of the project Humiburg? So again just to restate the purpose. The purpose is to rewrite parts of it in Golang and the priority I would say is very high because again it affects the scalability of the system but right now we're kind of prioritizing deep containers a little bit higher than that. So it's something that we're working on. We have actually patches out there that builds the foundation work for being able to port. And we're really just talking about the object server that we're going to port and maybe some of the data consistency background demons. So it's important. It's very important to us but there's more important work like the deep containers that we need to tackle first. But it's ongoing parallel work that is already ongoing. I thought I saw here and there. Being a corporate of OpenStack I understand it means that the release schedule, the release dates are tied with the releases of OpenStack or does it have its own cable? I don't think so. Does Swift have a separate schedule from the OpenStack cadence? I guess you can say that we do. So actually a lot of OpenStack projects are following sort of their own cadence where they're able to just release their own component and when there is an OpenStack release it kind of gets bundled together and we just tag a specific release that we did, the last release, and we say this is going to be the Queen's release for OpenStack. You actually bring up a nice issue which is Swift typically actually gets called out in a nice way nowadays that you're actually able to run OpenStack Swift outside of OpenStack. You don't need a whole OpenStack cluster to run Swift. Swift is its own storage system that you can run without any actually OpenStack component. If you have your own authentication system you don't even need Keystone. You can just spun up a Swift cluster and it does not depend on OpenStack other OpenStack components. Can I change the policy in great time? Yeah, so going to repeat, make sure that I understood your question. So if you have a three replica system or policy and you want to reduce that policy to two replicas for it to lower the cost. Yes, you can, without downtime. Without downtime. Yes, any more questions? All right, cool. Thank you.