 Thanks for coming everyone. This is the Swift 101 intro and architecture for beginners So if you want to know what Swift is how it works, maybe even how it might look in a real production deployment Who's using it today and all that kind of stuff. You're in the right place Go back one go back one. My name is John Dickinson. I'm the project technical lead for open stack Swift And I've been involved in the project since it began and I work at a company called Swift stack I'm the director of technology there and with me today is the CEO of Swift stack. Hi, I'm Joe Arnold CEO of Swiss stack John and I met because we started working on some of the early Well, John was on the original project team for Swift at rack space And I got a chance to work with him on some of the early deployments right after Swift was open sourced And so what we did is we formed a company around building making Swift great at the same time Building a product around making people making it easy to deploy scale and manage a Swift environment and what Swift is if you're not familiar is why we're here. It's why we're here It's an object storage system So if you're familiar with Amazon s3 or rack space cloud files, which is Swift That is what that's what that's what the system is and that's what we're going to be talking about today Okay, so one of the really thing exciting things or things I'm excited about this week is how much there is about Swift all week long And these are some of the sessions that are after this one So not even counting some of the things that went on yesterday We've got at least nine more sessions sometime during the week on Swift itself And a couple of interesting things here is if you are somebody who's interested in hearing what other people are doing in Production with Swift we've got some case studies that are coming up for real production users If you are interested in the development side and say I need to extend Swift and add some some of my own Functionality we've got some things on there like the middleware stuff and one of the ones I really wanted to point out is on Thursday. We've got a workshop on deploying OpenStack Swift just a short little session you can show up there and Well, this one's a different one And and you can and you can do that But we also have a full-day workshop on Thursday a little bit off-site of the summit But very close at the Ellis Hotel it's a map here and you can go to swift workshops that event right calm and do that That's gonna be a really really deep dive full day long into building OpenStack Swift and getting it running on your own and really playing with it You know playing with failure conditions talking about network diagrams all of that kind of stuff So if you're really interested in doing in learning Really deep information technically about really product putting it in production. I would really suggest you go to this And finally we've got a book so we've got in a rally book that is OpenStack stores with Swift and it is available today and Every day at the Swift stack booth just right out here We will be giving away a hundred copies and you can pick up yours for free If you show up if the first hundred people to show up otherwise Go to swift act comm slash book and you can get the link to order it online and we have we're doing more There's book signings today at 2 45 and then tomorrow at 3 30 if you want to swing by the booth Well, there's gonna be an extra batch that we're gonna reserve for that So let's get started. Why are we here? so when When when I was when Amazon first came out I was happened to be at a place where we were doing Ruby on rails and deploying Ruby out applications and we built this platform service on top of Amazon and It completely changed the game because now people could build applications. They can deploy it They had API's as a service you could you can get in for infrastructure via an API and It really changed the games it changed the games for the economics and it enabled developments in organization to have a lot of agility and then specific to storage what happened was the the object storage services that came out as part of the cloud cloud infrastructure meant that you could have lots of Applications all attaching themselves to the same pool of storage and you had this multi-tenant storage platform rather than having an application with a dedicated storage environment and And that's what that's what cloud storage enabled that's what first S3 enabled and now that technology is available for people who want to have private environments of their own object storage system and The reality of the world is that as applications have changed we have seen a massive massive increase in the amount of data that people actually need to store and unstructured data is what is Dominating that increase in storage itself. So what is unstructured data? Basically, you think of it as just the common everyday stuff you normally use like documents photos images Web content if you've got applications that people are using on their cell phones and taking pictures user-generated content Playing online games that sort of thing where you've got a little piece of information you need to store sometimes a large piece of information You need to store that sort of data is just exploding as far as the amount of data. We've got a store and so The crux of the problem is that you can't avoid solving this problem You've got to have something that's going to be able to store your data and also scale and so when you do that You've got a couple of choices But the reality you have to have something that is on more than one computer someplace you have to have a distributed system So when you're solving distributed problems, you've got a choice to make You can do one of two things You know you always if you've got computers talking to one another You know you always have to be able to handle some degree of failures You've got to be able to hack handle what happens if for example a network connection dies or something times out But the answer the question is what do you do when that happens? Do you go for something that's a really strong consistent views like I know exactly what's going to happen? and I know The the entire system has the exact same view of the data at all times or Do you say you know I'm gonna relax that a little bit and I don't have to have the The answer right now, but I know I'm gonna get it, but I can go ahead and always respond to requests so this is summed up in something called the cap theorem the CAP that you see there and What it says is that in a distributed system you have to make a choice You have to choose two of these things and you can't sacrifice the partition tolerance What this means is that you have to make a choice between an eventually consistent system and a strongly consistent system And both of these systems have extremely good use cases one is not better than the other So for example, it's incredibly important that in a block storage system You have something that is strongly consistent You don't want for example the underlying storage underneath your boot volume or that's underlying your database to suddenly change Because something came back online later on But that's not the case with this massive set of unstructured data that we see it doesn't really matter that That your mom's picture that she uploaded to Facebook is actually the same thing as your cousin's picture that he uploaded to Facebook It doesn't matter that the strict ordering of those Is always enforced so you can have something that's eventually consistent and this allows you to have a very highly available system It's always going to be able to respond to your requests and you're going to have something that's going to be very very able and easy to scale and So that leads us to the question of well why Swift Swift is an eventually consistent distributed storage system And what this gives you is something that is able to scale globally. You can have one cluster with Locations all over the world it allows you to have simple API access for applications So it's easy to consume and application developers can actually offload the hard problems of storage to the storage system itself I've got a great story around this and I'm gonna talk it about in just a little bit and from the IT side What's really cool is that they can now get away from those silos that Joe was talking about and Start working on pools of storage which gives them both the cost savings of being able to share share storage across the organization But it also means that you can They can they can respond in a very agile way They don't have to spend a lot of time provisioning and carving out pools of storage for a particular application And then worrying about how to scale that instead They can just treat it as a service IT can manage a service Developers can consume a service and the final most important thing that I think is incredibly important And it's why we're here this week at the open stack summit is that it gives you an ownership of your data Everybody has data and you need to have ownership of everything that touches that all the way from the hardware to the software That's doing it and the tool chains that the actual tool chain That's accessing the data and the only way you can do that is something that's an open system That's one of the things I think is really important about open stack and something that Jonathan Bryce was Really mentioned and hit home in Hong Kong last fall Saying that the key part about open stack is the open We've got an open system that people can participate in and actually influence the direction And that's what gives you ownership of the things that touch your data So who's using swift? There's a there's a lot of people using swift and What's really unique about swift as a storage project is that the people who are working on it the people who are Contributing to it mirrors what's happening in the rest of the open stack so what that means is there's lots of companies that have built businesses that are providing services that are contributing and are part of the swift community and Contributing swift code and so it means that when when functionality is being developed. It's being developed by Multiple people multiple companies all contributing towards that effort as a result I think it adds a lot of comfort for people who to pick swift as a technology because I know that there's gonna be a broad ecosystem to support Let's talk about real-world use cases see other people, but how does it fit you and how does that actually map? There's one use case. I want to talk about here first. That's a pack 12. You may know them as you know that sports Group collegiate sports group if you're like me, it's go sports But what's really kind of cool about this is their use case is that they've got literally hundreds of video streams They've got to do they're basically a TV station and they've got to record a bunch and broadcast a bunch of Sporting events every year something on the order like 800 a year or something like that So their story was that they had a an existing sand that they were putting all of their data in just great You know say hey, you know, I've got a lot of video. I need to throw it in my sand. It's gonna be great well, they started running out of room on their sand as one does and What do you do? Well, they archived some of that they put it on tape and they put it on a shelf Great. Well, they're still running out of space and it's getting kind of expensive. So what do you do? Well, they built a swift cluster and they were able to migrate all of their data that was in their sand to their swift cluster Which is really great because it allowed them to really lower costs But the really interesting thing and the thing that's really exciting to me about it is that it's not just cost savings They were because they were putting their video content into a highly available system It actually enables the opportunity for new revenue models for them So because Swift is able to run on commodity hardware and just you know cheap white box is someplace It means that the marginal cost of them taking their archive tape video footage and putting it into their highly available Swift cluster is less than the potential revenue. They have of actually doing that So the really cool idea is thinking you know what I want to go see that 1994 football game Well now if somebody's doing that they don't have to think about well What tape is that on and how do I do that? You can't monetize that but they can actually say now with a highly available system Look, we can do this and we can charge somebody $5 to go watch that or something like this So I'm really excited to to watch them and see how they're gonna be able to take advantage of this new sort of highly available system And we have two more two more use cases right after this one on the third floor Fred Hutchinson is gonna be presenting a use case where they're doing HPC and providing an economy file store So you can hear the nitty-gritty of how they implemented it Georgia Tech tomorrow at I believe Time ten o'clock. Sorry the time is on the beginning one and we're gonna talk about their use case as well So there's two more use cases that are in the program at the show that you can go watch And one thing the last one I want to talk about this is something I heard about once I got here to Atlanta This is really kind of cool So there's a site out there called adrift.org.au and it's a really kind of fun site You click on any place in the ocean and it shows you what the dispersion pattern for ocean debris is over the next ten years It's a great scientific thing and really important for understanding how our world works But you may have heard that there was an airline that went to an airplane that went missing and it was apparently a big news story And they were looking in the middle of the Indian Ocean and figuring out well if this crashed Where would the debris be it turns out that the Guardian found this website and promptly swamped their servers because you know slash dot effect essentially and so They were looking at it and say well, what what can we do here? Well, we can scale out our engine X servers. We know how to do that That's a that's a known a known thing, but you know, it's actually operational complexity and things like that And they realized the data that we have on this is already being stored in Swift What if we just Through that data directly to the browser. What if we let the browsers directly connect to our Swift cluster? And that's what they did boom problem solved They did not have to worry about scaling out their web applications because it was this the available storage system that was Offloading those hard problems of high concurrency to away from that storage application. This is something. I was really excited about I want to talk more to these guys. I was just really kind of cool to find this week So let's talk about some hardware. So you want to actually put this in production What does it look like? There's one thing that I'm working on right now is building out this a community test cluster And this is a really cool set of hardware. It was Donated by Intel and HGST We've got some of the new healing The helium drives six terabyte drives in 12 of those in one you on these these these servers here and It is this is kind of off-the-shelf things. These are using Intel's new avatar processors Really low power really high connectivity. They've got you know the 12 SATA ports on it and two 10 gig ports and four One gig ports and it's all in a 400 watt power supply. It's really exciting stuff So we're building this out for the for the community right now But if you're wanting to do something else There's other kind of things you can have as well For example, you can look at just the kind of generic for you chassis that gives you a Lot of a lot of storage within a good form factor using basically standard off-the-shelf hardware You can buy this kind of product from lots of different companies out there This is something I believe was from Fred Hutchison. Yeah, and and there's also there's there's many other vendors that all have configurations that are Available that you can buy off the shelf from all the major vendors that are ready to go to run Swift and can be bundled and installed right on those so Depending on which manufacturer comfortable with going there's many options and then later today There's gonna be another session just talking about hardware selection in Swift So once you all have you once you've bought your hardware you need to figure out. How do we actually put this together? What do we what is what does it look like in real life? I mean like I can conceptualize this but show me what it looks like well Here's an example Let's say you're starting out with kind of a moderate sized Swift cluster a couple You know half a dozen boxes in a couple of racks in this case Let's go from the top down what you've got up at the front You need to have some sort of load balancer situation solution You can use something that's open source like h.a. Proxy. You can buy something that's From a vendor out there depending on what your needs are Then you've got your normal Aggregation switching that you're doing and each each rack is gonna have their normal top of rack switch in this example We've got a couple of proxy servers I'll go into detail on what these pieces mean in just a little bit Well, we've got a couple of basically you can take you the front-end servers in one rack And then we've got our storage capacity in another So what and what this allows you to do is really optimize what it is that you need do you need more network connectivity and client Throughput let's add some more proxy servers. You need some more capacity You can scale that independently and just plug in more servers with hardware So what happens when you run out of space on this? Well, let's add another rack and add a little bit more of capacity and Even if you want to go beyond that you don't have to just add new racks all the time You can't even just backfill your existing capacity and Swift is going to be able to automatically take care of this for you And that's what we're going to get into next. How does that how does Swift actually work? Let's go over some of the the architecture levels of how it's put together and what's going on So I mentioned earlier that Swift is has a pretty simple API. It's a REST based API and All that means is that it uses standard HTTP verbs and response codes. It speaks the normal language of the internet This is what a URL looks like that's referring to something inside of Swift the big parts here I want to pay attention to are the account the container and the object an account is basically Your your playground your your place to put stuff It could be say your your Organizational's business unit place to put stuff point is when you get an account on a Swift cluster now You can create containers inside of it and an account keeps a list of what containers you have and a little bit of metadata The container similarly keeps a list of what are the objects you have in this particular container? So given an account on your system You may create a container to call images and then you're going to upload cat.jpeg because the internet's for cat pictures and That's going to be your object name. So what does this look like? You actually want to create it you do a put request and you're going to get back a 200 response or 201 created When you do a read object you do a get what this means and as demonstrated by the adrift The adrift site is that you can just this normal browsers You can just talk to this and it's very easy and very easy to integrate into existing Languages SDKs tool chains applications all of that stuff So here's here's the parts of Swift. This is this is it. It's a very simple design very modular allows you to do a Lot of really cool stuff. So first we've got a client the client talks to the proxy server the proxy server is responsible for implementing most of the API and Making sure that the communication with the storage nodes is handled appropriately The storage nodes are responsible for actually persisting stuff to a disk So if you remember that diagram from earlier, we had some proxy servers in one rack And we had the account container and object server storage nodes on some servers in another rack Let me ask you a question here. Does the client ever talk to a hard drive? no does the Client ever have to worry about say a particular object server failing No, so what really is great here is that you've got the proxy server Which is abstract can be publicly exposed and is abstracting away the the complexities of what's the state of your distributed? System and you know hardware failures and timeouts and network load things like that and completely making that Transparent to the application and more importantly the application never has to think about well What directory did I put this in or what hard drive is this stored on it? And do I have to figure out how to do this distributed lock in some POSIX file system in order to coordinate across all my million mobile phone Applications and things like that. You just can't do that with these sort of systems and that's what Swift is doing It's abstracting away these hard drives so that it makes The failure handling automatic and transparent to the user and the and the user the application there simply consumes it as a service So what this design gives you is something that's highly scalable you can add It's linearly scalable because there's no single point of failure You keep adding more and more and more and the system continues to Perform and improve and get better and it's in that way. It's just a it's a fully distributed system So we need to figure out What does that give us Swift is optimized for massive concurrency across the entire entire data set which means that if you've got You know really hot content you may need to put some caching in front of that But if you've got content that's kind of that traditional long tail power curve looking access pattern Swift is absolutely amazing at that that sort of thing. So how do we figure out ultimately? Where to put it on the hard drives? We've got to take some object your your image your video your your documents And we've got to put it on a hard drive someplace just to actually make sure we can store it So to do that We use something called consistent hashing consistent hashing is is a pretty cool thing because it allows us to easily scale out our capacity But let's talk about what is what is hashing and we're all familiar with this because we all grew up going to an encyclopedia And looking something up So if you had to do a book report and you wanted to learn about birds what volume in the encyclopedia do you pull out? B because that's where it hashes to you just kind of look at that prefix and you know what it was But if you're gonna go do something on octopuses to pie then You're gonna go to oh, and if you're gonna go look at something in the zoo you go to z's so that's a very easy understanding of I know Exactly how hashing works. Okay, I'm gonna go here It's in each of those shortcut to figure out how to put my stuff into a particular bucket Well consistent hashing is similar to that what it does is it uses a hash algorithm to Splay things around what we call a ring and it's conceptually a ring because when you get to the end and you add One it just goes back to the beginning which gives us a ring So what happens is for example you take a hash of a particular thing to figure out where it goes It goes to a point on that ring and then it starts for example. Let's let's go clockwise and Find the next node that's been entered into the ring now that has responsibility For that particular data. Well, this is a really good system This is something that is proven at very large scale But there's some few things that we can do to improve this this sort of scheme Instead of putting things essentially randomly around the around this consistent hashing ring What we can do is we can evenly divide our space and assign things to even partitions and what this gives us is a very nice Even filling of all of our hard drives. So in Swift what happens is we hash this stuff up We take the prefix of that hash just like the first few letters of an encyclopedia article And then we can directly look this up In our ring which means that essentially we figure out cat.jpeg that's going to hash to for example hard drive number 728 and then having a hard drive number 720 or partition Let's say 728 partition 728 We know is going to be on three hard drives one seven and nine and that's our three replicas of our system So we store three replicas of this of the data in in Swift to ensure that you have high durability and failover when hardware fails This is how we choose what hard drive it goes on We've kind of got a Tearing system at the very bottom you've got hard drives. These are your ultimate failure domain they fail pretty often about 5% a year on average and You need to be able to account for that. So we're gonna put it on we're gonna put it on multiple hard drives But if we have more than one server We also want to protect against server failure so that you can not only handle that failure But also do in place upgrades without ever having to worry about taking your whole cluster down So in this case if we have more than one server we want to make sure we have It assigned to drives that are on different servers and then we can even group servers into what we call availability zones Most often time in deployments This is mapped to a rack because a wrap rack has either a single power supply or a or a single top of rack switch That's a physical failure domain that you want to protect against and finally you can group racks oftentimes into a data center Or a region places that are geographically distance and don't really necessarily have good network connectivity between them or maybe it's just expensive So when we choose something we choose it as uniquely as possible So in this case we've chosen three hard drives and you can see it's played across Both it's we've got one region But we've got two zones and we've got three servers that have responsibility of this and in this case We can go up to the level of losing an entire rack and not have to worry about unavailability or loss of data So the way we do that is by like I said Storing more than one copy in this system and generally three is a good balance Although you can separate that you can change this on the fly and swift but the whole cluster has the same replication factor Normally, it's three. It's a good balance of durability availability and cost We also consistently continually check the check the data with checksums So when something is stored we take a hash of that and check some of the entire data and make sure that that is The same it still has to change checksum sometime later This protects us against say powering the or losing power to a hard drive in the middle of a right You don't want to have that file system corruption. I'm causing problems and then we we do that by automatically scrubbing the data walking the data in the background to To make sure that it's still the right data and then the replication process that is running on swift is making sure that when something is becomes unavailable it will automatically copy and Another another replica will be pushed to the appropriate location, which means that as an as a sys admin if you get If you're on call this weekend and a hard drive goes down and your pager goes off What do you do? Well, it's 3 a.m. On Saturday or Sunday morning go back to sleep Take care of it on Monday swift is already going to take care of it for you So we're obviously an open-stack project So how do we fit in with the rest of open-stack? The important thing here that I want to talk about is that we are a Piece of that entire system It's not something that's a replacement for your VMs or something that you're going to plug in directly to Yeah, as your boot device, but it's something that participates and provides a solution for scaling you and matching your Infrastructure directly to your applications use cases So just as you need to have scalable and dynamically Configurable compute instances and be able to attach networks and and block devices to those you also need to be able to have Access to a storage system for your application that can dynamically treat that storage as a service So within the other open-stack projects absolutely working with things like keystone and salameter and glance and sender and All of those other projects so that the entire system works better together now one thing I want to point out and it is a little bit of Interesting within open stack is that Swift because it is not intimately tied to those computing pieces can be deployed separately from say Nova and sender But it certainly works very well with them. So where do we fit in? How does that work? What do we provide beyond this? Swift is an open-source object storage system and that is what Swift stack is providing here, but we're also adding on some of that the monitoring the management the The needs that you have to integrate that into your IT infrastructure so we've got Swift at the core there and You put that on your own hardware and then we added in a little bit of some some monitoring to let you know What's going on right now? And then you now have an out-of-band management controller not that we're going to see your data But just that you can know what's happening to my cluster right now And then once I know what's happening to my cluster and when something goes wrong you could say how do I respond to that? How do I fix that? How do I integrate this into my monitoring systems? That's it wide. How do I integrate this into my auth systems? How do I integrate this into my capacity management plan and doing chargebacks and and that sort of thing Those are the piece the pieces and the tools that Swift stack is providing alongside of Swift So I want to be clear here When Swift stack is contributing to Swift what we're doing is we're pushing all of that storage engine code upstream And that our product is out of band and not we're not hosting your data or anything like that So where do we go for more info Joe? well, so there's the So stack we have a website that goes over the architecture You can check that out at Swift stack comm slash open slash open stack hyphen Swift There's API docs So if you don't get a chance to get a book which has about three chapters dedicated on how to build Applications with the Swift API how to build middleware with the Swift API There's also API docs that are online and then for contributors There's an IRC channel which is pretty active and if you want to get the book come to the booth or Find it go to the URL so stack comm forward slash book and we can get you a copy Yeah Now we've got about 10 minutes left for questions And I think we've got a couple of mics here and I will definitely repeat the question so we can hear it on the video and Make sure that everybody hairs it. I think there was first question over here That's a great question So the question was about multiple proxy servers in front of one Swift cluster so that you can have multiple geographies and things like That and the answer is absolutely of course The proxy server that's exactly the way it's built doesn't matter if you're in one location You're gonna want to have more than one proxy server anyway just for ha probably behind some sort of load balancer But in the case of a geographic cluster, for example, let's say you have one Swift cluster And you have a location in Amsterdam one in Hong Kong and one in Dallas You certainly don't want all of your European customers to end up going to Hong Kong for their routing So you may put something like that. It's external to Swift Build up some sort of anycast Geo DNS sort of system and then you can route that to the appropriate region and Then Swift is going to be able to take that Swift what up by by Swift is going to be able to take that I mean the Swift proxy server will receive the request and know how to store that on the particular In the right place within the cluster now I want to go into a little more detail on that because the global clusters are pretty exciting and Joe gave a great Talk on it yesterday, and you should definitely look for that video The the thing about global clusters is we've got a couple of different ways to scale that out one is to say that I'm going to optimize for Local access and so I can I can read and write directly from that location and say that I'm going to write all of my copies Immediately in this one location, and then we're gonna asynchronously replicate them across and we can do that on the reads to choose a copy that's local and then the other way you can do that is you can say that I'm just going to make sure that when that write request comes into Amsterdam It's going to be real time pushed across therefore. We know it's going to be available in all of the locations immediately Question here. Yeah, this is probably elementary. I'm trying to learn but How would you characterize yourself versus no SQL databases? I mean to me it seems like there's some cross crossover there I'm sure they're not identical, but That's a great question So I think the the pity answer there is that you know when it comes down to it everything's a key value store, right? But the you know comparison to those sort of things a lot of times those sorts No SQL databases are designed for a couple of different things one. They're designed for storing very small values Because you want to store just you know a few bytes or maybe even you know a couple of k or something like that And number two you're probably going to try to do a lot of relations between the data So you even if it's some sort of map reducing you want to be able to dynamically compute across that Swift is not designed for those sort of things Swift is designed to be able to store Up to and beyond multiple gigabyte files works. Okay for small files as well But you want to be able to have this more It's kind of generic storage pool that it does and it's responsible for Storing the data not doing the compute on that now an interesting thing And there's a talk later this week on it is there's a project called zero VM Which is exactly designed to allow you to run your own compute jobs on the Swift cluster itself right next to your data So something I'm really interested in continuing to look at That's something that's going on inside of rack space right now So definitely keep your eyes open for that sort of thing, but in general the answer that I have is it's designed for more Larger data that's going to be heavy on the just the kind of static content that can grow without bound rather Than the things that are more dynamically changing and need relations all the time So if I understood what you said is another way to put that is Swift is more dealing with Almost opaque blobs exactly exactly that of like data you can analyze exactly exactly over here How does latency and throughput of Swift like varies with under like underlying storages So latency and storage on the network connections between inside of the cluster So between the client and the swift and client with the real storage underlying How does the latency actually changes due to Swift? So Swift has been demonstrated to I mean it powers some of the largest storage clouds in the world So I mean it's been demonstrated to be a very performant system for those highly available use high concurrency use cases Normally the first scaling points you're gonna have in a swift system Well, actually the normally the first scaling points are the the latency or the the bandwidth available between the client and the Swift cluster itself. Normally, that's the constraining factor, but inside of the cluster Oftentimes the things that get saturated is yes the network latency there So most of the time people are deploying on 10 gigabit networks Because it's these days is pretty cheap to get that and just deploy that everywhere compared to what the performance you get and There's also quite a few, you know buffer sizes and kernel tuning parameters you can do Overall Swift is not designed for single stream throughput. It's designed for massive concurrency across the entire data set And I think that's the optimization factor we have there. You had one in the front here How do you stay consistent in the case of concurrent updates? Well, we don't so well then we kind of do so the point It's eventually consistent So the way we do our conflict resolution if two people are uploading the exact same named object But different versions of our different copies of it different data It's it's a last-right wins and so even in the split-brain scenario if you have it's uploaded here And it's uploaded here and the one that's gonna be that was uploaded last is the one that's gonna take precedence when that split brain is restored Now why don't we do why don't we do one more for large objects does the latency has like Does Swift has latency impacts? Do the fact that like replication is going to take place For large objects due to replication or like so Uploading the the the large objects and saying that the the effort of putting them into the multiple locations is causing you serious latency Is that an understanding? With large objects your less latency sensitive because you're streaming a large object up to the system and the important point is streaming there So we're not taking it and then copying it and then Copying it to the locations and then giving you a response You send of your 5 gigabyte object you send us 64k we stream out 64k. We're not spooling. We're not doing anything like that So it's very it's very It's gonna be it's determined by what is the latency of your network in your cluster That's that's what it is. Why don't we do one more question for the gentleman who's been waiting and then anyone else We can come up and we can have a discussion on the side Um in the Amazon space there there's products that actually translate from file system protocols to s3 So the question is is that a bad idea and is there anything is there a counterpart product in the Swiss space? Yeah, so the so if you're putting so To answer the latter question first Can you put a file system on top of an object storage the answer is yes But it means that you're not going to be able to provide a truly POSIX file system with locking and there's pros and cons of that We off Swiss stack offers a file system gateway for our customers to help them bridge the gap They want to be in objects, but they also want to have one foot in a file system So we present sits in NFS, but you're still not allowed to lock objects and then for And so if you take a file system and then put objects on top of that you're you're doing something very similar You're you're not getting the same scale out properties like we just and all the benefits and the concurrency as As you as you would out of a native object storage By by putting a gateway that speaks objects in top of it. At least that's our view Okay, thank you very much. So I think we yeah, I feel free to come up and have questions I'm not my name on Twitter Joe is Joe Arnold at Twitter and we'd be happy to answer any questions you may have