 Well has been working on it as long as I have Hello Hi, I'm David gets I'm a cloud files developer. I've been working on Swift for four or five years. It's kind of hard to remember and This here's Michael Barton. Hi Michael Barton my principal engineer at Rackspace. I've been working on Swift for almost six and a half years So yeah So we're gonna be doing a talk called omg objects the unscaling underbelly of OpenStack Swift Basically going to be like some of the Problems we've had with Swift over the years And go through some of the solutions Basically I'm gonna go through and complain a whole lot about stuff that happens and because I get all mad about that and then Barton's gonna come Through and with some solutions, which is what he does. So it's kind of like a one of our team meetings here Okay, so starting off first We're not trashing Swift. Just want to get that out of the way Skipped OpenStack Swift does scale very well so cloud files is the first Swift cluster has developed at Rackspace a long long time ago and We think that public big cloud is a primary use case for it, but we're pulling out the most common use case for it At least from what I've seen at the conferences and stuff You know and Swift that can do many many many things. This is just one of them So at Rackspace we have six clusters on four different continents Our daily peaks are around 20,000 requests a second We have over 120 petabytes of capacity over 75 billion objects 50,000 accounts, which is important and we all run that with less than 10 ops and admins. So we got a whole lot of stuff and What I want to focus on is the accounts what makes Rackspace or cloud files Tricky is the different kinds of clients that we have Some people look at us as just a kind of like a cold warm storage type thing Where they try to you know, they just do lots of puts and those objects just kind of sit there in mind business and for a long period of time and Other clients because we also run a really big CDN out of our cluster They you know, they'll do smaller objects or they want a lot of range requests for videos and They expect really fast Performance and this makes it difficult because you kind of have to solve for the generous for the general case And just as an example There's one of our clusters where over the years was just it never really had problems as kind of like our golden boy type cluster And the reason why is because we had an internal customer That used us as a back-end for like email and stuff like that and it was kind of like that cold warm storage So they're a multi petabyte cluster and it's just kind of spread out the load on all the nodes and all the other customers just kind of read the benefits of having like a whole lot of object storage object nodes serving less hot objects with lots of traffic I'm still worried you picked a picture of Amazon to put on there. I Don't think it is Amazon. It's not all warehouses are Amazon Yet, yeah Okay, so here's a very quick introduction to Swift Basically, there's this proxy server layer and we're not going to talk about that one at all and then there's three Internal services underneath there. There's the account the container and the object layer So basically a client does a request into the proxy server and depending on the type of request It'll just send that down to one of those services like if it's a DB listing or containers the releasing of the objects or an object Put or delete or whatever which goes directly to the object server and we're going to deal with scaling issues with each one of these services Except for the account layer. I just put this because I thought it's kind of funny We love you account layer. It never goes down. It works great. I can't think of a single scaling issue We've ever had with our account layer And there's a reason for that is because at cloud files We limit the number of containers a single customer can we have to five hundred thousand which kinds of You know, we just limit it until because and that's why we don't have any poems there So the object container listings on the other hand, we do have problems with These just as a background the container layer is what holds all the listings of an object when you put an object in the Swift Then you can do it you can hit that object directly But if you do a object listing goes to container databases and get served out of there Every single container in Swift is a separate independent SQLite database is sitting out on some server someplace So the proxy server gets the request it sends it over there loads of the SQLite database Spits out the result and that's kind of how it works The good thing about this is there's no centralized database for all the objects in your cluster So it's horizontally scalable and I put ish Because within one container it is not horizontally scalable and The problem with this is that containers can get very very very very very very big Especially for a SQLite database We have containers in the wild out in our clusters that are hundreds of gigabytes in size with over a billion objects in them So you could see that you could look at this one direction and say wow Swift scales to That many objects per container, but the problem is is that we don't have any way of stopping these things from growing The good news is they still perform relatively well. We don't get complaints from these customers I guess they're used to the eventual consistency model But in many cases we have contacted customers saying hey, can you shard up your containers and then some of them go back and say? It's public cloud, you know, well, this is what we're doing so Yeah, so this is still a really big problem for us The ways we try to address it are with sort of non-technical solutions working with customers on their architecture also with Sort of medium technical solutions like rate limiting rights per second We've tried gluing together ever bigger SSD rates to try and keep ahead of the problem We figure eventually we will need container sharding. There's another record named Matt Oliver who's working on that He's not working with us, but we're hoping that someday he'll save us From yeah So that the Feature where this is presented the biggest problem that I can think of at X-Base was with the expiring objects feature Which is the feature I love to hate and Because it's a neat feature some customers really like it like we have customers that like Send us up video game screencast and then they'll save them for a couple months And then it'll be auto-deleted and stuff like that Basically the feature is is you put an object in the Swift and you can set a metadata header It says delete this adder this object automatically after so much time The problem with this is is the way it was Implemented is very very very container level Container layer heavy It just coincides so excited that the time when we got SSDs at Rackspace is we came up with this feature and I think we just kind of got giddy with how fast they were compared to the spinning discs that we just Just thought that the container layer now they're on SSDs. It's not going to run out Which it did so Here's a just a quick introduction into what happens with a non-expiring object put Basically the proxy server sends the put with the object down to the object server It saves the object and then it tries to sends requests up to the container database And if that container database is unresponsive It'll save an async for later on and we have this background day And we'll go around and send those up at a later point in time and it's super cool as part of our eventually consistent framework and We can you know the customer can just keep on putting objects in the container server can go down for a short period of time And they'll never know the difference But this is what it looks like for expiring objects, and I think I missed a couple arrows, but basically There's a whole second async thing can be created for the kind for the expiring objects marker And then we have this daemon runs in the background. It's constantly hammering the expiring objects container and and then trying to find objects delete and then when it does delete them Well that turns into an expiring objects delete down the road So it kind of like triples or quadruples the load on the general container layer And on a single container layer you get something like this This is This graph is in number of asyncs in the millions This is 1.2 billion asyncs that we had put into our cluster One time a while back and what happened was is we had this customer start putting in a whole bunch of objects And they sharded out the containers so they get multi hundred writes per second and every single one of those had X delete after header on it so those went up to the container databases up here and everything was kind of going long happily for that whole time we You know work just fine, but then they started in September say 2012 then September 2013 All those objects started deleting so then we had this container day and daemon going and finding all these objects And then sending deletes into that same container layer so overnight this container layer Which is handling hundreds of crisis like and it's not going to do having to handle twice that with the daemon and it just stopped working and It was doing many of these a second and they piled up quick and we got over a billion asyncs Which is just not good So me and Barton and crew Well, this really still isn't fixed either We're still behind on asyncs not on asyncs on expiring objects We don't have asyncs piled up quite like that, but we do it we still are way behind We've put in a few fixes to Swift to try and improve db throughput We've upgraded sequelite most of these are performance problems not really tackling the architecture of the problem So, yeah, we we we've still tried taking the can down the road on this problem chasing after performance We really need to re-architect the whole thing to decentralize it There's a spec out there by Alan Irwin right now, and we're hoping that he'll save us Yeah on this particular problem. Yeah Okay, moving on to the object layer, which is the bread and butter of Swift It's the simplest part of Swift It's basically it just sits on top of the object servers and reads and writes bytes It hasn't changed a whole lot. It's been shuffled around a little bit But you know the API and change much it hasn't changed much in the last 60 years And it has had the same general problem for six years which gets worse as your scale Increases and that is inconsistent performance This is a graph of one of our big internal customers sending inputs into our system And this is a response time on those puts Now the majority of those puts they're all the same object coming from the same endpoint Coming into the same container. I think Yeah, it's same container because it's SLO segments and the line at the bottom, which is 99 99% of the puts the request time is hanging around 12 seconds 10 to 12 seconds But then about less than 1% of the time It'll go to some bad node out in the cluster and they're getting plus 4,000 second response times same object basically 12 seconds 4,000 seconds So, you know, you could say to yourself. Well, you know So cloud solution and it is and all that but this customer They're putting segments of an SLO and some of those SLOs might have a thousand segments in them So from their point of view they're putting a thousand but from their point of view They're just putting one object. Well, if less than 1% of those fails then for that one object They're getting it's not less than 1% failure. It's more around 30-something. So this client was seeing around 30-something percent failure on their puts Because of this inconsistent problem So Yeah, the object server is tied closely to the hardware. It has to be it's basically the lowest level you get in Swift and drives do fail especially when you have tens of thousands of them which we do and the file system can freeze up and This stuff happens on any system but what we found is that the Python object server becomes unresponsive too easily and That the only way that we were able to fix this over the years was to have really great monitoring Which we do and really great ops which we do and but there's still just people and they got to go out there and fix these things and That's the only kind of buffer that we got right now until Yeah, so I Paid somebody five dollars to draw that hummingbird Well, I asked him to make it look like he could be friends with the go-go for so that's what I got to So a couple of years ago my friend Florian asked me to go to go for con with them I decided if I was going to go for con I probably should have written something in go So I decided to re-implement the Swift object server because I thought I understood it pretty well and I thought it could be improved on and Yeah, what I ended up with was something by the time I got to go for con It was really benchmarkable and I was really impressed with the performance. So I Kind of spent my spare time over the next year or so Getting it to where Could be dropped in for the Swift object server And what we found was that it was like super fast Two times as fast for most operations. It uses about 75% less CPU It's also a lot tougher because go that what I drew the body for this one Just keep on the software development Yeah, so Swift has a huge problem where you know disk IO can block other requests going on We all know about that problem. It's been a problem since the beginning Go uses real OS threads threads for the disk IO. So, you know, you get isolation between requests Bad drives don't lock up this with the the object server the way they do and Swift at all We also implemented a really simple kind of error handling where it's basically a Simiforpe or drive that limits the number of requests that can go to them and That really cuts down on the number of timeouts we get from the proxy app server This is a graph of one of our production clusters. That's the number of timeouts per hour You can see what hopefully you can tell when we switch from Swift to hummingbird the the number of timeouts are dropped They're basically zero There's another up switch. We did this is just one server That was where the the Swift object server had basically become completely unresponsive all because of one drive So we switched a you can see the number of timeouts to that server Dropped to basically nothing and the number of requests it was serving went from basically nothing to a Bunch is that per hour? Yeah per hour So hummingbird has helped a lot with the object server, but we still have another problem Yeah, which is replication so So replication is What does everything? You know if you have a node go down and because of hard drive fails Well that you switch out that you get new gear in there And then the replicator will go find some other copy of the object and put it over here so it's kind of the magic behind Swift and And it's extremely important, but it has the same problems of the object server is written in Python back With our sync to move the data arounds. This is the our sync version of the replicator, which I'm talking about here But it can get slow Very slow This is a log taken from one of our production nodes that had some slow drives on it And it wasn't doing very well Probably need to be switched out, but this is just a log line where this replicator was saying that it would not finish a pass on that object node for 32 days and That's just not good So here's a couple examples of how that's not good This is something this is a healthy Change so somebody puts an object and it's out there on his three primaries and it's perfectly fine This is the box represents the primary nodes So that three that object is sitting on his three primary nodes and then you do a ring change because you got some new capacity You're getting a lot of customers. You add a new ring change. So that new Say that new server is going to be number four and this object is going to be should be moved to that new server So it can fill up and the server that was on can drain and so you have your balance again so Three is sitting out there where it was, but your replicator finishes a pass and move three back to four And then it deletes the one from three and everybody's happy again. You have three copies of objects. You're filling up your new node rejoice Well with slow replication the same thing happens step one same step two same but Say that one or two replicates to number four and Three is still sitting there because it happened to be on that 32-day guy Well during this point in time you have four copies of your object sitting out there so we call this over replication and this is a temporary problem that happens all the time and it's just a thing but In some of our clusters We're adding capacity and adding capacity adding capacity and this over replication step just kind of lingers and When you're some of your servers are at capacity, which is why you're adding new capacity replication slows down and things just get kind of worse and Basically, these new nodes don't fill up quick enough and your at capacity nodes don't drain quick enough Now let's make this worse because it's fun You have your first this is the good side you have your first object You do a ring change you're back to your one two four idea But then they send a delete to the object So that delete only gets sent to the primary nodes number three doesn't know about the delete because it's not supposed to Be there. It's a transitionary state But replication on three worked this time So it it sends it's gonna do its put back to one two or four and then it finds out that there's a tombstone there that This object should be deleted. So it's like oh fine. So it just deletes himself rejoice well What happens if The object sitting out there the delete happens and a lot of time passes before your replication window completes Well, what happens is is that the tombstone sitting on one two and Four eventually gets reclaimed We just we clean those things up after a certain amount of time three still sitting out there He knows nothing about anything he's sitting out there all by himself Then finally sometime later three happens and he goes out and talks to one two and four It says hey, do y'all have me and they're like I've never heard of you and they're like oh, okay Here I am so he copies out to one two and four a brand new copy of himself thinking Oh, I only have one copy left of me and now I'm triple-repleted. I'm so happy Well, the thing is is that one two and four shouldn't be there anymore. They're gone the customer is not paying you for them anymore and It's not in the container listing, but there's the object is it's unaccounted for it's taking up space and this is where everybody screams like Luke Skywalker especially me Yeah, so All this boils down to is that replication needs to go fast and it needs to be super easy to watch and to find problems and everything else so After we had so much success with the object server kind of our two Long-standing goal the whole time was to switch order replication and here's what hummingbird could do with replication So one thing was that we got kind of changed the model of replication With Swift we only repushed data from one server to another And It turns out that if you just send back a little bit of data You can help a lot for example if you're trying to sync an old data file to a server and it turns out there's a newer tombstone over there You could just send back. Hey, we have a newer file and then the sending server can know to remove that data file because It's outdated don't need it anymore That really helped us a lot get rid of handoffs you can see here Where we rolled the hummingbird replicator to each of these object servers These are these all object servers all had a lot of data. They needed to get off the server What no, there's a graph of capacity on it. I know they're trying to drain That's a graph of amount of free space on the server and all these servers needed to drain data You can see where we rolled hummingbird to them. They were able to you know turn around Go from basically having flat Available data to actually getting data off the servers Yeah, we added new nodes But we couldn't fill them up and the existing nodes that were at capacity were not draining until we released on the bird another change we made For some reasons well because of our sync again Swift will If it's small if it's replicating a partition to multiple peers, it'll walk the file system three times It'll read files three times Or up to three times and send those to the peers with hummingbird replicator we actually walk the file system once and Replicate to any peers that need the files as we go The hummingbird replicator also gets the same speed ups as the object server where we're using, you know, 75% less CPU We get isolation between The various threads of execution because each drive gets its own go routine Today three of our six clusters are running hummingbird. It's made a big difference in those clusters Yeah, for example the customer that I had the one with the 4,000 seconds Yeah, they're wearing some of our normal clusters are getting 65 percent Success rate in the cluster would played hummingbirds above 99% success rate So having we're in the future we're gonna continue to work on replication is not really done But it's at a point that we're obviously deploying it and it's working We'd eventually like to replace hashes pickle We've wanted to do that for a long time and we think this is kind of a platform where we can play around with that a little bit better We haven't ever done replicated policies just because we don't use them But it would probably be pretty easy to go and add those back in We have a prototype for the pro for replacing the proxy We don't think Like actually replacing the proxy would probably be a Herculean task That we're not getting prepared to take on But the speed up we get from the prototype is so much that we're kind of interested in maybe see if we can get a reduced set of the functionality Working so that we could support maybe like Glantz or our CDN product those sort of things on Sort of a simplified proxy that would take a lot of traffic off of our Swift proxies that use Really a lot of resources That's pretty much where we want to go with hummingbird. Yep So we got some time left over for questions Yes, or there's a mic So Ashish not currently IDC had a question on so there's been a lot of talk about erasure coding in Swift and You didn't mention anything about it. Is it something you're neglecting completely in your future direction or something you'd consider? We haven't really looked at it too much. Yeah, basically what happened was is Mike started on this prototype and He kind of talked about it and it's kind of like a running joke around the office that ha ha. Yeah, we're gonna move to Golang and so it kept on going there and our problems didn't go away and Eventually we after several months we kind of switched over to fix it to get it working and our small team of developers went and tried to implement everything that Rackspace uses and That's as far as we've gotten so Yeah, we're just we've just we have just thus far done what cloud files does Clay has a whole notebook And he's like on page 93 It was like four pages don't be jerks Okay, so you guys talked about some stuff you talked about the hash pickle some other things you want to do in the replicator But I think the object server maybe I Mean I and I guess you kind of got to this with the ec stuff But I mean aside from other some of those other compatibility stuff for you guys is that layer done the API? Are you feeling you know, I mean you've obviously replaced it in three clusters. So how close does that you think? It fulfills our needs pretty well Yeah, I mean, you know, it's a drop-in replacement one thing is that we deployed it in our first cluster over a period of I think four months and We you can go in and you can just install hummingbird and you can just run it You can actually run it out of your existing Swift object server config I think it still works we actually have a separate one for hummingbird But you can do that and you can just turn on hummingbird and let it run for a day and watch all your problems Just magically go away and then you can turn it off and go back to the other one and your masochist Yeah, just for kicks, you know, and we did that, you know, we installed it and we made sure everything is going but it Hummingbird does implement a new replication model, but it supports the old one So you can you can run your hummingbird object server and the Python object replicator can talk to it, right? but So you can just deploy it on one node and all the other Python daemons can talk to it But if you want to run the hummingbird replicator daemons, you have to deploy the entire cluster With hummingbird nodes because a replicator daemon can only talk to a hummingbird object server So yeah Have you guys ever like follow up on some of those the slow requests So on a you had an object server that you're getting an example You had 90 drives and one of them was bad and the the impact on at the in the the Python processes whenever you're sharing everybody in the same event loop You know if you've got right there, you're trying to do non-blocking IO and you just insert a bunch of blocking IO And you know everybody traps out it's so unfair And of course the go you've got the threads much better, but I mean those drives are still slow. They still got to be slow Do you still see? You know you said 99% success rate on that that one customer So what is the impact on the bad requests going into the the go objects? so I'll share two examples the first one was the one we put the drive on and this was kind of like a perfect use case for Hummingbird as opposed to Swift One of our ops guy was he just there's no the one responsive and where it's like well Let's just try it so we put it on we got it running and you can hit a disk usage URL Which will show you the error limited which is I love that thing, but anyway, there was just one not node There's just one drive out of 90 on that server that Hummingbird had to error limit All the rest of those drives are perfectly running fine, and that's why there's such a huge You know good thing happening well and one of our other clusters It was badly performing and we tried to put out Hummingbird out on there, and it's exactly what you said There was some nodes where So first off I mean the read timeouts did go down But it didn't like solve everything wildly because there are some nodes out there They're just messed up and there is like a whole bunch of drives on on several nodes that were bad So you look at one server and the disk usage They'll be like four or five or six or eight drives that are all bouncing around all over the place And Hummingbird while it did slightly improve the situation didn't solve that problem because you can't because you're sitting on top of hardware There wasn't any like unintended like secondary consequences of like threads being leaked or the go-routines aren't getting closed down correctly or You know some sort of like CPU spike or something It was bad for the requests But the go-core was handling it that the load goes through the roof But not anymore some Python Well since we're error limiting it doesn't really do that Also, we're kind of conscious of the fact that we could Basically shut down a drive without knowing about it So we're trying to figure out how to avoid that if we just start rejecting our who has to a slow drive we could basically Shut it out without Purposely removing it from the ring or knowing that we need to do maintenance or any that stuff So we're trying to figure out exactly how to monitor. Yeah, some of the error limiting that we've put in Yeah, that was the one kind of fear the whole way through is that we had ended we could start getting these like silent Bad guys out there and so yeah, we're working on separate new types of monitoring Like there's a swift drive on it and all that But we're working on With our ops guy really closely Get a next generation of that Can I switch to some questions? Yeah, there's somebody in the back Yeah, just you know Thank you, I work a lot with service providers And I guess what you said worries me a little bit because it seems that if you know if I was to advise them to From an object store to build it on Swift That if they don't have the sort of capabilities that rack space is obviously built up over a period of time That if they are gonna extend beyond a fairly simplistic on-premise Swift object store they're gonna run into a lot of problems and And basically have got an awful lot of learning to reach the sort of expertise that you have Yeah, I mean like We got to over a hundred petabytes capacity six clusters for no and all that stuff that we got to that on Python and And I couldn't imagine that there's some software out there that at that scale doesn't have any problems It's just that this and and the other thing is is we have 50,000 customers And we always hear about from the bad ones right but whenever our product guy goes out and talks to our customers Even our big customers. He'll go and he'll like send emails out to our top ten customers. They're all happy with us you know like It does work really well and But I think I think you are To do anything non-trivial your trivial you are gonna have to be something of an expert It's not the easiest system in the world to run. Yeah, I think anyone who runs it would agree with that Yeah, but I mean I would just say that yeah, I mean we have hundreds of nodes 270 terabytes of capacity per node tens of thousands of hard drives I couldn't imagine. I mean, it's just that's a concept complicated problem Okay, so so basically you're saying you could do it But it's going to be complicated and and I need to go with eyes open and perhaps if we just follow that up If with all the replication issues if you start to distribute things as lots of service providers want to say you have maybe two copies on Prem 1 500 kilometers away again, it seemed with this I mean I know about the eventually consistent But I kind of get the impression that you know if you've got four six 16 terabyte drives whatever it's going to be you never ever if one goes bang You know, let's face it You're never ever going to get to the stages never be consistent ever again. Well, so that slide I put up with a bad airline That was the worst one I could find when I was doing this and that's actually a cluster that was fairly small and For several reasons Just got a ton of traffic Our bigger clusters that are more spread out. There's less partitions per drive There's more object nodes talking to each other the replication times don't look anything like that But there's a long distance though. So let's say it is 500 miles. We don't do anything over long distance And so I have no experience Everything's within a cluster so long distance. You'd think it's a bad idea I don't have any experience with it and I've heard other people talking about Swift clusters that do it and they're perfectly successful Yeah, yeah, I mean like I say that there's probably a reason why nobody ever talks about the object layer and the object replicator It's because it works almost all the time but You know We have a very big cluster. Some of our nodes are sitting out there for a long period of time and Basically, you know, I mean it's like it's kind of like the The billion objects per container problem. I mean how many people have that problem, but we're just having to be one of them Any other questions? Clay Okay, I wanted to switch to replication so Yeah, so so that was really interesting so What do you do you know off the top of your head like what is your current reclaim age? Are you still running the default of a week or It seems like yeah, just make replication happen in a week. What are you that's the best slide perfect. Yes. Yeah, everybody's I mean, do you get some sort of warning as your I mean are you are you monitoring your replication cycle times? Are you able to see like does that just happen like immediately you just suddenly replication cycle times a month? Whereas it had been running it, you know three days and then when you did the last capacity adjustment It took, you know, five days to push out that Yeah, no, it does not happen immediately and in our bigger clusters That wasn't one of our medium-sized kind of heavy traffic not that big yet clusters It's kind of like this adolescent phase that seems to kind of happen But and it was like it was at capacity it was like things were going wrong in that cluster for reels and So that's when I that's the one I looked at for this thing and most of our clusters Yeah, you can find that stuff and you can also check your jobs log and look at those things and you can find the drives That are going slow and then you can use that your ops can go say oh, let's go find that drive But so like these are all problems. I don't know if they're all problems, but to a large degree You know, they can be solved with the fixing bad hardware better monitoring blah blah blah blah But we're just trying to and you're always gonna have to do that no matter what but we're just going to you know We're just trying to get around that and Like I say the getting rid of the hash is pickle idea was always That could actually increase Speed on puts a lot and not having to have this file system and we're kind of wanting to do that if we could prove that hummingbird that we can get rid of our sink and and And yeah Cool. Are you guys still working on that as part of making any progress? Yes So not this yeah, yeah, we're still working on it and Then but the thing is is yeah, that would it would make it not backwards compatible with Swift, which we have not done yet I think we do about oh, that's right. He had a password. That's not true. Never mind a Mike what do you think about on the For the proxy layer you said that the prototype you was looking good for you But I'm curious You know one of the reasons that goes so attractive at the Audra server layers You've got that runtime where you can push you just do the blocking IO in a threat But even the network operations are they are they also in blocking in threads or is everything they go into an event loop? Okay, so in the proxy every is a scheduler that uses either real threads or even an event loop based on very similar to how event like works Like if you were to do the threaded this guy, oh It's very similar to that, but it's all baked into the runtime and it's very fast and very good. Yes I'm just there's somebody with a question in the back right there So back to the container shouting thing um I would think that there are two two worries that can happen there one worry is the actual unbalanced ring of The container ring and the other one is the ability to serve requests or puts to a container that is like really really huge What would you find is your primary problem between the two Would it be the capacity or the ability to work with a large container? They're both problems usually we see more problems with the capacity just actually filling up a drive or Filling it up enough that we can't do the work. We need to we don't have the scratch space That kind of stuff. Yeah, I mean there is seems to be a bigger problem than like actually we don't seem to see a lot of problem Personally, I don't see a lot of problems with the replication of those big Containers or insert speed or any of that kind of stuff. Yeah the two problem Are customers who who use those big big containers they probably never do a container listing You know, they just they know it's eventually consistent. They know that it takes a little while the problem the other problem is is if you get a Single drive that has two of these behemoth containers On your SSDs is that SSD will be really full while your normal thing is it's much lower. Thanks Okay, great. Thank you very much