 Good. Testing? All right. Welcome to a year with Cinder and Sepp at TWC. It has definitely been a very adventurous year. And that's kind of what we're going to talk about today if I can actually get my slides to move. There we go. So basically, what we're going to cover is, first and foremost, all these views are not from developer perspective. It's a typical systems administrator, storage admin, comparing it to stuff that we've done with a lot of the other vendors and typical standard back office IT deployments. How we came to evaluate our storage, what was the pass-fail criteria, some different things that we're evolving through with IPv6 and doing traditional storage arrays versus grid storage, and then how we came about using Sepp. Also, we're in the transition of adding additional back ends. And there are some kind of troubling issues whenever you do that. Who are you? Oh, by the way, I'm Craig DeLat. I am a senior engineer with Time Warner Cable, and this is Brian Stillwell. For those of you that don't know what Time Warner Cable is, we sell video services in broadband. OK, so about a year and a half ago, we had an organizational change. And our new VP came in and said, hey, I'm going to change how we're doing everything. We're going to do OpenStack, which is great. So after three or four cups of coffee and Google, I found out what OpenStack was. So after we go through that whole transition, we came up with, yeah, we can do this. We were a big EMC shop. We're like, yeah, it supports whatever. They'll support us. We pay them enough money. Then he comes back and says, well, it has to meet X, Y, and Z. Then we start looking at this, and we find out that the matrix, a lot of things weren't necessarily supported, even though they said they were. So certain things like live migration, how it does snapshots, just different things. And it's definitely if you go to OpenStack.org and you look at the Cinder matrix, it has a pretty intense checklist of what's supported where. So basically, go down three months down the road, we're testing a lot of vendors. We had multiple vendors come in, install stuff in our lab. We're going through everything. Everything's working great. And then we start getting into issues, because one of our things that we do is live migration. A lot of people in the cloud don't necessarily support this. It's the cattle versus cow. We don't need to actually, we don't need to worry about this VM. They have one over here. But for us, it was a must-have. So one of the things that also led us to this is the rack anywhere. Rack space, floor space, power was definitely something that was an issue with us. We had very restrictive power requirements per rack. So putting up a big iron array definitely did not suit our needs. Plus we have to go into multiple data rooms. OK, so after testing out multiple vendor arrays and working with some very skilled Cinder developers, we realized that there are some shortcomings with how Nova does live migration. So we were to scramble. We had a deadline. We had to support live migration. We had to deploy Cinder for OpenSack. And we came about to using SEP with dumpling. Basically, this was the only thing that supported live migration at the time. And it was kind of like an emergency ditch effort to just get live migration OpenSack deployed. And it also allowed us to take, instead of going through a purchasing process, getting a lot more equipment, getting it racked, going through the network intake process, we had stuff racked that we could just steal and deploy SEP. OK, so this led to our first SEP design. And this is where it went really wrong for us. Basically, our customers are used to almost unlimited performance. We had a lot of arrays. We had a lot of RAM. We had a lot of fiber channel speed. That kind of went out the window. We stole some Swift nodes and threw some cheap SSDs in there for journals. And basically, at that point, we had SEP. We're like, yep, we're done. We're good. We have live migration. Well, then you start getting into some of our customers who have unlimited performance needs. And they're like, hey, why is this slow? So we kind of, by the way, this is a four node cluster, very, very small. So we ended up having tons of performance issues down the road. So actually kind of backtracking, we went through the live migration testing with SEP. It was the only thing that supported us. And I kind of covered that earlier, so excuse me. All right, so early life with SEP, it's challenging. It's awesome. It's great. But if you're not used to this and you're used to a lot of vendor support, you are going to have some challenges. So certain things like out of family upgrades, they really, really do require a leap of faith. If you motor through, you're good. And usually, that's what you do in your dev environments. When it comes to production, you build in a lot more safeguards. This can actually hurt you more than help you. Be prepared to scare your coworkers. I see some of mine out there, and I cannot even begin to tell you how many times they're here like, is something wrong with SEP. We run Glance, and we run Cinder on SEP. And out of nowhere, they're here like, our customers are complaining. So once they start understanding it, how the whole everything kind of bounces out, they're a little bit more OK with how the whole upgrade process goes. And this is probably like, I don't think Eric's here, but one of our coworkers is very fond of animated gifts. So during our first upgrade, we ran into some major issues, and hopefully you're going to take away from this and not do some of the same things we've done. But with your first production upgrade, this is basically how I felt and probably how you may feel. You're just kind of getting slapped around, and you're kind of scared. And like I said, if you kind of motor on through, you're pretty good. There we get an animated gift. OK, so our initial SEP cluster was definitely, it was small, it was inadequate. We kind of stole hardware from a different use case. So we were like 60 OSDs. Our journal ratio was five to one. We didn't even test out the SSD card. So I know there are some other people in the room, they have great blogs on which SSDs do well, which ones don't, and we ended up with some that don't. In our raw capacity and usable capacity is there. And we went with 7,200 RPM ATA drives as well. So at this point, I'm going to hand it off to Brian. So this is my first public talk, so I'm a little nervous. So I've actually been using SEP for a little over two years, but about six months ago, I switched to working at Time Warner Cable and came into this cluster. And literally the first day, before I even had my email set up, we were running into problems moving off of our legacy hardware onto the new hardware, which was like a wholesale replacement of all hardware, so we could actually return those nodes back to our Swift team. So it wasn't a very big upgrade, but we did increase it by about 50%. The drives changed from 1 terabyte, 7,200 RPM drives to 1.2 terabyte 10K SAS drives. So we did increase the usable capacity up to 30 terabytes. So what went wrong? Quite a few things. So we identified a few of them right away. The journal ratio to hard drives or hard drives to journal ratio was rather high, a 5 to 1, with 10K SAS drives just didn't seem good enough. So we marked it down as something that we wanted to change. So having that high of a ratio could work pretty well for a cluster that primarily sees read traffic. But with RBD, it's a little bit more right traffic. And not quite sure what else I was going to say that, but. So also one of the things we noticed was placement groups. We started with 512 placement groups per pool, but with the size that we were planning on growing the cluster to that was not going to be enough. And then one of the first major issues we ran into was the VMs lost site of the storage. This was actually a problem because we replaced every single mon node. And all the new mon nodes came up with new IP addresses, which Dan ran into over at CERN. So we ended up having, well, I'll do that on the next slide. And then we also identified that Lake Sea tunables were enabled, so we had to switch that, which required some data migration. Quite a few of these, like enlarging your placement groups, lots of data migration, changing tunables, lots of data migration. And then we lost site of our storage again, which changing the tunables, the compute nodes actually still had the old version of CEPHON, because we upgraded the cluster, but forgot about the compute nodes. So they were no longer able to talk to the storage. So what corrections did we make? So we started the ordering process to get more SSDs that takes a while. So we wanted to get that started right away. We reused them on IP so we could actually get this cluster responding back to our virtual machines again. Placement groups, so we increased those. I think we did it like 64 placement groups at a time. Large jumps just completely killed our performance. Switched tunables to the Firefly tunables, which provides better data placement. And then also need to make sure that all systems were upgraded to the version of CEPHON, not just the cluster. Then a few of the settings we found was setting no backfill, so when we were making all of our changes, we didn't want to start the backfill process and then make another change and restart the backfill process. Disabled scrubbing and deep scrubbing, so that way we don't have additional background processes using the hard drives while it's trying to resize. And then we have a couple of CEPHON settings, like reducing the number of backfills that can happen at a time to one, max active, also set that to one, so that only one recovery process is happening per OSD. And then the recovery op priority to one, which is the lowest setting possible. I kind of wish that there was some way to use the CFQ IO scheduler to really make the recovery IO an idle level. But it seems like right now the only way to do it is just reduce it to the minimum number you can. So then that led us to the second expansion, because we actually needed a little bit more storage. So we actually had 20 nodes, but because we didn't buy the right number of SSDs at the beginning, we ended up having to borrow a bunch of those SSDs and put them into the nodes to get them to the proper size. So we actually have six SSDs now per node, with 18 OSDs per node. But we still had our original five nodes, which were completely different configurations. So we actually reduced the number of used hard drives so that we could get to the 3 to 1 journal ratio that we were shooting for. So this is actually a fairly big jump in the size of the cluster, going from 75 OSDs to 189. And usable capacity went up about 150%, it looks like. But we had more problems, different performance issues this time. And then we actually ran into a problem where giant had come out, and we were using a CEF deploy. And all of a sudden, these new nodes that we were adding were coming up with the giant packages instead of the Firefly packages, which the rest of the cluster was on. So I believe that actually caused us a few other issues. So we had to make sure to downgrade that again. So the corrections we made this time was we decided we needed dedicated mon nodes, added a couple more options to improve performance, which I've listed below. OSD recovery max single start, and op threads equals 12. And I tried to look those up to remember exactly they were, but they just slow things down a little bit more for the recovering process. And then we started work on replacing CEF deploy with Puppet CEF, so we could get a little bit more consistency with our packaged installation. So we've actually gone through quite a few different expansions as the hardware has become available. So the SSDs finally came in. We added those to the rest of our racked nodes, which finally got us up to all 20 nodes that we originally purchased. And the expansion was about 50%. So it wasn't quite more than doubling that it was before. But definitely noticeable a lot of traffic, especially since we had our customers start using it more and more. So there was more data total on the cluster. But still had some performance problems, which we had to troubleshoot. Luckily those are going away as the cluster is getting bigger and bigger. And then we also started removing the OSDs from the servers, so that way actually we could, yeah. So if you see in the top right, just one level down, basically we dropped those down to just the OS, like removed all the OSDs off of it, so that way we could reconfigure the hardware. So the corrections we made this time, we started well, yeah we started working on the CEF deploy, being replaced with Puppet CEF. But we also had an option to, when we bring in the OSDs, we bring them in with a weight of zero, because we found peering during having a whole bunch of OSDs doing peering at the same time kills performance for our customers. So that's the OSD crushed initial weight is zero. And then the most recent one, which is also kind of the smallest, but is actually just finally trying to get our cluster into the right configuration that we should have done in the first place. So if you see in the top right, we've got three nodes now that only have the OS on them. And that gave us, well that gave us actually mon nodes. Dedicated mon nodes, because we found that having mon nodes on the same hardware as your OSDs has performance issues, because mon nodes use a lot of F-sync operations. And yeah, this upgrade was only nine drives. So not big, but still something we had to do. And then, yeah, dedicated mon nodes, finally. And so we actually use our mon nodes as kind of our admin nodes, which I believe a lot of other people also do. But this also keeps the admin keys, like the keys to the cluster in a single spot, instead of spreading them out all over the place, because we had, well, when I started, we had them kind of scattered all over the place, and that's just not good to get the keys away to the kingdom. So we have actually multiple step clusters. So all the things I just went through, we did twice. So once in East and then again in West, usually the second time, it went a lot better. But we also have two small staging clusters and two lab clusters, and each member of our team has the ability to create a virtual step cluster inside of our OpenStack Cloud running on top of step. So I think Dan was saying he didn't understand what that meant, but it actually allows us to test things like public changes and different ideas like cache tearing and, I don't know, like primary infinity, I think it was. And I think, is this yours? Oh, yes. All right. So earlier, I kind of mentioned about going to multi back ends. So basically the next hurdle that we run into, and this is something we just ran into, I think in the past month in our lab, whenever you add in your second back end, all of a sudden your whole sender configuration changes. So this is kind of one of those you need to plan for it from the beginning. The main issue is you take a lot of default values and you kind of take the lazy way out for lack of a better way of saying it. And this hurts you because whenever you change your service type on the back end, whether it be a vendor or SEF, you have RBD, SolidFire, 3par, it changes how sender looks at it. It's no longer the default. And this can run into a lot of issues when sender is trying to move the line. It thinks it's default, which is no longer a valid type. And you can run into a lot of connection issues. So one of the things you really got to do is if you ever plan on expanding to a different tier or anything like that, even with an RBD, go away from the default right off the bat. It's going to save you a lot of trouble. And not all lab testing is really going to reveal what's going to happen in production. Our lab cluster was, I think, four nodes. And we only had a handful of OSDs. And it doesn't equate to whenever you're moving PG's around. And it's rebalancing and doing failover testing. It's not the same as a 30 node cluster or even an 8 node cluster. So looking forward, we are looking at providing a couple new storage tiers to our cluster, because right now it's very homogenous. So we're looking at a performance all SSD cluster and capacity a large hard drive cluster. We're investigating kind of emerging drive technologies like NDME drives, some of the ethernet drives that are coming out. I'm pretty excited about those. And see where a new store goes. So you don't have to worry so much about journals, at least that's how I'm understanding it. So takeaways. Don't start small if you're going to go big. Order the right number and type of SSDs, because the original SSDs in the Swift nodes just didn't have a really slow on direct IO. Determine the right number of placement groups early, because having to split your placement groups and expand on the new ones, you're moving all the data of your cluster multiple times. Get dedicated mon nodes from the start, because of that sync problem. Be careful with mon nodes in OpenStack, because changing IPs is not a fan, because Libert doesn't like this. And set up grades. Don't forget the rest of the nodes, your compute nodes, your monitor nodes. All right, so we really told you a lot of stuff that kind of went wrong. So what makes it worth it? What makes it worth the effort of going through this and the growing pains? I've done attached storage for the past 18 years. This is probably the first time that I've actually come back around where I feel like I have total control of the cluster, how it operates, and it's just how expandable it is. So one of the things is we're not locked into vendor specific hardware. You can go from your name brand servers to your generic servers. You can go as few OSDs or as many. You can go different tiers, different drives. You're not locked into anything that a vendor is typically going to make you kind of commit to. Scaling across racks, rows, and rooms, this is a big issue for us. We're almost out of room in one of our data halls. And we're going to have to go to another data hall. Well, guess what? Seth can do that. We don't have any limitations. We're going to have some microsecond latency. It just kind of works, and that's the beauty of it. Data migrations. Nasty data migrations are a thing of the past. For the past four or five years, I've been migrating off legacy end-of-life arrays onto new arrays. And by the time we get on those, they're almost end-of-life. The nice thing about a Seth cluster, you just kind of racks new nodes, and it just automatically rebounces the data. So whenever your hardware is end-of-life, you can basically wait for it to die. And when it dies, you buy a new piece of hardware, a racket somewhere else. Data gets moved on to it. You don't have to worry about going in between vendors or anything like that. And this is probably the most important thing, is we actually have a say. There are a lot of Seth people here. There are a lot of Seth talks. Open a bug. If it's a use-case bug, you can sit there and say, hey, why is this not doing the right thing? Like an out-of-family mon node upgrade, for certain things they have, yes, yes, I really mean it. Well, if you're doing an out-of-family mon upgrade, that you know is going to cause some issues, like why is that flag not there? Why is it not consistent? So if you run into features or something that you think should be there, open a bug. And the other thing is, there's a session today hosted by Dave that is going to actually help us gather more information and hopefully collaborate. We do drive testing. Other people do drive testing. Why is it not all in one area? Where somebody can go out and look and say, I don't need to go recreate and test an Intel drive for a journal or a 7,200 RPM, 6 terabyte drive, or so forth. So it makes it easier to configure your first cluster and keep it out of that growing pains effort. So hopefully you guys are taking away that this is definitely worth the time. It's worth the investment. You've got to be bold. You really got to make it kind of how you envision it. So we're going to open up for questions and comments, but you can always reach us. I think we're both on the Cinder channel and some other, most of the OpenStack channels. And if you see us around, definitely you can pull us over or ask us questions. And I think at that point we can actually open up for some questions if anybody has any. And I'm going to shamelessly plug the other Time Warner cable talks coming up. Next one is right next door. Can you please speak into the mic? So one of the things I was wondering is, as you're expanding, did you see some of the bottlenecks moving to the network? And did you have to look at that? And what kind of network we're using, like a back end? Yeah, so we're kind of lucky. We built this all from scratch. So all of our ports are 10 gig ports. With all storage, you just have to choose where your bottleneck's going to be. So ideally, if I can get the network to be my bottleneck, that's where I want it. Because that's just, I can go blame the network team and say you got to upgrade us to 40 gig ports. So yeah, you got to watch with that. Because we're getting into some solid state nodes. And if you're going to go dense with the amount of drives you have, you have riser cards and different stuff. So yeah, when you're sitting down and designing it, choose your bottlenecks wisely. You don't want it to be on you. You want it to be on somebody else. If I could just ask one more. I see one thing that you're going to do is go to an all SSD type of array. You see the bottleneck moving to maybe CPU utilization. And at that point, where you're going to have very fast IO, will you shift it? Not from the network, maybe this time, but maybe to CPU. I definitely see that, some of our early investigations, like basically by the fastest CPUs we can get. Yeah, we're going to find that out. I think it's going to be more backplane issues than CPU, to be honest. I mean, it'll be close. It'll be one of those two. But again, if we eliminate that, if we go to a less dense node, it comes down to CPU or network at that point. OK, thank you. Question about live migration. You said that's really important for you. Did you manage to know a part of the storage also with Ceph or did you do something different for Nova? No, so with Ceph, excuse me, we have Glance and Cinder. So we're doing no ephemeral storage. So everything, all of our RBD instance volumes are on Ceph. So that's the magic. Now Nova itself, like Livert, requires a shared volume or SSH abilities in between nodes to copy certain files over. So yeah, in terms of Nova, it's all booting off of Ceph. OK, thanks. Hey, just first to comment, thank you for the talk. It sounds like you guys are having exact same issues as us and probably a lot of other people. So it's kind of refreshing to hear that. Quick question, have you guys looked at tiering at all? I think I didn't hear that in the talk. So it's kind of on our roadmap. We've actually, and this is actually another nice point of Ceph, we're actually treating it more like a development lifecycle roadmap than an end-of-life hardware roadmap. So as new things get deployed with Ceph, we're rapidly kind of evolving our cluster. So it is on the roadmap. I think one of the things we've been talking about is waiting for something like the NVMe drives. It just allows us to build kind of some faster, denser cache nodes. And then there was other talk that we spoke with Red Hat, who we have a support contract with about if you want, you can just put all your primary PG's on just your SSDs and that really speeds stuff up. So we haven't decided where to go. We're a two-man team supporting 60 some Ceph nodes and that's only like a quarter of our job. So yeah, because we have a lot of say in live migration and it's how some of the other stuff works. But yeah, tiering I'm excited about and we're actually starting to look at RBD cache and I know there are some issues with RBD cache and how the client accesses it, whether they're gonna use direct IO if they're using a Linux box or if it's gonna be cached in the file system local in the instance. So yeah, we're trying to work out all those bugs, but it takes time. Okay, thanks. Hi guys. I'm curious how you actually expanded the cluster. Did you sort of just put it all in production at once or did you do one disk at a time or one host at a time? Well, okay, so that's a learning curve. So whenever we actually brought the first node, it automatically brought everything in and we didn't even want that. So we actually wanted to take a tiered approach, bring in one OSD and it caused major issues. All of a sudden, like I said, be prepared to scare your teammates because all of a sudden you're doing something that's normal and everything gets slow, customers complain. And then once you actually start getting to a point and this is something that you were talking about is you get past some of the peering issues which is the biggest bottleneck for us. Well, peering at the same time as the backfill recovery process just seems to be. But the bigger you get, it's now where we, I mean, we have it scripted. We brought on a hundred and, or what was it, no, 60 some OSDs and it never even blinked. So that's because we got the process down right. Basically stop all backfill process from going on while we're adding in the nodes. Yeah, there are some studies. And don't do too many at the same time so you don't have a peering process going on one OSD when you're bringing up the next OSD. Yeah. And then once it's all done. So you've scripted to bring in one OSD at a time. Yeah. There are some flags that you should, some things you should disable. And Biome is definitely reach out to us and we can tell you exactly what we used. Cool. And if you know a better way, please let us know. Yeah. No, I think that's an area for improvement, I think. Okay. Because this is, yeah. Thanks. Hi, you mentioned performance. What kind of actual aggregate throughput did you get out of your array and how did that change as you scaled up? Out of the cluster? Out of the cluster. So it's kind of, this is where we're kind of like, we differ in opinion. So as we tested out, I'm used to using VD Bench, which is a SBC, you know, storage performance council, you know, standard. A lot of people are using FIO, which again, I'm happy with. Some of our customers use Bonnie plus plus. So, you know, we kind of see, there are different ways to test the characteristics. I look at just IO. He likes to look at file transfers. So right now our cluster, it's not necessarily that it's like super busy, but it's doing large blocks. So we're doing 9,000 IOPS or something on average, but they're very large block sizes. It's not like 40,000 4K blocks. They're, I don't know, probably 32K or bigger. Some of these ranging almost up to 512. So it's, I mean, it's performing. I mean, we're, you know, right now we're not, we have no complaints and we're running some very intense stuff that you wouldn't think you run on the cloud, like logging applications that are always writing. The customers just aren't complaining. It's like 300 terabytes, I think. Yeah, well. Or no, say 300 gigabytes per instance. So, you know, we're seeing the throughput. And again, it's just, it comes down to the drives. Eventually we're going to run out of, we're going to run out of performance. Well, the other thing we did is we did 10K spindles. So we didn't do any 7,200 RPM spindles and that helps out. That kind of came from my performance, or my background doing storage. Always plan on performance, never capacity. Like customers always just assume that you're, they're getting a Ferrari. So, but they don't want to pay for it. So I know that probably doesn't totally answer your question, but I mean, if you have us, if we can get more specific afterwards, we can definitely. Hi, good morning. I'd like to ask, what's your approach for self-deep scrubbing? Do you delay it, or do you just let it run on a weekly basis? And do you also experience performance bottleneck while deep scrubbing is running? So, we did initially with the very small cluster, but as the cluster's gotten bigger, we haven't noticed it as much. Could be because it was added in so many different stages that are all happening at different times. Yeah, we were actually talking about this last night and we're not seeing the issues that other people are. And we actually think it's blind lock, that we're not, all of our volume groups aren't doing it at the same time. So, there are a lot of good resources or pages that you were reading and they say about modifying the clock. So whenever you're spreading out how it's doing deep scrubbing, because if you do get all your... Not modifying the clock, more like just scheduling the scrubbing process. Yeah, so, yeah, you can change the countdown timer. So, yeah, so, I mean, we really don't have our center volumes or our instance volumes are really the only two that are heavily used. Our glance group doesn't get used that much, so whenever it's deep scrub, nobody ever complains. So, yeah. Thanks. Two related questions. How do you handle the resiliency issues? Like, when the drive fails, do you get notified? How do you replace it and does it automatically sort of replacement drive start getting used automatically? And the related question is, how do you handle the monitor servers going down? Do you have to ever replace them or how do you handle the reliability of non-server during the upgrades? Okay, so, I'll handle the first part if you handle the second part. So, we actually built out Isinga. So, we're, and there's some really free, nice, self-health checks that are out there. So, we get notified of an OSD or a node going down. Before, you know, any of our customers notice any kind of issues at all. And with the size of our cluster now, our customers don't even notice. We failed a whole node because of a bad storage adapter. We didn't fail it, it failed itself. Well, yeah, it failed itself. And we actually recovered from that in less than an hour and a half. It was four terabytes of data that moved in an hour and a half. So, that was pretty awesome. And our customers, we were still passing our 9,000 IOPS and nobody ever complained. So, and that's with all the recovery traffic. But, yeah, and whenever we actually have something that goes down, so for instance, we actually marked this out of our puppet so it's no longer alerting people and trying to wake them up and at the same time, we set it to not auto boot. So, whenever we bring it in, it's actually gonna be on our terms, not because whenever you bring it in and it's gonna start repuring and changing the crush map, you're gonna run into some issues. And you just wanna, like I said, have it on your terms, not like it, they replace the card, it automatically comes up and in the middle of the night, one of our teammates gets a page for no reason and they're mad at us. So, we try to avoid that in the mon node. No, I was gonna go onto that a little bit more. It's like, that's one of the areas I wanna look into because before I came to Time Warner I was actually at Photobucket and we had around like 10,000 hard drives and auto support is great. So, like the drive fills all send a day or two later. It shows a new one, it shows up in the mail. You pop that in, throw the other one in the box and you're done. I wanted to get to that point and that's some of the work that I'll be working on. But the mon nodes. Yeah, filled mon node. We just upgrade them one at a time. So, take care man. Yeah. Yeah, so we have, we can actually evict a compute node in an emergency and you know, reappean and use Puppet and Ansible and our Cobbler server and we can rebuild it right into SF mon node and like Dave said, less than 20 minutes. So, that's one of the ways that we kinda try and recover. But if a mon goes down, you're really, it depends on how many mon nodes you have. You're in a state where you really gotta react. Yeah, so I mean my question was more if you have like a large number of nodes. Yeah. And now the mon nodes are like say five or some number, small number. The chance of all of them going down at once might go up. So, how do you handle that kind of? Of mon nodes or both mon and OSD? Mainly the mon nodes because. So, if you have, I mean, if you have all your mon nodes go down or you lose quorum, you're kind of, you know, not in a good place at all. It doesn't. So, and that's again where you actually, I think it's important to reiterate like you gotta have a way to spin up an emergency mon node pretty quickly. Because if you actually have, you know, four of your five mon nodes go down, well, you know, you basically lose your Cinder operations and you're gonna run into some RBD stuff where I think it's actually gonna start, you know, eventually stop passing some data. So, yeah, you really gotta have an emergency situation or emergency plan in place for that. Okay, thank you. Hi, first of all, I wanna echo the sentiment. Thank you very much for the candor. Okay. It's an awful lot of times some of the sessions wind up, the only thing missing is a revival tent. So, I really appreciate the actual, you know, nice candid. I do have a couple of questions, however. I missed the applications that were this fault tolerant. And the second part of that question is what are the opportunity costs for another system that doesn't have such a low utilization rate for the costs involved in the time? So, I missed what applications are running, who the customers are that can tolerate this. Yeah, so, unlike I think how a lot of people deploy this, our customers can run anything. Our vice president sat there and we're onboarding customers and it's stuff from video content to web portals to logging applications and some people who are trying to run databases. So, our stuff is all over the board. It's not just like it's an imaging system that's capturing like thousands and thousands or millions of pictures and graphics. It's, our IO is completely scattered and random. So, you had mission critical applications on the system? Absolutely. And they were able to tolerate failure at every upgrade level? Yeah, early on we actually had some issues with a message queuing system, but as we actually expanded and our cluster got bigger, that whole window minimized. Once we streamlined the process to figure out the proper way to go about. So, we're running customer facing applications. So, we got 20 million people that have email and different things while that's their interface. So, based on that goes to the second question. Were you asked as part of the justification for going in this particular route for a cost benefit ratio of doing this particular solution for those applications versus a more traditional approach and what the cost differences were? Cause you have, for every drive you have that's usable, you've got two others that don't. Obviously, that's the way it works. But that's a very, that's like a hard drive tractor pull. Yeah, so, this is like a shameless plug but the changing culture of Time Warner Cable, like the guys that are speaking to that are the people that actually are writing the checks for the stuff. So, for me to justify, hey, we're gonna have a bunch of uptime and we can actually evacuate nodes. We do changes during the day. It's no longer nighttime changes. Like, we don't have a bunch of burned out engineers working day and night. Like to them that's worth the writing that check. So, but that's definitely a question for them. I mean, they'll be happy to field that for you and yeah, I don't have. I just say what drives I wanna buy. I just was curious. Thank you very much. All right, I think we're out of time. Thank you.