 So hello, Riverbun, how are you doing? I'm Nidall, I'm the head of the data duplication village. We have a repeat speaker here, this is Andy Klein, he is the individual who has come to us several times, three, four times now to the data duplication village and spoken for us. He works for a company that does hundreds, if not thousands, if not tens of thousands of hard drives, more every year, and he's come to talk to us about those hard drives, their reliability and how they compare with SSDs and a number of different things, but instead of giving away everything, I'm gonna introduce you to Andy Klein. Andy, thank you for joining us. Hey, thank you Scott and appreciate it. And like you said, for the last few years, I've been coming out and talking to the folks at the data duplication village about hard drives and SSDs and so on and so forth. And let me give you a little idea of what we'll talk about today. And by the way, anytime you have a question, please just stick your hand up in the air and we can get that out of the way. I'm old, you don't need to know that. We're gonna talk a little bit about the data we collect. It's about hard drives, it's hard drive failure data, hard drive capabilities and so on and so forth. We'll talk about failure rates. I'm gonna do SSDs for a little bit. They're kind of popular these days I've heard. And then I have a bunch of other oddball statistics that I'll just throw in there about hard drives and things that we found out. And then finally, we're gonna put all of this in action. In other words, how we make use of it moving forward. So kind of fun to do. All right, so a little bit about the environment, all right? Right now we're currently monitoring about 250,000 or so hard drives. So that's not a bad day at the office, right? About three exabytes of data that we're dealing with. We've been doing this for the last 10 years. Hence the title, right? So we started back in 2013. We record the data every day. You'll see that in just a second. And then we can analyze the data and figure out what kinds of patterns we might see in the information that we're there. So I forgot to bring my hammer. We have a bunch of different data centers there. The fun one is the data center on a boat. Excuse me, a barge. So this is actually kind of cool. I just brought this along. It's got nothing to do with the fact except it's full of hard drives. But it's actually water-cooled. There's no freezers or anything like that in there. It sits on the San Joaquin River in Stockton, California, which is kind of nice. Saves us a bunch of electricity. Doesn't kill fish and so on and so forth. Okay, which is all cool. But this is where things need to go in general, right? Data centers like this, guys, what you're doing, use a lot of electricity, do a lot of interesting things to the environment. This is our first step. So it's one of the nice things that we're doing around here. All right, so let's get to the data, okay? Again, I said we've been storing it since 2013. We use a package called SmartMart Tools. It just takes something called SmartStats off of every single drive. Every hard drive, every SSD out there for the last 20-some, 30-some odd years produces these statistics. And then this package will read that, okay? What we did is those statistics, and I'll show you some of them a little bit later, all right? Every day we snap the statistics off of every single drive, and that's what we've been keeping. And it gives us the ability to track a drive and its behavior. So for example, one of the statistics is temperature. I can monitor the temperature of every single drive I have, all 250,000 of them, okay? I can tell when something's getting interesting inside there. I can monitor the media inside of every single drive, and I can understand that we're having trouble failing and writing in a certain area. So we can do all of that kind of stuff, and that's what we've been doing, all right? The data is open source. So you can go and play with this if you choose to, all right? It's a bit to download, but you're more than welcome to do so, and people do. Interestingly enough, the drive manufacturers do. And you'll see why in a minute. So what does it look like just to be, just to kind of get everybody on the same page? You can see, oh, I lost my colors. Oh my goodness, it's horrible. So here's the data, it's just laid out, right? Row, data with columns, it's no big deal, nothing spectacular. You can see smart one normalized. You get a normalized value and a raw value for every smart statistic out there. There are potentially 255 of them, pairs like that. No more, any given drive doesn't use any more than 40. And that's a pretty high number. And we'll look at some of them a little bit later, like smart one is a read error rate, smart five is reallocated sectors. It'll start to be interesting. Smart nine is power on hours, how much that drive has been running, all of those kinds of things, right? Now, the data there, you can see there's a failure in one of the columns, that's how the data marks a failure. That drive failed that day, it was, and it's gone. It won't be there tomorrow, all right? And that's how we know. We know that that particular drive, that particular model, SD 800, so on and so forth, okay? Failed that day. Now, each row for each day is what we call a drive day. And that's important to understand when we start to do aggregation and we start to figure out failure rates, right? Speaking of failure, so what is a failure, all right? Now, most people would say my drive failed when it stops running, that's a failure, right? We have two states, we have what we call a reactive failure, which is exactly that. I can't talk to it, it doesn't give me data back, it's broke, okay, for all of that. And then we have something called a proactive failure, okay, we're monitoring these on a regular basis. You might get what's called ATA errors coming at you, which are electrical type errors. You might fail an FS check, a file system check, or combine that with the smart data that we were just talking about, all those various statistics, and you go, no, that drive is on its way to failure, and I think I need to get it out of there. Now, why would we do that? Why would we care, right? The reason we care, one of the primary reasons is, if I can take a drive out before it rolls over and dies, I can clone it. Cloning works really well, right about here, you get about a 95% success rate to clone a drive. And why is cloning important? Because you know how long it takes to rebuild the drive when you put it in, like at a rate array, and you pull one out? Not bad with a one terabyte drive, maybe it's a day or two. Gets kind of, okay, with a four terabyte drive, because that certainly gets to be four or five days, or six days. We're dealing with 16 terabyte drives right now, and that can take weeks. So, I don't want weeks. I can clone it in a day and a half, and have it back in the array, and running, and keep up what's called durability on the whole system, right? So, you can start to see the data we collect, and the decisions we make, but either one of those will produce that little one in that column, and that drive is gone. Okay, it's fail. Now, how do we compute this thing called annualized failure rate, right? Which is what we publish. You'll see that number as we go through the thing, right? We take a group of drives, a cohort of drives, in this case, all of the models that were active as of 630, 2023, okay? All of the different drive models. And we give it a period, Q2. And then we just obtain the numbers for those things, and we do some math. Drive days, number of drives, running each day over that period. Failures, how many failures? These are real numbers, by the way. I'm not making these up, okay? And at the end of Q3, Q2, excuse me, we had 240,000 drives we were looking at, right? And then we do it a little math there, and we come up with a number, and it says that's our annualized failure rate. And then we publish that number, and we hand people the data, all right? And say here, now, sometimes people say, hey, why don't you use drive count? Because that's what I would do. I had 10 drives, one failed, I have a 10% failure rate. We can't do that, because that makes the assumption that you had 10 drives over the entire period you were looking at. We don't, we're constantly adding drives in and taking drives out, you know? So we have to use drive days instead of drive count. I tried publishing the reports one time without a drive count, and everybody yelled at me because they really want that number, okay? But it's irrelevant. So what do we do with the data? We put it in some quarterly reports, some of you may have seen it. We do it for hard drives, we do it for SSDs, which we'll talk about in just a minute. We give you the data for the most recent quarter, we give you the data on an annual basis and a lifetime basis for all of the different drives that are out there. It's kind of useful. If you want to go there, go to the blog, type search for drive stat, you'll get effectively eight years worth of reports. And you can see what the heck this is all about. It's free, we don't charge you, we don't track you, or any of that silliness. If you would like to get a newsletter you can join in, I'm sorry, that's what the marketing people do, but that's about it, okay? So a report kind of looks like this, okay? It's a table. We write some fancy words around it and make it look pretty. I pulled out a couple of things in here because as people start to look at this, I want you to understand what you're looking at, right? One of them, the first one there, the HGST one, right? It's got a 13% failure rate and people go, that's terrible, I would never want one of them, okay? Well, that's not a good conclusion because there's only a total of 8,000 drive days, right? That's not a lot over a quarter. You really want 50,000, 60, 70,000, start to get a number that feels good, all right? So not much there. I highlighted the Seagate one because it's our old timer and the data duplication village guys have been using the six terabyte drives for the last few years or asking you to bring them in as the case may be and that happens to be, look how old that is, 98 months. That's not bad, right? Eight times 12, 96, little over eight years on average in a data center, okay? Being beaten every day, all right? A good little drive. It does bring up one point and I want to point this out if you ever do look at these reports, okay? And you see that model number, that ST6000 DX000, right? And you can go, wow, that sounds pretty good. I'm gonna go get me one of them, right? And you go to the store and it's a ST6000 DX005. They're not the same, okay? Different model, okay? They may look the same. They may have the same specs and everything like that. There's a reason that Seagate and all of the other manufacturers change the model numbers and they don't tell us. So if it's on there and it's a certain number, you like it, great, okay? But if you can't find it, the next model over, okay? Your call on that. In other words, the data doesn't, the models don't transfer. It doesn't go from one to the other. So in the most recent quarter, we had a total of 31 different models. These are the ones we're using in our data center. These are active. They are drives that we are storing customer data on, okay? So they are in use and they're treated all the same. They're all given air conditioning every day, right? There's a guy who comes out and dusts every once in a while. No. Another chart you'll see is what's called Lifetime. That previous one was quarterly. This is Lifetime. This is a whole lot more accurate for the whole drive, for the picture of the whole drive, right? Couple of fun ones there. There's that HGST one. There you can begin to see this thing called a confidence interval, all the way in that last two columns there. That is, it's a statistical thing. You don't have to care what it is. All you have to care about is the delta between those two numbers, 0.5 or less. Maybe 0.6 or less, okay? If it is, you got a pretty solid number in that AFR column, annualized failure rate column. If it's not, you don't, and it's that simple. So the first one, the HGST, that difference is 10. Not 0.5, 10, okay? Which means that AFR 656 is, yeah, okay? We just did some math, that's all, all we did, okay? Statistically, that's not a valid number. We publish it, but it's not. The one right below it, 0.2, okay? All right, that failure rate of 0.59% for that drive is pretty solid, right? You can probably look at that and go, that fails at less than 1%, that's a cool drive. I like that, right? You can look at the one, the last one there for Seagate, oh my goodness. Look, no drive failures, that's great. Yeah, it's great, but again, a pretty wide number there, so it doesn't have enough history. And I tell you this, because people look at these numbers all the time, and then they go, you know, I want you to, you know, they complain, and the confidence interval is there for a purpose. So take a look at it, use it, make it a part of it. Now, one of the things that people talk about is, you know, what can I use this for? All right, the data itself, okay, that we publish. And I always tell them, it's entirely up to you, obviously, what you want to do with it, but there's nothing out there like this. The only other data you get is manufacturer's data. And every manufacturer I've ever seen either has an annualized failure rate of 0.9 or 1%, depending on how they feel, okay? And it's never higher, and it's never lower. All right? So this is a little bit helpful in trying to talk to you. Now you can make all of the arguments about, hey, different environments, all of that kind of cool stuff, right? All right, we do treat these things nice. We don't put them in an external case and stuff them under the bed, okay? You know, so, but they do a lot of work for us. All right, just to make your life a little easier, I put together a list of things that we call gold medal winners. And I'm gonna actually start from the bottom of the list just to kind of show you some things here. The bottom three, the ones with the little three there and friends, those drives are, those are really great numbers, okay? Those are really great numbers, but you can't buy them. You can't buy them new. You can buy rehabs. Rehabs all bets are off, so don't, okay? So, and if somebody's trying to sell you one of those is new, they must have had them in a warehouse somewhere for the last eight years. You go up a little bit to the 12 terabyte drives. You can kind of see the numbers there. You can still buy all of those, but two of them are from HGSTs. The two from HGST are actually rebranded as Western Digital now, okay? Western Digital bought a chunk of the HGST business five or six years ago, seven or eight years ago. The top ones are all pretty legit. Four teens and 16s. I haven't seen that very top one available for sale in the US, so maybe you want to go down in the second one, okay? But those are solid drives. They've been performing for us. They got more than two million drive days, so they've been working. You know, make sure, whatever choice, use the data however you want to help make decisions. All right, a little bit about SSDs. So, last year I talked about this at the group, and we had this question about SSDs and reliability versus HDDs, right? Everybody said SSDs are much better, but nobody had any proof, right? So, we went and we started to take a look and see what we could do. But you always want to compare apples to apples, so the only place we had SSDs were as boot drives. But we also had HDDs, hard drives as boot drives, right? So, we wanted to compare them. Now, boot drives you might think of is, turn on the machine, spins about 30 seconds or whatever, and then it just sits there and does nothing. Ours don't do that. Ours are actually used for daily, for temporary storage. The drive stacks actually get downloaded onto them, and then get offloaded about once a week. So, these are drives that are working, all right? They are reading and writing on a regular basis and deleting files and all of that kind of stuff. And you can see the counts aren't spectacular, but it's about 4,000 drives we did the analysis with. And last year I presented this chart, and I asked the question, I says, well, what are we gonna do? So, what we had done here, let me give you a little background, is we tracked the failure rates over time for the red, which was the hard drives, remember the boot drives, and the SSDs. In the first year they were in existence, the second year and so on and so forth, as they aged, okay? And we got to year four and we didn't have any more data for the SSDs because they had only been around about four years. So, we said, well, what do you think's gonna happen? Because they kinda tracked pretty closely together there, right? We said, do you think it's gonna go up? Follow it up? You think it's gonna be in the middle? Or maybe it's gonna level out, right? And the answer, you know, and we had people guessing and everything like that. And the actual answer is, they were all wrong, they went down, okay? The SSDs, and I will tell you, I'm getting ready to do the next report for SSDs, and the number continues to be that low. So, SSDs do in fact seem to be more reliable than HDDs, which everybody said it would, but at least we have a little bit of math now in a practical environment to put it to rest. Now, at some point we'll put SSDs in the data center, we'll start using them, we'll compare them and all of that, but that's, right now it doesn't make it sense from a cost perspective to do that. So, SSDs pretty solid though. Yeah, okay, so you can see the numbers right there. The other thing we looked at over the last few months is the smart stats, remember? The statistics we catch for each, we capture for each of the hard drives. Tell us things like media, temperature, and so on. Well, SSDs do the same thing, they report that data, okay, and we can record it. And nobody was making any sense of that that we could find out. People tried and we tried to say, well, what are we gonna do? So, we took three of them, three SSD models, right? And I listed them there. The Seagate, the WD-C1, and the Crucial one, they were all 250 gig SSD drives, okay, so they're all identical from that perspective. And we analyzed the smart stats and we said, what the heck do we have, all right? What are we looking at when we look at it? Is it any good, okay? They've been doing, the hard drives have been used in smart stats for, oh, I don't know, you know, 25 years is, you know, so what about SSDs? And we found some things that were common between all three models, right? Power on hours, yay. Power cycle count on and off, temperature, that was good. And then a couple of them, SDD, SSD ware leveling and unexpected power loss. Those were the only five attributes that were common between those three SSDs. And so, for example, the Seagate has 20 pairs, only five are common with the other one. That means 15 are unique to the Seagate. Why? Okay? So, we tried to figure out what were interesting ones. And I wanna talk about a few of those today just so you can see, it's a little bit of a wild west right now as it relates to SSDs and smart stats, right? Here's a good one, lifetime percentage. Now, in most of us would say, oh, I get that, okay? That means how long do I expect this thing to live? Where is it, perhaps, in its life cycle? Okay? That's a really, really cool thing. And they calculate that. All three of the vendors, the three different vendors, use a different smart stat. Okay? The WDC one and the Seagate both basically compute it the same way, okay? Something called programmer race cycles, available blocks, math, okay? And then they get a number, an SSD, a number that starts at 100 and comes down, okay? So, 100 is new, new, new, new, 99, we're still pretty new, 90, really good, okay? In the case of the Western Digital one, when it gets to zero, you gotta, it's over. It's past its life percentage time. You really should yank that drive. In the case of the Seagate, it's 10. That's their threshold value. And if the number is zero, that means it drives in read-only mode, right? Now, okay, so that's close enough. The folks at Crucial decided that they weren't gonna count down, they were gonna count up. Okay, okay? So, it's lifetime used. So, the number comes up as zero when you buy it and it goes up as you use it, all right? So, I don't know, you know? So, one of the things that points out, by the way, is that tools like Smart CTL, which is what we use to do this, right? Now have to pull out, I have to pull out three sets of numbers and do my analysis if I wanna compare things. And then I have to remember in the case of the Crucial one to flip, right? And that's kinda cool. The other thing we saw is that all three vendors, and they're not the only ones, it's not, Micron does the same thing, Dell seems to do the same thing, all of the different vendors seem to have their own tool that you're gonna get. Now, by the way, that's really cool. If you buy in a Crucial drive and a Crucial SSD, you got three or four of them, you really like it, it's a great brand, you go and you use that tool to do your analysis and see the health of that drive and excellent, right? But when you're in a business that has actually more than one type of SSD, it starts to get to be a pain. And when you're in a business that has 20 different types of SSDs, it's a pain in the neck. So one of these days, I hope they kinda get their act together maybe make these a little more consistent. Another one, some interesting ones that I found as I was digging through these, right? WDC has something called Drive Life Protection Status. Right? And this is amazing because it has this thing like it's like a trajectory, it says, okay, this implies that they have, they can calculate their drive life and then it implies that there's some curve that they've come up with and embedded into that so that they can do this projection, okay? But nowhere have I been able to find what that curve is. So is it tied to, for example, the warranty period? Or is it whatever, right? Is it tied to load? So they have a curve like this and if you're using your SSD too much, it could be above it but okay, what's too much? All right? Endurance remaining, okay? That's kinda fun. The, with SSDs and stuff like that, it's a twisted way of looking at the protection status but there's no curve. The last one is actually really cool, okay? This is, the folks at Crucial have this built in. It's a little thing called RAIN, which is kinda like RAID except on their SSD, okay? And what they do is they hash out your data and they store the parity off. And so if you as a customer suddenly start to use your SSD and the data can't be found, okay? In a normal RAID, they can use the hash to recover it and that's all completely invisible to you, right? Really, really awesome, okay? They didn't say whether that parity was actually part of that 250 gigabits or it's added and you don't see it, okay? But the overheads for something like that has gotta be 10, 15% of storage to pull it off. But it's a really cool feature because again, as a user, you wouldn't even know it, okay? They go read something, they couldn't read it, they use the hash and recover it. Ma'am, really enjoyed that one. One of the questions that came up while I was doing a bunch of research was this one. How long can you store an SSD without power? You take it, put it in a closet, right? We all do this, we do it with hard drives and we expect them to last forever, okay? We expect to pull that hard drive out after five years, plug it in and spin that puppy right up and I don't know, sometimes that doesn't happen. Sometimes you gotta get out the hammer and hit the top. Sometimes you gotta get a new hard drive. So SSDs, okay? The number, the number's been all over the place but I'll just say the current thinking is about a year, okay, is the maximum. Now, that seems to be tied to how long the drive, busy the drive is, how tired it is, how much it's been used before. Crucial actually documents that and has a little number, so all the way over on that side. I get my left and right screwed up all at a time. You can see, if you look at smart value 202 and it's a zero, once again, I like to count down and count up, zero, the maximum storage period a year, okay? And as that number goes up, they continue to reduce it. So if that number's 50, six months, if it's 100, a month and you're stretching it, okay? The other two vendors, okay, don't have similar attributes that they use for the same kind of thing, but I can't find any documentation which says that's the way it works, all right? So I can't trust it, but it seems to be that those two pieces are built that way for it. So if you're thinking of taking a hard, an SSD and putting in an archive for a little bit of time, you may wanna take a look at that number for the SSD and see how long you think you're gonna put it there for. If it's been used a lot, you may not be able to get away with putting it in there for a year, all right? All right, so a little bit more about, oops, okay. No more about SSDs, apparently it's a computer's pissed off. So we'll get back to hard drives, just some fun stuff about hard drives. A few months ago, there was an article published by Secure Data Recovery, it's another company in the business, and they came up with this number which said, hey, we look at 2007 failed drives and the average age of a failed drive was two years and 10 months. And that, I looked at that and said, oh my, that seems really low. I was thinking, excuse me, it's gotta be higher than that, right? So I took more data, okay? And over the years, we've had about 17,000 or so failures and I said, what's the average age? They failed that. And it came up to two years and six months. And I went, you just look at something and it just doesn't seem right, right? So then I said, well, wait a minute, I have a bunch of models that haven't finished failing yet, okay? In other words, they're still in production so they could fail tomorrow. So I took out all of their drives, I only had used the drives that we didn't have in production anymore. So our data set, those numbers weren't gonna change and it rose all the way up to two years and seven months, right? Which still feels low. It just doesn't feel right. And so just to kind of give you an idea of what we're trying to look at here, the ones in pink that are pink, okay, that have zero drives remaining, those are the ones that I use for that data set that got me two years and seven months, right? The other drives still have plenty, we still have plenty of drives in service. So we can't really use those as a calculation. And if you look at them, for example, the four terabyte drives, the average age of those is seven years and one month. So anything that fails out of that set is certainly going to be older than two months, two years and six months. So what I'm saying is, as those drives start to come out, I expect that two years and six months or two years and seven months to go up to some other number and we'll track it over time and see what it looks like, okay? This is just one of these things that you look at the data, you get an answer, you go, I just don't believe the answer. And then you go from there. All right, a couple of little fun things that we like to do. Failure rates by, in this case, drive size. So we take all of the drives that are given size. Our 10 terabyte drives are doing terrible. Okay. It happens, they get old, they hit the wall. We obviously had a lot of strategies in place to deal with all of this. It's not an issue. We've been around for 16 years. So we've seen all kinds of different interesting patterns. But I bring this up because this is what we publish and this is what we show and this is the data that's out there. It also gives us an interesting idea to see what's happening. So the one that was the surprise was the eight terabyte drives. So the last couple of quarters, they suddenly started to jump up. And so then we had to go in and dig in and see if it was a particular drive model. It wasn't, it was all of the drives. And we had to go dig in and see if it had to do something with software changes that we made or location they are and we had to get all of that out. No, what's just happening is they're just getting to the end of their life and the failure rate is starting to rise, right? But we publish this kind of stuff on a regular basis. We also do this by manufacturer, okay? And I always put this up and everybody looks at it and goes, Seagate sucks. And no, they don't suck. They're nice guys too. They're all nice guys. We have a good relationship with all of them, right? But you can see the numbers are rising and they've always been a little high. And so I wanted to do a little bit of math for you just to show you that sometimes it isn't just the failure rate that matters, all right? And we publish this model. It's a simplified version of what we do internally to figure out who we should buy drives from and how much we're gonna pay, right? And I'm actually gonna apply real data to it but that's the model. You can kind of walk through it. We pay so much for a drive and model one, model two, model three, three different prices, right? We buy so many drives. They have three different failure rates. The cheapest one has the highest failure rate and so on and you can see the number of failed drives to replace the drive, it's 300 bucks for labor. We assume all of them are under warranty, five year warranty because it's a five year period so we're not gonna pay and buy any more drives but if we did, it would be the same for everybody. And then at the bottom there's a total replacement, total cost, so the drives plus the maintenance. Again, a little bit simplified but it'll work for us. And you can see, model one, oh my goodness, I saved almost $200,000 by buying those drives over the lifetime of those drives. That's really, really cool, right? So let's use real data. So I put Seagate, Toshiba and WDC, these are 14 terabyte drives, right? This is in our data center, real numbers. And I wanted to break it down. So the first thing I did is I started down here and I said, I'm gonna do this math backwards. I'm gonna make the answer the same and let all of the other factors start to change going back up to the top. In other words, I wanna get a number at the top, right? So it's gonna cost me almost $2.6 million to buy 10,000 drives and maintain it for five years. And then I put in the failure rates, the actual failure rates for the lifetime for those drives, okay? And I put them in and those are real. I didn't make them up at all, right? And what I come up with is the price I'd be willing to pay to buy those drives, right? So I'd be willing to give Seagate $230 and I'd be willing to pay $245 and so on, right? Now, that's easy, okay? And it makes some sense, but there are other factors which come into that like I said 10,000 drives. So if I go to WDC and I say, hey, I need 10,000 drives, what number can you give me? And they go, I don't have 10,000 of those, I have 12,000, okay? And then you go to Toshiba and Toshiba says, well, I have 8,000 but I can't get them to you until a month from now, okay? But I'll cut the price down. And so you have all of these other factors which make it in. But the point is, okay, this is why we have drives from different vendors, okay? When you do all of that kind of math to figure it all out. All right, so, again, we publish this data on a regular basis. We have all this smart stats. We have more data coming, okay? Which will give us the opportunity to share more with the community because that's what it's all about. We have something called the Vault ID. I didn't talk much about what a Vault is. A Vault is just an organizational unit for us. 1,200 drives in 20 servers. A drive on each server is a tome and so on and so forth. I'm not going to spend any time with that. But what we get is, we can isolate problems then down to a Vault which is, again, that logical unit or a pod which is a storage server. And start to see and do the analysis. Now, our ops guys have been doing this for years, okay? I just didn't have the data to share with you. So we'll be able to share that. We'll also have data center. So you know that boat that I showed you earlier? We'll be able to say, is the failure rate any better or worse out in the boat? Versus our other data center in Phoenix because it's kind of warm down there. Or our other data center in Sacramento. Or the one in Amsterdam. Is there a problem because the electricity is different? Who knows, right? And then we'll be able to look at things like failure rates by server type and data center and even things like temperature, right? So on that boat, okay? That's using water cooling instead of air conditioning. What kinds of things are going on out there? So those are kinds of things coming, all right? Lots of fun stuff that we look at. We've been doing it for years. The data's free. It's out there. You guys can play with it if you want. You can read the reports if you want, okay? You can also do analysis for us and send it to us because I'd love to share that with folks. That's what this is all about. So, all right. If there's any questions, I'll gladly take them now or I'll be here for a few minutes and we can take it from there. Any questions in the audience? Any questions in the audience? Thanks, Andy. If you have any questions, just come on up to this mic. That way we can hear you and they can hear you on the Defconn TV as well. I have a one quick question. What drive do you use at home for backups? I have a Seagate drive. So those are HDDs, I assume? Yeah, well, yes, they're all HDDC gates, yeah. Okay, what size? I believe it's a six terabyte. Okay, thank you. My question's about the SSD and its storage of data. Was that based on power to the chip or the bit being written to the year life? I'm not sure I understand the question. So your chart showed that if the SSD wasn't used for a year, it could start to lose data. Was that meaning that the bit was written to or power applied to the chip? Power applied, okay, yeah. Thank you, yeah. Have you done any special analysis on how vibration maybe influences lifetime of drives? Yeah, so that's a good question. Vibration's a tough thing to measure. Technically there's a smart stat for that, but it doesn't seem to be populated very well for it. And so what you end up doing is, in our case, that's part of what we'll see with the server type, because we use different mechanisms in the different servers to lock the drives down or hold them in place. And that may be one of the things we see. The other thing I have is anecdotal information from our maintenance records. All of this data, by the way, that I produce off the drives gets validated against maintenance records before we publish it. And the maintenance records sometimes have that as a reason, drive vibration. And because drive vibration actually not only bothers the current drive, it usually bothers the neighbors as well. So it's a hard one to say, because we should have a stat that gives us that, but the stat seems to be garbage. And so we have to rely on just other things. When we were building storage servers, the storage pots, okay? We built multiple versions of them. We had like four different ways that we tried to minimize vibration. And that's one of the things we're gonna take a look at now that we have the storage ID. And so it'll be fun. So good question. I don't have a really good answer, sorry. Good to know, thank you. Kind of in that same vein, have you looked at other correlations, not just with things happening in the building, but things happening to employees, the personnel, kind of looking for other explanations for drive failure over time? Yeah, so that's a really good thing. Outside of the technology and the human element a little bit, interestingly enough, the maintenance records show that the most because they're all trained to do the maintenance records a certain way. And they actually have a script that they have to follow and things of that nature. And when they don't, you can see it. And so Larry is the guy who runs our global data centers. Him and I have that conversation from time to time. I'll be reading through it and I'll go, you know, Barney doesn't seem to be doing real good on that stuff. He says, yeah, Barney's no longer with us. Okay, so they do have a process that they're supposed to follow. It's relatively consistent, but the reality is that sometimes things just don't go in a straight line. So it's an interesting point, but no, we don't. That's the best we can do. All right, any other questions? If there's not, I will gladly, like I said, I'll be down here if you have anything else you wanna ask and I'll gladly share anything we want. Check it out. It's fun data, okay? There's no harm, no foul. You can use it, you can don't use it, whatever. Thank you, Andy. And the data duplication village is on the other side of that wall if you haven't been there before. Thanks, everyone.