 Thank you all for coming in. I appreciate it's standing remotely. It's awesome. We're gonna have to have a bigger venue next year and everybody's just having conversations if you could keep it down for our talk please. Thank you. I have the pleasure of introducing the talks at the Data Duplication Village. This is the first time we've had talks this year and I love seeing your response. Thank you Paul for coming out and I also have the pleasure of introducing this individual Andy Klein from Backblaze. We have been paying close attention to the reports that he has been generating for the past several years but four years now, four years on a quarterly basis he's been generating reports to a hard drive and putting him out for society for the community to be able to take advantage of especially us so we can see what happens in the way of failures and drives to look for. So without further ado Andy thank you very much. Welcome. Thank you Scott. Much appreciated. Alright so they did let me in here with a title of marketing so I don't know how that happened. I snuck in under the radar. I actually do a long time ago I actually coded for a living. I played systems administrator for a while and all of that and then I crossed over to the dark side and became a marketing person. The good part is is I'm hard to bluff from a technical point of view but I still am a marketing person. So take everything I say with a grain of salt. Alright so a little bit about I have to do this. We're gonna talk today a little bit about our environment so all about drives and stuff like that right? How we measure a failure? Alright because we do that. I'll walk you through some of the stats we have and then we'll do some fun stuff like look at enterprise drive versus consumer drives and helium drives versus airfield drives and a little bit about the idea of can you actually predict failure on a hard drive? Alright if you have enough statistics and enough numbers you can predict anything and this we'll finish up with temperature. Just so you know for those of you who know back plays this was the original storage pod we built okay for those in the back that's plywood okay that's how we prototype the first one you can see it up on a rack over there in with that lovely little piece of delicacy which cost nine times as much by the way so we built that and then we have actually changed into those pretty red ones that we'll talk about in a few minutes. So our environment and it's important that you understand our environment just a little bit because people look at this the data sometimes and they go that's not how my system is and this is it and I have two drives and one failed and you guys don't know what you're talking about okay this is our environment it's a data center 60 drives in a chassis there are now systems with up to a hundred drives in a 4u chassis right but that's how we do it and then we actually logically group together 20 of them 20 of those chassis into something called a vault so when a file comes in it actually gets sharded across those 20 different chassis right so that way we can lose because of the way this is done with our own we created our own encoding erasure code stuff a mechanism and we actually open sourced it by the way so go to get hub you can look it up if you want to steal it excuse me borrow it excuse me make it better it's out there for you guys to look at it's but it's a it's a excuse me 17 three encoding mechanisms so you can lose up to three three whole systems before you lose anything any data and long before that ever happens okay we're we're way ahead of that okay but drives fail and this is the kind of mechanism you have to build when you want to scale the system right and because drives do fail and we'll see a little bit later how many okay right now we're storing about 600 petabytes of data so just slightly more than came in the last few days but I am amazed by the way these guys deserve a hell of a hand okay if if they do an amazing job to get all of this stuff replicated for you guys for anyone who's doing that so I think that's the first thing to do let's give them a hand okay because it is it is hard to maintain all of this and then they have the little fail drives over there to keep you track it's kind of fun we have more than a hundred thousand drives that are in operation right now so and we stick them in those wonderful red chassis right and that's that's us okay and you've probably seen this shot on the internet because everybody loves to say look at a data center and it's all pretty in red right a little story the reason they're red was because when we did the very first one and built it the guy called us and said what color would you like them because they were just metal we said I don't know and he said I have some red literally it's that simple and we said okay okay there was no guy down there with a good thing going hey give me a give me pms color 127 no it didn't happen he had red so we stuck with it and it's worked out really well it's pretty cool collecting hard drive data so we use the smart mon tools package many of you have heard of that to do that all right the data we collect is the smart stats off of that and I'll show you exactly what we get in a few minutes we collect it once a day we actually scan the drives multiple times a day all right and there's a reason we're scanning we're looking for things that are going wrong but we keep a copy of the data for each drive for each day we've been doing that since April of 2013 okay so that's the data set that's out there if you would like the data is public we publish it it's on our website there's the URL there for you you can download it I think it's over 100 gig these days worth of data we explain to you how it's laid out how you can go in and look at it and we even give you some sequel files to go and play with if you want to go do that to go chest and do your own thing so if you ever if you're ever curious and you got nothing to do for a couple of days you can download some of that stuff now what do we have here I thought I fixed this slide oh well date serial number model this is for every drive once a day right capacity you can see that and then we carry the smart stats so smart stats carry a normalized and a raw value for each of the statistics that are out there there are currently 255 sets you could get not all of them are ever are used matter of fact roughly only half are used that we're aware of from the drives that we have and we collect all of that we store all of that so if you want to know what you know on Thursday June 13 2017 what the smart raw value was smart to raw value was for that drive we have that it's in that data set okay I don't know why you would want to know that but an on button okay I think that's a little bit better I think I'll stay about this far away from it too all right now the most important part from the point of view of what we're talking about today is the failure button so what's a right and that's what you can see there's a failure value all right in that record that means that that drive failed on that day okay the way we do it okay is we actually will scan for drives and when a drive comes up and it's not there we go and try to figure out why it either failed or sometimes a lot of times what happens is they took the system down for some reason perhaps migration data migration or whatever the case may be but we do get failures as well and so the only thing that gets marked there is a one for failure a system drive can disappear from the data because again it's being migrated maybe we took a pod out for maintenance purposes or something like that okay so what is a failure okay and this is the part that's really important the first two are easy right didn't spin out we can't see it the rate array won't do anything with it whatever the case may be right that third one drives people nuts I'll be tired right because it's it's our educated guess from the smart stats values that we see and other things that are happening with that drive so it failed it throws f-sec errors okay and we see it happening consistently we pull it out and we'll pull it out we'll mark it as a failure we'll also by the way before we mark it as a failure we run it through two levels of testing on the back side which are basically nothing more than a quick reformat and then a long-term format which beats the live and tar out of it that we have from one of the manufacturers who shall go nameless and if the drive passes then it gets put back into services it passes both of those so we're really confident that a drive has failed by the time one of those three things has happened to it and and like I said we don't they use it now for those of you who are familiar with our drive stacks okay we have been recording the data and this is the data you're seeing is through the end of June right we I could go in and pull it through today but it I actually pulled it at the beginning of August and it was no real significant difference so I didn't bother updating all of these slides you can see the kinds of things we've had a hundred thirty two thousand drives we've had in play yada yada yada right the thing it's important is the failure rate everybody looks at the failure rate he looks at the failure rate everybody looks at that they go that's the number I care about I don't care about all of these other numbers and there were two basic ways to compute a failure rate and one of them is wrong and let's see if you can figure out which one is wrong some people do it this way number of failures divided by drives times a hundred right hard easy formula and given that data I just showed you the failure rate for all of those drives is 5.62 percent some people do it this way failures divided by what's called drive days are a count of each day a drive is in operation right if it is not an operation i.e. it failed and it's gone it's no longer counted and that's the way and you can see same kind of numbers right drive days divided by failures but now you get a number that's almost half as much 2.56 percent right which one is right well we use the second method all right we use drive days and the reason is for us all right method one that first method assumes that every drive has been operating the same period of time and that may be very valid for you in in a situation where you have five drives in a in a in a rig in a NAS box or something like that and you want to do that computation that's a very valid computation right we don't we have drives in and out of this system all of the time so whether it's a failed drive whether we take a system down for maintenance whether we put in a new system another when we put in drives now we put in 1200 drives at a time right that's what one of those vaults takes 1200 drives comes spinning up right the lights dim everything happens right so that's why we use that method because it accounts for the fact that we have drives in and out of the system all of the time so if you're ever on our comments and and all of that kind of stuff and people are yelling about this they usually are thinking the first one is hey why why doesn't that work and that's why it doesn't work I want to make sure I stay on time now thank you just just because other people have in fact taken all of that data that I talked about and made and applied other models to it okay so we have an annualized failure rate we create and we'll see that in a minute right for those of you who have ever been in the medical bit and side of thing biology side of thing there's something called Kaplan Meyer which is basically a how long will something live versus how often will it fail right and Simon Aaron I'm gonna get his name right Aaron me okay he's from Sweden you can see down there where he publishes it each quarter he updates it we publish this data he publishes it this is pretty boring little chart but he's got a lot of other ones that are a whole lot more interesting but it basically this is the all of the drives we've ever had over time and the chance that they're going to survive after so many days so one year two year three or four years you're looking at something in a but about 88% okay so that's the failure rate over time how long can something be expected to live you put in a drive today is an 88% chance it will survive four years that's that's the kind of thing that this and doesn't and he's done this by the way for all of the different drive models that we have so it's kind of cool and the technique isn't very hard if you know how to do it so a number of other people have done so fun stuff with the data but this is a really good one and he does a really nice job of explaining it measurements okay so when you look when you look you come in quarterly we publish the drive stats and like you said we've been doing this for about four years now this is everything we got here's all of the data there's no hiding it's everything and each quarter we publish two sometimes three different looks at one is a quarterly look tell me everything that happened in the last quarter only the other one is a lifetime look given all of the drives we currently have running how they've been doing since we ever put them in and then the last one is every drive we ever owned tell me how it did over the entire period right so you'll see the data if you go look at the stats and stuff like that you'll see all that cool stuff but that's how you look at the data and so a lot of times we make a mistake by the way and when we have published a report the first thing is the quarterly numbers and people have short attention spans so they look at the quarterly numbers and they immediately just say oh my goodness so-and-so's drive only had had a zero percent failure rate oh my goodness they're a great drive oh well yeah but there's only 20 of them you know and it's only they've only had a whole total of 539 days of which there are there so make sure you pay attention to the data the quarterly data in particular we use the quarterly data as just a mechanism like a vector is a drive moving up or down in how it's failing over time all right and that's a good that's a good vision point for us to do I will tell you pulling these stats and doing all of the magic that we do and stuff like that that's like a part-time job it's like it's like another thing we have to do and so it's we do it and it's great but it's not like that's in my job description anywhere so right now on the other side okay you get some really good looking things because you start to get decent numbers of drives and decent number of drive days so for example I know there's some six I heard there's a couple six terabyte drives over there that particular model seagate which you cannot get anymore by the way sorry that's failure rate annual life's failure rate is 0.87 percent less than 1 percent that's a pretty good number all right that's a pretty good number there's some four terabyte hgst drives that are stellar 0.26 percent that's amazing all right that's that's like you know it's 100 drives a quarter of one failed over the year you know that's that's pretty stellar numbers can't get those either more anymore we we bought I know you can't because we bought every single one of them western digital who owns hgst had in warehouses everywhere all right we sent guys under trucks and in corners looking for the boxes of hard drives and we bought every single one they had because they were great and then we opened up a data center with them that's what we do if you know one thing about back plays is we are frugal we started and we continue to use consumer drives all right which we'll talk about in a minute lifetime stats remember I said the stats now I have stats and I go back this is over the entire period of time right a little less information but you can kind of see that's the math we calculated before okay that's a fun number 7,437 drives failed since we started keeping stats anybody else have that many failures right unless you're from google or facebook or one of those guys should probably by the way their number is going to be a whole lot bigger than that you know I we crow we have 600 petabytes yay right those guys have I don't know 100 times that it's amazing I'm still proud of what we do anyway what we did I did there is I summarized that for all of the different size drives so if you're going down to the store what might you be thinking about now we had a really good run with two terabyte drives right really good every single one we got in the place is great right and then there was a thing in 2011-2012 called the Thailand drive crisis and we had to buy a lot of drives and they were all three terabyte drives during that period and let's just say that the drives were not as good and we'll leave it at that and but you guys and the numbers are the numbers right now you look at this you'll go wow look at those 12 terabyte drives they're doing great remember we've only had them for a year or so so if you start to think about how drives fail they do seem to follow a bathtub curve so there's a little infant mortality at the beginning and then they then they kind of settle into a really nice low rate for two three years then they start to bump up at about three and a half to four years and the failure rate starts to go up from there right and drives seem to follow it the interesting part about that is the infant mortality rate for us has gone almost flat at the front end of the curve for some of the bigger drives I don't know if it's because they're making better drives they're testing them well whatever the case may be but we're just not seeing the same level we're seeing a really interesting it's it's almost it's almost indistinguishable from the middle of the curve now so so yay for the drive manufacturers for that let's see oh my manufacturer everybody always asked which manufacturer makes the best drive it depends right HGST most of the drives we have for them with the exception of one mud model are four terabytes or less and that was generally before they were acquired they have so that number kind of fits that model the early seagates not so good the late seagates pretty darn good and we don't have enough Toshiba drives yet I do have 1200 14 terabyte Toshiba drives in the warehouse that are going to be deployed any day so that'll help some of these numbers but that's what we have so you look at the failure rates you know people everybody wants that answer should I buy seagates right and there's there's I could divide the room in half and this half is the western digital half and that half is the seagate half right and you guys can yell and throw each other stuff and everything like that and then there's a couple of Toshiba guys down at front and nobody pays attention to them anyway so so I where I'm not going to try to solve that problem for you okay the other thing that gets in the way is our environment is our environment right it's a data center we treat these guys really nice I get they go in a nice chassis right they get tested they get put in there it's their air conditioned right we we monitor the electricity going through them it's all filtered and everything like that I don't know if that's the same environment you have at home okay okay I'll just say it but this is the way the data we have a little more stuff lifetime okay so for the drives that are currently in the data center today running there's 98,000 of them and you can see the failure rate right a little less than 2% all right still 4,300 of those of those drives have failed and and that's what we consider to be the most relevant thing I can't go back in history I don't really care about one terabyte drives anymore because we don't have any we just got rid of the last three terabyte drives like two days ago and the only reason we even had them the only reason we even had them was because they were in a rack there was four four pods full of them in a rack and they were in an area where we don't have we don't go I mean it's it's the way that this particular data center set up there's like a hint there's like two or three racks that are just all by themselves at a corner somewhere and it's and they're caged and everything and we were going to build out the rest of that at one point and we ran out of electricity I guess is the best way to put it all right who would have known it right but they and it was it was too expensive it was actually cheaper to go and have another data center than it was to drag more electricity into that existing one and this is the kind of fun math you have to do you know so no I don't want to go spend the $500,000 to have PGD drag another you know 100 megawatts whatever it was into that data center let's just go open one in Phoenix and it worked out but these these happen to be in a rack one of those racks in the corner and those drives just never fail for some reason neither they were in a little set there but they finally they finally left we had a ceremony for them they will we'll even we'll do a little blog post about them and everything like that because that's kind of goofy stuff we do so just all of those models right lifestyle stats of operational models just so you can see the ones that are really kind of fun if you if you stared all the way over at that right hand column which one's the best well it's the 10 terabyte C gates all the way down at the bottom right those have been really rock solid drives and C gate doesn't make 10 terabyte drives anymore they they skip right to 12 they made those for like a week I swear and we bought them we bought 1200 of them and we went to go buy some more because what we do is we run a sequence of things right very typical data center kinds of things you have to think about we put in 20 20 is what's called the tone it's one drive in each of those 20 different pods right and we see how they perform and if they work if they if they keep up with everything else that's going on and and all of that me and then we'll say great then we'll build a whole storage pod out of them and we so we add another 59 because it's 60 in there right and then we do it again and then if we're happy with that we like the results of that then we'll go out and we'll fill a vault with them at 1200 and and that's where we did with that with the C gate ones but apparently we took too long because they decided they weren't going to make 10s anymore and now they're making 12s and 14s are coming they don't they don't I don't believe they have any 14s yet so so sorry can't get those it looks like you're going to need really big drives next year anyway so it's amazing the other one that does really well is the HGST about a half a percent you can see those two four terabyte ones towards the top which are really good and rock solid mentioned the C gate sixes were pretty good too the C gate eight terabyte ones we'll talk about those in just a second that's a really nice thing because one of them is consumer drive and the other one is an enterprise drive so I think it's important understand what we care about okay because it's probably it may or may not be the same as what you care about when you buy a drive all right we care about cost number one the rest of them you can gray out it's almost that much right but that's the second one as I mentioned sometimes power is really interesting to us so for example when we put in the enterprise eight terabyte C gate drives they they were almost one and a half times as much power as the consumer drives and when you're running on the ragged edge of the amount of rack power you have in a rack all right you can't do that because otherwise you have to you can't put in 10 strip pods in there because ten times you fill a whole rack right you can only put in six and that is that's not good for density right so power is important but then C gate has a really nice capability in there called power technology or something like that it's on the next slide which allows you to adjust the amount of power that you're going to give that drive when we could get it in there so cost right now for us someplace around 2.2 cents to 2.25 cents right you can actually get a better price on that every once in a while you'll go down and somebody will be having a sale at Costco or wherever and you'll be able to get it it'll actually math out to be less than that but a long time ago we used to buy drives at Costco now if we show up and say at Costco that we need a thousand drives they don't let us have them so we buy now straight from the manufacturers or close enough so that's about what we pay okay someplace in there the other things in there fits our usage like I mentioned earlier if we put a drive in now and it just doesn't work it fails and and that happens sometimes we put a drive in we put 20 of those drives in there and they just can't keep up okay there's something that's not working right in the environment we don't use it why why beat our head against the wall right failure rates do matter so you saw really nice low numbers we can tolerate anything in a single digit failure rate okay once you start to get above that you're starting to play you're starting to roll the dice really hard on your durability okay so anything above a single digit kind of failure rate the lower the better right right now we're running at about 1.1 1.2 percent and that's a really nice number because it keeps the durability going and and their durability is remember they're sharded across seven across 20 different things 17 and three is a mechanism so I can lose three whole systems like I mentioned earlier that's part of the durability that you do the calculation with but if I have drive failure rates that are 12 I start and rebuild times that are now starting to approach two weeks right on some of the large drives right all of a sudden a math starts to get funny so we like single digits warranty I don't care about warranty we don't care about warranty it's it's almost not worth it for us I know it's worth it for most of you but when a drive fails the time it takes us to go and fill out all of the information put in a box send it off hopefully get it back okay and they're going to send you a refurb drive right which I really don't want you know warranty is not interesting and then the last one for us is speed okay we we're real the way when we build that array of 20 there are 60 of those in a vault I have no trouble accepting data onto those disks none at all right the gating item is is you can't get me enough data I mean it's just it's that simple yeah yeah there's a I don't have the slide with me actually it's probably on my computer somewhere but we've seen it when we first started it was around 11 cents um and and then it's over the over the 10 or so years we've been doing this it's come down and you can literally see drives by size do that and they come down and it'll start and it'll go down then the next one will be introduced and it'll be a little higher and you got to weed it out until it gets down to where the the previous one was and then and and they they're just consistent the only time that broke was during the drive prices and that broke hard okay we um our drive prices through normal channels went up 3x yeah so there's a that's a really good point um and it has and the other thing that factors into that is density storage density in a given spot okay so one of the things we've been doing over the last three or four years now is migrating from the smaller drives to larger drives so I take out a four and I put in a 12 I just got three times as much storage for approximately the same cost okay now I got a bunch of four terabyte drives but I got four years out of them right I turn them over and they get recycled I don't they end up in a pile in China hopefully not um so that's a good point about that and there is exactly that kind of math we do so so let's compare this is a eight terabyte drives okay by by the same manufacturer two different models once a consumer one once an enterprise one about a year ago we did it the failure rates in that second column you can see where they were now you can see the current failure rates that's within the margin of error by the way so you're sitting there going hmm I could spend uh $129 to buy that eight terabyte drive all right maybe 159 or I could spend $429 to buy the enterprise drive I wonder which one I should do if I'm interested about failure right um for us it didn't matter now for us also let's just say the price of those things is approximately the same okay you can't do that I can do that okay because I buy a million dollars worth of drives at a time um okay um so but that's what we see out of the data so if I was looking at that and going what would I do I might think a consumer drive because really it's the failures is about the same I will tell you a little difference that we've seen and this is anecdotal I don't I'm never going to write this down anywhere right the consumer drives seem to have this tolerance for things happening to them inside like bad sectors that have to be remapped around and all of this kind of stuff right the enterprise drives don't seem to have that same tolerance when they start to go they just go they don't they don't give you a whole lot of notice okay it's kind of like I don't feel good goodbye okay it's it's just an observation I think it's just because the situation I think I'm going to put a drive in a consumer system and consumers are they're not it's not going in a data center it's going in it's an external drive and you have it next to your thing and you drag it around you bring it over to Aunt Molly's house and you drop it on the floor you know so they have a lot of tolerance built into them but if you're making the decision for yourself about what do you got to think about these are the things that you need I think you you know might want to consider right the warranties of course are different all right typical enterprise one is five the consumer ones are two at one point during the drive crisis they were one year and and if they could have gotten it down to like 90 days they would have done it during that period consumer drives are really much less expensive for just off the shelf all right enterprise drives have a lot more features I have the power choice technology that's what I was trying to think of for example from Seagate but they have a lot more things you can tweak in the firmware all right to make that drive perform that fits into your environment really well they are absolutely faster to read and write absolutely again we don't care because that's not where the bottleneck is there's the bottleneck is just getting the data to us you know and we have plenty of network for that so it's just a matter of waiting kind of sitting around doing nothing but on the consumer side they do use a whole lot less power out of the box and again mentioned more for your failure so much is right for you yeah so they seem to do quite well okay and like I said and then and then they get sick and die and it's it's that fast it's like it's sometimes it's it's hours you'll see the first little it'll throw an absolute check or something like that and then it goes offline you know two hours later and you go I didn't have time to look at it you know they're just any HGST by the way but they seem to be the same way in its behavior we just thought it was an HGST versus Seagate thing but it seems to be the same it seems to be an enterprise versus consumer thing all right helium we'll have plenty of time for questions and getting all of this stuff out of the way so helium so any large drive now all right starting at about the eight terabyte drives although there were some six terabyte helium drives but starting with about the eights and moving on up are going to have are going to be filled with helium they finally got that technology right it was they were trying for years to figure out how to keep the helium in there because it keeps wanting to get out and and they finally figured it out they even created a smart stat the smart stats now to measure the amount of helium so for example HGST smart stat is 22 and it's 100 is a raw value is the value and anything less than that means it's leaking and they have a tolerance number but they haven't told us what it is but we have a handful of them running in the 90s right now so we're trying to figure it out the Toshiba drives we just got in the 14 terabyte Toshiba drives are helium filled drives and they have two numbers 23 and 24 and they measure helium at two different levels inside the drive a high they call it high and low I think it's above and below the ladders the platters and it's the same kind of thing they and it's the same same kind of thing they're actually in many ways still learning what that number means to them a little bit because it is a fairly new thing for them so what do we what can I tell you about that so we have some helium field drives on the top we have some non helium filled drives on the bottom one of the funny things about the bottom we'll talk about temperature in a minute the eight terabyte airfield drives ran hot they they did they ran three four five degrees Celsius hotter than the lower end drives and it's just there's so much going on in there the helium filled drives run a little cooler they run back at normal levels and that makes sense that's one of the things they talked about and right now okay we don't see any difference in this annualized failure rates between helium and air all right which is which actually bodes well that means they picked a good technology they move forward it didn't cost them anything you can see the different failure rates out there and you can compare it now if this were going to be a perfect test the drive days there would be but roughly the same and they're not right now so it's not quite apples to apples but it's pretty good all right and there's enough data there to start to actually get to that kind of conclusion that it looks like the helium drives are going to have a reasonably are going to be able to perform at least as similarly to airfield drives they still cost a little bit more or a lot in some cases when we bought those hgst ones there we bought those like four years ago three years ago about three three years ago and those were about four hundred and fifty dollars a piece and that was a crazy number for us we bought 45 of them so we it was the most expensive storage pod we'd ever built but they're doing okay they're in three years now three plus years now that's a really good annualized failure rate after three years and they're hanging on so we'll see we'll see what's going to happen we're going to track them over time and see if over time the helium drives continue to maintain you know that kind of performance that kind of failure rates with that uh let's see what else um yeah so we'll continue to do that uh those are the two uh eight terabytes we we have um i was going to throw the sixes in there too but they're really like I said there were a couple of six terabyte uh helium field drive models but they didn't make them in any quantity and they kind of experiment with it with them they really started to do it in the eights is where the technology and if you buy anything above that now that's what's going to be in it okay um chances are it's going to have helium in it uh interesting little thing though both of those models up there for the helium are what's our enterprise drives or enterprise class drives um it'll be interesting to see if manufacturers continue to build large quantities of consumer drives uh in that size um the reason we buy the enterprise one is again the price is about the same in quantity okay so we buy a bunch of them um I don't know if I could buy let's say 12,000 consumer drives right now you know if I wanted to buy uh 12 terabyte c gate consumer drives I don't know if I could buy them I don't know if anybody would sell them to me got it um so and that's that's part of the that's part of the way that drive manufacturers manage their channel this they might you might be able to go to Costco and buy one or two okay um we we did that when during the drive the thailand drive crisis we went into Costco and best buy and bought drives off the shelf because we couldn't get them anywhere else um um yeah but I don't want to do that to try to buy 12,000 of them um there's well I was going to say there's not enough Costco is what there probably is um all right so nothing about that uh smart stats and you actually predict right if a drive is going to fail or not so we tracked five stats by default right we did we've been doing this for years we talked to drive manufacturers and lots of folks and I said hey these are five good ones all right and so we tracked these numbers um and one of the things I did a few uh a little bit ago was say all right if there's an operational drive if I look at all of the drives that are running right now how many of them have one of those one or more of those attributes right better greater than zero and that's it's either zero or a number better than zero zero is good anything else is bad right and just so happened about four point two percent of them were like that so then I looked at all of the failed drives and I said well then how many of the failed drives had the same thing right 76.7 so you if you're a stats guy you're looking at that going that doesn't feel like a very good predictor right those five little stats it kind of looks like it it's obvious there's really big gap in there but you're you're not sure you'd like to see that number on the other side be what 95 right a couple of deviations out you start to feel good about it so so really smart people okay not me um over at IBM uh Switzerland uh got together did a wonderful little paper a couple years ago um and that's where you can find it if you're and if you um uh if you're not good with math it's so it's a fun read um but what they were able to do by drive model this was the amazing part right by drive model show that you could actually predict with that kind of certainty okay when a drive was going to fail and that's pretty amazing if you start to do think about it right so wouldn't you like to know what the 97 percent degree certainty three days ahead of time that a drive was going to fail that's awesome right now you have to calculate that for that particular drive is a cg4 terabyte drive right then you got to calculate that for every single drive and it gets one of these interesting little things of well somebody's got to have a lot of drives to produce the data to calculate it so that everybody else can use it okay so um but it's interesting that the drive stats that they and that's exactly how they did this they looked at all kinds of drive stats that we had they used our data to do this with um and that you are able to get pretty good now do you look at the hg st not so good that particular model I had three days out 84 percent I don't know if you really want to throw away 16 percent of your drives that are good right I don't I think the other one's pretty cool like three I could deal with three percent right so it seems like there's some way you can calculate this right um I like I said uh this is it's not my day job to do all of this stuff so we're trying to run a backup company and cloud storage company so if any of you guys want to do it we'll give you the data is there um uh to do those kinds of things but it is interesting now I've heard people by the way talk about drives that as smart stats and say it's a bunch of garbage it's a bunch of garbage the different manufacturers though they just they don't they just spit out numbers who cares right I don't know that doesn't look like a bunch of garbage right um that looks pretty good and and they were really if you look through the paper that they did um they really spent a lot of time with it uh last things last temperatures right just for the fun of it because everybody asked this question a number of years ago google did a study right and said eh temperature doesn't matter you just crank up the heat turn down the air conditioning you can go right so we wanted to figure out if that was true because I don't mind saving on air conditioning right especially since we built the data center in phoenix um so the average temperature of operational drives for us is you can see around 77 uh 77 degrees or so I converted to Fahrenheit because we're in America um but um and you can kind of see how that chart is and there's a handful of them that run at 45 degrees but um you know which is pretty warm by the way you're going to really start to see um but all of that is within the range of a drive right and this is taken by the way right off of uh the smart it's the sensor inside the drive so this isn't like in the chassis or us sticking a thermometer on top of the thing or anything this is inside the drive itself um and so but all of those fit within the parameters that they give you of the operational range of a hard drive right so none we've never had a drive fall outside of those parameters all right um now the interesting part would be failure is there any fit correlation to failure so I broke it down by drive model this once again everybody cares about that and you can see the hg the the three mott manufacturers aren't even close right how they fail um you know hgst down at the bottom it doesn't look like there's any real correlation it could be anywhere along there the seagate one maybe but not much um and the western digital one is a batman cow so that's what I see I don't know what you guys see um the the front thing is is towards the end which is where google spent their time talking about once you start to get above 40 degrees celsius you actually do see that bump up of drive failure right but we just don't see enough drives fail there um to to actually say that's what happens right they seem to fail in other places it'll take a while for example uh the western digital one at 30 degrees celsius is 18 percent of their drives are failing there right that's not much above their normal temperature so I don't know I don't think there's any real correlation uh between the failure and during a normal range of operation uh once you get up there all right so you know you know what we talked about so I'll leave uh I'll leave it with questions uh since we got a few minutes here for questions anybody got anything yeah no no oh sorry uh have we done any analysis for what file systems uh and how they might affect drives and so on right um the answer is no we use our file system uh now I'm going to remember what it is but it's a standard one uh I can't remember what it is but it's a standard one you everybody and their brother would use um uh so we didn't invent our own or anything like that um I do know some folks who have tried that um because they do some really funky things as it relates to riding blocks and so on but um but we just the drive that's what works with the drive and it works so you know no difference with file systems that I'm aware of so uh anything else so so the question is um is since we started putting our numbers out have we noticed that uh the consumer drives that's gotten better or failed less uh yes we've noticed um uh while I would like to say I'd like to take credit for that uh because sometimes transparency is a is a very good thing um uh I I don't think Seagate is sitting around you know in board rooms going gosh they just published their data uh we better get better um I also think by the way they learned a lot that the Thailand drive crisis was really an awful event uh for a lot of reasons um and they really took it and they really got hurt during that for a lot of different things publicity everything like that um so I think what you're really seeing over the last few years is just them making drives now with a reliable set of parts and so on um I'd like to believe we had some influence on them making better drives um uh and we have good relationships with Seagate for example with all the drive manufacturers maybe they just give us the good ones I don't know um but um you know but we have observed that yeah that's a fair observation uh anything else yes how do they keep helium in the helium drives uh that's a really good question I they I I know they spent a lot of time creating the case that goes around it and how they pack it I don't know the mechanics I was reading an article a few weeks ago about it um about how they did it because we got the Toshiba drives um and the history and how it was actually I think western digital which got the first commercial versions of them out um but I don't know the the mechanics and I don't think they share a lot of that there's this general notion of hey we did it and we used the flexi core of loa ba marketing name thing uh to do it and um but they don't give you the specifics of you know hey we coded it here and we did this year and we plugged this gap here and and all of that so I mean it has to it can't let anything out helium is just going to leak out it can actually leak out through a lot of substances uh you know so yeah so that's right there is no flow so they've had to reinvent the drive a little bit um the the question was is with air drives air helps uh helps the heads a little bit there um and helium doesn't do that because there is no flow um uh basically how do you get rid of heat okay uh we that was one of you know so those are the kinds of things that uh they have managed to figure out how to do um but they can't lose the helium that's in there they they put it in there it's sealed in and it's not like they come around and plug in a thing every so often and add some more helium so so yeah there was another question so we actually wrote our own erasure code um so it's it's like rate um but it's it's I'll just say it's like rate but it's different and um there's we we did publish that uh and how we did that and we put it up on get hub if you want to read it but it is that same it's the same kind of a notion of sharding something across um you know X number of devices uh and having to be able to use have so many of them to restore the entire thing uh and so it is that notion of what rate is we used originally rate six um and so a lot of the storage pods still run rate six um but all of the vaults run our own erasure coding uh anything else you got uh how are we going on one one last one so and fun question um how much uh if you're looking at like S3 uh you know they seem to they charge a little bit more than we do how much of that is profit right um so I know they make a lot of money and Jeff Bezos is making more than our CEO um so um you know now they have scale to enormous area so that certainly adds cost to some level or another um they also subsidize some of their other businesses with the money they make and and all of that um I think we like we do a really good job they do some things better they have a lot of compute capabilities and all of that so I'm not going to tell you we're the same service uh but for what we do we try to make it as economical as possible um and we'll always do that so um and even though our CEO will be poor all right thank you very much