 Hello and welcome. This is SMART 8 Update, a permissively licensed alternative to SMART control presented by Chuck Tuffley and Michael Dexter for EuroBSDCon 2021. I am Michael and you should also see Chuck in the video. So I will give a quick little overview of how we got to this point and Chuck will take over for the technical difficulties. In my day job, I help with storage systems and I see a lot of disks and I talk to a lot of disks. And the general conversation is a bit like at play school where it's like, hey, how old are you? How are you feeling? Give a temperature. And that's it. It's a nice quick exchange. But using the de facto standard tools that the whole world uses, no one's ever been fired for using them, SMART control, part of the SMART MONTools project, some pronounce it SMART cuddle, some spell it out SMART CTL, but I'll try to be consistent. It's a bit like getting a full medical record every time. So it's like, oh, hey, you haven't contracted malaria since yesterday and your hemoglobin levels are still good and a whole bunch of things that don't make sense to the user. So in all practicality, I often want to know, okay, here's a new disk. It is from a large popular vendor. Let's see how many hours are on it. Because if it's new, it should have between like one and a zero and two hours. If it is not in fact new, it could have say 30,000 hours on it. And I've seen that. And so asking those simple questions and getting unambiguous answers is in fact really important. And related asking for a temperature may indicate if say a fan has failed. And if you have an array of like 16 drives and four are really hot in one corner. Well, maybe this has nothing to do with drive health, but you've lost a fan and you have that those tea leaves to work with for diagnosing that. So in looking at the tools we have, part of the problem and I forgive the SMART control team for this is that the vendors have not been very good about the information they provide. There is secret information. There are fields that make sense to them and no one else. And that's just life. There is misleading information. If you use a Seagate Barracuda, one of the first entries is this huge value of errors. Well, those are internal and you should ignore those. But if you ask a new user to sit down and check for errors with this utility, suddenly it's like, my disk is dead. It's like, no, no, ignore that. Like, well, but you said, okay, fine, then you go very carefully read the tea leaves between a few other values. So that's been frustrating. That's not helpful. And so if you've ever watched one of my talks, it was a bit of a theme of sort of eunuch's visceral epiphanies. And my epiphany with SMART control was, wow, this tool is not optimized for either human readable output or machine readable output, which leaves the question like, who is this tailored to? Because I'm not feeling quite where that's headed. So I suppose developers can be like that, where they implement a spec and one thing leads to another and we arrive at this point. But that's been frustrating. And SMART control to its credit finally over the years received JSON output. So you can further parse it. And I touched on that in another talk where with a bunch of JQ and other things, you can finally more or less unambiguously get that one piece of information you want. But that's a whole lot of steps for what can be a very simple value. Just one little integer that's like, hey, how old is this? So I started having experience that I started looking into, well, what are the options? What's out there? What's built into the OS's and found things like, wow, that you can use cam control to throw values added on the slide here. I have an example of if you throw this carefully crafted string at a drive, you might get back results that are machine readable, but with a little help can somewhat give you the information you want, which is promising, I suppose. It's not super user friendly, but if it's unambiguous and consistent, fine, I will wrap that up in other tools and it tells me what I want to know. OpenBSD and NetBSD have ATA control, which does have some rudimentary SMART control and SMART information. And I looked at that now that freeBSD is relying on, say, the cam nifty geom structure and cam. It's not necessarily compatible with anything on another operating system. So on freeBSD specifically, it takes a bit of a fresh look at things. And so in that research, I compiled like a dozen pages of everything I could find on the subject. That led to a paper I wrote on the subject and go ahead, Chuck, next slide, and eventually a presentation at AsiaBSD.com. So in looking at, well, first, what's terrible? We want unambiguous information and sometimes very little information. And if we fast forward from whenever it was that SMART control was looking at ATA drives, pre-SATA, pre-you name it, there has been progress in the world, such as OpenZFS, which has a lovely syntax for very specifically and perfectly walking the line between human readable and machine readable, specifying a specific parameter. And I'd be example of like a dash o for option this. I want that or maybe comma separated and you can remove the header from the output or tab delineate it and just many niceties that probably didn't occur to the original developers. And for years prior to the JSON output, one would maybe do a few passes with regex to first get rid of those commas that they put in two places between the size of the block device values or a few commas there and a few elsewhere and a few parentheses here and you strip those out and it was like, no, not cool. So from a usability perspective, okay, I daily use something that has a better output. And in the big picture, it's like, well, okay, everyone uses SMART control, fine, but everyone used gross and along came mandoc and it's like, well, let's rethink the problem. Let's understand the problem first, address it and dive in. Now, fundamentally, when it comes to say importing SMART control into say ZFS so that you could have values presented on demand about the health of a drive, there are some very convoluted external scripts to do that. But what if the license permitted it to be included seamlessly and it did one thing well rather than all the things that that package does. So I screamed from the bell tower, hey, I think there's a room for improvement. I am not a developer. I cannot implement that short of those scripts to throw a few raw commands that drives and get them back and hope for the best. So that set the stage for this. I put that out into the world. And Chuck kindly responded to my cry for help. And in the big picture also, hey, no amount of turd polishing on SMART control will change the license, such that if you want it to be in base, if you want to be integrated in other projects, not going to happen. There's no amount of optimism, no amount of horse trading that will make that happen. And early on, I thought, okay, fine, let's talk UNIX. What if SIS controls were there for device name and then the SMART value by a numeric value? And so I pretty much put everything on the table and said, okay, let's rethink this and moved on to like, well, should a virtual machine have synthetic SMART data from the host so that it has some insight if it's on safe and reliable storage or that's something to reach out about and then we're going to have trouble. So that was the context I established. And again, Chuck kindly came to the rescue and I'll let him take over for the technical details. Thanks, Michael. So, yes, I was one of the people that was listening to Michael's talk and to give some context to it at the time I was working on a product that tested ATA drives. And one of the features was it retrieved SMART data for it. My job was to then support NVMe. And when I went to look at how NVMe did SMART data and compared it to the ATA infrastructure, they just differed wildly. And in the name of getting something shipped, I kind of shook my head, held my nose and said, okay, we'll just do a totally separate interface for NVMe because I don't have time for this. But with Michael's talk, the question came up again is, so would something like a live SMART be possible? And so in the course of that investigation, this program that we're going to talk today about called SMART is the code platform for me to test out my ideas. And for better or worse, it became a little bit useful. And so I've been adding features to it. So you want to play along at home, there's code up at the GitHub. My apologies, it's not quite current. I'm having some sort of weird get issue, or I could just say get issue. And there's also something available in previous deports. All right, so let's talk about SMART. So SMART, if you aren't aware of it, stands for self monitoring, analysis, and reporting technology. This gained popularity with ATA drives. And the 30,000 foot view of this capability is to have the device provide information to you as the user that as you look at it over time gives you the ability to predict drive failures. Now, when you look at what data comes back, it doesn't look anything like what you see in the output of something like SMART control. At its very fundamental, the data is either a list, or a structure, or some number of log pages, all of numerical values. And just to keep things interesting, ATA vendors do not agree on what a lot of these values are. But still, when you look at it as a whole, it's all providing health data. And so I started to believe that actually getting ATA and SCSI and NVMe to report similar, or to present the health data in a similar way should be mostly possible. How hard could it do? All right, so to understand kind of some of the complexities here, we're going to need to dive a little bit into the various protocols that SMART supports. And you'll get a very good sense of why ATA is not the same as NVMe, which is not the same as SCSI. Like I said, at a high level, all of these protocols support returning health data from the drive, but you get that data in different ways. So with ATA, you're going to issue this smart read data command. On NVMe, on the other hand, they have a smart health information log page that you would issue a get log page to get back. SCSI is a little bit similar in that it provides all the data in log pages. But instead of having one log page, it feels like a billion. It's more like seven. Now that we have this data, we can start looking at it. And the content is mostly different, although if you squint hard, it starts to look a little bit the same. For example, ATA has this concept of a right error rate. On NVMe, you have these media and data integrity errors, which include rights, but they also include other things, and it's a raw count and not a rate. And then on SCSI, you have the total uncorrected errors for rights. So while they're all talking about rights, they all kind of talk about it in a very different sort of way. The other thing that gets interesting here is whether these protocols, the health reporting is standardized or not. Both NVMe and SCSI, the contents of the log pages are defined by the standards group. So for NVMe, that's the NVM Express Technical Working Group, and for SCSI, it's T10. ATA, on the other hand, doesn't have a standard for the contents. They define the mechanism to retrieve this data, but the contents are completely up to the vendor. So each vendor is allowed to decide which attribute IDs they're going to support, and then they're allowed to decide what each attribute ID needs. So this fundamentally means that if vendor A supports attribute 174, it can mean something entirely different than vendor B who also supports attribute 174. So in trying to digest all of this, I started thinking about, well, what would I want? And so naively, you look at these as very different things, and so it would be very easy to fall into this trap of saying, well, if the protocol of the device is ATA, then I'm going to need one of these ATA buffers. If it's NVMe, I'm going to need an NVMe kind of buffer and so on. But when you look at what would I do to create a library, this sort of, if you went this route, this code would end up expanding everywhere, and everything would end up having to special case each of the different protocols. And this felt kind of unwieldy and terrible. So instead, what I came up with was a protocol independent structure for this data, what I am calling a data model for smart. And the data that you get back is in these self-describing buffers, I'm calling maps. And the other part of this is if you didn't want to actually incorporate a live smart into something like ZFS, it would need to support multiple different operating systems. So being able to split this code into OS dependent and OS independent was going to be important. That has other advantages in that it lets you take advantages of system specific capabilities. All right. So let's take a look at each one of these things. So for this data model that I'm calling dumb, to kind of understand how we got there, let's take a look at each of the protocols in a little more detail. So for ATA, you're going to issue this smart read data command. That is going to return you a list of attribute IDs. And each of these attributes are composed of some raw values, flags that tell you how to interpret some of the data, the min and max seen for that, and some threshold values. And each of these elements that gets returned are between 8 and 48 bits. NVMe, on the other hand, has a smart health information log page. So you would issue a get log page. But the data that comes back is not a list. It's this packed data structure. If you're familiar with C programming, it looks a whole lot like a C struct, and that each element has a position in the structure and a byte size. And the data that comes in each of these elements are numbers. And they vary between one bit for flags that tell you whether something has happened or not to really big things like that are 128 bits. For example, the number of blocks that have been read and written over the lifetime of the device. SCSI is a little bit like NVMe in that it reports information in log page, but instead of having a single log page, it has seven or eight. So you have errors for reads, writes, verifies, the last in errors. It can also report temperature and start-stop cycle counts. And then finally, there's this informational exceptions log page, which is supposed to tell you whether or not one of these exceptions occurred. Now, when you get this data back for one of these individual log page reads, it doesn't have attributes, but it sort of feels a little bit similar. It's listed as a bunch of parameters, and each of these parameters has a code. And in that code, you have a packed data structure, which SCSI has is a little bit self-describing. Now, the data that each of these has are either numbers, they're big end-in, by the way, so that's fun, or they're strings. And just to keep things interesting, all of these things are variable length and size. So when you boil all that down and you try to come up with this common unified model for health data, the thing that made sense was health data is now a list of log pages. Log page will have an ID and a description. And inside the log page, it consists of a list of unique attribute IDs. And the attributes have things like a value, description, size, et cetera. And in thinking back about the previous slide, you might say, hey, that looks a lot like SCSI, and you'd be right. So here on the slide, you can see that I have kind of an example. So this, the health data for this device has consists of three pages. They have attribute IDs and some values. And in some cases, they also have text descriptions for each of those attributes. So now that we have this generalized model, let's look at how that applies to each of the different protocols. So for ATA, it doesn't have a concept of log pages. So to get that page ID that we're using, we're going to actually use the value from the command feature field. So when you issue a command to an ATA drive, that's how it tells what kind of command it is. So for the smart read data command, that common feature value is 208 or hex 0x D0. Now, ATA does have attribute IDs. Who knows what they are, but it does have them and they're unique. So that's good. And then we're going to use the returned raw value and some guesses as to what the descriptions actually mean. So then when we run smart against an ATA drive, in this case, ATA0, you can see these three things. So that first column shows 208. That's the page that we just read. That middle column are the attribute IDs. So you can see that the first three are 512 and 175. And that last column are the raw values. NDME has log pages, so we can actually use the log page ID of two, which is corresponds to the smart health informational log page. Now, it doesn't have attribute IDs per say. So instead what we do is we use the byte offset to each field as the attribute ID. And being a byte offset into a structure, that'll be unique. So we're all set there. And then we use the value in description as defined by the NDME spec. So here when we run smart against NDME drive, in this case NDA0, and we look at it, we can see that we have the log page ID two in that first column. The middle column is the attribute IDs. And to help you understand how that works, I've printed a subset of the NDME specification that describes what comes back in the smart log page. So you can see that the critical warnings are at byte zero. The composite temperature starts at byte one and is a couple of bytes long. So that's that attribute one. And the available spare starts at byte three. So that gets an attribute ID of three. And so we can see that my available spare is 100%, which means I can abuse this device for many years to come. For FreeBSD people, note that this is NDA0. FreeBSD actually has two NDME disc drivers. There's NVD and NDA smart requires CAM, which we'll talk about a little bit later. And so it needs to use the NDA. See your NDA man page for more details. All right, then for SCSI, it has the same concept of a log page ID. So we can go ahead and use that. And then instead of using attribute IDs, we'll use the parameter code to uniquely identify the health elements. And one thing to note here is that parameter codes in SCSI are numbered from zero to N. And they are that for every single log page. So every single log page will have a parameter code zero. And a lot of them will also have a parameter code one, which means that you can't actually use this attribute ID to uniquely identify a health element like you could for NVME or for ATA. And then finally, we use the value in the description as defined by the SCSI spec. So when we run smart against a SCSI drive here, DA0, you can see in that first column, we have a list of the different log pages. I printed out a subset of the SCSI spec that showed how to map the page codes to the log page name. So you can see we have the writer, counter here, the verify, non-medium errors, and the temperature. So let's take a look at the temperature a little bit. When you go over to its section in the spec on table 351, you can see that parameter code zero is the temperature in Celsius. So that first line of data says that this is the temperature log page, and that 31 is the degrees in Celsius of your drive, which means that it's running very cool. I think if I'm doing my Celsius math right, and should be just fine. Okay, so now that we understand how this model works, that we have these list of pages, pages are filled with attributes and attributes have these values, let's talk about how that translates to the mechanics of LibSmart itself. So as a user of the library, you need to establish a connection to your drive. So if you want to be able to retrieve data, you need to be able to refer to it. So what LibSmart does is it abstracts out that connection to the lower layer stuff and keeps you fairly oblivious to what is the contents of that R. And that was a fairly intentional decision to try to mitigate ABI issues. So my belief was that as this tool progressed, I would learn things and be smarter about it. And so I didn't want to expose too much of the inner workings and break things like ABI. Now this is where the device independent dependent layer kicks in. So the handle actually translates into these two pieces. And the thing to note here is this is actually allocated by the very lowest layer, the device layer. So here we can see the code on the right hand side at the very top level, you as an application would allocate one of these handles. It's a void pointer. So there's really nothing your application can infer from it. But this is the handle that you would pass back to every subsequent LibSmart call to let it know about the underlying details. Now internally in the device independent layer, so at the library level, this smart handle turns into three different things. So one, it tells you what protocol this is, is this ATA, SCSI and NVMe. It stores some information about the device. So what's the name of the device, who's the vendor, firmware revision, that sort of thing. And then finally it gives a list of pages. And the pages are really, these are the pages that the lower layer needs to go query to return all of the smart data. So for example, in NVMe, we saw there was a single log page, so this page list would have a single entry. For SCSI with seven pages, this page list would have seven entries. And then finally, the device dependent layer, it's allowed to define its structure in the way that it wants. But what it does is it uses that smart, that common smart structure as the first element. And then it hides all of its device dependent stuff after that, such that when it allocates this and passes it back to the higher layers, they can just operate on the smart data and not really worry about the internal low level details. So now that you have this connection to the drive, you want to go and read data. And the data that gets returned is returned in something called a map. So this map is a self-describing buffer of attributes. So this includes both the raw data that's returned by the device, the number of attributes in that buffer, as well as an array of attributes. And the latter one is sort of a predigested version of the machine version. So here we can take a look at the map structure and that smart buffer, that's the raw data that's returned by the device, the count, and then this array of attributes. The attributes themselves, so the high level goal is that once you return this map, you have this array of attributes that you as a consumer can then go through, be able to identify which attribute you want and get some information about it all without understanding anything other than how to traverse an array. So what allows you to do that is the contents of each of these attribute structures. So this gives you a way to uniquely identify them. This is the tuple of the page ID and the attribute ID. And it also gives you a pointer to the data. One of the things that I was sensitive to was when I retrieved data from the device, I didn't want to make a second copy of it just for the application. So instead what you have in each of these attributes is a pointer to the original data buffer where this data resides. And then some information about how to decode it. So this will give you the number of bytes in that field, some flags to help you determine how to interpret it. So for example, is the data in big Indian format? Is the data actually a string and not a number? And then finally, we have this raw pointer that gets you access to the actual data. ATA has this concept of thresholds that I don't really want to get into, but long story short, it turns out that this map structure can actually help you interpret that as well. So at the lowest level, the device or OS abstraction, this is currently only implemented for FreeBSD. And besides being a FreeBSD developer, one of the reasons that I picked this is because FreeBSD has this storage device interface called CAM that makes this whole process easy. It simplifies a lot of the details of opening up a device and passing commands back and forth to it. But I did want to make sure that I had gotten the abstraction right. So I did do a proof of concept for Windows using a really cool library from CK called OpenSeachest, which allows you to do various disk operations across a variety of operating systems. And so while it wasn't straightforward to use, it was useful enough to convince me that my abstraction was correct. So at the lowest layer, there's very little that needs to happen in terms of an API. You can open the device, you can close the device, and then you can read a particular log page. So moving up to the device, the OS independent portion of this, we look at the lib smart. So this is the thing that would actually be interacting with the device abstraction. So here, again, that kind of forces some of the same sort of API decisions and that you're going to be able to open and close the device. The library will also tell you if smart is supported by the device. So while many ATA drives do actually support smart, it's optional. You have the ability to read the health data. This returns that map that we were talking about earlier. It gives you a way to free that and then some helper functions to print out the smart data as well as some of the device information. And then at the very top layer, we have the application smart itself. It actually does very, very little. So it's responsible for the command line option processing. It does all of the lib XO, set up and tear down. And then it's the thing that makes those API calls to the smart library to open and print and that sort of thing. And here on the side, you can see the health output from the smart command and the various options that it gives you. So let's talk a little bit about the output format. Now keep in mind that the original motivation was to collect values over time. So imagine that you have a fleet of servers and you want to be able to predict when those drives in the various servers might fail. So you're going to know the specific attribute IDs that are good indicators or that you think are good indicators of drive health. And you really don't need all of the decoding and other attributes that don't have anything to do with that. And you just need to store the raw value. And you're going to look at this over time to understand trends. This would be called via cron, something like cron or perhaps you have a fancier monitoring framework like Prometheus. So here we can see that with smart, I can say, hey, I only want attribute ID five from device ADA zero. And all it does is returns that value. Easy peasy. Now I referred, I mentioned lib XO earlier, all of the output formatting is driven by lib XO and they describe themselves as a library for generating text, XML, JSON and HTML output. So again, if we go back to our previous example of looking at attribute ID five, I can actually tell it, hey, use lib XO. And this time I want to see that output in JSON for ADA zero. And what we get back is some ADA is some JSON output that shows that attribute. So at the top level, you can see we have a drive drives have attributes and this particular attribute that we're looking at has a raw value of zero. Now you can picture lib XO as instead of using print outs in my underlying code, I'm telling XO, this is how to describe the data. And it knows how to translate that to the various different formats. So here we have a little bit more detailed version of what that JSON output could look like this time with some of the device information. So you can see that this is a Samsung drive, the revision for it, serial number and then the attributes. And each of these attributes now you can see that we have the page ID, which is the 208 that we know and love, the attribute ID itself of five and a raw value of zero. So after a fair amount of arm twisting by Michael and Alan, I caved and added some attribute decode because mostly because I got tired of explaining what attribute ID five was. So smart now can actually decode some of these attribute IDs. So for SCSI, the text comes from the specification. NVMe is very similar with the caveat that it's smart enough to look at the drive spec revision and it filters the output according to what should be there. And then for ATA, it's complicated. So here you can see we've told smart to do some decode and instead of seeing attribute ID five, hey, now we know that that's the reallocated sector cap. So now let's let's go back to why I keep waffling on ATA decodes. So again, remember that the smart attribute structure for ATA is specified but not the attributes themselves. So if the spec doesn't tell you what each of these attributes are, what is that? How do you decode them? Well, you end up having to get the definitions from each vendor. And just keep in mind that that vendor A can decide that attribute 174 is a temperature thing and vendor B can decide that attribute 174 is for flash health. And really the only way that you can tell what attribute ID 174 means is by doing a match on the drive model in the firmware. And so things like SmartMon have formalized this into these regular expressions that take both the drive model and the firmware vision, apply this regular expression to it and then have a table that maps that to what these values mean. So inside of SmartMon tools, this is something that they call the drive database. It's about 6,200 lines of code and it compiles down to about just over 28 k bytes. And one thing to note is that it is GPL. So if you go back to one of the earlier tenants that this should be, have a permissible license, this means that using SmartMon's DriveDB directly wasn't really an option. So what do you do? Well, okay, funny story. So the ATA vendors were very frustrated by the Wikipedia page that describes Smart because at the end of that page there is an extensive amount of decoding of what each of the attribute IDs mean and by and large, they're wrong. So the vendors tried to get those definitions changed but the sticking point was there was no authoritative source of authoritative source of what those definitions should mean. So smash cut to IN CITS-TR54 or Smart Attribute Descriptions. This was basically 8 or 10 of the ATA vendors getting together and documenting the agreed upon definitions. So for all the vendors that agreed on what attribute ID 5 meant, that spec says that's a reallocated sector count. And since that's a publicly available document, that's actually what Smart is using to do the ATA decodes. And the future, I hope to have a text-based Drive database probably similar to what SmartMon does, but that's a future TBD. And here you can see that in the case where the IDs aren't agreed upon by all of the vendors that it just prints the raw values. That was what I had. Thank you for listening to our talk. Thank you for EuroBSDCon for letting me babble about this for a while. If you're interested in the code, the main development is done on Heptapod, which is graciously hosted by Octobus. Like I said, there is a GitHub mirror of this data, and you can get it through ports and packages if you're so inclined. Here are a couple ways to get a whole of us on email and Twitter, and now I will throw it open to questions. I have a question. Go ahead. Chuck, at this very moment in time, what is the best way to get, say, two or three attributes at once? Does one rerun it, or is there a comma-separated request on individual attributes or something else? Yeah. Yes. So the eventual goal is for that attribute to be a comma-separated list of page and ID values. That will happen real soon now. So that is one outstanding thing, but that is the eventual goal to make it a little bit more like CFS. Cool. How did I find LibxO? Actually, it was fairly easy to incorporate into the app. So initially, I used straight-up printfs to do all of the outputting, but the documentation for LibxO is good, and it's fairly straightforward to get it to do the right thing. Someone in the chat is re-litigating using, oh, smart control and JQ and all the little things that I'd conclude are still not super user-friendly, given that it took me hours upon hours to just get the syntax right to get a simple parse to get one value. So hopefully this relieves some pain. Yes, and I apologize. I have a fairly good-looking patch queue that I'm trying to push, and I've done something done. So I will work on that after this talk, and that's basically going to consist of all of the decodes. I wrote a man page because I was feeling very guilty about that, and probably something that will happen fairly soon is being able to list more than one attribute on the command line. Chuck, has anyone stepped forward as an individual or organization to say, hey, that looks cool. We could use that, be it large streaming companies or otherwise? As with most open source tools, I don't really know how many people use it. I know I've gotten some specific requests. So one of the features that I got pestered about at a MeetBSD was to add JSON output, and so that's where the libxo came from. And I do know that there's another package in FreeBSD that actually uses smart to collect information about SSDs. Oh, cool. Is that a name you recall off the top of your head? No, but it has SSD in it. Okay, I like that. Have you seen any contributions from the outside world? Yeah, actually, I've had a couple of full requests to fix a couple of things, and some bug reports. Awesome. Yes. Contributions, welcome. If you find this useful, let me know. If you think it's the worst thing you've ever seen, let me know. I'll field the hate mail, you can get the kindness. So chat room, any questions? We're at your disposal. I do see a perhaps it is SSD report, SSD underscore report in fresh ports, which mentioned smart and SSDs. Alan's asking read questions. I'm going to ignore them. So you touched on your proof of concept to Windows. What plumbing would be needed for other operating systems? Should this take off? Really, it comes down to a way to read data from all of the different protocols. And like I touched on, CAN makes that very easy. They have a common interface for sending all three commands. At one point, I looked at what it would take to port this to Linux, which has three different libraries depending on which protocol you're trying to get at. And that seemed a little, that wasn't the thing that I wanted to focus on, but Linux would probably be the next thing to go tackle to see if I couldn't get this running on Linux as well. A few typing indicators. Rhymon, don't be shy. And Chuck, what's the number one thing people can do? Easy, test it, pound on it. Please test it and let me know what I did wrong. Unless, of course, you have a good relationship with SSD vendors and want to get me specs so that I can actually come up with a drive database that won't leave missing values. Indeed, vendors are very shy about sharing, say, those specs on consumer drives and quite a few people run consumer drives, such that if you are in a position to obtain any of that, hopefully properly, please do reach out. Awesome. Great. Well, thank you very much. I think we're out of time, right? Yep. Thank you very much, both of you. And pleasure. Thank you, Chuck, for all your work on this. Is it off to the social event from here? And does one do that on the... There's the closing session starting in about 12 minutes. And the closing session is here or it's in... That, I don't know. It's on the schedule as in here, but they don't know. Well, it's kind of vague. Yeah, okay. Ending you there. And I'm posting a list of space just in case. Which is a cool tool, I admit. Yeah, it's going to be interesting. But yeah, for the SMR drives, disc info can tell you about what the size of the SMR chunks are and so on if the drive doesn't try to hide the fact that it's SMR. I've managed to avoid getting any of the drives that were, you know, submarine into the channel as it were. And so I've not had that misfortune, but you know, they can usually tell you, you know, is it drive managed or host managed and what's the size of the zones are and so on. I guess it was... I was saved by the fact that I was buying Seagate drives and they promised not to ever make an SMR NAS drive. Well, perhaps make it but leave it as such. Right, but like the ones they sell specifically for NASs, because SMR is like the antithesis of NAS. They're like, yeah, NAS drives are meant to be used in RAID and SMR is not. And so we will never put the wrong labels on stuff. Because did you see the Freakus more recently with the SSD vendors changing up the flash without changing the model number and being like, here, this is actually quad level now, not tri-level. And it'll be much worse. But we didn't change the model number. We have Kingston to thank for setting the stage on that years ago when a really nice low-latency drive changed firmware and you had to carefully watch the version number to get the right one. And it was just cool. And I trust the recording will kick us all out before the final session. I think so. I don't know if they did it for the final one, but okay, I'm happy to answer questions until then and reminisce or cry on shoulders about disc output. But the too long didn't read the smart control utility does not simply concatenate some giant text file hiding in the drive that tells you all this information that it's far more complicated than yeah. Otherwise, somebody would have written a replacement a long time ago. Exactly. We wouldn't have had to trick Chuck into doing it. Motive, not arm twist or trick. Inspire, that's the word. Inspire. Anyhow, let's see who's... Good turnout. Hey, all. Yes, I can. A quick answer. Look for Peltor, I believe, but let's see three. You are welcome. And when you're shopping for that hearing protection, note which ones fold into themselves and which ones do not because that's handy when traveling. Should you ever go to another conference in person? Someday, hopefully not too far in the future. And pro tip, earbud earphones underneath those are like studio quality alternatives to big fancy recording studio earmuffs. So that works. Yes, I've done that on an airplane where they provided the big ones with noise canceling that plug into the airplane, but then my little earbuds inside them playing my audiobook to fall asleep. Exactly. I don't know if they're that foldable. There was the decibel, the DD, look for DD. Think that one did it. I could maybe run up and get those during the next social break, especially that's on the other platform. Decibel Defense was another one, but they're pretty rigid, not as comfortable for a few hours of work. But that's another one of those things that's a small investment. So even if you just grab a few or if you have a local industrial supply company, you can go try a few of them and just feel how soft that pad is, see how they collapse and look at the decibel rating. That's from there, it's purely subjective. And also sometimes the weight, if they're too heavy, they can be uncomfortable. That's true. But if they're too light, they might break too easy. Yep. So confession, 90% of my podcast listening is on a lawn tractor mowing the lawn with the earbuds and the massive flight crew hearing protection. So all of my associations with like 2.5 admins and BSD now are somehow visually related to lawn.