 Time here for more in systems and we're going to talk about hard drive benchmarking. But I thought it'd be kind of fun to start with one I took apart here and play around with it. I've actually got some platters out from another one and a little laptop drive here. And of course the salad state drive. And what we're going to talk about today is benchmark. Now benchmarking in Linux is not too difficult to do if you have the right tools to do it. And we're going to talk about a few of those tools that are available. And it's important because one of the things you want is when you're setting up a storage server or even just setting up a singular hard drive, whether that be solid state or the still relevant in August of 2021 spinning rust rise, because well, the capacity of these is still much higher than most of these in terms of your dollars per gigabyte cost. So you know, benchmarking and tuning to make sure that you have the best performance and the best setup is still really an important factor when you're building out these systems to make sure they'll behave in the way they want. Now I've said before, there are lies, damn lies and there are benchmarks because obviously there's a play on the lies, damn lies statistics phrase that's popular because you can lie with a benchmark and sometimes marketing people do. And what I mean by that is simply if you don't understand how the came to the conclusion of a benchmark or the tools that were used in there, you just see some high number that was plucked by the marketing people to stick it on the box of the product that you're looking to buy. My goal today is to educate people how to run the benchmarks yourself so you can instead of having kind of a subjective like this feels faster to a more objective of these are the parameters. And as you tune these parameters, it will give you the exact results and then you can better understand how you've designed your storage servers, how you've set up your hypervisors and be able to really fine tune these things. Warning, this is a little of a rabbit hole. I will cover how to use the tools, but for any of the tools I talk about, you can certainly and I will be leaving links down below to the write ups on these be able to dive much more in depth on these and start really playing with a lot of parameters and noticing how there's a lot of little tuning things that can go, especially as you play this out with larger scale systems. But that's part of the fun of doing this and learning it before we dive into this video. If you'd like to learn more about me and my company head over to Lawrence Systems dot com. If you'd like to hire share a project, there's a hires button right at the top, which does include hiring us for storage consulting. If you just like to support this channel other ways, there's affiliate links down below to get you deals and discounts on products and services we talk about on this channel. Now everything I'll be talking about to be leaving a link down to below and starting with this quick primer to get you started with doing some Linux storage benchmarking. Here is the FIO man page, which offers an amazing array of parameters that you can really help fine tune your testing. But to keep this video in scope and shorter, we aren't going to cover all of them, but I will leave a link because I encourage people to RTFM much as I do to read all of these things. We will though talk a little bit about this as we run some of the tests. And this is the output. So if you want to know or have run this tool, but just not sure what they are, this is kind of a primer right here for this. We'll talk about it in brief, because once again, it gives you a ton of little details in here. Also linked in here will be the package for IO top and IO ping, which will be running as well. Those are just to kind of watch what's going on while you're running these benchmarking. It's just kind of an insight tool and can actually be a helpful debugging tool when you're troubleshooting storage. Now before running these tests, what will we be measuring specifically? We're going to focus on input outputs per second and your throughput. So your IOPS and throughput, what about blocks and file sizes? And this is all related. This is one thing about storage. There's a lot of parameters to come up with what these benchmarks are telling us. The goal in benchmarking is to really see if the storage system has been optimized to suit your intended use case. For example, a system optimized for writing, reading lots of small files, lots of documents and logs, for example, will benefit from a smaller block size, but writing and reading large files, videos and large backups will benefit from a larger block size. Another example would be a database server, which may have a large database size, but due to the way the transactions are committed in a database, it may benefit from a smaller block size. I'm bringing this up because this is one of those parameters. Once you've determined how much storage you need, as in you've got the volume of storage, then you have to fine tune different block size parameters. And this comes in the way if you're using something like ZFS where you can tune this on a per data set. But why would you tune it one way or the other comes back to the application and use case? Why does that matter when you're benchmarking? Because right here, when we talk about this benchmark and let's just jump right into this random IOPS one that we'll do here, it matters because we set the block size and the file size. This way we're simulating, let's say we have a database that's four gig and writing a bunch of small 4k commits, how fast can it do that? And this gives you those parameters you need to start calculating that piece of information. Quick prerequisites before running the test, besides the obvious of having these packages loaded in your Linux distribution, is making sure you have enough free space. I have definitely not looked a couple times when I was doing these tests and realized oops, I filled the drive up. So make sure you have enough free space because you'll end up with some really weird test results and crashing a Linux system. Check and pause other workloads that may interfere with the results. In the case of the way we're testing this, this is running inside of a hypervisor because I wanted to show a couple more expanded parameter tests. But that is something to take into consideration, especially when you have inconsistent results. Make sure nothing else is running. Sounds obvious, but it matters quite a bit to make sure you're testing things properly. Understand your workload and what your intended use for the storage is. As I said, you're testing and you want to optimize your benchmarks so you know that it will perform with the applications you plan to use on here. And tune anything you might want to tune to such as the block or file test size. And that's what we'll be doing in here. So go ahead and just copy this one and switch over to this. Now before I run the command, I want to use iotop right here to show the disk reads and writes top is the application monitor iotop very much like an application monitoring tool is specifically to watch disc read and disc write per application great debugging tool and fun because we can demonstrate what's going on here. Now this is the parameters that we have here. So we have sync, FIO, ran repeat, IO engine, direct name, test file, block size, and file size right here. Now this is the read write test, but this is singular. And we're just going to see what happens when we run it in a singular manner like this. So not real impressive, a whole whopping eight megs a second for this particular test. But this obviously depends on if that application it's telling you that a single application that has a four gig file with a bunch of 4k small rights is going to operate at this speed. But generally speaking, there's usually many applications that will do things like this num jobs equals, we'll put a parameter 20 in there, this will take this particular command and run it simultaneously times 20. So it's the same four gig file, it's same the 4k rights, but now we're going to have 20 in here. So starting 20 processes. And this is why it's fun to see iotop because now you can see that up at the top here, there's 20 processes running. And we also have a substantially faster disk speed showing. Now this is where things can get a little bit confusing, because you're thinking, well, the disk speed still seems relatively slow, but it's actually being loaded up quite a bit by all these tiny little rights. And this is how it's performing. But this is not indicative of the maximum performance we can get out of this particular drive. Let me explain. So we're going to go ahead and cancel this. So this was your random right test here. But what does a sequential right test look like? What about when we want to write sequentially a small amount of block size of four mag and do the read, write, write and ramp up time for so it's almost the same one here. But what kind of speed can it get from sequential? So let's run this command here. And you'll actually see the drives are operating closer to the 400 bouncing back and forth when it clears the buffers. So we're seeing 400 and 500 commits on there. What about when we increase the number of jobs equals, we'll say five. So we're, you know, writing five large files to this. Well, now we're all the way up at 700 600. So you can see there's a huge difference in speed on there, but this is now measuring a large streaming performance of this particular drive, not measuring the small rights on here. This is why benchmarking can be so confusing to do because is the drive fast? Is the drive slow? Isn't really the right question. Can it work with the workload that you want? If you are serving up video files, for example, to a bunch of video editors, this matters a whole lot. Can those editors write these files simultaneously? How many of them can write them simultaneously? Well, if we had five video editors or maybe even bring it up to 10 video editors. So if we have 10 video editors, what is the thorough put that they are going to see writing out their large files that are many gigs, and we can keep tuning this till we kind of get the workload idea that we're doing. So we can go back up arrow and say block size for mag size for G or maybe the size is going to be eight gig files or so on and so far. So you can keep tuning it. What happens when it's at eight gig? How does this file system perform? And you can start narrowing down each of these and understanding these different parameters. Now, there's not a one size fits all. Of course, it's not like we are locked into only doing one thing, but there are some tradeoffs when you adjust these type of tuning parameters, when you're setting these things up, of whether or not you optimize for the smaller or the larger block sizes and how it may perform. But having these tools and understanding how FIO can kind of give you some understanding on that really is helpful. Now, going back and forth, there are read and write options in here. So we do have like mixed random workloads and sequential read tests for thorough put could be a little bit more confusing when you have things like caching going on, because they can even optimize it. But this is also a way to dive into tests of whether those caching optimizations are working and same parameters, but this happens to be a read one on here. Now, what about automating all of this? A couple different options you have there. You can actually do a lot of scripting and dive into ways you can automate things around FIO, which is pretty cool. But if you want to go full scripted on this and dive even deeper into it, that's where I don't know if I'll do a video on this. But what I do when I do my testing is the pharaonics test suite where you can build a series of automations. And by the way, the pharaonics test suite also has FIO in there. And this is like a results file of what it looks like. Well, so I'm doing a little too far. And then you can start playing with all the parameters. And actually, this is for an upcoming video I have on NFS on true NAS versus ice because you have covered it before I'm covering a game with true NAS 12 and an upcoming video that depending on when you're watching this may or may not have been released, but this will give you that better idea of when those parameters matter for each side. As a matter of fact, when you're comparing two things, having an automated system like pharaonics is great because these tests took a long time to run. And while it may look like there's some winners in some categories, you can start to see how some parameters were favoring the NFS and somewhere favoring now, you can build out the parameters, build out some automation to it. So you don't actually sit there for hours waiting for this to get done. So you can make a video on it. So this would be the more advanced side if you want to go a little further, but getting the basics with FIO pretty simple right here just to try any of these. As a matter of fact, we can go and look at some of these random workloads or set up this sequential right here. So actually copy this one. Copy run this test again. But then we'll say jobs equals 12. There we go. 12 four gig ones. See what kind of performance that may give us quite good. This is where the I already know because this is running at the true NAS side is probably cashing some of this on the back end and giving us a relatively high throughput on there. But before you say, well, cash is cheating and things like that, it's actually not. And I have that people I get that people want to say that like you should be able to exhaust the cash each time. But also you may not want to exhaust the cash because the ideal situation is to have the cash and the cash hits be there for repeated requests for large files. So you do get a little bit of fuzziness in the parameters because depending on what else the cash has in there. And in this case, because the system is dedicated just for test benchmarks, it's not running any production environments, it was able to really peek out a lot of data here with a reasonably high IOPS. But you know, that's the nature of it when it's able to serve that up. Obviously at some point you exhaust the cash. But in order to do that in here, we could just make the test much bigger. So if we change this to like a 16 gig file, 16 gig file, keep the block size there. And let's see if that's enough to exhaust the cash and actually not hit those same numbers anymore. Right. So it took a little bit to write all that file out, let's switch over to here. And we can see right here, how we did the throughput for the writing wasn't incredibly fast, it kind of scoped off a little bit here and sloped a little bit down. But the reading and this is undoubtedly if I log into the TrueNAS next. And as I suspected, you can see the cash going up, the hit ratio being low for when we wrote the file. But as soon as that file was written to the drive, it's able to start sticking in memory to meet the demand and requests over here and keep us at a really high output coming back from it. So it's probably not touching the drive so much because it's able to pull the repeated requests for memory out and give us a really high IOPS on the read on there. So hopefully this serves as a good starting point. So where you can get started in running and getting baselines on the drives and storage devices in your Linux system so you can start tuning those parameters, especially when you get up to the hypervisor level, because you have network attached storage in this case, who running things over network tech storage means changing not just storage matters, changing network settings in between can also affect how much thorough puts you get or the light latency that you may achieve and IOPS you may achieve on there. But storage tuning is definitely an art. It is a lot of parameters and a lot of different options. So there's a lot you can dig into. And if you want to go further and more than just baselining a one single system, you want to do those more options that are into building automation, do check out Pharaonix. It's a test suite I have used before. And of course, many people use if you've spent any time on the Pharaonix site or really almost any benchmarking site, they talk about high performing servers, you'll frequently find Pharaonix and it does a lot more than just disk benching, but it does kind of build some automation around FiO. So you can have a whole lot of parameters and not have to write a whole lot of batch scripts to get all those output. And of course, it uploads them to the open benchmark site. I'll leave a link to that if you're not familiar with it. But I think a lot of people are going to be Oh, yeah, they've seen that in any of the articles over on the Pharaonix site where they've wrote that and it's an open source program as well. All right, leave links to everything I talked about in the video down below. And let me know what you want to know about storage in servers or questions I can answer if this needs a follow up video for certain questions. But boy, this can be a real rabbit hole to dive into. I at least want to give people the tools to get started and places to go read more. And of course, a forum is where you can reach me if you want to discuss this further or things that I may have missed or some of your favorite storage tools. Go ahead and reach out to me in the forums. It's a great place to have that discussion. Thanks. And thank you for making it to the end of this video. If you enjoyed this content, please give it a thumbs up. If you'd like to see more content from this channel, hit the subscribe button and the bell icon. To hire a share project, head over to LawrenceSystems.com and click on the hires button right at the top. To help this channel out in other ways, there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts and offers, check out our affiliate links and descriptions of all of our videos, including a link to our shirt store where we have a wide variety of shirts and new designs come out well randomly. So check back frequently. And finally, our forums. Forums.LawrenceSystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel. Thank you again, and we look forward to hearing from you. In the meantime, check out some of our other videos.