 Now, normally I say Tom here from Warren Systems, but today I'm going to say Tom here from Warren Systems and Tom here from CNWR. I am in the CNWR Toledo Lab. We still have CNWR Southgate. That office still exists. Today I'm setting up a few things and doing a little bit of testing for some customers and we're going to talk about some of the servers and give you a little behind the scenes look. And hopefully no one's configuring this right now, but we have some 100 gig connections that we have set up on these devices. These are actually 40 gig adapters because the 100 ones haven't come in yet. It does have 100 gig adapters. We're going to talk about the project we're doing here for a client and give you some ideas of what goes on when we're not just making videos about some of the homelab stuff. This is the enterprise side of the company where, well, I set all this stuff and put into practice all the different things that I talk about on my channel, which leads to feedback that some of you might be interested in, especially when we take a look at all the NVIDIA cards in there. And I'm actually kind of surprised it's not too loud at idle, but this thing once it gets warmed up is really loud. So let's dive into it. Are you an individual or forward thinking company looking for expert assistance with network engineering, storage or virtualization projects? Perhaps you're an internal IT team seeking help to proactively manage, monitor or secure your systems. We offer comprehensive consulting services tailored to meet your specific project needs. Whether you require fully managed or co-managed IT services, our experienced team is ready to step in and help. We specialize in supporting businesses that need IT administration or IT team seeking an extra layer of support to enhance their operations. To learn more about any of our services, head over to our website and fill out the hire us form at LawrenceSystems.com. Let us start crafting the perfect IT solution for you. If you want to show some extra love for our channel, check out our swag store and affiliate links down below that will lead you to discounts and deals for products and services we've discussed on this channel. With the ad read out of the way, let's get you back to the content that you really came here for. While I can't tell you the name of the client, I can't tell you the use case they have for this setup. This is for their engineering team that collects and analyzes data out in the field. They have specialized software to use of the GPUs to form in-depth analysis. This system is running Proxmox per the client's request as they are already familiar with it and we have it configured to pass through each of the GPUs to their custom software that runs for each of the VMs. The field data they are collecting is all stored on a 45 drives XL 60. And this is actually the second one they bought from us. As with many of the projects, the client did not tell us what servers to buy for this project. They let us know their budget, their needs in terms of CPU, GPU, networking and storage. And then we came up with the Super Micro as being a fit for this particular setup as there's not a lot of servers that can support that many GPUs right out of the box. So this is a direct order from Super Micro with all those GPUs in it. This is not like piecemeal together. Now let's talk about those server specs. This is a Super Micro AS 4125GS-TN RT. You'll find a link in the description below to that because they have a page for it where you can get all those numbers and more details. It has one and a half terabytes of memory, a pair of AMD Epic 9254, 24 core, 48 thread CPUs, giving us 96 threads in total, eight NVIDIA RTX A6000s, only two micron 1.92 terabyte drives, two 10 gig on board. And then we added a Kinect X6 100 gig adapter to this. There's also four 2000 watt titanium level power supplies and hotspot fans. Now the 45 drives XL60, I've reviewed those before. You'll also find a link to that down below. We have this running true NAS core. Once again, it's because this is what the client is familiar with. It has 512 gigs of memory, Intel Xeon Gold 6230, 26 core, 56 thread, 20 terabyte Seagate XL drives, 60 of them, giving us 1.2 petabytes of raw storage. Now, of course, the reason these servers are at our lab and not at the clients is because they should be tested prior to delivery. And that's where I'm really happy we did our testing because I learned about resizable base address registers or resizable bar for short. This is the PCI capability that allows your device, such as a discrete graphics card, to negotiate the bar size to optimize system resources, enabling this functionality can result in performance improvement or in the case of the super micro, as we learned, it can force you to reset the BIOS and you can't get into the BIOS to reset it. So you have to just pop out the battery. That sounds simple enough until you realize the battery is under the entire backplane under all of those cards, which then resulted in a new project called we're going to take apart this assembled server. This was interesting and the people at super micro were amazingly helpful in letting us know exactly what BIOS setting not to change again because it does result in this machine having to have that battery removed. But hey, me and Jason had some fun taking this apart because we learned a lot more about how it works and get to show you a little bit more in depth of what this server looks like taken apart. Now, the testing of the 45 drive server went perfectly smooth. We test these anytime we order them. We ordered quite a few of them in 2023 and this one, well, it went smooth as well. I'm not saying we've never had a bad drive. I'm as happy when we find that drive prior to it getting delivered to the client, but 60 drives out of the box, 60 drives loaded up with data. I use FIO, which is built into TrueNAS from the command line to load up some test data, do some performance testing on the drives, put them under load for hours at a time, see if there's any errors resulting in none means, well, it was ready to ship and move on. But then we wanted to really test the Nvidia cards back in that Super Micro server. I figured the best way, once we figured out that little bios problem and took it apart and put it back together and we want to make sure everything worked, was to pass them all through and do some testing with Hash Cat, which the first thing we learned is we don't have 4,000 watts available on one circuit inside of our lab. So we had to plug in two power supplies on one side of the lab, two power supplies on the other. And then me and Jason dove into testing with Hash Cat and the results were, well, pretty amazing. It is, again, just a pretty basic install, because I think the customer will probably end up blowing this away, but we wanted to install it and deliver it to them with basically a technology preview example of what we intend or they intend to do with it. The customer's use case is actually to run, I think, eight VMs or eight VMs on them, each with a single GPU passed to it. Yep. I couldn't be bothered to do that and I couldn't be bothered to mess with the windows. So I did a single VM and I ran Cali through it. Passing through the GPU is relatively simple. You have to go into the Proxmox shell and basically blacklist the kernel drivers for Nuvo and NVIDIA. And once that's done, the PCI devices are available inside of the actual VM itself. You can see that I have all these PCI devices passed through. And if we go to like this one and click edit, you can actually see in the list here, if I hit the drop down that it's an NVIDIA RTX A6000, right? Like I passed the base one through and I pass all of the functions of it through. I'm not trying to do any sort of map device, although Proxmox does support doing basically VGPU mode. I'm not doing it in this case. When I passed all eight of these devices through to the Cali guest, then once you're in the Cali guest, there's some configuration you have to do the same thing. You have to basically blacklist the Nuvo driver so it's not trying to use it for X. Once that's done, you install the NVIDIA drivers in the CUDA toolkit and that's when the real fun starts. So a couple of things of note here. This server has a bad habit of blowing circuit breakers. It draws it blew a that in the 45 drives box together that Tom is also covering blew the circuit, a 20 amp circuit breaker yesterday when I started hash cat in full mode. So I had to move half of its power. It's got four power supplies. I had to move half of its power to another outlet. It was bad. It's this thing. This thing draws so much power. I can't believe it. I believe it's over 3000 watts when it's fully running. Yeah, it's just insane. It's a lot of wattage, but all that wattage is fun when we want to run a hash cat. It is. You want to run some stuff. Benchmarks. I don't know what one in particular here you were looking to run. I can run, I guess, hash cat. That's B-O, which does just the basic benchmarks of all of the all of the different types of hashes we can run here, right? So we're going to run some MD5. We can do MD5 in the giga hash range, giga hash of second range. It is fast. I we actually had a hash of Cisco hash we were attempting to crack. When I got in this morning, we were it had run for about 18 total hours and we were 50% through the eight character brute force space, right? So like by the end of that, by the midday tomorrow, we would have brute force every possible eight character password combination. So in less than in about probably two days total, we could get through with Salted MD5 the entire eight character password space, which is insane to me. I mean, look at how fast it still is. Even at shot 12, we're still rocking out there. It's a pretty good speed. I mean, you're at that speed, you're getting through the entirety of rock you in a matter of second. It's not slow. It is very, very, very fast. I can show you the hash I was actually cracking before I'll just run through it with rock you and see if and see how long it takes to get through rock you with that hash type. There you go. It's running. The funny thing is it actually takes longer to initialize the devices and compile the kernels than it takes to run some of these these attacks essentially, right? So it's basically compiling and building the engine to do bcrypt. So that's going to take a bit to load in the the actual code into the device that can do bcrypt. It's on device five. So it'll take a second here. We'll see if I can start to hear it is I Can you hear it from your office? Oh, yes, I could hear it from my office when I got in this morning and that room was hot. I could probably almost hear it from Detroit. Yeah, there it goes. It's running. Now, this one's not. I don't expect this one to be as fast, right? So one of the things is, is again, we don't paralyze very well here, right? Because we're doing that. So we're on status here. So we're doing, I mean, this is killer hashes a second. This is actually pretty slow. So we're only, I mean, we're 39 percent through the word list now. But the word list, what the rock you is what 14 million passwords, I think. So we won't get to it. So we're at the very end of it. We're 84 percent through. So this thing estimates it'll complete in five seconds. So it'll, there we go. And we got it. So, yeah, in a matter of what it, that was a minute and 24 seconds. If you look at the start and end time, we went through all of Rocky with BeCrypt. So this machine is very, very fast. How many passwords is Rocky? 14 million, 14 million. Yeah, 14 million. Yeah, 14 million passwords in a minute and four seconds. And I think like a solid 45 of that was compiling the kernel to be able to run BeCrypt inside of the actual CUDA device. If you run it again, does it recompile or is it already got it? No, we can run it. Yeah, it goes immediately because it's already got it compiled. It caches those. So if you actually look here, I was looking at this earlier. So, yeah, it's already got it compiled. So it's got an optimized kernel for, you know, type. What was that type? 2600, I think it was. It definitely has the kernels optimized and pure for the various device types, and it works really well. Yeah, that's an impressive system for sure. It is. Now, once we were done, we boxed up a super micro, removed all the drives from the 45 drives because, well, 60 drives add a lot of weight to the chassis, and it's not pleasant trying to move it. Plus, we had to install it in a rack, but 45 drives is a tool of system. So just sliding all the drives back in had us up and running in no time. I will do more project videos like this in the future based on some of the feedback I receive. I don't do too many of them because my goal is always to create tutorials and learning, but this is also the other side of what I do, and I wanted to kind of show that for people who are curious or people who say, well, TrueNAS isn't used in enterprise companies, is it? And the answer is absolutely yes. Not just from the consulting we do, but I wanted to kind of showcase a project here where this is a very large company and they are using TrueNAS, they are using Proxbox, which I don't see as often in enterprise companies, but this is what they are familiar with, and ultimately that is the goals to give the client something that is very familiar and usable to them as a solution and a package. And they've already tested this in their proof of concept they have with several other machines before contracting us to build out this big setup that we have in there, which now they have over two petabytes of storage and that fancy super micro system and a lot of data to crunch. It's just really cool seeing it all come together. Love hearing from you, leave your thoughts and comments down below. Let me know what you loved, what you didn't love or if you like videos like this. I'm always curious and listening to feedback. Like and subscribe. You want to see more content from this channel. Sign up for my newsletter if you want to keep up with some of the latest things that we're talking about from technology and some of the videos I got going between this channel and my business technicalities channel. All right and thanks.