 Only three minutes late We put this on mute, you know you caught up listening to something and you're just like you can't stop well I Was working on things and listening things and Get deeply interested so it takes me a couple extra minutes to get started going. Oh, yeah, and then I'm like yeah Let's do this But we're gonna talk about the ace magic first get that out of the way Cuz I know that's probably why some people post have you know post this live stream being live I've clicked on the link so I'm gonna be talking about that first. I'll go into a few other topics And we'll read a couple of comments on the main PC's may PC have been criticized for having suspicious software low like a version of Chrome that doesn't respond to setting changes. I don't know if that's sure not um You know, that's a little bit confusing cuz I believe that the settings will change it Just chrome will pick up its old settings if you reinstall it you just got to go through and clear it But I wouldn't trust any of them We'll just start that at the beginning to just use them out of the box with whatever random software or shipped on them I don't really do that. That's um, but definitely one of the concerns I would have I have Ace PC peek-a-box me PC Proxima's cluster up the first one started in Windows. I switched to the SATA drives To blank the ones. Yeah, I mean if you're just wiping them and that's the first thing I do anyways I don't spend a lot of time It just comes with really any of these PCs outside. Well, even OEMs When you buy them from the large companies, we generally rewrite them right away I don't trust whatever garbage they may or may not have put on them and so we don't yeah I just don't bother with that All right, here we go I'm gonna go share the screen about the ace magic. So let's get that started right now share screen Cuz I've actually been using this one for a little while and Let me give you a little background here So ace magic reached out to me and says let's send you a mini PC We said okay, and I reviewed one. They've now sent me a couple of them They sent me the one I did the review on like a dedicated video another model then this one was sent to me as well I Said yes a few times because why not but I there's nothing that I find really significant about them I'm also not the most expert person at doing hardware reviews. Sometimes I feel These mini PCs are all kind of the same especially because with ace magic They have more than one name and I brought this up before it makes it really hard to figure out Because they also go by the name We'll share the other tab here Cam Rui, they are the same ones without different It's Cam Rui or is it ace magic the specs are the same the box is the same the logo is different The website's not great. I hate things that constantly want to chat with me But hey, that's apparently how their website works, but let's go back to the actual ace magic one The weird thing is they brought it up again to me like hey You haven't really reviewed this one that we sent you and you know I brought it up in a couple of licensees before that I was using it this one I'm gonna call my finalized review of it because I've been actually using it for quite a while running XC PNG It works great actually Functionally wise. I've had no problems at all with it. This is the system when it's currently running. We pull it up. It's this Running XC PNG. This is the host that has been running. It's got 32 gigs ram 16 cores We'll get a little bit more detailed in the specs here, but it's been running for a few months I just restarted it because I wanted to take a few photos of it But let's run down the specs list and kind of cover it with see Ryzen 7 7 7 3 5 each This is one of the reasons that I wanted to review it was I like some of the Ryzen mini PCs because they're very low power It has the built-in and integrated AMD rate rate on it 32 gigs of RAM 512 gig SSD. I actually think this one. Let's go look at the storage on it real quick here I actually believe This one, okay It only has 512 and I couldn't remember one of them had a said 512 but actually shipped with something bigger in it Which I was like, hey cool That's nice that they sent me a bigger one. This one did ship with a 512, which is fine little SS a little NVMe in it Solid state go down here. This is one of the weird features I don't know why it has this but it does they equipped it with wireless charging So I can set my phone on top of this and charge it. That's weird, but we'll go with that Dual two and a half gig internet dual HDMI out. So yeah, we got a couple USBs USB-C power and That I was so confused when I seen this because I'm like, why is it got this little symbol on it? Let's share this tab instead. Sorry is wrong tab But I was like, why does it have this little symbol on it? What does that symbol mean? It turns out that's wireless charging. I was like, oh, I can set and I you can you can set a device on it It's got chi charging built-in Novel I don't know the real reason for that But it has it I Think of it more of just a convenient mini piece. I guess it could be a small workstation. I Don't know. It's not bad though. It runs XC PNG perfectly fine and the two and a half gigs work great on it So that parts solid there. They like I like that they show the chi charging on it right now But one of the problems is in it says it right here Coming soon for this model. I've had it for a while. It says coming soon, but it also says sold out They worse for sell for a while then they're gone. I can't even find them on Amazon So I can't even offer you an Amazon link for it, which is also weird for them to be Bugging me about a review when it's essentially not easy to obtain this particular model the Other things about it though is I will say it is quite fast is quite low power. So those things are you know Those match up in terms of that actually I forgot to Get the power specs I guess I could really quickly go plug in The device but it's do they have the wattage on it? I'll pull the wattage of what you get actually at the plug So I can go do that real quick. I Just got to run over to the other side and plug the kilowatt into it to give you an idea But you know, it's not a bad little box So the complaint I'm gonna have about it Let's go over some pictures before I plug in for the kilowatt on it to show you the actual wattage So I forgot to pull that up But in case you're wondering from the photos here I if you're wondering did Tom shoot this on his pool table. He sure did this was absolutely Take photos were taken on my pool table because it was convenient We'll take a look inside of it real quick there. It does have a nice little the heat transfer pad on the SSD You get the Wi-Fi inside, but if you notice you have spots here, so you can plug in a couple more and I think these actually are these MVME does it? See if that's in the specs there. What speed are they? Okay, yeah, they're all three Proders further site. They're all three MVMEs. I want to make sure they were full MVME speed But the boxes worked really well running Linux. I've stressed tested it I've on you know all my little tools I've been running on it and it's actually worked quite well I haven't had any issues with it. One thing I'll say is the piano glow black piano black gloss I guess you call I've heard it referred to that a lot. It's really glossy black Everything shines on it. And if you notice from the even the first photo There's a fingerprint and here's from a different light angle. Look how that fingerprint on there I had to wipe this off so many times just to make it look like this the It is a fingerprint magnet for sure, but they're not bad little boxes. I mean, that's my thoughts on now I think these are still reasonable deals for a lot of homelab and I think they're probably for the power you get out of them in terms of how little wattage They use which will get to the exact wattage in a moment versus how much They cost versus how small and quiet they are I think these are gonna start offsetting some of these larger servers as being popular in the homelab I know they don't have the features You're gonna get or the storage you're gonna get with like an old Dell R630 or something like that But on the other side, you're not dealing with something that's large heavy pulling lots of wattage pulling lots of heat and Bulky these you can build a small cluster of them for pretty reasonable now I'm gonna cover the one big downside I feel about all of these mini PCs that aren't passively cooled and that is gonna come down to the fans when I reviewed the other one I have let me pull that video up real quick, but one of the other ones I reviewed I kind of I harped a lot on that fan because I just don't feel That the fan and that was the Go in model here when I reviewed the go in. Yeah, that that's this little box here I've got a whole review I did and the mini fans on there are adequate to keep up with it almost when you're using a 25 gig I was able to overheat the 25 gig on this one this other device only has two and a half gig and Obviously if you just use a two and a half gig this thing never overheats But my problem is what it happens in three or four years if you get a little dust in these fans How serviceable are they because it it would be a nightmare to take any of these including the ace magic one apart Because it just doesn't come apart easy. That's the one big downside. I see that it didn't really think about service ability I think most of their goal is like how cheap can we make this box and send it out to people and you know They have a few other other rise in many PCs on their site and but they're all designed the same way They're not easily serviceable They they maybe want you to make it easy to pop a drive into them, but that's about it Outside of the drive. You're like, yeah, if the van goes bad It's it's a kind of weird little custom fan that you might have to get for it Me throw go through and answer some questions that people may have about it Real tech nicks. Ah the nicks in this I believe are Intel two and a half gigs. Let me double-check that. That's a good point Me SSH into it and we'll get some of the hardware specs pulled up There but I'm 99% sure the nicks in this device are I think they're Intel based. Let's figure that out. I thought that was in their notes, but it's not One of the other ones I reviewed had the Intel next in there And I was like you that was one of my problems with it I can't remember what it's another if you look at silly other ace magic. I reviewed I don't plan to keep doing too. They want to send me a lot of these As you get bigger as a youtuber, they just like hey, let's start sending you all these and you say yes to a few But then you realize you don't want to say yes to all of them At some point. I'm like, I don't really need all of these things not my Expertise or actually not my desire. Oh Womp-womp they are real tech network controllers. Oh wait. No, I'm wrong ethernet controller device i2 5 so they have Intel ethernets in there. Well, let's go ahead and I gotta share out my tab here So this is actually interesting though To onboard nicks real tech is for the Wi-Fi, but we have the onboard nicks are Intel So we have two two and a half gig Intel's on there. So that's definitely a big plus in my opinion Yeah, three MVM me and a mini PC is not bad 3m twos I I agree completely and you know Wendell me and Wendell have both talked about this and I Wendell from level one techs Building little clusters out of these mini PCs. They're low wattage You're making up for the potential pitfalls of these things maybe not being long-term reliable by building them into some type of Redundancy and running things on there And I think that's not a bad idea and then you can put some redundancy on your storage by having You know, you could set this up in a pair with two MVM ease or put three of them in there And maybe even you know set up a raid between them It's not gonna have any hardware raid But you can set software rate up have some redundancies of your data on there have one boot drive and a pair of other Drives that you set up in a raid mirror. I Think it's a pretty pretty neat little device build three nodes to CPUs those Amazon CPUs He chose has 72 cores five two gigs RAM Yeah, you could do quite a bit on your homelab stuff The only thing is a lack of after sale support by ace magic people says anything about hardware. Oh, yeah I don't know how good the after sale supports gonna be at all. I I almost would go in with the assumption of no I Don't outside if you buy these two Amazon I feel you have better protections and they've sent me links before and I didn't post them They wanted me to use their link to send it to their site Personally, I buy these from Amazon before I buy these from their site That's my opinion and you know, that's one of these things about these reviews. I They can send me the box. I'm gonna review it the way I want to review it I mean, I have to disclose because I'm very honest about all my reviews that you know Someone sends me something but this is actually why I prefer to buy things myself. So I don't even have to disclose that I could just Say whatever I want whether I like it or not like it. I can completely just say this is my opinion There's no implied bias. I mean, I do like when they company sends me stuff that can be nice It's convenient. I'm gonna buy it. So I'm not likely this is not a device I'd likely to buy myself. I have a lot of other I have a lot of servers that's part of my problem and I end up Sometimes I'm done with these giving them away to employees and things like that to or they just end up in the lab somewhere else. I Started playing mini forums at my work for generic PCs register salesman paint They're cheap plenty powerful for what we do haven't had one fail yet in two years I've brought this up before Because co-tom has these boxes here. I got it. I've just had this one on my desk as an example one for a little while This is the actual co-tom one. The only fair failures We've seen is some of the nicks just die That's happened to a few of them. The other things we've seen is they use the cheapest possible MVMEs so if you have something that ends up with a heavy amount of read write sometimes they have gone bad but not often but More than more than expected, but you can pop in a normal MVME and solve that as well. So that's kind of yeah I have a couple to tell you that I used to like CP and G boxes The box you showing looks nicer though. Oh, yeah, the the these definitely run really nice for that 3m twos 6 8 core perfect little unraid box. No boot dis required. Yeah Mostly sketchy stuff in Amazon as well as mostly break first week or never. Yeah It's like they work the ones we've gotten Outside of dry failures. I don't think well. No, I take that back I think we have one at the I stay have one at the CNWR office. It just mystery died not long I think it lasted just a month or two and just mystery died Maybe it was a little bit longer. I don't remember though. It was just long enough that they couldn't send it back That's why it's sitting on a shelf but it's I don't know it does it's What you're saving and the for the number that we bought it's not that many But it's still higher than if we're looking at like Lenovo boxes and things like that I see a lot of mini PCs and doctors off stash profess them out to the monitor. Now. That's another thing one of the Little things I like is is it Lenovo that makes some some of the Lenovo mini PCs and Dell Dell makes some too Though those are actually pretty nice and you can get those on the secondary market pretty Reasonably priced we actually got a hell of a deal and oddly they were new They were on clearance when we were buying some their sale was like half off We bought them for a completely for a client project We bought a bunch of the HP really tiny mini ones and I was like, oh, this is this is really cool You know, I can buy some of these for the clients and the price was killer They were better than the Dells at the time and I haven't had any problems at all The clients we deployed those two and I mean I come to the model number I don't let me look them up on Amazon Because we've ordered a bunch before they are the Lenovo think centers Stop sharing so I can switch Yeah, these little mini PCs and They're solid You can find and these are new prices for them. You can find used ones for pretty cheap These little boxes are quite reliable though And now you don't have to worry about you actually will get warranty and support on these models These I think my friends over at hardware Haven have covered and done some reviews on these on that YouTube channel Yeah, these are another option for some of the mini PCs Does it make sense to skip two and a half gig and go to ten gig sec instead? Yes, but and my problem and the reason I say but I definitely think yes The challenge you run into is when it comes to these mini PCs. It's finding ones that have The ten gig on them. I agree two and a half gig is a weird Middle standard like just give me all the way to ten But finding little PCs with tens on them has become kind of more challenging They always seem to stick the two and a half gigs in there. Oh, let's see On all my customers, I hate I have supplied them HP or Dell under the desk monitor on the back them out to point smart deploy clean easy So, oh, yeah, these those are really Really solid Yeah, finding a ten gig ones is a little challenging now the other side of it though The other argument I could make is one gig works for the majority of things If you look at what I'm running on there like right now. I I don't have much I have a Hold on me switch to that particular pool That's right. I got chasm on here for example. I Chasm can be connected at one gig. It's gonna work perfectly fine It's not like I need to have this connected at ten gig for the services. I'm running I have a you know, some random things I run in Docker All the little connections matter of fact if you look at my analogy and I look at all the different things I have running in here like my fresh RSS and everything else All of that's connected at one gig even my media server is only connected at one gig I don't need ten gig to my Media server ten gig is great if you need like a hundred TVs streaming 4k stuff, but reality is I usually never have more than one or two TVs streaming and You can have more than that on a one gig link. So one gig even works perfectly fine the only thing I need ten gig for the only thing in my network is First my video editing ten gig is a necessity or video editing would be painful because I'm shooting everything in 4k So the video files are huge. So I do need that I need ten gig for my hypervisor connected to the storage So all the different machines boot fast enough But even that's kind of limited because it kind of depends on the virtual machines that you have on there Yeah hardware haven since some really good videos on that whole topic What about 10g? We need 10g networking How you doing bearded it dad Yeah, NFS for VM disc that you probably want ten gig for Why not 40 gig we have we have some hundred gig stuff in the office I'm gonna talk about in a minute here. How many customers insist on 10g 40g or whatever I mean look at the metrics we use at 1.1 gig during peak hours Yeah, we had a customer that really well I mean they're getting it because they have the budget for it They're we're putting in a hundred gig infrastructure for them They just wanted a future proof it and they have the budget for it So we're putting it in the reality one things we told them is you're not gonna saturate your hundred gig because you know They're they're getting some of the 45 drives XL 60s from us and we have a few already they're buying more and Yeah, having a 45 drives XL 60 which is Here, you know 60 drives in the system still ain't gonna saturate a 10 gig. Sorry Yep So I'm going to go plug that into the kilowatt real quick so I can answer the question and be done with the ace rise in part of this Oh, by the way, I will have all this time indexed I'm using some tools to read time index all the live streams after I'm done with them But let me go plug it into the kilowatts so I can get the actual Usage of that on here so go ahead and I'm gonna put the little countdown spinner thing on it So I'll be right back We have our numbers. I just gotta refresh this phone real quick Catch up on the comments. Let's see here. I have 10 gig at home wonderfuls backbone. My servers doesn't matter Yeah, so let's hold on here. Let's Get this How many watts do you think it uses? We'll play we'll play this as a guessing game Someone I'll throw the numbers up here So the peak wattage When it was booting up the VMs everything else hit 52 watts That was the peak wattage then it idled It was bounced around a little bit went down to 12 then idle all the way down to 12.2 watts So 12 watts at idle at the plug via a kilowatt device and 52 watts was the max it pulled on there So not bad. I Didn't know the reason I've been using it is because it's such a low wattage and really you can't hear that device It is silent. I guess I made it make a little bit of noise when the fan wears up But it's minor. It's not as loud as most as some of the gaming laptops are deaf I've heard have louder fans than this device. So Yeah Yeah, the mini PC Fun stuff now the other things I was working on This is me over at the CNW our lab This is the lab in Toledo, I was there my son was there to practicing his YouTube face See see me taking a picture But we have a few things going on there So let me may show you a picture of the server. This is the video I'm working on next is the Where's it at here? I? Thought I had a few other pictures of it. I have video. I just didn't have much on my camera It's this bad boy right here This is the one that is going to be I'm gonna do a video on this one This is the server is about a hundred thousand roughly a hundred thousand dollars We're this is not a one for a client we have and Oh, yeah, I did it's the R286 DME later and we'll talk about that so we can get your PfSense going If you didn't know me and me and the beard IT dad know each other, so We we're all part of the not-so-secret Creators group of other YouTube creators that are in the tech world So a DME so we can chat about this so we can get your project going because I did review it And now it's kind of chilling out so maybe we should co-lab and We'll figure this out we'll get the PfSense stuff done Let's see Hey, Tom. I I have heard that some high providers don't really support the new Intel PE core hybrid processor architecture I Don't know enough. I can't answer that right now That would be a good question to post in the forums or even in the xcp and g forums of how they're handling those particular cores Then so crazy can't keep up. We should catch up sometime We just you know how to DM me with us Well, if you can join us tomorrow, we're gonna do the We're doing our meeting tomorrow there so Anyways, you know what I mean? I don't get too off topic on that Morning from the land down under But let me see if I can pull up. I have a few other pictures of this particular server I want to show you people. What's in it? This is the other side of what we do me limit all these here. We go Make sure I can show all these photos Yes All these photos are safe to be shown Yeah, this is what's inside that server. These are all the Nvidia cards inside of it This is a pretty wild. These are all RTX a 6,000 and I think that's right I'll have the right model number when I get the final review done But yeah, this is definitely a wild server for running things like hash cat and stuff like that It's definitely pretty cool playing with this We we were trying to figure out the best demo to run so we started running some hash cat demo Because why not what are you gonna do with all these GPOs besides crack passwords? We were let me look how many passwords how many passwords in the So 14 million passwords. There we go we were using the Common password list called rock you and running this password list which This data set We can Yeah, rock you contains 14 million unique passwords used in 32 32 million accounts. I think this takes less than a minute to run through all the hashes And it is wild how fast that can go it is Yeah It is crazy how you know, we did this is what a demo was running a several several different things in hash cat with all the GPUs I Also think it pulls around 3000 watts when all the GPUs spin up we we don't have any we we would measure the wattage for you But one of the challenges is first it started blowing fuses. That's our how we know how much it pulls So it has four power supplies So we had to split where each two power supplies pull from one from one side of the lab one from the other side of Lab so it wouldn't blow fuses And we had to make sure everything else was turned off Yeah, that's It is pretty wild how fast that how fast it was for doing that So this is gonna be for a pretty specific job for the client, but I mean this this is Probably a $300,000 project for the client between the 100 gig networking the XL 60 servers they have more than one 45 drives XL 60 petabyte of data server And in fact we can log into that that I have access well log into the other one too, but just not right now Yeah, what is it? I gotta get the password to it. Oh The joy of 110 volt in the USA a lot of our a lot of the server stuff ends up being 240 But you're right that the start our lab does not have a 240 in it. Oh, it's probably worth worth mentioning I think I got if we jump through these pictures There's what the back of the unit looks like Yeah, it's a beast of a system. It will take It will take few minutes to boot up. I do like that. It says that We did have to take it apart because we had problems with it. That's gonna be part of the video. Oh, these are building pictures people pictures That's there's a server pictures. I Did another video where I talked about the building tour, but we did have to take it apart to get to the Insides of it. I do have some video of this because we had to get to the Stupid battery if you can see I know it's a little blurry, but you can see the battery in there Yeah Fun stuff though working on these things This is some we I don't know how much I it's kind of weird doing mini PC reviews also talking about $100,000 servers and things like that But you know, we cover quite a few things We we we use some of them for some of our lab and projects But we also, you know build these out for our clients pretty expensive things I try to figure out what does or doesn't fit on going on the channel the storage stuff for sure though It's going on the channel. I talk about that quite a bit. So let's get the I need the password. So So do open All right, so now we can log into the other device This is my next I got to get this done because That's why I got to work on this is my project for tomorrow This has I got 450 gigs for caching on it You know, that's a good size ZFS cash right 450 gigs Storage wise we have You get the screen readable there we go 700 usable capacity with six raid Z3's and That's still put that puts us at only three point. I got what yeah 24 terabytes of data that I set up on this Yeah, the this server is pretty Beefy what I want to do is rebuild it as D raid. So I want to pop this out I did all the load testing while it's in raid Z3 and I want to swap this over if I got time I'm gonna get this done tomorrow to a D raid setup and see how fast it re-silver So I want to do a re-silver with raid Z3 and a re-silver using D raid So I can try to figure out what's a more effective way to Do this there's trade-offs It's not like D raid is the answer people think because D raids available It becomes the answer to their problems that I'm like, it's not that simple It's not like D raid automatically solves other issues that you'll run into because D raid has a lot of trouble with the smaller rights but then you can put a Metadata drive in there to help with that. So there's there's always trade-offs. There's no, you know Easy answer for it Yeah, this is this certainly has that big disk energy here 726 terabytes What you can do in a homelab many PCs you can do in a business enterprise just scaled up So all the topics are good most people see fancy enterprise equipment too often Yeah, I try to make things more relatable because I think a lot of channels get caught up into ooh the expensive fancy stuff and I I don't really I don't really like doing too many videos that are like these unobtainable I like to show people what things are how they work how I look at things But I don't like to just show hey look I got expensive thing that to me is not not as good for I don't know I guess it gets the views It's the same reason like people driving around in super rare exotic cars that cost a million two million dollars Gets the views people like watching it, but I don't think that's as educational and most of my focus is education You do know you have 240 at the panel right electric extra runs on 240. Yeah Yep, absolutely budget for one dual CPU server, which is best hypervisor with churnet hypervisor churnass or virtual or churnass or bare metal machines on it, you know my My real preference is gonna be to have churnash running on bare metal that is just always my preference I don't like my one of my other friends. I made friends recently the guy who runs Jim's garage I think he's got a great channel I haven't watched his video on it because he just published it on virtualizing churnass But he does it right because he passes through the hba if you're passing through the hba Okay, that's a reasonable way to do it. You really want it to have bare metal access to the drive So you don't have problems. And so if you're passing through the hba, I think you're good But you're also adding complexity to your life. Do you need that complexity? How much are you saving with that complexity? some people I get it if you only have A more finite budget, but you really want to run some of the churnass functions Wrapping it all into one machine can be a way to solve that problem But you don't want to play with the liquid setup. Are you talking about the liquid ones that wendell from level one text been talking about the um That stuff's pretty cool. Or are you talking about the liquid cooled setup because those are cool as well Got tail scale pf sense accepting routes I'm gonna wrote route is the same set as my pf sense made of vlam But bsd doesn't like uh two routes at the same space Probably has a problem with that um It I don't know. I haven't run into that on tail. I haven't tried to mix Matching routes and see if tail scale has a problem with it. Um when it crosses boundaries on pf sense I don't know there's a possible though got two servers 144 chair by raid Running access camera station. The one is 90 full takes forever to be put as 50 full. How would you troubleshoot the slow one? um Raid six is just going to be slow. I don't I don't know about troubleshooting. It's going to be That much data on your I It's not going to be great raid 10 would probably be more ideal So I don't know it's not really a troubleshooting. I think you just Um You may you may have run into the limitations of the hardware that you're using You may have to start upgrading it One maybe an indexing problem Is the other issue once it gets 90 fold it may not index it very well This is a jason slago makes a really good point there Some file systems uh in zfs can is problematic. You're going to get more fragmentation with the file system When it becomes that 90 full As the pf sense user is an decade appliance better at 10 gig uh SAP support like I know right now on customer hardware some of the cheaper 10 gigs are not available to repository updates I mean If you're buying an appliance from netgate that has 10 gig I guarantee the 10 gig works on it So maybe I don't understand your question. I mean the support from the netgate appliances is Flawless for their 10 gig because the device ships with 10 gig And as always jason if you want to pop in feel free You you have access to this Both tb core instances are in proxmox with hva pass true works great for me Um, I'm not ready to go to scale yet. Yeah, I mean It the hba pass through is not a bad way to do that You're supposed to be working on the code for codemash. Oh, I didn't mention that I'll be at codemash next week So if any of you are going to codemash jason's teaching or speaking at codemash with uh john hammond They're doing some Our reverse engineering stuff I'm looking for it. It's gonna be I'm sure it'll be a great talk Oh, it's running on windows Uh, I'm sorry Yeah, I I don't I'm assuming it's probably running like exact vision because those are all those are windows based That's uh Might be where the problem is with it might be an issue All right Now Start driving a few other topics here I will mention Because uh more and more people have been signing up for this and I did send out my first newsletter That if you go down here Oh, this this is fixed. I don't know who fixed it. I don't know if I did or my web dev did I can't remember but this is fixed now someone pointed out. This was broken last time and the email address This is for my newsletter. So if you are interested in joining my newsletter, that is a thing now that I got going and I'm got the Newsletter going out for anyone interested. It just covers my videos and different topics and a few other things Can you add remove metadata and pools? Uh, like you can a log drive and true nas I don't think so. I think once you add them, they are permanently part of it because data starts living on them That's why it recommends putting them a mirror. So I don't think they can be removed I I'm going to do an updated video in there Oh answering the question Jason posted here if it's ntfs. This is the person with the 90 full drive It's probably a fragmentation problem rate six is going to be two x reads plus one for every right And the seek time in those are drives is going to be terrible with it greater than 50 percent. Yes That And with less than 50 percent is probably uh indexing it right to ram. Yep I have a 10 gig pf sense with the 25 gig switch under my limited 10 gig without layer three support Well, you can't route if you have a 25 gig switch You can have 25 gig on each network that has all those devices attached to it What you're not going to do is route it through the pf sense at 25 gig, but also why would you What is the use case by which you'd want to do that? You don't route your storage I repeat that a lot because I run into a lot of times where people start routing their storage You put all the devices that you need to talk to 25 that need to talk at that 25 gig speed on the same network And that's how you avoid that problem You cannot remove meta jj threshold. Yeah, I didn't think you could I didn't think there's any route any path to do that Routed storage is a high unavailability layer. Yes Well, the client file server is routed through the firewall, but the server's obviously not yes Yeah, it it happens a lot. It's one of those. Why did someone build it this way? They weren't thinking properly One more thing i'm going to mention here uh back to my website because I threw this I just added another book. I gotta fix the text. I got the text Like the text is good on this one, but I added another book to the reading list. Uh, this particular book here is quite good Tracers in the dark by andy greenberg Um, I just finished this one. I definitely give it a recommended list. I've been putting on here any of the tech books I read and This particular one. Wow. It really dives into the history of bitcoin and tracking down criminals using bitcoin It's it's got some controversial topics because the question is how much should you look into bitcoin? The company chain analysis is centered in this a lot They're one of the first but there's more companies doing this now that really dove into how trackable all of your bitcoin transactions are and in It started in the beginning where people made some assumptions Made the assumptions that it was not that traceable like it was very anonymous It turns out bitcoin in the infinite ledger that is that Is just that an infinite ledger of all the transactions and they are able to Absolutely, they go through the process that how law enforcement uses it How they were able to bust all these different crime rings the it covers like the silk road The alpha alpha bay and several others It's all the different takedowns all related to how they sorted out and identified them But also how they were able to trace all the people's transactions So, uh, one of those, you know, I finished a book and I like giving some of the book recommendations for people that might be into Diving deep into technical topics and that's what my book collection is focused on on my site is I read a lot of books But when I read a tech book I add it to my site right there. So something Have you read the art of computer programming? Uh, oddly, no, I have not that is not in my list that's But I've never been good at coding I've read I've read the art of deception and the art of intrusion. Both of those are really good books I may I read those so long ago. I might want to read them again Uh ghosts and the wires. I've read those are the three kevin mitnick books. I read. I don't does he have other books? maybe Cult of the dead cow Really like cult of the dead cow. That was good. I'm a huge fan of cori doctrine. I threw a couple of his most recent books on here I like his fiction too. So cori doctrine was liver brother My kids have read that I That's one of the ones I recommend a lot of people. It's a great one to start with. Uh, it's enjoyable Even if you're not a kid But this is a good one too algorithms to live by I like this one quite a bit the computer science of human decisions algorithms to buy is all computer science decisions and how they apply in real life Solid book. Uh, I I've made some notes from that book If you want to know the origins of the internet where wizards stay up late if you You would be shocked at how much Uh, it was packed into that book. Like it is so much great history and detail. I'm glad there's a book to kind of Categorize all of it. So yeah Uh, when you I say routed storage, I mean don't put the storage network on a separate network So you have my storage server should be on the same subnet as the thing that wants the storage So my xcp and g server wants to talk to my cheer nas. Those are on the same network They're not on separate networks. They don't route through my pf sense The number of people that want to route their storage network through their firewall blows my mind Um smb over vpn sucks. It can be done, but I don't recommend it That's one of those why things I get a video where I actually show how latency affects Uh, dramatically your performance over smb. smb is not really well suited for high latency connections So smb degrades rapidly when you have any latency on your vpn And you already have a lot more latency on a vpn than you do for a local network So if you get a little bit more latency or a slightly slow connection You have an exponential problem because it just the protocol itself wasn't really designed to deal with that You're tempting me to come on Yeah, you know where the buttons are Uh By sub nano second storage appliance, uh, and then route through firewall sounds like very good enterprise compliance logic Ha ha ha Uh all that helps a lot thinking locally my land traffic is going to go through router and back. No not You gotta remember The gateway is there for when things got to get to the other side of that gateway You do not when they're all on the same network pf sense for devices on the same subnet nothing happens They're all on the same subnet Jason's trying to procrastinate doing uh really Writing all of his stuff. He's he's got to present next week. So he's writing it now because The homelab show was great when tom and jason this week jason should people are people are peer pressuring jason now Uh, look at a move to papa west from windows. Uh, why did you pop us over our disc shows? Um, it works great. It's polished I don't really have a lot of other answers like all the little shortcuts for papa west just are really nice Once I switched to it over a bunch to I just like the extra little things they did I think since then a bunch who has probably gotten More polished papa west polished it up first Then once I went to I played with a bunch who kind of recently and I said, oh, that's nice But i'm already using papa west. So why would I switch? I know where everything is it works really well And I don't have any problems with it He just well jason's part of the community too. I mean jason's as much As much if not more of a nerd than I am about it, especially networking stuff. That's jason's jam. So Of course, so is programming and reverse engineering and hacking and uh, and running a tech company So me and jason it was easy for us to partner. We already had a lot in common Uh, my the mac book I have is an m1. So no, uh Mac but mac book air m1 you cannot run papa west on that Now there is a linux project I've already forgot the name of it. There is one to get linux running an apple silicone I think that's pretty cool that they're working on it. It's not feature complete yet. So There's that uh, let's see here All right, cool Yo, aren't you supposed to be doing something else? I'm trying to figure out how the windows form designer works That's what I'm procrastinating. I'm writing Windows applications and dot net I have all the reasons in the world to procrastinate Yeah Doesn't uh papa west have an arm build uh The apple silicone is very unique Uh, assi a s. I don't know how you say it. Yes, it doesn't have gp drivers yeah That's and that's the challenge. Uh, my understanding is they got most everything working I'm I'm actually curious to try it on my mac book at one point in time like when it gets a little more I mean, I I love the apple mac book hardware It's You know apple is almost playing on a different level with it when you compare to some of the other laptops That's why you have one apparently it runs. Uh, you can do it in parallels Yeah But Yeah, it's kind of uh I don't know if the market will catch up because like when mac mac's, you know, what the m3 is out now They're just playing a different game when you compare them to the windows laptops. It's just I don't know. It's it's they're so fast or so smooth. The battery life is insane I I can't buy a laptop for the especially for the price point that competes with my mac book air I'm by the expense like mine's expensive. So there are definitely cheaper laptops. Yeah All the mayor is a great price value like Spot it's they killed it there and because to me I always look at it because I don't like I don't use it for any of the horsepower It's a terminal to me. It's a terminal with like an amazing 12 or 13 hour battery life I think matt was impressed because at the management meetings the off-site management means he's like you didn't plug it in I'm like it's still got 50 after using Mind us. I mean mind us too like It's yeah insane. Yeah Uh, I see some of people get the framework ones Yeah Those are nice. I mean arm is the secret there right like intel Uh, it is so power hungry compared to arm Yeah, and it goes a step further when when you have one company Designing the hardware and the software. It's going to be so much more well tuned Than other ones. This is one of the reasons the raspberry pi is so well tuned for applications Versus some of the you know clones if you will of the raspberry pi The raspberry pi has had enough time to live an ecosystem people have customized the hell out of it. They've tweaked it. They know exactly What to expect with that hardware and how to get the most out of it Uh, jeff girling did a great video talking about like the high alternatives and all the downsides of them Trying to get some of the other the little quirks you run into it If you have something more than basic application support you want Yeah Hey, I figured thing out I hate windows development Uh, I actually I popped on to talk about Oh, sure. That's gonna be a nice guzzie. Why not? Yeah, absolutely. Um I'm curious how efficient a framework 16 extra battery will do I don't know. I I think the frameworks are pretty neat, but I they're not they're not going to be at the mac level yet because it's that arm that it's the arm thing on there That's the thing but smb ice guzzie. This is a good topic. Let's why should we route these? Let's repeat this to the class. I mean you latency, right like I mean on top of complexity Right your storage. What is one of the biggest metrics you look at when you're sizing storage? You look at round trip latency, right? Like you you really want less than one millisecond response time and you're not going to get that through any commodity routed device There's some data center stuff that maybe you can approach that right, but it's just not great Yeah, it's It's a challenge we run into way too often in the the business consulting that we do Oh, you're having problems with your storage servers. Oh, why did you set it up this way? Why did you tie it through this? I mean admittedly mat like You know mat's the one that added Routable interfaces to most of our storage networks because he got tired of having to go through jump boxes to basically manage things that are too stupid That dedicated management interfaces But then he you know at some point another somebody's mounting ice guzzie volumes over like that router interface. I'm like don't do that Yeah, that that's where you get yourself into trouble. Um You can use remote network storage next cpng I mean you can but once again, you don't want to route it. That's not the good idea A different topic is going to be if you're sending things to s3 from the zenn orchestra backups That's different. That's not the same as router interface. It does have that ability it's got some interesting abilities actually because uh, they added the ability now to clone backup repositories so you can Back up to your local one with low latency And then clone it and stream the data to another one so you can have your off-site backup stream to a completely another backup location um, and they do it that way because The backups themselves because they can and they've gotten better even with higher latency links, but they they pull your delta backups using nbd From the system, that's how they get them really fast But of course they their last blog post talked about the latency challenges that are created when you have it and they've actually Figured out how to even optimize it more when it's under a high latency link and also let you churn it up faster when it's under Low latency situation Yeah, I mean so vmware has some secrets for stretched like these hand clusters and things like that And there are some prior prior sort of things you can do and when accelerators do exist, right? Like if you want to go down a rabbit hole go look at uh The windows like branch storage stuff that basically does like local smb caching of remote volumes to speed up and deal with some of the latency problems That's like a day of your life. You'll never get back and then you'll try to configure it It's like a month of your life. You'll never get back But yeah, I mean even microsoft understands that that's not the cifs and smb aren't a great Solution there's too many acknowledgments. So it's too many round trip times Yeah, when you look at this and this is I did that video I think in mid last year about smb or vpn and I played with a fun feature inside of pf sense If you if you go to any firewall rule in pf sense There's an advanced option if you scroll down the bottom of those advanced options They have a few ways you can introduce jitter latency You can actually insert that as an option in the rule and I just use that option to Create latency to show you and simulate what a network would look like with latency on it And you can dial in how much latency you want And because of the way smb does all these back and forth acknowledgments like hey, I got a tiny bit of the file I sure did and it goes back and forth Sending that that gets to be an exponential problem It's actually one of the reasons why the time calculation is so hard when it starts calculating the files You'll watch a big file. It goes. Okay. I don't have as many acknowledgments But a bunch of little files will get the numbers flying all over the place for it to try to guess How long it's going to take to get a transfer done? Yeah Yes, correct. I mean there's a lot of I think it I'd have to Double-check, but I think like file creation is like six or seven individual round trips and latency just heads up Is there something on the internet that recommends routing storage why do people do it? Yes, there's a lot of bad forum posts The the Venn diagram of people that don't understand the difference between layer two and layer three and route storage is almost a circle That's the good answer for that. How about routing storage over layer three switch? No, you shouldn't If you have a layer three switch that is able to have like a six on it and could do it You could there's there's a solve for that, but now you're talking about switch so expensive No, I mean you can do it right like it's Uh At some point or another serialization delay actually matters right like it takes time You can't even on a local land going through a couple of layer three switches You're still looking at a couple milliseconds typically right like when you you know trace rats through all those hops because You know think about oh now we're going deep network nerd Right think about your transit transversals up and down the osi stack Right, you got to go your application layers got to get packaged inside of an ip well an ice guzzy packet Which then gets uh put inside of a udp packet, right? Which then gets put in an ip down to a datagram physically converted and serialized over the wire Then each switch in the middle has to undo that at least to the udp layer Right or I mean a layer three switch may only take it to the ip layer, right? It may not do layer four may only inspect the ip header But it still has to then re-encapsulate it on the new map. It's yeah, it's not going to happen instantly Yeah, you always don't think about all the How the bits flow in all the things and all the steps Me and Jason were talking about this the other day It's kind of because we're both old school. We've been doing this for a long time but we think of things that almost like I would say like the machine assembly language because not that we write code and assembly language I think about it from the fundamentals of that. How is this bit going to get from where it is? How are these ones and zeros going to turn into the next piece to turn into the next piece to eventually come out to be That piece of data As a whole and I think if you if you take some fundamentals of it like peel the mystery back for how these work You go. Oh, that's logical. It has to be broken down in these ways You know or even uh, like Jason said people who may not know the difference between layer two layer three Understanding those osi models. They matter everything is fundamentally still built on this for all the networking Even though it's been years since I've studied them It's still there's still as relevant today as they were some old book. What do you got here? Oh, yeah tcpip illustrated volume one. How old is that book? Oh, uh Oh, let's see 94 1994 yeah, wow Yeah This is the one though if you want to like if you want to actually learn and understand it This is the one I would probably get it's the one I tend to use for in uh networking classes Internet working with tcp Yeah, like dug Comer. I think it's a similar vintage um Does the rule for not routing started apply to user shares We have a file server on a network and users accessing A workstation. Well, no Maybe it's a little different. It's complicated That's yeah, if you have a very large network you may have segments of your network and You may not have a server with multiple network interfaces on it So you have a network interface in each network and then you may have to do it that way I mean when you talk about, you know Companies with thousands of users. Yeah, then you have a you you have to start diagramming it a little bit differently when you lay that out Uh Here even even worse See antsy programming language look at the authors Brian and oh Dennis Richie. Holy crap. It's k and r of k and r the k and r c standard Yeah, that's a classic there Uh I mean this one was type set. I'm pretty sure on like a typewriter Unix is a registered trademark of AT&T. It says on the author page. This book was written in 1978 This printing is from 1988 this version of it. Wow. That is a classic there I may I may go back and maybe I'll get some old books and read them It's been a long time. If you want any of these I'll bring them in and grab them read them and give them back Yeah, I I learned on a lot of those old books. I don't know which ones I read. It's whatever ones were in my high school which were Back in the 90s. Like that's what I had access to Um, here's an easy one. What is the best practice for on-prem window servers running dhcp with unified gateways and switches? We're on them. I mean just don't don't let unify take over your dhcp Let if you have a windows domain Windows should hand out the dcp and the dns generally although understand the caveats of that because depending on what day of the week it is and what Segment the moon is in it may or may not require a cal for every device. It's giving a dcp lease out to Oh, yeah, anyone that actually does it, but if you actually read the license terms I believe you are technically required to have a cal for every device that gets an ip Yeah, I remember um, I was laughing really hard. I was listening to paul therot go over licensing changes. He does the Windows weekly podcast. I got to meet him in person. He's a super interesting guy and He was talking about how hard it is for anyone to really solve the licensing He goes no, there's no such thing as a fortune 1000 company. That's fully compliant. It just doesn't exist Nope every every single one I dealt with that's gotten audited is out of fine Yeah, microsoft just they've made it so complicated. It's impossible tasks to be properly licensed And even if you were to achieve it you've only achieved it until They're six months later when they changed the licensing rules a little bit Uh generally though you want walled off you want your PC network like where your workstations and things are that should be a network If you have a bunch of random iot things that shouldn't probably be on the same network as all your windows computers maybe Maybe the printers will be on that same network because printers prefer it but Segment things out. Um that aren't your windows or things windows needs access to like printers Maybe scanners printers are definitely the big hated part printers Yeah, I figured out how to add a picture now I can rick roll the class Ah, it's like hippa HIPAA That's a different set of complexities We're we're getting into the other fun stuff the cmmc complexities and No, brad had a hippocomplexity before he left him and lori lori was getting mad at the person She was talking to on the phone because they didn't understand it. Oh I Love when doctors have asked me what are you the hippo police? What are you gonna turn me in? I I'm just letting you know The laws are here Maybe I even had at least it's been a while since I had someone uh tell me that although I've had someone that wanted me to Uh go over their hippo but when I sat down with them It was one of the quickest times I've ever left a meeting They wanted me to backdate all the stuff to say that they were compliant six months ago I'm like, what do you mean compliant six months ago? I just met you. I didn't know you six months ago Well, we're kind of going through a little problem right now. Oh Well, I can't help with that. Oh, we need someone who can and I you came highly recommended. I say well I uh, I don't like jail I come highly recommended for the same reasons. I won't do that Yeah, I said so they said but you can't can you recommend someone? I said, I don't know anyone shady enough to do What you need done? I said, I'll be honest. I just don't have friends that low They're like, but it's a big deal that we have this uh dated about six months ago right now because He might lose his license or something like that. I'm like, yeah, that's definitely a you problem. I'm sorry And I left I never heard from them. They were gone I remember noticing because I know where they were located and I noticed I don't know It took a long time. I feel like it was almost two years later that the signs were off the building Yeah, I mean the problem is is that for many many years HIPAA had no teeth Right, so like no one paid attention to it and now they're starting to actually pay attention to it So people are like, oh, I don't have to pay attention to that because no one's ever gotten fined and now people are Yep, they're getting fined and some of these people have committed some egregious Slip ups and poor behaviors and things like that Oh fed ramp. There's a fun one Oh Please no, I think I would walk away. That would have to be really there would have to be a lot of zeros on that for me to bother Yeah, that's It's a big no Backdated HIPAA compliance. I'm sure it's a big pharma tech bro. Uh could start out for them. Oh, yes I'm sure there's definitely some big pharma tech bro, right? It is Uh dive into that Uh question for you Jason what There was some dead mini PCs. Um, which ones died on you? You said some died Uh, I've had several of the top tunnel ones die Okay, that's the top 10s. I couldn't remember the names. Yeah, I don't think they died I think this is shitty SSDs they use in them or what dies and we've had several of the SSDs dies So like I think that uh, and they overheat like I the I think if we were going to use them going forward, I would take each one of them I would open it up double check the thermal paste because like the only thing cooling those things is a thermal contact pad that takes it to the chassis Uh, and if you watch any of the sth's videos, he's also noted that a lot of times they're just not bonded very well Yeah, and then I would just order them without ssds or ram because what they send you is trash Yeah, they they're not It's kind of weird. I like the latest ones serve the home reviewed the solid state box But it only has a c300 in it that processors just not fast a lot of work done I like that one that he just reviewed with an atom processor like as long as you're If you're really looking to use it for a router that thing is a workhorse Yeah, well, that's that c300 and the reason he brought up the uh compared it to neck gate and to true nas My true nas mini that's what it's got is that same atom processor in there That's an old school workhorse and it's got the qat crypto acceleration, which is going to be good for your The I love the fact that it's got the connector for a direct attached storage on it too Yeah, well, I mean the wrong connector, but a connector nonetheless Yeah, I do like it's I love that it's a wrong connector It's like clearly Oh man, we got this connector on this reference motherboard. We paul. What do we do with it? Oh, it's got we'll just put a hole in the back of the case It this is great But I like that it's the internal connector you're connecting so 100% It's not made for external, but it works. I was wondering where you got that cable It's um, they have an internal external cable that run through the back Or does it come with one? Uh, I don't think it comes though, but you can they're findable. I guess people sell them because you just make a hole in the back of the chassis Yeah, okay because they I don't know people do weird things actually, um Wendell did a video Recently were I think he did it one with Gamer's nexus as well He started breaking down the mystery connectors all the different breakout cables that they have now for all your PCI to mvme adapters and just the weird shit you have to go through to get these things connected But the fact is they make them that's the there's so many random ones Yep There's some neat stuff they got on there, but this box is pretty cool Yep, that's yeah, that's the weird connector Mm-hmm, but this has two and a half gig and ten gig on it and if I'm not mistaken, um This one's available on amazon. Let me pull it up real quick Okay, I think we've confused the masses now looking at michael's comment Oh Did we don't route block storage there? I said it. Oh, there we go good clarification Uh, you can you can route Samba or other file related storage things nfs, uh samba understand limitations late high latency links locally routing smb is going to be just fine WAN routing smb is going to be meh Uh, because it's very sensitive to latency Yeah ice-cozy Really ice-cozy and nfs just avoid routing those. Yeah, I mean you can route. I've routed nfs plenty But again, like understand the limitations of doing it nfs is not a block level, right? So like Uh ice-cozy fcoe uh Those are probably the only two you're ever gonna come across you'll probably actually never come across fcoe It's so it's mostly just ice-cozy. Yep, and I have a video about um Storage design. That's what I titled it. I did it only like a month ago I put graphics in there of what you can and can't route and what that all means To help explain it to people a lot better. Um, but this little box. I'm sure of the home This is impressive for 359 bucks. That's a lot of connectivity on this box Yep 16 gigs ram 5 pro gig SSD for 350 bucks There's not much cost savings to go to the what is there 20 bucks to drop it down to like a eight gig Just go the 359 ones the Best deal. I'm not much more 400 bucks. You get 32 gigs of ram I really like that you can get these on amazon and not have to order them from alibaba and eventually get them And I love that they're passively cool. I debated about buying. I don't have a use for it I just think it's cool. I thought about buying it I know they're they are neat They I got that picture so perfectly aligned. I didn't know if they just photoshopped the face Oh 100 percent We can't even trust the images on these Uh, but this does have mvme on the inside and two sata connectors Now you can only figure out how to get one of them connected because there's not room for two drives inside of it But they did put there's two sata connectors on it. So there's that There's your second one right there. Yeah, that's pretty cool I wonder what the boards are. I mean, so these are clearly just reference boards are buying from somewhere and stuffing in a case Like I wonder what they were made for I wonder a lot about that too because I've always were what where's the other device that they clone those drums They engineered it. Yeah Originally, yeah, I think some of them are probably hmi's honestly Right like uh or some sort of industrial thing or you know, these could be I don't know like I I don't know what else that would be for Like I that there are so many weird design decisions Like why is there a sata on that when there's no reasonable way to mount r driving it like why? Why is there an s o dim that doesn't fit in the case? Like yeah comes with free random bios firmware. Yeah, you'll never get an update for it, but I don't So this question comes up a lot What is the likelihood that someone engineered a backdoor into the bios that implants something? It just seems it's statistically improbable. I feel it Yeah, we we need to do a whole thing on threat modeling. Yeah, I think that that should not be in your threat model That's Yeah, it's Bot netting is the worst you're going to be worried about right like no one's coming after your homelab data Yeah, and like I think we said this before no one's going to burn their o day to uh crack your homelab either You might come home one day and it's mining bitcoin and it's part of a bot net But that's about the extent of your problem. Now what we did see this is uh, probably three or four years ago I thought this was funny In it's what makes people go that step further of maybe there's a bios problem not likely but not likely at all but the One thing they did do is they shipped with a backdoor version of psense shipped on these And i'm like, why would you trust their load of ps sense? I mean most of them the hardware inside of most of these things I think most of them would support running u-boot or one of the other like open source for bios design There's there's nothing super secret about it I don't think most of them require binary blobs to actually successfully start up because they're not using Proprietary things like the pies pies are actually really terrible about it. I think most of them require binary blobs to actually boot Yep This common modity x86 hardware does not so you can almost certainly just load net U-boot or one of the other core boot something like that on it if you're worried about it Ah Yeah, one of the alleys had a rat pre-installed. Yeah And That came up when I first was talking about like how these little uh mini pc ship with windows on and i'm like Well, the first thing I do is format it I don't know I didn't spend any time looking at the windows on it people ask you if I looked at it or Scan to see if they put a backdoor and i'm like why i'm gonna format it Matter of fact, if lenovo ships me one i'm gonna format that one too I why don't want the vendors that we have garbage on there I usually want a nice fresh clean install with whatever we want on there We used to do that when you now use emmy to handle that and Instead of blowing the os away reinstalling it that we have just emmy Formulas that just clear all that crap out. Yeah Which is another way to handle it But if you're a home user and you don't have access to something like we do like emmy bot Just format and reload it and clean it and load it the way you want The other thing too is like one of the ace magics has that little stupid screen on it Yeah, matter of fact, I got that one sitting here I reviewed this one too. This has got that Yeah, I feel about that the same way I feel about like a rgb like my my case on your head is really nice rgb in it It's always red and I never look at it. Yeah This uh, you can't see it because it's off because that's a screen on it because apparently that was a Unnecessary thing and the software gets flagged by windows defender and removed as soon as it boots and updates You have to tell I think there's I think they finally got it not flagged But I found a forum post that I linked to when I did the video Someone's kind of reverse engineered how the screen works on it. I forget it's like an arduino and something else someone figured out what they used and figured out how to control it and It started off someone. I think they just copied someone else's open source projects Someone sounds some other source code. I get up for it Which not surprising they didn't put a lot of effort into it Just a weird choice. They uh make on some of these mini pcs. Yeah Uh, I broke my code Chinese mini pcs have mailware backdoor built into bios or something on board and hardware. No, I just don't see that with them It would be too easily found and caught for one uh You would It's it's actually hard work to do that and they're trying to get a low-cost pc out there Their goal is not necessarily do that now The software they install on the systems that might be a little different because there's plenty of garbage mailware And I don't even know if it's on purpose. Sometimes it's just garbage software. It's not necessarily meant to be nefarious They found the lowest paid worker that could get windows loaded and said you Go figure out how to get windows to work on all these. What's the license? I don't know. You have to go google and figure that out How to get windows on these pcs I mean, there's a bunch of oem keys that you can use that will they may or may not survive activation But they'll allow installation I didn't think about this someone pointed out uh with the oem keys The reasons some of them are so cheap because I get offers Um for people want me to promote their oem keys and I always think that's kind of funny But why are they so cheap a lot of them are bought in foreign countries? So they're playing the exchange rate games um And I didn't know that uh, I think it was one goal that was explaining how that works Yeah, so that's actually very much a thing I hadn't even considered uh because a lot of a lot of these enterprisey softwares have differential priceless and in developing countries they will sell it considerably cheaper and offer way more incentives than uh, then Selling it in the u.s. And there's actually other games you can play too because partnership level right like uh, is a Microsoft we're a silver partner in various pieces, right? We're not cool enough to be gold or anything But the requirements for me to hit gold in like a developing country are a fraction of what it is in the u.s So if I could go sell a bunch of developing country keys to somebody There are there would be very shady partnership reasons to do that. Yeah Thank you very much for donation 10 gigabit even if you don't saturate throughput can get benefit of deeper q lower latency At least that's how a colleague justified Maybe not yes Your latency usually I mean 10 gig and one gig don't have big latency changes Yeah, I mean, well the serialization has to happen 10 times faster. It's actually not 10 times faster because uh Ironically enough most 10 gig that you're gonna run is not actually 10 gig It's actually a four bonded 250 megabit channels or two 2.5 gigabit channels, right? So you're gonna have the serialization delay of a 2.5 gig ethernet link instead of a 10 gig ethernet link So two and a half times better, but we're talking about nanoseconds here Yeah, it I pointed this out that it's so indistinguishable when you even When you think what the muxing that has to happen versus a DAC cable versus like a fiber cable There's different because it's got to convert each one of these but they add such small amounts of latency to it They're not significant. Yeah. I mean in the other half of it q Right, like so I will I will give you my qos story when we're talking about cues here qos Is a tool for deciding Whose life is terrible when things are bad? Yeah If you're if you're at the point where you're dealing with cues Then you've probably already lost and we have a client that is running into that on non traditional I'm traditional like I think that it's a three par maybe a storage array And it it's running, you know, there's an IOPS limit on that because it's spinning rust and it performs fine And so you hit the that IOPS like cliff and then as soon as you hit it the whole thing over there Yeah, it's the same thing like the deeper q. Yes in theory. You probably have slightly bigger buffers on 10 gig Do you need? Slightly bigger buffers probably not and and you can definitely find enterprise class one gig gear that will have just as big a buffers Yeah, so not so much Um Most of these little boxes I think it's referring to uh, they don't have the money or time to create back to our bios Yeah, I mean it takes some time if you want to create those this was the argument when um, that who is it that wrote the article They bloomberg when they came out with there was allegedly these chips and backdoors on They uh hit them for super micro. They've never redacted They've never confirmed they kept saying evidence was coming and any quit talking about it and made hopefully We're hoping everyone will forget about it But it takes a lot people took the time to show what it would take to do what was claimed in the bloomberg article that Super micro had been infiltrated by companies Installing these backdoors on these ships and sorry i'm on there That is no small feat to be able to pull that off There's a lot of complexities in it. I thought that was kind of cool The hacker community really showed like this is what it would really take to do that and you're like Oh, that's actually a lot of engineering and even after you do it It's not like a sure-fired wake you have to figure out how to get in and get around the os and inject something into it Just being on the board doesn't make it easy To insert something into the os. Yeah It does happen though, but yeah, like it's not Oh, yeah, let's say he's got their means. Yeah to what end though, right? Like, you know, it's I go back to a thing that I think I said on your vlog It was relatively controversial which was like the penetration of things like micro tick inside of enterprise is relatively low Right like it unless you have a wide audience Right these boxes the target audience for these boxes as home labs Like it's not like Ford is going out and installing hundreds of these crappy top-ton boxes because at their volume They would make their own Yeah Yeah, it's you know I I remember I had a co-worker many many many years ago He was always obsessed with that. They were going to come get him I finally let him know and this is true for some of the other people dude You're just not that important Like they're not gonna the the the might of the government is not coming down on top of you to uh You know, he was a why why 2k doom prepper? They were going to be specifically after him because he knew too much. I'm like, no, you don't I've worked with you. Sorry, dude You're not the one they'd ever be after Yeah, the same thing they're not there's not a lot of value in the home labs For in terms of hacking not that that's some excuse to let your guard down But in terms of are they going to burn their zero day to get to you? This is I think this is a good topic Maybe me and jason just to dive into some time. It's just threat modeling things What are the real threats that you should be focusing on? You know, what are your priorities? Which ones are really? I've seen people go on total rants about locking down every single unused port at their home And i'm like is that really your threat model someone's gonna come in and plug something into my rack potential lawyers office We were we were pitching at one point or another telling me they don't want to move their email to microsoft Because they won't they don't want microsoft to read their email yet They're running an out-of-date on patch version of exchange on site and it's like that's The microsoft reading your email shouldn't be in your threat model because guess what you're reading your email with off your exchange server That microsoft wrote i'll look that microsoft wrote if they want to read your email they have nine nine other ways to do it Exactly, they don't know these running an out-of-date exchange server someone besides microsoft's going to eventually end up reading all your emails Yeah, I mean same thing the msp space is notorious for that. It's like oh vendors are all going to get us popped Yeah, that's probably true, but like the the applet the rmm with our msa on it in the reuse of passwords It's probably actually going to get you popped at least evidence suggests 80 of the time That's what's going to get you popped like fix all that stuff and then we can worry about your vendors getting you popped Yeah, i'm not sure why you would have Unify using dhcp relay. Why why would you have unified between your windows servers? You should probably like post a diagram of the network in the forums or something so we can better understand and answer to question I mean do you have that many thousands of users that you've got to segment your network? And if you got a network that big is breaking it all up with unify the best solution The other thing is is that what you'll often find is that on a lot of these systems you can't run dhcp relay and dhcp server on the same system Uh, they just it's unsupported on most on actually a very large number of Products to do that because they both use the same port. They use slightly different daemons, right? Like so It's very very common for you to not be able to do both on one device, right? Ah, the real threats users. Yes, some people Unpatched exchange for sure Well, I think I have this thing to the point where I can actually write code now to validate the key Which is good because I'm going to reuse 90 of this code for both challenges. I'm writing Oh, okay. So you're writing the ctf for codemesh Uh, so I'm kind of so we're teaching people to reverse engineer I stood up actually if you go to a ctfd dot kd8bfn.org you can see my ctfd And He's a little slack. I'll send it to you in slack. Okay It's my ham call sign, which means your your viewers will be able to find where I live, but whatever You have a unique enough name. You're not actually that hard to find it. You already know that They figured out where I live not that that's hard either Uh, I figured out where your office was playing the uh, Google image dork game rather than uh, uh, then finding it What was that thing we were geoguess or kind of thing you were doing at myset con Yeah, so basically the labs are all going to be in a ctfd and then at the end of the class for like the last 40 minutes I'm going to have like two or three other challenges off of crack knees And I'm I'm going to give a give $50 gift card away cool Yeah Someone says you've given your call sign. Yeah, the chat has a long memory. We'll just say that They bring up stuff I talked about like two years ago people like what happened that thing you did too What do you know what happened? I forgot I did it. That's what happened to it Uh Yep, lots of those errors Yeah, I'm like the easiest guy I'm like the security guy that just doesn't Try to protect their identity right like I'm not that interesting docs me like Yeah, that's it's annoying if someone tries to pizza you or something like that or swat you like I'm worried like but I don't think you and definitely not I run with the crowd that I expect with swat No, we we we generally pull a more mature audience. I think If you're into the kids and games that's where that Because I've had a friend that had to deal with that a lot, but then um There's no question as to why he was an asshole running a gaming server Yeah, and he was definitely part of the problem and he's like Yeah One of our sheriffs lives like across the street around the cul-de-sac a bit And so I'm sure this could be a fun game for them if somebody did that try to track them down Yep, I don't know my ostin's pretty good. So I don't want to find out but Yeah, no, it's one of those Painting about things because I didn't think to um I wasn't thinking about it when I got this house to Put it all inside of another name. Yeah. Yeah I think that domain doesn't even have domain privacy on so Like several of my domains don't Yeah, there's always people who I don't know they behave badly I just uh ignore them and move on with my day. I got plenty of other things to do Oh, what else do we got? I actually never did your four-hour live stream Oh, I don't know what I'm gonna do the four-hour live stream not today that today Oh, definitely not today. I have I still have to go run three miles. So I'm putting that up too. I'm procrastinating my running by coding and I'm procrastinating my coding by being here So it's what is that called productive procrastination we're still doing something right we're still participating in the community and It's fine. I actually again I while I was sitting on this call I figured out enough Uh Windows for I use one form is to set a wpf but or a wfp, but I think I have enough to actually make this thing work. Yay People signing you up for an mlm scheme. Well, it turns out I attend things like chamber of commerce events It's kind of the same thing. They there's enough people there always trying to sell me some soap on a rope bs Which by the way, I realize I've said soap on a rope a lot and people don't know what that is that that's related to Amway I was talking to eric and eric says why these erics is like, what does that mean? And I'm like, oh amway because I know what amway is like the whole mlm thing He's like, yeah, I said amway's product from the early days was soap on a rope so Ah fun Sign me up for kaseya sales calls. I don't think they will call me much these days kaseya No, are you banned from their events officially or unofficially? I don't know. I've not tried to go to another one to find out We know we know kaseya doesn't like sliggle They do not He's hit a few other companies up, but he your goal is to better them Yes You may first aggravate them Yes, I Usually if I have a way to affect the change I'm trying to make without being a lot of mouth at Asshole on the internet. I will use that way When it fails, the second way comes out Yeah Uh, you got four nodes server chassis. I want to build a new I like those four node server chassis Uh, there's a couple companies making those now. So You'll have to come to the buckeye. We have uh, so it's a twin It's one of those super micro four node two boxes. That's one of the uh Our hosting the production hosting environment at buckeye. Oh nice. Yeah. Yeah, let me know Maybe yeah, do you think the uh, when we move out of the other location, they'll let us do anything there or They don't like already I mean, yeah, sure come with me come in and film. It'll be like the third week in january Yeah, and then and they say hey never film here again. Like good relieving I don't want to kick me out. Oh Oh, okay. All right. Well, bye Um four node chassis. I want to build next to be decent. Would you put each chassis in its own poll? Would you bother putting all one I put them all in one pulls? It just makes it easier Once you define all the networking and xcpng is defined by the pool not by the host So if you have four nodes that are the same It becomes arbitrarily easy to put them all in the same resource pool that way anything you define You don't have to define four times Then if you have a shared storage, even if you don't have shared storage The pool is a good thing But if you have shared storage now, it's even easier because if you add the storage it automatically adds it to all four of the nodes I got to do an updated video on networking because that's where people definitely get confused with xcpng They'll start to try to set it on the host. You don't set the networking and a host you define it by the pool Where that gets more confusing is when you have mismatch hosts Because if you you define eth0 eth1, etc and what you want attached to them But if you're not sure what eth0 is on your other ones or you want them to be different I have a script that will set them differently. It's it's not hard to do you go the command line You can flip flop them. They have a utility for it if you don't know what you're doing You're gonna have a bad time Yeah, and sometimes you add extra interfaces to a pf sense box it reboots and all your interfaces are renumbered that one too That happens in vmware as well. Yeah. Well, uh, yeah. Yeah, it was it was in vmware and it was what yesterday Yep, we're building up the update of the lab Yeah, I still um Uh, does xcpng support fiber channel should I think so. Yeah for any of the supported cards. I don't see why it wouldn't like yeah Yeah, I mean q i almost certainly cumulogic and emulex cards like I forget who bought each of those things But they've been around and i'm sure that they're a supported one Q logic is marvel now Yeah, they actually have a lot of different storage support For xcpng because they even do have native some native sef stuff. Oh q lot, uh Interesting Oh cadmium that's who that's who bought uh And cadmium is required by marvel So if you go to their site, they have them all and broad cam bot emulex Yeah, they have what is thin provision. What is thick provision? Uh different storage types block base, but then the cards themselves, you know And if you have a go Just present any of the storage to them you can uh like if you Present it in linux as a drive for example because you know, we've got a few clients that are just running del raid arrays We just present all those drives as a drive and then let the del system handle the raid and it works fine Yeah Well, it works as good as del raid works, but for the client use case. It's good enough Yeah, the actually the opposite problem is usually a problem more for me Is that like getting the raid cards to give you single drive volumes so you can run Cfs on them is a pain in the butt you in that case typically just want an hba and not a raid adapter I've been working on networking docs for xcpg the ad remove was recently done working on next steps to three name interface. How nice Might have a pr for this weekend. Yeah, there's one thing is nice is they're very open to um The community can edit and push some of the documents I've helped them a documentation and things like that and building things on there. They it's a Yeah, it's a very open community Which has been their openness with the community and the contributions back have really helped iterate a lot faster uh That is universally true Multi-path you might need to add the devices You have to go to like wizard school to understand all the options to configure multi-path because sometimes you need a ula Sometimes there's proprietary things. Yeah Um I've done, you know, I never did a video on it, but I did it because I did consulting on it I turned on multi-path and got it all set up. Um, it's not hard to do an xcpg. They actually have it all Inside the ui. It's not hard to do but Depending on your back end storage There are things you have to do for it to work correctly. So uh, it's an example of you're using a true nas back end Uh, even one of the like say you're using a high availability. What is m50? I think I want to put two controller nodes in it There are almost certainly options you have to present To your multi-path D if you're doing any sort of multi-path stuff to make it survive a reboot Or the timeouts won't be right like things will With it's less a big deal for true nas but things that are like a controller base where one one controller holds the Is primary for the lun and the second one is secondary and it has to go through both like you it It's not Is easy as just set it and forget it it'll work It will work sub optimally and it may fall over when one of the controllers goes down Yeah, yeah, once you get into the complicated m40 and 50 the dual controller systems three-part I've never tried it on synology. I think synology supports multi-path. Don't they? That's yeah Yeah, and alex is right there traditional sand problems I mean, almost all the vendors release documentation on uh, how to set up how to set up multi-path D in red hat 678 pick one right that closely matches the debian version or I guess xp and g is uh sentos based on the back Anyway, so Just go find the docs for your vendor for the version of red hat that the version of xp and g are using is upstream and follow Those docs and it'll work just fine Yeah, they mpio the confusing parts is You can't just have them on the same subnet. They got to have two subnets into your nas and then time together where that's true Yeah But it creates confusion because people were trying to force it onto the same one Yep, or they start trying to do leg In And not realizing how the hashing works and they're like, why am I losing one link? It's like because everything hashes to the same like hash value. Yep No, don't do that Because someone had Made an attempt and was broken. That's how the consulting call came in So I set it up and I was like, okay, make sure I understood how it worked got it working Then sorted out all the things they had done wrong. Yeah, that's I mean That's a very common dumb that I see people do when they're doing ice guzzies They their inclination is well, I'm just going to run this like any other networking where I do multiple links And I aggregate I'm using like leg b here lacp. You're one of the aggregation protocols Not realizing that I only have one data stream here. So it's not going to go any faster. It's only going to saturate one link Port forwarding will be a cool video with two routers. One is for dial-in. Uh, I know Come on. We could probably there's a poor master of the basement. We can need to dig out Yeah I that is kind of one of my I want to catch up on a few things that I have I've to finish the data center video I started editing it and I once again, I'm doing other things instead of When because the other big boy servers are leaving I have to finish anything I'm doing with those Are are amazingly 14 million passwords in a few seconds server I I want one but man those the video cards are the cheapest I could find them I'm like, maybe I'll just buy a couple of those the cheapest I could find them is like 3500 bucks a piece. Yeah Ah, that's a lot. Yeah The bigger problem is and maybe the chat here will know like can you just grab random, uh Like I don't think you can just put gpus in random servers because you don't have the power to drive them Yeah, they have to have the right, um, yeah All the power. Yeah, because of the connectors and things like that Uh, I mean that has eight. I mean Again, I think we're we Made that server cranky with the power draw on it Yeah, about what actually happened, but uh, I blew the circuit breaker I blew surge suppressor circuit breakers on multiple surge suppressors I started splitting the power up eventually blew the inwall circuit breaker Uh, and then had to take extension cords to get the half the server's power supplies to another Uh circuit breaker Yeah, it's It's needing all that connectivity that we had to make this thing work Oh, it must have gotten mentioned somewhere Yeah, yeah We I think me and you mentioned it. I think in the uh, uh, uh, yes, there is definitely a picture It Nothing was damaged. Uh, I would have fixed it if it was damaged Uh, a little bit of a magic eraser to the top of the case jesse because there was a little bit of mark on it But uh, nothing was harmed in the in my burn-in testing of that server Yeah Those Those video cards are expensive. That's a huge part of that. I mean, that's roughly a hundred thousand dollar server right around there Yep. Yeah, I think it was like 85 or something like that their cost Yeah Ah fun stuff Kiss some character Yeah, it's fine. It's uh It was it was my dumb mistake. Uh, uh, when I picked up the power cable to plug it in I'm like, man, this power cable is pretty thin and I didn't even like I didn't even think twice It was uh, some real burn-in testing and some Yeah, um Where do you source something like that from and you buy right from super micro? I got ingram Ingram, I mean, uh, super micro We sat with them to do the specs because getting validated servers that do that stuff is harder than getting servers Uh, there are a lot a lot of considerations. Like do you have enough pc i u lanes? All right, like do you have a proper ram config to actually feed data these things fast enough to have enough power? It's not just a matter of like buying a commodity off the shelf. I mean first of all You know your first challenge is this has eight eight six thousands in it. So you have to find a motherboard that has eight uh That has eight x16 pc i e slots on it start there There are very many of them Yeah, it just makes most of them It you don't arrive at all really quick on these. Um It's interesting how they vent and how they cool as well The fans have a pass through to each other. Yeah So it looks like they're too close to like wait a minute if the fans are here How does the next one get fed through this fan? Yeah It's it's designed to work this way. It does not overheat. It's crazy loud though Yeah, and as a matter of fact, uh, they put a space heater in the men's room I don't know if you noticed it say when you were down Uh, and the reason is is that it that heated the room it was in in the adjacent room where the thermostat is I'll put solid 15 degrees above the office ambient temperature. So like no heat ran for the rest of that zone So, okay, I was wondering why there was a space heater in there and that explains it. Yeah I don't know how much does that thing weigh A lot heavy. Yeah Epic cpus aren't cheap and consumer chips don't have enough lanes. Yes. That's that's absolutely one of the challenges Number of PCI lanes. Yes And not even not all the epic chips Uh, some of the newer intel ones also have enough lanes to do it But yeah, it's interesting Travis it was warm in the bathroom Yeah, so this client did not need uh, they didn't need NV link I asked because if you can see you can see the connector for it on top there and the towards the back It's just a connector that connects the two cards Yeah, all the cars they didn't need it. Uh, so the use case of this Is there were actually it's got proxmox on it. There will be eight VMs on it and each of the VMs will have a single GPU Passed through to it. I believe is the use case. Yep. So there is no cross card things going on at all Nope, it runs. We'll say some scientific data engineering To process. It's a very custom thing. This is not none of this is really off the shelf Like what they do or anything Yeah I and probably as expected for Uh, companies that can buy a hundred thousand dollar servers for their labs. Um, where they do engineering work in science So definitely interesting client, uh, the They already have um that 45 drive server. They've got another one coming and they're probably gonna buy some more They've The networking kit next I think because like these servers have 100 gig in them We and we got a I didn't have any 100 gig optics, but we definitely had some hundred or 40 gig DAC So this and the 45 drive box. We definitely connected to each other 40 gig But yeah, I mean these boxes have 10 and 100 gig in it. That's it Yeah, uh, the server lifts are cool. Um, I have videos of them at the data center So when I get the data center video, they have the really cool Uh hydraulic lifts for the servers and it's neat because they attach to the racks to rack them in and load They got these cool slides on them. Yeah Yep, I want to set up that xe Uh from patchbox. Have you seen that? No It's like the one thing of theirs that uh, although I did they they gave me a channel con a bunch of the uh, dev mounts Which are like, uh Rack screw alternatives. These are the patchbox people Yeah, yeah, they reached they recently reached out to me asked if I want some product that I haven't replied to them yet So I can say yes. Yeah, I mean the setup that xe is it it is cool because it's like it's not quite a server lift But it gives you a way that I would be comfortable doing it to you by myself That way Uh, basically goes in the rack and you can set the server on it to push it into the rack then you get Yeah, it's here This is the part that they always want people to promote. This doesn't make a lot of sense to me I don't understand that. Okay. It's not just me then. Oh, it's like I can make this pretty by sacrificing a you. It's like I don't want to do that Yeah, and I don't once the cables are done and nice Like it's a lab thing that might be fun. So apartment party wants to say yes because it's a fun lab thing I don't see it as a production thing Yeah Well It doesn't make sense some of the bigger vendors like uh, uh, cisco and the nexus stuff They have like some sort of like their cable bundles Or like eight go into them and you can unplug and plug in like groups of eight Right. So they there's already solutions to some of this stuff But this is cool for setting something on there. I I do like this. Yeah, that's cool And I think the uh, and I think the dev mounts are cool too, honestly Uh, yeah They're novel like I I hate cage nuts I have smashed my fingers and cut myself on cage nuts so many times that If you can use something like that, it's really cool But you know for a lot of use cases it probably just won't work because you know, it's a captive I'm not going to be able to mount like rack rails with them because they have to screw into it And the post won't work but for like switches. That's great Um, the ones I like the best and you may have noticed them that are in our lab in south gate Uh, what's that company makes the plastic ones? I love them. They were great. I rack studs rack studs. Yeah Man, we put like 200 pounds on one of them. We we couldn't believe how much weight we did some, uh sheer testing with them by We made a stupid video where we we brought a bunch of Lifting weights in and we're just setting them more and more on shelves and these shelves were You know teetering and it finally ripped them out But I was like we put more weight than you will I mean You just don't usually have servers that weigh that much and especially because you're like you said Usually even your heavy sysco stuff doesn't weigh that much for uh switches. So those are solid I've always hated the cage nuts man. I I got so many like times when I've had to mess with my finger over the years Yeah, yeah 10x the cost of the patch cables. That's Right. Yeah, that's the other side of it. I want to pay more and lose rack space I mean, they're cool. Like I If I don't know you buy one for the rack you show everybody Yeah And it's I think this might be fun to have when you're moving things around because it's my lab but don't But yeah me lanes leave the cables where they're at and yeah I'm testing different switches in my lab and things like that. Yeah, okay Like I'm I'm racking it. I'm putting it in because I want to demo this model switch Like it's for like literally demoing stuff, but not production Because I'm only going to do it once. Why would I spend this much money for a modular cassette? Yeah If it was yeah, but it costs Like I can't if it was one right, but now I have to buy a cassette for each one of them And it's pointing out there like 10 times as expensive as a patch cable. Yeah And that stuff should be static like look at this $570. Yeah, that's like, you know how cheap patch cables are Yeah, this is yeah comes out to be the cassettes $22 per cat per patch cable I don't get it. Oh, I almost just said something. I would have gotten me in trouble for sure. Oh Oh someone says can beam do proxmox pms. I don't It doesn't have proxmox support. Does it no I could probably make it work the same way I could make beam work, but the restore path isn't great Yeah, which is like, you know run the linux agent on the host use pre and post scripts to like, uh, basically Run the proper commands to issue the snapshots in right Then back up the directory with the snapshots in it and then, you know The restore that's how ironically enough that's how I'm backing up an oracle oda Or we were using Unitrends it was exactly that But it's not supported and it's not awesome Yeah, not not an ideal situation Oh Yeah, how to waste your it budget by a bunch of these silly cassettes So how do you go up backing up proxmox? So she's already xo Um proxmox has their proxmox backup server. I've never used it um It's not as if my understanding it's still agent-based. It's not as integrated as their Uh, it doesn't integrate with proxmox ve it's like a separate product It doesn't integrate the same way xcp and g does Yeah Which is weird to me because you think if the same company's making it, but it's by the way proxmox makes an email server It's another one of their products has nothing to do with proxmox ve. It's just another product they make That's interesting Yeah, if you go to proxmox these are like separate products made by the same company they're kind of interesting so You have their virtual environment ve you have their backup server This is a separate standalone backup server. It doesn't have anything to do with or even need this to run And then you have proxmox mail gateway They're all separate pieces of software by the same company called proxmox kind of strange Oh, they're playing barracuda Yeah, let's just yeah mail gateways open source email security solution protects your mail server against that Like I said, it's kind of strange. It looks like here's a mail gateway product Here's a backup server And here's a proxmox and these are all set completely separate things I've never looked at the pricing on it over a hundred a year complete feature set 15 tickets a year community edition that only features there we go Spam and virus easy deployment dns s&p gray listing block you know, I mean I don't know inbound email I don't know if it's going to be that much better. I feel The companies are going to be better at it or still like we use uh mimecast or one of their competitors That's just going to be proof point proof point Yeah Because it really comes down to is it good enough just pulling these real-time block lists from spam house and all the different places Yes, that is actually probably about 70 of the blocks that we get are from those lists Yeah But the more complicated ones what you gave a percentage before just how much of our percentage of email doesn't even make it to us through Somewhere in 40 the 40 to 60 range never even gets to like running data. So you never even get a subject Or any better information it gets rejected Uh before you even start the envelope yeah, uh Vincent backup and restore performs backups or stormage for xcp and g and proxmax never heard of vincen before Cable assembly. Maybe that's it I've spent this whole time looking for this like cable So at the track phone data center down in miami Their switches they run in the nexus 9000 series big sisco iron and they're using the fabric extenders Which all connect via you know 100 gig back to the main thing But they had some sort of weird thing where they had one button they could push and unplug groups of eight ethernet cables Right, so I I thought Hubble made it I don't know who made it but it was awesome because it's like you got a 96 port blade here Right a two u blade that's got or two chassis blade. It's got 96 ports on it But it's not 96 ports. It's like 16 groups of uh eight Huh, oh and you're looking for the device that would just let you do that Okay I did see someone clever They 3d printed it, but it's kind of neat when you have to swap out a switch But you want all the ports that go in exactly the same locations Uh, they 3d printed a block that you would just move all of them to so you can put them all back where they were That's an idea Uh with broad chemical As you're going to back it all up a analogy that I'm currently Doing with vmware. No, you're not going to this is another thing. It's going to break. Um Synology doesn't have a native integration for backing up proxmox or v or um xcp and g either Now you can use with xcp and g because of their integrated backups You can use Synology as a storage target to store your backup data But you don't there's not like a native tool in Synology that does it Uh vmware has been around so long and they're so big Lots of people have native tools and that's what makes it a shame that broadcoms kind of Rattle under cage and breaking things because You it's it's like this chain reaction. It's all the ripple effects of I have this and it's integrated But that'll break if I move away from vmware But that's what they're hoping is you'll go well, I don't want to deal with the breakage How much more is it? Let me bend over further How how bad is it going to be? I don't know double the license price. Well It's it's this expensive to replace all the things that depend on it and All my staff is trained and knows where all the buttons are in vmware. Do I want to deal with that or I'll just It's cheaper than changing. That's what that's how what they're gonna bank on Yes, and that will work for a while until people stop learning vmware And then it will get expensive because the number of people that know how to support it. Well, we'll go down Uh and but but that ceo of broadcom that's there right now. He'll be long gone by the time that happens and that's why he doesn't care Yep, no, there's all these different consolidations with these large large companies Um, they're all in it for the short-term gain. It's like He's he's going to be rewarded based on the short-term gain of what he does during his tenure and he can bounce and then he's got this cool pedigree of You know all these accolades of oh while I was ahead of broadcom I increased revenues by blah blah blah blah blah blah blah. Isn't that place a Deeming pile of crap now like two years after you left. Oh, that's not my problem. That was the next guy's problem While I was there. We were making money. I left and I thought I must have screwed it all up It's just it's so weird to think of how that works And it aggravates me and that's why I don't think too much about corporate business companies I unfortunately have to deal with the results. I will rally against their business practices Um, at the end of the day, I won't spend too much time noodling around on it because just it's nothing good's gonna happen Pan do it quick net plug. Let's look that up Yeah That could be it Pan do it makes a lot of nice stuff That's it. It's discontinued no That wasn't quite it It was similar to that Oh, maybe it was it. Okay. Yes. Yes. That's it I keep seeing ones that are fiber. Oh wait. Yeah, they've got ones that aren't Yeah, that was it. It looks like they just continued it. Oh It makes a fiber one too. Yeah That thing was awesome. Oh I know where to look for it flea bay I'm sure you can always find all the weird stuff There I found a picture of it. Oh, did you? Yeah I keep finding the fiber ones. They seem to be more popular All right, I'll send you the url here I make your civil customers from vmware to xcp and g Some vms are backed up with agent for vm rest is Integrated back with xcp and g. Yeah So that one did six But you could get it with that connector on the left on both sides That's why is it? Potato picture here in hands It doesn't enhance it doesn't get they disable right click so you can't make it bigger Control plus will make it bigger Hold on let me just see if I can find a better image by searching the part number. Yeah On someone with a better site Oh like pan do it themselves So you could plug that left side into a switch and then you could plug and unplug that whole grouping there at once That's cool And you could get that configured on both ends of it, right? So like it could go to your patch panel and then I mean I don't need I need that I don't need patchbox. I need that thing That is cool I like this pan do it qbe something something It's pan do it themselves has them. No No, some of it's I haven't figured out if it's all discontinued about some amount of it's discontinued They still have the product page here. Oh, yeah the product. Okay. I found them Yeah, they still sell them. They still sell that one anymore That is cool Isn't it I like this Now I know I want to patch my new my box Fine if I'm one of these in ebay Yeah, I mean they're all so expensive, but like so we did some cabling for the local hospital system. We were doing some uh basically Installation and integration work for them when they built a new office building and I think we installed I didn't do any of them, but I think matt matt actually a lot of them For 500 switches across the various things and then had to patch all the things down And they were buying their patch cables in a box that reminded me of like a normal cable box In that the cables just had little connectors between them So you would just pull a cable 10 foot cable out of this instead of being individually bag They were just in a cable box you pull it out you twist it off and then plug it in and use it I think they were called quick patch huh These are cool though all different part numbers five feet links Huh easy patch. That's it easy patch or tonics easy patch And I'm like why would somebody want Like was company called La Grande they're like two or three times as expensive as normal patch cables are and I'm like Why would somebody spend that amount of money on this like fancy boxy patch cables? The reason is you don't have to Undo terror plastic bag and undo two twist ties in every one of them Okay, yeah They're all kinds of what is the the box they had there Yeah, there you go. You found it. Yeah So you could get them in colors and then cat six six a and then you know sizes And you know, there's 40 or 50 of them in each box Yep I get it Yeah, so do I Like huh the things Yeah, it's weird as you can't get those easier that way But like you said when you got like 400 things to patch it's time is money Yeah, well, that's I mean that's what I'm like, okay So instead of being like a two dollar mono price patch cable. This is like a four dollar patch cable or five dollar patch cable But if it takes, uh, you know, $80,000 your resource 90 seconds to pull all the stuff out of it like it may not really be that much cheaper Or Yeah, uh, what do you thoughts on a 45 drives homeland stuff are I think the hl 15 is awesome. I have one. Um, it's working great. I haven't had any problems Well, I can't say I have zero problems with it, but they're not they have nothing to do with 45 drives one of the There's uh, uh, it's a super micro board and the board. Um, it has no link lights on it, which is weird But that's not has nothing to do with 45 drives. Just tell me a new board Um, I well, it's on its way. It hasn't it hasn't arrived yet But it's not a dig it matter of fact if anything it's a how good the support is the 45 drives as I told them I like we'll just ship you another board. I'm like, great Oh, yeah, we've had no issues at all with to my knowledge with 45 drives support support like we've had We've had some bad drives Uh, anytime you buy 60 We've I think with the exception of the most recent one We've always had one or two bad ones Because that's the failure curve of hard drives like it's really tall And then it's very small for a long time Yep And if you look at the back plays reports, they'll say the same thing like The instance of them failing within the first 10 hours is high and then it's low and then it goes back up again Uh, but we had a backplane bad too and they sent parts out. They don't question They don't even want the parts back in many cases. No, they just want to get it fixed Um, and we've now bought Uh, one or do we buy two? Stornado's or one just one. Okay We got to do an updated review. I did a review before uh, before it got installed But you know, I like to follow up reviews because reviews are one thing reviews after out in the field that's where we want to follow up on them and Your assessment of that tornado is it's really fast. It's really fast I don't know what else to say about it Yeah It just works they they make I think they've done such a good job of taking more commoditized hardware and building a machine that's affordable I mean you think about that petabyte server the matter of fact the one i'm logged into right now um It's this server is right around the 50 60 000 mark for a usable storage volume here Yeah, I think that's probably about right Yeah, I you can price it out right on your site like I went and priced it out It's right about 50 000 and you're talking about 726 terabytes of fast storage Not fast in quotes. I mean like you were actually able to push it pretty quick, right? Yeah, um, what did I post because this is I posted our slack. Yeah, you were getting like a couple gigabits per second That was not bytes. I presume right. Yeah It'll actually be somewhere in that data, but yeah, it's it's wild is how fast it is And I think that is actually a uh, uh, I think we put 128 gigs in that one. So it's a little more Uh, and it has 100 gig nick in it. Yeah That one will be the real problem because it's got the 100 gig nick and then everything else is on board So the on boards are only gig Uh, the on boards are 10 Uh, no, this says the on boards are the there's an included dual port nick on board Normally they ship with 10s out of the box, but we upgraded them to 100s So I don't know if that's an addition to the 10 board. They normally put in it. Let's all let's look. I can I have it pulled up Uh, does it tell me what you know, you know zero and you know one are going to be 10 gig or one gig almost certainly Yeah, because these are the 10 ones No, those are hundreds I'm sorry hundreds But can you pop a shell on it? Yeah, hold on. I'm just I thought you could choose the link speed on here. I guess they don't have that option inside of this I have config dash a Oh, look kubernetes bullshit There we go No, there's no I'm trying to remember what the equivalent command to e-tool is on uh, previous d Oh, this is linux Oh, it's oh, yeah, then just run e-tool Yeah, assuming that's installed Let me Uh, and then you got to give it a dev so just give it slash dev slash, you know, or just you know, give it a you know two No, it's 10. Yeah, I thought they had on board 10. No, is that 10? Oh, you know that nope. You're right. That's not gig. That's gig. It's one gig. Okay, so it's got a pair of gigs And then a pair of hundred gigs Yeah, they need a hundred gig to get this thing going uh, so I'm gonna pop that guy avaya switches are fine. They're the cli and it's kind of wonky. They will move data if that's what your goal is Uh, no one is installing them because I'm pretty sure they got bought by extreme He has that who purchased them they're old Those are probably actually honestly old, uh, uh bay network switches or nortel Yeah, but I thought somebody bought the switch line out of it from avaya maybe not Nortel I haven't heard that word in a while Oh, yeah, bay networks nortel bought by networks and avaya bought nortel Yep Ah, man the memory lane of switches Price out power usage. Yeah, those old those old servers don't uh, they don't or old switches don't sit power That's for sure. I have I have a Lenovo switch I was going to do some testing with um, because they were they were cheap at the time, but then the pandemic but I got it from the guys at tech supply direct. They sent it over and I was going to review it because there was like no switches in stock during the beginning of the pandemic But it seemed like kind of a lack of interest in it. I don't know there wasn't enough demand They sold enough of them, but Because they just had a big surplus of these poe Lenovo switches. They're loud as hell though That's one reason I didn't like it like it just yeah, it only has one mode of operation And honestly, they're no different than any of the China Minium ones you're gonna get it's just some reference platform. They're pops like cumulus or some other crappy os on and that's what they're running their web UI was bad and slow and yeah, it's all it's a reference platform Yeah exactly Lenovo made a switch. What kind of Lenovo clearance the whole lot of switches at tech supply direct care In HP said to the same thing But at least HP was honest about the fact that they didn't make most of them They're just relabeled so like you could buy fiber channel switches from HP and the model number ended in b or e And if it was ended in b was a brocade and if it ended in e was nemox Yep Well, welcome to del's world as well. They did much the same thing lots of just rebranding It looks like they still sell there's still a switch available on their site Lenovo. Yeah, really? I'm not I've not seen them in the wild No, me either Someone bought a bunch of them someone some Lenovo rep made is Quota No, this is not exactly correct all the large pc builders don't make switches. They all brand switches as their own They'll stick your label. Let's be real. How many of them actually make computers. I think foxconn makes a lot of computers I think foxconn makes a whole lot of computers Dell release notes for pat switches are random. Yeah Yeah, I actually don't see there's still pages on their site on Lenovo site for switches But I don't actually see that you can get to them off the main site anymore Ah, so it's probably all discontinued Yeah Foxconn all the things I think foxconn makes the apple things too Yep Oh fun. Well, I think you want to wind it down unless you know don't forever. It's been two hours Yeah, and she's waiting to put the kids to bed. She's like, I want to say good night to the kids Yeah, and you got to do how many miles you got to run tonight three You can do it outside or inside. Huh? No Uh, I Will typically run down to about 37 degrees outside. Uh, it was there was ice on the way to the car So definitely will not be running outside. It's like 27 out. I think yeah It was it was nice for a minute out like when the sun was out when I just got back to my house But I'm like, yeah, I'm the same way. I like hiking. I don't mind hiking, but I won't I won't do anything too active I don't like that cold air hitting me just bothers my lungs too Yeah, it's I'm getting old. It's more of I'm afraid of stepping like the last thing I want to do while running is find ice Yeah, entry that would suck 2 a.m. Where where does that put jimmy the it guy, uh, uk Yes UK area 9 plus 4 is 13. Yeah, we're gmt plus 5 right now. So somewhere around gmt Yep gmt minus 5 I guess so yeah, I want to dive into uh, kaspersky. I want to do some more reading. Uh, They're they just did another dump of research on the Uh triangulation attack from iphone. It's really interesting uh, there's apparently kind of sort of a back door and iphone and What they found Is really interesting because the back door that was used in this exploit No one could have reverse engineered Someone had a map. How did that map get out? That's the question because it's a map of where all the uh All the hardware is mapped to io memory devices and I get it's it's really clever the apple engineered it that exists For sure. Oh, yeah, I mean apple built it. Yeah, but the weird part is apple built it and apple's latest patch Just what everyone's confused They removed it seren's like why was it in there for so long? Was it for testing reasons? Why would you leave this enabled because you obviously were able to remove it when kaspersky pointed out the problem with it and that people have maps to it so you fixed it but It kind of leads to the question of why did you leave it enabled for so long if it wasn't necessary If it didn't have a breaking thing by breaking this, uh, Maybe they didn't know Well, someone said that as a suggestion But I think the count it is how this is very well written each one of the ioports requires a specific hash before it will accept the command so they took the time to guard it Oh So it was built purposefully They probably had to reengine it's probably an appendix Yeah, like a leftover organ from an earlier time when they required it They've re-architected the things that no longer need it anymore, but no one but the software developer that wrote the new thing Yep mark the old thing for deprecation And no one never removed it right because tech debt and then until it becomes a problem And then it's like oh we probably audit like we probably ought to release a patch to fix that one last thing That's to realize on this thing and then get rid of it Yeah, it's really interesting. I mean because you've worked in some of the enterprise software dev how That stuff can get overlooked. It's not necessarily nefarious It's just one of those Yeah, it's like it's the old appendix problem. Like we don't we don't know we need it, but we still keep drawing one Yeah Oh, they changed the hash each well the hash okay, that's also potentially explainable because the changing the hash every generation could have just been a build uh build output products like it could just be in the build pipeline that could go Again, it's there could be one there could have been one subsystem that was still relying on this old crutch They had and so it was still part of the build pipeline and then the security researchers are like, ah, we we okay Now we need to get rid of it They reengineer the one or two things using it and then they finally remove it Yep It just because just because there's an automatic thing that happens every version It doesn't necessarily mean a human's involved. They're almost certainly not at that scale. It's almost certainly a build Uh pipeline Yeah, yeah Yeah, it's a uh, Steve Gibson did that's what got me intrigued. I listed his I wanted source somebody link source because This uh, if you have a video, it's on link it to me because I need something that's about 36 minutes long to watch on the treadmill Oh, yeah, I'll send you this one and skip. I'll tell you where to skip to in it because it's like a long podcast But if you listen to him at 2x, he sounds like a normal person. Okay, if you listen at 1x, you think he had aneurysm Yeah, no, that sounds great. So I'll I'll send that to you All right, we're gonna bounce everyone. Uh, if I was talking about the latest security now episode with Steve Gibson So if anyone else wants to join in listening, what that's what we're gonna do after I've already listened to a petition to listen to all He's running. Thanks everyone for joining. Thanks for having us. I'll time index this uh after my new ai tools have been doing a great job of Taking a two and a half hour live stream and making sense of it With chat it does great chapters It'll have like where we talked about ice guzzy and everything and I wonder how many times it's gonna count that we said don't route your storage I need a shirt that says that We'll just reference the shirt every time like hey, don't route your storage Somebody said you needed don't write your storage shirt. So oh, I'll put that on my to-do list. Yeah. All right later folks. Yeah later