 Welcome to vlog. There's a number 354 and technically because I seen a comment of me being live It twice in 24 hours, but it's actually been three times. I was live this morning with Ninja one their IT horror fest you can watch it on replay and That was live too, so I've done that live stream. I did some meetings. I got some videos done and I'm pulling my hair back Because it's I went motorcycling for a bit. So my hair is kind of all over the place People have so many comments if I let my hair down or when I don't let my hair down that is Odd to me. I don't know what it has to do with all the tech videos I do but the commentary it amuses me at least and so far no one's been too creepy but Plenty to talk about and no, I'm not really gonna go diving back into the PF sense one That's why I have a separate video for that, but I will be doing a upcoming video on the PF sense topic is I Have a feeling based on the backlash that They're you they may fix things or they may fix it. I don't know. Well, it's a play about your I don't know You can't predict the future You can infer something might happen. You can hope something might happen. So I I'm actually gonna be talking to one of Other people tomorrow and yeah, we'll just see all that lands and maybe we can solve that But that is not a topic. I felt like dwelling on again because I spent so much time on it yesterday Nonetheless, I did get to go to a data center and that is a really interesting thing I Can show you what can I show you technically? I got to get a lot of things approved things I have approved are gonna be like this photo here Share the screen But there is a ton of Issues With filming there and they let me film there, which was really cool. So it's just yeah a lot of fun What's the breakdown for yesterday? You just have to watch that video from yesterday basically PF sense made some license changes and If you're running Nicky hardware, no big deal if you're not running neck your hair where you're running PSS plus You don't have to do anything But they remove the free version. That's all Let me see what other photos I'm able to show from the data center that are approved Not much else at the moment because I told them now. This is not a sponsored video with this data center this is completely me going there just to talk about it and and Discuss it in a fun way. This is actually I actually asked permission from the security people after I had they Do I was there because I had to Essentially go through a check-in process get my ID get my badge and I said hey I'm obviously they're like what's in the bag. I'm like, it's all camera gear So I took a picture. This is in the lobby as soon as you walk in no photography use of cameras This is what I talk about a lot of the data centers This is the first thing they say no, you cannot do this. This is like as soon as you come in there They're reminding you nothing here without the express written consent of the building management and of course, that's what we had So it's gonna be a fun video to dive into. I'm hoping next week to have the video finished to the tour there's actually probably gonna be more than one video and Because there's a lot we talked about we ended up talking Way more than expected on things I thought I'd be in there in and out and it turned out to be a Lot more end up spending like a couple hours there It was a lot of fun. Now the fun things are going to be talking about how they do power delivery How they do and I can show you from the outside. Actually, this is a picture. I can't show this is an outside picture How they store in these tanks the water how the cooling works There's just topic after topic that's gonna be kind of interesting about the way They think about energy and efficiency in these data centers the way they operate it was just so many good discussions that we had that are Really? Maybe you didn't think about it and you wouldn't think about it unless your data center We've gotten into topics a little beyond and this isn't gonna be really part of it But I thought it was interesting. They were talking about how some data centers build themselves next to places that they can sell The waste heat too so as they pull all the heat out It's kind of neat because they actually sell it to adjacent companies that need extra thermals It's just all this interesting planning that goes back and forth into building a data center how you think about the energy management I think they get they did release and let me see is it's on I think this is on Twitter Let me see if I can the they posted some pictures. Oh, here we go. I can share this one here and And Yeah, they also had their photographer taking a couple pictures. So they released a couple. I was like, yeah, absolutely Let's talk about this because we were having a really good Discussion on energy and inside of these blue This is the part I want to talk about these are flywheels inside of these blue boxes. These are We had a really in-depth discussion about this. This is the picture. I posted where I'm looking down I mean, this is I believe they call it a tier 3 plus data center. So this is a big place and This is something that you have to feel an experience where I'm standing there I'm in awe of just the volume of air moving by me But it also because a lot of these servers are running Those Nvidia cards and this is something this data center is able to do This is deft by the way the data center that I'm at they can handle these massively high wattage servers that just crank out heat at an incredible level there's four power supplies in these and They're just pushing heat. It's like standing by a blow dryer It's just pushing at you, but they're able to keep the air moving fast enough because of the volume of air So there's all I'm gonna make sure I have all the numbers and stats and we were miked up So that's where you're you're actually able to hear audio and everything from this one I put it together. I was actually shocked the audio came through as good as it did It's not gonna require as much fixing as I thought so I'm gonna be a fun for sure talking about it I'm really looking forward to it. I know a lot of you probably are gonna find it interesting One of the things I was gonna comment on and I got to scroll back to it so the servers that you were using are the same servers that Well same one that we have here That'll be doing a video on shortly That we're building this for it. Well, it's built. We have this for a client This has four power supplies in it and then what do we have in this? There's a one two three four there's eight of those Are the a66 hundreds? Yeah, it's just wild We blew so we can start this server up But when we were running hash cat on it Jason was running hash cat and it blew the breaker We can ramp the server up so much wattage that it started blowing breakers So we had to distribute how the power supplies were plugged in just so it wouldn't blow breakers in our lab You know, I can't remember how much kilowatts per rack But it's in there like I don't remember it But it's that was things we talked about they can run more kilowatts per rack than other places can that was in It's a conversion of if you have this many kilowatts of energy going into the servers It comes back out as heat a a pretty good percentage of it So they limit some of the racks based on how much cooling this is just a general rule They have and this particular data center is able to do some really high-end things these servers were common there and Like rows and rows of them and when you start talking about these things fully loaded up can be a hundred fifty thousand dollars each And there's just stacks of them and that's kind of cool to think about and I believe each of these uses 3000 watts each something like that. So yeah power delivery is wild Less about cooling and more about how many times you can change the air in the room Yes, and they that's one of the things that was really interesting is how they do air movement is Not with a normal raised floor. They have a really tall raised floor so they can actually move air at that volume So that's what made it kind of interesting. It was really cool kind of amazing how much power those and videos use absolutely Many data series are now data because Watson cooling per rack has jumped a ton. Yeah, this is where this data center is Absolutely not dated if you look back at the pictures. They they shared. I mean, I'm standing in the cooling room here There were the chillers and stuff are The rules were I could take pictures of it from a large-scale infrastructure I was not allowed to show you like individual pressure sensors So that's why all my angles are kind of wide angle like this But it was fascinating because they're able to absolutely move and cool it and they have Contingencies for everything that was a lot of what we talked about and these are stuff that they can talk about As all the levels of contingency of what if this happens? What if this happens? What if municipal water isn't available? well They have a way of dealing with that too So that's gonna be it's gonna be just a fun discussion. I think of how autonomous this data center can be with outside forces It's also kind of an interesting challenge. It's happened years ago Well, it's happened more recently than that, but it just doesn't make the news. There was some minor fires at data centers I know Because one of my friends was affected by some type of outage I don't remember exactly it was a fire But what happens is and they were discussing how they break everything out into different segments So each part of the data center can operate independent of the other one that way when fire marshals have an incident at one Part of the data center they may want to lock out the data center That won't shut this place down because they have everything on like two Everything's doubled and it's doubled on each end of the building So one cooling system from one side of the building can service the whole building one cooling system from the other side Can service the whole building and if you divided them somehow in between they can still keep functioning There's just so redundancy after redundancy after redundancy that they had put in place in here and I thought that was just it's Complicated in some ways It's just really budget of being able to build a place over time that can do what this place can do I think to the point where useful waters are available for long enough. We need contingency plans power would amount out Long to zombies banging at the doors. No water main breaks are common Maybe not where you're X. I think you're on the west coast But here a water main break that shuts down water for some period of time Maybe 12 to 24 hours because of winter. That's the thing that happens here if you're in the colder states This was in Chicago. So it's the same weather we get essentially here in Detroit, which means yeah frozen water main breaks That's the thing. So they have contingencies to deal with that Which is great because I mean you don't really want the data center going down because of that You know one of the things I wanted to talk about today, too it's just open source in general and This is kind of a challenge if you make a product so good that people use it, but then never wants to Support it like they don't want to donate money to it And this is a little bit touching on pfSense, but they're not the only project with this problem Xtp and G is one of those as well is how do you get people to? Spend money with you so to speak by support contracts when I don't use them It was a it's a topic that comes up a lot for many projects in the open source world and I think it's gonna take some rethinking of these companies because One of the challenges like if I buy a support contract and that challenge is huh I didn't use support for the last two years The product works great all the updates didn't break anything It works exactly as expected like I clicked update and updated and in all the moving parts of this work great so now what and That is a challenge. I think some of the open source companies that are really getting good are having I I think Oliver's got a blog post Oliver Lambert over at Xtp and G I believe he posted his blog something along those lines of just some of the challenges that can come with it of Hey, we would like to give you the entire product for free for to use We'd like to support development of the product, but when people are just happy they Don't tend to buy support Steve from security now Steve Gibson talked about this the software that he uses to build Spinrite the Operating system basically the guy ran out of bugs to fix so no one paid for updates because there weren't any more bug fixes and the Company closed because there was nothing more to add to the operating system It did what it needed to do and now they're not doing it anymore now when there's always innovation that's great And I look at Xtp and G as an easy example of this They're constantly innovating But if you're not buying a license for it that innovation still has to come from somewhere You know, there's still writing code and if businesses and this is something we've run into because we do consulting We really encourage these businesses that you should buy a license like Bob, but it works so good. I'm like, yeah But at some point You got to support the developers to do it I think more people need to get a little bit more conscious of that and it comes back of course a little bit to the Pfsense controversy Well, and I said this before love or hate neck gate the reality is and this is just something That there's not an easy solution for me pulled up in my forums Share this tab This is where you can find out actually a drop a link to this for anyone who just wants to see it This is a this isn't my forum post. This is all the licensing stuff and When you look at free BSD who puts the code together for this and its neck gate is contributing greatly to Specifically the things related to firewalls Netflix is up there for the don't for the contributions as well And this is just a challenge of open source if you have these people juniper's doing in what is juniper at 6% of The commits came from juniper 8% came from neck gate Clara systems who does a lot of storage servers pull them up to open source development Reimagined and you know, these are companies that are contributing back So open ZFS and free BSC support they commit things not necessarily for firewalls, but for storage For any one of these this is the important part is how do all these code commits create the product? And this is just a weird dynamic that Happens where they're like oh Pfsense is the problem and neck gates the problem and I'm not saying they're perfect I'm not excusing stupid behavior like changing a license over Charlie Makes us hate that's a bone-headed move to do it the way they do it, but somehow we need to figure out a way to keep the products going because the one of the challenges is the people from and people love to mention open sense open sense is using and pulling all the Packages that BSD is contributing back. They use the same PF filter So when neck gate updates and enhances all the PF filter features that comes downstream So people like open sense can build on it open sense is not listed here as pd part of the contributions back They're not forming the same level of contribution back there That means that you know everyone switching to another system But never supporting the people who are writing a code these things die and this is how software will start to Trickle down matter of fact This is the problem overall with BSD is a lot of things a lot of things are built on BSD But with a lack of contribution back and a lack of commercial support from big companies BSD is shrinking from what it was versus Linux. So it's just constantly growing constantly moving So this is where it just becomes something I want people to be conscious of but there's not an easy Snap your fingers answer on this to keep these things going the foundation does a lot and I would say BSD is far from dead This is matter of fact, this is the source for all this data. So The this is linked in here they talk about projects this How to get involved they talk about grants they have they're not That small there's still a lot going on here in the free BSD world, but Like anything it has to continue. It can't shrink further. It's one of the challenges in there Okay to pay for feature updates not okay to pay for bug fixes. I don't agree with that and here's the reason why What do you define as a bug fix? Did I write bad code or did someone figure out that the code itself is Vulnerable because of a certain attack and someone has to fix it or what if an underpinned library such as open SSL Needs to be updated. Is that a bug fix and And who's gonna pay for that someone has to re-spin and recompile it to fix that Free BSD and specifically pfSense was not affected by this had a problem that someone found in ping Okay, that means someone has to go update all the things that were pulling those sources down and rebuild them now pfSense wasn't affected because they use D ping her without ping and this is You know, there's a lot of carefulness that goes into the way neck gate builds pfSense Which is one of the reasons I like it I'm less worried about a supply chain problem because they're very careful about only putting in what needs to be Put into the firewall and not anything else. They reduce the threat service this way. They're very conscious of it It's one of the things that matters a lot. So I don't agree with not paying for any of the bug fixes some point You know, there's bugs. You can't write perfect software That's really hard and the developers continuing on to do it now if they make an egregiously stupid bug That's maybe different. Maybe I shouldn't pay for egregiously stupid bugs But what point do you declare something egregiously stupid versus just a bug? Age-old problem think about selling razors or light bulbs with intentional short life Um, we don't seem more true nasty base commits And I'm kind of curious about that as well because I thought I X systems was higher up there on the Lists of people who donated to BSD Maybe they've changed because this is this is from this year and maybe I X systems is really doubling down and focusing on Really focusing on the side of the Linux I'm sorry, why was I drawing like they're the Linux true nasty scale side the other thing that Kind of makes me wonder and I don't know this and I'm gonna ask someone because I know people that Do some of these things is if I X systems isn't doing it directly Do they partner with Clara systems because Clara systems and I X systems have a you know common interest in ZFS features on? Free BSD that's their their common circle on there. So yeah, I'm curious about that The unity stuff was just short-sighted and stupid finding Funding through government infrastructure budget as a utility Yeah That's a whole different That goes off the rails in terms of that's not likely to happen Now I do believe that what I will say about in terms of government funding I've always been a big advocate and this is not popular here in the United States but I think there's a few European countries that believe this where it's the Government money should fund open source code because the government needs open source Why because there's a lot of commonalities like if you look at cities for example, they all need some of the same management tools Why don't we build it once open source and we have you know government funds funding open source development of the tools the government uses? I've always been a big advocate of that I don't think it's really that I'm aware of I've not seen it happening here in the US But hey, hopefully one day it will Is the graph BST in general specifically free BST wondering the contributions Graph would be similar to open BST and that BC. I don't know I'm certainly not a BSD expert. You don't like the true Naskoo he feels very limiting I That's yeah, maybe I don't know I'm fine with it It's easier than managing ZFS from the command line, but I have some future videos Are we doing on the Houston OS? I really and I've done videos before on Houston Houston's ZFS management is great if you're looking for a good way to manage a Linux server running ZFS Houston is definitely the way for that I feel the difference between a bug and an egregious is the failure to identify and act on a flaw. Yeah, it's splitting hairs though Governments use more open source. Yes something else worth noting I seen someone asked this question and Yes, make or tick is an option for firewalls, but man, I don't know that it's a good one My grotesque interface is not great. You their documentation also less than great and I've found it a little bit quirky now that being said there are people who have an expertise in it And there's also people who are talking about BIOS and a few others You know these all come up But one of the challenges with a lot of these is the learning curve and documentation This is one of the reasons PF sense is so popular. There's a lot of documentation There's a lot of people who have write-ups not just neck eight tons of write-ups You could find for how to get something done in PF sense. It's has a massive community Now if PF sense or neck eight more specifically keep screwing up Yeah, they're gonna break that community and it takes a community of these testers that bring forth And this is one of the reasons IX systems really gets this really well They realized by keep giving you TrueNAS for free They have the biggest testing base of any storage software platform on the planet end of story There's not another storage platform that goes through as much testing as the TrueNAS one Because so many people can use it for free people have discussions and forums. It gives feedback to the developers PF sense is the same way. It's an extremely popular firewall Very well tested from that aspect of it as long as they keep the community going. They will have testers They just got to not screw up somehow that's become a little bit harder than it should be but And open sense is not bad, but there's some challenges open sense and I gotta dive into it I am probably gonna do at least one video on open sense I'll take the time to really play with it just to explain a few things to people But I don't like the updates of it like I ran it for a little while and I was like man This thing updates way too much and it feels like you're doing the testing and as I understand Unless you buy their business version you are doing the testing which that's fine. How much is open sense? Anyways, I should probably pull it up Mikrotik licensing is rough. I I thought you didn't need licenses with the Mikrotik ones Maybe I'm wrong. I Just know any time I've used looked at or tried to use rato at rato s. I'm not thrilled with it so where is the Throw this up here. So we're talking about Downloads sign up for ET pro telemetry Where's the buy or is the official shop? So here's their systems So you can get the software so it's Open business edition and they charge Set euros Eros to dollar I don't know the exchange rate. So this is It doesn't I'm assuming this is three hundred and fifty nine euros Which should be about three hundred seventy nine US dollars am I reading it right? Yeah, open sense funds their systems with hardware the same way PF sense does there's there there's a lot of parallels to how their businesses work That's their funding model is selling hardware and selling licenses because Commercial firmware repository offering selective upgrade path access to the OBA free access like professional plugins such as opn central Some of their documentation I guess is going to be like their e-book So yeah, it's just roughly with the conversion about four hundred dollars. It looks like if I'm understanding this Pricing. Oh never mind. I'm an idiot. I Three hundred ninety seven dollars. Oh, just like I said about four hundred dollars. There's a US dollar mark here Now they have do they have an addition version here? AWS software licenses So they're based out of the Netherlands. Yeah, I mean, this is how they fund it is their Selling support selling these here's you can buy your support licenses Four hours of support $600 so roughly two hundred dollars an hour I don't know. Are they a twenty four seven support? I don't know what their support options are Services branded version annual subscription and here's their security appliances Orange is cool. Yeah, like I said open sense is a little bit different Dollars are a bit more than euros It's kind of funny everyone threatening to go open sense that is forked from the CE version It was forked a long time ago. So they're really different now than they used to be Just hired my first employee for my MSP Wanted to say thank you for Brett and Jason for the BT videos. That's awesome. Congratulations on that But open sense is three years of support Let's go back over to the page here No, this is just a three-year license not support So there's no support in this by the way This is just the license to run it in a business their business edition is $400 to be able to use it for three years Not this that's not support supports different they sell support separately Well, you said I may sit down and like put a video together kind of comparing there is a difference in a way They that this is sold compared to PF sets it similar because they sell hardware And I guess that's a question to this the hardware come with Licenses you get free one year. Okay, so you get one year free if you Get it if you get their hardware What's the difference in pmsense CE and pmsense plus The so the pfs I have a video where I break down the difference because for some reason that gate doesn't have that The biggest thing is going to be boot environments. So boot environments open VPN data data channel offload and IP sec phone profile export and an Amazon AWS wizard to connect you So there's not a dramatic difference between the pmsense plus. This is why going back to CE I did the video today to show people how easy it is to go back and I lost boot environments by doing it It's not the end of the world The hardware does look nice, you know, I'm curious because I don't think they manufacture the hardware if someone knows where Exactly these come from Like who's the motherboard manufacturer on these because I'm willing to bet they're just stickered, you know They're not actually going through the manufacturing on this Well, let's see here. I'm trying to figure out what processor CPU cores Power requirements boy, they're really vague on this. Why does it tell me CPU cores, but not the CPU? You know, I you'd think I'd For a 19 inch rec mod I'd be able to find out what processors in it. I Mean, we know that it's got four cores Maybe someone and why is it still, you know, it's DDR3 not DDR4 That tells me it's an older model But there's not much else in here about it. So and what does this cost? Hold on? It was a price at the bottom buy $800 for this little box and these are all rendered. This isn't what the box looks like What does the box actually look like these are rendered pictures of it? Does anyone have one here? I know or never mind why I'm asking all of you. I have the internet and if I type in OPN Bet someone has one of those I'm on eBay. Well, oddly. No, baby No, let's try putting in the exact model number here. Nope But if we did this images So there is someone here on YouTube who seems to have one, but you are the internet. Yes, all of you are But yes, someone at least does have a video. It looks like they have on one or work maybe Benchmarks details, but not There's not a lot you can find about these Great. He seems to be the only person who's done a video on here and this is from a year ago Yeah, I don't know. There's not a lot about their hardware for anyone If anyone wants to email vlog Thursday At Lawrence systems.com if you have a picture of this or just know someone a link That's a good place to send that information to just fired off the vlog Thursday at Lawrence systems.com Let me actually I think a couple people had questions that ended up in the vlog Already, I didn't answer them right away People ask a lot of them. I guess it was actually quite a few of them People were asking about the PF sense licenses. I got a few vlog Thursdays on that and You know, I might try and read this one on the air because I don't know What they're asking Let's see. Oh, someone just emailed one. Let me see what this person emailed I'll read their question live on here. They literally just came in a little bit ago Watching YouTube channels well two others to focus on unified networks and we have two offices are thinking about a VPN Yeah, for some reason I'm able to get the computers to ping another Okay, everything else pings pulls up on their network opposite network get all computers all MS do not go through The network sharing of course does Doesn't work due to not being able to ping. Well, no It's because they're not talking to each other. Do you have any videos perhaps how to fix the issue recommendations? Basically, I'm trying to set up a host server on one side with a static IP and Let the second and future branches link to it for white files, etc Actually, this is an easy one to answer Who knows what's wrong when you have Default set up windows computers on each side and you create a VPN with two firewalls between the sites Do who wants to take a stab at what's wrong? Cuz it's actually a common problem with newer versions of windows because windows actually got better about security and And I'll give people a minute to answer. Let me take a sip of my beer and All right Either the chances going slow or no one feels like typing windows firewall by default only only will allow Access to networks that are local on the same subnet. So the answer to the problem is Windows firewall, this is Just something that I mean you can set policies or whatnot to make it Different than default and push it out through like a domain But if you're and I'm assuming what this person's doing is they just got computers at each end And he's like hey things can ping across the VPN, but the computers won't ping each other therefore the shares are also not working and yep, that's just as simple as Updating the windows firewall to allow the other networks to come through The inside is just as nice from the network a 3d a 30 embedded main board interesting Nope not in that hole punching The VPN Because they're not doing that they're you're doing a site-to-site VPN So you're you're routing the two networks between each other and windows will understand the routes as presented by the firewall That's not the issue, but by default windows won't let shares come across without you adjusting the firewall on it These are not Linux problems So it doesn't come up for Linux issues, but we'll come up for Windows problems. I Think there's that question OpenSense company and boards. Oh, perfect. Thank you. Someone had sent this in. Oh Well, I Don't know how much help more helpful. This is let's let's talk about this Once again, there are 3d renders. Let's see if we have some details in here is an embedded interior demanding for 24 is We read this properly the net board a 10 revision tool 3 Dash CN is an embedded main board for demanding 24 7 applications Let's see Is there specs? Oh AMD embedded. Okay. Interesting. They went with AMD Zoom in a little bit here. They have a brochure. I was hoping for real pictures of it. They got a brochure So they have details here, but still doesn't really tell me much. I Just got the processor model. That's really one of the things I was after I Swear they have renders. I wonder what it looks like in real life Hey, nice. They're using core boot thumbs up thumbs up for core boat. There we go AMD embedded. I have no idea what the speed of these is Wow That's not impressive at all That that is not a fast processor. That is that is sadness there For threat for cores for threads is sir Try to see what else is in this chip? So it's a 1.6 April of 2013 That can't be right That that's really old You're asking is there a way to do Bonding over SD when I think is a question you want So SD Wan is a marketing term as far as I'm concerned because it means so many different things If you're asking can you do bonded connections? I guess what's the goal and I? Have a video on SD when because it's a marketing term So we can figure out which type of SD when you're looking to do if you're looking to bond connections and Balance them. There is WAN load balancing in PF sense if that's what you're looking for But you can't cumulatively bond to connections in PF sense Like two separate ISPs for example to do that Good evening sir. Thanks for homelab purposes. Which one or more SLC with our DNS s stateless DHCP I don't know. I don't use DHCP six someone will get angry when I say that but I don't have No, I'm not that DHCP six or IPv6 person Said a raspberry pi. No to be fair. It's not supposed to be fast I want to see load idle if load and idle efficiency numbers, which too many reviewers failed a benchmark Yeah, I mean you want a low base power because it may not be needed to be high-powered excuse me all the time and Yeah, it really comes down to what its use case is They probably never sell it never update the page could be end of life date a last order is third quarter of 2023, huh? Oh Oxide racks are generally available neat. I think I looked at them a while ago Looks like no desic So this is a little bit of history if I'm not mistaken The other company is actually They started supporting their this is where the whole And I'm not an expert on the history of it. Maybe I can find someone who is but However, you say it to see so they wanted to sell This is back when they started splitting this company's been around for a while and they wanted to start selling hardware but neck gate was selling hardware with PF sense and This is where the arguments started and if I if I remember the history correctly, and this has been a long time they said no we want to be able to sell PF sense on ours and they said no we sell PF sense on ours and That is what kind of caused kind of a schism if you will so these people started selling Inforked it now. They have open sense So they have open sense and they sell it with their hardware and neck gate has PF sense and sell it with their hardware and Originally, both of these were you know the original fork of all this is going to be monowall. So yeah, there's there's a lot of history there I'm not an expert on exactly the nuances of what went on but that is you know some of the history that they Their inter I think it even says an open senses website still somewhere if we go to the No, I'm not shop that open sense.com. I think it's go to open sense.com, right? Oh We started oh org easy to build free BC firewall blah blah blah Somewhere there. Yeah, open sense project is founded by them so this is There we go founded by Them so that's kind of some of the history of that company and so they're a hardware company that supports open sense So they can sell their hardware to support it plus they sell the licenses as I mentioned earlier there. I Had a downgrade CE because second USB stick with a config But use a second. Yeah Yeah, I guess I could have mentioned it. I Not everyone I didn't want to go through that extra part of adding like five more minutes of explaining To get the second USB. I think people who know how to are more savvy know how to do it wouldn't question it But you know it either way it works. I Don't know that I'll do more open sense videos now because I guess that I don't really use it That's the big challenge and last time I used it I just got aggravated with how many updates it had that was kind of annoying to me The AMD Jaguar processor is also used in the Xbox one interesting. How's the arm arm is doing well? Yeah, I see this is that I pulled this up before This one here. That's I believe that's the link you just sent me from the net board a 30 embedded That's just a really old board if it's the one, you know, unless I'm confused about it. They're I Feel like they're purposely vague in here. Okay, this one's newer. It has a DDR4 in it. Um, is PF sense actually open source the problem with building for arm is Recompiling all the BSD stuff to work on arm. It's not just the interface the interface is actually, you know, it's all PHP That's the easy part. It's all the underlying components that are expected to To see certain functions in an x86 processor Recompiling it for arms not a possible task, but yeah Hey, Tom Sidney Australia. Hopefully you didn't miss much. What about BIOS? Uh, what about it? So people ask a lot about BIOS. I'm not likely to ever do a video on it BIOS has really changed a lot and You either get the rolling release or the LTS release with it. There's the first thing They're really focused on the commercial side of the world. So there's the rolling release legacy LTS release. I think you got a free download Those rolling release fit my needs Where is the current one what are they charge they charge for like the stable version if I'm not mistaken I think that's subscriptions. There we go So if you want a stable version It's 6400 a year or if you 8,000 a year right now if you'd like a Stable annual subscription for it. So there's the first thing with BIOS now if you're just going to go and download it And say I want the rolling release where I'm on the beta all the time You can download this the thing is one if you start going through all their documentation and stuff like that This is all run from the command line to get any functionality out of it. So you have to be Good at that, you know, like here's our getting started I'll actually just go here to Like Setting up NAT As long as you're fine writing firewall rules like this. This is you know show NAT rule translations Set NAT destination. I mean, it's not rocket appliance, but Are you fine with figuring a firewall from the command line? That's most of what you're going to be doing if you're setting up things in BIOS That's why I don't spend a lot of time on it. I don't think there's enough demand for That you know what I mean? We've got really nice web interfaces on a lot of different firewalls now people become used to the You know menu driven I'm not saying that there's no merit to doing things from the command line But it depends on what you do for a living if you're someone managing a firewall in an enterprise level environment BIOS is really popular at the data center level Awesome And it's your job all day is just to punch around rules in a command line If you're someone who wants to go through be able to set up a nice menu get it going and then go about your day That's a completely different use case. So It's that's really the problem with BIOS I would want a firewall with a GUI. Yeah, that's the whole thing Uh BIOS is awesome. As long as you're fine with command line firewall That's that is going to this is how a lot of this works. It's going to niche down There are lots of people who are comfortable with a really basic firewall Some people want some advanced features, but you start saying who loves a command line driven firewall That shrinks the crowd down quite a bit. It's the same reason not everybody writes an assembly You can say lots of people like the program who wants to write an assembly. Oh only the the real the really In depth people who want to take the time to really learn something like that And you can get a lot of benefit of writing things in assembly But obviously that doesn't bring it to the masses That brings it to the few who take the time it takes to learn and write something in assembly BIOS will bring out version 1.4 of the year with a GUI Yeah, but they're GUI in Unless they've made incredible enhancements. It seems like you had to go to the command line to get anything advanced done I would not say it's really an alternative to pfSense. That's just It it's Like you could do some of the basics. I mean it's been a little while since I looked but I couldn't see it seemed like everything was like Oh And then you got to go back over to the command line to get this done and this done And that's not going to create an experience and most people will just get aggravated and go Well, I'm just going to start doing it from the command line or I'm going to use something that just does everything in the web UI Uh, you're looking at the a10 a20 listen the website the epic Uh, okay epic a core a30 is it's a network dc model. I'll assume it has even more power. Okay The mvcp is similar to pc engine sports interesting A single build uh work current. Yeah, but it's the work on rba processors But will all the functionality when it comes to the network driver's work? This is what's going on under the hood of pfSense is the its interaction with the network driver So, you know, we'll take any arm board and say hey, it needs to have two network interfaces at least Then it has to have those interfaces working well enough to Function inside of there. So it's not just that will it run on arm? Will the functionality be there to make pf filter work properly? The these are the nuances that kind of matter for running it on there If you love sysco asa then you might like bios. Oh, yeah If you're if you're someone who's already familiar with like sysco firewall It's not a hard pivot to go to even even micro tick micro tick You can do quite a bit from the command line on versus their web UI But that's a learning curve that it depends on the person you ask With some of arm's recent action I wonder if they're applying some more pressure to neck gate. I don't know There's no reason a cli can't be ported Uh to gooey. Well It's Can be it's just about time any of these are just about the time it takes to do it Um, I'm not likely I don't use bios. Um, it's just niche Maybe one day I'll get into that niche. I don't got time for it right now If I had the time to sit and really learn it and get good at it I mean, I could muddle my way through it, but who wants me muddling my way through something It's really not worth my time It's also pretty much not something the small or even media business will use It's kind of niched into data centers And people who want to play with it in homelab because they want to get a job at a data center and That i'm not going to do the best job of representing either one of those Have you seen any of the data room tours on the channel Uh custodian data centers. I've probably seen one or two of them Can h a and p f sense do geo filtering? Yes Well, uh, or What are you asking about ha like the the geo filtering is done with pf blocker and g So if you're asking is when it's in High availability does it work? Yes Yeah, bios came from viata a bunch who also came from viata the um The edge series was forked from that Oh ha Is ha proxy So can ha proxy do geo filtering pf blocker does geo filtering? So if you have pf blocker and ha proxy, you can geo filter Yeah juniper is also bsd based We demand a drug a a drunken struggling tom Well, this one beer is good Um, I like these but not not on the um drunkenness. No Being drunk is not my thing So not not likely to happen Uh, what else they happen? Oh, it's just tech talk. Okay. I double check to make sure I covered the things I wanted to cover Maybe the other thing I need to cover Hey, look an actual picture Let me throw this up here someone else something this So that's what it actually looks like neat Because everything else was those renders. I just want to know what the board physically look like But I think this is the older one, isn't it? Yeah This is that older processor Can be a sense to ipv6 geo filtering? Yeah Actually can you know because you can do they have both in there ipv6 and ipv4, but I don't know how good ipv6 filtering is I just don't have an answer for that one Yeah tnsr also does not have a You does not have a ui that is true If you don't get drunk, how do you screw up your tech somehow? I'm able to do it. I can break things without being I can break things sober It's a skill Uh, someone invests to use hardware asic base router replace your psense. Would you recommend other than sysco? I don't know Don't really have any thoughts on that Thank you for the nation richard regardless of how the neck eight stuff spins. Thank you for coverage and effort Have a cold one on me. You know, my cold one's almost empty Yeah, I guess 48 does have 48 48 net 48 does have some of their own Um, I think a few of their models have their own chips in there You just got to deal with well same thing with sysco. You got to do what they're licensing on it Without being drunk you call you call it karma when you break it. Yeah Uh, don't drink and deploy guilty of this in the lab. Hey, why not? Uh Yep, but not their ssl vpn. Yeah, the the they need to refactor all the code over at 48 for that. That's just a disaster that's Yeah That's broken Oh speaking of broken. I seen someone posted my I have not I purposely haven't because I was busy working updated to the latest version of the True nas Let me pull it up That's was my next projects is get my systems up to date, but people have been talking about some problems So they uh The latest version of scale Someone linked and we find it in my forums Someone linked me to some issues people are posting about So it's on my to-do list to test this And to do where did it go? There we go Suck initializing app services now We got a person reporting the problem Is there a solution here for it? So yeah, didn't uh More people chiming in they have the same problem. I just gotta Well, we'll see I'll I'll give them another day and then I'll update it Like I said, I'm not completely sure Are they gonna have a point release that comes after? That's kind of the question Fixing stuff drunk, uh, but not being able to replicate the issue. Yeah Hold on. Hold my beer. I need to replicate an issue No, I actually don't have to embrace it. I can keep me I can keep making people angry for a very long time by saying I don't care about ipv6 Oh, let's see So I rolls keep things dialed in and running it smoothly. Yeah, uh, picked up mario wonder for switch yet No, I don't even have a switch Is there any spc that can handle lsi Uh hba cards You you know, you could use the zima board because it's got the extra connector on there I like the zima board. I want to do some more projects with it and actually the Those are new one they have this the blade, but they're also working on a I do like their website. It's just pretty cool. It's called the zima cube So this is something else that they're working on too. Is there Zima cube. I I think zima's got some cool stuff on there for building it I'm really looking forward though. My new nas is going to be the 45 drives nas I got one of those coming the zima edge Oh 40 net has a pile of cbe's for sure Zima board xcp and g. Um, yeah, I've already well. I know I didn't try it. I've been running um Excuse me. I've been running the I'll switch which land there we go log into this I got to finish my video of one of these little ace magic Boxes, but I've been running this with xcp and g it works really well Now I know lots of people are excited for me to try proxbox. Unfortunately This box doesn't like proxbox and uh, I tried loading it a couple times. I reloaded it twice I kept getting the same results. It does not like the network interface in it. It's kind of annoying I don't really have a solution for that I I don't feel like spending enough time to figure out why Because it doesn't see It sees the network interfaces, but will not link with them. It's a puzzle So yeah, someone if someone knows because they're intel I think they're the i226 chips on these if someone knows why it doesn't like these because From a quick google search Proxmox has got the latest kernel and supports the i226 chips the two and a half gig ones But what happens is it gets an ip address Then it tells me the link goes down and that's where it stops And uh, I haven't figured that out yet Happy birthday chris and Uh, lube. I'm sorry if I say her name wrong. Lupita Uh shout out. So happy birthday. If it's your birthday, uh feel free everyone wish, uh christmas See tomorrow is mine and my wife's birthday. You Married someone with the same birthday as you that is kind of cool The cube looks nice. They said zima board 2.0 is coming next after the ads are released neat Uh, what is the point of buying support from people who don't know how to do security in the first place? Kernel patch for 226 is that the solution for um Open sense is that there's a kernel patch for it Uh proxmox 8.04 fixed a kernel issues that worked fine Uh that worked fine in proxmox 7. Ah So what version do I have I guess what version did I download? I did this last week So was it the it was last week's when it failed. So 802 Yeah, june. Yeah, because it was a june 23rd. I So someone Where do I get the latest because the latest according to them? Is 8.02 So how do I get the How do I get the latest release if they have an updated the latest release to fix the problems that are in the release I have These are questions. I don't have the answers to maybe someone does So 804 fixes it but if I'm loading off 802, how do I get it? How do I get the updates on there? That's a challenge Yeah internal apt update except for the fact that it doesn't have internet. It doesn't have access to the network Proxmox offline updates. How hard is that to do? How to update without an internet connection. Is there some simple way? I don't know. It didn't come up as the first search results I mean, there's people discussing it I mean, I could you know, try to mirror their whole repository onto usb and modify At some point. I'm not going to spend an hour or two hours playing with it just to get it to work I guess this prospects work with the usb network card. I could try that Yeah, I know it's just debian But if you don't have network access, you kind of have a problem of getting it updated I might just not try it on that box. I I know I've got other boxes. It'll work on I just was aggravated It didn't work with this box because I'd actually set up with the ace magic I wanted to do and I'll still do this is just going to be on a different system The ace magic I wanted to try because it's a pretty fast system. So this is a cheap relatively cheap rise in system, but I have another system coming that might be a better one to test this on So what processors in here? This is the yeah amd rise in seven seven three three five What my goal was was really simple because this is something the audience would like running benchmarks The same benchmarks with the inside of virtual machine running on each one of these systems to see what the results are I think that would have been an interesting test, but I need the same machine to run the same software I don't feel like spending hours messing with it. You can also install an upgrade. Yeah, that seems tedious too Or install another system and then move to the system board. Yeah Well, no, then I got a problem with the network cards not lining up and then I got to fix that But I do have I have more servers coming because I have some other stuff for doing matter of fact That giant server I showed earlier that has all the nvidia cards in it is also running proxmox because that is a very Specific request from a client If I want a lab with cml genus re even g and packet tracer is the sxi my only option I don't run any of those so I have no idea Uh to get the same machine get a pair of inexpensive del r630s and match the cpu and ram and are Yeah Well, I'm just going to take the same machine and reload it. I'm I have a newer machine coming Uh from another demo Just downgraded back to ce from plus through my config xml and the installer drive and it loaded My config on first boot So as we get mtu 9000 some developer will make make up mtu Uh, yeah, don't fragment. Oh, that'd be fun Got a b-link rise in seven 70 40 hs set up headless test out for a client very stampy performance, you know I probably got him pulled up somewhere I have some benchmarks from that device and Yeah, the the little rise in seven system is actually Reason I got a benchmark this to compare it to some other things But I'm impressed with how fast a lot of those things run on there like loading Uh, even the iops it gets is not terrible for a single drive system. Of course, it's mvme It's not it's single nvme. So we're not dealing with any type of redundancy here But for building out your home lab, I think it's actually a pretty clever little box for that Do this reset Like watch this vm boot up man. It takes no time at all for the vm's to boot on this. It's just it's just boom boom boom done Yeah, I think these rise in many pcs because they're so low wattage low heat quiet small They're kind of a no-brainer for building a home lab stuff on if you want a little quick test environment That doesn't take up a bunch of room and doesn't pull a lot of watts These are kind of an easy answer for that compared to I mean, it's not exactly the same as running an r640 or some other high-end enterprise server But do you need that to do what you're trying to do that's it comes down to need I mean, you're not trying to run some 24 seven production environment. You're going. Hey, I'd like to learn docker I want a device I can learn docker. I can learn some Setting up some linux boxes and virtual machines on I want to play with a handful of things or learn ansible This is a great tool for doing that If you're a little single machine, we can't side by concern Well, which is important for the human factor feels more snappy be renders more smoothly No I don't do anything based on human factor. You do things like this You run the same feronics benchmarks test with these same exact parameters and then you answer the question You don't answer it with Uh, it feels snappier. Uh, so no, that's why it doesn't matter. Didn't you this More it's more deterministic if you just run the same test the same way each time Uh, palm is proxmox offline mirror Hmm, so there's maybe a way to do it with the Look up proxmox Share this tab terminology getting help license. There's a license for it Okay, it's just open source. Okay. That's good and solve via apt offline subscription keys Yeah, I could read through this I don't care. I'm just going to try it on a different server because that's an easier That's an easier answer than doing this Proxmox users know how to get lightweight 2d GUI solution without needing pass through Uh, when they'll do the video in usp 3.2 usd hard drive controller sees you working great. Yeah, those are pretty cool I think those are going to become more popular as that gets more stable one thing about a xeon e is that effectively Uh, a 10 nak made into a server less cores more fast Uh, much quicker to work with the cloud managed slow parallel workloads. Yes You may as well heat your office with servers in the winter. I can agree with that Pharaonics benchmarks are good for a raw computer which is important in most use cases most important to human factor Uh, like configuring a windows server I I don't understand your debate here like It computers are deterministic they process at a set speed and there is overheads that a virtualization platform may create The reality is someone actually did this night. I've been a long time I should look up the research he's from a couple years ago and they tested several different hypervisors and They were all within a few percentage of each other There was not a major in terms of cpu and memory performance Most of the problems in virtualization come down to the drive emulation performance. That is where there is a wildly different amount of numbers and It is a little bit slower on the zen side for single vm accessing the drive but That's because of the way zen isolates each one of these calls Oliver Lambert's got an entire write-up that he did in my forums I referenced it recently, but if you look up proxmox, uh, we're more specifically kvm versus zen There's a write-up in my forums on this The parallelism is really good in zen But the single performance of a single vm is not as good But the reality is you don't build a virtualization platform to get single vm performance You built it because you want to run all of your vms and get good performance. So that's kind of you know It's it's something that matters if you are looking for the utmost performance proxmox. I believe And I have to look at where these numbers land proxmox had slightly faster i o performance But that was for a single vm provided. No other vms were taxing the drive Once you start taxing the drive they kind of equal back out Um Morton from my playhouse channel ran a heat and a heat vent in the server room to his living room to warm it in the Dutch winter. Yeah, there we go Well, the variable clocking it's still deterministic. It's still As the cpu You know provided that the virtualization software can ramp up the cpu if that's an option in there for variable speeds the turbo speed if you will That's still deterministic the turbo speed provided is taken advantage by the hypervisor means your benchmarks should be really the same between hypervisors Minus whatever overhead a hypervisor puts in between for the cpu That's kind of what that comes down to And I know my test of showing boot up speed is probably the least relevant test we could possibly come up with As to how this works, but let's do something like hey, let's Um, oh, I don't have a backup on this. I'll build one real quick backup overview Dem W and 12 demo system Delete that grab grab that one Save get rid of that We'll make a new backup The backups and things like that work quite fast on this so It doesn't take long to back these up so I can have an extra copy of this This is actually the nice thing about the way I run xo Not on these machines. I just have to load xcp and g on them And then reattach them to my zen orchestra and then I can just restore backup move operating systems around and It works really well the two and a half gig works great on these. I wish it was 10 You know, but it's not so that's life and even without being a 10 gig It's only going to take two minutes to back up this virtual machine So not really that big of a deal I want to put an open sense firewall behind my RTA x8s at you. Um, my box doesn't have a wi-fi card So I need to use the acus for my wi-fi. I want open sense to take care of everything else I mean, you can double nat it. Oh, how can I test a backup? Untested backup is not a backup. Your words not mine. Oh, that's easy To test a backup. Well, we'll actually do it here. So as soon as this backup is done In 49 seconds, we'll restore one So in 37 seconds, we'll restore it Um, nobody cares about a backup that works. Everyone wants to restore that works Which means The best way to do it is actually test the restore Uh, what's the easiest way to connect xcp and gvm gui to a rub browser or some kind of remote desktop? Uh, that depends on what operating system you're running. I'm running linux So I don't need a desktop at all if you are running windows rdp is probably the easiest to do That would be my guess. I would go rdp if it's windows If you're using linux, you can use x to go. I've got a video on x to go. I haven't used it in a while, but We're going to go ahead and stop this one So we don't have to well, we can leave two of them running because I can always Create it, but here's my uh Debian 12 demo and we're just going to go ahead and do a restore So, uh, let's no, we'll just start the vm after i'm not going to generate a new mac address So let's see how fast it'll restore It says in two days, it'll restore. Okay, less than a minute. It's one of those things when at first uh Do that What was that? Taking another sip of that beer. Yes, I need more beer. I'm out of beer. Well, I don't need more beer Uh, do you have separate data sets on true nas for snapshot backups next cpng? Yes I have a lot of true nas system. So I've got to spread across multiple true nas systems If I create a separate backup with health check Is it going to duplicate the backup data from my original backup the job that runs hourly? You do not want to do an hourly health check That would be a lot unless you have a really really fast system The way you would do it is Go here And by the way, it obviously worked because it's booted. So this one worked fine Here's our debbie 12 youtube demo if we look under general it says restored from backup So you notice one restored. So we'll stop it and destroy it remove But what you do is if you have your backups here and I have a video on this So this is my It's not enabled right now But let's say we have an automated backup that we can say here's another schedule backup retention for this is one We want to do a health check on the local storage So this means go ahead and run that health check on this one as well at a different schedule So you can have your health check schedule. And maybe we have this just get more implicit Um Not every day but on Sundays, maybe I don't know we'll just say Sunday maybe saturday saturdays are better than sundays, right? So sundays at these times hit okay Now we know we have this schedule and we can enable each of these. So this is one backup schedule This is the other backup schedule that has the health check in it. So this one does a health check this just keeps three and Now You hit save and you you run them separately and it's not going to duplicate the data It'll follow the retention policies on there Not going to enable that though save Yeah, that's the reason they did it so you can have one backup job that does it all That's an easy way to do it If we a month go ahead issue with proxmox changing network names after reboots for their test revealed that the standard dbn Had the same issue replacing it all as well Yeah, that's I've seen more than one person. Maybe it was you uh It's definitely It's a thing Uh when networks don't line up the way you want them to be Hey, I admit this is marginally prettier than uh parish Welcome Veronica. Good to see you here For you, though, don't know who Veronica explains are go and subscribe to her channel Definitely, uh some really good topics on there covering a lot of linux and even some fun commodore things which I think is really novel I love the I'm I'm someone who does not possess or want to possess because it'll become an obsession for me really old hardware but I love living vicariously through other people who post and Teach me about old hardware and things you can do with it. I love the vintage computer stuff I just don't want to start collecting it because once you start it seems like it becomes a problem I don't need that problem, but I love learning about it. I love thinking about Matter of fact, I'll give a shout out to also if you haven't watched retro bites A kind of documentary style about retro computer stuff really good knowledge. They drop uh on things too But yeah, check out Veronica explains channel definitely always fun to Learn from the linux things she does and also from the vintage hardware. She'll uh run around and talk about so always fun times I brought it up in a live show me you and j. Yeah, okay. Maybe it was you who brought it up I've seen a brain uh brought up for more than once Uh, I will drop a link to her youtube channel. It is relatively easy to find Uh, great sco because there is turns out there's not another person named Of Veronica that then puts explains after their name. So it becomes relatively easy to find her channel Oh, adrian's digital basement. Yeah, definitely. Um I'm one among the people I watch uh, you know, I could run down lots of I like uh computer clan Does some great videos. There's so much, you know, this is a lot of what the youtube I watch is I don't watch a lot of tv Really, I'm not a big. I don't watch any shows, but I love I my hours of watch time Watching other creators talk about things, especially when it's the vintage computer stuff Definitely where I spend a lot more of my time online The um, I mean, I enjoy the the weird It's some of this fun esoteric things like the fly pentop computer. Definitely strange Um, some of the history of apples things were really interesting I thought this was actually if you have a g4 cube video, this was weird There's a lot more to this as a product than I ever knew And the history was uh, equally interesting. So yeah He did a great job of covering the weirdness of this Not everyone's into vintage computers, but you know, that's uh, Yeah, katha ray do Absolutely, but one of the other things on here and this is the Pull this back up the devian 12 youtube demo I haven't walked through this in a minute But this is if we wanted to do a health check for those you don't know what this does Let's go ahead and just build a new backup job vm backup yt demo health check h8 health check So it's like that vm health check Health check local store or whatever. We're not worried about the schedule on this Great, great, great, great. All right And we'll just use it as a delta backup because we can do it faster that way Uh, if we want to do more than one of them And the we'll set the retention here to we'll keep three of these Hit okay Hit create True nas is the target Create awesome Now when we run this backup job, this is going to run a little bit differently here So we're going to hit and kick this off because it's going to run a backup But then it's going to validate the backup by doing a restore and letting me know the result This is one of the really cool features that Um, I really like an xcp and g is the fact that it runs through this entire process It's not just a backup It's going to do the backup and restore because hey, you know We don't want a backup that says successful We want to restore that works because no one cares about the backup that works They want to know that the restore works Uh eight bit guy. Yeah eight bit guy is definitely, uh, interesting Uh, can I back up to s3 azure blob native? You know s3 is one of the options in there It's not going to be the fastest one Um, I have a question about pfsense Well, then this is a good place to ask that Agents little basement another is curious mark I should probably pull this up as a task So it's almost done with the backup Now it's trying to run the restore Uh, s delete from cisternals does work on refs if you're stuck with refs fine You need to free up some underlying storage Hmm, not sure what that comments a reply to But while that's doing there the backup remotes is what they're referred to is s3 is an option so It's in beta right now. I I don't use it much s3 is a nice option. There's use cases for it But it's not going to be your fastest option your fastest is really going to be nfs That's why this is set up with nfs because it's just a really fast way to get these backups and restores to go As a matter of fact, let's go to our backups And see our health check demo So now we can go through the entire process here of what happened So we did the we'll zoom this in to make it easier read We snapshot Transfer health check start end duration a few seconds Speed vm start vm start success duration two minutes type full So we went through we did a backup then we did a restore and then we destroyed the restore So the health check was successful. It's just really Novel that it can do all of that and this is the fully open source version compiled This is not the licensed version if you will they give you a lot of features for free 100 time. We have not tested the backups yet all the time Uh, my he proxy is no longer working on pf sent system version anything I can do to make it work again I would look at the logs to figure out why it's not working. There's usually errors in there Oh, yeah, Dave's garage Me and Dave talked about he uses true nas and we had a discussion about that Dave's got some interesting insights Oh curious mark has a great set of videos on the Apollo flight computers. That's that's pretty neat They actually got one working. That's that's super retro And don't forget you need the guest tools installed for the health check to work right Yes, they're looking for the guest tools. So that is true Yep, if you don't have installed they will not work Turn it back on Well, I mean wind this down does anyone or does someone want me to go longer? It's either either a I go get another drink and keep the live stream going while I play with stuff or b I wind it down and go watch some youtube Is now you guys give me all of you have given me more suggestions on more channels to to watch I'm particularly uh, want to look up this curious mark one But it's funny because it Finishes a curious mark Apollo Oh, this is actually not even it's just a little bit older of a video. It's still really cool Neat. Okay. I'm distracted Whatever you do go for another beer party times full send. Okay You know what I I still and I did make this so you guys can watch my Tom's going to go grab another drink screen This is I remember where that is Great videos on sun systems neat Having a blast first time in our community cool Apollo play computers were analog computers very interesting. Yeah, I'm that's I cued that up That looks like a good playlist to go watch all of those Where is my video that plays there he goes. I'm gonna throw my be right back video So I'll play that while I go grab a drink. I guess I'm back Music makes you want to dance Uh, the Taylor and Amy show you know I Taylor and Amy the reason I first found them was this Oh, I think they're TRS 80 video So for those you aren't familiar, uh, yeah, I I was like, oh Especially is this one is more specifically which I've watched this one. It's funny. You can see that I watched this one I watched this one of my other youtube name. This is the cocoa 2 in the ram upgrade video And uh, yeah, they bring a lot of great humor Along with it and uh, branca did a fun song with them at the vintage computer festival. So that was definitely great Oh, let's see might be on this, uh Asianometry Here's mark to actually get the working simulate moon landings and the actual hardware some new equipment And software to make components in space neat Still have a 46 dx around here. Yeah Do you know if you can transfer bhp evm's what are b? Are you asking a beehive m's? And no not that I know of Uh, why did you downgrade c instead of ump of sense plus? Um, I just did it to show people that it can be done It's not that you have to I wanted to prove that yes This is how you can do this So if you would like to do this you can and people were wanting to know what the path was If they don't want to stick with psn's plus z vowels Can't be true. Yeah z vowels and raw disc images are kind of the same Wife and I just sing happy birthday and cut our cheesecake Awesome cheesecakes good, but I did do The reason I did the uh sidegrade if you will was You know, I just wanted to change this is the one I changed it on I I still have the backup if I wanted to roll back to it So if I rolled back to this version or if we look at the backup I did a backup because it'll revert everything back to the previous and Do it but it doesn't matter for our lab I switched this all over, but it's not like It's not like a really big deal to me for our lab stuff The way we treat our lab is we have a ton of these So we go to networks here This is what the lab side of my network looks like and I'm going to do an updated video on this Of why we have all these different belands attached and this a lot This is what allows us to easily and dynamically rebuild a network to match a clients before we do our setups And uh, that's you know, it's just a real important feature on there You know, I'll say it again, too And I don't know if you've written a channel earlier bronco, but I guess uh, Chris and his wife both share the same birthday, which is novel. Am I am I correct about that? Um, that's interesting But back to something I want to show here This is something that's in our office right now Or at least this is But this is an easy example of this you know, this is another project going out for a client And we matched their network So all these things are actually predefined with all of their IP address ranges whatever the client has requested for the pre-work setup that way when you know We're building it everything out in our lab. We have all these different We call it wan lab and lab 102 one oh One oh one 102 and 103 and then inside of unify. These are all corresponding vlands So we can match a client's network. So we it doesn't really matter if we have pfc or plus in our lab on there Born on the 29th of february. That's an interesting that would be an interesting date um Oh, okay, so, uh So the next february 29th is gonna be in 2024. I didn't even think about that. That's interesting Yeah, have you done an ipv6 only deployment not likely yeah born on february 29th One birthday every four years That would be interesting for sure What else was I gonna talk about? Let's see here Data center data center data share stuff. Oh, I do laugh at all the signs in the data center That say things like no tailgating You're gonna have to figure out um You're gonna start searching what those errors are and post them in the forums to figure out what you changed in ha proxy to cause the error Managers pushing for dual stack network work. Yeah Yeah, this seems like more work to me and yes Uh, not sure when you change the thursday stream, but I love it. Um, are you referring to the time that I changed it to? I just been doing it later Oh, I did do video of this Uh, they had a degauser The data eliminator degauser and the data destroyer. These things are pretty cool Um, they are for crushing drives. So that was uh, it was fun to play with it Charging it charges up and then degauses the drive. It's it's really It it's interesting. It's louder than I expected. Um, when it does the degausing You updated and it just don't work. I don't know somewhere in them logs. Um There's gonna be an error message in there somewhere of what is what it's hanging up on So you can find that error and figure out What broke because it's an odd problem because I mean I I updated my he proxy and it's working fine. So Uh drive crushers a crime against fantasy Yeah, but they have to uh, it's a compliance problem It's one of those things like I I like the one we have we have the pure love. Matter of fact, I have a video on it if you go to my channel Buried somewhere in one of my old videos It's called the pure love hard drive crusher I wonder if they still make them It's actually a good business problem. We'll show up there and we And it just crushes the hard drive Let's see. Do I have a link to there? This is from six years ago. Wow Yeah, the company still exists This is a neat device for crushing hard drives I make my son use it mostly Uh, do they charge customers to use it drive destruction? Well, this is what's interesting of how the data center works and This will be a part of it that I I'll figure out how to iterate in What they do at deft they have a warehouse if you will of all kinds of drives And everything so if you have something in the colo now when you put something in a colo You own the hardware you you are renting a rack in a colo from them But you know, let's say you wanted to use chicago colo, but you're not in chicago chicago is for me about a four and a half hour drive if you have a problem with one of your Devices, you know hard drives go out for example, and you're like hey, I need someone to replace this drive Do you send a tech out there by four via four hour drive or instead? Do you Contact the team at deft and go can you swap one of the drives in this bay? And they will do that and what do you want us to with the other drive? Well, that comes down to compliance. They will crush the old drive for you Uh, so it's just a service they offer to help maintain Everything in there, so I think that's great, but for compliance reasons in some companies There's just no risk that they want taken on drives. Well, that's going to be the compliance and you know the The companies that care more about their data Is sometimes some of the movie companies and things like that They are they are so paranoid about a frame of a movie linking out that they would rather crush it than anything else Oh, I agree d-ban and things like that do a good enough job to get rid of the data, but Compliance and insurance doesn't care If it's not crushing wrong drive What size are they or what size are the crushing drives on there? Their website looks the same as it always has Videos they have my video on there. No These are their videos Which is fine It works really well. They got it on a cart. Um, but yeah, it's a neat device Simple and effective. That's how I look at it. Like you just show up at this and start cracking drives Any opinions on seph? You know, that was the other topic. I was going to bring up, but I didn't get my graphics done So I did a in-depth training with 45 drives on seph. I did a two-day in-depth training on seph I know a lot more about seph than I did before and I'm impressed with it but I'm also It's harder to set up than I thought in terms of how it works There's just so much more to it. So there's a lot of complexity in how they handle things Especially the way seph clusters work, but it's a really solid system once you set it up, right? It's extremely nice. So I'm going to do a Few videos on this as a topic and I'm going to probably work with 45 drives and building the videos out because of the way the nodes work is Intriguing to me as how I'd say it like they built it for resilience seph is amazing but It also like the minimum the minimum 45 drives recommends for a production system It's going to be four nodes if you aren't at least doing four nodes Don't bother if those four nodes also aren't interconnected at at least 10 gig or faster Then you're going to have some issues There's just some challenges like the way seph does its healing the way it distributes the data Also comes at an expense of you need to have the hardware synchronizing very fast You have to have the network infrastructure set up to do this So it's just one of those things that it takes a lot of planning It takes a lot of hardware to get it going. But in the end you end up with an extremely resilient system In terms of the failure of losing a node. So I think seph is an amazing solution But it's also has its complexities and it actually I it kind of needs to be compared I think to zfs because it still doesn't do some things that zfs does kind of smoothly like snap does seph does snapshots in a very similar way to zfs but seph does not have the same level of replication services that zfs does so to Replicate a let's say we built a seph cluster to replicate that off-site is not as simple as it is in zfs So there's definitely some changes on that It's not one. It's not a one-to-one seph um no seph definitely has strong data integrity guarantees it I would say it's it rivals zfs, but But it's implemented in a very different way. So it's not exactly how zfs works It's the way the osd's work the way You it's it's one of those things like I it needs to be explained because it's not It's the way it breaks out all the groups of the the pgs get broke out um Is there a good breakdown of this? This is what things like if you start looking it up, there's a lot of people have a lot of explainers on it Like how data is stored and stuff There's a lot to learn I spent I spent two solid days Working on this I was like wow. This is amazing, but also learn where some of those limitations are on it Yeah, you really like four is the minimum to start That's that's one of the first things about seph to properly be implemented Is going to be you're going to need four nodes Yeah, uh, well in five nodes to start if you need redundant gateways. There's a whole another thing Yeah, redundant gateways how you want to present it. It's um 45 drives to the seph training. Holy crap. I've been watching your channel since before you remodeled the office lab and mounted all the orange cable covers What in still seph has some struggles with certain things that's going to be For example, how seph handles If you suck a bunch of databases on there with a Lot of 4k writes for example seph may not be the fastest way to get that done You can overcome that by buying faster and faster hardware but there's some challenges with seph compared to zfs in terms of performance and seph always values data integrity above all else But to get the real performance out of seph I mean you can go easily up to a hundred gig backplane to get these seph clusters To sync at a really fast rate. That's where the challenge comes in Every drive in seph has its own daemon running in linux This means that they all talk to each other, which is great except the Amount of communication at that creates sorry my back edges That extra level of communication Created there Means you have to have a fast interconnect to get all the data synchronized because seph doesn't release back When you write something to it that the data has been written until it actually is That's where the slowdowns can occur. So so by Valuing the integrity as it should means there's a natural slowdown Now what I want to do is put together an explainer video because this is This is a fundamental problem with any type of cluster storage. This is not a seph problem This is a cluster storage integrity problem This is also a zfs vdev problem to verify all of the rights to the drives to each leg of the vdev So there's a lot to it Problem with four seph nodes is you can't do four nodes on three proxmox nodes So you need five nodes so it'll having an odd number of everything. Yeah Seph won't work over 5g data connections. Seph can have trouble over even Lower data connections Because when it's trying to examine the cluster health, it will think the cluster's out of health When we did this demo, there's a whole there's five Virtual machines will demo lab if you will with 45 drives There's five virtual machines That they spin up if your computer's not fast enough to run all five of these virtual machines with enough speed You can actually run into a health problem That is just basically latency between machines because of the heavy i o that it creates Like putting sink always on your pool, but even slower. Yeah seph is You know, it's kind of is and isn't a homelab topic I think people get excited because they they love the idea Of beautifully clustered redundant storage But the reality of it is actually harder to do it's not it's not that magic It's just it's just like whatever would ask me about hyper converged and hyper converged is the storage marketing term Just like sd wan is a networking marketing term I care about how the storage actually works Not just oh, this is hyper converged. It's one data store Don't look behind the curtain of what's going on. We sold you a hyper converged solution That's not That's just a marketing word to me. You know hyper converge great you glued all these storage boxes together How are they glued together? How Is the integrity being verified and what happens if it breaks? hyper converged systems that break Is it easy to restore? Is it hard to restore? Is it easy to get all these drives back and sink? What happens when there's a failure? That you know in really important you know Gluster is an interesting topic into itself Because gluster is a competing clustered storage solution, but seph seems to beat it by a lot So it's kind of interesting is How seph is doing compared to gluster because that's what true nas went with was gluster But they seem to be the only ones i've seen that really seem to be pushing gluster as a way to cluster the file system So it's kind of yeah Yeah, hyper converge compute VMs and storage on every node But what's the glue that holds it together? That's always my question about how these things work I because unfortunately and this is the nature of being a consultant people don't talk to me about systems that work Wonderful, they call me because they have a system that doesn't work wonderful anymore And then trying to figure out how to get that set up properly again And what broke you start asking the real questions of what glued all this together and where did that fail? Uh gluster had some limitations even ltt stopped using it Yeah, if you were going to build a large environment for someone doing You know video production. I would definitely go seph and More specifically if you look up and we'll pull this up right now. I I should talk about this. Um, Let me show this to the right article Yeah, september 20 23rd tree. So just a couple weeks ago, uh, I think Hey, look tom lorenz is in there along with a few other people window wilson level one text So, uh, I'll drop this forbes article for you But they talk about the 45 drives team how they're disrupting storage with this and specifically and we'll pull up the company real quick Do so your founder of moonshine post days 45 drives the redefining vendor custom relationship and day space They empower to scale with their technology and support and losses here a transformative approach Uh I'll throw a little secret. That's our client too We work with moonshine and 45 drives and all these companies So they're all related like these are some of the projects we work on and if you're not familiar with how big Moonshine post production is where do you think some of these petabyte servers end up? You know that we buy Uh, it's a really interesting discussion and then you know, you build these things out with seph You can build these things out to scale 45 drives is probably One of the foremost experts on seph and I they've done a great job of doing that You know, I really seen your praises for it And that's why I'll probably do a video with them On the topic of seph because it's it's a fun topic I just need to have some more graphics put together and things like that to Present it visually For pro through large files at a local level. Gluster is faster. I don't think that holds up Um Zfs might be faster like native zfs might be faster I don't know that gluster is going to be faster for that. I don't know Recently updated the stuff cluster. Yeah now that's something I learned a lot about was how their update process works. Yeah that is It's relatively smooth because they will let you even get I think it's up to two versions behind and still allow an upgrade path So that's really um Really important Why did it be 40 drives? Why did it come in 42 the answers to life? Yeah, that's true Is that rum or bourbon? That's this for those wondering 45 drives is the answer to life in the universe everything plus three hot spares. Yes I can go with that But the whole seph system is just really It's slick. There's no doubt about it man. This is uh Oh, wow, this is a really old seph blog They do a lot of testing. They do a lot to show the difference There's a lot you can do to set these up. There's It's a lot of complexity in the setup of seph. This is one of the challenges. It's not like or near as simple Is zfs is complicated, but I thought it was complicated until you compared to seph. You're like, oh no zfs is really easy I I'm I thought I knew a lot about zfs. I'm like, oh it just I know a lot about it, but it's also not near as complicated as setting up seph So there's good and bad when you set this up, but hey, you know It's still fun to learn all this stuff They've got a lot of good documentation. I'll say that for the seph good community good documentation And 45 drives. I think I tweeted this to pull up my twitter You can use their houston system to deploy seph. This is what their training also does and By the way, this isn't just a ui This is a ui that builds ansible playbooks for your deployment It's really slick. I thought this was a really nice way. They implemented this so instead of Some ui that magics the back end the ui Develops playbooks that you can actually go back and edit yourself or you can edit them inside the ui again and rebuild your playbooks So once you set all these things up you go, oh, hey, I want to build this here And you're like, all right now I got this built this bill Oh, I want to change your configuration so you can go back and rebuild the playbook again and redeploy your seph cluster So it's definitely um I'm impressed with how well it works. Now. I will be doing because this video is going to be simpler I'm going to talk about True nas versus synology for high availability storage I'm keeping it narrowed in scope because those are going to be more popular less complicated easier to deploy solutions and their kind of turnkey ready Things you can easily deploy yourself type of solutions now specifically i'm going to be talking about ix systems with their hardware Because they have a redundancy there and of course i've recently reviewed the sa 3400 d From synology So i'll be talking about how these devices do the different ha and some of the advantages and disadvantages of doing them Synology actually has two options, which is kind of cool. Synology will let you do ha between two individual synologies or They'll have the sa 3400 d and similar models where they have two motherboards and a single Synology and it was a good discussion. I have with a client who was kind of going. Well, what's the difference? I they first started going. I think I want two separate servers So they're two completely separate and i'm like that's fair and that's a good way to do it If you took and they make an sa 3400 for example So if you took two sa 3400 not the d models not the dual model You could put two of these together and have redundancy And it will flip between them or you can buy one with two motherboards The real difference is if you buy two separate ones, you got to double the number of hard drives you need Versus two motherboards are talking to both sets of hard drives Which means you don't have to buy twice as many hard drives So that is a it's not quite as redundant, but it's pretty good It's it's still you're not going to fault tolerate yourself against a catastrophic failure of the backplane that glues the drives together But that's usually not where the failure mode is the failure mode is not usually the backplane can be I'm not saying it's never has been the backplane. I'm saying as you go down the statistics of what fails Hard drives most frequently power supplies next motherboards on the list backplane Not usually Synology doesn't win good for simple stuff It was funny when I was wandering through the data center plenty of synologies in the data center. So definitely We already know because we deploy it through a lot of large commercial clients Synology makes them really solid rack mount stuff like there's their rack mount devices are great. I We just don't have failures on their rack station stuff Matter of fact, we've been going through trying to find even with the consulting clients anyone who complained it went back like the reliability is Really impressive with Synology like their their products have come a long way. I've been I've been really happy with them I can agree with this that I just don't trust butterfs as much as I trust cfs. Absolutely. I'll agree with you I think Synology does things in an interesting way because the way they're working it in Synology Log into my Synology real quick so Synology Has butterfs and where am I going to see that? Was it the storage manager? No wrong spot. I went to click storage Until tom's getting tired right storage manager So even though they have butterfs They don't the raid is managed by The nadm the linux raid management at the bottom So it makes a big difference like because they're doing it that way you don't have to worry as much about The problems that are often seem to be associated with butterfs Which is going to be hey butterfs isn't working because it's managing the drives So they're handling the raid separately and Synology's been working with butterfs long enough And the fact that we haven't had problems and failures through a numerous versions of upgrades From Synology. I really feel that like they've got a good grasp on what they're doing with it You've had excellent luck with qnap and I would call that luck qnap has been a disaster when it comes to Security they've they've been really bad about it. I covered the fact that they even ignored security researchers Which really was annoying? Like when someone's telling you in really trying to get a hold of you That your your stuff's broken and you are not answering them in a reasonable way. Yeah That's definitely a challenge that with the qnaps. I'm not a big fan of them for security I don't know much about their software. We just don't see that many of them Hyper backup for business s3 is a solid solution. I've deployed I have deployed many that way. Yes You know the Synology storage themselves if you if you use the Synology's Was it c2? Synology c2 works well I'm thrilled my my Synology just it just runs active backup for business It runs some of the backups for all of my gmail and everything else. It just works really well As a matter of fact, it backs up my studio computer. I No problems there at all So if I need to do anything in my studio computer if I need to restore it or anything I completely trust it my gaming system, which I'd never turn on bring out Was last time my gaming system was on 1014 so a few weeks ago I booted up my gaming computer and it backed it up Hardware spec is much better than Synology and you can run free nas now I've thought about taking a qnap device and running free nas on it Because I think that makes sense. I just don't think the qnap software makes any sense It was the connections per seconds that had the change. Okay Neat. So you got it fixed. That is awesome Yeah, Synology also includes great m355 backup. Yes It's impressive that they can do both of these so well and you can do both of them on one box It's not it's actually not even that taxing to do these on there I'm really under utilizing this Synology. I should I should load more things on it I just don't know what else to load on there. I should you know, I've been wanting to Find a good project to do things like Uh Maybe organize my book library I don't have a ton of digital books But I have enough of them that I could organize them in a better way So there's some things I could probably do cheap usp 2tb as offline storage just easy to scale out uh, I mean usp storage And now wendell did the video and as we mentioned earlier in the stream here where he covered how to use it for Doing um The the external drive with the usb's in it But I don't know that the sticks the usb 2 sticks are going to be the most reliable way to do storage Is s3 secure s3 can be set up securely So that is not is s3 secure did someone configure s3 to be secure or did they Leave the bucket wide open That's usually the bigger challenge Yeah, Synology actually does have uh esxi host backup. So if we go back over here to our active backup Uh, you can do both vSphere and Uh, microsoft hyper v so you can connect it to either one so you can actually connect and add a Uh, vm vmware vSphere system in here. So I think it's just nice that they built that in Yep, you can host the piehole as well I have a portainer system is is one of the reasons I don't have It's set up in my synologies because I'm running portainer But uh, there's a couple write-ups I've seen where you can put portainer Inside of the synology as well I haven't updated my portainer system a little while. It's not running that many different things Tombear went to where did my portain ago? There it is so let me Close too many windows because I'm losing all the things that I have open And then we'll open up this I forgot what even I have in portainer right now probably not much Because I started testing stuff and then I got sidetracked with like a million other things I know I've got uptime kuma running in there Oh, I got chasm and youtube download running inside of here Chasm is pretty neat. I've been talking about doing a video on it. I just never got around to it Jay from learnlinks tv did one Uptime kuma is in here so that The stack was created outside of port yeah Portainer is a little bit confusing because of the way they call them stacks when they're really just Docker compose files. That's a little bit confusing but Your thoughts on using a analogy box for camera system and also smb file share I Keep them separate and the reason why Is remote access so for example If you would like to open up your Synology surveillance station for remote access That's fine, but don't do that on the same box That you're doing really important things on If you're not opening up, it's probably a doable solution But for our clients if we're running You know the Synology surveillance station system We're not going to also run their file server on it because they usually 99% of the time want external access You're opening yourself up a bit now I don't know of any flaws But nobody knows of any flaws until they're found in one of these systems being open so they work well Now you can bounce a Synology surveillance station off of a relay Which will limit some of the exposure, but they'll so limit your bandwidth You kind of should open it up so it has a direct connection to the device There's good and bad with that of course another alternative is using I think you can use tail scale I haven't tried tail scale with surveillance station, but it should work fine But once again if you're opening up the system that runs surveillance station And it's also the same system that runs your file server with your incredibly important things If there's a flaw in surveillance station, it may pivot over to your file server and cause you a headache So it's not something I really recommend Backblaze is used by a lot of people Yeah, my thing is too. There's there's a reason my Portainer instance says you know This deck is limited because I was just building things in docker compose and then I attached portainer to it So yeah, I found docker compose to be so easy. I just haven't really switched it Uh, don't know if you address this I have a whole separate video. I address this in live streams yesterday Look for my yesterday's video on that topic Uh, it is 10 o'clock and I'm getting I'm getting to the end of my day Nobody knows any vulnerabilities until they find them. Yep. Yeah, I'm still using Backblaze myself. It's one that it's still back blaze is really inexpensive for storing things. So storage calculator We have it on her site or was it no I was looking for they have a storage calculator Somewhere did they take their storage calculator down? Huh, I guess they don't have it on here no more. They used to have a comparison to some of the other cloud places For uh, their pricing. I don't see it on here now But it's still It's not that expensive to store things here. Oh, there it is. Just I just had to scroll down What's your monthly downloads? What's your total day of stored? And you can start, you know figuring it out. You have 195 terabytes of data. So s3 is going to be 29 000 dels a year In matter of fact, you know, it's funny just to look at how quickly storing A thousand terabytes of data, I mean This is what makes the uh Back buying 45 drive servers so Cost efficient cloud storage is expensive I mean, you need the off-site backups, but we do have some large clients that replicate between Their sites they have two geographically separate sites and they'll duplicate between them when they have massive amounts of data because it just It's like well Do we put this in by the way? This is also the argument for a colo is do you buy a colo space? For x dollars and stick a 45 drives in there for all of your off-site backups When would you recommend kubernetes over docker? I see you know, I see people talking about the reality is um the problem kubernetes when it fails is really really really hard to fix And when sometimes there's updates it breaks things in kubernetes, which is really really hard to fix How complex do you want things? I always start with the least amount of complexity and go from there so That is that is really What's important to me is that it's a supportable solution That is huge for us as a business We don't sell solutions that are overly complex to support because that means the support will cost more It's not that kubernetes isn't the bad solute is a bad solution. It's just do you need a complicated solution? Someone asked why we deployed cfs for some of the drive systems we did I'm like this is easier support than sef sef is great sef has a lot of complexities, but What the clients demands are and what their needs are this will work perfectly fine and is easier to support Well cloud prices only go one way, you know it drive prices may be dropping but cloud prices are always on their eyes let's see Just got my psns plus or today a little bittersweet given yesterday, but uh, George is a great guy cool I do like the back plays hard drive stats. They're very interesting You should open a colo location. I mean it only takes several million dollars to get one started properly Cloud prices might not go up as fast if we all outlawed physical drive destruction yeah Maybe not Well, I'm gonna end it here is me been going on for two and a half hours and uh, so I think that's enough of Me talking tech. I love talking tech. Don't get me wrong, but there is a point at which I become tired And I'm reaching that point. So usually start my day at 5 a.m. So by 10 I am ready to watch some youtube hit me up in the forums hit me up on the socials I'm not a hard person to find I've left this up the Vlog thursday at larchsystems.com easy if you want to email me Um people who want to just reach out and say hello If you want to hire us larchsystems.com great place to go over there It'll actually the forum takes you over to the cnwr team Which is also part of you know where I'm at. So Absolutely love hearing from all of you check out my other channel business technicalities if you want to talk about the business side of what we do Oh, we put that all on a separate channel And uh, love hearing from all of you leave thoughts and comments down below and thanks. I am I am tired I didn't realize it's like 10 o'clock. I think it's time Also, I did get to do this thing. Oh I'll do the last couple minutes off topic tom Um, what does tom do when he's not? Doing this this is I was it was a happy day I went for a ride in the woods sometimes I go for a walk in the woods But riding my motorcycle in the woods was actually uh today's project. So that was kind of fun um Then I got all muddy and they had to wash my motorcycle. So that's what he did today Uh, all right. Thanks everyone later