 Yeah, that's I. Hey, people tell me if you can hear us. This is amateur hour. I've only done 343 of these. So. So welcome to blog Thursday number 343. Tom and Jason debate about virtualization and tech talk. If you're looking for some merger talk, Lauren stop video slash biz me and Jason been rocking out business videos over on business technicalities. I've made that URL short because I can't spell technicalities consistently. Yeah, I spelling spelling is hard. Yeah, we need AI to spell better for us. If you see me looking over here, this is where the control is we're using our my main camera. That's why we're not sure we're assuming that we're 44 seconds in and people can hear us. So type of message. Hello, a hello. Actually we'll go all the way up to the top here. Good morning. Pushing a little early for you because I know you're in the West Coast area of the world. It's afternoon for people in the UK. Sure, the people in the UK like that is earlier. Yeah. You know, I got to say something. No, I have just a ton of French followers now. My Twitter engagement is a lot of French now and just kind of like, well, that's kind of novel. I, you know, by the way, the, the something we're going to talk to is XP and G. They're also a French company. So is CrowdSec. And there's one other big company that we started working with that's French. That's open source. Seems like a lot of open source stuff coming out of France, which I think is cool. It's, it's funny that we're both in hunter shirts. Oh, yeah, yeah. That's completely coincidence. We didn't plan. Yeah. Can you run your fingers to your hair? So this started as an accident was one, I was since I injured my arm and I can be out of the sling because I can support it. I can't reach though to put a ponytail. So I let my hair down and then everyone's asking me questions. You'll get PTI presumed or improve the range of motion. Yes. Yeah, I can do this now and it doesn't hurt as bad. I've been doing that. Actually, if you look at this little PT sheet of do this with your hand, do this. So yeah, I got all kinds of PT stuff I'm doing. For sure. All right. First couple of questions for people that actually me throw this banner up on there because I always encourage people. I've been, I've even been replying to some of the emails because I set this up as a separate email, which is the blog Thursday at LawrenceSystems.com because sometimes people just want to point something out or send something to me. And I know you're not one of the usual spammers because purposely this is not embedded anywhere. So the scraper's okay. They got to scrape it off a screen, which it seems to be slightly more challenging. But if it's not, if it doesn't show up in any of the thumbnails, it has to go through the whole video. Yeah, they're too lazy for that. This any stuff. But so this is a question someone just sent in. Hi, Tom, hope you're doing well. Shoulders doing better? Yes, it is. So they're comparing the tail scale performance tail scale versus zero tier and a number of videos you have been using tail scales, wondering what brought it down to that side of the fence and updated thoughts and any similar systems like Nebula or NetMaker. I'd also be interested to hear cool use cases for mesh VPN overly networks that you can talk about where you're employing them with clients. I might have a couple client videos that I can talk about with that. I will admit I bet Jason hasn't heard of this one. This one is really cool. If you've not, they're branding in, so you tap on string error, throw that in here. NetMaker has really got a pretty website. Yeah. But what I've heard from people is it's kind of a little hit and miss. And I haven't tried it myself. It's a automated system. So you just point to nodes and it automatically builds all the tunnels and builds a wire guard mesh network, which because it's n times n minus one divided by two tunnels you end up with. Yeah. So it's kind of cool. It's open source and you have to build your own. Well, you get to host your own. So good and bad. It's a little bit more work to set up than the other ones like zero to your tail scale, which have their own coordination server. You get to build your coordination server in a cloud so you maintain control of the networks. I think that's a cool feature, but it's also, I don't know. I got to admit for an open source project, that's a website usually don't see one that nice for something that's 100% open source. I just haven't spent a lot of time with it. A few people have told me it's kind of buggy, but I will admit the reason I went for tail scale, a lot of it just has to do with the fact that it's built into PF Sense. And there's kind of a, I don't know if I'm going to talk about this, I don't know if it's worthy of a whole video, but I did switch to a MacBook Air. And yeah, and you like Mac. And yeah, he likes Mac. We're both, in matter of fact, we both have the same Android phone, but we both, I fought Mac. I would have told you I hated it for a long time, but I tried it and I said, Oh God, I like this. Yeah. So for me, I ran a Buntu like I've never really been, I've used Windows systems and I've had to, but I switched to a Buntu and then I honestly got tired of fighting with graphics drivers. So because of that and because of my needs are not that great these days, but I do need Word. I do need those apps that don't run well on Linux. And as a business person, you know, it became, I got tired of messing with it. I need things to just work. And so the problem is with the Buntu and that is I tinker too much and then eventually it doesn't work and then I lose productivity while I'm trying to make it work again. So I switched to Mac to try to kill all that stuff off. It turns out, pretty sure the graphics driver problems don't go away in Mac. It's a hardware problem. It's not actually a driver problem. So I still have GPU crashes and stuff like that that happened. Less so on this M2 one, but it's just a good experience. I'm a FreeBSD guy. The user base, user space binaries for OS X or Mac OS are all FreeBSD forked. Julian, whatever his name is, was a big part of that when it first happened. FreeBSD core contributor got hired by Apple. Yeah. And to kind of bring it back to the VPN side and be having a MacBook Air, which also has tail scale support. So I have tail scale on my MacBook Air. I have tail scale on my PF Sense. And I also have like, you know, if I wanted to watch a movie when I travel, I've been using it. I have tail scale on my phone. You put all this together. And I think tail scale, I'm going to do like a long term review. I wanted to use it for everything for a while and go, damn. And I have zero problems with it. I should do it. I still use, I got one of the humble bundle VPN deals back in the day for like lifetime access for 200 bucks or something. So I've been using that when I need it. But I am definitely curious and will probably end up setting up tail scale at home. But I mean, I'm Maraki here at home. So I'll just set it up behind that. You set it up behind it. But it's one of those ease of use. And we've consulted with a handful of clients on it as far as like commercial solutions. The one that, there's a grocery store chain that we set them up with it. And it was really impressive. And because they had a problem where the install company put the same network at range, so they couldn't VPN. So every store was cookie cutter to same right down to the ranges. So this IT guy that in here, he goes, crap, how do I solve this? Because I need the database backed up between all these two one server turns out an overlay network is a perfect solution because it adds an extra adapter. So it doesn't care. And then they can use the tail scale IP addresses to specifically bind and do the coordination between all the inventory everything. And it built an incredible network. And we've had a few other random consulting calls from just we've talked to the team internally of IT. We didn't do the implementation. We just kind of explained how it worked. They go, Oh, that's it. One of them. I kind of like they watched my video and now didn't need consulting. They go, they almost like couldn't believe it worked that good. They're like, Yeah, it's kind of neat. So I mean, to me, this is just the next evolution of DM VPN, right? So I'm an old Cisco guy by trade. And I we were talking about DM VPN. It's been a CCI topic for eight years now, right? So it's this isn't new. It's just made its way down to the open source world more recently. Yeah, it's pretty neat. Um, hold on, someone had one more question. I want to lose all joking aside, no small business was learning if there is an add on to PF sense that you recommend that I have. As far as like the I have my small business video set up for PF sense, I think it's from 2021. Those still apply load PF blocker. Be careful with Saracada false positives. We literally got burned yesterday. We burned ourselves. All right. There's, um, Snort actually has a little bit better of a rule set now because if you snort, they have a pull down in PF sense that just automates it to like it says real aggressive as in spend all day tracing false positives or not so aggressive. But PF blockers not bad. I specifically my PF blocker, I call out like, Hey, block all the tour sites. Um, and I can't remember CIS army has their list. There's a few lists and I have a video called blocking threats of PF blocker. I did in late 2022 or early 2023, but the list haven't changed. Well, the list have been updated with the names of the list the same. So those are the two things that recommend as first packages for PF sense for small business. What are your thoughts on paid like emerging threats or some of the other paid. Yeah, those are, I think those are good. Um, but for small business limited protection most of the time is business email compromise and silliness. So they're cool, but I think they're for like a small business. I think they're like 80 bucks a year or something. They're not expensive. Yeah. And it does right. Emerging threats obviously it's a group. I think it's Talos, right? Isn't that a lot of it comes out of Talos and they are on top of and yeah, a lot of it's probably not aimed at small businesses, but it will definitely catch it. If you're getting hit by an ODA that has been caught. Yeah. And hopefully if you're small business, you don't have any ports open. So it's just less of a concern in general. That's usually not true. But yeah. In a perfect world, you won't have any ports open. The second question came in and I this is one of those wording things of what people said versus what they want. And the way they worded it's a little funny, but this may trigger some things in people's heads to understand the question better. Is it possible to route multiple private IPs on the same port on the public IP of PF Sense? And if the PF Sense has supported what option would you suggest? Well, you're asking for probably more like HA proxy where you will set up a proxy that redirects on the back end to get the front end where it needs to be. So 443 is all I have open, but depending on the matching URL, and I do have three tutorials on HA proxy, even including one for doing wildcards, and the third one is kind of not a tutorial. It's all the things you probably did wrong. And of course that starts with fixing DNS, because most of the consulting we do on HA proxy is your DNS was wrong, your matching was wrong, you misspelled the URL, that's a common one, because it has to be that URL. Host headers. Yeah, the host headers, the SNI has to match. If you didn't do the matching, you paste it in wrong, or you're connecting to something that has a self-signed certificate, you didn't check that box of don't validate certificate, because HA proxy by default wants to validate certificates on the inside, and of course they're probably not valid search, that's why you're using HA proxy. Yeah, we use, I mean I've got good amount of HA proxy experience, but I tend to use Nginx for that, but they both work fine here. Yeah, if what you're trying to do is web, that's what you want to do. I mean you can do, so the NAT rules, the back and underlying NAT system supports a combination of source and destination IP, so you could have this source IP on this port gets this, this source IP on this port gets that. I'm not sure that PSense exposes that in a way that you can actually do it in the UI, and the troubleshooting on that's impossible. Yeah, it can be tricky. Don't do it. Yeah. Tried Linux on a video creative, music didn't, it's, I've been, since it was painful 10 years ago, I switched to Linux on the desktop. That was a lot of work back then. Here in 2023, I barely ever fire up my Windows virtual machine, it just rarely happens. DaVinci Resolve is my go-to, and DaVinci Resolve is actually, if you were to talk to them, they'll push you towards either a Mac or a Linux version over a Windows one. They really like and they work so well on the, the new M2s and things like that, like it works awesome. I'm in the process of migrating. Proxbox appeared DB and 12 LXD, now that it's in Bookworm Prospery. Maybe it's still learning, but looks promising. Is it LXD or LXD containers you can do in Proxbox, do you know? LXC. LXC. Yeah, I think Jay from Learn Linux TV has a video on LXD. I'm not using it, so I don't have, I like containerization, but I'm not, I'm not up on the latest terms of this question here with any comments on it. So, yeah. IPv6, man. It's the year of IPv6 for the last 20 years. Yeah. I remember when it came out. I know, like my hair wasn't gray when IPv6 came out. I mean, I still, so I mean, I think the biggest adoption problem there is I still come across plenty of hardware that just doesn't support it, or if it does, it doesn't hardware switch it, right? So it's like the performance of throughput on, we'd have to look at like the NETK firewalls. I'm curious to see how much of performance hit it is to go to IPv6, because some of the network cards don't do TCP offload. They don't do a lot of the features that you get with IPv4. Yeah. So you end up giving out a lot of performance because a lot of it goes to the CPU then. Yeah. It just becomes a problem. Home Assistant, Zigbee MQTT XCPNG. Well, the problem with XCPNG, and it's not a problem, it's just extra work, is if you're passing through one of these Zigbee MQTT, which happened to happen in my hand, you end up with a problem of USB pass through, mapping it. It's not, there's documentation how to do it. I have not, I don't have a documented video, but there is documentation in their forums about it. I do like it on a Raspberry Pi. That's my preference. A Raspberry Pi worked pretty well for this because it's always on, and you can use things like Wake On Land. I have a lot of my systems. I just power all off, and then from my phone, I have it connected to Home Assistant, and it sends a Wake On Land packet and wakes up all the systems. That's how I turn everything on that's here. But I really like doing it on the Zigbee and Z-Wave. And this little device, has Zigbee and Z-Wave on one USB device. It's like 30 bucks. And I've put, I've already been replacing light switches and everything in my house on this. I have an upcoming video on Home Assistant, but if you go to kit.co.slashlaurancesystems, and it's linked in the description, I have an entire parts list for all my Home Assistant stuff. So if you want to copy my build, even though I haven't done the video, I did do the, I did do the part where all the things I know that work with Home Assistant, because I think that's the hardest part. Home Assistant, stupid easy to load. The next 25 hours of labor is all, does it work with this switch? Does it work with that switch? How do I get this configured? It's not the loader receiver. And it works, but it's not a great experience. Yeah. So, yeah. Figuring out which one's the right one. All right. This is on my to-list. Check MK. I'm going to look at that sometime in the future. I've used it extensively. Oh, really? Yeah. Yeah. We've deployed it in production a number of times. It's cool. It does, if you're going to run Nagios, run Check MK. Yeah. It comes with a bunch of tests out of the box. It's pretty easy to configure. The UI is kind of gross, but I mean, that's part of the course for all of those monitoring solutions. Yeah. I'm a fan. Cool. So, he has more experience than me. So, check MK. Thumbs up from Jason over here. Well, let's see. One of your recent videos, XTP and G, we're talking about a new utility to migrate from VMware. Mention you made to a video on the process, any chance that's on the roadmap? Yes. Absolutely. Me and Jason, now that we've got the merger done, like we have to do business things so we can afford our lifestyles and buy servers that we can load XTP and G on and load VMware on and do migrations. That's absolutely. Matter of fact, Jason's, me and him, we're going to coordinate on a video for how to do upgrades. Jason has a whole, in his email, I said it looked like a Windows activation key. He's got all the VMware certs. And so do several people at CNWR. So, there's a lot of VMware knowledge that I don't have, that the merger has now brought and we will bring to this channel. So, yeah, VMware videos are in the future. And of course, that also means VMware migration videos are in the future. But the tool itself has had several updates. They now can support migrations, warm migrations, incremental migrations. They added a whole, like they've, this has been what they said is, it removes the pain points. I talked to Oliver and his team and it just removes the pain point for people go, I really want XTP and G, but I'm all in this environment here and it's always supported importing like the VMDK. Well, you know, I think you had to use QCOW. You had to use. There was conversions. Yeah, the QEM. They automated all the conversion. Yeah, they converted. So, you can point it and suck all the VMs out and bring them over. And they've got some really, they've really enhanced the tooling on that. So, yeah, video will be in the future for that. Newer to virtualization, how hard or straightforward is it? Or even possible to copy move a 2012 VM server to a new server? There's no cluster. Like you want to go P2V, physical or virtual? Virtualized, yeah. Yeah. I have a video on how to do P2V. When you go to P2V it back kind of the same hardware. Remember, you need to shrink the hard drive first because otherwise it won't fit. Bryce, that's a call out to you if you're here watching because you had to save that day. Yeah. OpenSense has NGINX package with configurable security rules. Interesting. Oh, so a WAF. Okay. Yeah. I mean, I run, we tend to, I tend to run mod security for that kind of stuff with NGINX. Yeah, Xavix, I don't use it as much anymore. As a matter of fact, if I'll share this tab, there's some little point out that, hey, Tom, your dashboard's old. You're running an old version of Xavix. And it's up to date in terms of security. But it's because we used to use it to monitor all of our PF senses. Now I just use it to monitor mine. It was like we tried it, we set stuff up, and it was novel and now we don't. Yeah. I prefer check-in-pay over it, honestly. Yeah. And that's why I want to look at it. I guess to go some of the updates since Xavix is kind of breaking things and I need to rebuild it to the new version to stop that, but I don't care enough. Because it's just monitoring, it's monitoring my PF senses and that's it. Let's see here. How do you set up a clinic, set up a client to get a PM with audio? I think PMC supports it. I would use... Don't use VNC. Yeah. Don't use, don't use VNC. What's that other tool? X to go. Yeah. X to go is way better. Okay. For your RDP will work. I think some variants of VNC, real VNC or tired, one of the VNC variants, I think supports audio, but from a security standpoint, don't use VNC. Yeah. And when we get around to it, once we get the debrief done, we want to talk about VNC and the security incident with a client. That's going to be a debrief video we do at some point. So VMware versus the XSI, we're going to do a more concise, not just a live stream on this, we'll do a more concise video because this obviously is cool and I talked about this before, but it doesn't really tell the whole story. So me and Jason, we'll talk about some of the nuanced differences between this. I think a lot of it might be because I'm bringing XCP and GE to the CNWR team. So Jason's not as familiar with it as I am, of course, and I'm learning the VMware from him and together we want to make some videos to kind of talk about the comparative differences. We'll do that in a more recorded, organized format, but what's your initial take, though, is everything's in a weird place. Everything's in a weird place. So I told you this when we talked about doing this today that some of my experience is poisoned by a Zen experience on an Oracle data appliance. They've since moved to KVM for the virtualization in more modern versions of it, but the ODA on top of it is disgusting. It's a giant conglomeration of really crappy Perl scripts that configure all the Zen stuff and it's running an old enough version of Zen that a lot of the features just aren't there. So that poisoned me on it a little bit. So I've been running, I've got XCP and GE running at home for a bit now. I have, so I am now running Proxmox and XCP and GE at the house. I have not migrated my Proxmox VMs over to XCP and GE, although I'm going to, right? That's, that's my plan is to build a two-note cluster there. It works. Some, there's, I, there's some oddities about it. And as you said, some of them are just a lack of, oh, that's just done differently there. And some of them have to do with how pooling and live migrations work. It's just, I'm used to basically being able to like, pull the VM and just drag it to the new host. And it pops up with the migration screen, click, click next and you're done. Yeah. And it's all done in XCP and GE like this. I pulled something just so you can see there's, I ran H top, just so you can see the VMs not restarting or anything like that, but we can live migrated over here to the other one, select a migration network. I'm going to leave that alone because I'll let it do that. But you can, if you have a specific way you want to do it, but we can do that and we'll go here. Takes, this is a 25 gig interconnect. So it migrated within, what was that about 10 seconds? Yeah, how much land does that host have? Not much. Yeah. That's, I mean, that's going to be the limiting factor is you got to basically snapshot memory and copied over four gigs. Yeah. Hold on. Let me probably not four gigs and use. Let me try something here. Let's actually do this. I just, I got a hike. It's going to show my email address. There's a way I want to change the filter view on this, change this, and we want to change it to defaulting to the lab. Actually YouTube. There we go. Then we'll go, that was just YouTube. There we go. Then we'll add the YouTube tag. There. Now when I click home, I don't have everything on there. Security by obscurity. Yeah. Security. I just don't feel like, I don't want to click on the wrong one. This is a production system. There's like people working that will stop working if Tom screws up. But let's do this here. We'll stop the VM. Well, I could do it without stopping it, I think. Well, no, I got to see what the lamp ram is set to. Let's give this 32 gigs of RAM. Now actually make it use 32 gigs of RAM. So run like a sort of benchmark or something on it. Yeah. Actually, you know what? Do I have a pharaonic system? No, that's on another server. What is that tool? Stress will do that, right? I think so. Yeah. So 32 gigs. Actually, how much RAM do we have on this host? Let's use more. How much free do we have, I should say? That's the number. Yeah, you're fine. 32 is fine because we're going to be, so you're going to have to transfer 32 gigs of RAM. So I mean, that should happen in a best case scenario. You should be able to transfer that in about 12 seconds if you map that out. Yeah, but we'll give it 40 gigs. Yeah. 50 would be the number because it's two times you're in the bandwidth, but it's fine. 40 is fine. 40 is plenty. Yeah, and it does have the ability to do dynamic on here. So that is actually an option. I've got a whole video about how the dynamics works. It's committed, right? So in VMware, that's a reservation is what we call it in VMware learning. Yep. So you should pull that comment down. Oh yeah, probably. So when you do the comparison, you should include Jays, Proxmox sites. You get all three. That would be a good thing. I'm sure I could grab one of our guys in the office so we could even throw Hyper-V into the mix. So for what it's worth, I load Hyper-V. Me too. We've used it. We've got several client systems that are on it. Every time I use it, I swear a lot. And I'm like, they took everything from VMware and made it dumber. Yeah, I don't see it as a long term. Like I feel like at some point, Microsoft is just going to drop it. Well, so it depends on how you install it, right? So if you install it with just core, I don't consider it a type one hypervisor, right? Because it's a full windows install underneath it. But there is a way to install it where that's not the case. Try it. I thought VM state, VM heap. Sure, if there's a, yeah. You want it. You want the M, yeah. Yeah, dash M. So how many CPUs that add to? Give it like eight if there's two CPUs out. Yeah, but M is for memory. So we get a C. Well, you got to give VM bytes. So you want to give it dash FVM bytes and give it like 32 or gig or... YG. Type in bytes wrong. 32 capital G. 32 G. Well, I should keep that box busy. Did I do it wrong? So you want it like that? No. Or do I not need the M? I just put VM bytes. Oh. VM dash bytes. Yeah, it's dash bytes. Capital G. It's just down. It's just capital G. Yeah. Just work with it. No, it doesn't work with lowercase G. So much trouble. All right, we're back. Oh, geez. I clicked the wrong button. All right. The good news is this stayed in the show. Is this stressNG? Yeah. Okay. Does it need a time? Give it like dash C2. Yeah, it does. It needs the number of CPUs I'd give it a dash. No, we don't want it. I don't care about it. Yeah, it works. IO would be fun too. Yeah. So give it like dash IA10. Yep. Let me change this to a G. It needed a time after it. Yeah. Is that all we're going to run for 10 seconds then? Yeah, I just want to make sure it runs. Okay. And actually it uses the memory. We'll do it live. We'll do it live. Figure out how this thing works. You got to put our back on our comments view there so we can see people. Are you there? Uh, we are back. We hope. We hope. All right. It did a thing. It did a thing. Did it use memory? No. It did use CPU, but it didn't actually consume the memory. So let's figure out why it didn't. The end bytes. Do we have to go to dash M? Also? Maybe. Maybe we need to dash M. Yeah, go to dash M1. All right. Because we didn't have, we didn't give any workers doing memory. So we'll set it for a longer time and actually do the migration. Yeah, that's a more fair test. Although I still don't see a lot of memory usage. Yeah, we don't see the RAM usage. Run each top in another tab over there. Well, run it for like a minute so we can see it. Oh, 60. So M, I don't know, let's put 8. Does that help? Hogs to CPU 8 VMs fail. Out of memory. Okay. So we can't do 8 processes each consuming 32 gigs of RAM. Yeah. Oh, I saw a spike in something, disk. Yeah. Okay. So it's definitely spiking the disk. Yeah. A lot. So we're hammering on the drive, but we aren't seeing the memory usage. So is this NFS? Or so one of the questions I had was local storage, right? There didn't seem to be a good way to do a migration that required moving the disk and the host at the same time while it was live. Yeah. So if you want to move this to at the same time, you can. You can migrate this even if we chose a completely different server that's outside the pool, for example. Can you choose a local disk on that or does it all have to be shared storage? Nope. There's a local disk option right here. Okay. So it's on rise in one so I can choose local storage and that'll migrate it to that local storage of that device. But of course, that'll break the ability to move it between there. Okay. So in VMware, I can do both, right? I can tell it to move storage and host at the same time. Yeah. And it will do that. Yeah. Technically, well, let's do that. Let's migrate this over to local storage because it's on rise in one so we can move to rise in one storage without an error. Yep. Oh, no, migrate. VM, we want to migrate storage only. So we do a storage only. So rise in one, hit okay, and it should migrate. Should be reasonably fast. This is not a very big VM. Yeah. Just a base install of Debian. Well, now for that thing still consuming CPU there, or disk there, then it would be, but I think I'd probably stop by now. Well, we can let's do it while it's migrating this here. I'll set it for like 10 minutes. Yeah. All right. So now it's got a high load, which should slow this process down, maybe. It should. It's suddenly got a synchronized drive. Yeah, well, now it's been a long two seconds. Yeah, we messed up the time estimation. We arose it because the stats here say somewhere it's doing something. We got a lot of CPU load. Yep. We just jumped up to four minutes. Yeah. But it's still going. It's still chugging along. It's chugging along, yeah. Yeah, so it'll do that. Can you set the interconnect time between them looks like? Can you monitor that in there? Not in here. It doesn't tell me the interconnect. Okay. Yeah, so you can't monitor like the network usage of the host. Actually, maybe. Of the host. There's a CPU usage of the host. There's a network throughput right down there. Yeah. So you can't actually do it. Oh, internal error. We broke it. We broke it. Oh, internal error. We broke it. We broke it. We broke it already. Yep. It failed to migrate. It's under too high a load. So it probably said there's something going on from what we predicted to what happened. So if we stop it, it'll probably finish migration. No, I bet you canceled it. I bet you the task is gone. Yeah, the task is gone. Yeah. But it did that because it probably it's got some error catching internet. If it predicted something and something changed, it said, all right. So now we did break it live. But moving storage, you can move storage. And once it's on a system, if we were to migrate it over to one of these, when you do the migrate again, it'll automatically move the storage. OK. Because this is on a shared storage and it knows this storage right here is shared. Yeah, it'll just move the CPU. Yeah, it just moves the CPU. So there is a methodology. If I have two standalone hosts with local storage to migrate between them while it's live. OK. Yeah. You also have the ability to do warm migrations. I forgot. I did a video. I haven't done a video on it. But it's a new feature they added. It's a way to coordinate all the migrations in there. And they also have inside here. Yeah, because that's equivalent of going, if you have DRS and going to maintenance vote on VM where it will automatically evacuate everything off the host. Yeah. Yeah. I also have these here where you can build jobs. And when you build the jobs and the scheduling, this will actually let you do things like if you wanted to migrate based on time of day, move things. They've got a whole scheduling engine for this in here. This is something that I think you can do in VMware, but I think they made it kind of slick here because you just create a bunch of time things. You can do some of that stuff in VMware, but only if you have Enterprise Plus licensing. So a lot of the really cool, fun stuff is gated behind Enterprise Plus. It's called DRS there. DRS actually can do cool things like minimize power. So it's like it will migrate everything to various hosts and then shut unused hosts down and then turn them back on when the load. Okay. Yeah. So you can actually bring hosts up and down depending on load. That's kind of cool. Oh, yeah. We're hitting the memory now. Yep. Well, this is what's interesting. When you do the memory management, it compresses it. The ballooning. Yeah. It does the ballooning and sends it back and forth to the minimum use. That's why we're trying to make it consume more memory for doing that. Yeah. I presume it uses a kernel driver to do the ballooning and not just a user space process. Okay. Yep. It's got a kernel driver for it, which is also built into Linux. And the ones from... They're signed if you... Because you can load the Citrix ones. So the weird relationship they still have is Citrix has the signing key and VATES does not. So Citrix drivers work perfectly fine in Windows. Yeah. VATES isn't bothered because basically they don't need to. Like Citrix does it. And Citrix is well supported by us because VATES is the one developing all the TPM stuff that's going into Citrix. So they kind of have an uneasy relationship because essentially this project was spawning off of Citrix trying to turn everything into a license. Yeah. And so they pull a red hat and then it'll be contentious. Yeah. Well, VATES is pretty clear on everything going to be open source. They've a long time open source advocate for things. But yeah, that's... I like the backups being integrated. Yeah. That's neat. It's novel. One, I wish for Veeam support. We can make it work for Veeam support. It's just a little grosser because we are looking. So full disclosure, we're starting to deploy at least for our own internal like BDR devices we put out there. We're... And you know this, we're putting XCPNG on them and managing them that way. We want to back them up. So we're basically running the Veeam agent on the host. Right. And then we can use the local backups in there and then we back the backups up with the Veeam agent. So it's just a two step restore then. I can't directly restore it to hardware that way, which is in the case of these is actually perfectly fine. Yeah. Yeah, maybe and maybe they'll have Veeam support sometime in the future. I don't know if that's on the roadmap. That's a Veeam problem. It's not a... Yes, Veeam has to rate the integration. The good news is Zen's been around long enough to be documented that Veeam support's probably not a big deal. The big advantage is shared storage over live migration with local storage with shared storage and cluster. You can automatically restore Veeam if a host dies, can't do that local. Yeah, I mean... Yeah, HA. Yeah. Yeah, you can have it automatically restart on there. Yeah, I mean, by all means, shared storage is the way to go, right? Like, and they've got like... Oh, the exosan stuff is half baked, I will call it. Yes, that is because exosan 2 is supposed to come out late this year. It might be a third baked. It might not even be half baked. They admit and if you read, they even have a whole write up on how bad the spaghetti code is. Yeah. Their own admission, like this is spaghetti code. Like this doesn't work the way I thought it would and I don't think I would use it the way it actually works. Yeah, but when they come out with exosan 2 later this year is kind of the release for that. They're going to have a new version of Zen. They're going to dark mode. There's the most exciting feature because that white is just blinding. But the exosan V2 is going to be substantially better. Yeah. Like they're also rewriting the exo API. So the underlying storage management is being rewritten too. Yeah, you should have that next question because we can talk about that too, right? Because this is not unique to either one of them. So generally, I do not. I use a dedicated network, at least a VLAN. It's not necessarily dedicated physical infrastructure but all of my iSCSI traffic lives on its own VLAN. Typically on the host has its own network adapters associated with it. If it's 10 gig, that may not be the case. But I generally try to keep storage traffic on its own because you're 100% right that you end up with all the storage traffic leaving the host. You build a storage network. That's what storage networks are for. And it's fine. It works fine. Yeah. Or a new fiber channel. You could definitely do fiber channel. I mean, it's still a thing. And you can pick up cheap fiber channel stuff because no one wants to do it anymore. So you could definitely just use fiber channel too and then it's not hitting the network. I mean, still kind of is. But yeah. And these, if you notice the network here is 192.168.20.225. We have a dedicated storage network at the office as well. It's its own dedicated separate thing because not mixing those. This question has come up a few times because someone asked why I don't define them in PF sense. I'm like, why would I? I do not want my any chance of my storage network routing to the outside world. I do not route the storage network at all. That's just not. It has no default gateway. Yeah. It has no routing device on it. And there's no DHCP on it because I statically set everything. A lot of my networks don't have DHCP, right? Like management networks tend to not have it. If I don't want random hosts appearing on a network, I'm not going to run DHCP. Yeah. It just solves a lot of the problems because we go to the production VMs. They're also split here because if you look at the networks, this is why the XO itself, one leg, this is where it talks all the VMs and moves them around. The 20 network is where it bounces them around for the actual storage. This is how the backups are so fast. It's completely separate storage. It just makes life a lot better. Less risky. Yes. To see everyone features like Serene, Incident, yeah. Yeah. We do sure backup most places, instant virtual machines. I think we have that some places. We do all sorts of application or stuff. That stuff's not that hard to do. It's a little harder. We're doing Oracle backups there too, right? And so it's a little harder because you have to integrate with R-Man to get Oracle backups and the Kriask and all the other stuff that goes along with it. But yeah, if you're using Veeam, you should be using all those features. Yeah. That's part of the beauty of Veeam. We're still using that we brought over the MSP360 because people asked what we're settling on. And honestly, the goals eventually get to one backup, but it's not hard. The MSP360 works. My choice for backups says we hated that one the least for the three woman. It wasn't that I loved it. I tested a Cronus. Did it work? Yes. Is the dashboard horrible? Yes, it's really, really bad. And it's like they turned into an upsell machine. Yeah, no, they did. Yeah, I hate that. You want to use us for cybersecurity? No, like if you're the best in backup, then go do something else, right? But like when you're middle of the pack and backup, like, what are you doing? Right, and Comet backup, I don't know if they were fixed this, but we looked at it and one of the problems, Comet's really cool. I was really close to pulling the trigger, except they have a way, but not way, they don't have a way of getting the new clients on boarded without everyone sharing a key. And I'm like, this is a problem. Like this is a security issue. And because of that, it kind of ruled them out. And it's like, yeah, we've had people mention that before. And I'm like, well, you should probably fix it. Yeah, it probably has to do with signatures and stuff like that. It's probably not easy to do. They have to use subordinate keys and whatever. But what you have to think about is, if someone were to pop one of your clients and everyone had to share a key, they're like, I didn't get a key to this client. I got a key to your clients. So this was actually a problem back when I had my CVE in ConnectWise is that there was a lab tech deployment key that was common and you could get it off any of the hosts and it basically let you register your hosts with lab tech. So they changed it so that is no longer given out willy-nilly. I mean, we'll probably move your clients to Veeam eventually on the grand scheme of former clients, on the grand scheme of processes and tools that we're going to integrate. It's pretty low on the list because what you have is working. It's mostly doesn't require a lot of work to care and feed. There's very few tickets because it really is. I mean, we monitor it, but it's kind of a set it and forget it because the agents work fine. They just work. The setup's not difficult. It's easy to navigate. The dashboard is easy to look at. And yeah, sometimes those are like, it works. We've done restores. We've restored things for clients and it works. We do have one co-managed running Ninja backups, I think, still, but Ninja backups have. So since that, what's the names in charge of it? It's gone through some major refactoring. They feel bad. They're going to watch this video and they're going to see that. I went, oh, when you said it. Oh, I did too. I'm the one that look, I was just honest with them. I said, I can't use this product. I'm on their advisory and go to their counsel stuff and things like that. I told them and they took not just my feedback, but others that had experiences and they really put some work into it. So it's worth revisiting. But yeah, that's a different topic all together. Also, if you want us to, so if I want to do a tools video because there's so many more tools to talk about now. Yeah, I think we have 36 different things users can log into. There's just a lot. And it's, I think people, we want to give you some insight into the back end of running the MSP. And there's just so much, like I think all of you for the kind words of saying Tom had a mature operation, but there wasn't everything being done that could have been done. And so expanding further has been really interesting and insightful from like, hey, look at all the things we do now. I finally, I joked with Sean. I was like, I'm finally a connect-wise customer. Not finally a ninja customer. And he's finally a ninja customer because we like ninja for the RMM, but we manage works. They've done a lot with the managed the way you put nilier on top of it. I think your guys actually are starting to like manage. They do. At least, yeah, the employees like it. I don't hate it. I don't love it, but I don't hate it. You don't have to do a lot of it because I've exempted you from time sheets and all the other things that are, yeah. Yeah. How easy is it to pass you a PCI device next to PNG, next GPU, HPA? It's not hard. The write-up is pretty clear where you LSPCI from DOM0, you find it, you pop it into a list and exclude it so DOM0 has no access to it and you figure out which VM you want to. They do not have, because this request comes up from home users, it is not a priority for them right now to do tooling in XCPNG to allow device pass through. But if you like it better, Unray does a better job of this for home users, and so does TrueNAS, it's actually getting pretty good at it because a lot of people want their TrueNAS scale system and the Kubernetes thing they built, Kubernetes Docker coordination. They actually have pull-downs now, so your cards can be passed right through. I think Proxmox does it too. I think they have a, yeah, I think Proxmox 8 has a UI for it. That's one of those little questions. Yeah, it's one of those things that I think if you're, when you're choosing it just as a home user, you don't care about the scalability and cool things I talk about in XCPNG. It may be that Proxmox is a better fit for you because you're like, I don't want to do that command line thing, Tom. I want to pull down. Yeah, and now as we go through this, some of that stuff is that there's a lot of things that you still have to go back to the command line in. Once you get out of the common use path, you're back to CLI land for a lot of the XCPNG stuff. Yeah, oh, your Raspberry Pi isn't powerful enough. So my suggestion, I did a review of the Zima board. The Zima board is way faster than the Raspberry Pi. It's really cool. Orange Pi, I think is another one, right? Or orange something. So the problem with those, Jeff Gehrling did a good video on this. The problem is support sucks on some of them. Yeah, it does. It does. You just have to accept that. Yeah. If you buy it off of Alibaba, the support's probably not going to be great. But I just did a review. There's a couple of Turing. Turing's one that's stalking me right now, but I think those are only compute modules. I don't think they're full. Yeah, Jay's got a giant Turing Pi setup. Yeah, yeah. That's probably why it's stalking me, because I watch the videos. Yep. So the Zima board is pretty cool. I did a video on this. It's 179 with 8 gigs of RAM, and it's x86. Okay. There was a Rock, was another one. Yeah. The Rock x86. There's a couple of companies that make these. Matter of fact, in the box over there, I just did a review. Those, I can't pick it up to Carrie, so that's, you're going to take that back to the office. Like this? Nope. The other little ones. Yeah, the little ones. Those are those little mini PCs. Those are really awesome. I did a review of both of those. So that's how we're deploying now for BDRs. Yeah. Are those little 1.1 liter PCs? Yep. We're not 1 liter. They're the bigger ones. The one with like 4, 6, 2, and 8 gigs in them. But either way, the Zima board is a good choice for running it. It's got USB 3. It's got a PCIe connector on there. If you don't need 8 gigs, they started like 119 for the less than 8 gig model. So Zima boards are kind of a good alternative to the Pi. Libra is another one that I've seen. Yeah. There's a handful of those mini PCs. Odroid. That's the one I was saying. Odroid. Odroid. Odroid has so many models. Yes. And like basically every configuration you could want. And Patrick from Serve the Home, he's got so many good reviews of some of those boxes. Patrick just dives deep into it. He's awesome at it. We'll answer a few more. Eventually we got to do other work. Proxmox needs better UI for SRIOV. Yeah, probably. I don't know how many home users need it, but... We could talk about that someday. The unnecessary complexity I see in some of these home labs. Yeah. Some of them really insist they need a SEF cluster to run their Plex system. Yeah. And then when it goes down, they put in a consulting request to get it fixed. I'm like, why were you doing that? Yeah. Do we use Teleport to manage NMSC? Not right now. Teleport's pretty cool. I don't know what that is, so I haven't looked at it. You've actually... What is that called? Jumper? I saw it. There you go. Oh, that. Okay. Yeah. I see you've seen this before. Teleport's pretty neat. Christian Lempa. He's my friend from Germany. He's done several good videos on this. What they did was they copied SSH. It's based on SSH wrapped around their Teleport tool. It's all open source. It's really slick how they built it. It's probably just SSH for the wrappers. Yeah, and they're very open about how they do it. It's essentially SSH with wrappers, but from a control standpoint, the thing I don't like about it is an agent, and I don't want to have to run more agents. Yeah. That's what stops me from using this, and there's a couple competing tools for it. I just don't want to load any extra that I don't have to. Yeah. We've been talking about that, even with regards to the XAPNG stuff, because what makes the most sense for us to do is to run a centralized Zen Orchestra box to manage all of these, but I am very afraid of that attack surface. So we have not done that to date. We might we? Yeah, we might. We'll end up tunneling it in, only allowing it from the IPs, access to it from the IPs externally that need access to using some sort of SSH tunneling or a tail scale or some sort of other methodology to get the traffic back, and then very much limiting access to that from internal networks, from trusted hosts and that. So you'll have to go through a jump box to get through it. It's kind of a complicated thing. It's limiting the threat surface as the really big piece that you want to make sure, and of course, back to being agent-based, I don't want to load extra agents, especially on my XAPNG Dom Zero, because now I may have a conflict that causes and breaks something else, so I'm troubleshooting a problem I created. And that's kind of our... There are other ways to solve that. You could just do an SSH with port forwarding. You could do... There's a bunch of ways to do it. Yeah. SSH VPNs are still my favorite combination. Just VPN in, SSH in. Keep it restricted to certain IPs. All right, we've got a few more questions and we'll wind it down here for coming up on the hour. In the hour, yeah. Home users, and I need it, really, to post some Rack and Lab info in the forums or something. Yeah. I mean, so over the years, my Home Lab has gone from a ton of hardware down to... I think I've got three 2U servers running, maybe two 2U and one 1U that run in the Rack, and I could get it all down to one. At some point, I tinker, but I need stuff to just work to, right? So it's like I try to avoid... I have lab environments that become complex, but I'm not running production stuff on them. No. The team, and this is actually cool, I'll pull up their YouTube real quick, because I didn't watch the video yet, but it's 45 drives. That's what I want to do. I want to get that back to go at some point. So the good news is, these are the... Let me find it. Well, I'll just leave the title up. But you can find this video. They had a challenge posted to them in the CEP forums, and it says, can we achieve 10,000 IOPS using CEP and SATA SSDs? But they built this out, and walk you through all the steps they do. This is a long video. How many boxes is that? Just one box? Probably. But what I like is they walk you through how they step through the process, how you build it. They're really good on CEP. So what kills you there in the end is the native command queuing depth of SATA is only two. So you don't get any right combining, and that makes it... I guess for IOPS, that's fine. Were they able to get 10K IOPS to the disks, or was it hitting the memory buffer? You know, I didn't watch the video, so it's not my to-do list. They just posted this. I'm gonna watch this too now. Yeah, and by the way, I need to invite you next time we do a round table with them. I'll bring you on there. They love hearing ideas and stuff throughout them. I won't say, but when Wendell was on there, Wendell has a clever idea that they're gonna build at the lab. It's gonna be part of the Creator Summit that I can't make it to because I canceled my travel plans there, but there's some cool videos that are gonna come out of there. I'm gonna fly out separate to go hang out with the people who are at 45 Drives and go to their building. And maybe when I go, you wanna go too. They were gonna give a whole tour of the facility where they build these things. So... With a rack of 3D printers, because a bunch of the parts seem to be 3D printed, which is kind of cool. Yeah, they're just neat people. They're really, really strong engineers there. But they build some of these ideas out, but they also talk about the complexities of them. And one of the other problems is if you want a good SEP cluster, it's not just SEP, you usually go, hey, I'd like to have several systems clustered together. The interconnection between those becomes the limitation of your speed. Synology actually has a really cool HA system where you can take two systems and create... And I have a analogy at the office that I'm gonna be doing this on, again, because they sent it to me, but it's got dual controllers in there. But the interconnected controllers becomes the limitation for your IAPS when you put it in that mode. That's why things like Infiniband exist, right? Because they're very low latency, very high speed, basically just PCIe interconnects between two systems. Yeah, so that's where people think they want it. But then when you start realizing all the components to make the cluster work, you'll know why we deploy way more ZFS than clusters in SEP. Eventually, our beam target storage will probably move to SEP, honestly, because we are... Yeah, I have so many. I'm up to like three or four relatively largest true-nest core boxes now, right? At some point or another, it makes sense to combine them all into one SEP cluster and just expose a giant NFS volume. The jury's still out. After the more time I spend with the 45 Drive Team, they used to do cluster. And they said, look, I said, cluster was cool, but we had like almost lost clients over this and because it couldn't scale to the performance. And I think they told me like they have over like 20 petabytes of data with a client. It's more like object storage to me, right? It's like if you're storing... It doesn't... It seems to fall apart because we looked at using it for a bigger enterprise-y client of ours that where we have a job batching system that lives into... There's four copies of it running, right? So basically we do blue-green deployment. So we've got the active one, the passive dev one, and then there's the secondary sites. There's four copies of it. We ended up just doing a centralized NFS server because Gluster fell apart with the 200,000 files that live in that one directory, right? It's just it could not handle a high number of inodes in one spot. And that's exactly what their experience was. SEP handles this differently because they have storage spaces that are dedicated to managing that. So you can manage it to say, all right, this is the extra caching. They said they would have with the client that they switched off a Gluster. I think they said they would just like... Just to open up some of the directories was a one to two minute pause. Yeah, yeah. It took... So that's what killed it. We set up this Gluster stuff and then I did LS. And it took literally two and a half minutes to get an LS response. I'm like, yeah, this isn't going to work. Yeah. It's just it's not practical. Yeah. I don't know where you're at, Jason. I think you're in Canada though. So morning. Morning. You said we should chat. You connected with me on LinkedIn. So say hi sometime. If you want to talk about videos and stuff, he's actually done... I think he's got a few proxmox videos that he did. Another YouTuber. Let's see. Two Synology Doctor Hose. I still have H.A. Firewalls, multiple switches, MTSP, for G backup though. Interesting. MSTP versus PBE, RTSP, right? Like I mostly have moved away from MST because of the... Unless it's multi... Unless MSTP isn't what I'm referring to. Yeah. To me it's multi-spanning tree protocol. Is it multi-spanning tree protocol? We're guessing... We have a lot of acronym collision that we're going to do it. Yeah. Because I think that's just called MST in most vendors' worlds. Yes. You can run up to 16 spanning tree instances and then you just map VLANs to them. Yeah. So you're saying the thought of cluster storage has lost its luster? Yes. Yeah. I mean, I think for certain workloads if you have a lot of large files for a video editing workflow, it might actually work fine because you're dealing with few large files. It seems to be a problem with metadata essentially and how long it takes to do the metadata lookups and collisions. And I think Ceph is a little bit different because it's more like block storage than it is individual. It's just different the way it's architected so it's less of a problem. Yeah. And the way TrueNAS is doing it, people aren't going to like... I've talked about this because the number of people ask me about it but then they realize how they implemented it. So TrueNAS scale and we'll pull up their site because I'll show you how the wording is. I like TrueNAS scale. It's not a dig at scale, but it's an implementation dig. So, you know, they're building this whole scale out structure which is cool. And they talk about it, but, but, but, Docker, VM, blah, blah, blah. The way they do this is the only way to make this work is through their... You have to license their cloud controller. There's no interface for Gluster in TrueNAS scale. So this is where you're... We're going to split hairs about open source. TrueNAS scale, the open source server still is built on open source and Gluster's open source. They're just not going to give you a management interface for it. You have to buy their close source. Yeah. Okay. So that's how they're getting around the fact that it's open source. The board is going to cloud manage this and then it's not bound by any of these licenses. Interesting. Yeah. So you go through and you buy a license for the coordination server and then you can host it yourself. They'll host it for you. You can host it yourself. You can host it on a TrueNAS scale system. It's just a VM that spins up that connects through APIs that will talk to this. And it's kind of cool. It gives you more than just... What is it called? True command. They have a whole core enterprise. True command. So the True command system is what manages all this and it connects all of them. It gives you a cool dashboard. You can connect all your TrueNAS systems to it. It's got a lot of features. Okay. So we have a request from somebody that we're currently trying to fulfill right now that wants to not run multiple... Will this talk to core or only scale? No, both. Okay. That might be a solution to the thing we were just talking about. Oh, yeah. And this is what... Well, if he wants to use it, this is what he has to run. As opposed to using Gluster. Yes. Well, no, this coordinates the Gluster. Okay. So it's still Gluster on the back end, but this is the Web UI for it. Okay. Will it do just... If you don't want to do Gluster, it will do centralized management of just the TrueNAS... I mean, that may be all easy. After, honestly. Yeah. So it's cool. I actually demoed it when it was in the beta and I didn't like it enough at the time. And I was very honest with them and there's a reason there's not a video. I'm like, sorry guys, I'm not going to do a video because you won't like it as I told them. And I'm good friends with the people that I existence were a reseller. I said, when it gets more polished, it's actually got a lot more polished. Last time I looked at it, it looks good. They did a nice job. The way you... Like, it's pretty now. It's not like this... It felt... It looks like TrueNAS. Yeah, it looks like TrueNAS. They found a common theme. They've put a lot of things there. They're very open about the pricing on it. The subscription pricing. And it all is subscription. But this is what I know the homeland people are going, but Tom, I thought we were going to get Gluster with TrueNAS scale. Like, you are. As long as you buy the license for this. It's not expensive. You already could have Gluster with TrueNAS scale. Like even... You just don't get a UI around it. Yeah. You just have to run it off of the command line. So that's the one thing is a little bit... And the only place that even has any UI for Cep is the team at 45 Drives. They built some that are pretty cool. They're CepDeploy. They have a video on... I think they call it like CepDeploy, which is... They built it on top of Cockpit. It's pretty cool. I like where they're going with it. I think there are other Cep UI front-ends too that do exist. Okay. I've not looked... Besides them, I haven't seen many people doing it. It's not... Almost anyone who does Cep, like it's used all the time. It's a really popular system. It came from the open stack world. So because of that, there's open stack stuff that supports it. It's going to be your DevOps engineers, fingers on keyboard, punching away at it. It's... And those people don't want a web UI that would slow them down and remove the automation. So... Yeah. Most of those things are not intended to be... They're intended to be configured with automation, not with humans. Right. So that's why there's no UIs for them. Exactly. All right. We'll spend an hour. Thank you all for joining us. This was fun. As I said in the beginning, Lawrence.video.biz, if you want... We've been doing a lot of talk about the merger, all the behind the scenes on that, for those who are curious. We will be covering some of the VMware stuff, though, more in-depth. I'm excited to argue with Jason about VMware. I'm probably wrong about a few things. I've used VMware. I'm not completely blind to it, but he knows it way better than me. But we want to do more XCPNG as well. So... Yeah. Fun stuff. And we want to do that VMware upgrade video. I think that makes sense. I put that in our list of ideas. That one we should start getting done quick. So I'll get the groundwork laid for it. I'll build a couple of VMs. We're going to do it in VMs first, and then if they want to tinker on hardware later, they can. But that's like, I want to do it on hardware. I'm like, no, you don't, because it's like an hour to reset it between every time they play it. Yeah. Play, getting with it. And that's when I build my scripts for my videos. Like, I always tell people, if people watch me work, you would know I'm not as good as I look on video. I'm editing out all the bad. Seriously. There's like 10 times Tom went through this, Tom typoed it. Do you think I leave the typos in? I do. Yeah. When I do them. Yeah. Because I hate editing. Yeah. Editing. Editing is hard. Matter of fact, editing, though, is how you learn how to shoot better. And because once you've made a lot of mistakes, you try really hard not to make them. I'll just rerecord things rather than edit out the mistakes. I recorded the video that's uploaded right now that I'll get published tonight. I recorded it twice, because I realized how many mistakes I made. And I'm like, I ain't editing that. It's actually easier for me to read. It's only a five minute video. So it's like, it's easier for me to say this again in five minutes and spend 20 minutes editing out the things I said wrong. All right, man. Thanks, everyone. See you.