 Hey, we are live welcome rich. How you doing, man? I'm doing well Tom. How are you sir? It going great This is a Follow-up if you will rich just finished an amazing series that I watched all of Especially the Nutanix one we'll get to that in a moment here because that one that one was surprising I didn't realize how that worked But rich did a whole series on all the different alternatives compared to your background in vmware You covered proxmox xcp and g new tanix hyper v which I don't think anyone I mean there's at least a few videos you'll find but I don't think anyone else Did hyper v so thank you for doing that It was an interesting I mean the hyper v was probably one of the more interesting experiences just from the fact that I Strongly don't like hyper v. I think in the beginning of that video I was even like hey, I gotta lay my cards on the table here. I am not a fan of this software Um, and you know, there's there's redeeming qualities to all of them and but some less than others I was surprised to see the amount of like upset hyper v people that I made Yeah, which is you know, that was good. But so learn that was a fun experience um The Nutanix one I was I think the last in your series and by the way that series is linked in the playlist down below So you can watch all of them including a talk me and you did just in general about open source hypervisors One of the things about the Nutanix one that really stood out and all these have nice little screenshots back and forth So you you don't just talk about the product. You actually give us a demonstration of how it works You took the time to set these up and play with them Uh, but the Nutanix one so people can know right away because I think this will knock it off that list one no pass through to um Hyper converge or die. That's it. That's the only option you have Yeah, uh, which I found kind of interesting and I brought this up to the 45 drives people I did a live stream with them yesterday. We're talking about proxmox next cpng and I was like, yeah, they're not friendly You're 45 drives. They don't support anything with their own hardware. Like there's no Using a storage or an important variables. Well, they're no friend of ours Yeah, I was at a conference just a couple weeks back and there were a bunch of storage vendors there and I was having lots of conversations about you know, the future post bmware with a lot of these guys And you know, a lot of them have strong opinions about what's what they think is going to be the next big replacement for vmware, but at the time none of them Mentioned Nutanix at all and I I didn't bring it up to them because at that point I hadn't gone Deep and tried to figure out, you know the ins and outs Nutanix wasn't later I'm like, oh that's because they have no play here because there's literally no value in them trying to sell a shelf when You know, you can't use the sand with it. So it was an eye-opening experience for sure Yeah, the the other interesting thing. Um, I thought this was a good It was an opinion piece that was in a register. I didn't read all of it But I got the headline and I think that they got they did nail it on this one They said, hey, you said other large scale companies looking at what hock tan got away with And wondering how much they can get away with And it wouldn't shock me if some of these other companies go Well, if we're popular and we're locked in how much more can we raise things? This is not just about hypervisor This is about, you know, your big iron that runs because the hyper is all going to connect somewhere like to all the switches And everything else how much how much were those companies go? Well, guess what you're not switching out these nested switches You're not moving away from all the Cisco gear Right. Can we just add that much more to the license? It does feel like we might be At the beginning of a major shift and change I saw even in our my discord people were upset about the esu changes that microsoft made for windows 10, right? And the the 60 dollars per device for the first year 120 And then just keep stepping up and up like it does feel like there is A realization in business that hey, maybe we can put the screws to people and they don't have a choice because we've got these Not monopolies necessarily, but these really solid products have great products and not a whole lot alternatives to switch from Yeah, you know, we were one of the things I was pricing out because we we have so many people and you know My bias fully disclose is going to be towards xcp and g because this is something we're using or Doing production Systems with this and just before we started the life You know as you're talking we just built a bigger lab at our company cnwr just for Teaching more of our staff. We got 17 techs and we're training More of them up on how to do these migrations walking them through the process We were we have a bid we're submitting. I worked on the solutions designed with this It's all you know nice high-end sysco switches which wow the licenses I was like, oh, this is like 10,000. They're like, oh tom. That's not that that it don't work for 10,000 That's just a switch That's the that's the metal box, sir. That's the box. There's a license they have to put on there Oh, it's 27,000 now the license. I couldn't believe that's like once we added for every all the features they wanted because this is Five well just under 600 vms. We're looking at high density super micro servers We'll be in the quote I mean this is a half million dollar quote and then there's the labor on top to migrate all the vms And we've already done this for some of the companies. We've been doing it for more and more bigger ones This one we're doing some of the lift, but it's common these companies They we some moved a while ago and they kind of have to because they said it our business model because they host and resell That service they wrote some custom software each one has to run in its own instance And they said yeah vmware is not making it tenable We would have to pass that cost along to our clients that is substantially higher versus xcp and g is a much more affordable uh choice for us and they Is one of the ones you reviewed and I think you'd like you'd said if you're coming from the vmware world That concept of how it works is kind of there Yeah, and you know you're speaking of the uh those uh third-party sellers of Services, I sure you saw in the news about how in europe There's starting to be some rumblings about how this change in licensing for these uh companies from vmware is essentially Would put them out of business And so there's hope I wouldn't I wouldn't bet on it But there's hope that there'll be some investigations into whether their pricing is is appropriate for resellers of of Systems, right? Yeah, I mean because they're essentially they're not a monopoly but when you have only a couple companies colluding It's just I believe the term is oligopy. You have this small company The small um doing it. It's still it's not like a company doing it But if they collude didn't say well, we're the only people that sell 100 gig switches Uh, you can choose from one of these three vendors and we've all to set down it and while we were playing golf together Decided right how much we charge Exactly Yeah, they're they might look like they're independent companies But they they all play at the same golf courses and they I mean everybody has back channel communication So you know that they're talking about these changes Yeah, and they're there's somebody in golf. Uh, they're all playing so hot tans going. Hey, you know what? You know how I solved my my deficit. I mean, I just raised the prices. It turns out Yeah, just do what he did, right? You just raise prices and you make more money. It's weird I can see him and michael dell on the golf course. Oh, yeah Right It's bringing it back to hyper v for a moment because one of the things I want to point out and one of the reasons we Kind of don't like hyper v as much and I think you highlighted some of this But the dependency it has on windows. There was even a february 2024 update that specifically borked hyper v Then in march, we had the domain controller problem that was causing a memory leak that also affected a host that run hyper v I I just Don't believe microsoft releases the most stable builds when it comes to their updates and this becomes a problem So many people will answer will don't update until several months But if there's a security update this falls out of my own compliance that i'm Providing for my client even if we did isolate the base os and things like that that's still not the best way to do it Um, it's one of the reasons I think hyper v is just kind of out because for all those flaws with you know The pricing with vmware updates your vmware don't break the vm's. It's typically right. Yeah, typically it's not I mean there's it and we're talking over years. There's very few times you can point to I'll say the same for xcp and g i'm less familiar of proxmox I do know there was when i'm going version seven to eight people had some networking issues in proxmox The proxmox is just debbie and underneath so if you have a good understanding of linux You can work through those networking issues that came with it Yeah, I you know that's always been the the crux of windows right is that And with hyper v i said in the video too like i always kind of felt like for microsoft hyper v was Something that they did because they didn't want to be left behind during that era when virtualization or data center virtualization was becoming a thing and Even when looking at how you manage it and the expectations between using like system center sc vmm for example, which is a beast into itself for practically the only Reasonable way that an enterprise could manage it down to just the simple Virtual machine manager that they provide which is just mmc. It just it never felt like they gave it the love I do think that there is potentially a use case for it for people who Are microsoft shops microsoft shops only mind you i think that that's a that's a key one Yeah but they're looking at Minimizing their on-premise and shifting to like azure stack hci and you know getting their on-premise workloads because Off to azure because in the end i think that's basically where that stuff will and microsoft will eventually make it so that it's just a fluid Way to transfer on-premise workloads into azure and then you're paying for your consumption fees in azure instead of doing it on premise Yeah, hyper v is the gateway drug to azure yeah I remember a time when it wasn't that way and boy do i miss those times Yeah, i i don't pretty i mean we know there's a long life cycle So it's not a direct concern that microsoft is going to drop support for hyper v Isn't i think it would support as well 2028 the current iterations There's no and all the research i did there's no indication that microsoft's going to stop using hyper v In or providing hyper v as part of the windowserver os is it just seems that there's a lot of Engineering efforts going on to kind of merge between the the hot you know We'll call it multi cloud or you know from your private cloud to azure But that integration in azure is where they they see the the real benefit from but the free hyper v version is Done as of like server 2019, but it's i think it's supported up until 2027 2029 so like that Yeah, i i love because there's a comment from a microsoft employee And i i think i brought this up in one of my older videos On this related topic that they said we don't have the resources to put into supporting a free version and i'm like Your company's worth how many trillion you don't have the you don't have the desire We'll just use that Yeah, you are sitting on cash on hand more than any other company pretty much besides maybe apple and Oh, no, we can't we can't afford to support it man. It would just it would bankrupt us Yeah Yeah, that that's all political decisions inside the the walls of microsoft, right? And and being that it's closed source and all the reasons i put the word open source alternatives in here is the uh, the nice thing about open sourcing Go back to the xcp gene proxmox for example The fact that they fully open source this and Citrix is an example for those that don't know the history Zen server is the first open source hypervisor and it's part of the linux foundation zen the hyper zen the hypervisor embedded in the kernel zen server and some of the various Renaming that citrix did citrix send server as they may have called it Citrix became essentially a steward of the project because it's like anything It's not that you just need the hypervisor. You kind of need some tooling around it So they put together all the tooling around it. Uh, they were not a great steward of that open source project That's how we got xcp and g in 2017 citrix made some really horrible decisions That really made the community quite angry But this is the beauty of open source. Well, it provided there's enough community effort around something you can fork that project Take all that source code and build it again and build it into the product that you want And they've completely removed. There's still the remnants of it in there if you look Some of the licensing stuff, but they're all blank It's all the licenses unlimited because they took out all the Artificial requirements that kept getting added in by the folks at citrix is unwound all of that and have their own spin So when you look at things like xcp a shop proxmox They give me a little bit more confidence for a better future because of their open source nature Versus like new tanix, like I mentioned, uh, it's proprietary. It's a public iterated company They're a company that is also if you google new tanix for sale I've commented There's non tech wall street articles You'll find where the company has been trying to find new investors or perhaps a buyer And uh, that is how the rug gets pulled out to you. This is what led to that with vmware. So Yeah, and you're a hundred percent right like in in my personal feelings I do Maybe this is partially My my wishes and wants for the world, but I want to see The open source hypervisors succeed and Not just succeed, but also like just supersede the closed source stuff when it comes to virtualization because You know, we with with zen with uh with kvm We certainly have great foundations already to build upon and it's really more about how These open source companies can build the best interface the best user experience the best um feature set that will then Basically make the the big guys potentially like competitive at a minimum because I still think that there's a there's certainly a big play in a big place for specific things like uh Like the closed source stuff from vmware, and I'll hand it to vmware and I'll hand it to Nutanix They have really fantastic global support. They're available all the time. Um, you know We said a bunch of time so far, but like my my background's in vmware. I've been Vcp certified for years like I never worry about what i'm getting from them and When it comes to needing someone to answer the phone and take care of things And I think that that's one thing that I really hope to see the open source guys really build up is that level of Of comfort and support that we can basically come or rely upon and then they make some A true competitor for the rest of the corporate stuff that's out there This is not unfamiliar territory because this is what red hat built their company on was hey the code's free We're not free our labor is not free the customization you want done or the support you want isn't free. Um, I really I wish they would have sold to ibm. Oh, that's a different topic But the the red hat concept I think was a great proven business model this is something that can be done it can be done at scale and I think we are starting to see that and It's funny because I believe the risk factor that companies have to think about might be changing a bit because there's definitely this old school mentality many large corporate companies have open sources a risk to my business Who's going to be responsible for it? Who do I point the finger at but as we watch and even in a security realm we can't even We can't even attempt to keep up with how many avanti vulnerabilities there are like I just another one today. I'm like Do they I mean how many cvs is this thing have just in 2024? uh Then combine the fact that the companies especially the ones that are just going to be paying for the new vm We're licensing same thing. They're like what the risk they just faced was a Nonpredictable base cost to their things so they now got the rug ripped out from under them So actually not using open source feels like a risk to your business to some extent Like these companies have not been behaving well if you could say You know either by lack of security updates in some side Raising a price is randomly and I really wonder what the future of vmware looks like because the layoffs didn't I My understanding based on following people on twitter was they laid off a lot of smart people and the engineering people that push that product forward Where does that leave it for any future development? Or is it going to be the broken toy of the future with no one updating it and once they're done You know the way these companies work broadcom who's A tech company mad is really a private equity company masquerading as a tech company It's a it's the right off like well we raise prices It suddenly got less popular because it wasn't a great product anymore And then we did a circle and we moved it over here and we took a tax loss on the five year on it We made our money. We squeezed the juice throw throw it away over here Yeah, that's I sure hope that doesn't end up being the case but you know if I think there's been a lot of haymated, but I mean In the in the early days like we like when you and I had our conversation like in the early days I I tried to stay positive when the broadcom thing was announced and hope that you know This wouldn't be another computer associate. It wouldn't be another semantic and friends I was wrong and I can admit that I'm a man I can admit when I make mistakes and so that was it I I was the the fool in that and so I I it's again hard to really separate my like emotional feelings towards this product that I've been using for a very, very long time But yeah, we we live in an era now with where if a corporation can come in and 5x 8x 12x Your costs for your licensing for something that is the foundation of your business It starts to make that look less tenable And you start to look for alternatives and maybe that's really Where we'll see a big uprising from the open source stuff to really fill that hole and I sure hope so Yeah, because the the risk has definitely been Slapped in the face of me these companies are going how much more do we have why are we giving the IT department more money? What are we getting out of them? Oh, that's just a licensing fee We you know in yeah, it doesn't look good. Um Did you decide and we can ask this and by the way, there's about 300 people in the live stream right now That's awesome. If you have some questions go ahead and throw them throw them in the questions We're here to take dude does say live q&a here But did you decide what you want to use in your lab? So oh my gosh, so um Not yet. I hate to say but uh, I just bought some brand new hardware. Um and because I well So at home right now, um for those who don't know me and the stuff that I'm doing I have a two node Vsan cluster. Yes. I know two nodes is a waste of of resources You can you can tell me how bad I am in the comments And I mean I I do I am moving away from vmware for a variety of reasons notwithstanding but um as as as my primary virtualization platform I bought a four node SMC box with that are gen one gen two scalables because I really want to experience um xcp and g proxmox and even new tanix at their full uh clustered hyper converged, um approach just to Uh get a really good feel for it because you know, you can do these videos And I all the videos were basically a single host like I didn't I didn't have the resources to build up a cluster really throw this as a cluster That's why the whole focus was from esxi in in the sense of a singular hypervisor and really kind of Want to know where the the true strengths Comparison wise for at least for myself and professionally Lie within those products when you have four nodes right or three or four nodes to run again So that's that's my plan after that point then for me at least there shall be a winner whatever that I mean, I I personally Lean more towards the xcp and g side for a lot of reasons like the big one is is it feels more like What I'm comfortable with where I came from Um, and I'm really looking forward to the major changes that vates is making to the user experience because that's one of my least favorite parts about but But also, you know, I've heard things about the way that proxmox in the reliability of of seph and stuff like that And so I'm interested to see if that's One guy's bad story or if it's really, you know, the distinct issues and failures are real So there's a lot that I want to I want to figure out for myself and uh, so I'm looking forward to those Options here. Yeah, I'm gonna throw it up on the screen here to share with people I do have a proxmox cluster I set up So I am working myself on some updated videos on this I I did this even before I did the 45 drives video. Of course, I still have my xcp and g1 setup. I did learn That I'm working on some of the fundamentals for how proxmox works and one of the Like it's easy to migrate a vm in proxmox So I can have this vm and I can easily migrate it to the other storage in there But what I've learned is it works a lot different in proxmox than how that works in xcp and g And maybe you can answer this question if you know If this host dies, this is even though it's on shared storage this gets stuck on here And I thought that's interesting unless you have a full ha when you only have a two node cluster It's stuck versus an xcp and g These are all database driven. So there's a database that holds all the vms Which gives you this option I can say start on even if one of these hosts goes down I can start it on the other host to just tell it which host I want to start on Which I thought was it starts to I started breaking down something like those fundamental differences And I think that's uh, I don't know how much you dove into some of the back end out And that's kind of what my next thing is to talk about it Yeah, and I honestly did in a whole lot But I think as as someone mentioned in the chat just a second ago like Effectively speaking all of these companies that are doing Hyperconverged or they're doing shared storage are pushing you to be on a three node Like even my two node vSAN vCenter setup that I've got cluster It's basically just duplicating the data in both locations in case One goes offline. That's how they handle it. Then there's a witness or a you know to to break that quorum But you know, it's it's a good question and like knowing that that xcp and g is Making it available on on all or I'm assuming that's just the two node configuration though Right tom for the xcp and g that it that essentially duplicates the information because they would have to right Well, see this is where it's kind of it goes down to the fundamentals of how they work And I think I have a video where I did this and I made some Broad assumptions, but uh when I was talking to people about proxmox. So then I said I got to really learn this Um, I know I always said that I don't know proxmox very well, but I know xcp and g I see me and g Um, you always have one host always has to be a master and there's no concept of that because it doesn't need to be Well, unless you're running ha in proxmox But even if it's a host of one that host of one has to be a master a master holds a database The database is all the configuration settings the network settings all the status is the vms where those vms live Where the storage lives the ram and cpu and all these assignments to those Then when you add hosts one or any more hosts you add get a copy of that database But they queue off master you always talk to master all changes are sent to master and relayed to all non master nodes From two to a hundred that you have in this cluster It makes it interesting and you can repromote any of them to master But that's what allows as long as the vms on a shared storage Any host can die without a j and you can say just start it over here or start it over there It's not doing anything but moving the memory when you do a migrate But they're all in sync of the status of that host It's a it's a different concept versus in in proxmox. I learned everything's under I think it's like at cpbe There's there's it's all creating your standard linux style kvm stuff under there So when you do a migrate it's just saying hey copy from that node that's running over to here These and then migrate the memory and cool now we got live migration But the difference becomes if that node goes down it goes well, I don't know where that is anymore the node doesn't respond Uh, I have the shared storage so you can rebuild a vm and point it at the shared storage and work But that's not the same as we'll start on the non-failed node And if you only have a two node system, which I know most time if you're in production You're gonna have a three node ha or more So it's gonna be different but it those little concepts Uh, it's those little nuances. I've been learning as I go through it And like you said, this gets really challenging to do all of these I mean the amount of work you put in just how many hours you think you have into your entire video series I want to let people know we as youtubers who love to bring you content Do you know how much time it took us to set all this up? Like I've got two complete labs I got a people I got more people working behind me to help set these up too Yeah, it was it was quite the well, you know, because I wanted to do it the best I could in in the right way. So It's really hard to I think it I've been asked similarly How much is it time does it take and like I I stopped counting after a while because if I counted then The the imbalance would be too great. Like the finished product is is 20 25 minutes long But there's like a solid two weeks of human effort, you know going into it, right? And you know and it's never just as simple as like, oh, I just spun it up and and got it going like no I I tore it down when I made a mistake and I didn't like that and that's not following best practices So go back in make sure I'm following the best practices that way when I do this and giving people the right the actual Correct way because it's really important to make sure you're not just Just going out saying oh, I think it's like this and it's fine, you know, and yeah Yeah, you know, I mean these these sort of things take a long time Well, funny good I'll just say I the reason I learned how proxmox works better is I started just unplugging stuff in drives And I created all these failure modes because it's not about does it work out of the box? I fully expect all of these hypervisor work out of the box I want to happens when I yank a hard drive out because that's what actually happens Something breaks something goes down power loss of one node So power supply goes you guys start simulating the failure modes because it's not the success brand new hardware modes that matter It's how do we recover from a problem because that's what we do is tech people because it's not all sunshine and rainbows out there Yeah Yeah, if everything were perfectly we wouldn't need to we wouldn't be here, right? We just set it up and replace it every so years after it got outdated But yeah, the reality is these things just do random things So we're I'm building a series of proxmox clusters and putting some basic workloads on them And just letting them run and letting them run updates xcp and gz easy for example consulting on it all the time We have tons of and we have this running at our clients. So Clients have created we have a couple that created I'm like, I didn't I would never have thought to use the system that way But it's not documented to use it that way That's why we're consulting now because you've broke things in a way that we've never thought possible You someone created like you just know this would be a problem vmware too Someone started a snapshot policy had thought it they want to have 24 snapshots attached and go Well, the performance sucks now and I'm like, yeah That's um, you have to track the differentials between all of these I will answer a question. I seen in here the thing missing for me Us at a company is basically seamless pooling load balancing proxmox seems Not to have this at all xcpg you somehow still have to choose a host to start the bm on Actually, there's an entire they just updated the load balancer in the latest version There is a load balancing tool you can set up inside of xcp and g There's actually some load balancing policies that I'm less familiar with but I know they exist inside of proxmox Um, I they generally in proxmox recording. This is from the 45 dries people. They actually spoke to this Um, it does it based on memory it can start Shuffling them which makes the most sense because you're more often memory constrained when you are dealing with vm's You're like, oh, I need too many vms on this pick your host There's not enough memory to run them all cpu's a little bit harder to figure out because cpu's it not as much as constant It's like I need it because someone did a database query So we had this peak where a query and then it sat idle for the next four hours where it didn't need to do anything But both of them do have functionality for that and I will mention Uh, I'll share my screen really quick on this I we were diving into this as an xosan replacement The proxmox has sep which is amazing. Uh, the xosan supports the different distribution types using lin bit So it's similar to the erasure coding where you can actually Have multiple hosts lose multiple hosts have parity between them And uh, it's stretchy. So this is in they've updated all their documentation to reflect this by the way This is i'm looking on the public documentation pages Uh for how their xosan system Uh works for anyone interested Is that a topic but the sep stuff is really it's it's scary And we talked about this under 45 drives because they're the sep experts whenever people say where can I learn about sep? I'm like, there's a playlist on a 45 drives channel. I went through the training with them I was blown away at how amazing sep is like I understood it in concept But I did a two-day training with the team at 45 drives And uh, I love sep. It's amazing. Also. It's as complicated as you may think it is In terms of sep one things that I've heard because I haven't spent a whole lot of time Like I said, that's my my next big challenge on my own is Uh to really leverage the quality of sep you need to have a certain amount of like storage targets, right? No, it's essentially it's not just like you sure you can do a sep of of two nodes But isn't it like three or four or five something like that that you really get the quality out of it for a production environment And the way they 45 drives sells it like if you call them up because they will do solutions And we partner with them to build solutions for our clients that are built on sep uh The 45 drives team was gonna say you need a minimum of four servers to build out a solid Resilient yank whatever you want pull the plug and no one will notice a type of you know hyper Amazing storage cluster built out of sep. That's if you want it done Well, and they still agree to this day It's hyper converges a cool story But you want to separate your compute and your storage and the reason why comes down to If you want all four of these servers to replicate that data Perfectly. So when I write I have perfect replicated data across four servers They call it a public private network inside of sep. It's kind of the terminology But it doesn't mean public private like internet the if you have a public network of Uh 10 gig which is going to attach to your vmware system or your uh, whatever you're using You know xcp and g proxmox, etc The private network you got to amplify that So you have to have a back end fabric to connect the service together That's going to be at least 25 or 50 gigs that way as that right comes across It's got time to commit because if if i'm Could be misquoting this double check with the 45 drives people, but it it only does Synchronized commits There's no way to let go of that until it's been synchronized across whatever your erasure coding is So if it says this data must live on two servers three servers, whatever that is until it's actually committed You don't get a commit back So you can actually have this incredible slowdown Of the servers if they don't do it and because of the way you take a certain number of data drives That may be spinning rust and then you'll put ssds To hold the metadata and certain functions of your sep cluster. So there's like, hey, here's where the data is Here's where the here's where the metadata is Here's where the actual data is and it's a calculation done and this calculation is done across all of your sep clusters That's why when you separate it you have this compute cost of we have to calculate where that data is We have to calculate the parity for it. This comes us all the way back to what you said about proxmox and the discussion I think me and you had before about this um Proxmox did an amazing job Hats off to them for the engineering it took to put it in and their documentation does tell you to build a More robust network in order to get sep working properly They did not hide anything they did what you should do But what you can also do is have a single network interface Go build a cluster on my desk over here turn sep on and tell it to use All of it across one interface and the moment you actually were to put this in production You'll notice that if something was a database server Doing a bunch of small writes sep is trying to go let's calculate these rights and hyper converge it across your storage Oh, you also want to do the vm itself needs some compute time. No problem. We'll give you some compute time Something's got to give Then you're not you didn't set up a fast some network between them So everything just goes wait wait. It's not synced yet So next you know you watch, you know your wait times go up and up and up and your things going at a crawl You're like what happened here and the 45 drive seems actually shared stories of people who built And broke well built what they thought were good production proxmox cluster And they said yeah the q depth was so small on their consumer drives They used that no production workload would actually work on it the moment you started doing any small rights The whole system would go high load and just go to an absolute crawl So there's this good and bad of yes, they built it in but you still got to think about it from a big picture design Yeah, yeah, I mean that makes sense, right? There's there's always something to be said about making the user experience so easy that it's just a single checkbox and magic happens I mean that we we do live in the future, but we don't live there yet Not for that sort of stuff like there's there's always best practice for deploying that stuff and and you you hit on it every real big virtualization Company that's making gear has these best case or best practices to use when you set up your storage and interfaces and the storage network Or dedicated Sand if you're you know, you're going that route so that you make sure you don't run into io Problems that kill your your performance Yeah, and someone comment to this. I just wanted to go back to the documentation The word erasure coding is not used in here. I use the word erasure coding No, that is not the exact terminology for how xo sand works They call it disperse six Which is very similar to rate six so they have all explained here in their documentation Erasure coding is not a word used inside of here, but it I said similar to erasure coding I hope I said it like that So people can understand i'm not Uh, it does have the ability to do things like this like building your ray 10 like system with Distributed replicated versions of it. I know it's not the same as erasure coding But is for people who are if you're in the sef world I'm trying to make a similar equivalence because you can connect xcp and g to sef But it does not it's not built in like it is to proxmox proxmox In hands off to them. Like I said, I'm really impressed with the work. They uh do on this But you can set up and by the way hats off to proxmox for giving me a health warning because of my improper osd count for setting this up for not having the proper amount I these are single drive systems. This is not in any way ideal And I'm glad it warns me like hey, you have a health warning here You you really can't suffer you used sef, but you can't suffer a failure here. They're not wrong about this I'm glad they have it that is exactly what it should say So thank you team at proxmox for allowing me to set it up with a couple clicks, but also going you didn't do it, right? Big warnings. Yeah, big warnings down there. I'll just from right here um I use cockpit for virtual machine management, although it's less feature rich compared to proxmox Copics virtual machine modules all they want to thank you Grayson for the donation and uh, I'm there's all kinds of Other options. I guess you could say I've seen people roll their own thing and Some people I someone told me they don't need a whole management interface because they wrote a bunch of stuff in terraform and ansible And then I'm like the hats off to you I'm glad you have a team to manage it But you made some really good points that you should probably bring up again one of the things about Any of these is cool proxmox is devian based, but you know, we work in the enterprise space Does everyone uh, who does all these windows management? Do they necessarily are are they good on the command line out of the box every tech is right? Yeah, no, right. That's honestly and I imagine when we have this conversation here I can't wait to see what what what the chat says about this but because Guys, I think it's really important that you have a Especially on the console side of things a means of basic troubleshooting. That's not just the command line I totally understand that if you're using proxmox, you probably have more linux uh staff that is capable of handling it but Like tom, like you're saying like the the company I currently work for right now we have a staff of of four guys three guys and They have varying levels of experience some of them are really strong in linux and some of them are really strong as windows admins But every one of those guys is responsible for a p1 call at some point, right? And so if it ends up being a situation where the the host is lost It's it's web manageability Then you need to hop onto that con the command line console either either you're gonna get there via You know an iLo interface or remote access or you're in person But if you have to jump online to search for answers how to solve things You are slowing down your turnaround time and that doesn't make them bad engineers. It just means that the product doesn't have maybe the level of support and configurability that it should for a console, but yeah, I Strong believer that you should have like I love xcpng's console Yeah, it is amazing and I can't stress more like when I hadn't used it and when I started with xcpng and I set up my the first server and and ran I was like I can't believe that vmware didn't steal all of this like all of it It should have been stolen all of it because it's not new it's actually this has been baked in for years I mean you can go to the you can go to years and years ago 10 15 To the citrix days and that same council being able to be able it just if I got a reset a network management interface I don't I mean I know in proxmox. I can go to xc slash interfaces. I have a debium background I We have a lot of texts Not all of them could not every one of them. They are brilliant texts Some of them are azure certified some are vmware certified We got a good variety of people there But not every one of them could also go to as you network interfaces and hand type out a new interface or modify something as well Uh, I'm not gonna lie. There may be a couple that would not use them very well so Then uh grant nanos on there. I know yeah, but and it's uh Minor because we had a priority well as priority one type of things We had a tech today and it we solved it right away But it was all those he didn't realize of the way linux services had to be restarted To see modified a config and he's like well, it's still not working. I'm like just restart the service He goes, oh, yeah, I remember you guys told me that before you know, we were training them So it's all those things not everyone realizes how linux works in I get it with proxmox But being able to and when I did my admin video, I I liked one of the comments Someone goes that tab auto complete is amazing because when you go into and type in like xc For starting the command line and in there and say vm start and I say vm equals I can just tab tab and it'll list all the vm's available on there and say so from the command line Start up one of these vm's I went through a hole that my admin video to kind of show people You could you you could actually completely manage it for the command line Maybe not ideal, but there's someone out there that you can bake some scripts together But honestly, I want to show that it's easy for people who don't spend a lot of time in the command line To run a few commands to get a few of the things that they may run into on a daily basis a A host failure that they're not sure why the hosts are out of sync with each other And there's some simple tools to say resync these hosts or Join these hosts back or how do you designate a new master when something went wrong and the master didn't flip to another host in your setup, so Those are being able to sit tab complete You know like you don't have to know all the ui people like uids. It's hard. I'm like no no tab auto complete Just tab tab it'll spit them all out for you You can list them all and make your life a little bit easier by admitting this Yeah Yeah, I I I still think that it's important that that like you're talking about with your your Staff like you need to train people for the platforms. This is not like throw this out there and say good luck guys You know you don't run a business that way But you also need to make sure that that you're providing The the platform provides the the features and tools that that a variety people could support You know and that's that's that's I think there's nothing There's nothing wrong with it and anyone who says if you don't know linux Then you shouldn't be using proxmox is doing proxmox in the community to service because that you want to be inclusive of everybody Yeah, right and that's that's how you make a better product And in you know an easy example is going to be so what the client that we were we're bidding on for this large project Um, they're very technical. They're software developers though They developed a product they resell their product through a software as a service It runs in uh several data centers that they got across a really nice infrastructure all with vmware and we're moving to xcp and g But the reality is their day-to-day is software development. They're They know how to run the software. They know how to write their software um, but they're actually all their software runs in windows and it's a Really interesting application they put together But they don't have necessarily and they don't have to look at vmware vmware needs some updates They load some updates. They say yes it research the host and uh, it shot There's all a chase it up. So it'll you know evacuate them move the vm's over things like that It all does it through the whole v-sphere. They're using all the vmware stuff So they're evaluating and have decided xcp and g is where they want to go and the same thing They're like cool. We can do this through the web interface. We can hit the patches It's cool that the command line's there in case of that emergency, but uh, ideally and these are some very skilled Developers they don't want to have to Learn linux because they're not going to use it very often. They're only going to use it if they find a problem so And we're on there and I seen someone ask this question Does xcp and g and proxmox require more command line troubleshooting than vmware? I don't know that more There's vmware is essentially linux under the hood as well. It all comes down to what are you troubleshooting? What went wrong and what are you what are you chasing down? I've You know vmware and xcp and g share a common idea Where the hosts themselves are bare. There's is little the minimal amount running only what's needed and no more versus proxmox This is why proxmox is super popular in lab It's uh debian underneath and it's proxmox on top of debian if you look at the repositories are debian So you can apt get your way into a level of hell Yes, you can I've seen people I mean how no one in the corporate world will think about adding a file because here fun fact and don't get me wrong This is cool. This is the tom being excited cool I would be horrified if I found a corporate environment using this You can load sef and then expose your file shares and do samba You can load samba on proxmox and make it your file server alongside your vm and you're like that's neat But also if you had to support a corporate environment, would you would you enjoy that? yeah Just just run your open source windows actor directory services on on your proxmox. So it should be fine. Everything's gonna be good Yeah, I want to I mean we can do some fun things in samba. We can just time like that. So Is it someone says, uh, please don't mix proxmox With xcbg xcb is an absolute winner replacement for vmware But proxmox is better for small home office. Do you get a real estate thing to do with cop or a container? Oh, uh I don't know. I think the best thing to do in a homelab environment is learn I I if if I it's just how I said I'm horrified if I see a home if I see a business environment trying to run samba next to their Inside their proxmox. I'm like, well, I don't this is not great But uh homelab, I mean I've seen some homelabs. They're built some amazing house of cards, but it's an amazing learning opportunity Let's do it. So hats off to people in the homelab that build some of the most elaborate Like it's rube golberg time That's a good example. I mean, that's what it's for, right? That's I mean you can have a homelab that helps you Specifically target being better professionally, right? Which a lot of people I know I know a lot of people who are into homelabs Are doing so because when they go to work, they need to make sure that they know what they're doing And they can't play At work and production system And then there's people who are just masters of self-hosters and they built the like you said these massive house of cards On top of systems that you know, they're clinging through because they're just Going to have fun and get better at it. So that's that's the magic of it. Go for it, right? Take your proxmox and go to the moon with that. Just do it I actually uh, wendell from level in texas. He calls it the forbidden router I believe it's the title of the video. It was just talks about virtualizing Your routing platforms like pf sense and things like that. It's a fun deep dive Once again, do I want to see this in corporate? No, but you get to learn a lot virtualizing it You start to understand how the packets come through You go, okay. This is how it happens when you virtualize network adapters. This is what q and q means Oh, wait, there's overhead wait. I had to change the mtu These are all those things you start to understand better because you took that endeavor of diving into it So I always encourage people to do these things. I just don't encourage them to install them as a production system Yeah Yes, follow your best practices learn how to do it and then follow the best practices people Yeah, um, does proxmox allow thin provisioning in ice guzzy? Hopefully xcpng closes by release to my knowledge Neither proxmox nor xcpng support thin provisioning uh for ice guzzy. I don't know if you're aware of that That's that's my knowledge as well. Yeah um Someone had I remember seeing a write-up of why open source ones don't do thin provisioning in ice guzzy I if I could find that uh, someone was on reddit had a really they clearly Had done their homework and I really talked about the history of it and why there were some challenges in making that happen um I don't remember what the reasons were but there's some just technical challenges and the answer My answer is generally use nfs But the people using or asking for it an ice guzzy often because especially I've run uh, one of my home that friends I chat with on twitter. I can't remember. He's had some older Uh proprietary hardware, but it's really nice. He got it out of You know his corporate environment, but it only presents ice guzzy It's a really fast all like santa rays got but he goes it only presents ice guzzy It's kind of a hang up for me because he goes I I don't I he says I could replace it It's expensive and it works really well. I'm like I get it Oh, what else did we have for questions here? Oh, yeah ice guzzy does not ever do thin provisioning drive has to be formed everybody I host I want to say someone had a write-up on how to build lvm thin provisioned inside of proxmox Proxmox got some weird things you can do but um I don't know that they're like the book way to do it Yeah, I said that's probably and that's the nice thing about linux in general Right, there's a lot of ways to slice it up. You could probably Do a out-of-band mounting of shares and then when you the presenter the operating system as you know Like you said now vm and then you can thin provision inside of that But it's technically being backed by something external But yeah, that doesn't feel like the the good way to do it but Yeah, you know with like vmware that sort of stuff they they've got all their va ai and all that sort of like connectivity and integrations with all the storage vendors so all the ice guzzy stuff and and I'm pretty sure tom I can do thin provision. I have to double check Where does it? Yeah, I think I can provision an nfs as well I can print thin provision all day long everywhere. So that sounds more like a driver issue than it, you know, they're they're set up than than a Technical issue, but yeah, well, it's all those. It's always just some weird thing. Maybe I I can't remember and maybe it wasn't this but maybe it is this but I don't remember this being in the right But maybe there's some proprietary There's a license on if you thin provision and or somebody has a patent on thin provisioning ice because he who knows That would be someone someone's got to make their money before you can get that picture. Yeah The answer really is use nfs, which is thin provision And I I should just be testing on this to validate how this works exactly but essentially True to ask can thin provision a zeval It will be called thick provision But it's thin provision because of the compression and everything of the way it works in zfs Um, I believe you don't have to claim as much space. It will show the space So you can put these larger mounts Which the big challenge you run to when something isn't thin provisioned in a hypervisor Is when you have to do snapshots each snapshots going to take the size of your 60 gigs It's going to say even if that snapshots only using a differential of like one gig of change It's going to need 60 gigs again and you go do a couple more snapshots next You know, you're like wait a minute this vm this particular vm and its snapshots take up this large amount of the ice cozy Top that off when you start deleting them You need at least as much space available to coalesce them back into their nature and The inability to coalesce is something we run into where people set it up ice cozy or use some type of You know ice because the only advice they snapshot too many times you tried to lead them like yeah It doesn't have enough space to coalesce. I'm like yeah, I can we got to figure this one out We're gonna have to like back to vm up and restore it without the what snapshots like there's not an Not an easy solution for that you've you've done something off book again Every every virtual admin Uh has at least once had their their snapshots consume all of their physical storage Then and crash the virtual machine if you haven't done that. I mean are you even Into a virtualization. I mean that's someone makes a snap says, oh, I'm gonna I'll destroy the snap in a in the day after I make sure that my my modifications worked and then they forget And it's a day by server and you know, it's it's now down everything's down Yeah, it It's all those unexpected things because what happens is the snapshots if they can't coalesce And you have a job that runs on a regular schedule and we have a client like anytime I see the ticket comes in I don't have to read the client name I know by the problem they created because I don't know why they we've talked to them a couple times about this They think snapshots are great ideas for backups Um because they're always messing with things and then they set them on a schedule to do something Like oh, we're gonna have a snapshot like every 10 minutes and uh, then The way the system does it waits for a free time to coalesce them back on there And if you are under high demand, there's not a lot of free time So the io load is too high on that particular nfs mount It won't coalesce right away and then it has more to coalesce and more io time and more to coalesce things You know, it's like, yeah, there's 37 snapshots on this 1vm and now the system's not working again I'm like, I know which client that is they did it again Yeah, they forgot they said they said a job that says snapshot every saw. Yeah, this is where we're at now Sounds like they're confusing storage snapshots and virtual machine snapshots, right? Yes. Yes. They uh, it's It's that's the that's the fun part about consulting, uh, for some reason they they come up with ideas I I would have never had so we've actually learned how to troubleshoot scenarios that I didn't ever see to create in my lab Like why would I do that? But now I get to figure out how to fix it Yeah, oh, let's see seems there's any other questions in there Is new tanix any good we answered this at the very beginning Um new tanix is this black box of their own hyperconverged solution that doesn't support ice because he Uh, doesn't support. Well doesn't support external storage. We'll just say it doesn't support fans Uh, and doesn't have any pass through so if that is if you want a company Will we call them the apple of it? Like It's it's the hood's locked for an open in the hood. You get what you get You can only twiddle these couple knobs and you pay us a lot of money It's it's funny. Yeah. I mean except for the setting up of it So the one thing that I we didn't get talked about with with new tanks in terms of like the configuration of it I find it the strangest out of all of the systems including xcp and g in fact that that uh That the um the virtual machine management is done in another you know a vm, right? But when you do a new tanks deploy you get one iso You install because they have a community edition and you can install it and it'll work on a single host That's what I was using you can install that that a hv hypervisor and then you wait and in the background The elves are deploying a separate virtual machine the cvm That is where everything happens. So unlike, uh, I mean it's similar to xcp and g in the sense that the That zen is just doing virtual station options But you don't interact with the a hv hypervisor all Zen, you know you xcp and g when you get on the console You can you can essentially do all the things you need to do by the console with a hv It's like no everything that happens here happens in their hypervisor or they're in the cvm the controller virtual machine and it was a A departure for one i'm familiar with the other thing that is kind of baked into it that is weird Is that the amount of overhead that takes to run the cvm and all the hyperconverged functionality is very high like Minimum of 16 gigs has to be assigned to that that cvm and without a cvm. You have no Nutanix. So it is uh There's a commitment involved. I think that a lot of people will not see that works beneficially for a homelab necessarily but you know it It's it's it's an interesting approach. I love their gooey though their gooey is It's beautiful. That's the It's so pretty watch version video on Nutanix. I mean if if nothing else, it's pretty so It exactly it is pretty. It's it's got nice html 5 animations and there's a built-in like Game you can play on it, which is interesting, but oh, yeah, it's got 2048 built in the top. Yeah Just I find that just fascinating. You don't see that anywhere. Um, maybe they just have a little extra time on their hands Yeah, it's cute, but it's nice. Um, I don't like I thought I did think that was a nice feature on there And any of these ui's can always They're often written by technical people. So they don't always get the polish or the flashiness that you may see Um, which is a shame. I I like though when I the new xo light just got updated again today I think or yesterday, um, which is their light interface for xcpg I won't do a video on it until it's a little it's very rough right now But what's not rough about it is xo light and when they come out with xo 6 are going to be Um using that new dark mode interface that looks really nice. They actually hired a dedicated They've got vates has gotten to be pretty big as a company the people behind xo and xcpng They have a dedicated designer who's really pushing this forward from a very they're doing it slowly But it's pushing forward with a whole design philosophy. They've already set out You actually can they've got it all in their github. You can see all the colors They chose how they're going to label menus how they have all the elements like they put this whole Philosophy of design and now they're pulling from it. So when the the Technical people writing it they have elements like this is the applied element that works So this is how you have to fit it on the screen within these design parameters Which is kind of cool because this is one of those things that often get built is the haphazard design of technical people We don't necessarily think in art space But when an artist applies rules and says here's your design elements and how you're going to find these are the color schemes One makes it easier for technical people because we just have to write the menus that come up And someone else designed the color scheme for those type of menus and those modal windows and everything else On doubt which I think it's kind of cool I super happy to hear that too because that's the right way to do it right there's I understand that Like you said, we're technical people We think about the the UI necessarily maybe is the the last step after all the hard work has been done Like let's just let's just throw something out to get it done But like there's studies right there's people who go to college For years and study in how to design and build out user experiences and then make them Standardized across all of the products and VMware did that a long time ago too Yeah, I mean you you log into ESXi and you get the ESXi console you go look at vcenter They share same componentry and that kind of stuff right there's like That's that's what a business who's serious about their product does and I think that's awesome Yeah, it's a big lift on the back end because you have to hire a dedicated resource Whose job is just to make things look pretty when you think about that from a business standpoint You're like that person's not going to be cheap They are going to set the cadence for how this is done And keep them up to date but in the end what that looks like long term If you've got the vision for how you want your product to look it's so much more Wonderful user experience when I go I can expect the same similar elements across this product line. Thank you. So I'm not searching for it and I will say Good UI design is it makes the job easier, especially when I have to teach people. Oh, no There's like the sub menu sub menu. Oh, you got this farting claps and I had to go start over again. Please don't do that Right click over here and I got a context menu, but if I right click over here I get the browser menu. That's it. That's my my complaint about proxmox So there's context menus in some places but not others And yeah, I mean that's that's how you do it. And then you have that design language and then when someone learns that interface They can Follow through that product and if you think about it If I if I put on like my sitting in a conference room listening to the sales guy Demonstrate the product right show you the interface That's a first impression first impressions are really important if you're looking at something and it looks amazing Like okay, I'm more interested in seeing what this has got behind the scenes versus going I'm going to focus on the fact that that looks like it came from 2015 Yes Yes, and I I've heard the complaints and I get you there's that I will say I see I can't remember if was one of the people in my office. I think they call it the fisher price interface They're not a big fan of the xo interface. They're like This is like a fisher price interface to my kids might use or something like that I laughed like fair enough. They go to get a couple of the box of crayons, you know, I'll take the ribbing I'll take the I'll take them jokes Do we know when um the new xcp and she interfaces are going to become This year is your plan as far as I know They they haven't you know like developers do they don't set hard dates on there? The good news is from a design philosophy They they they plan to really make it so you'll be able to switch back and forth between them So you'll actually have one of the updates whenever that update comes that starts bringing us to new elements They're not going to say here it is it's going to be here's an option to switch to it Here's always an option to switch back in case you have a problem So they're actually going to keep them dual then there comes a point where the old interface becomes essentially Static in terms of it's not getting any of the new features Because the other one becomes feature complete but for those of you that they're they want to do like a really good transition Um, so they put a plan together to do that as well, which I think is actually pretty cool on there um Any chance to revisit harvester Ah, I I felt harvester not to be the most complete system I don't think I don't look at it as a competitor in this space, but maybe I'm wrong about that I don't know It's Exclusively homeland people asking me about it, but I don't know what it offers in terms of features. Have you looked at harvester at all or Only in the context of running um containers, right specifically Yeah And you know, I I still kind of subscribe to the well you can virtualize be You can virtualize the linux box and run your containers on that versus having harvester be the platform So I haven't I haven't dug into it, but like you I'm hearing a lot about people asking about it as well Yeah, it just didn't feel as complete, uh, but I think tech I know tim another friend of ours. I think he's got a video on it. I should probably revisit on there um, I I don't know it's uh, I think it doesn't compare and feature So the ones we've kind of mentioned before but maybe it's worth looking at. I mean, it's always One more thing to play with Um in an cpg pool. Can I add a machine that has a different network card? Yes, but Yes, but you can't have mismatch network cards. It's less than ideal You just have to remap them so they're applied to the same network So every machine in the pool ETH zero needs to be plugged into that same network across all of them It can be a one gig it can be a 10 gig But as long as ETH zero ETH one so on and so forth are all plugged into the same network It can work. Uh, I've got mismatch ones in a pool right now for my lab We have a few clients because they had some older machines and we wanted them in the pool Sometimes because you can move VMs even when they're not in the pool You can actually just build them separately This one of the things that I always thought was cool about xcp and g and this goes back to the Citrix days You can take two separate systems not in the same pool and migrate VMs between them So they don't have to be part of the same resource pool to do it So the first question is do you need a resource pool between these two? If you don't need it and one system is particularly different Then you can do that of note if I have five nodes and four of them are modern processor One node is not modern processor. Maybe it's a few generations older Pools go down to the lowest common denominator So the dumbest one in the pool makes the other ones just as dumb This is the best way to describe it But the moment you kick the dumb one out all of them auto upgrade to full features because if there's certain Features that aren't available it makes them not available for the whole pool Which is a good reason often not to have one old machine in there doesn't mean you can't also be talking to it But it's one of those things that It may not be the best idea We're going to go a few more minutes here and take me to the last few questions But yeah, it was almost Peaked up close to 400 people on here. That's amazing. It's a lot of people on popular topic here It is it's a big topic for a lot of people. Yeah, it sounded like 350 It was like 379 380 and it kind of I watched him peak and go but it was really cool um What's the best approach from moving a large enterprise from vmware to xv when you have a learning curve for current staff I'm testing now um Yeah, one nice thing and something rich said and I'll let him comment more on this as well When you're moving people across like that Because of I think there's once you tell people Part of the training we're going to offer for this company that's we're moving almost between 5 and 600 vm's for They don't want well, they would love it if we would move all the vms But actually what they want us to do is train them how to move the vms train them on the system And then be there as a support plan. So it's usually hiring a third party To do it unless you want to designate someone going your task is to become Is now as well with xcp and g is tom is right now go watch his videos or go watch them You know, it all depends on different company, but companies aren't always willing to dedicate a resource To that level of training. So they usually hire a third party consultant to come in Teach them fill in the gaps give them the overview and kind of be there as a backstop while they do the transition And learn I mean how you guys because in your corporate job, you're making some change as well I mean, it's probably how you guys would handle it Yeah, it's it I think it really depends on On the company's workload the type of people they have staff, right? Like some companies like you mentioned with software engineers They don't necessarily want to be that intimately part of that But if you have an engineering staff like basically you would write seed ride through some of it and Either you get to the point where you feel comfortable doing the rest of these yourself Or they are there and you have them as a backup to assist you Or you let them finish the job and they give you run books afterwards that people can use then to To follow any sort of necessary things that they didn't pick up as part of the process But I think that in a good department with that has the actual staff for it You have people who really love This stuff anyway, and so they're going to be there watching and trying to insert themselves so that they can take over But you know, it depends on it's always dependent on the org right and who they got for their their staffing Yeah, that it kind of comes down to that. Um, this is just a quick one. Uh for years, uh, Ken, here's a donation for the next barbecue. So thank you very much. I much appreciate Asian. Thank you mark Barbecue sounds good. It's almost summer here. Uh, let's see And I I agree like harvesters cool vm manages them And it's more of a technology demo proving that kubernetes can manage vm something like xcp and g I There's a love hate. I have a kubernetes. Like I know people that work it I swift on security has one of my favorite tweets. Someone asked me what kubernetes is I tried teaching them now neither one of us know what kubernetes is That's great It's a beautiful system when it works, but uh, read the reddit debrief Why why reddit went down last year with a kubernetes update and it comes down to too long didn't read kubernetes is hard Even for people who do it for a living It's it's beautiful when it works But it's also a fairly complicated system and it comes crashing as much as it's a self healing amazing We're just going to build all these servers and they're going to self heal and we're going to spin them up And it's elastic also when it goes wrong it goes completely wrong Um, if you listen to the cloud, I think it's called the cloud security podcast They recently had on a google engineer Um because google runs a lot of kubernetes But they talked about the level of engineering it takes just to understand your logs of what goes wrong in kubernetes It's it's a long podcast. It's a really insightful of how they have to decipher things But you're also like wow that is there's as much as there's tooling to make it work This person's a security engineer who has to find out when it doesn't work and even uh, they talked about how they found When someone hacks your kubernetes It's so hard to unwind that because you're like it's whack them on. Oh shut down the server. It's infected now That one's infected. It's it's spawning them everywhere and he walked through the process of how to unwind Hidden stuff in kubernetes that people put in there to inject. It was there's a wonderful podcast It's called the cloud security podcast Uh, it's about things like kubernetes and google and it's like your mind's kind of blowing you're like wow That's okay. There's there's what we talked about here in the home lab. Then there's some engineering jobs that being Uh rich due because we work in the corporate world, but then there's google the hyperscale Exactly Yeah, we don't plan that space. They're they're way above. Yeah kubernetes They need kubernetes to operate how they operate, but wow, there's uh, when you start thinking about the complexity of it Bring down. I don't know that I'd want to fix a broken kubernetes cluster It's above my pay grade. We'll just say because I don't do it for nothing Oh, let's see. Uh, is any other questions you see, um There's one question about backup that just popped up there from alamone From all the solutions. Which do you think has the best backup features? I'm going to say xcp and g's backup system is extremely complete because they have But yeah, my proxmox backup is its own product. It's a separate product So it's hard to compare as far as like natively integrated xcp and g did a good job because it comes with xcp and g proxmox backup I don't know if you've ever used it. Not many videos on it No, there's there's a lot of people say they like it, but I I haven't touched it Yeah, I haven't really touched it either. I think j from learn like cb is well He's got a video, but it's an older video because they've got a new version out So I don't know how how much has changed since he did that video like two or three years ago um, so I don't have enough direct experience to really Dive into it. It's uh, I've heard people say they like it, but like cp and g I can speak from experience Uh, I do know they have a feature that is not in proxmox at this time is Uh, they not only can back up your vms You can build it so you can point it at that same backup repository and restore it to a whole other environment as a test validation With automation, so that's how their backups aren't just like we got deltas and replications They have all the cool, you know, usual things you expect from a backup That's just table stakes at this point But they can do backup validation and have it restore on a completely separate environment So you can say hey, uh automate all this test restore and some of you notice will all these vms Restore because nobody wants a uh a backup works everyone wants a restore that works So you have to validate your restores and they have an entire validation system built in That to me is like that's it This is the same thing you expect and you have this with beam and uh your commercial tools. They come with validation Doing the restores Datto before because a incident was a popular company and one of the things that I don't have was this kind of cool You could their backups used to restore and send you a screenshot I think there's a few other applications that do this They'll send just out of your booted vm and send it in an email to say yeah, we validated it He's validated in their cloud. Uh, so you knew you could do the restore So it's kind of there's I'm going to say the backups are really really good in xcp ng Yeah, the one thing I wanted to mention to you and to any of the viewers because that's awesome and bring up beam uh actually had a conversation with a beam engineer and uh, who shall be named nameless and He said that they were so close to releasing their x or their proxmox backup agent for beam and That I pressed them on the xcp ng because like okay cool great, but what about xcp ng and the guy said Unofficially they are aggressively working towards coming up with a mechanism that will work with xcp ng But there's no no release time for it. So There's that company is working on something. Hopefully it comes out and it's it helps people who are migrating from vmware Or you know, whatever because that's what beam really wants to provide is a ubiquitous platform for Cross backup and restore across anything that they can target So that was interesting to see and he also said to me that his final law was like They don't listen to us. They don't listen to the engineers and what the engineers say They listen to the the forums the people in the forum So if anyone out there and this is just to know anyone who's thinking about xcp ng who's in the vmware world using beam and really wants to keep that that That long pole in the tent in their virtualization Get into the beam forums and start telling them where you're going to what you want because those people are apparently listening And no, I don't work for beam. So don't ask me. Well, let's uh, let's go a little bit um further on this Because this just happened. Um the other day this happened on march 25th. Oh, look at that Oliver Lambert has said he'll dedicate resources to work with the team at beam I will throw this in it's not hard to find it's in there. It's in the xcp ng blog post but there's obviously There's some discussion going on here. Um, they're willing to work together on this and he's storming gauntlet down That he's willing to Oliver the team at bates is willing to work with beam beam has got people replying So, yeah, there's Oh, and here's a fun one here the s n The new version the new api for the storage here right here from this is yesterday that this was posted Look for a blog post next week as an announcement on that. So that's great But yeah, uh, follow this for i'll throw it in i'll throw a link into the Chat for people who just want to jump right over there. But yeah, we we use beam at the cove and i'm wearing their shirt Actually for people that might want to laugh That's a great shirt. Yeah. Yeah, we we we we make a version of this shirt that we'll throw available on in my store at some point, but um It's cnwr. We're we're a beam certified I forget this. I don't know the beam sir because i'm not beam certified I'm not the person to ask but we have numerous technical people on our team We're a beam certified cloud storage writer. We have our uh, we have a whole beam storage repository in our data center So we've got we use a lot of beam. We'll just say that we we pay a lot of money in that company With licenses and things like that, but um, so yeah, we're interested in this too It's been a hiccup because you want one dashboard to be able to manage your backups and it does create some challenges for Someone who's doing what we do and what I say when I say we I mean cnwr We manage we have over like a hundred businesses we manage and some percentage of those are using hypervisors And we need to be able to back things up and I seen someone had question about like what's a good application aware backup Well, once it goes sounds application aware beam is Not cheap, but also known really good for their ability to do the full application where backup. That's why we Um use it. This is one of things. I mean backing up the vm's and creating Restores from from the xo backup. Hey works great and everything else, but We have some companies running some unique databases. I I can't try either a I got to shut the entire vm down to back it up. I cannot trust data in flight. I need application aware Once you've tried to restore some of these databases from a Snapshot where things may or may not be in a fully committed state. Yeah, that's a it's like cool The vm's back, but the database doesn't start because xyz or the transactions are messed up and now it doesn't work So you need application where backup I get the need for beam. So Yeah, that's I think that's the big thing is the transaction logging especially if you run a sequel Right that's being able to go in and say quiesce. We're gonna go ahead and start your backups So that's that's what it's at. And I also think that it's it's a Having the ability to run xcpg at home and be able to back up your stuff for free Is fantastic right now. I'm backing up my vm more stuff, but I had to spend like $8,000 on a Synology So, I mean it's it's a sunk cost. It was only $8,000. I don't run beam at home But like you get that functionality built in and it's solid. That's that's fantastic And I guess the same thing for proxbox and and a pbs proxbox backups are right is But I it you have those solutions Get it. You don't get that vmware. I'm never gonna get that with vmware No, and one of the nice things is when we sell like xcpg to a client who we're not managing your backups They're managing their own backups their own environment Because there's no multi-tenancy concept for backups inside of xcpg But when you're a manager's frider, you kind of need the multi-tenancy dashboard I can't just put everyone's backups in one giant pile that would not there are some different challenges that come with that We actually would have to build out a system for it and a way to monitor that system Not that we can't tool it. There's labor costs involved in tooling it beam provides that for us with a multi-tenant dashboard Everyone's segmented one of the concepts you have to really think about is if someone infected a backup for one customer How do you stop that from propagating to others beam has methods by which this works? Not all backup companies are made the same Um, that may be a fun topic of ways you can pivot off of backups because this has happened before where There's a common key that a manager's frider might use to set up all the backups of their clients But it turns out that same key gets you into the backups and can pivot through all the clients and uh Yeah, there's some backup companies. We actually I I have a list. I once I ruled out years ago I had I won't call them out yet because I don't know if they've ever fixed it But we actually told them the reason we couldn't use their backup service is because of the common key problem I said we found a way to extract your key out of a single client. They're like, yeah, that's how it works And I'm like, no, you need to build a new key. Well, that would require a whole another retooling of ours I'm like, yeah, but if I can get a key out of one client and I'm a threat actor and I pivot Now I can go into those backups to start deleting them for other clients or extracting the data, which is terrible idea so I don't want to get you off topic on that. We're we're gonna wind this down here. We could we could go on and on The longer you work in enterprise the more you start really having to think about this stuff There's so many attacks on all these things. We have to make it all secure. It's It's an ever-ending battle Absolutely any final words you have for the audience here, rich Uh, not really. I think that uh, one thing I'll say is that I I thought I was done with the series And I am officially but I'm I'm being asked to make a round up so I think that in terms of the The the four that I made there'll be a fifth even though I said there wasn't going to be a fifth and And try to compile a big list. So if you want to get the TLDR from the stuff I did just watch that one when it comes out Yep, you don't know the other ones you did like you said, they're linked down below in the description Subscribe to two guys tech. I got his previous videos on this. He's got another video to you by the way He's not just the virtualization guy. There's actually a lot of great Thank you. I appreciate that. Yeah, there's a lot of great content on rich's channel Uh, so like his subscribe over there, uh, check all that outlook for his new video that he'll be doing It'll be the round up which good news is he doesn't have to rebuild all the servers What we love for using content, but it's gonna grab all the b-roll you've had I kept it all just for this. Yeah, haven't haven't purged it out yet Yeah, man, that's that's Storage topics. See the reason youtubers especially as tech ones love talking about storage is because we have to store all that b-roll We created everything up. So It's uh In matter of fact, I I kind of laugh because what did Linus say because Linus has that new storage video and Like no matter what server he builds because of the scale you start operating at when you're at Linus tech tips size It's always we builds a new is he called who knock I think storage server and we hate it just Because it doesn't do all the magic or we hate it because it's too expensive. Yeah It's the fun the fun topics around that but thanks everyone for joining. Thanks for all of you Coming to this topic as I said Taylor's channel watch the next series. He's gonna do make sure you subscribe so you don't miss it All right, thanks Thanks, Tom. Thanks everybody