 I'm here from Orange Systems and when I did a video about XCPNG versus Proxmox, it generated a lot of discussion. This is kind of a part too because I wanted to share more knowledge and not just for me, very specifically it'll be from Oliver Lambert. He is the CEO of VATES, the company behind the XCPNG and Zen Orchestra project. And this really goes to show their community engagement level because he watched my video which is exciting and also, you know, had some more information if you will to throw in the list here. Now I, as I said in the last video and I'll say it again here, I am just here to share knowledge, share my experience, occasionally drop a few tutorials here on my channel and I always like to cite sources where I get information from so you can read further because sometimes I can condense things to this level to keep the video brief but there's a lot more context. And among the context though that was confusing to me is people tell me that the Zen is dead or that the Zen hypervisor is old dated technology that is not in use anymore, etc., etc. There seem to be more people complaining about a lot of things but they didn't bother citing their sources and then on my live stream the other day I kind of mentioned as well that can someone, you know, give me some more context and then Oliver Lambert did a really nice write up about the, well, context for all of these things. So we're going to go through some of that, talk about a little bit of history between the two different projects and some architectural differences and of course I'll be, you know, adding all these links down below. So if you don't want to listen to me, you just feel like reading everything. Absolutely you can do that. So too long didn't watch the video or too long didn't feel like reading. Watch the video, whichever way is right for you. Just carry on with me while we jump over to Oliver's forum post. Now this is over in the Lawrence Systems forums, but I will completely thank Oliver for taking the time to post in my forums. But please note if you have a lot of questions or want to participate in the community with XCP and GNZ and Arcushia, they do have forums that are also linked down below where they're very active and engaging with the community, which I think is a really important aspect. But we'll start here. Just watch Tom's latest video. Thanks again, Oliver. And wrote a large comment given explanation and YouTube then disappeared. This is the reason I have forums because yes, you can spend a lot of time articulating a reply in YouTube and I don't know what happens to it, even as a creator, I do not have wonderful insights into how their algorithm decides what's a good comment and what's a garbage comment. That's a different topic all together. So I suppose Tom could probably make a video version of this. So it's more convenient for people to listen to watch, too, for the TLDR. But hey, why not both? There's a forum post linked down below and here's the video. Now first step right here, Zen versus Zen server. I have undoubtedly conflated these a couple of times because of naming nuances, not on purpose, not to mislead anyone at all. But this is the clear distinction between these and it's laid out in writing from Oliver. He is the source of truth on this piece of information. Zen versus Zen server. The hypervisor is just Zen, not Zen server. I know I've conflated it at least once or twice over the years. Zen server is the whole platform made by Citrix and the source of the fork that gave us XCPNG to be even more precise. It's called Zen project. So I'll make sure to add those words so I make sure I don't complete them in my head and hopefully you can do the same. And since Citrix didn't want to give the Zen brand to a project hosted in the Linux foundation, because yes, that's where it lives right now to avoid business confusion. This project is a true GPL v2 open source project with contributions from Citrix arm. A lot of these see no below. Susie, AWS and so on. There's also big names in the pre disclosure mailing list. Members can be seen here at the Zen security response. By the way, security policy as Zen project is one of the best known in the entire IT world. Those users are in general big Zen users since they must patch some issues before it's public. It gives you an interesting view. Now, this particular comment right here. Why arm and Zen? I thought this was interesting in the automotive world. Zen is a perfect fit. You can statically partition the hardware, i.e. one motherboard with multiple CPUs assigned to various VMs. For example, one VM for entertainment, one for ECU electrical assets without security, just nor memory data leak since the memory is truly isolated and Zen controls it. I thought this was interesting from a standpoint of other use cases that you may not be thinking of. And certainly ones I didn't cover in that video because they were way out of scope of that particular video. And to add more to the complexity, this is the naming complexity Citrix renames in server to Citrix hypervisors. Don't want to call that now. So that did add to the confusion. But they're the same thing. It's what Citrix has gone forward upstream and downstream. This is also a question I've seen often for people without deep knowledge on the Zen project or Citrix and project is the upstream like the API called X API. Those projects are hosted by the Linux foundation. Everyone contribute to the Zen, as I said before Citrix hypervisors and server is the downstream of those projects. XEP and G is also a downstream and at the same level as Citrix. So they both use a Zen project as part of their upstream pulled into the projects themselves separately now called Citrix hypervisor and XEP and G. So there's in case anyone's curious about the clarifications or relationship, it's not exactly a fork of it's it's running parallel to would be probably an easier way to describe it. So no XEP is not just downstream of Zen server. It would be just part of Zen project. Once and then the conflation issues. Now this is the one that I think people may find interesting. And if we go over here to the Zen project and we look at right here over 10 million people use Zen with an ecosystem of more than 2000 commercially certified partners. How do you get that large number? And it's going to be AWS. But then there's people who seen an article that lacks all the context of things. Again, let's go and zoom in a little bit on this right here. What is the nitro hypervisor? The launch of C5 instances introduce a new hypervisor for Amazon EC2. And this is right from Amazon. So I'm not like trying to read some news article because the news articles conflated things a little bit if you go and read them where they say Amazon's moving to KVM, which isn't exactly true either. And we're getting the context in just a second. A certain section of Amazon is specifically C5 and everything else because it works is still using Zen. But is it? That's where we'll get into this. It's simple. AWS became the first public cloud provider thanks to the Zen project. It's a fact since Zen GPL v2. Note you can modify the code without contributing back upstream since as a cloud provider, you aren't actually delivering the software. AWS did that a lot because they wanted to protect their competitive edge and virtualization space for their cloud service. They also built their own tool stack, which isn't public nor open source. Anyway, at some point AWS realized they had the power of building their own stack from the hardware to the software. They practically invented the concept of DPU data processing in it and thinks that are huge experience running Zen at that scale. Millions of machines decided to experiment things from scratch, building their own dedicated hardware to get the most out of it. But they custom software stack on top of it. Funny enough, writing a hypervisor isn't that hard if you know which hardware will run it. What makes Zen relatively complex is the fact that you can run at most all eight x86 platforms in the world. Believe me, it's really hard. Hardware is buggy as hell. You can constantly need to adjust various bugs from the hardware itself. BIOS, UEFI or firmware. It's a nightmare Linux is full of workarounds too. But with that opportunity, you get to run your own hardware. AWS started from a blank page. Nitro, their hypervisor isn't your usual KVM. This is where it's a little fuzzy that they're actually using KVM because it's so stripped down and so customized to AWS's use case. They are not doing the same thing. We're doing servicing businesses. Even if those enterprise businesses are fairly large, you can pretty much guarantee they're not as large at this moment here in 2022 as AWS. So it's going to be extremely different. And for those of you in the home lab, which I think are the most well, loud commenters on a lot of this, it's probably extremely different than what you're running in your home lab. Back to the topic here, though, the design is really great, but don't expect to find this in your distro with regular hardware. That's kind of the gist of that. Also, this is the AWS switch from KVM, as I commented, not true. Also, because they won't replace all their instances with Nitro. Most certainly is probably still running on Zen because it's there. It's secure. It's compatible with any of the machines they can afford off the shelf. So AWS just added a new solution to their stack. They did not replace their stack. So that's an important just for concept for those of you that wanted to do the arguing about the numbers of who's got the most installs out there. Well, technically Zen does, but I'm not here to tell you that's why you should use it. And AWS is barely a reference as why you should use one product or another. They're not like the example here. But why is KVM, why is KVM around? KVM success is undeniable. There are multiple reasons that some are technical, but there's a cost to that. We'll talk about that in a moment down below. And some are due to the main players, that being Red Hat. So while the Zen project is hosted over here at the Linux Foundation, it's part of that, the KVM project. You can see a little Red Hat logo out there and they are supported by Red Hat. They were worried because Citrix not the best maintainer of some of the projects at the time in their earlier days. So, you know, Red Hat wanted to have something on there. Zen was the first open source hypervisor. It could have been the unique player in that area, but it's not the case today, since you can see a lot of KVM deployments all around the world, but why KVM just exists. Because when Citrix acquired Zen source circa 2007, everyone was afraid. Citrix does not have a great track record with open source since their core business is a virtual desktop and working with Microsoft. Imagine you are Red Hat at this time and you are a Linux kernel champion and there's a hypervisor made by Citrix going everywhere. So, you're reacting, you create KVM, a module on top of the Linux kernel because you know very well, you know Red Hat's big contributor there, to do virtualization. So, you are not dependent on Citrix doing work with freshly acquired software. Well, just leave it at that. That's completely legit reason I completely understand it. Also, you're Red Hat, so you're pushing forward everywhere. After all, you're the number one company in the open source and Linux world. So, you integrate it within a product so it's easier to use and serve virtualization context. On the other side, Citrix is in fact not caring for virtualization market despite having great product called Zen server because remember, Citrix doesn't care about server vert market. They are all in desktop vert market, more margin, bigger dollars and their core business after all. And that's where KVM started to get more traction. And both of these projects been around a long time. So, neither one is, you know, small, both big, both been around for many, many years. This is where we can dive into the technical aspects and this is where I'm very interested in some of the core differences. The other reason is technical sense. It's integrated push by Red Hat people started to rely on it but also contribute to it. From a technical perspective, KVM is far more permissive than Zen. So, you can do fun stuff relatively quickly like for an IO and such. However, this comes at the cost of less isolation due to a less isolated model architecture than Zen, which is a true type one hypervisor. Zen designs closer to ESXi, KVM is more open bar. But this aspect is also part of its success simply versus security in terms of adoption and development. I'm not saying KVM isn't secure. It's a different design, which is inherently less secure. Is one better than the other? Hard to tell. There's no silver ball. And this is very true. If you have a more open design with a lot more lines of code, you can say, look at all the fun things I can do. And then you can also see the risks that come with it. People, you know, sing praises of wire guard is another thing I talk to my channels being more secure because there's less code, but open VPN is still more popular because it has all the features people need. So having less code maybe doesn't have the features, but you can see where there's different arguments for both sides of it from an architectural and neither answer is just right. There's not just one or the other and that's that. And I know it's slightly out of context to talk about VPNs versus that, but I'm just trying to make a few rational ideas here. In my opinion, it's not what matters. And what matters is the integration and a product easy to use. We have an edge on security aspects with XCPNG, but we have to do more work to develop new features. That's a really important aspect right there. Another aspect I love the Zen is its size. Now, you gotta remember, there is an precious beginning. Oliver, as I said, he is the CEO of eights who's head of the Zen project. So of course, he's going to favor the Zen project as I do because I use it as well. I'm very open about my bias towards it. I'm just here to drop some knowledge and leave you some links, but in give my opinion on all of this. Another aspect I love is its size. It's not a perfect metric, but if you give an order of magnitude, Zen is the entire code base, is 2% the size of the Linux kernel in terms of lines of code. This allows for a very interesting thing. It's a true type one hypervisor. You work first in the hardware on Zen, which is kind of a microkernel in the end. It means the attack surface is really low. Anytime you can have less code, there's just easier to audit. That's just a fact. Since it's not that big, you can actually read the whole code base and understand everything. This is a fantastic way to discover what a hypervisor is and move further to improve it. This actually does help people get involved in projects when there's less code to learn to understand a project. Also, the Zen project isn't too big either, meaning you can actually make things moving faster if you invest in it. Linux KVM contributes in just the most of IBM and such. All right, let's talk about Zen's future. Zen is not dead. Neither is KVM. Neither one of these are dead, despite the people who seem to be arguing in the comments over this. This is where it matters. Here at Vates, we are committed to become one of the biggest maintainers of the Zen project. We invest a lot of R&D into it, because, yes, there's a lot of things to do. That's why our XP and G dev team doubled in size since last year, and the number of 10 dedicated devs, three new devs this year, at this place, we clearly outnumber even the Citrix Zen server team very soon, if not already the case. That's also why we're not afraid or anything that could happen with the Zen project, even if Citrix decided to stop at some point for whatever reason. It's really hard to guess their intentions. Zen project is fully independent, open source project, and it found a new welcome home at Vates. And I was singing praises, and still am, of the automated backup testing that's occurring in there. This comes from both their community engagement and the rapid development, being able to add features in there. So they're very committed to the things that people want to see from the community, want to see in this project to make it easier, especially because so many of the people using XP and G and Zen server are very large companies, as I mentioned, running large numbers of VMs. So they focus a lot first on that use case of managing large scale infrastructure. Now, let's talk about Proxmox versus XP and G and Oliver's thoughts on that. I've heard some heated comments at LTS Tom. We also had the same in France when someone on Twitter previously using Proxmox decided to switch to XP and G. Some people from the Proxmox community went angry about it. I don't really understand why some people feeling threatened by other people's choices. And I feel weird with that, too. People seem to over criticize occasionally and, hey, that's their choice to do so. But I'm, once again, just sharing my knowledge and my experience with the product. And I mentioned even Jay from Learning Life TV has a whole series on Proxmox. So I offered people entire training and opportunity for that from someone who also is a Proxmox user. So I and Jay feels good about both projects, by the way. So, you know, me and Jay are aligned on a lot of that. Back to here. Here at VH, we deeply respect Proxmox project since it's fully open source. I don't understand the level of respect against... I don't have the same level of respect from a corporate policy, from a company leader on the virtualization market. Yeah, we know who they are. Always pushing their prices to the roof. That should be enough clues to figure that one out. If you love it good for you, nobody's forcing you to switch to XP and G. But respect their technical level and their engineers, obviously. Because there's a lot of good engineering in other products, even the ones that are closed source. There's a lot that goes into, not just the hypervisor, it's everything around it that gets complicated. It's not just virtualizing just some machine. It's everything around it. And that becomes the bigger challenge. It's also important, I think, to understand difference in philosophy between the both platforms here at Vates, we want to build the most integrated virtualization platform. We are 100% dedicated to that because that's what Vates does. That's it. Vates is extremely focused on just delivering the Zen Orchestra and XP and G and everything around it. I mean, code contributions related to it, of course, as well, but that's the product they make and not other products. That's just something I would really want to hammer home that you take a team that's entirely focused on one thing and from the CEO on down. I think you get a really good experience with that. That's been, you know, me watching a project progress over the years to see where it was several years ago. And I can go back and watch an old video to see what features it didn't have. And I'm so impressed with all the cool things it can do here in June of 2022. So let's go a little bit further down and read this couple of last sentences. Another difference is a philosophy to be truly open, not just the code, but with our communities. Zen Orchestra and XPG are direct result of users feedback. We made our best to simplify user contributions on GitHub, but we also work with partners, hardware, software. We truly want to build an ecosystem. It's not people coming to us, but it's truly us coming to different communities as this very post. And I don't think you can get any more real than the engagement you have with them on Twitter. The fact that they took the time to register, log in, and type this up inside of my forums and not just comment back to their forums. I mean, this is them being as engaged as possible. Finally, the other big difference is the technical aspect beyond Zen and KVM. We want to provide an integrated experience that doesn't require any Linux knowledge. That's why the XAPI, API of XP and Zen Server is great. Creating a storage doesn't require any Linux commands, same for network and such. So it means to not be tinkered with like you can with Proxmox. This is a direct consequence of the KVM Linux philosophy where the hosts control and the entire ecosystem, which is not the case in XAPI and G. Your host is in fact Dom0, which is already running on top of Zen. It's a little bit of a difference because that's why I say you don't want to modify it too much. Someone said it was proprietary and it's fully open source proprietary is completely the wrong term. It's just not a full Linux distribution with all the features, as I said, when you're using the XAPI and G it's because you'll build a series of hosts that are very stripped down. But that also reduces your attack service because there's less in there. It may reduce certain opportunities, but they build them in as feature requests. So they only build things that are necessary for the hypervisor as opposed to all the features that come with the Linux distribution. Keeps it lighter weight, keeps the attack surface down. It's just a different design philosophy, but it's not proprietary. It's still completely open source. And you can still tinker with it if you want. You can grab the source and spin it up and modify it quite a bit and many people have, but a lot of people in the enterprise space like a very homogenized, clean environment that does a singular focus thing. That's why it's so popular in the large enterprise space where we've done a lot of consulting with it. And so has the team over at Bates. Hope they answered a lot of relevant questions. This is a foreign post that you can comment on too. Notes on the other parts for some questions that were asked. Network drivers, E100 versus real tech. This doesn't matter as you're using PV drivers. When you boot Linux, for example, the emulated drivers are only used before loading the kernel, i.e. grub, the Linux while making such a PV driver. So there's no emulated NIC, no emulation at all. So there's also no NIC speed limitation. You will be limited by the PV drivers, mostly CPU speed your host to the ZenPV calls. E100 only matters if you're on Windows and you don't want to use any PV drivers maybe because you're still running old Windows versions. But I've talked about this, how to get the Windows update drivers to work and different CPU models. If all your VMs are in the same pool, there's an automated mechanism that reduced to the CPU features to the older CPU. So there's no issue doing live migration between VMs on those hosts. If you want to migrate between different pools, there's always migrate from the oldest to the newest CPUs. More instructions on the destination is a problem versus the opposite. Indeed, if you do, your VM will be booted and some available CPU instructions, if it suddenly disappears on a destination host. The guess will crash when trying to call them because they don't exist. And one little thing I wanted to mention on the security for those of you that haven't heard of Cubes OS, they also, for the same reasons, use a, you know, a Zen project-based Zen hypervisor here inside there. And it's also for that, you know, really restrictive isolation between all of their... You can probably find it's not something I'm planning on reviewing some people talking about Cubes, but there's a reason they choose Zen as their core as well for some of those same reasons. Now, as I said, everything I talked about will be linked down below. So you can dig into these things yourself, gain an understanding, engage with that forum post, head over to the XTP and G forums and engage with there. If you want to learn more about their project and their product, follow them on Twitter. They reply and engage there. They really do care about the community. They have taken the time to interact and engage. And I think that's just a really important aspect to me. And I welcome, if Proxmox has some thoughts they'd like to share and you want to participate in my forums, they're pretty easy. Head on over there and do a post. So it's not me telling anybody what to do. It's just about sharing some knowledge and thinking about the different aspects of these architectures and letting you know that both of them are very alive and well. Both of the core hypervisor projects and both of the projects that are XTP and G and Proxmox themselves. So there's still plenty to go. Either one is still a good choice, as I said in my other video. Comes down to what fits your needs. Links down below to everything I talked about. Head over to my forums for more in-depth discussion or even an in-depth discussion with Oliver Lambert who took the time to write everything up in there. All right, thanks. And thank you for making it all the way to the end of this video. If you've enjoyed the content, please give us a thumbs up. If you would like to see more content from this channel, hit the subscribe button and the bell icon. If you'd like to hire a short project, head over to laurancesystems.com and click the hires button right at the top. To help this channel out in other ways, there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts, and offers, check out our affiliate links in the description of all of our videos, including a link to our shirt store where we have a wide variety of shirts that we sell and designs come out well, randomly. So check back frequently. And finally, our forums. Forums.LauranceSystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel. Thanks again for watching and look forward to hearing from you.