 Okay welcome back everybody Next up really interesting talk here from Karen beer. It's going to talk about bringing Zen to Centos 6 Which has been really one of the more interesting things that's transpired in this past year Karen beer How many if you guys have used Centos I'm guessing a few people have right okay. How many of you started using Zen because it was in Centos Okay In 2007 Something interesting happened Centos 5 was was made available Thanks to the work done upstream and some of the work that we did as well But what it really did was that this is the first time there was a complete open-source solution that he could use and abuse And do whatever you want to do in whichever way you want to do you could brand it sell it retell it You could use it internally you could use it with you know commodity storage on commodity machines To achieve stuff which otherwise in the past you'd have to pay a lot of money to a company called VM Where for this is really the first time there was a stable virtualized platform that you could just use it and worked out of the box With the best part about the whole about the whole you know ecosystem that came up around Zen and Centos 5 Was the fact that the community Created itself screaming shouting whether they wanted to whether they didn't want to it didn't matter the community happened on its own Because then was integrated in with the system There was there was no external component that you had to get in from somewhere It was all a part of the distribution you could install it and install it was smart enough to work out Hey, I'm being installed into a Zen environment I'm being I'm being installed into a PV Zen environment or a HVM Zen environment and it would make decisions for you It would help you do things like you know get the right Dry via modules up and things like that so it would do it would do some really cool smart stuff which meant a lot of people started using it and Then these people started blogging about it and then these people started talking about it And then started writing how to is about it and they would go to these Linux user group meetups and tell other people about the fact that hey I've got four computers on my computer I've got one machine and I can reboot any of those four at any point I want and I can do whatever I want It doesn't make a difference to the other machines. I Guess the point that I'm trying to get to is that these are people who weren't buying into virtualization because of something They're read on a you know in a white paper and an academic setup. These are people who are really finding benefits of virtualization and You know small abstract examples of this was a bump into somebody who's a who's a doctor in and So how many of you guys have been to India? So it's a couple of people have been to India and survived the experience I'm guessing the empty chairs are the ones who didn't quite survive the experience There's a couple of things that that are quite sort of you know back to basics in India and the medical setup You know the whole social government supported medical infrastructure is one of them And I was talking to this young doctor who's what I say young. I'm not really that old myself But youngish guy Who's using Zen on centos 5 to and what they were doing was they were building these VMs Which would contain medical records for statistics and for models for medicines that have been proved by their state department And they were shipping these VMs out On USB keys by post to all of the hospitals and all of the doctors who needed this information And all those guys would do is they would get a dot CFG file They would drop it into 80 C's end and do XM minus E create and Bob's your uncle and that's it If anything failed they had yesterday's or they had last week's image. They could fall back to this is not really Classic virtualization solution, right? This is not really the kind of problem domain that the guys that Citrix were really working towards right or at that point Zen source, but it's you know ingenious stuff that people came up with And that sent us six that equation changed a bit as in you could still do all of this stuff But you had to use something else Which created a bit of a problem and my top kind of addresses more of you know what we did how we did it Why we did it kind of a thing rather than the bits and bobs about it And I hope and I hope you guys sort of you know get some value from that So a bit about me. I'm based out of the UK. I'm based out of London. I Love the rain. I don't know why people have a problem with it It was terrible, yeah, I mean I had to shut my machines down in the in the garage and it was getting so warm I don't know what's up with that. I don't know why people live in places like, California It's like, you know, where do you run your servers? How do you cool your servers down? I've been involved with the centros project for about seven years and again because I Was doing something for my employer at that point which didn't quite work on any other platform And and the amount of money that red hat wanted to support us was was way they were like, I mean There were about four or five zeros more than we were ready to pay for it And centos seemed to be like a good place to be so, you know I kind of joined the community got involved with the thing and and seven years down the road and still around I love bullet points as you'll see through my talk and I and I love Trappist beer. I Haven't quite gotten around the whole IPA infatuation in the US. What is that all about? Everybody here seems to be wanting to drink IPA And really harsh IP as well. I think you guys should all try Trappist Belgian beers. It's like this is a whole new world up there So I've given you a bit of the background already that you know, sometimes five came out a lot of people invested in Virtualization around it to do some really interesting things I know there was there's a school project in in Brazil, which which does some stuff with virtualization as well where they deliver classroom content using Zen PV domains and They ship them out the ship or the examination content on a Luxe encrypted LV volume Which we can put on and the day off the test the teachers get a text message Which has the decryption password? So and then they kind of know that hey nobody else can get to this test and the test is delivered as a as a web app From a Zen Dom you Which is so it's like I think going back to the point It was a lot of really creative crazy stuff that people were doing around You know virtualization on five which wasn't possible earlier or maybe it was possible It just wasn't that main line. It wasn't that mainstream people weren't really doing like crazy stuff with it Red Hat had an agenda had a Policy had a principle had a marketing goal of pushing intellectual property that they own right and another point when six came out KVM was what they were doing KVM was like the in-house kind of thing So the message that they were pushing out was that hey, you know, we've just shipped six and that's fine You know six is great, and then there was you know lots of people think well hang on I can't run a dom zero on center six or rel six and And the feedback or what you know redhead marketing kind of was pushing out was that's fine You know so you can still run a dom zero on five and you can run your doms used on on six So you can still have the goodness of six you can still have you know a batchy two-two and you can still have PHP 53 or whatever and you know what's wrong with running your dom zero on on five you carry on running it There are a few differences between the Linux kernel two six eight and Two six thirty two right One of them one of the really really beneficial ones is Okay, let me not let me not let me not that slack off a bunch of guys A lot of people worked on this code right between two six eighteen two six eight two two six thirty two and they all really good features to have One really cool thing is power management about two six thirty Had a had a massive rewrite And when we did some tests and this was within our own centers build systems We've got like six or seven machines. We did a few tests and we found The guy who's giving us space to host those machines said that you can't use more than eight amps of electricity And then we upgraded those machines from five to six We suddenly found hey, we've got enough electricity to squeeze in another machine, which we didn't have earlier So what's going on with that? And on Dell gen 11 gen 12 machines We found that the power utilization of the center six kernel was between eight to twelve percent lower than power utilization of the same workload of The center's five kernel, which is a massive benefit to have Perhaps not so much if it's just me, you know on the laptop But if you've got a hundred machines, you can you know ten percent of electricity bill is a good way to have What is also happening was that a lot of tooling that was coming up a lot of the management stuff that is coming up now is coming up around six And five isn't really supported that much like a lot of the really cool stuff with puppet and chef and foreman and Kettle all of those things are being driven towards six and you want to be able to manage your base layers with the same tooling Right, so you want six on that So people are trying some interesting things to work around the solutions right to to work around this thing the number one was moved to Debian Right. Hey Debian's got Zen To seven years ago when I got involved in the center's project This is what we try to do at my workplace as well. We said, hey, you know, we've got 150 machines running my sequel shorted setup We're doing some really cool stuff. It doesn't matter what we use under it, right? Let's move to Debian It took us six hours to find the Apache configs Because this was a group of five guys who had basically used red hat technologies since I mean my first red hat install was back in 96 right, I think 41 or something And it's just you know, it had kind of worked and then you know Stuff kind of worked and we knew where things were we knew that whole 86 is conflict kind of stuff So what three or four days down the road the boss was you know So how's the Debian migration plan going on and you were like not very well This is this is going to need like, you know a lot more resources than we had planned for But the biggest challenge was that none of the tooling that we had in place None of the provisioning tools that we had in place none of the PXE configs We had in place none of the kick starts that we had in place none of the my sequel tuning stuff that we had in Place none of our PHP front layers we had in place and okay I admit I have used visual basic in the past and at that point we did have a very large visual basic 6 application That used to talk to this my sequel cluster of 150 machines even the drivers for that didn't work with Debian and We were like, you know, this could be a massive rabbit hole We have no idea, you know, what's at the other end of this and we don't even know when we might come out of it And I think a lot of people who tried to do this including some of the really large hosting companies Who tried to go the Debian root found that hey with the investment that they already had with Zen on five on cent five It wasn't really going to work out moving to Debian. They would have to either rethink the entire architecture or Move to KVM and they're like look, we've done the Zen thing for a while. We see the wins At this point Zen was also about seven years mature six or seven years into the majority cycle KVM was about maybe two and a half years To a point I think where Red Hat had only started supporting it in production about a year and a half before this point A lot of the things that you could do with Zen a lot of things that people already doing with then you couldn't really do with KVM So you had to come up with workarounds so that plan didn't work out very well, and you still needed new tools because when Zen shipped with five everybody invested on XM everybody invested in scraping XM Everybody invested in writing their own little API layers because all developers love doing that, right? If you get five developers into a room, they'll come up with 14 APIs and the next day They would have re-engineered three of them And all of those APIs were using XM we did a very little spike thing where I went around and spoke to a lot of people at hosting conferences So you know what virtualization do you use on Santas like Zen? Okay, so how do you how do you actually you know interface with Zen? Yeah XM, but do you really use XM? No, I've got these bash scripts which call XM and then we scrape what comes out and we do stuff here So again KVM was going to be completely new tooling even if you were using live, but it was going to mean complete New tooling and then the way you provision and what your provision was going to change Right, so that both of the steps weren't really working out very well So what we did in the within the project was we said hey, okay? Let's pull in our resources. We have a bunch of smart people ash people What can we do and then the first thing that a lot of people within the center's project as you everybody knows And a lot of people don't really realize is that the center's project is an independent entity from the center's Linux distribution As in the center's project are the bunch of guys who create the center's Linux Distribution they're also the people who do things like forum support mailing lists IRC Help out in bugs help out with code extend the system all that kind of stuff But the core values the core principles of what we do within sent us is that we don't promote sent us as a platform We promote what people do with sent us Right, so you so it's a user focused stuff. It's a completely user focused Ecosystem also, we're not developer and feature led the entire platform the entire sent us project moves based on user expectations and problem We're not looking to build new features So if you've got if you've got a little app that needs yesterday's version of lip curl and tomorrow's version of Libyama and you know The libexml 42 that's coming out in six days time Then sent us is really not the platform that you want to be using, you know There's there's a lot of other people who try and solve that problem But if you want something you can deploy today, that's still going to be here three years from now Then hey think about sent us. It's a it's quite a viable platform for that kind of a workload Which is why it also makes fantastic? hypervisor platforms and Our entire cloud strategy from the project side has been that hey, you know, you want to run the cloud Excellent everybody wants to run a cloud you want to run VMs open to windows whatever you want to do You know instances fine, but when it comes down to considering hypervisor platform consider sent us Install it today five years from now. It's still doing what it's supposed to be doing So the problem we had was that we don't really have the developer depth that was required to make this happen so we started talking to Lars and and David Nellie and seeing hey, you know, what what can we kind of do? So we had a couple of initial meetings with the Citrix guys We had a couple of meetings with people we thought represented the best user base And we brought in a bunch of people we said, you know, who are the ideal candidates that we would want to target Who are the people who represent the workloads that we really want to go for? And obviously right up on top of the list was me. I was like, hey, I want to solve my problems and then everybody else who was privy to this conversation said, you know, I have You know a corner. I have a niche ever, you know, like my own corner case and I want to bring that to the table So we kind of spoke about that a little bit. We did, you know, make make installs We tried to build a few things see how the patches could work We did something like take 41 at the time Zen for one and try to apply it to the last 15 cent or six kernels I was like, yeah, this is not going to work out So we did we did a few things and we came up with a set of core targets that we wanted to hit We said it has to be Zen for based Because that's what everybody really wants to be on right now. It has to have a stable kernel We can't get into a situation where we building yesterday's LTS or we building, you know, yesterday's mainline And at this point, we've also said that 2632 isn't really going to work out Especially with the red hat patch set it was going to be too much of an effort getting Zen for back in so that has to go And then from the user side, we wanted to preserve its existing infrastructure So we wanted kick starts that worked on five to still work We wanted like spacewalk satellite installs to still be able to consume it We wanted people to be able to install centers the way they install centers and then consume Zen on top of it And you wanted to target the needs of the existing Zen users. You're like, hey, you know Zen is doing great stuff in feature terms in terms of, you know, so I think to this particular audience I don't have to sell Zen, right? I think everybody's kind of bought into it But what we really wanted to do was not get into a point where it was feature led We didn't want to get into a point where, you know, we were building and releasing every Zen release all the time And then expecting all of the users to hit the upgrade Treadmill as it were we wanted to get something which is kind of you know going to stay stable And the process we adopted was the thing that we started doing in CentOS now and it has it has it has it's a double-edged sword I know some people in the communities aren't very keen on on us doing the sort of thing and why we do it But what we really do is we try and get people who represent the user base people who represent the problem domain on a private mailing list We try and find the people who can help them solve those problems on the same private mailing list And then we try and build sent us infrastructure around it because what happens is that there's an expectation In the user base around sent us branded code going out And even though you know you can have like blink flashing red lights around it that you know This is dev code do not use this people will still use it Right and doing this basically meant that we could build the process we could build the code We could get to a point where we knew we could actually ship something Before getting into a situation where you know 500 people are using it And we have to then sort of trying to call them up by saying look you're gonna have to uninstall this Or your machine is going to get hacked because we're not supporting this anymore Right and if 500 people are using it three people will have ever get in touch with you the 497 other silent majority You'll never have any contact with so we started using this process and I think it worked out really well For for what you're trying to do here. So we said okay two six thirty isn't going to work So if you're going to branched on to three point four LTS The main line LTS colonel for for the time being we'll try and get to a two-year production cycle Which is what you know based on user feedback that that was good enough That's what people are really looking for people didn't want the 511 year cycle for for this particular solution They were like to use good enough ensure that during those two years all upgrades and in place so we test for those updates And we'll build every even generally so we'll do we did for to we'll do 4-4 We're not actually doing 4-3 unless somebody extra tricks, you know picks up the phone and says you know Whoa, we've got a massive security issue. We're going to have to do this So the aim is that you're only going to build even updates and and work with that The colonel is basically the bog standard LTS 3 4 with a few zen specific patches including the block tap 2.5 How many people know block tap 2.5? How many people know that how many people I guess I didn't know there was something called blocked back to 5 I knew there was one there was two and there was going to be a three And then turns out there was a 2.5 as well Which some of the people said it's critical for them to be able to move to this platform and and the Citrix guys Actually put in the effort to get this going on that colonel From the Zen side we wanted to make sure We wanted to make sure that we don't fragment the communities Whatever we were shipping we wanted those guys to be able to go back to the Zen org or the Zen project or dog Infrastructure and actually talk to people there unlike Let's just Samba for example like the Samba shipped and sent us is about you know We two and a half three years adrift from from in line So you go to the Samba communities and say I've got a problem with this The first response you get is you need to install the latest release And we wanted to not get into that situation with Zen So what we've tried to do is we try and promote as much as possible The communities to come up around the Zen project rather than the Zen communities coming up around sent us project and then fragmenting it The ongoing plan I think we've touched on this already is you know deliver Zen for even releases do a two-year thing And and deliver and maintain the ecosystem around Zen sent us rather than around sent us with a bunch of people who use We use then Our primary user testing was a hosting company you may have heard of Is anybody here who hasn't heard of that? No somebody here, okay? Our second big user case was the guys who were helping us doing the trench work was another hosting company that some of you may have heard of and Which was which is the two large hosting services companies with about 180,000 clients one of those two without going into names had a 30,000 hypervisor cluster that they actually deployed this on and tested as well. So we knew that it actually works We had and there were a couple of small vendors as well There were a couple of people who build solutions around Zen and they kind of came along and said, you know How can we get involved and I think when we released we had about 80 odd people Representing different interest groups who were involved in the in a test release process Our biggest challenge has been engaging the users and This is a there's a bit of a numbers thing at the end Which I think people will find amusing as well But what what we've kind of realized is that we're hitting about a 1.8 to 1.1 1.9 percent hit rate in engaging users. So for every hundred users out there There's a one and a half person who actually talks to us Which is a bit of an interesting or an awkward position to be in because what's happening now is that the people who work with us within the Keywear release process Even a lot of them have now gone away and done a little bit of work around the edges Which which isn't really being fed back in and that's a massive challenge And we're using what people would normally do in the community to address that if anybody has bright ideas if anybody has you know Comments on how we can fix that please come and find me The the other problem has been engaging Wenders and providers a lot of people have started building and developing solutions around the Zen for Center stack It would be nice if some of them actually came and spoke to us because some of them Including small you know one-man two-man three-man companies all over the world are building stuff Which is otherwise open source and we've actually got a process in place that allows them to release the code into the centos ecosystem Giving them access to a massive user base But how do we actually reach out and how do we get them in through that door has proven to be a massive issue? So again if anybody has ideas if anybody has views on how we can help solve that please come and find me I would love I would love to have you know a better ecosystem that helps people succeed around around Zen for Centos In terms of future plans we're looking to expand scope We've got Seth and cluster FS coming in by the end of this month and that's specific to The functionality needed who within the Zen for stack as used by cloud stack and open stack So this is not so this is not Seth and cluster SFs going into genetic Centos repositories. This is Seth and cluster FS going into Targeting so at one of the law at the previous the dojo before last David Scott from Citrix did a talk on Open stack running with Zen for Centos Zappy and Seth and there's a video of this online on the centos project YouTube channel I think about a 1500 600 people have seen that video I highly recommend everybody goes and see it in about 20 minutes He got the entire cloud stack stuff up and in a stage where he could scale it with the whole stack Which is which is which is a really good place to be so Seth and cluster F has actually targeted that particular platform What we're also trying to do is eucalyptus and open nebula are both kind of community driven Cloud controllers, I'm sure most people would have heard of at least one of the two But both of them dropped the ball a little bit on Zen support and we're trying to engage with both of those Communities or both of the developer communities to try and bring support for Zen for Centos back in I think the open Nebula guys showed up at the What's it that Google hosted Zen hackathon? I think they've actually put in the code has gone in and the new release has support for Zen for Centos I'm working quite hard on the eucalyptus guys to try and get them to do this as well because they've kind of bridged over to only supporting KVM And we're trying to facilitate we're trying to kind of get resources going so they can start that checking this as well We've got an ISO which is ready to go. We didn't release it with six forward We're going to release it with six five which is pre booted pre loaded with Zen So it's a Centos installer which has the Zen code built in plug it in put up your machine And it'll give you a Zen first install on it I think James touched on the fact that we're trying to get zappy and most of the work has been done but we hit a roadblock in I think People change environment changed a lot of other stuff happened So we quite I'm quite keen on sort of rebooting that effort and getting it in before six five comes along And the other thing the final point in here is that we ship a custom version of LibWord into the Zen for Centos repositories, which is a different version of LibWord from the generic Centos six KVM LibWord And that's because we're supporting XM and XL and we also have the patches in for getting zappy and through LibWord in the Version shipped with Zen for Centos And Red Hat hasn't been hasn't accepted those patches in yet So the the LibWord that's in there is better than the LibWord that you can get in Fedora or Debian at the moment in that everything works including Somebody's going to probably kill me for saying this but I do know that spice kind of works as well Sometimes on Mondays when your machine is facing north, but it does it So I mean the support for the whole thing is there and XM and XL work out of the box And you can do remote management and you can integrate it with any LibWord script and stuff Okay, I've Lost something maybe hang on. Let's see this works. So a Little bit of a story of numbers, right? Centos 5 was released quarter one 2007. A lot of people enjoyed it much festivities around it You know Zen works people came up with some interesting things hosting, VoIP, HPC and in embedded Areas were what what I think based on what you know digging around I've been able to do with the biggest adopters for Zen on Centos And all industries which were traditionally Either not looking at virtualization at all or we're looking at massive, you know, window lock-in kind of solutions So it was it went well for many many years and I think I don't have the number up there But off the back of my head. I remember that at one point We were greater than 1.6 to 1.7 million installs where people were using Centos 5 with Zen at various points But in the first week of Jan this year I did I actually went through an exercise for a couple of days and try to come up with a number that hey Are we actually solving this now this problem for me myself or is it really somebody else out there? Just like me who's got exactly the problems that I do and it was kind of important that if you're going to commit resources You know, we do it to something which is worthwhile Because there's so many things you could do Let's find something which is worthwhile and the worthwhile was that we came up with a number which we can extrapolate a little bit Around and we said 750,000 estimated installs out there in January this year We're running Zen 3 on Centos 5, which is a big number Which was like that that'd be really good problem to solve for these guys and June 20th we released it and Kind of quietly there hasn't been a lot of fanfare around it because we've let it grow organically And I think the plan was that by the time we get around to having more support from the Zen projects out of things when Things like you know Zen server becomes a young install and things then we'll start making a lot more noise about it You know and things like that but September 10th I did I went through the same exercise that I had gone through in early Jan this year and the number of unique IPs that were tracking those repositories So this is not drive-by hits these are people who are actually hitting yum against the Zen for Centos repositories was about 87,000 I'm not sure how many people will be surprised by this But it was 98.3 or something like that IPv4 and there was like in the four guys who use IPv6 All of them were using this repository So I'm going to kind of finish off. I don't know how we're doing for time During the fun There's a there's a wiki page that gives you all of the details. I'll make sure this presentation thing gets posted as well I know a lot of people are going to be looking at the numbers thing to tweet around which should be fun The the effort is completely run off the Centos work list Join in the fun. The the git repositories are public the git repositories are available to anybody and everybody who wants to use them We're going to try and push them out to GitHub so that people can fork and do their own thing the Workflow that we're trying to set up around the zappy rpms is going to be on github. So good github.com slash centos slash Whatever the zappy stuff is going to be it's not there yet, but that's what we're trying to get to build system access Community access all of that stuff is available. It's in place look at the wiki page You should get you should get basically all of the bootstrap information you need Thanks, of course to everybody including a big thanks to the Citrix guys who at that point weren't really sold on what They were doing with open source or maybe they were they just weren't admitting what they were doing This was last year in November December a Big thanks to them for actually reaching out because the big problem We had a roadblock that we had was we did not have to develop a depth to do things like hey Can I get blocked at 2.5 over into the kernel? I actually had a go at it for about three days later I was like look this is going to need to be somebody who's looked at this code before Who knows what the hell is going on so and it was it was it was really good of them to kind of reach out And help us out with that sort of stuff also one of the things that is key And I haven't put it up on this on the slides there is the kernel that we ship the 3 4 kernel that we ship is actually curated by a Bunch of people and in that bunch of people include kernel engineers at this is in a project So if you look at the change log, you'll see guys at Citrix comma doing commits as well So should you trust that kernel? I think you should you should trust it because it's a main line It's an LTS main line So you know what's happening if the security fixes happening upstream they're coming through if the Zen specific stuff The Citrix guys are pushing that through I am kind of hopeful I don't know if this is going to happen or not I'm kind of hopeful that in futures and several products They actually end up using the same kernel which makes our life a lot easier because we can just use those sources then so the The the plan and I know Johnny's working on this He's done a spike as well is that we want to start rebasing three four to three ten sort of March next year Because there's a couple of one of the big reasons why I'm caught keen on still pushing Zen Even though a lot of people keep saying hey, you know when KVM works. Why are you kind of wasting your time? It is something like what Larry touched on that's then because of the way it is and because of what it does And because the way that it runs kind of above the Linux kernel It lends itself better to the whole disruptive hardware situation that we ran at the moment So even on the Intel platform people are engineering around what is sort of established conventional engineering practices And in many of those situations or many of those problem domains then just fits in as a better hypervisor than KVM does so for example, you've got a MIPS based Network switch you want to drop OBS on it. You don't want you know, not all software on it But it's fine. How about you know something from Bay Networks from like 1932 So, you know, you have the option of actually dropping in as an hypervisor and being able to do a bunch of this sort of stuff I'm not sure if this was NDA or not, but I haven't signed the NDA so I'm fine with that Arm of course is big and that's coming through and we trying to bootstrap arm on For sent us purely because we want to deliver Zen with the center solution Because we think the Fedora guys redhead guys are going to do whatever they do with arm anyway But what we want to be able to do is deliver that hypervisor there because we think when you get to situations like, you know 288 cores in a single box But each core has limited processing capability each core can only hit 1.4 gigahertz and can't do you know Context switching becomes a bit of a challenge. It's nice to have the Zen wins on some of those platforms as well A couple of links if anybody's interested My email address is on that if anybody wants this t-shirt, which I think has got the Vintage logo. Let's not say the old logo the vintage logo of the Zen project Send me an email send me an email with your address and I'll make sure and your size And yeah, so American sizes please and I'll make sure I send something out to you So the last slide I've got before I wind up is I I Started the whole talk with you know that the primary focus of what we were trying to do was solve the problem That whole bunch of people out there whole bunch of hosting companies out there were running Zen on five Right, so one of the key issues that you know, we wanted to hit was hey How can we just you know get to a point where the guy puts in a USB key and reboots the machine and Six minutes later. He's running Santa six and all of his VMs are still there all of his storage is still there It was a good dream to have And we tried quite hard to make that happen But it is tricky because because of the number of changes and because of the way So one of the wins that people had with the XEP platform was that because the kernel is different and because access to that Domain is controlled by a Citrix very few people go in there to install for example Oracle DB right Whereas on centers five because it's native because it's already there So the so we had three guys who said they would really love to help out with this testing stuff One guy had an Oracle database running on his Dom zero and he like well hang on. What are you? You know? Is that is that a good idea? And he did set up a dev instance of his platform and he didn't go through a couple of iterations of trying to do the migration Script it wasn't going to happen. It wasn't pretty it wasn't going to happen One of the other guys had this high performance my sequel stack on there and and somebody else was running his DRBD stack on the Dom zero which is which is not really a Corner case it's it's you know Lot of people would want to run like cluster LVM on the DRBD or something and use that as the storage that they exposed to the Dom use and we found that you know just the migration from DRBD 8 1 to 8 4 Was was painful and you had to synchronize it in a way where you would have to bring the cluster down anyway So then it was a case of what can we do and what is the best way that we can make this happen? So we have a migration script Which will take a seven plus five machine and you will end up with a seven plus six machine And it does some crazy things like non-district content will get our synced away to slash opt slash deprecated So if you've installed with stuff that make make install it'll find all of the bits that are not managed that didn't come from RBMs Todd ball all of that stuff up and leave it for you to handle later Which effectively means your machine may not boot if you're running for example if you've been so-called PKI certified by Certain large companies whose names don't need to be brought up and they've done make make install open SSH installs Because that hits the right version numbers. You may end up with a machine that doesn't boot Hopefully not a lot of people have done that kind of thing but It's something to look at it's something to consider it goes to three reboots because it does it changes stuff at different points And during that during those three reboots everything on your machine will will be down the General line that the project toes is that if you can't find that script on your own Then you shouldn't be using that script Because it will cause damage, you know, and it's not the kind of thing like you know C panel Hey, I can just go in there type in a command and I end up with C panel. That's not how this works And because of the fact that it's intrusive and because the fact we say that you know There's a good chance your machine is not going to come back up We don't link to that script anyway We don't talk about that script anyway like if you can find the script which means There's a good chance. It's a it's in a git repository and not a very hard place to find But if this is a problem that you guys want to look at then then there's an option But what we say is hey migrate your VMs instead Just if you've got ten machines take one out install sense six on it migrate your VMs and then cascade down and just do that loop And I think most people would be familiar with that process, right? We haven't tested live migration from let me okay Let me rephrase that we have tested Live migration from three to four, but because this is not something that Zen project supports We usually say that we haven't tested it In certain cases if you're just using some very basic stuff if using fireback stuff and just raw storage or VHD files It does kind of work But I believe that official line is that you know don't do live migration is not probably going to work So just offline it and it should go through So is anything anybody wants to talk about any questions that anybody has or would anybody like to know anything about Zen on CentOS or about CentOS or About Zen I did have a really nice demo where I had a I have I actually have hopefully I still have nobody's run away with that machine I have a machine which has about half a terabyte of RAM and One of the things that that's really cool, and I've done this in the past is bringing up a thousand VMs on Zen in about it takes about 18 minutes for all of them to come up because you know Malik takes time when you've got that much of memory and then DHCP takes time when you've got that much of memory and So Because I believe you didn't have time I haven't done that also when I was sitting here in the morning and I was looking around it Who's here? It's like maybe bringing up VMs isn't going to be a very interesting thing to see right? So I think most people have done that already So kind of cycle that out. But is there anything else anybody wants to talk? Yeah, I was curious if you had like a Best practices for optimal performance like You know how much memory Per VM or what to put on your Dom you at Dom zero You know how many CPUs to put on each In I must be really growing old I can't remember the day but in March this year Roger Bow who works with Citrix did a talk at the CentOS dojo in Antwerp Which is recorded is a video of that available on the CentOS project YouTube channel and the slides are available Where he did 45 minutes on how to performance tunes and for CentOS and he went into exactly all of these things He went into things like you know, how do you how do you solve the numer problem? Yes, how do you get your storage stuff sorted out? How can you boot in various places? What you know? So even things like everybody seems to have this real big infatuation for QCOW and especially the breast QCOW But it's really interesting to see under real workloads. What happens to the back end with that even things like if you're running PV stuff on Zen4 you still need QEMU instances running in your Dom zero for some of the some of the backing stores So what impact those have and how to manage memory in your Dom zero discovered all of that stuff really well And I won't even start going into some other things. Okay. I that's exactly what I would like to get If you drop me an email, I'm happy to send you a link to his talk. The video is there And his slides are available as well And it is a fantastic talk. I just I don't even want to kind of go in there And if you really want to kind of get the whole stick experience Two sessions after him was Jaime mellis from the open ebola project doing best practices of KVM based VMs in a cloud environment Which is also quite interesting because a lot of stuff they were talking about was was common And a lot of the things that one guy brought up is are also relevant to the other setup And what the other setup brought up was also relevant to the other setup I think both of those talks are worth looking at In sync and it's about two hours of your life. I think I highly recommend it Thank you. All right. Well, let's thank Karen Bair for this wonderful talk