 Well, we have the pleasure of having Misha Peters come and talk about the open VST hypervisor in the wild There'll be there'll be time for a question and answer. It's at the end So please hold your questions until then and we'll leave a few minutes at the end with that. Thank you It's a little bit weird that you're introducing me though Being the guy that's written it all and some other people here. So good morning. Welcome. I like the intimate Room that I'm in. It's really nice And so I'm gonna talk about the the bees the hypervisor open bees the hypervisor that I'm using for open bees the Amsterdam First I have to apologize. I'm presenting this from a Mac I was not planning to but I'm flying to the US and I don't like to take my personal notebook into the US So that's why I'm using on my word notebook. This is my personal notebook so that you know And a little bit about me I Started way back when at excess roll ice being the Netherlands When it was like this in the in the picture And since then I started working for for vendors So I worked for for companies like like blue coat and foundry that people might know And also started around that time with free BSD Later on added open BSD to to the mix and when Mike started working on the on the hypervisor. I thought oh, this is great. So let's let's do that So I'm gonna talk about how it all started the setup the things that I've Seen what users are experiencing so hopefully you get a good picture of What's going on and I've doing some hosting in call location since 99 as a little bit out-of-hand hobby That that snowballed But what about you? So who's using open BSD? Cool Who's using VMM VMD? Who's an open BSD Amsterdam? Yeah, all right. Cool. That's that's pretty good. Awesome So how did it all begin? I? Was always using Forms of segmentation or virtualization I'm always looking for something that was running on the BSD's properly or useful So I'm I started with jails still using jails. I really like jails Then came be high for me at least I do like be high I think it's a little bit too complex So I was very happy when Mike started working on on VMM VMD, and I just moved everything Since I'm not running any Linux VMs anyway, so that helped The other thing is What I noticed on my call location site was that a lot of people were moving to the cloud So a lot of people would just leave Remove their hardware. So I had some rec space left. I have some hardware left. I had spare IP space IPv4 space I must say And I also always wanted to have a domain with something BSD And I think Hessler has taken a lot of the cool ones and probably some others here in the room Okay, and then thought Amsterdam came along. Oh, okay, maybe I can do something with open bsd.amsterdam So that and also wanted to find a way to actually contribute back to the community I'm not a C coder or or coder that much. I'm a spaghetti scripter So I can get things done but not in a very structured fashion And I also wanted to see and and and help out a little bit how far we can actually Take this thing and where things were were breaking So that's how it all started. So on the on the right hand side, you see a picture of some of the machines that we're using and Domain for shits and giggles. So I registered it last year in June And that's when the whole are a pro actually March and that's when the whole thing Started kicking off. So where is it? It is in Amsterdam It's actually in a data center in Amsterdam. I don't live there, but all the hardware is there It's in the access role data center, which is KPN these days And it's all running primarily on del R6 tends connected to a foundry switch connected to a foundry router I worked for foundry. So I have a love relationship with foundry. I get the CLI I've tried juniper doesn't really stick And I buy these things on on eBay. So it doesn't really cost a lot of money So it it started on Twitter. I started talking a little bit with people on Twitter say, hey What what can we do? Would you would you host a VM? And the first machine I brought online Which was some of the spare hardware that I had it's a box with with a gig of RAM There's still some VMs running there And this was actually one of the first let's say official tweets that I send out And I also wanted to know what people were willing to pay. So I Send out a poll on Twitter and say, hey, what are you willing to pay for a VM that looks like this? So a lot of people are around the five euro this in euros five euro mark and I went, okay, that's that's fine So started building VMs Based on that. So they're five euros a month and of the donation or of the the yearly fee I donate five euros to the OpenBSD foundation ten euros. Sorry Then I was able to get rid of some VMs that I was running on this machine that I brought online And this was actually the machine that kickstarted the whole thing This was able to take 50 VMs On it and out of that came the first donation to the OpenBSD foundation, which I thought was really cool And we I was able to donate 400 euros based on the first I think first month of running this So a little bit of the statistics so you know where we where we are The last donation was 370 euros 850 last year 1700 year to date total of 3,500 euros I'm running eight hosts. I still count the first one even though it's it's Doesn't really I want to move them. There's only 10 VMs currently running on there But it I still count it and there's roughly 280 VMs so we had some renewals already and what I'm seeing is that the amount of renewals Are the amount of people that don't renew actually get replenished by people that get a VM So it's pretty stable and I really want to break through the 300 no idea why but it's this mental thing that I want to get So what do you get? You get an opinionated VM if you've been on on the website. That's what you what you see you get an opinionated VM So initially that meant I decide the install sets I pre-configure a couple of things and it's running OpenBSD VM on OpenBSD I'm not going to give you a Linux VM or or anything else. It's an OpenBSD VM Now with 6.6 coming and sys upgrade. I'm gonna let go of the install sets I'm just gonna install everything so you don't have any issues there I heard Theo yelling at a couple of people on are not heard saw Theo yelling on the mailing list if you want to I also asked that question To exclude certain sets from the sys upgrades a process and you went don't do that Okay But what do you actually get so you get? 512 mega RAM 50 gig disks IPv4 assigned via DGP even though it's a static IP address that's assigned to your VM And you get an IPv6 subnet slash 56 that is statically assigned during the whole install process And what I've noticed during doing this. I'm very used to just assign my router as the gateway And so I started adding v6 addresses On the gateway and then all of a sudden. Well, I cannot hold more any v6 routes That's annoying. So what I started doing was actually the host is the gateway for all the VMs in regards to v6 so v4 still goes to The router itself because all the VMs are in a slash 24 I'm thinking about changing it because I'm hitting some snacks with bridge and I had some interesting discussions already with some people but that's something that I'm Considering or at least investigating if there's a way around it But personally, I do like layer 2 is Blake in the room Sorry, I do like layer 2 It's easy Anyway, so what does the setup look like? I try to use as much in base as possible. So it does influence Some of the design decisions that we've done That I've done So this is pretty much the list. So I'm using pearl How I deploy so I've written it deploy script in pearl Using VMM VMD, of course the HPD auto install site Scripts during the auto install process HDB to serve the sets and then sensors D And I mentioned this specifically because I figured something annoying out which I'll mention later To keep track of the of the discs primarily and of course vi So what I'm doing with pearl is I started with a very nasty script Because when I when I announce, hey, you can now get a VM I had this this half baked form and I had some information in there and I threw it in a file And I went, okay, so how do I actually get all these files instead of doing this manually? So I built really nasty pearl script and yes, you can debate if you can write nice pearl. That's fine but what the the pearl script does it will Maintain and create VM.com it will maintain and create the HPD.com It creates an install.com for every single VM. So every VM is a fresh install. It's not a clone of an image I've also noticed some other quirks. So it also maintains do as comms now And it's responsible for the user creation as well as the the VM creation So as soon as I run that all these things happen and have some some flows how that how that works So the VM of comms if you're familiar with it, I assign a switch a bridge I always assign an owner to a VM so you can connect to the console yourself. You can stop it. You can start it You get a disk and I also assign a static Mac address that Pearl script creates because with that I then assign the v4 address. Let's say statically The dhpd.com Similar so I have that same static Mac address I assign an IP address and I also say that It's now auto install. So as soon as I start I boot up that that VM I must say that when it's a new VM. I also have a boot line here So it boots from bsd.rd So then the auto install knows that it has to kick in Then I have an auto install script for every single VM that I built again based on the Mac address So if you provide me with your your host name, I put that in The v6 addresses are assigned your username that you provided your SSH key I cut and paste in there and then these sets here That will probably go away and this will be plus set plus sites Star moving forward And because of that I have to do continue anyway. Yes verification. Yes So that's then what the VM is being being built on Then I'm using site tgc to actually Do some pre-configuration that makes it easier for you makes it easier for me So it sets the install URL to cdn.openbsd.org and it also changes the The clock is it a format or a method or how do you call that clock the the hardware? Okay, the source of the time counter of the current, okay and The reason why we're doing this is because the default is i8 3 Okay, cool. So I'm setting this because there are some synchronization. Let's say challenges Within the VM in regards to to the clock And then I set ntpd and also I turn off the sound demon because well, you don't really need it So that's in the in the site File then I'm using htpd on one of my hosts to actually serve all the all the sets and every single host has an htpd running for The install files. So every single host has its own little little world in regards to that The reason why I wanted to mention sensors is that a lot of the examples that you see you actually Get like if disk fails then run this command in the sensor d.com. I figured out Luckily only one disk failed that that doesn't work. You actually need to Call a script out of sensor d in order to actually get any action out of it So I either an email that's being sent or whatever the action might be so I luckily call that Quite quickly so I was able to replace the disk without any problems But then I started digging into the whole sensor d in the config and how that works So I so this is now the way that I'm using it. So the command is on top sensor d.com It it calls a script with a bunch of parameters that are also there And then it sends me an email saying hey disk is okay at boot and when the disk fails It sends me this is not okay. So I know to replace it So how do I actually deploy it's not fully automated There are still some manual steps that are there and Currently that that works for me. So as I mentioned every host has its own little world that it exists in so I have a Convigoration for a specific host. So I define the prefix of a Mac address Mac address that they use IP address IPv6 address Where all the the file should go And what the default setup is for a VM? So how much memory disk if it's one disk two discs what the image format is what bridge They they need to use or or uplink need to use and so on so based on this I just have a single script that I copy to all the hosts and then based on the Convigration it knows what to do and what the what the defaults are. I Thought it was a good idea at the time so the flow in itself is I have a Contact form on our website So you put all your information in there that sends me an email in in rightly Formatted I put that in a text file So I look hey which VM ID is still available if it's full I take a different host I paste that in and then I run the deploy script. I restart the chip PD I reload VMD to actually take the new configuration and then I start the VM and I hit a Because pixie's not working And I think Philip does a whole talk about how he packer eyes this But as I said, I like to use a lot of things in base and and for me just hitting a and waiting for it to finish It's fine. I can probably wrap it around expect or or do something else But it's for me this works and I don't have to deploy like tens or twenty VMs a day It's it's one or two max roughly In average. So this what the what the contact form looks like you can select if you want to have 512 Mega memory 1 gig How much this space you need? What the what the image format is I default now to Kiko key cow to So that I save a little bit on the on disk space, especially if I move it And then based on this I put everything in the in the in the text file Which roughly looks like this. So it's just Quoted text that I just push into pearl or that pearl parses so that it knows what to do When I then run deploy deploy the PL it creates the the install file it creates a user Creates the at least that's what I'm seeing. It also does a do as in the dhp But these are the things that I'm most interested about if the The user it could be that the user existed already So then I have to fix that or something filled with the with the image creation And if you select that you want more disk space it will create two disks here or two images here and Once the whole install is done. I run it again and that script also removes the install.com file Then I do VM control reloads. I restart dhpd and then I start the VM and as I mentioned I hit it So what did we find during this whole process? One of the things is that very early on The owner of a VM always had to be in group wheel now personally. I don't find that a problem There's there's this unauthorized section in the upgrade manual that says if you don't have access to the console you can do this We don't recommend it and then yeah, people can just do that. That's not a problem And then people went. Ah, no, you cannot do this. You have to have console and yada yada. So I went, okay Right, what can we do? Is there a way that I can define another user group? Or another group where I can assign users so that they these users can look at Their VM instead of adding everybody to wheel which I thought was not a good idea So right thought that was a good idea and he really quickly managed to to put that in so from then on I used VMD underscore VMD users to put all the the users in so they can actually Control their own VM instead of asking me The other thing that is quite interesting if you're starting doing anything with with virtual machines is you need a tap interface By default and there's four So if you run more than four you need to create your tap interface and this this was initially a head scratcher Why is it not working because it wasn't really clear that it was failing on on an interface So on every new host that I deploy I run this this for loop that just creates up to 50 tap interfaces So I always have enough because an average Host runs Around 40 VMs because that's another thing that we that I discovered quite quickly is that if you especially on older hardware If you add more VMs weird things happen this interface bridge Memory So on the Dell R6 10s. We are using only up to 40 VMs and that works reasonably well Another thing is people were supplying me with a SSH key, but I didn't want to send them a password over email saying hey, this is your root password I One way that I wanted to do this and from an automation perspective is I wanted to do this during the auto install So what I've been doing is actually I generate a Random password with Jot. Thank you Roman for that And then I added to the authorized keys at the end as a comment So that as soon as you log in with your SSH key You just look into that file and you have your root password and the password of your user I thought it was clever. I was very happy to see that Reich also thought it was clever So he borrowed that idea But there's probably and this also started Before do as was there because now you could also potentially say well this user gets Set up into do as with a no pass and then you can do whatever you want So that would be another way that I would probably maybe do that today Another thing was in beginning Was stopping VMs was a little bit of an issue If you stop everything in one go, it's not a good idea So I was trying to actually emulate like like normal shutdown. So what I've done is I parsed VM control through a for loop with org looking at all the machines that were started And then would stop them and wait for 30 seconds to stop the other one Now you can just do that with VM control stop a W so it would Stop all and it will wait for that VM to be stopped in most cases that works really really well Another interesting challenge is actually Starting VMs. So I'm doing something similar, but instead of using stop. I'm using start When it was a clean shutdown of the host I use 30 seconds if it crashed I use 90 or more Because what is really bad is if you have two Worst more VMs that are going through an FS check That will slow things down dramatically and it's not pleasant for the VMs that have started So I do Hopefully I can see if it was a clean shutdown or not But then when it wasn't not I use 90 seconds Another interesting one Especially in the beginning I saw that a lot of VMs were were disappearing so they would have connectivity and then all of a sudden connectivity would would go away or You could not reach it anymore. So I Claudio pointed me to RQ sys control That by default sits on 50 Now in normal environments and just for your machine in a layer 2 network It's not usually a big of a problem But since I'm pushing a slash 24 onto a host with 40 VMs That becomes a little bit of a problem. So we saw a lot of drops and I started playing a little bit with the numbers. I've started with 512 and I still had drops I did 756 still had drops So I increased it to 2024 and it seemed that that is solving a lot of the connectivity issues that we've been noticing So what are users experiencing because this is a little bit on on the on the side that I've been Stumbling into things but on the user side. There's of course also quirks and things that we had to get used to So in some cases and this all depends on how Or how the VM boots and what stage it boots and what was happening on the host, but sometimes the the VM gets a little bit skewed from The clock hurts of the host and when that is a little bit off you get a lot of clock drift within the VM so one way and The way that that Paul is is fixing this is he looks at the the clock drift Of the VM and if it's too far It just reboots the machine to see if it's a little bit closer now with 40 VMs or 50 I thought that was not such a good idea So I fixed it like very rudimentary where I suggest people just to put our date in in cron In general this works. Well, some VMs Have to screw this a little bit lower And I have one user running alpine and he's just yeah, he's on his own you have to he has to sync his clock every every second Another thing is that I got a lot of questions about was there's a high interrupt on The CPU within the VM and I have a page on the website with all the known. Let's say challenges issues and this is So I'm using some of the information that might give me it's like an accounting Error, but I did get some question. Oh, hey, what's going on? I don't have anything started It's something wrong with the host. So I started compiling that list of things And there's also links in my presentation that point to explanations of Mike or Reich about some of the things that we've seen So when I Managed to figure out this RQ thing a lot of the connectivity problems went away and it was great And then all of a sudden on a Friday No idea why I started getting connectivity issues again, and it wasn't just one host It was all of them including my own without not too many VMs running and I've been scratching my head since But one great way of fixing this is just either Ping from cron so the tricky is the tricky part is that as soon as it's it's gone You have to start some connectivity from within the VM to the outside world So the way that you can do that is either do it in cron every five minutes, but at the moment it's it's Pretty bad that that doesn't even help anymore So what I've started doing was just starting T-mux with ping minus I to my gateway and for now That seems to solve the problem a little bit But that's a yeah, that's a hatch scratcher Another thing that we've seen is a VM can become unresponsive So nothing to do with connectivity But it would either consume a lot of CP resources And a VM control stop would not work. It would just be hanging in in stopping mode So every once in a while if I see it I fix it for for for the user But sometimes I get an email from someone say I cannot reach my VM. Can you do something and okay? so I need to figure out a way that That user they can run VM control stop, but they should also be able to just kill that And I thought about doing something with kill and then you go, okay Yeah, root and then do as so the way that I worked around it was with P kill Because what you can do you can actually define the let's say the string or the process Specifically that you can kill So that's why I said the pearl script now also Manages do as comms So this part is in in the do us this configuration and this is then what you can use and run as a user so you can just Kill your own VM if need be Because I have all that information And I think some people have already used it So my wish list what I would like to see in the in the future is working PXE That will save me from hitting a um Last clock drift no clock drift. There's there's things that are that are possible there Another thing that I would like to do is move away from from bridge So either looking at switch or layer three And we had some interesting discussions already in the last couple of days around that One of the things that I would like to do is automate more So the whole flow of getting that email putting it into a file starting the VM I would like to to do that more automated so I can Get that VM deployed a lot quicker and I really want to hit that 300 just for my own fun, I would say so I do want to thank a couple of people because I could not do this without especially Mike and Reich and Carlos and Stefan and Claudio and Jasper and Ori um So these are the main committers and contributors to VMM VMD And we figured out yesterday that there's around 47 people that actually made commits to VMM VMD So that's that's quite impressive. And I also want to thank roman for the artwork for pushing for helping for Getting the word out Wearing the t-shirt to get it with Reich and some and and and Fred and of course You the users that are actually getting a VM on on the platform and see how far we can drive this thing So that is my talk Thank you very much And this is without background image. So it's a little bit easier to read any questions This one's dead. Oh, no, it's okay If there's any questions microphones right up here and I also have more suggestions or feedback or Say, hey, this is stupid Don't do this Not too many though. Hello. Hi. Thank you for the talk What is do Have you considered giving a prefix to a vm or I have seen in the config that you are basically giving a vm One ipv6 address No prefix a prefix and is it slash 64 or was it 56? Yeah, uh, but that's to the host not to the not to the vm. No, that's to the vm. Oh, okay, okay That's generous and I I do of course Generate already a slash 28 But the the host has a 56 Okay, sorry vm vm Yes host has 40 56 prefixes I Do have any private network options or do you think about adding the possibility of building a vpc? For a customer to have two vm's in the private network. Yes Um or more obviously. Yeah, I I thought about that I had Of all the people that are running a vm. I had that question once Um, I also had that question if there could be two vm's active standby active active So I've been thinking about either adding a second interface or adding a sacred network segment so I can run carp or Thinking about it, but the nice thing is not a lot of people ask for it. So I don't have to think about it too hard Yeah Yes So any plans to oops any plans to add any other Locations like new data centers or anything? I don't have the domain. So No, it's the the the nice thing is Uh, I think someone said hey, this is a great idea. I want to do this in London And then he worked on the numbers and he went oh, I cannot really do this Um benefit that I have is since I I do hosting and co-location. I have rec space. I have IP space Uh, I have traffic So a lot of the costs are already carried by my own company Otherwise, this would not really be a available business model With only five euro vm's You then have to also do like the The 20 euro a month vm's then then it could potentially work, but I think what I but I personally like about the product it's it's Community from the community for the community. So I don't make any money of this I don't want to make any money of this. It's it's self-sustaining To an extent I guess And because of all the money that that came out of out of a host I bought the new server and then that machine bought for or paid for the the other machines So it's it's it's self-sustaining And I think adding a new location regardless where that would be Probably would not work economically Okay. Thank you again for great service You're welcome So this this is a bit like Testbed for the technology, right? That's how it started There's there's a disclaimer there that that it's it's an active development things break But people don't read So there's also dns running in there and there's email environments running in there So there's I think there's even some people that are running production type services on this and I go Okay Great so now but yeah one two five scale for let's say if someone is more interested in okay, I AWS doesn't support officially open bsd and here is someone who gives a really nice environment for this Then from one to five How would you rate the production readiness of this? Enterprise I would say try it And then if you like it pay it Um I run all my production environment in in vmm and vmd as well Um, I I suffer from some of the quirks that that that the whole platform is suffering from um, but for me it works right because The my own company is also a hobby project. I don't have to make a lot of money out of it um, but if if especially if you do something with payments I would not do that if it's a website or forum or our internal communication platform Probably yeah, there's ways around to to work around the connectivity issues that that's not really a problem Do you collect some data about the uh, let's say average uptimes of these vms because you said they Can become unresponsive or stuff like that. So I don't know if vmctl shows data like this, but Maybe you could do some nice graphs or whatever um, so if I um Get any telemetry out of the vms uptime and things like that um No, I don't I do get some telemetry out of the host um In regards to cpu load and and traffic Um, but nothing on on on individual vms uptime that that's that's an interesting one. Yeah Yeah So martin otain, let's get cracking on agent x Thank you. Sure five minutes I have some questions for you about this thing Okay No, no not really. I'm just want to make a uh, uh statement here because like, uh, whenever this vmd VMM thing started And when mike shocked me the first lines of d message and back in a bar in berlin at some point I think I was thinking it would be so nice and excuse me for the word to have some cloud kind of thing That runs on an open vst hypervisor because all this cloud stuff is unavoidable these days and We ported open bst to most of these platforms and then you still run on a shitty linux or well, maybe on a free bst, but It if I cannot trust the hypervisor when Yeah, one is it and then but not just doing this for testing really run a cloud like platform opinionated vms or something like this and then Later you just showed up and did it And it helps us so much because of these things. I mean these features We get many feature requests for things that are weird or pointless, but from you it really helps moving forward And so we need more projects like this Really, we need people who whatever do something similar and this happens deploying vgp routers from open bst or whatever just doing something like this and And then actually you're the one who proved that you can run the stuff in production to us develop us. Yeah That's pretty amazing. So my my cutos for this year publicly. Absolutely So economically, it's a great way to do this do this in other cities things right Hi Do you use or would you use? vms open bsd vms on other people's computers as a user Would I use vmm on other people's computer? What do you mean? Like, you know, you host vms and you have users that use them. Would you or do you personally use vms? Which are hosted on computers controlled by Other people. I actually I don't I don't host. I don't know any other Environment where you can do that where you can actually have a vm on open bsd vmm Somewhere else, but would you if I provided you with the vmm? Would you actively use it depends how much it is? If it was free, sure Okay, sure. I'm dutch of course free Okay, and another question about virtualization open bsd Was up until a few years ago completely against any kind virtualization and even then, you know, you need some kind of contra naysist Conturnation If you run it on your own hosts you own, but now we see the the open bsd moving in another direction And you know hosting providing vmms Can you explain a bit a change in the in the politics that would not really be a question I can answer but Mike maybe If you start turning an orphan on again The whole topic of how vmm vmd started is probably A topic that they talked that we should have later because we're almost out of time and it's going to take much longer than two minutes But i'm happy to have that conversation with you or indeed anybody that's in attendance If you'd like to know more about the platform how we built it what our goals were the history around it I'm absolutely happy to do that just find me later later today tomorrow. I'll be around But it probably will take more time than what we have now So happy to talk about it later. And there's some good talks already online that you did. Yep One more It's not not really related to the open bsd vmm part of things, but do you do anything with Do you do anything with with reverse dns, uh either v4 v6? Sorry say again. Do you do anything with reverse dns? Absolutely. Yes. I I need to because there's people running their mill environment On open bsd. Amsterdam. So yes, yeah Not by default That can be something that i'm considering But currently it's just fire off an email or put it in the the note field in the contact forms Hey, just put reverse dns to this domain and i'll i'll fix it. It's uh, it's it's not a big deal But not a lot of people actually request it Yeah, yes exact. That's what's my reaction as well. Why not? But yeah Oh, actually, I have someone that doesn't have a domain So they they said, yeah, I know I want to do something with mail, but I don't have a domain Okay, I can give you a host on on open bsd. Amsterdam. Oh, great. Let's encrypt done. So yeah That's also an option. You can all the uh, you can also have a host name dot open bsd. Amsterdam I would be happy to reverse that into your vm as well. Not a problem Thank you very much. Thank you