 Hi, my name is Guido, so I'm here to present Guenetti, which is a tool we've been developing at Google in the last few years, and I'm here to present it in a slightly different way. Basically, since I was tired of presenting the usual of what Guenetti does and how to use it, I'm going to present a bit of how I use it personally on my laptop in a totally unofficial way, just to make my project, my technical manager more scared about this. So what's Guenetti? Basically for people who haven't heard about it, it's a cluster virtual machine manager, which actually means that the cluster is not of virtual machines, but is of physical machines, machines we call nodes, and the cluster is used to schedule virtual machines. So you can use various hypervisors. We now support Guenetti 2.0, KVM and Xen, and you can use, you will be able to use LXC or normal CH routes in 2.1 when it will be released, or if you are very brave, you can use it now. And we use various types of backend disks, so our main development is on DRBD and LVM, but we also support files, although it might have some improvements. It might use some improvement. We support different operating systems through an API, so basically what you can do is you write a bunch of scripts, and they can install your operating system under Guenetti, and this can be used to install almost whatever. We have an example operating system, which is open source, and it's based on the bootstrap, but anything else can be created. You can use static images or install your own, not Debian based or whatever. And yeah, basically you use Guenetti to install these instances and schedule them on nodes, on a cluster of nodes, and then we have features like live migrations without needing any storage or network or shared storage between the machines, and we use DRBD to do that. So in this way, you can use basically cheap hardware or normal server hardware without any special features to create a network of virtual machines. And as I was saying, we are going to talk about how to create and dispose of development machines on a system like a laptop rather than a cluster, but because, I mean, I thought everybody already knew about the cluster stuff, and it was kind of boring, so. Maybe it's me, and I've talked about it a bit too much. If you want to talk a bit more about it later, just ask questions and we'll figure it out. So development Debian release status. We have Guenetti 1.2, which was our former release. We don't use it anymore. It's in Lightning. And of course, I mean, anything which is in Lightning is already old, right? So we know that Debian Stable has this feature. XenHVM doesn't work, but Houston is trying to fix it in one point release or something, and you can only use Xen. Only now only Xen.PVM. And the last latest stable is Guenetti 1.2.8. It has a couple bug fixes more from 1.2.6, but in general, just don't use any of them. Okay, Guenetti 2.0. This is the current stable release. We are currently at 2.0.2, but 2.0.2 is broken, so we are using 2.0.1 also instead. This is the release I would recommend you to use. We just uploaded it to Debian. We didn't have time before, so it was still 1.2 before. And yeah, basically, it went through new very fast. I think I had to bribe a few release managers for that, but yeah, no, actually, FTPmasters. And that one has support also for KVM, and that's the one we are targeting for Backport Sorgh soon, since at least you can use Debian Stable and a newer version of Guenetti. This also makes sense, because mostly in a Guenetti environment, you're going to use a machine just for Guenetti, and then use services on the virtual machines scheduled on Guenetti. So it makes sense to have a newer Guenetti there, because it will mostly be your only or about only service on the machines. And then we have a development branch, which is called Branch 2.1.ingit. That contains all the new things. It has a design doc which outlines what we're going to achieve or has some version of what we're going to achieve, which will be different anyway, but at least you can get some ideas. And if you have any feature which you really want and you are willing to implement and you're willing to sign our evil copyright thing and contribute patches, this is the branch to look at, unless it's just small fixes or minor features which can go into Stable. Let's talk about the structure of Guenetti a bit. So basically we have a bunch of demos, because Libvert had one and we wanted to have more, so we have at least three. The first one is Guenetti Master D. This runs on the Guenetti cluster master. Each cluster has one master node, which is where you give all commands, and you can fail over this role from one node to the other, should one node break and so on, because we want redundancy. And this demo manages the Guenetti job queue, which is basically the way you interact with Guenetti in the 2.0 world. So you submit a job, which is, for example, create an instance, and the master will execute this job. We'll see how this works later. And job can execute concurrently. Guenetti Master D handles locks and things needed for this to happen. There is one cluster config. It's present on the master. The master has the blessed copy, and it's also copied to a bunch of other nodes to make sure that it's not lost, should the master die. Of course, you should back it up, because should all your cluster die for some reason, you never know, right? And it replicates data, as we said, to other nodes. There is some data which is replicated to all nodes, and all of the data is replicated to a subset of the nodes, on big clusters. You have three nodes, everything is replicated everywhere. But in order to scale, we want to replicate only parts of the data to all nodes to have less traffic. Guenetti-noded is a very simple node. It's just HTTPS RPC, and it's used basically to give commands to the nodes. So the master D has the actual decision-making process, and then tells the node something like create an LVM partition, and the node does it. But this doesn't have any state. It can only query things from the system and report them to the master, or execute operations as the master requires it. Guenetti-RIPI is used, it's also an HTTPS demon, but it's used externally from the cluster in order to actually interact with Guenetti. So if you want, you can just interact on the system with scripts, which we'll see in a minute. But if you want to build a web interface or something else, or graphical GTK interface which interacts with a Guenetti cluster, you can do that through RIPI. RIPI doesn't support all operations which are available in a cluster, just some of them, but it's very easy to add new ones. Because basically, you just need to decode an HTTP request and submit a job using the library which is provided. So if that's needed, 10 lines, they say. I never touched this code. And there are more demons to come because we want to be better and better. So there's some design for, for example, a configuration demon into one which will be able for you to query quickly configuration variables from many nodes in order for it to be available even if the master dies or something. But this is in the design doc for one, for two one, and we don't have it yet. So it doesn't make sense to talk too much about it. Then you have scripts. So as we said, if you don't have your own web interface and you just want to use Guenetti normally, you can use this script to interact with the cluster. And these are also simple scripts. They basically parse your command line option, generate an opcode, which is a Guenetti operation, and then submit it to the master demon. Or for some exception, execute it directly. This is usually GNT cluster. At cluster level, you can create a new Guenetti cluster or execute some cluster operation like get information about the config, change some things and things like that. And you can also change which the master is. This is one of the operations which doesn't use the master because, of course, you need to create a new one. Then you have instance commands. Instance commands are used to actually do your real work, which is to create virtual machines, otherwise you don't need Guenetti. Node management is to actually extend your cluster. When you create it, it has only one node. But if you are working on a cluster level, which is not what this talk is about. But in general, I mean, if you use Guenetti on a cluster level, you use more than one node. So you extend nodes or you can repair them, mark them as offline and do these kind of things. Job management. So most commands just wait on the console and just pretend they're doing the job. Even if they're just actually pulling the master and asking it for information about the job status. But you can also submit jobs. So you can say, create this instance and I don't care. Just give me my shell back and do it. Job management allows you then to query the job status and to see if your job actually succeeded, what the output was. If there was an error, which error it was and so on. OS, we said we have an OS interface, which is documented. So you can just dump operating systems on all nodes and they will be visible by Guenetti. Of course, if you're developing your own, it might be that you think it should be visible, but it is not. And GNT OS will tell you why. Basically, it will tell you why it's not conformant to the API in our opinion. Or if it's not present on one node and things like that. Or it will just give you a list of available operating systems. GNT backup. This is used to basically export instances in the node file system. And it's used, for example, if you want to back up an instance, do some work and then recreate it and things like that. It's broken some of the time, so I wouldn't rely on it too much. But it mostly works in stable releases. And GNT debug is just used for us for debugging. I'm not sure if you'll never use it, but you can have a look at it and see if there's anything interesting. Other than that, we have some tools. This is mostly tools we use internally, which we thought they were handy. So we're just shipping them out with the rest of the source. LVMStrap is used if you have a node with many disks to just bootstrap an LVM block device dedicated to instances. GNT expects to zero that its block device is actually dedicated to instances. Rather, it's LVM block device, the volume group is dedicated to GNT. So if you have a volume group for the system, you should keep it outside. Of course, I don't do that and I just ignore GNT errors. But that's another way to use it. CFG shell is basically a way to edit config data, just use VI. CFG Upgraded is used to upgrade from one, two, to zero. Hopefully, we won't need it for two, one, but it's too early to know. And yeah, basically if you have a one, two cluster, probably you want that. If you don't, you better ignore it. And HTools is a nice implementation of some external features. It's basically not part of the GNT project itself. I think mostly because it's written in a different language and things. But it basically contains things like automatic allocation and cluster rebalancing. So if you have many nodes, you may want to make sure that your instances are actually equally spread among the nodes. Or for example, when you create a new instance, we have a feature which is called IAllocator, which allows you to allocate an instance on whatever nodes the cluster thinks are appropriate. But the cluster doesn't have such information. So it actually asks an external script to tell it which nodes are appropriate. And HTools contains an implementation of this. This is currently not in Dabian because of some missing dependencies. But we're working on having it in hopefully soon. In the meantime, you can just download it from Git and compile it by hand or ignore it and it works anyway. GNT is a development testing environment. So this is the main topic of the talk. So this is how I use, I was using GNT on my laptop a few months ago in order to showcase some networking infrastructure which was kind of weird. And I wanted to have an example of it before actually acquiring real machines in such an environment and things like that. And this is how I use GNT. So at the same time, there was a call for papers for Debcon when I thought, well, why not? I can talk about this. So as I've written, don't use it at home. Or basically use it at home, but don't use it at work. In general, don't follow this document for a real GNT installation. This is just random things I did on my laptop. In general, you should use our install guide if you have a real cluster which is a lot more appropriate for a full operation. So how do you do it? First of all, you just add a bridge to your machine. Normally, in a cluster, you would add a bridge which is connected to your Ethernet interface. On your laptop, you don't really want that. You don't want your instances to be on the network or things you may want to nut them. So what I did is just created a dummy interface. This is to trick the Debian if-up-down scripts to actually have a bridge ports, a valid bridge ports to understand this was a bridge. And then just create a bridge interface with the same IP address. So it will just inherit it apparently, so it works. And basically, yeah, I just, I don't have an auto line. So when I do, when I want it to work, I just IF-up BR0. It will IF-up the dummy interface. And then it will start DNSmask, which will do DNSmask rating for the instances, and it will do the HCP in a way that will show later, which is very handy. And yeah, this is a restart just because basically on my system, I don't run DNSmask at all. I have changed the init script not to work unless it's called from here with a small trick. Securely bind your demons. You don't want your connected demons which happen to be running as root on the network. Usually, I mean, in a production system, you may want to do that, but then you will have firewalls and things, and they won't be exposed to the internet. So your laptop is usually connected to conference networks and things, so you don't really want that. So you're just going to bind a node and an API to the local address on the BR0, which is not connected to the internet network, and that kind of works. Of course, you'll need some IP tables rules later because you'll have to enable forwarding for the instances to actually work, unless you want to use a proxy for them, which also works. Another trick is to set in default, gannety-rappy equals bin-through, which actually kills rape. This is a feature that I discovered after we refused a patch to disable rape, so this is a way to disable it. But yeah, IP table rules. This is basically after we enable forwarding, we want to avoid that forwarding actually allows people to set her out to our laptop and talk to our machines. So basically we drop invalid packets, and then we just accept, so there's a default somewhere, which is drop, of course, but I haven't pasted it. And then we just accept traffic from the instance network, which is this is Lush24 in this case, and from one of the allowed interfaces, which are the tabs for KVM and BR0, this is different because it depends if you're using it in bridge mode or not. I was using it sometimes in non-bridge mode, so I added the tabs. And yeah, and then you also want to masquerade your traffic, and this way the instances can access the internet, but the internet cannot access them. But this is all normal stuff. I mean, you know about how to do this. And then, of course, you would need DNS implementation and things, so you don't want that. You just add a few ATCOs and everything works. So you want one for your host. We gave it IP 254, so we added it there. Then we have one for the cluster. We would have one for each node, but we just have one laptop, so no nodes. And then we have one IP, per instance. That's it. Also, Ganeti doesn't like it. If your ETC host name contains only your local handle, like myhost, it probably wants your whole myhost.mobile.example.com, so just give it, and it will work happily. Then we just install Ganeti. This assumes CID right now. Backport.org should work. Otherwise, you can backport it. It doesn't have any particular changes because it's basically a source-all package in Python, so you just need the dependency, and you should be able to install the CID version, even on Lennie, I suppose. I haven't tried, but. And yeah, then you want to customize Ganeti, instance, the bootstrap, versus a configuration file. You need to see default for that. I'm not sure it's the right place, but whatever. Now it's too late to change it. And basically, if you want more than one distro, so you want, I don't know, CID, Lennie, and squeeze, all of the same system, you need more versions of the bootstrap, so you need to do some twiddling by basically copying the bootstrap to some local areas, and then calling a different config file or things like that. This is kind of unfortunate, we realize. So we will change this in Ganeti 2.1, but it's too early for now. And we don't have it even in Git implemented it, so it's just a design. You can't use it really. So for now, you will have one operating system for each version of the thing you want to install at the same time, or you can change the config file every time, but that's kind of handy sometimes. Using Ganeti, so you first of all, init your cluster. No LVM storage in case your laptop doesn't have LVM, or you don't want to use LVM, and you would just want to use a file-based storage directory for your nodes. You want KVM probably, because Xen will just fry your laptop, apparently it doesn't know much about CPU levels, and it doesn't play nicely with mobile machines, and KVM is so handy, you can just run it when you need it, and otherwise it's just your normal Linux, and you don't need to twiddle with your bootloader and things. And then, well, that comment is kind of tricky. It's not actually what it's going to do, but what you need to do. You have two options. Basically, either you use your normal Debian kernel for KVM hosts as well, or my option is usually to compile a non-modular kernel, so just monolithic, no models at all, and only the drivers which are needed for KVM used, so it's a lot more lightweight, and it's good for your virtual machines. But I mean, it's not mandatory. If you actually use the Debian kernel, then you may want to change a few parameters, enable the initRD, and install the modules or the kernel package also inside the instances. Then you can just add that instance. It's a very simple command. You can ping it. You can SSH into it. And if you use a couple of hooks, this can be made all automatic, like the instance getting its own IP from the host's file, and we'll see how. And for example, getting your public SSH key so you can SSH into it without giving any password, because by default, it doesn't, or putting a password inside, and so on, which brings us to these hooks. So basically, normally, you just have your operating systems very, very basic. You can change the source for the instance, the bootstrap in order to personalize them, but this is kind of inconvenient because this is a Debian package, so you either fork your own or you submit patches to us, and so on. So we just developed a hooks model. You can basically plug your own scripts inside many GANET operations and also OS operations. So for example, we have some example scripts, and you can write your own. It's very, very easy to write. They read environment variables and then change things in there, and they can do basically whatever. So if you are not careful with a hook, it runs as root on your node, so you can just delete your whole laptop or do bad things. So be careful, but apart from that, for example, the GANET eaters hook will create an ETC eaters file, which will plug the instance host name with the MAC address GANET has generated for it. This is a hook for GANET. And if you use it, DNS mask can read this eaters file and can provide the HCP addresses to instances, merging basically what hosts contain and what eaters contain. So an instance comes up with the MAC address GANET has decided, asks for the HCP, and verse an example, interfaces hook, which is actually an OS hook, which is used in order to automate the networking setup. So just create an interface which comes up with the HCP. It will ask, DNS mask will look up in its eaters, find that it's a known host, and so serve the known address from ETC host for it. And we don't need any big demos. DNS mask is very small and in this case. So how do I use this setup? If I want to play with new RC kernels and experiment features, for example, when Linux containers were just coming out and they wanted to try it, but it was just 269 RC something. I didn't really want my laptop with that. So I just created a bunch of KVM machines and started testing it there. You can do different networking setups. So on your bridge network, you have basically a network of machines. And there you can experiment with panels bridging. You can actually disconnect your instances from the bridge and put the routes between them. And then you have instances which are kind of in the same network but not really. So you can have lots of nice things to try before buying real hardware. And yeah, you can use them as auto builders or to port things like if you want to quickly make sure that both your AMD64 and I386 version of the package are built and work, you can quickly test build them and upload them from your development machines if you don't want to wait for the auto builders to do their job. Or I mean, just for testing or for example, for desktop application, you can install X and then connect via VNC to the instance and try your desktop application without changing your stable system. And yeah, you can test new network protocols, demos, new things and so on. So yeah, this is just a showcase. It's not basically what we use Gennady for, what we use Gennady for is actually to provide services and to loan virtual machines to people who provide services with them. But I thought it might have been interesting to talk about it. If you have any questions, okay. They suggest that I mentioned the cluster size for normal deployment. So normally we deploy with around 20 machine per clusters. Four, four, yeah, no, okay. Normally we use the deploy with 20, now we're moving to 40 and bigger. So we want to make sure Gennady scales on the big numbers. But we also have small installations. Luckily, we have one or two nodes, well, two nodes is the minimum to have the redundancy. So we also make sure that it works nicely on small clusters, we don't ignore this situation. Except when we do and we figure out that there is a bug and then we fix it though. Gennady two is a much more complicated structure than Gennady 1.2. Gennady 1.2 didn't have the master demo, didn't have a job queue and so on. So Giddo's talk I think very nicely highlights that, yes, it still works on small installation even Gennady 2, which is a heavy weight in that sense compared to 1.2. So it still manages to scale from one to 40 and we have the 2.1 will bring it even further. So when they will scale it more and I'll be a part of this, I will also make sure that it continues to work on my laptop because it's so handy. So don't worry about small installations. Wow, we also managed to have you do some sports in between. Last time I checked, which was before Debconf, I have to address, Gennady 2 wasn't in Debian. It is now. It is now, okay. It's just in seed, as I told before during the talk, we uploaded it during Debconf. And we had a very quick run thanks to FTP masters through the new queue, so we're now in seed. Yeah, they say that it's useful, it's simple to use from source as well, but on the other hand, why would you? I mean, we're Debian developers, right? Okay, there's someone on IRC, which asks how about storage migration? Storage migration, yes, Gennady supports that. You basically can live migrate an instance, as we said, between one node and its secondary, but you can also replace the secondary with another secondary or replace the disks locally on the primary or on the secondary. So if one disk breaks or one whole node breaks, you are still safe and you can failover your instance or migrate it away if it's still up. So if the node went down and the instance was running on it, of course, the instance is now down and you have to restart it on another node. If a node went down and it was a secondary, you can just replace the secondary with another one and the instance will continue to run transparently. So this is pretty nice. If you have a cluster, well, this is storage migration, right? Yes, you replace disks and you migrate your storage from one place to the other. Oh, yeah, very suggesting that it might be, can we migrate from one storage type to the other, like from DRBD to files or to LVM, but you can't do unless you twiddle manually with a config file. Or yeah, or you export your instance, but you have to shut it down first, which you probably would do anyway in such a case, but yeah, you can export your instance and import it back supposing that works. Yes, go ahead. Okay, I think it's on, right? Okay, the saint also on IRC asks that what he was asking exactly was about the whole virtual machine configuration and disks from one physical disk to another? From one physical, so wait, repeat, please. So you have two machines, what's the case? That's... Okay, sorry. From what I understand of the question, is he wants or he or she wants to know if there is a, if it's possible to migrate the whole virtual machine config and disks, you know, the virtual machine disks, from one physical disk to another on the cluster? Yes, so basically what you can do is if you have on a machine more than one disk, you can just disable one of them for new allocation, new LVM allocations because you think it's broken, then you run and replace disks locally on the machine and your LVM would then allocate the new disk on another disk and then it will be created there. So the disk would automatically be replaced by gritty, so you can do that. In your pages, you mentioned that it wants to have a full qualified domain name in the EDC hostname file. Yes. But that's highly uncommon. Most tools require just the hostname in EDC hostname and that's what also the name of the file is. And they do a lookup get entry hosts for that name and gets the full qualified domain name from EDC hosts and services like that. So I think there are reasons for that. I don't know if you want to chime in why we exactly require the full qualified domain name in the EDC hostname. Because other systems might break because of that requirement. Sure, so before GANET 1.2 and GANET 1.1 we actually didn't have this requirement. However, most of the node identification functions, so what node I am, physical machine, what role, what is my role and so on, relies on the hostname. And we have seen that if you have mixed FQDNs and non-FQDNs machine in the cluster, it's much harder to debug problems. So just for consistency, we wanted to pick the same way on every node. So GANET needs one configuration. And we said, well, let's choose FQDN because that's the more clear. But hostname is the wrong file to store the full qualified domain name. It's meant to be only the hostname. As far as I know, the POSIX standard doesn't say this. The POSIX standard doesn't specify which. So there's a long running debate between FQDN versus non-FQDN. And we decided to side with FQDN. Most of it, I mean, what I figured is that this didn't break anything on my laptop, as far as I know. And for nodes in a real installation, as I said, you most probably don't want to run a whole bunch of services on your node. You just want to run GANET and then run services on the virtual machines, because this is like the kind of thing that makes sense. Yes, but that requires, if you look at the source, for example, in Python, even a hostname, if you do S3's hostname dash F to get the full qualified domain name, you will see that this implies a heuristics. If the fully qualified hostname is not stored in the kernel, then it does heuristics. And that usually means querying DNS for nodes which usually run virtual machines on top of which the DNS server run. You don't want any dependency on an external infrastructure. So in our model, so just do S3's hostname and hostname dash F, and you will see that what hostname dash F does is do a lot more work. And for reliability, we don't want to depend on DNS on a GANET node because in our, not really sure, but we can try it later after the talk. And just to make it clear to the question about migrating instances, GANET tries and usually manages to abstract completely the physical machine from the virtual machine. So there's no one-to-one relation. Just right now, this VM run on this node. But GANET provides all the commands needed to move it completely away to take a physical machine out of the cluster, everything dealing with LVM configuration, the RBD configuration, everything. So you don't need to think in terms of, oh, I have the web server running on that physical machine. It's just a temporary association. And you just do a GANET instance migrate or failover and some other commands which are hopefully detailed in the admin guide. And it does everything related to the evacuation, move around and so on. Other questions, comments, remarks. It doesn't seem so, so thanks for participating. We are 15 minutes early, but I mean, you can have cookies.