 I'm giving this presentation with Robert Lazarus, he's a co-developer from Scotland. He's got about 10 years of systems administration experience in the Linux ISP world in the UK. And he actually designed and deployed the second cloud computing environment of the world. As you may know, Amazon's EC cloud was the first cloud, so he made the second cloud. I'm Jasper Kappel. I'm a technical consultant with an open source consulting firm in the Netherlands, Stone IT. We integrate a lot of open source products into one solution for our customer. I'm Reddit certified and I've been an active contributor to the Cobbler project for only about half a year and I've been using it for a little over a year. So compared to a lot of people who stand here, I'm relatively new to contributing to the open source world and I'm very honored to be standing here before you. Okay, why was Cobbler founded? Cobbler was founded by some people at Red Hat about three years ago to fill the gaps in the solutions for systems deployment. There are a lot of good products for monitoring solutions or configuration management, but there were not so many good solutions available for deploying both physical and virtual servers. So you had this one solution for your physical systems. You usually deploy an image or something with a ghost maybe and then you had a separate solution for your virtual servers. What are Cobbler's goals? We really want to make a tool that's very, very easy to use, but while we're doing that we don't want to give up any flexibility doing so and the tools got to be very powerful. It's also got to have as many features as possible and the real goal here is to create a common deployment tool for all system administrators or basically everyone whose job is or part of whose job is to install systems on a regular basis. So whether that's as a consultant or you're a systems engineer in some big firm, this tool was meant to help you do your job. Okay, why is Cobbler useful? Well, we're all lazy, at least I am, don't know about you, but I want to get my job done with the least amount of effort possible. Well, and re-installations, they happen on a relatively low frequency, so what you usually do is put a CD-ROM in the system, install the system, and well, it's NINEC, so it's still running at five years, six years, seven years going, but when you're at the company someone install the system way back when you then leave them work there. So you have a problem when that system dies or whatever because you then have to re-install the system and you've only got to hope there are specifications for the system available. Different persons each have their different methods of installing their systems. In my company we have about 10 technical people and it's safe to say that each one of us got its own standards to deploy new systems. So Cobbler is a great tool to standardize your installations and while you're standardizing your installations, you're actually also documenting your installations for disaster recovery scenarios. Okay, so to be able to use Cobbler, we have to define how all this information is storing Cobbler. In Cobbler we have a few types of objects. One of the most obvious is the distribution. It's the software distribution, so it can be Fedora or Red Hat Enterprise Linux or CentOS. Directly under the distribution in the profile is, in the hierarchy, it's a profile. The profile is basically, well, you can say we have web servers here, application servers there, our database servers and beneath one or more layers of profiles because you can stack your profiles to more specific needs is the system and the system is really that physical or virtual box you're trying to install. To help you manage those systems, we've got two additional objects. One is the repository. It gives you the power to mirror remote YUM repositories or Red Hat Network repositories and the image. Image is quite confusing, actually, but I'll get back to that later. Okay, what's the distribution in Cobbler? Distribution is nothing more than a kernel and an installer in its RAM disk. So most modern distributions, they've got some kind of netboot CD and what's on that CD is just a kernel and an installer RAM disk. And that installer RAM disk knows enough to start the installation. In the distribution, we also have a URL to where the software repository is located. So in the past, that would be an NFS mount of your installation medium. But in Cobbler, we usually run Cobbler import and point it to our DVD medium or point it to a remote FTP to download the entire distribution into Cobbler. Cobbler scans it, hey, I've got this installer kernel, I've got this RAM disk, I'll just create a new distribution from it. Okay, profiles are our main container object in Cobbler because it gives us the ability to build a hierarchy of systems in it. Cobbler, the profile container is a child to the distribution, so it gets a few values from it, like where's my inner RAM disk and such. And it can be apparent to another profile or there can be systems directly underneath it. The profile also defines the path to the kickstart and that's really, really useful because the kickstart is, at least within the Red Hat world, the only way to automate your installations. The profiles also provide for property inheritance. So if I define, for example, a network gateway for web servers to use, and later on I decide to override it for, say, a few systems, I can do so. A particular example is useful for low-manager configurations, for example, where you have one standard gateway, which is your normal router, and then you've got these low-manager machines that act as a router for the web servers. So when you've got property inheritance, you can do really powerful things. On the systems, well, it's obviously a child to the profile, and this object in Cobbler represents one actual system to be deployed. So be it physical or virtual, one system in Cobbler is one system in your environment. So if you want another set of systems, you can clone this system and edit the properties of the system and create a new system. But the system contains network information, so you really can't have two of the same systems. Although you can, as I previously explained, override values inherited from the profiles. So repositories are not in Cobbler's main workflow. They're not created with Cobbler import. Cobbler import is the easy command to get you started on Cobbler. But it defines a package repository. So if I run a Cobbler repo ad and then give a URL of the URL to Fedora 10, for example, Fedora 10 updates, what I can have Cobbler do is run a repo sync, and it fetches all packages for Fedora 10. So my thousands of systems in Cobbler don't have to go to the internet separately to get those packages. You can assign repositories to systems. So you can say, my web service are already at Enterprise Linux 5, and I need some extra software for it. Luckily, there's extra packages for Enterprise Linux. So the service who need this extra software can have those repositories assigned to them. You can either point Cobbler to a remote repository, or you can have Cobbler mirror locally for you with a repo sync. This is a bit confusing, actually, because a lot of people ask me if, oh, Cobbler can do physical machine cloning. Oh, actually, no, it can't, because images in Cobbler are not system images or ghost images. They're more like floppy images you can add to Cobbler to have your system net boot those images. This is actually very useful. A lot of folks use these images to update their HP-proliant firmware, for example, without leaving their workstation. So you just assign this firmware update to a system. You say to the system, OK, you do a new network boot. We can have Cobbler, if we want to cycle the power of that system to make it happen, and then it boots into the firmware image. When that's done, the camera for the system reboots, and the normal operating system starts again. OK, Cobbler is not alone. Cobbler is more like the server utility for this solution. We also have a client utility, which is cone. It stands for kickstart over a network, and it resides on physical or virtual servers we install, or on virtualization servers we use. So I can, for example, install cone on my Xen servers to install new virtual machines. It's also possible to trigger a reinstallation of a system from within that system. So I just say cone and reinstall self, replace self, and then it would download its image in a RAM disk, update its grab configuration, and then reboot into the installation. So after that process is done, I've got a fresh system again. On Xen server, I would say, cone, I'd like to have a new virtual machine. Please pick that system for me, or that profile for me. The profiles have usually got the virtual disk size defined on it, how many virtual CPUs should be assigned, how much memory should be assigned to the server, and cone will just start off the installation for you underneath that's done through word install. And one of the latest features in Colors is that we've built a tiny configuration management in it. It's not as powerful as tools like Puppet, but there are a lot of use cases where Puppet for an environment is really overkill. So what we can do here is use Puppet's cobbler's power for templating these configurations, because we've got a templating engine in cobbler. Every kickstart we define is either just a kickstart or we embed a cheetah code in it. And cheetah is a very powerful templating engine. It's written in Python. So essentially, I can just write Python in my kickstarts or in my snippets. Yeah, one back. No, no, one back. Yeah. I skipped a few here. The top one is really important. Right now, we have support for CentOS, Debian, Fedora, Red Hat Enterprise Linux, Suzy Linux, and Ubuntu. Our user base consists of, I think, about 95% of Red Hat-style distribution users. So the CentOS, Fedora, and Red Hat Enterprise Linux deployments are really well tested. Since last release, support for Debian and Ubuntu was added, but it's fairly new. But although it does work, it's not just yet as powerful feature-wise as the Red Hat deployments. And we can do a full medium import for Debian or Ubuntu now. And for Suzy, we can basically kickstart installations now. We've also got DNS and DHP management, which I'll tell you about a bit later. Power management, and we're actually using the Red Hat Cluster Suite fencing tools for that. So we can add a power interface to a system and then, say, from color, say, to the system, reboot. Or if the system doesn't respond, actually what happens is we go to the APC power switch or we go to the DRAC interface or to the ILO interface and we say, reboot that system. And then optionally reinstall it, because obviously the system wasn't behaving like it should be when I have to do this. One of the most powerful features for Cobbler at my company was that we can use a Python API and XML RPC API. This essentially means that you can do whatever you want with Cobbler. You can script it according to your needs. So it's not actually a standalone, you don't have to use that as a standalone tool. You can use it as a framework as well. A few more features in Cobbler, authentication, authorization, ACLs. Authentication is pretty obvious, maybe. I can connect Cobbler to my LDAP tree or Kerberos domain. Authentication is useful to define which users have access to which objects. So if you've got Cobbler split up, like I've got profiles for my main office, my branch office, and some branch office on the other side of the world, and I give my sys admins in a branch office access to Cobbler, I don't want them to be able to reinstall any servers at my main location. So that's what I can do with authorization. ACLs go even further because with ACLs I can even define that one certain junior sys admin can only edit specific properties of a system. So if he likes, he can, for example, change the DNS name, but he cannot touch the IP address configuration because that could create conflicts in my network. These are some of the features we're working on for the next release. The top one is a big one. We're aiming for performance enhancement so that Cobbler can scale up to about 50,000 systems. We also plan to do a few API upgrades for spacewalk integration because Cobbler is being adopted as the provisioning backend for spacewalk. Some of you may know or may not know spacewalk is the upstream open source version of Red Hat Satellite Server. We're also adding some features required for Fedora Beaker, which if I remember correctly is some sort of test slash QA suite, and we're planning to implement Windows deployment. That's actually really cool. I know it's a big Linux crowd here, an open source crowd, but the more open source utilities we can use in a mostly hybrid environment because a lot of companies have these applications and they need to run on Windows. Well, if we can manage those systems from Cobbler, in my opinion, that's great. Now, well, we might want to add, actually, we do want to add support for a lot more distributions. My personal wish list are free BSD, net BSD, and so on. We'd like to add some web interface improvements. I think we're actually going to submit that project for a Google Summer of Code. Support VMware ESX because we can now support Xen. We can support KVM, QEMU. These all work. But the other major virtualization project out there, of course, is VMware ESX server. So we'd like to be able to deploy virtuals to ESX as well. The S390 mainframe support actually has been added last week. So this will move to 1.6 probably. Also, because this is a community project, users can just inject any feature they want on the roadmap. And if they submit patches for it or it's a really good idea and we wanted to do it ourselves, we just added to our roadmap. The top line here, maybe we should rethink naming our top object in copper distribution because I don't think we can name free BSD or Windows distribution, really, but that's something we have to work out. Okay, a little bit about our development process. We aim to release a major color release that's a feature release, essentially, about every two to three months. We're using an old kernel development scheme. So 1.3 became 1.4 stable. We're working on 1.5 now, which will become 1.6 in hopefully, say, a month, maybe two months. The roadmap is not really fixed, as I said. So if you've got a really cool feature we need to add, just join the mailing list and send us patches or share your IDs because copper was meant as a tool to be as feature-rich as possible. So I know there are a lot of custom deployment systems around there, like a shop running its own PHP kickstart generator, for example. If we can consolidate all these features into cobbler, we can build a really, really flexible versatile deployment tool. Okay, scripting through the API. This is where it really gets fun because this shows really how flexible cobbler is. I can really, essentially, I can script anything cobbler. And my company, we're using this to integrate this with our main management tool, and we're using the XML RPC to add systems to cobbler, to delete systems from cobbler, to generate new profiles or whatever. In conjunction with this, we're using FUNC to trigger new installations or to trigger creating new networks on our virtual citizen servers and so on. Because cobbler has got a database of its own, it's storing all the systems and profiles and distributions are stored in YAML files, so you can use cobbler as your configuration management database if you like. On the other hand, you don't have to because I know a lot of people say, I've got a cobbler working CMDB already. That's fine. If you write the glue, you can just have your current CMDB push the information to cobbler and then start deploying. So you're basically keeping your management tools and are just adopting cobbler as your provisioning backend. Okay, this is a small API example. I hope the code's readable in the back, but this is basically to show how easy it is to use the API to do some scripting. So what we're doing here, first we're importing the XML RPC library and my own library that can do really neat stuff that I've written. I don't know what, but we defined the URI to the cobbler server. We opened a connection using the XML RPC library. What we want to do is get the system, my site, my site.site.local, and then for that system, we'd like to generate a kickstart. I've taken this from a small script a colleague of mine wrote to generate a Nigel's configuration from a system stored in cobbler. So in this example, my own library.do underscore magic would do is look at that generated kickstart, find a partitioning bits in the rendered kickstart, and then create checks for disk space, for example. So in my kickstart, I can see I'm creating a partition for root, I'm creating a partition for slash far, I'm creating a partition for log files. With this function, I can generate Nigel's checks for those systems, for example. Okay, this is where Robert Lezher's is going to take over for the cobbler use cases. Good morning, guys. My name is Jasper Menchers, Rob Lazarus. Thanks very much for getting up this early in the morning to come see our talk. I'm sure you all had a fun time last night and didn't really need to be up here. So I'm here to describe more of the cobbler use cases. Now I'm a sysadmin, I'm not a developer like most of you. So I'm here to tell you why cobbler is useful to you today, why you want to go home and install this and use it in your systems. Now at the moment today, you're either going to be installing your systems by the old fashion method or CD, putting it in your system, clicking all those buttons going through. Or you may have moved on a little bit more and you've got your HTTP server, you've got a Fedora mirror, and you've got your own kickstart file on there. And you've got your own TFTP server. You might even be doing something like managing your repos with M repo of some other tool. And it feels a lot of metadata. You might have a database behind that and you're thinking, why should I bother using this tool? Well, hopefully this will show you why. So first use case was the first place I used cobbler, which was to deploy the systems for the cloud computing farm that I wrote when I was working at the last ISP. Now in this sort of case, you've got thousands of systems that you may need to be reinstalling daily or even early. And they all need to be pretty much the same. You might have a web server class, a database server class, and they all have pretty similar properties. You need to be able to do this reliably. Also, if you're in the cloud computing environment, you don't want to be doing this yourself. Your customers really want to be the ones doing this. So you want to be able to hand over this entire process to them and know that it will just work every single time. Also, in a cloud computing environment, if you've got thousands and thousands of piece of hardware like Amazon, like Google, or like big ISPs, you're going to be seeing hardware failure on a regular basis. Now the statistics we worked at the last ISPs, we figured for every 100 machines, we should be seeing a failure every single month. And your most ISPs are running on thousands of systems. So that's a lot of failures. You've got to be able to get those systems back up and running very quickly for your customers. It's one of the advantages a couple of give you. Also, if you've got a big grid system, so say you're a university, like Edinburgh, or one of the European universities, and you've got this big massive grid, and it's compelling your code, doing your nuclear statistics, or whatever else you're doing, you're going to want to tell that build system that, hey, I just lost that piece of hardware there, so you might have nauseous doing that. But when it comes back up, you want to be able to tell that system instantly, hey, I've got it ready, it's installed, let's get using it. Again, going back to my ISP routes, this is one of the most horrible things I've ever been through. I woke up, I'm on call, it's 2 AM, and the entire customer web farms are there. So you're talking about, say, 20 systems. Now, back in the days before Cobbler, this meant at 2 AM, I've got to sit there, take down each system, because most of the web farms work, you don't want to take down the whole thing, but you've got to take down each system individually, reinstall it, sit there whilst the install happens, and I'm not thinking straight there, this isn't going to work well, and the errors are going to be made. So with Cobbler now, what I can do, is I can write a script with the API, it shuts down the systems. Now, if the systems weren't rooted, we could be using Cone to do a reinstall self, or replace self, to then just, now what Cone does with that is it puts an entry in the grub menu, reboots the system, and then on that grub menu entry, it has a small bootloader environment, which then reinstalls the system, but in this use case, if our systems are rooted, we can't trust them. So Cobbler gives us the power management through the API to shut down the system, and because Cobbler gives you the facility to turn on and off your net booting for your systems, you can have all of your systems all set up to automatically pixie boot, and they won't reinstall themselves unless you told Cobbler to do this. So, with your Cobbler script, you've rebooted your system, it's came back into the install, it started reinstalling automatically, according to your profile, with all the software, the additional repositories, and everything else you need, then when it comes back up, because we've got funk integration, which lets you run remote command reliably on the systems, if you haven't seen that, I suggest you have a look at it, and then puppet integration, to make sure that you're, to ensure that your system's configuration is exactly as it should be for your users or your web farm, that all happens automatically. So what I can do then, as a systems administrator now, rather than back in the old days, I wake up at 2 a.m., I notice why web farm's rooted, instead of having to stay up for eight hours and sit and reinstall systems, I run one script, and I go back to bed knowing that in half an hour, an hour or however long it's gonna take, those systems are gonna be reinstalled exactly as they were originally, their configuration's gonna be perfect, and I've lost no sleep. Now our third use case scenario, this is the one I'm working on now. I work at TomTom, and one of the things we do is location tracking based on mobile phone data to give us more accurate traffic statistics. What we need to do there, is in every mobile operator, we've got to install a cluster of machines. Now this sounds perfectly fine, surely you just fly, rob your sys admin, out to your mobile operator, and he'd install machines. The sad thing is, that you end up in scenarios like I've got here, where the mobile operator in our case is far too paranoid to let rob the bearded, long-haired sys admin into their facilities, so that's not gonna happen. So I've then got complete and utter monkeys who have never seen Linux before in their life gonna install systems for me. So what I give them now, instead of giving them a CD with a kickstart, which it might be an hour on, they have to figure something out, I give them a portable hard drive to plug into one of the systems in the cluster, it's got the cobbler server up and running, so that system boots off that hard disk, it's got cobbler server all set up, then all they need to do is turn on each single one of those systems, they get installed, configured, and they're ready to go for us, and that monkey never has to touch anything or do anything, and that way, we don't have to worry about the systems being deployed or configured in the wrong ways, everything just works for us. What this also lets us do, is because we've got the local repository mirrors, we don't need to worry about the bandwidth coming in out of these sites either. But because cobbler has a replicate functionality, so what cobbler lets you do is, you have your central cobbler server, so in my case, in our office, I can then replicate multiple cobbler servers from that, so I've got that central configuration management database for all of our mobile operators all over the world, and it holds all of the records on all those systems, and I just replicate that entire system, it's a trivial operation, takes me about five minutes, and suddenly I can deploy the entire world's worth of mobile operators, very, very trivly. It makes systems administration, it makes the installation of these systems absolutely trivial for me. Next slide. Now, this is the thing I really love about cobbler, okay, it makes my life easy at 2 a.m., and I lose less sleep, but the great thing I love about cobbler is it's been really, really easy to get involved, and that's why I love open source software, and this is what I'd now encourage you to do. A year ago, around about the same time Jasper joined a project I did as well. Now, as I said, I'm not a developer like you guys, so I've maybe written a patch or two, but what I was able to do is contribute what I actually needed to the project, and we were able very quickly to get that done with Michael Deham with great photos like Jasper, and suddenly I've got a product that's usable to me, and that's for me, and from a business perspective, the big win for open source software, and what you guys provide to the world, it's fantastic, so come and join us if you're gonna use this. We'll listen to all of your requests, feature requests, or if you've got patches for us, and we'll get them in there quickly. It works really well. We're a really open community. I know some open source developers are not. I've had to deal with some recently who they won't accept patches. They're rude. Please, please, if you are an open source developer, be open to your users, because they're the ones who are using your product. They're the ones that you're actually sitting there doing this for most of the time, and Cobbler has advanced greatly, and now the reason we've got this fast release cycle is because we're such an open community. Thanks very much again for getting up this early morning, listening to us practically along. I seriously appreciate it. Now, do you have any questions? Yes, yeah. Absolutely, so one of the things I've got in TomTom is we all have Linux workstations because they're all Eclipse and Java developers, yeah? Now, we don't have a central NFS server in the home directories because the latency would be far too high. Everything's on the systems, but with the authorization and authentication ACLs, I give the developers the ability to reinstall their workstations. The last thing I want them to do is wiping out the home directories. So as soon as the system's installed, I put a parameter in Cobbler to say use existing partitions, and then there's a little snippet inside the kickstart template, which I couldn't have done before with a traditional standard kickstart and mirror. I'm able to put a little bit of code in there to say, okay, if I've got this parameter against this system, it's to use the existing profiles and not wipe out the home directory of those developers and kill all their code because I don't want to have to be restoring backups. Yeah, I've got better things to do in the time like rebuild. Anything else? Oh, sorry. Yeah, so if anyone else didn't hear what our friend here was saying is he got, say, 200 new systems in a day. And again, you've probably got some tech in the data center installing those in, and you just want this to all be automated. You don't want him to have to go into Cobbler, add the system because something will go wrong. Yeah, humans are, we all do. That's why we've got Bugzilla. So what Cobbler allows you to do is register new installs automatically. So they can boot up that system, select the profile for it, so it gets all of your common software, if it's your database server, for example, and then that automatically registers that install within Cobbler. So you don't have to have someone sitting there or even an API or a little script adding all those systems in. It can be done just as hardware reaches your data center. Sorry if I was to use a fluffy phrase that'd be just in time delivery. That's one possibility, yeah. What I'd like to work on in the next coming few weeks is bootable CD that reach your system configuration and then adds that configuration to Cobbler. For example, your system MAC addresses. So you can then edit networking properties of those systems in Cobbler and then have it reboot into the installation. So I guess the main thing is, as long as you get your systems up and running, it can SSH to it or trigger any other remote procedure to get things going again. You're good. Any more questions? I like this, great. Sorry? We can do failover with the Cobbler replicate command. We can have information starting Cobbler on both systems simultaneously. Replication isn't instant. So I would have to do this in my Chrome tab for now, but the Cobbler service basically just a Python process. So if I want to order two headbeats, that's absolutely no problem. It's same with DHCP. If you've got two DHCP servers running on your estate, as long as you've not got any range addresses on those, they'll quite happily cooperate. In theory, the protocol says that range addresses should cooperate as well into DHCP servers. In reality, it just doesn't work. Even with the latest release of ISCD-HCPD, you will have errors with that. But yeah, if you stick to single systems in there, running two Cobbler servers on your LAN is fine. Yep. Okay. So you're saying if I've got, say, so actually we've got something, we're very similar in our dev environment to this at the moment in TomTom. So what we have is a bunch of servers monitored by Nazios, and then we have a Nazios event handler script, which can then do reinstallations. Because we've got Cobbler's API, those reinstallations are simply seem predictable, so I can trust Nazios to take care of those. So say, for instance, our build farm suddenly needs new systems. We're running far too hot and the developers are gonna be sitting waiting in desks, waiting on builds happening. I take over other infrastructure systems or whatever spare pool of hardware I have, and Nazios can then just reinstall those systems. I don't even need to think about it. Again, the great thing about Cobbler is it allows me to be as lazy as possible, which is what a good system admin tool should let you do. So yeah, something I don't wanna talk about too much because I work for TomTom, but it's something I'm helping the UK ISPs do, is do this for the web farm, Jeff. So say you've got your 10 systems behind your load balancer and suddenly one of those customers ends up on national television, newspaper or whatever, and you need 10 more systems. Now ISPs have generally got spare pool of hardware because they like to sell hardware very fast. What Cobbler can let you do with that Nazios event handler is Nazios sees a load on the web farm going up and presumably you'll have multiple monitors there for this, then with that event handler, it can just install new web servers, bring them online instantly. Now, it's not a trivial script, but you can see with Cobbler API how it can be done, yeah? It's not difficult. Any other questions? Please do. This is actually a function that a normal kickstart provides. You could have in the pre-section of your kickstart a little script that detects your hard drives and then writes out a partitioning scheme accordingly. So it can see, hey, I've got one normal rate one SAS disk, for example, for my operating system and then I've got these two SSD disks and I'm going to use those for my var partition, for example. So yes, that's possible. Any other questions? Okay, thank you very much.