 Hopefully that's the right button. But this is pretend cloud or how to accidentally in open source project. My name's Mike Ruckman. You know me as Roshi on IRC. I work for Red Hat specifically on Fedora. I'm on the QA team with Adam Ways and Tim Flink. And I work on Fedora full time, which is awesome. I'm an accidental cloud working group member. The cloud, when we did the Fedora next thing, we needed a QA rep with each of the working groups. And I got cloud. And somehow I ended up on the cloud working group and actually helped them make decisions, which I think is hilarious. But it's been working out so well, so far. Some of my dislikes, I dislike bells and whistles. I like things simple and I want them to work. And I don't usually care about the things that make it shiny. I mean, I typically end up using mate or I3 as my window manager. The bells and whistles don't do it for me. And then also, I really dislike magic. Things that I don't understand how it works or it seems like magic, I dislike them because I want to know how they work. And my experience with cloud and specifically with test cloud has been that. Because so much of cloud is summed up in the, I only saw the trailer, some movie. And the guy's like, nobody understands what the cloud is. It's pretty true. I mean, we hear about it all the time. But a true understanding of how to build one and how it works isn't typically typical knowledge. So we're going to go to the intro. The first thing, and this is just for the purpose of this talk, I've got an argument with somebody on IRC about what cloud was. But this is just for the purpose of this talk, is what I needed to do on the QA team was test and validate our images. Because for a long time, the amount of testing that was being done on the cloud images sometimes would just be like Dennis Gilmore booting it once in EC2 and saying, OK. And that was the Fedora-22 cloud image, which isn't nearly enough testing to say that, hey, you should run your production on a recently released cloud instance. No. No, no, no, we have never said that you should run your. Well, we try explicitly not to say that, I guess. But that was the extent of the testing we had. And so for the purposes of this talk, the cloud is literally just some means to boot an instance so that you can SSH into it and make it do stuff. It's none of the glue that allows all of the automagic scaling or anything like that. This is just booting the raw QCOW or raw image somewhere where you can test it. So the cloud is collecting lots of user data. That's what cloud is in real life. And that's why I say it's almost a joke. But when we were, cloud really isn't at a high level conceptually, it isn't that different from what we've been doing for a really long time with VPSs. We started out, and this is how we did mass hosting. We did it, and it was a Model T. And now we're to the point where it's a Tesla. We've gotten really good at provisioning virtual machines and making them useful for us, allowing us to scale up. In the old days, we've got a server and a hardware house and stuff like that. Luckily, well, it's just moved up so it's glowing. Yeah, exactly. It's adding automation to it. So that's how we're going to be looking at cloud today. And I want to go over what goes into cloud image. How does it get its information? What makes it different from a traditional install? And really, the stuff that goes into a cloud image isn't that different from a traditional install. But the way that they're built is you run an install on a VM and then you take a snapshot of that virtual machine and duplicate it for all of your instances. So you can actually build your own custom cloud images just with Fert Manager. And you can run your snapshots and whatnot. And we'll get into some of the intricacies of how a cloud instance is a little bit different than a normal install. But that's basically how they're built. It's just a snapshot of an already installed machine. So you'll have all of your, well, I recently learned, you don't actually have all of the normal packages that you're used to. But whatever is installed in a typical base fedora install is usually going to show up in your cloud images. There's been some transition to move, like breaking the kernel package into kernel core and then the kernel extras was to bring the kernel package size down because cloud doesn't need hardware drivers. Like cloud doesn't need a lot of the stuff that shifts with your kernel for every install. But one of the key things that makes a cloud image useful is cloud init. And cloud init is a software tool for pulling in identity information about that instance. But yeah, it's not really that different from a normal install. You SSH into a cloud instance, and you're going to have most of the tools that you're used to at the command line that you have on a normal install or a bare metal machine. So this brings us to metadata. When you do a normal install on bare metal, you give the machine a host name. It pulls its IP from DHCP. You tell it what users and what the passwords are going to be on it. But a cloud instance has no way of knowing any of that. So your metadata is all of the identity information. So you've got an instance that's basically just a blob. And then it has to ping some kind of service. And this can be like on Amazon. It's at IP 169.254, 169.254. And this will pull all of your instance data to that instance. And then we'll give you your SSH keys, also any user data scripts. And you say, hey, the first time I provision this instance, I want you to create these users, install these packages, configure these services. And that's what really allows cloud to be able to scale up and out is the metadata service. So you can get it. Each cloud provider hosts it a slightly different way. But cloud can do network resources. The way that we use it in test cloud is you can actually do a local disk, a local file system. So we end up creating an ISO with the metadata in it and then attaching it. And then cloud in it is smart enough to go through its sources and pick out the one that's actually available. And I'm not there. Do you know the order of precedence on that? Because it's network? I haven't never realized it. It's in the source code somewhere. What? It's always in the source code somewhere. No, I don't recall. But yeah, in cloud in it, it doesn't only just give instance information. You can install packages. You can do pretty much anything. You can run any bash script. You can feed it a bash script and have it run through. But it also has some useful directives so that you don't have to do everything in bash, which makes it so that your cloud in in scripts, if you've been installing certain packages in all your instances, and you want to try it on, let's say, a Debian-based instance, your cloud in in script will transfer pretty easily to a new distro, which is one of the nice things about cloud in it. I like the popular stuff where there's a resourceful package called an absolute unit. Right, exactly. Well, in Ansible, it's much the same way. And a lot of what you can do with Ansible, you can do with cloud in it and vice versa. So I've run into a couple times where cloud in it, like the package was failing, but I still needed to configure my instances. And so I was able to do that through Ansible. So how does test cloud fit in? Well, test cloud is the homebrewed electric moped of clouds compared to the Model T and the Tesla there. This is test cloud. It handles the giving you your metadata and instantiating your cloud instance locally on your machine. It runs off of QMU, Libvert, and it just makes it a whole lot easier. Because one of the problems that I ran into as a new QA guy is I had access to a couple of open stack instances that were often running out of storage or there was some other issue with it. And I didn't want to use EZ2 because I didn't want to charge the Fedora Project money just for me testing if I could do it somewhere else. And so what I had been doing is I looked up some documentation and tutorials on how to just boot a cloud image in Vert Manager. And it was a royal pain. It was like 12 steps that I had to run each time I wanted to boot a new instance. I didn't know it was going to be like passwords, for example. Yeah. I didn't see the metadata. So I was spending 10 to 15 minutes each test just to get it set up and ran. And I was like, well, this is a waste. So I started scripting it out. And it was originally just an easy dirty Python hack to do it all automatically for me that way. I didn't have to. But it requires Vert install, QMU, Libvert, and then also it can, by default, spin up Vert Viewer so that you can get a GUI. You don't have to use SSH if you don't want to. And it's recently been ported to integrate with Libvert. So now when you start an instance in test cloud, it'll show up in Virtual Machine Manager. And you can manage all of your machines like that, which has proven to be a whole lot easier. Because when you're just doing raw QMU commands, it's really easy to lose instances and leave them running for days on end and not even know that you have them running because you weren't paying attention. And the QMU instances, they're like 25 arguments. Oh, yeah. A QMU instance is almost a whole terminal full of text. And that was the other thing I was doing by hand each time. Like, it got really old really fast. And because you're like, wow, this QMU instance interface changes constantly. You usually look very upset. Yeah, exactly. Well, we just recently did the porting for that. And it's become a whole lot easier to use. But the potential uses for test cloud, initially, I was just using it for testing. And it was one of those things where I had an itch and I scratched it. And it never occurred to me that anybody else would ever want to use this because it was just a dirty Python hack by a non-cloud guy. I was taking information from tutorials and just jamming it into a Python script. But it turns out that other people also had the same itch. If one person is going to issue a lot of people, you have to issue. It's true. When I told a friend of mine that other people were using my code, he just laughed at me and said, ha, ha, now you have an open source project. He's like, welcome to the club. That sounds weird. So the other thing I've been using the test cloud for and other people have used test cloud for is setting up licenses. For instance, I'm able to test software stacks and why not on a cloud, while I'm a plane, when I'm not willing to pay for the $400 minute wireless access. Also automation, did any of you guys go to the talk on tenure? Well, tenure is a small continuous integration application that Cushal DOS is trying to use for testing cloud images. And it actually uses test cloud as its engine to boot instances and SSHN and run the tests. And then the larger project that tenure is going to be integrated into is called Taskatron. Has anybody heard of Taskatron? Yes. Okay, well Taskatron, like right now what we're working on is disposable clients. Because right now Taskatron can run tasks, like any kind of generic task. But one of the things we really wanna be able to do is run destructive tests. But currently the way it's set up, you can't run a destructive test because then it kills the slave. But now we have, with the disposable clients working, we're gonna be able to spin up an instance and run the destructive test to report the results all in a safe manner and test cloud is the engine that's being used to do that. And then I have reports in the wild of somebody at their day job using test cloud to deploy their production services on their internal network. Which I do not recommend. It says in the name. At all. I do not recommend this at all. But those servers have been up for, I think 11 months now and they're working fine. Cause really all test cloud does is it handles the prepackaging and then it passes it off to live for. So it's, I guess you could do it, but you have it from me, the author, don't use this as an appointment. Don't use it as an appointment. Is that an e-pap, an M-E-M-D cloud? Yeah, well, I was asked recently if I had any plans to make test cloud its own proper, like small cloud service and I don't. That's an order of magnitude higher than somebody looked at it and they're like, wow, you're like a web UI away from an OpenStack. Probably work better. Well, and that was one of the things I was looking at is I didn't have, when I started out testing cloud images, I didn't have hardware to run an OpenStack instance of any size. But then also getting it to work was a non-trivial task. And I was like, I don't want to have to maintain this day in and day out just so that I can make sure that I can install packages and test services on our cloud images. But yeah, now it's demo time. Oh, don't worry, it's recorded. I know, right? So what this is gonna do is I recorded output with a script, has everybody heard of script? It's awesome. I can't believe I didn't know about it before because I've done live demos before and that was a really bad idea. How's the cell? Script. It's just script that's installed on your machine already. It's a really old tool. I guess it used to be used for turning in homework. Like, you turned in a timing file and the output and I was like, man, I would've preferred that way to turn it in homework. But it's going to install test cloud, which is currently the only build is in is in Cobra. Right now it's still pending package review to actually get into the Fedora for Popscores, but it's gonna install it from Cobra and then it's going to download an image, a cloud image and then boot it up and just run through some basic tests. But this will give you a feel for what it does and I'll explain where it's saving information and how it's getting in. Because in my first iteration of test cloud, every time I ran a test, it downloaded a fresh image, but now it'll download the image once and cache it and then use that image as a backing store. So each of your instances is fresh but it also doesn't eat up all your hard drive space. So you can have, is everybody familiar with backing stores like Q-Count to backing stores? I've seen it each and every user. Well, it's really nice because now for test cloud, the instant storage is just the delta between its backing store and what I've installed. Oh, so it's like a snapshot when it's stored separately. Exactly. You know, and you can run that over a network, you can use backing stores from a lot of different places. Oh. The recent versus the third manager. Yeah. No, you can, it's a, it's a commemoration. Very much a new AP. Yeah. No. Yeah, I kept all my time. It's the corner of this typo. Oh yeah. Well, you know, I wanted you to know that this actually happened. I didn't just, you know, fabricate all of this. Yes, you did. But it's in cooper. It's going to install a glit guest FS for it install for a manager. Initially, the way that the only way that I could get to work with Q-Mew was I had to extract the kernel initial RAM disk that way I can modify boot odds before it booted. But now this handles all of that comfortably. And I guess next time I should do this on an 800 by 600 screen that way it doesn't look that bad. So it's like, when I saw it, you know, glit guest was like, what am I modifying in this? Is it an invalidated test case? Yeah, all it did is extract the kernel. Right. And you don't need any more. Yeah. Yeah, it doesn't do that anymore. But the CLI interface is, you know, test cloud instance and then create, stop or destroy. You know, so you can create one and then you give it, you give it a name and a URL. And it'll go and fetch that image and it can do local storage or, you know, from Koji. And I ran it, like I've learned a lot about Livevert with this entire process. Livevert's really hard to silence. You know, I have explicit warnings in there. It says, don't worry about these Q-Mew driver errors. Livevert is whiny and has no method to shut it up. Like, you know, so basically I just have to keep pounding Livevert until it gives me the domain name that I know that has. But then it'll return the IP address of your recently spawned instance, which is one of the things that has been a pain if you've launched something in Livevert. There's no easy way to tell what the IP address. Yeah, and it's, you have to add a lot of stuff and do a lot of monkeying around. So my solution is I dump the XML of the guest you just created. I find the MAC address. I do ARP A and I search for it and then I spit it back. And, you know, it works. I haven't had any failures on that particular piece. And then, so the way Test Cloud works is, can everybody see that? Fun? Okay. So, Test Cloud has cash in instances, which I'm, apparently, I need to rename cash because I guess that's caused some confusion in the past. This is gonna be backing stores or something similar. There's a bunch of people that want cash, disks. Yeah, it's something. I just, you was using cash as a generic term, which I guess caused some frustration. Just to clarify, you know what the process is, our standard says, you know, DDS temporary or DDS current, like you need to be here and all. Yeah. So if I do Test Cloud instance list, it will look through and find me all of my running instances. And then I can pass all, and it'll tell me all the ones that stop, that are shut off. So if I do, I really gotta do tab completion for this. And now inside the instances, there's a directory for each instance. So if I go to my cloud, so IP address is just a flat text file with the IP address that it found earlier, because I have created instances and then gone away and come back and not been able to find the IP address of the instance that I created. So I started saving it in a file. But if we look at the metadata, you know, it gives it an instance ID and a local host name. So like you can go in here and change this for each of your instances. And it's all in a Python config object, but it defaults to Test Cloud and it defaults to a password of password. And it allows SSH password authentication because the default on cloud instances is to only accept SSH keys, which you can also use Test Cloud to inject SSH keys. This was just easier for me and has stayed there for the past year. For your reputation, what you need to do is just put it up. We need that one. Exactly, and it's all local behind the firewall, but you know, anything that you like, I tried to meddle with as little as possible. So anything that you can do in cloud net, you can still do with Test Cloud. You know, you can feed it custom user scripts and metadata information. And I still have some API work or some CLI interface stuff to work on to make it easier to do. Right now you'd have to manually edit the configuration object. So if we do. I'd like to point out that there's a bunch of rules like static exchange answers for what's the default password on the word cloud. Yes. And they apply it like through a 19. And actually there's no password on those. We're calling it. I answered your earlier question on your finger. I was number two on the list. That wasn't the part. Yeah, something is off on my... That's my SSH key password. You know, now we have a running cloud image locally, which isn't much, but it saves you a lot of extra time getting all of the everything configured. And it's gonna be, it's got what I think are relatively same defaults. So whenever you spin up a new instance, you know, it's gonna be the same user, the same password, you know, unless you configure it different. And right now this is only tested on Fedora images. I don't see any reason why it wouldn't also boot Debian, Ubuntu, CentOS, CoreOS, you know, whatever type of QCOW you could throw at it. But that hasn't been tested yet. That's something that we plan on doing in the future so that you can pick out, you know, you can test multiple images with it, which will be especially useful for anybody else that's using it. It's like in Taskatron, you know, if you wanna test that your package builds and also installs on CentOS, you know, we can do that in a disposable client. So you can also do image list, which will tell you all of the images that you have downloaded, which is nice, you know, on my desktop back at the house. There's like 40 of them. And right now it's just the file name of whatever gets spit out of code here is put on download.fp.o. But it also makes it really easy if you're, if you wanna reuse an instance, all you need is that name that you pass it as the URL and it will look and say, oh, I already have that and then use it, you're good to go. So yeah, any questions? What can I just say that you plan to remove like the need scene spice support or? Yeah, initially I had different arguments and commands that had to be passed into QMU and port redirection and whatnot to allow VNC and spice viewers to connect to those guests. And I haven't removed those flags out of the code yet. But right now it runs in VertManager so you can change all of the, you can twiddle those settings as much as you want, you know, for however you wanna view it. Yeah, then you think that's a nice use of app. Yeah, well, now that Lidford's handling all of that, I mean, it's automatic. You guys can see it and then you can like use test cloud to get the cloud around you to make, oh, it's all the desktop environments. Oh yeah, well, you know, that's kind of the plan because, you know, with Taskatron and disposable clients, we wanna make Task cloud capable of also running desktop environments. So like, you know, for GNOME, they have like a GNOME test suite that's relatively, you know, runs through a bunch of stuff but we could spin that up and actually run the GNOME tests on a fresh install in NMVM before we sign off on that build. That's the best thing I wanna look at that I'm the person who does the three standards. Yeah. I'd love to do some more quick testing of the day. It moves. Exactly, well, you know, I'm working on right now porting a bunch of the sentos tests to work on Fedora and going to run them via test cloud. You know, that way we can increase our test coverage overall, but that'll fit right into Taskatron and Taskatron, you know, when there's, you can write generic tasks with disposable clients, you should be able to set that up as a task anytime a respan happens. Well, or I can set it up to do some tests, boot the lab image and do a test cloud and let it go with that way. Oh yeah, no, it can do exactly. You need to test your lab images as well, so. Yeah. I think down the road it's gonna solve a bunch of problems. Yeah, well, that was the hope. But yeah, I really learned a lot about what is okay for me. Have on a quick dirty hack that only I use and how next time I think somebody might find something useful I should try to, you know, design it right the first time because going back and refactoring is not much fun. Yeah, so that's test cloud and you know, we're gonna be able to use it to, you know, so people wanna try the cloud image. So hey, install test cloud, feed it this URL and now you can poke around on the cloud image. You know, you don't have to have OpenStack running somewhere, you don't have to have EC2. You don't have to figure out the esoteric chicken sacrifice practices that you have to get it to run it for. And you said you don't like magic, but you didn't, you said it's all magic, come on. Well, I tried to explain it to you, so it's not magic anymore. But yeah, the source code is all, I mean, it's not a very big project. It's some wrapper commands and utility functions to handle extra data, but it's really easy. Patch is welcome. We've got a, I need to come up with a roadmap on what all we're actually gonna work on. But there's, we have a list of things that need done if anybody's interested in helping us out on it. Yeah, Adam's working on that right now, actually. He's got some packages to finish. I have to mention, if you're just going to test pilot, you'll see this like a company test pilot, yeah. Yeah.