 Hey, so welcome everybody, today we're going to be talking about OpenStack, Python and the Holy Grail. So we kind of talked about this in the guide as like a new proposal for image portability. And maybe another way of looking at that is, is like how do we actually go about making this building of a planet scale cloud an actual reality? What is the real thing there? Looking at how you use images, how you use many different OpenStack clouds, all looking like this is an OpenStack cloud, I know what that means. So before we get going, how many of you have actually watched Monty Python's quest for the Holy Grail? Awesome, apologies if you haven't. The human will be bizarre, but hey, never mind. So I'm John Garbit, known as John the Tuba Guy on IRC. I'm a software developer with Rackspace. I've been doing OpenStack stuff from a very early stage. I started out doing sort of the private cloud packaging back in when I was working at Citrix, sort of private cloud packaging of OpenStack. And now I've sort of gone to the other extreme, still using Citrix Zen server, but inside Rackspace on the massive public cloud that we have there. Awesome fun. My favorite color is silver, by which I mean, I do actually play the Tuba. I attempted to do some terrible graphics work here. I don't think King Arthur played the Tuba. It wasn't invented then, but never mind. Hey, so I'll pass you over to Brian. Hey, I'm Brian Rossmaida. I'm the Cloud Images product manager at Rackspace. I'm on the plant's driver's team. As you know, it's important to know your favorite color, so mine's green. And I don't play an instrument, but I do play reggae recordings on the radio. So that's us. Awesome, didn't want to feel left out, so take it away. Okay, so we have some questions to think about and you can see what they are. So we're interested in what's your favorite deployment strategy. We'll discuss that a bit. What's your favorite new cloud servers feature? So what do you want to take advantage of with portability? And then we want to discuss what's the best route to interoperability. And just sort of a hint. So as John showed you, we sort of changed the subtitle of this. And part of it is we've decided about partway through that everybody thinks they want image portability, but I think what they really want is workload portability. So we'll discuss ways to get that sooner. So before we kind of dig into this kind of thing, I wanted to bring up this whole idea of bootstrapping and baking and portability and how all that kind of interrelates. Like how you use your cloud and what that really means for what portability really means. Like what's the problem we're trying to solve? So the what is your favorite deployment strategy? I don't do that very well. I should have a big shaggy coat or some kind of, I don't know. Or a scroll, anyway. So there's lots of like Pat versus cattle analogies going around in the cloud, right? And one of these comes from the sort of this bake philosophy. So the idea is is you build this image, you test it and you've got this snapshot and this image in your system that does all you need. And then you stamp loads of those out all over your cloud, put them behind low balances, whatever you want to do. So you've got all your sort of like cows in your field sort of chewing at the request grass, I'm putting this metaphor far too far now. But you get the idea, this is kind of like you build the thing, you test it and you know what you're deploying and all these different things. So this is like the many identical clones, not to mix Star Wars with Monty Python, but you've got lots of good things going on there. Now, when it comes to implementing this thing in the public cloud, I just wanted to have a slight aside. So like recently, I've been doing a lot of performance work on the Rackspace Cloud and looking at how we make server builds as fast as possible for everybody. So when you think about doing it with a base image, you can cache that base image in loads of places. And it means that that download from glance to the local disk, when you're running your server with a local disk, as you kind of probably want to do with your cattle, because there's no point having paying for the extra remote storage. That deployment process is actually quite slow, because you have a lot of bits to get from one side of the network to the other. So it's hard to get the caching right on that. So it's interesting, I mean, when you look at boot from volume and you get copy on write technologies, they kind of get a lot closer to getting that very fast start with lots of identical clones. It's a bit of an aside, but it becomes problematic getting that to work really well. So the other approach is this kind of bootstrap idea, which kind of boils down to using Cloud in it and DevOps-y kind of things. So if you get your base image started, and you can actually insert like user data into your request, you can get your config management service to start configuring that instance to be exactly what you want. So you pull in your latest code from Git, you get all the dependencies installed, you wire it all together and you get your beautiful web server, say, working away how you want. And that makes a lot of sense for making something very similar from a developer point of view of what I have working on my laptop and what I have working in production. It's a very similar workflow, right? I use all my config management tools here and in the virtual world, then I go to a physical server, it's the same thing. You know, you've got the OS and you apply your stuff onto it. It's a kind of a nice way of working. So these are kind of the two contrasting strategies. You create the like image and you stamp out loads of the same image or you bootstrap lots of VMs. So you create lots of VMs doing this bootstrap thing. So we can talk about like what's best. And it kind of depends on what you're doing, right? We need to support both of these and work out how it works. But when you look at like how do I bake my image, I think most of the cases when you're trying to do that in a good DevOps way, you're trying to automate that, you end up actually bootstrapping the image that you then create and then stamp out across your cloud. So these things aren't necessarily totally fighting, but it's like how do you do that deploy and exactly what you're doing will depend which is fastest, like whether you can pull from the internet all your resources faster than actually distributing all your images around the cloud. It depends. It's an interesting mix of match. Anyway, at this point I'm gonna hand over to Brian to talk through kind of what we've been doing with a few of these things. I suspect it might be me first. So we'll discuss what your new favorite cloud server speaker is since I'm a product guy. I got to give a little plug here. So let me try the voice. What is your favorite new cloud server feature? So this connects in because most clouds are constantly improving the hardware, changing things, trying to make things better. So if you're locked into some images, right, some custom images, they may not run on all the new hardware. You might not be able to take advantage of the latest technology. So that's one thing to be aware of. So performance servers, for example, have smaller system disks. They're optimized for a lot of memory, for really good networking. And also the idea is that you'll probably attach your data by a volume instead of carrying it along and a disk or send it through files. So that's one possibility. There are also new bootstrap methods developing. So SSH key injection, use of cloud knit, as John was mentioning. And then just DevOps as an automation service. So just trying to make it easier to make the deploys from base images. Now one thing with portability. So in the Rackspace Cloud choice talk yesterday, you mentioned that we've exposed Blants in the Rackspace public cloud. So we've got version two of the Glantz API running. So you can do image sharing, which allows people to build images and then share them out to other accounts so other people can build servers from those. So that spreads it around. And there's also image import. So you could develop a custom image outside the cloud and then bring it in. And there's correspondingly image export. That's a little less interesting for this point. So we offer those. So it allows for more collaboration. And what's interesting about the image import and also the sharing is that we offer a base set of images, but they don't have all the operating systems every possible customer would want. And some customers have very specialized needs. So this allows a way for you to bring an image into the cloud if you prepare it properly, it'll run. And then you can share it out to other customers and they can take advantage of that. But again, if a lot of people are sharing the same type of image, then it makes sense to go to more of a bootstrapping point of view so you can use the custom image somebody's made but you'll have to bring all your stuff into it. Yeah, so just to go back to what Brian was saying, we've done a lot of work with making the bootstrapping work. I wanted to kind of say, sure, this is kind of a plug for the stuff we do. But what I wanted to say is really this is like, this is Rackspace saying we are committed to making stuff more portable. We see the value, our customers really need this and we really want to try and get this to work. So we've come from a place where we had a cloud before OpenStack and it was using, we've got our own agent that's specific to our setup doing the sort of configuration of these machines. We're working hard to get this, so it is something we can all standardize on and see how it's going. I mean Troy was saying this is kind of a, it's a tricky situation on working out exactly what people need. But for me, if I'm wanting to do DevOps kind of things, having SSH key injections, so you have password swing around, that's like super important. So let's standardize on the NOVA APIs, get that working. Cloud in it, you can add your user data in there. We've got cloud in it now in our base images, so you can actually start really working on that. And what we're doing at Rackspace, right, we're giving fanatical support, like we always have done to people working on a managed cloud. And we're helping people sort of actually get into this whole DevOps mentality for lots of the customers and helping them through this with a DevOps automation service. I just wanted to highlight kind of what's going on there. Cool, so our next bit is sort of, okay, so we've looked at this is how we use the cloud, these are the features that are available in the cloud. And now we're looking at, okay, so how do we actually get to this interoperability? What does that mean and how this is all gonna work? So we have to decide what, I guess I was supposed to do the voice. So what is your quest? So portability or bootstrapability. And it really depends on what people wanna do, of course. So some people wanna move between clouds, maybe for high availability or something. Some want to move from a dev box where everything's set up into a cloud or move from an old server or something into the cloud or move backwards out of the cloud back to on-premise for some particular reason. So the question is, you gotta move a lot of data. You've gotta maybe move images. What's the best approach to handle here? So, I mean, with moving data, I mean, you have to be careful because of course it can blow up in your face. Problem with data is it's very valuable, which is nice, but it's also usually very big. So it's a lot of stuff to move and get moved correctly. So if we use a DevOps kind of approach, it should be fairly easy to install data and move it around, probably by keeping it in a volume and then you can mount the volume onto different servers. Another possibilities and sort of the old style cloud has been to keep data on an image and then move the image around. And then when the server comes up, it's got the data there. But then that's a lot more fragile. So we're sort of not quite in favor of that. Yeah, that makes sense. I mean, and to highlight here, sort of in our cloud, if you're trying to do downloads from Swift, we have like optimized routes within the internal network to make that fast. We've got a service net there for those kind of things. And it's, yeah, I think this comes to sort of like, how do we talk about image portability? And for me as a developer, I kind of want to change the rules about what we're thinking about here. So if we think about separating, if we think about bootstrapping in the same way across all OpenStack clouds and we're moving data in the same way across all the OpenStack clouds, the question is, is that good enough for most people's cases? Does that get us an awful long way down this interoperability route? It's kind of, there's enough work there to keep us going. I don't think image portability is necessarily a bad thing. And it's something that we will need to work on. I'm just saying, can we do this other thing first to get the sort of really good DevOps style use cases working well across all OpenStack clouds? Also, this is probably easier. So as a developer, I kind of see, I want to do the easier thing, but I think it's gonna be, it's empowering people in the right direction, I hope. So, yeah, image portability, like why is it actually hard? I kind of just wanted to go through a few things. Looking at the sort of new glance feature and rack spaces made us look very closely in how people prepare images, how they get them into the cloud and how that differs for other clouds. So whether you've got Zen versus KVM, the Linux community did a great job with the PVOps kernels of making that much easier, but depending on your image, that can be a problem. When you've got a Windows image, you've got drivers in there for the hypervisor, how you get the right ones for the right platform and getting those switched. It's kind of, this all gets messy and the question is, do we want to bother with this as a user of a cloud? We can just use the base image from the provider that has all this done and we just use the bootstrapping. There's a great demonstration of Juju with Hyper-V earlier and seeing, getting sort of on the Windows images actually sort of using more orchestration on Windows images and other things. It'd be interesting to see how this all goes with the DevOps approach sort of moving at a pace in different places. So we'd kind of need to, I think it'd be good to sort of make sure that we're innovating in a way that's going to be interoperable between all these different things. So yeah, going on to this, what's this vision for common bootstrapping and just to kind of recap it is, using the cloud images from your providers, they've got all the bits and pieces that may be specific over time. I hope that there are no pieces that are more specific on each of these clouds. We get to use some of this. So as an aside, I just kind of lead the sort of Zen server support sub team within Nova and the Zen community has been doing some great work on trying to make it easier to use Zen with different things. So on the Windows side, there's a lot of work to try and eventually get to the point where Zen is actually hardware that gets presented to the OS that the OS can then get drivers for through the official routes and all these kind of things. So there is hope for getting this to be, you use it and the OS is intelligent enough to work out what's going on. In the Linux world, I mentioned PVOps kernels which are doing the same thing. They go, hey, your Zen hardware or your KVM hardware and they get the appropriate drivers loaded and stuff starts to work. A bit of a deep dive. I kind of just wanted to show that there is progress on getting that kind of ease of use within the image but for now, using the cloud image and putting stuff on top kind of just sidesteps all that argument anyway. But it's an interesting idea. And as we mentioned earlier, using cloud in it to sort of get the config management so that you've got this, I keep saying DevOps far too many times. If someone's doing like, what do they call it, like lecture cricket? I think we used to call it open place. Someone's going to be winning if they had DevOps so I apologize. But it's kind of, you get the approach where you're pulling in the Git code. You're using your infrastructure as just a programmable thing. You're using it a bit like software. And that really sort of makes it easy to port what you're doing from one platform to the next. And the overall thing is to try and make this the same for any OpenStack cloud, right? So I guess let's take a look at what's at the other side of the bridge that we started with, see where we're going. So we want to do a common bootstrapping process between the Rackspace cloud servers and the public cloud and then the Rackspace private cloud is a big game. As far as data goes, we'd like to do disk conversion and move it to Swift. And then we can have it optimized, downloaded to an ephemeral disk and then we want to contribute to the defining the core tests to make sure all this works. So we started out saying we wanted to make images portable. It looks like that's probably not as good a goal to go for as to moving to more of a DevOps type approach. There's that word again. And then because we're starting to go to multi hypervisor clouds, it's the target for what your image should be like so it can take advantage of the drivers and so that your VM will come up and be fast keeps changing. So, and the other advantages, if we use a cache based image, then build is really fast and then you can load up your machine with other things. So we're starting to think that architecture shouldn't really be on the image so much as on everything else. And that's what we're thinking. This is the proposal. Anyway, for those who are squeamish or don't really like brass instruments, you probably want to look away for the next slide. But we wanted to be good to open the floor to questions and what you were thinking about this. You can see my great graphics work of making a cloud and inside the other clouds. I should keep doing the day job of coding. This is for sure. But anyway, yeah, there's a mic here for questions in the center if people want to make an orderly line. So try to tie in the theme here are the Linux containers, are they the holy grail of this or they fit into this in any way? That's a good question. So like when you start comparing the Linux container world and the VM world, does this help unify things? Is that a fair statement? So like full disclosure, I really like Zen. So that's part of me that says containers are terrible. But I think what's really happening here is there's the right technology for different use cases in different ways. I think one of the great things that we as a company do at Rackspace is that we have managed hosting. So for those older style workloads where you really need someone that can sit there and hug your server, we have intensive managed hosting where that kind of thing happens, where you need that use case and where you need something sort of smaller than that and more dynamic and you need to start it quickly, then you've got the whole VM scenario, which is great. The problem with containers right now is how do you secure that for when you've got sort of people with untrusted workloads getting on there? So if you're trying to do like platform as a service and you're starting lots of databases, containers is great. But anyway, that's just my random rant on hyperverses versus containers. I think everyone has a great space to play in this market and it's just a case of working out what's the best technology for the best workloads. But coming back to this topic, as I decided I've completely distracted myself, is I think having that common bootstrapping kind of thing, it makes sense for containers, but it depends exactly how you use the containers. I'm a little bit torn. So there's use cases where you have a container per Apache request. So in that case, there's no way that you want to do a bootstrapping within that image because it would take too long. But to create that image, I think you would want to use an automatic bootstrapping mechanism to actually create that thing. So I think it's all part and parcel there. I don't have all the answers, but I mean, that's kind of my ramble and I should stop now because someone's gone on the question. You actually touched upon something I wanted to ask. Cool. So we started off thinking about images and thought, oh, we'll go down the image route. And then what we realized was that we didn't want to maintain images manually. So then we went down the bake approach. The problem with the bake approach is that so we use Chef and we have Chef runs and it does it and everything, but it's too slow. So the bootstrap for us would be too slow. So now we're going back to the bake process, the image process, right? So fry, so fry versus bake, right? So we were frying, now we're gonna bake. And so what we're trying to bring in is something that allows us to automate the image creation. So we have the Chef process will identify for all the different components and we just click a button and it re-does the bake with all the patches and whatnot. Now, from what you're saying about bootstrapping, obviously that's to apply, right? You have to make sure where the balance is. You can't just decide that I'm gonna go down the bake approach because that's gonna be great and it's gonna be perfect. Especially if you don't have it automated. So I mean, do you have ready have that process in place for people to say define what the end state of an image should look like? And so when you think of it, so when I'm thinking of a end state, I'm thinking of everything that I need for that role. I mean, we don't run platform as a service, it's like literally we give a standard VM and we put everything on top of it. But if I said to you, if we're taking a very simple approach, if you said, I'm gonna give you a Tomcat server, then the bake process is just dropping the war in, right? At this time and then making the image and spinning up a thousand of those. But do you have that automated today? And if you do, how do you do it? The honest answer is I'm not entirely certain. And I think, so it's an interesting point. I think that the, it sort of touches on whether you should use, it's an interesting point, because it touches on the difference between PAS and infrastructure as a service and other things. And my personal view is, is I think this space will get, the diversification in PAS is important because everyone has a slightly different itch to scratch. So I think having an API where you can go to it and say this is my war and deploy it, I think that does make a lot of sense in doing that quickly. So actually let me add something to it. Yeah. So make it a bit more difficult. So it's easy to do when you've got a state of system with no data, right? So you mentioned data there. So we have systems which are ridiculously large. I wouldn't call them cloud-friendly at all, right? But we put them in the cloud. So in those scenarios, obviously you're talking about attaching the storage and whatnot. And so yes, the war Tomcat one is a very good example, but imagine you had a Mongo instance, which has something along the lines of a three terabyte data line attached to it, which is like 80% full. Right. Again, your recommendation is to split it out and have the data separate and attach later on. I think it's tricky. So when it comes to moving your application from place A to place B, moving data is not an easy thing. But if you're trying to do that, you do need to move the data. So it might be more of a case of ensuring that you sync the data between A to B. I think more that I'm trying to say is if you have a big 300 gigabyte image with lots of data in and lots of programs in there that are all sort of intertwined into this crazy mess, when you're trying to move that to another cloud provider, if you export that image and try and import it somewhere else, you've kind of got two problems. You've got this massive load of data that's jumbled with everything else. And you don't really know that when you're running it in the other place, the image will work. So the argument is more if you have, I mean, I guess consider it more as two problems. So if you import that big 300 gigabyte blob and you import it over here, if you just consider that as I've got the data in, then you've actually got the data in there. And then you can think about, hey, so I create my new image over here and I ensure that I reinstall all my applications in a DevOps way over here and then attach that data to that VM. I think that's a healthier thing to think about. I guess it's my point because if you're trying to move this whole big blob with all the applications and the data are all mixed up, it becomes a much harder process to try and undo it the other side. I don't know. Last question. So we're actually in the process of looking at different providers to use cloud providers and that point you mentioned about migrating your data from your in-house cloud to a public cloud, we're hitting that issue. And the actual issue we're hitting, so the cloud init stuff is one example, so we weren't using cloud init. So I'll give you a bit more background, we actually use VMware as our internal cloud thing, right? So you get some free stuff with VMware with its post and pre-boot provisioning mechanism. We used to be in AWS, we used to use cloud init there and then we came in and we started using this. And at the time, there was no cloud init. So we have 50, 50 windows in Linux and there was no cloud init for Windows so there was that issue we had to resolve. Now we've realized that, yes, we have to use cloud init, we should standardize on that in images but the problem we're finding now is OVFs which are supported by VirtualBox, VMware isn't supported by OpenStack, right? Or at least it wasn't when I checked last for Havana. So in terms, now we have to do this conversion process of converting the image to whoever cloud you're going to. So how do you address that? What's the most portable image that you could use if you were gonna go down that route? Or? Yeah, it's a good question. I'm conscious I don't want to spend too much time on one question or one set of questions but so yeah, do queue behind if you want to ask another question is what I mean. But let's think about that. So I want to, I need to pull the problems apart slightly. So I think if you're in an approach where you can automate your install of things with configuration management, then pointing at a different images and fixing that up makes sense. Because in a way, you're pointing at one distro here and one distro here. That's quite likely to work when you switch distros as different problems but at least that's a much easier thing than thinking of I'm moving this whole image and putting it somewhere else. That's my kind of general thinking. If you've got, yeah, if you're just pointing a config management in different places, that's an easier place to be. Might be something you want to do anyway. Going to the specific OVS point. OpenStack does support OVS quite a lot but it's, is this OVS inside your image in the cloud? No, so... Or is this your private cloud deployment you're trying to use OVS with? When I was first looking at OpenStack, I looked at, oh, I'll just take, I know that it's supposed to be MDKs but we weren't sure if we, so one of our cloud providers said that, oh, can we have an OVF? And we were like, okay, but then we didn't want to sit there making an OVS conversion and then taking that as it is but then what your point was why don't you just take your process of creating images and just create a new one in their cloud. So that's the approach we're looking at but I was just curious, was that cool? Yeah, I don't think OVS should affect you in the thing that you're thinking about but we should talk about that more. All right, thank you. For sure, no problem. Thank you for the questions, that's good. Cool, more volunteers. So, talking about bootstrapping and the role of cloud in it, what your experience with the maturity of cloud in it and what are the platforms that you support and I'm just fishing here trying to get some practical feedback because to be honest with my experience, I've seen that it's kind of working pretty well on Ubuntu but when you move outside of Ubuntu and maybe Fedora then you might run into a lot of issues. So aren't we making some assumptions here like everyone is skipping over that cloud in it is gonna solve your problems, don't worry about it, it's portable, it's kind of consistent across platforms so my question is, is it? That's a very good question. So if you tell, I think where this gets harder actually, so the specific thing that I have a concern with myself from a personal point of view is when you talk about package names because you kind of want to say I want to install my package X and they're gonna be different between the two things. So that makes me concerned. So maybe what I'm thinking about is I think the conversation in the OpenStack community should be more about ensuring that we have solid API. In this particular case, cloud in it is not part of, it's part of the OpenStack ecosystem, sort of by association, if not by official naming, but it's the, I think what's more important is that we get a healthy API between OpenStack and cloud in it. So one of the things I'm interested in is that when you use config drive to insert the networking, for example, currently we actually insert an Ubuntu style networking file that then the REL stuff has to actually, and the CentOS stuff has to convert to a CentOS thing which is a bit of an odd API. So we are working in Nova, I got a blueprint up, actually there is some code up for it to try and have like a more agnostic description of the network so that we have like good APIs there. So I think, I worry, I'm sidestepping the question a little bit because I'm not sure I can answer it properly because I don't use it that much. I don't use it enough to make a good qualified statement across enough districts. But I think the more important thing is that we get a good API between the bootstrappy thing inside the image and the thing that Nova's talking to. I think that's the important thing. Have we got it right today? I'm pretty sure the answer's no, actually. We're still working on that. But I think, yeah, that's the important thing is getting the thing inside the image that responds to the metadata that OpenStack provides. It's that, I'm probably misusing the term here. It's a bit like, it's the interface between the image and OpenStack. Like getting that as a clean thing. We're spending, I mean, we're spending quite a lot of time thinking about the interface between our users starting servers. But there is this other interface, right, between the stuff that comes in here and then when it talks to the instance at the other side. Yeah, if you imagine Nova as a slice, you've kind of got user rest APIs to start the servers. You've got Nova and then stuff sort of leaks out the other side in some weird way for the instances. So we were talking about standardizing how we present config drive. We got much better at that. And I think, yeah, getting that flow correct, I think is much more important, right, from an OpenStack perspective than perhaps what's responding to it on the other side. So I'm conscious of the time. But for example, on Windows, there's cloud base in it and there's different cloud, effectively there's different implementations on cloud in it on each thing. So it's a problem of making sure that that all works in a cohesive way. Yeah, it's a good point. This is, this presentation is a lot more about, there's a lot more forward-facing vision than some of the bits and pieces. I think as a community, we need to rally around this and make it all work. So I'm afraid that's time. They told me they got rid of slides when I ran out of time, which is good. So yeah, thank you very much. And it's really bad because that's the guy who knows about cloud in it more than anyone else who can actually answer the question. But, sorry. Yeah, but thank you very much. Anyway, we will do this later. Okay, so I'll just comment on cloud in it in general. My feeling to that is, so A, if there are bugs on cloud in it, on Fedor or REL, those can get fixed. We can make that better. I'm not opposed to that. I know people use it at Yahoo in production on REL so it should generally work. My feeling as to how you customize something on boot is really that you need to know what you're talking to. And in many cases, that's cloud in it, but really what I think that the interface you're talking to is really, for us, Ubuntu 14.04, AMD 64, and all of those things will look like that. And then you know when you see an image like that, you can talk to it in a certain way. And so it's kind of the image that provides the API, not cloud in it per se, but you know, I talked to this image that way. I talked to Fedor in a different way. And then your packages have to fall out. The package difference is because of that. Yeah, so following on from that, then we would need to have a good description of what that image is between the different clouds, right, for that to work. Anyway, you've got to stop. But thank you very much for your time. That was awesome.