 AV gentlemen, are you ready to start? Happy with your life choices? Cool. Hi, so I'm Michael. I'm here to talk about deploying Nova. I've set myself an interesting challenge with this talk. What I did at first was I emailed approximately 20 senior OpenStack devs and said, if you could talk to deployers before they deployed Nova, what would you say? I got back a lot of staff, some of it mutually conflicting and turning that into something coherent took a fair bit of work. So I actually ended up writing this talk as an essay first. So I've ended up with a 3200-word essay that I'm going to present to you in 23 minutes. So when you do the math, so I have to talk at like over 100 words a minute, so it's going to be awesome. But I'll give you a link to the essay at the end because I had written this thing, I might as well put it online. So the premise of the talk is what would a Nova developer say to a deployer before they deployed? What mistakes do we commonly see? What do we wish people had considered before? It was too late, that sort of thing. So first off, in 10 seconds or less, what is Nova? Nova is the component of OpenStack that manages virtual machines and hypervisors and that sort of thing, compute resources. So it's intended to take pools of machines, preferably hundreds of machines, I imagine, and you come up and make an API request for a virtual machine, we pick a machine for you and do the thing. It's obviously more complicated than that, but like I said, I have to talk real quick. Okay, so let's get into the meat of the matter. There are a few things about people's operational environments that I wish I could say to them before they deployed. First off, consider what base operating system you're going to use. I know that long-term supported releases are very attractive, but depending on your vendor, we sometimes experience bugs in the software that we depend on, and sometimes it can take a very long time to get bug fixes backported to long-term supported releases. So you could think of Nova as really just a way of orchestrating tools you already had on your machines, to a large extent. And so, you know, if your OpenV switch doesn't work properly, we're going to have a bad day. A good example recently is we found a bug in QMU which corrupted the disk images for virtual machines, which is kind of a big deal, and it was a lot of work to get that fix backported to all the various distros. Tony Breeds did a really good job there, and some distros are better than others at working with us. But, you know, when you're picking your base OS, don't necessarily pick what you always use, or, you know, maybe there's a version of your base OS or your chosen OS that is targeted at OpenStack, and maybe it's worth if you're building a big OpenStack environment using a different distro from just your default for absolutely everything. The version of Python users use matters as well. We only test with Python 2.7 at this point, I believe. So if you were running on a distro with 2.6, for example, that would worry me. We also don't have complete support for Python 3 yet, so don't just assume that because Python 3 is cool that we support it. We publicly document the versions of things we use for our continuous integration, and so if I was going to go and pick a release of a distro, I would ask questions about what OpenStack was testing on and then compare that with what I was thinking about running on. Don't reinvent the configuration management wheel. There are community-supported puppet, Ansible, Chef, Juju, whatever recipes. If you already have, if you've already picked a winner in that space, I don't know, you're a puppet shop, then I think you'd have to have a pretty good reason to go and reinvent the puppet modules, right? There are ones that people have already written. They've put a lot of effort into them. If they're not perfect, they'll probably take bug reports, so I'd seriously consider not starting from scratch there. Log management is kind of a big deal, too. So the harsh reality at the moment with NOVA is you're probably going to end up running with debug logging turned on when you experience a problem. Our debug logs are crazily verbose, so we're talking about very large logs, and we're also talking about a system intended to run on hundreds of machines. And so a single request might hit many machines, and it's sometimes hard to determine what machine the problem actually occurred on. So I'd be thinking centralized logging. It doesn't have to be syslog. You could be using something like LogStash and Elasticsearch and stuff like that, but you do need to have a reasonable answer to how logging is going to work in a non-trivial environment. Monitoring is really important, but especially monitoring of memory on your controller nodes. So the Python services that we use to, sorry, that form NOVA can consume a fair bit of virtual memory, so you especially need to make sure that you're monitoring memory usage on your controller nodes. But in general, if a hypervisor goes offline, like a hypervisor node, you'll only lose the instances on that hypervisor node, and it would be quite possible without good monitoring to not notice that that had happened. The things about scale that my 20 survey respondees said to me. Think about your final scale now. I know that John Dickinson in the Swift talk kind of said exactly the opposite, but specifically if you're building like a proof of concept and it's going to be tens of machines, but you think it's very likely that you're going to scale to hundreds or thousands of machines because I don't know, you're a bank or you're CERN or something, then you probably want to deploy cells now, even though you don't really need it. And that's just because it's going to be easier to move to your final cells world later. We're actually re-architecting NOVA at the moment because we want everyone to deploy cells, even if you have a five-machine install. And we're going to make that relatively inexpensive. But if you think you're going to build a non-trivial environment, you should probably be thinking about cells early and often. SQL databases. This one seems a bit religious and I don't have a lot of database religion, but I'll try to pretend. Carefully consider what SQL database backend you use. So specifically I keep meeting people like we're a Postgres shop, we're going to use Postgres. And more power to you, but the rehearse reality is the vast majority of OpenStack Deployers use MySQL. So you're probably going to experience bugs that haven't been reported before and it's likely our CI doesn't pick up things that you'll experience. So again, you know, there's a value to existing operational expertise. Like if you are a Postgres shop and you really understand that, then that's cool. But you know, ask questions about what other people are using and what experiences they've had. There's some other things here. You know, we support read-only replicas. The database performance really, really matters. There are many user operations that block on database transactions. So if your SQL backend isn't performing reasonably, then you're going to have a bad day. It won't block Nova in that we'll, you know, run off and preemptively do somebody else's request, but your requests will block. And specifically at this point in time, do not run Galera. Now, this is a bit scary because it turns out that lots and lots and lots of people are running Galera at the moment and we want to support Galera, but select for update doesn't work properly on Galera. The right intent locks aren't propagated. And so we do see bug reports where it's simply that the locking isn't being passed around by Galera and it's all gone horribly wrong. In the essay form of this talk, I've included a link to a blog post from J-Pypes that explains exactly what this problem is. And it was only recently identified, so there are people working on it at the moment. And this is, I've mentioned this before. So SQL performance really, really matters. So, you know, presumably you'd be monitoring your SQL layer and making sure that it's responding reasonably. There are also options for different message queuing. So the way the Nova components talk to each other is via a message queuing system. This queuing system is not really exposed to, well, it's not exposed to end users at all. It is exposed to some other open-sack services. And there are choices. So, you know, RabbitMQ is supported, ZeroMQ is supported, and Cupid is the other one. Again, the vast majority of deployments seem to use RabbitMQ. So, you know, if you want to use something different for a good reason, that's cool. But maybe go and ask the operators mailing list if other people have experiences with that before you do it. Fundamentally, Nova is about hypervisors, right? What we're trying to do is provide an API on top of a set of hypervisor nodes and hand out virtual machines. So, when you get to picking a hypervisor driver, you need to pick one that is reliable and meets your needs. So specifically, if you have a lot of operational expertise with a particular hypervisor, that might be cool. But you also need to check that that driver for Nova behaves reasonably. So, for example, we have examples of drivers that are less well-tested than other drivers. We have examples of driver authors that are less well-integrated into the Nova team than others. And so, you know, just because you're... I'm specifically trying not to say bad things about specific hypervisors because they're all my friend. But, you know, if you're a big Zen shop, that's cool. But, you know, check into the state of the Zen driver in Nova before you decide that that's the thing you want to do. It turns out the Zen driver is fine. But, you know, there are other less mature hypervisors. I'd be especially concerned about hypervisor drivers that aren't in the Nova code base. The most obvious example is the Docker driver that's on Stackforge. But that's not totally terrible, right? At least we know about Docker. You know, we're working with those guys. There's some mutual CI happening. It's not the end of the world. It's on a path where we intend it to be in Nova. Even worse is I'd be exceptionally worried about running a hypervisor driver that the Nova team had never seen. So there are companies that will sell you proprietary hypervisor drivers that aren't open source, or if they are open source, they've never been shown to the... Like, they're not working with the Nova team. And that stuff is scary because the hypervisor driver's plug-in at a layer with no versioning in the code base. So, you know, over time, we'll say, hey, we need to do this thing to implement this feature and we'll change that driver layer. And for all the drivers we know about, we can verify we haven't broken them. But for the drivers that are out there that don't talk to us, we don't even know they're broken. We don't even know necessarily that they exist or anyone uses them. And that's scary. So this is a little bit of a nuanced topic. I provide, you know, a bunch of stuff in the essay form again. But it's subtle, right? You need to basically talk to people. Go and find other operators that are using the hypervisor you think you want to use, see what their experiences are, you know, try and determine whether or not the team for that driver is working well with the Nova team, you know, that sort of thing. So yeah, not all hypervisor drivers are created equal. Another thing I'd say to people is that hypervisor state is a useful debugging tool. I think it's often overlooked. So to pick on libvert for a second, if you're experiencing a problem with an instance, don't just look at what Nova's doing. Have a look at what libvert is doing as well. So I don't know if people know libvert, but you know, there's an XML config file that describes the instance that you then hand to libvert when you boot it. Those exist on disk, we just create them, right? And there's files in whatever directory you've configured for the instance that have interesting state. Like for instance, there's the console log for the instance, which might be useful. So, you know, don't pretend that Nova operates in isolation. You know, there's similar tools for Xan or Hyper-V or VMware or whatever you're using, but you know, if you're experiencing a problem, you might find that the hypervisor provides you with meaningful information as well. Networking, this one's a little bit easy because I can punt on the whole what neutron driver should you pick issue. All I really want to say is don't deploy Nova network. So, there are vendors out there who'll still happily sell you a Nova network open stack install. Nova network has been deprecated for a very long time. Nova network will be removed. Nova network will probably not be removed in Kilo, which is the current release at this point. Because basically there's some features missing in neutron that we need to have, to be able to spot absolutely everybody who uses Nova network. And we're still finishing like the, you know, we've got 90%, we need the final 10% kind of thing. But if you're a new user, you probably don't depend on those features that are missing. So start with neutron. Because if you pick Nova network, all you're doing is signing up to do an upgrade at some point of time that's going to be reasonably close to now and where you don't control the timeline. So just don't sign up for the extra pain if you're new to open stack. Testing and upgrades. Now I guess I'm assuming a non-trivial environment here. If you're building a five-nove proof of concept, then these things don't necessarily apply as much. But if you've got a reasonable size deploy, you absolutely need a test lab. I keep meeting people who, you know, the first time they run an upgrade is in their production environment and that stuff freaks me out. Like at one point I had to apologize to people because their instances were corrupted when they did that. And that was bad. But part of the problem was we had assumed that people would report bugs before they actually did it in production. So you need a test lab. If your boss is not sympathetic to a test lab, I guess you have a conversation about, you know, how expensive is it if this whole thing goes away because we didn't test it. You also need to test the database migrations on a copy of your production data before you do them for real. So we've got this SQL database, right? A NOVA upgrade may include multiple database schema upgrades as well. We do them in steps. And some of those can take, for example, an extremely long time to execute. And so some of our bigger environments have donated datasets and we actually do CI testing on the upgrades. But those datasets tend to come from public clouds, for example, where volumes aren't as common as they might be in your environment. So we don't guarantee that we mimic every possible deployment. And so I would suggest that, you know, just running the upgrades on your dataset before you do them for real would be a valuable thing to do. And you don't need a full NOVA install to do that. You don't need all the hypervisor nodes or anything like that. You just need something to run NOVA manage on and a machine hosting the database that approximates your real one. And so, yeah, because in the current world that we live in, the world stops when we do these migrations. And so if one of these migrations takes four hours, that might be a big deal in your environment. I'd also be worried about rollback of these migrations. This is a hard problem, right? You basically have two choices with the SQL database upgrades. You can either take a database snapshot and restore from backup when it goes wrong, or you could roll back. Neither of them's great. Like, if you take a database snapshot and the state of your cluster has changed, users have created and deleted instances, what happens to those instances when you restore a backup? But then again, rollbacks aren't great either because some of our migrations throw away data in a way we can't regenerate it. So at one point we were just storing enough data in the database to rehydrate things, but then people say, why is my database much bigger than it really needs to be? So in general, we have these people who do this thing called continuous deployment, which is where they're deploying trunk from Git before it's released. And the way they generally deal with a problem when we accidentally break the world is they roll forwards. If you're using your distro packages or something like that and you're upgrading every six months, this is probably less of an issue because it will have been tested by your distro and you'll have tested it in the lab. But think about how you're gonna handle database migrations. I think about your upgrade strategy in general as well, I guess I kind of alluded to this a second ago. Are you going to continuously deploy? Are you going to update every six months when we do a release? Are you going to wait until, I don't know, you're on an LTS release and five years later are you gonna upgrade to something new because five years is extreme, I think. But if you waited, say, two years, you have to step through four open stack releases. We only support the two most recent. So if you experience a problem in the first two of those, we actually, we don't even have the branch in Git anymore. We've got a tag, but we're not gonna help you. The code is too old. So maybe your distro will, but I'd be really scared of upgrade plans that involved waiting a long time between upgrades. But I also probably wouldn't want to be the first environment to do the six-monthly upgrade either. I'd be waiting for someone like, or one of the vendors like Bluebox, CERN, someone like that to have had to go first. But yeah, just needs some thought, I guess is what I'm saying. Another reality of the world at the moment is most people deploy Libvert as their hypervisor driver. So there's a few specific things I wanna say about Libvert. Libvert is an abstraction library on top of a bunch of different hypervisors. So you can use Nova with Libvert, but using Zen or KVM or whatever. And not all of those hypervisors under Libvert are created equal either. Most people use KVM, which means the Zen code is less tested. So if I was going to deploy Libvert, I would pick KVM. There are other choices with Libvert like LXC for example, but again, not as well tested. Specifically with the Libvert, you need to think about your instance storage options as well. So ignoring like more generally than just Libvert, in Nova there are two different kinds of storage your instance can have. There's instance storage, which exists for the lifetime of the instance and is normally stored on the hypervisor node, but that's not always true. And there's block storage, which is a thing provided by Cinder. And Cinder block storage is called volumes and they last past the lifetime of an instance. So specifically picking on instance storage, the Libvert driver has a plugable mechanism for those. Again, everywhere there's a plugability layer, some things are better tested than other things. So for example, lots of people want to have shared instance storage that they lose the disks on a hypervisor node, they don't lose the instance data. Many people choose to use Ceph for that. Ceph is cool, shared instance storage is cool. Again, it comes with a cost, right? So if you chose to do shared instance storage with expensive NFS appliances, the cost is money, the cost with Ceph is that it's not as well tested as just files on disk. And we have seen some bugs in that recently. We have some people from Red Hat and Inovance working on improving the continuous testing around that. So that should improve sometime really soon too. That's pretty much what I said before. So just be aware of how well tested your instance storage is. So I mentioned this before too, I'm being told I have two minutes. That's why I'm flustered. So the essay blog post form of this talk talks about this a bit more, it gives you some advice on how to evaluate whether or not there's continuous integration testing for your chosen driver, whether or not it's actually currently running. Whether or not there's actually someone paying attention to it when it fails and fixing things. That sort of stuff, how actively developed is your driver, that sort of thing. And I keep talking about this essay form of the talk. I actually posted this this morning, so this is up there now. I'm also sure it's not perfect and not correct because there's only so much time one can spend on something like this and it was already very long. So at some point you have to draw a line in the sand and say, you know, hey, this is complex enough. But I'm also very open to questions. So have a look at it, hopefully it's helpful. If you have questions or think that I'm a raving idiot, feel free to email me or talk to me on Twitter or IRC and hopefully it's useful to people. Do we have any questions? Yes, are we running a microphone? I could run a microphone. I'm self-running. You talked a lot about making sure you choose well tested implementations as an end user, how do you know? Is there something published by OpenStack Foundation that explains the level of testing? Is there some continuous integration system we can look at, that sort of thing? So yes, it's not perfect, nothing in life, as I suppose. That's kind of what I was talking about at the end. So yeah, there's a tool that Joe Gordon wrote that you can check out from Git and run and it will show you whether or not a given CEI bot is commenting recently. Our code review system and CEI system is all publicly available, so you can go and have a look at what's happening. But I think fundamentally talking to other operators is really important. So there's an OpenStack operators mailing list, there's an IRC channel, they have meetups, there's lots of user groups around the world. I'd be asking them, hey, what are your experiences with this or this is my proposed deployment, how painful is my life going to be? And listening to other people's experiences as well. But yes, there is advice online about how to determine if something has reasonable CEI. Yes, and there are active user groups. So specifically here, I think there's one in Auckland and one in Wellington, I've definitely been to an Auckland one. And they're mentioned on, they generally seem to organize themselves on meetup.com. But if you do a web search for that sort of thing, I think you'd find reasonable pointers. Anything else or should we go to the next guy who's more interesting? Going, going, gone, thank you.