 Well, we've got one more minute until 1150, but I think we'll get started looks like everybody's settled in here So me talking to you today about techniques for managing your open-stat cloud I want to actually take a quick poll of the room. How many of you guys are developers out there? Guessing a lot of you. Okay. How many of you are our management folks? Okay, good mix of both so of you developers, how many of you guys are really excited about You know digging in and racking and stacking gear and actually setting up open stack Okay, that's a little bit surprising. I have the same response and I'll show you in a minute what I mean by that Exactly. Oh, what about operations? Can I get a show of hands of operations? Okay, you guys are not gonna like my talk. I'm sorry for that in advance So just a little bit of background about myself and where you may know me from I wrote a guest blog On the right scale. Sorry the rack space blog that might happen a few times for this talk Right scales who I work for rack spaces like partner bars and they sound very similar and they both have RS in them That's a great acronym for them both I built an open-stack private cloud in my garage So I did want to geek out a little bit. I wanted to be an off-sky for a minute And put a couple of super micro boxes in my garage. It's actually just off to the left there from my screen and my circular saw is a 67 Cadillac's at endville that is in about a million pieces. So I like to tinker with stuff But for me the real interesting piece of setting this up was not as much getting the gear plugging it in installing Open stack and making it work quite frankly. That was overhead for me. That was You know a good experience. I got familiar with the innards of open stack But it wasn't really what I was trying to focus on so What I'm going to try to convince you guys today is that the best kind of management for your open-stack cloud is actually no management at all Now I can see a few people laughing and thinking what the hell is this guy thinking of course you need to manage your open-stack cloud That's ridiculous So what I'm going to focus on is actually two things and it's kind of first the ops piece, right? Managing your cloud hardware and software getting the things stood up making sure it's healthy making sure that your users can use it And then the other half is managing your cloud enabled applications And I'm guessing for the rest of the developers out there that didn't raise their hand for the ops question of being able to Set up open stack. That's what you're interested in right you want to be able to consume open stack You want to be able to run your applications on it You want to be able to have a dev and staging and QA and production environment that looks all the same that's API accessible That's what I'm going to be talking about today for the most part So to ensure that a little bit the first thing that I want to talk about is the ops piece, right? This is what you have to do to set up open stack You have all the logistics of racking and stacking gear Making sure you have power making sure you have network making sure all of those things are there for you And you install and you configure open stack And as a developer myself, I really appreciate that process, right? It's a lot of this a lot of command line stuff a lot of API accessibility You're making calls like this one or like this one or if like me you ran into trouble during your installation This is obviously not a very useful query But I spent a lot of time looking directly in the database, which is actually really cool, right open stack doesn't have a lot of secrets It's completely transparent You can go spunking in the database and you can rip out endpoints and kill images and do all sorts of really nasty stuff that you probably shouldn't do The next step though after you've got that all up and running and you've got all your gear up and You've got your cloud running and you've got all of your endpoints set up and Keystone has nova registered and glances registered with keystone and you can get images and you can launch instances and all that sort of stuff That's sort of when the fun starts, right? and In this case, I actually did the rack space Alamo installation It actually made it really easy for me because I didn't really want to know too much about how Open stack worked. I didn't have a lot of time to actually put that together So, you know, you've got the horizon dashboard in this case the rack space branded one And that's really great for both admins and for users, right admins can go in there They can set quotas they can add new users with specific permissions, right starts to abstract you away from The API from the command line tools from those sorts of things that are you know the overhead to actually running this thing? So again, it's cool That is what I'm going to try to do today is to convince you that the best kind of management for your open stack cloud is Absolutely, no management at all What do I mean by that? I know you guys think I'm insane So what we really want to focus on is this right? It's the application that you're running on open stack That's what's important. That's the thing that you're building the cloud for that's the thing that you're buying all that Gear for putting it in your data center It's to run your application and if you're doing it, right? Your application is actually set up to know that it's running in an environment where it can request additional resources through an API That it can health check itself that it knows that it's running in this place where it has all the benefits that you want From a cloud enabled application and from that environment So what I do is spend a bit of time talking about how right scale really does those pieces So the first piece of that is something that we call a server template So your application you've got no sequel databases. You have Relational databases. You've got your application tier. You've got You know batch workers. You've got all these things that make up your application What a server template is is that DNA or that description of what that thing is so for your database server, let's say What we have is actually a base image That actually runs on a variety of clouds open stack being one of them We've run on racks based public cloud, which is open stack base you can run your own private cloud and we've got an image for that that runs on KVM and It's the same known image everywhere, right? And we start out with very very little It's just sent OS or a boon to or you know pick the flavor of Linux that you like It's just that OS It's a specific patch level and it's an agent that we call right link. That's it. That's all that's in the image Obviously, that's not a lot, right? That's what you can probably get by launching Something in the horizon dashboard and by creating your own images and putting up the up there So the next step is to actually run these boot scripts that happen at startup So what this means is for your database server You've got this base sent to us image then we have configuration management code that runs and installs the mysql packages and sets your administrator password possibly pulls your database schema from Recent backup does all the magic that you need to just set this thing up There's also operational tasks that you can perform. So again for a database server We have in our base server template for my sql and we also support Microsoft sql and postgres and a few others There's the ability to set up a master slave relationship So we've got a little operational script that you can click on in the dashboard and you can run and you can end up with Those servers getting set up in that master slave arrangement And I'll talk about this a little bit more. I've actually got a demo which will Hopefully consume the majority of the time here I know they put me right before launch, which was fantastic. So I won't keep you from that So the other piece and this is maybe the more interesting one and where the concept of really managing your open stack installation in your Open stack cloud really starts to Fit into this conversation is that rice skill also gives you a bunch of tools for actually keeping track of what's happening with your application now I don't mean You know these sorts of monitoring metrics on your KVM hypervisor or on your Zen server hypervisor or on your Nova node Those are all things that are we're kind of just assuming those are taken care of right? That's that's operations job All right, we want to have this cloud up and running We trust that ops will keep it going but our application running on top of it has certain needs We know that if we're running, you know an API Bunch of workers for an API We need to make sure that those workers, you know can handle the number of requests that are coming in if they can't we need to add more workers We need to know that if a certain server a certain virtual machine is Running on a physical node that's having some sort of problem. There's network connectivity issues There's a bad stick a ram in it something like that Hopefully ops knows about that but if not we actually have visibility into that from the Activity and the sort of symptoms that you're going to see on the VM right skill can see that and rather than Submitting a ticket with ops with it and saying, you know help. There's a problem with memory on this instance You've got an API you call an API get a new instance We use that server template to configure it to be another API worker and you're good to go, right? You don't have to wait You don't have to ask ops. You don't have to wait for new hardware. That's the purpose of the cloud, right? It's that agility It's the ability to make those API calls and get resources as you need them Right skill gives you the ability to look at what's actually happening with your app so that you can make those decisions Because you've got an email or we can automate those decisions for you So I'll show you this in the demo as well and then I also mentioned the operational stuff being able to click on scripts that Will do those operational tasks and they're well-defined. So I Actually only have like two more slides after this one Going a little bit quicker through this than I expected, but that's all right I'm gonna go through and and do a little bit of a demo of the right skill dashboard Is that good for font size we'll get a little bit bigger here Let's do that as well not that Okay, so what you're looking at is the the right scale dashboard So what we're gonna focus on right is the idea of that application Centricity so if you look on the right hand side here We've got up at the top a list of a bunch of applications that I have running in this case. It's in a mix of rack space Public clouds and an open stack private cloud that we're running at right scale as well What I want to do is focus on this PHP 3 tier application. I'm gonna click on it and Hope that the conference why right nobody tweet or anything while I'm doing this, okay? Hope that the Wi-Fi is good for me. So this is what we're looking at This is a PHP 3 tier application. We've got up at the top the database server, right? And that's based on a server template. I'll show you a little bit more about that we're running in this case software load balancers HAProxy and Apache Apache actually in front of HAProxy so we can do things like Pre-writes and all that really fun magic that HAProxy is not so great at We chose to do load balancers rather than maybe software load balancers instead of maybe the You know f5 load balancer that you've got in your data center or if you're using a public cloud something like cloud load balancers with rack space or something else because This gives you the portability to run exactly the same thing anywhere So maybe in my data center in my private cloud I do have an f5 in the rack and I can use that for load balancing and that's awesome But if all of a sudden I have a problem in my data center Or I want to run the same application in your data center in a regional office something like that Maybe you couldn't afford an f5 for that regional office So by using a software load balancer gives you the opportunity to run the same thing on commodity hardware with your cloud, right? We focus a lot on that sort of thing at right scale being able to run the same sort of stuff everywhere There's certainly more performant ways to do it if you have an f5 use it. It's kick-ass But if you don't or you think that you're gonna need to move your application around this is the better way to go and Then down at the bottom I've talked about the load balancer talked about the database server. We've got two running PHP app server So this is a simple three-tier You know it's running, you know, let's call it maybe wordpress or Drupal or whatever you like, right? What I want to do is take a moment and actually take a look at the server template real quick and show you kind of the guts of that actually, I retract that what I want to show you is That The configuration management that we have on each one of the server templates is used to actually Configure each one of these things right so load balancers end up installing a chip proxy install patch you do all of that magic But they're for the most part general purpose. I told you that we are using a PHP application server server template Well, what if I don't just want to run generic PHP or I want to maybe run Zend or I want to Change my You know memory limit settings or whatever the case may be the way that you're able to do that is actually with inputs in the right scale Dashboard so we make these fairly general purpose and you can actually tweak a lot of the settings By actually changing inputs and probably the the most interesting one here Is going to be the repository that we use which apparently I actually said at the array rather than the deployment level but imagine that Where it says the repository Container the second one up that's in white I can put in my git URL there or my subversion URL wherever my source code for my application is I can set the branch for it and when I put a new server into this deployment. That's a PHP application server It's gonna inherit those settings So when it comes up a script's gonna run that goes in clones that from git checks out a specific branch And it's up and running So I'll show you a little bit more about specifically the app server here because I think that's really the interesting bit Because we put all of these things in the frame of a deployment of your application inside of this thing We're able to do some really cool things in terms of having your application and the actual nodes of that application Be aware of one another so This is the server template for the PHP app server And and this is sort of what it looks like in terms of the scripts that it runs and the operational tasks that you can perform on it The font's kind of small, but you know the first several steps are all housekeeping stuff We set up a swap drive. We set up logging. We set up Some right-scale tools that we're gonna use we make sure that Our Repositories are set up and make sure a patch is installed all these things that are you know expected for a PHP app server And then we start doing things that are really specific to the PHP app server, right? This is that do update code those inputs that I showed you where you specify your repository This script runs grabs that input goes ahead and downloads your code Make sure that it's set up and does all that magic and then the really interesting part Is that we go ahead and we find load balancers that are in the same deployment? So I mentioned that we're running HAProxy based load balancers software load balancers this script uses right scale makes a call out to our API And says hey, I want a list of all of the load balancers that are in my deployment I'm an application server. I need load balance traffic to come to me when it does that it actually this particular script actually Opens up an IP tables firewall rule so that those load balancers can actually send traffic on port 8000 Down to the app servers, right? We do that because we are a multi-cloud company And not every cloud that we support has the concept of security groups has the concept of the things that are starting to really Take off in quantum of being able to do that network isolation so far So we kind of automate that as well if you don't have any way to isolate your network We'll automate IP tables firewall rules on each node so that you have that sort of separation The next thing it does or the next to last thing it does Is it does another routine very similar to that it goes and Identifies all the load balancers and then even cooler it says hey load balancers Go ahead and run the script on yourself that adds me a php server php application server to this particular load balance pool And oh by the way, this is my dynamically assigned private IP address so it all happens Basically, well completely automatically without any user intervention so you get a new application server It's registered with your load balancers. It knows about the database server. It's up and running. It's ready to go So let me take you back to how am I doing on time? Let me take you back to the deployment And I want to show you just one more bit and then we'll open it up for questions and then I'll get you guys off to lunch with plenty of time So I talked about how we monitor and I just kind of clicked down into the load balancer that's running in this particular Deployment for this particular application. So this is what our monitoring looks like on the back end. It's actually pretty simple We use a demon called collectee. It's an open source thing just like open stack. So that's a great great deal there It's been around for a long time and there's a lot of pre-existing plugins for it Out of the box we you know monitor things you would expect like CPU and memory and disk use and network IO and all those sorts of things But on this one because it's a load balancer We've also installed the Apache plug-in so we can do things like look at Apache requests per second There's no traffic on this We can look at making sure that the HAProxy process is running We can check to see let's see all sorts of interesting things like Sessions disabled for HAProxy and you can also customize any of these and create your own So if you write your own application and you want to know about daily active users or you want to know about the number of times that Users open a particular container in your game, right? You can expose those things and monitor them and put them in the right scale dashboard, right? And then you can use any one of those monitoring metrics to do the automation All right, so this is all of the alerts We call them that are tied to this particular load balancer And you can see in this column these are all actually defined by the server template So these are common things we know that a load balancer is going to need We know it's supposed to be running Apache so we make sure it's running Apache and if it's not we do something about it You can also add your own here. We could actually add one for Apache requests per second and if it reaches a certain threshold we can do something about that We can launch another load balancer and register it with DNS We can decide that that means that we need more application servers We can decide that it means that we need to flush the cash on our database server So performs better or not flush the cash that would have the opposite effect But the point is you can make those sorts of intelligent decisions based on the monitoring information, right? So The the things that you can actually do from here is is obviously sending an email. I've talked about that a few times you can Run a script on the server that has the problem or on any other server that's in the same deployment And then this is also how we drive auto scaling so we can Automatically decide on based on certain thresholds to add or remove servers from a deployment So I'll stop for a second and ask if there's any questions because I don't have a a lot more prepared Yes Yep, good question. So the question is when I when I say by servers Do I mean virtual machines or the physical nodes that are running the hypervisor site? Unfortunately the terms end up being very interchangeable Specifically in this case. I'm referring to virtual machines. So this is all a very virtual machine focused application focused view So quite frankly, we don't care that much about the physical nodes That's that's Obstrov Yeah, it is it's application management. It's not open-stack management. Maybe I maybe I chose the wrong point to drive home So and so the question from the audience is why is why is this not management? It is it's management at a specific layer It's a management of the application layer The reason that I really was focusing on you know the best management for your open-stack cloud is no management It is no management at all is that from a developer perspective you expect the cloud to be there You expect the cloud to be managed you expect it to be up you expect it to be working What you're interested in in most cases is managing your application managing your workload running? Yes? Absolutely, so the question is Can you build different environments with for dev qa? production etc And have different properties for those so that's actually precisely what we do right So that concept of the server template and the deployment a deployment you can have an identical deployment for production for dev for staging and they all use the same Configuration management the same base image all of the same components to create that so if I click on my php3 tier here Which is loading it may take forever There's actually a clone button up at the top So I can take this say this is my product environment And I'm a developer that needs to test an issue that is occurring I'll get to your question in just a second make sure to raise your hand again, so I can pick you out And you need to test an issue that's happening in production You can clone this whole environment and it's going to take with it all of those inputs that you set at the deployment level Everything right this is exactly production and if you launch the database server and run that operational script It's going to pull in the latest Backup from your production database and you can actually operate on that if you wanted to right? So you have the bit of flexibility to set up that same thing There's no more well. I'm going to promote my code to production But my dev environment and my QA environment was a little bit different than its production. It is exactly the same configuration Let's take another question. Yeah Yes So there's there's ways to mitigate that the question is if we considered the the risk of Denial of service attacks so there are For a raise specifically, which is the part that we usually autoscale one that we add additional nodes to You can actually set a minimum and maximum right so At minimum I know that I need two servers all the time at maximum I don't ever want to run more than a hundred because my app doesn't scale that far or that's probably a problem Right so there's controls for that as well The other thing that you get is that you can set up alerts for all of these things So if you've got a bunch of servers that are a bunch of instances virtual machines that are coming up As a result of something at denial service attack or maybe better a success disaster You have visibility into that you can start getting emails and you can go look at and go Is this a problem is someone trying to you know did you toss me or is someone? You know did marketing do a really good job and driving a bunch traffic to my side There's a question back. Yeah Yeah, so the the question I'm going to reiterate it just to make sure I understand it is if we have a way to Install that agent on Linux operating systems and kind of give you the same monitoring tools Even if it's not necessarily launched from OpenStack or a cloud is that the question? Right so for also for monitoring the billing pieces as well. Yeah So the short answer is today. No, we don't We actually make the the assumption that instances that we manage are ones that we've provisioned through an API And that usually means through Cloud Stack OpenStack a public cloud something like that We're definitely looking at ways that we can provide that sort of functionality and feature set Because there are a lot of folks that are asking for people that have a mixed environment of legacy and bare metal on Hardware that they really kind of want to have some of these tools for So we're exploring it when I really have great answers for it just yet Any other questions? Yes So the question is the the scripts that I showed you are those out of the box scripts or Also, can you customize them etc? So the ones that I showed you are actually pre-built server templates that we have in our marketplace today So when I set up my OpenStack cloud in my garage once I got the Configuration and hardware and everything else figured out I registered that OpenStack cloud with right scale and I imported those server templates And I ran that same PHP 3.2 that you saw on my own cloud in like about an hour, right? So there are pre-built stuff that you can import you can also the same as I cloned that Deployment and it kept all the configuration of the deployment and then I could change things if I wanted to you can do the same with server templates So I can take that whole server template that definition of a PHP application server clone it and then make changes to it You know, maybe I don't want to Excuse me. Maybe I don't want to you know install certain components So I take those out. Maybe I have a specific proprietary third-party binary that I need to install I can write a script that does that and the scripting language is Basically any executable that the underlying operating system can understand so it's bash, Perl, Python, Ruby, whatever, right? And then we also have a really tight integration with Chef as well. I saw another hand over here does yeah Yeah, so the question is about the alerting metrics The expression language kind of look like it was you know if value is equal to less than greater than etc And the question is how flexible is that? Right now it's a per metric decision, right? So you can't say if this metric and this metric meet these criteria then do this For the most part that works just fine Especially as it pertains to health checking and making sure that you have sufficient capacity for an application So we've had good success with that There are certainly scenarios where you would want a little bit more intelligent decision-making And in that case The answer really is our API We expose all that monitoring data and all that alert data through our API And you can actually use that to make a slightly more informed decision and then call our API again to you know Effect change on your applications We're actually working on some really cool things around that as well that I can talk to you about after the session if you're interested Any other questions? They cut my slides off. I do have a QR code for you to scan if you're interested, but That's it. All right. Thanks everyone