 So, my name is Keith Bray. I have here with me today my colleague Randall Burt, and we want to talk to you about the experience of running heat at Rackspace for the public cloud. So, it was a little over a year ago that just before the Portland Summit, that Rackspace asked us to establish a software development team with two main missions. The first mission was to get involved in the OpenStack Heat Project, and to try to help further that project by ensuring that it had a native open declarative-based DSL that would allow innovation within the OpenStack ecosystem beyond what existing things state of the art was at the current time. So that was mission number one, just help out and try to make native OpenStack solutions that would allow for further innovation. Goal number two that we were given was to establish a product at Rackspace, one that essentially we would take the heat orchestration service, provide that to our end customers, and allow them to use that as a service running at our public cloud to do orchestration for things that customers want to do. And so, I'm happy to spend some time today with you just sharing kind of what we've learned through that process, some of the feedback we've gotten from customers. There's a couple things to keep in mind though. One of which is we fully recognize that while heat is matured to the point that we felt extremely comfortable running this as a product for our customers, there's still a lot of innovation to be had. We recognize that software configuration, if anyone just caught Steve Baker's talk, that the integration with heat and software config is still coming along and still has room to innovate. There's things like convergence where when you declare the state of what you want your running stack to be, that actually ensuring that the actual running state that's there ends up matching what you've declared. And so we're still pretty early on in that stage. We launched our service on March 12th of this year and it's API available to customers. But we've had some learnings that we've had through other ways that we've exposed orchestration services within Rackspace and so I'll talk a little bit about that. So the other thing to keep in mind besides the fact that there's plenty of innovation still to come in this space is that for us in particular, we're not trying to advocate necessarily with any of the things that we talk about today is things that we are hard set that the heat community needs to do or necessarily implement within the program. But this is feedback that is coming from customers. We're gonna kind of break this talk into sort of two parts. I'm gonna sort of go through things that we've learned from running the service that we've been hearing from customers and sort of feedback from that aspect. And then Randall's gonna go through some of the more technical details of what it's taking to actually operate this service within the largest public cloud, open stack public cloud. So with that, how are our customers actually using orchestration in particular heat like using heat at Rackspace. So one of the ways is we have traditional software vendors that are just trying to sell their application for people to purchase but they wanna sell it in a way that their application can be run in the cloud. And so what they're doing is they're working to create heat templates and then they're either providing their end customers with the template or they're providing their customer with a script or a webpage that allows them to enter the end customer's Rackspace cloud credentials and then they spin up that application for the customer through this script or providing the heat template and giving the customer instructions of how to make the heat API call. At the end of the day, that instantiation of a software application is something that ends up running on the end customer's Rackspace cloud account, not the software vendors Rackspace cloud account. The other way we're seeing it used is by actual SaaS providers who do software hosting as a service. And what they do is they tend to take their sales and ablement and purchase process, develop through their own website that the vendor themselves hosts an ability that when a customer purchases software through their site, that on the back end, the software SaaS vendor has integrated with the heat API at Rackspace to actually instantiate an instance of software running on the software vendors account. When they do that, what they actually provide to the end customer using the software is typically just a URL or an IP address and port number. And it's truly sort of a software as a service offering where it's fully managed and the SaaS provider since the instantiation of the application is on the SaaS provider's account, they handle everything about the details of the infrastructure underneath. And then there's the third bucket which is pretty much everything else. So we've got people that are just spinning up their own WordPress deployments to enterprises that are trying to do more complex applications. They want a repeatable and reliable way to spin up applications over and over again using orchestration. So if we kind of look at that third bucket, I don't have numbers on the slide and that's sort of on purpose. The main point that I want to get across here is that we're seeing a large number of people that are deploying single note applications and whether that's a blogging platform like WordPress or Ghost, that's pretty common. But then we're also seeing a heavy set of customers although to a lesser degree they're deploying what we would call a complex app. And that could be a multi-server. It could still be a simple app from the concept of being WordPress but in a multi-server configuration where they've got load balanced and caching layers and all this type thing or it could be their own custom complex applications. The other thing that we're seeing is a ton of interest from developer types that really just want a repeatable and reliable way to spin up what I'm calling here a framework. So this could be a PHP environment or Ruby on Rails environment. They just want to quickly spin up dev environments and then tear them down. So I'm gonna walk you through a little bit of just some of the feedback we've gotten from customers and please keep in mind this is not stuff that we're saying definitely needs to be in heat but this is just the initial feedback that we've gotten and keep in mind that there are lots of projects that are working to solve this at various layers and we'll talk a little bit about some of that. We've heard over and over again from our customers that they really would like to be able to use heat supervision to other clouds and you may say, okay, well, heat stack's focused on open stack and that's one of the things that makes it good. Sure. I think a couple of the reasons that I think customers at Rackspace are saying this, keeping in mind that we would like, oh, excuse me about the mic, that we would like all of our customers obviously to spin stuff up on our cloud but there's a couple of things, right? Most people don't go all in on one cloud vendor or they may make a company acquisition where they inherit cloud resources from another cloud and so they wanna use a single orchestration layer and have one place that they can go and actually orchestrate spinning up resources on more than one cloud without actually having to hit different endpoints of different cloud vendors. So that's been very interesting to hear that over and over again from customers. This also kind of plays in a little bit to the idea of multi-zone orchestration and I know the heat community's been talking about how to implement this. One other thing we've heard over and over again from our customers is across data centers or across availability zones, they wanna hit a single control plane and orchestrate instantiation of resources, backups, whatever it may be for various use cases across completely different clouds and that also includes the overloaded term hyper cloud but when people are trying to instantiate resources on a public cloud and at the same time they wanna instantiate resources in their own open stack environment that are running privately. So those two sort of play together both provisioning to other clouds and multi-zone orchestration. Another thing that we've heard over and over is about config management. So there've been multiple people in the community that have been working with Steve Baker from Red Hat. I know that there's been contributions from IBM and HP and several other companies working on integration of software orchestration. One of the pieces of feedback that we've really gotten from customers that have used cloud formation and who have used heat is that what they really like to see is a tight integration that is optimized for the technology that they already know and use. So software configuration is a great start and a great foundation. What we're hearing from our customers is they wanna see a little bit across that extra mile and actually optimize an experience for puppet, optimize an experience for chef, Ansible, et cetera. This actually led Rackspace as part of our learning and trying to figure out how and what we contribute back to the community and just testing stuff with our own customers. We actually developed a resource that's out in the public for heat that integrates with Chef Solo. So it's a resource that allows you in your heat template. If you're familiar with chef-like things, you can very quickly and easily within your heat template just declare the URL of where cookbooks are and you can declare the roles and recipes that you want to run on a node in sort of Brookshelf style ways. Things that the community that's familiar with Chef, the stuff that they know and use, they can do with that resource in the heat template and it looks very familiar to them. And we've gotten positive feedback on that. And so we're definitely gonna be working with the community to figure out what aspects of software configuration that remain generic because they're great building blocks versus where is there opportunity to extend to really optimize for a particular customer segment or group around a particular technology that they're familiar with and really wanna stick with. Another one that we've heard over and over again is AWS CF and compatibility. And so for many of you, your first thought is, well, heat already does that. Which is pretty interesting. It does. The challenge that we've heard from a couple different folks is that in order to have a service provider or a private open stack cloud, ingest the CFN format by heat that you then have to run the AWS extensions across some of the other projects, particularly Keystone. And so we've heard over and over again from customers that they'd really like a way to just be able to ingest an AWS CFN template because a lot of customers have already invested in that. And whether that's done through a translator that just translates the hot, the heat orchestration template format or not, I don't think they really care too much, but they would really like a way for that to come in. And for service providers, whether it's the private cloud service provider or public cloud service provider, having to run all those AWS extensions and ensure that specific agents and things that are on the box that implement AWS style things has been a little bit of a challenge for us. So another thing that we've heard around this, I'm gonna mention Tosca real quick. So we've talked to some of our customers about Tosca and I am a big fan of the work that IBM and others are doing to bring Tosca support to heat and help align those efforts. I think that's awesome. And I think it's gonna help a lot of folks. The feedback that I would say that I hear when I sit in customer meetings over and over again though, is that they're declaring that basically look, a standard is what people use, right? It's not something that you declare. And for their point of view, AWS CFN is sort of a standard that a lot of folks have already adopted. So again, when I tell them, well, look, you know, that's proprietary standard. We can't really innovate beyond what's already offered. They get it and that makes sense to them. So they're saying, okay, we'll give me a way to adjust my CFN template, but then translate that to hot so that when I need to, I can then take that translation and then I can start innovating and using these features and things that you're telling me about that I don't get with CFN, but give me a way to get there. Another thing that we're hearing over and over from customers is they want us as a public cloud vendor to be pretty opinionated about how they should do things, in particular, how they should deploy specific applications and how they should use orchestration. And most of our customers that I've been interfacing with and again, keeping in mind, we've only launched this API since March. So, you know, the amount of customer feedback, I'm sure there will be more to come. They want us to get very opinionated. And this actually led us at Rackspace to work with the community on figuring out how within Glantz or somewhere in the OpenStack projects that we could develop an application catalog. We're also working with the folks at Moranis to think about what that means with their concept of Morano. Well, so we decided to trial this with customers and we wanted to get something to market very, very quickly. So we ended up, and Randall's gonna talk more about this later, but we've essentially created sort of a proof of concept catalog that our customers can access, which gives them access to the Rackspace recommended way to deploy certain applications because what we're finding is that they want to quickly try it and start out and then they'll dive under the hood and look at the template and learn the format and the syntax as they go. But the first thing they want is, all right, just give me an example, let me push the go button and get this up and running and then dive under the hood. So that also brings me to confusion that a lot of customers have had when we talk to them about heat and orchestration about what is deployment versus provisioning and how does ALM fit into that? And so with the Rackspace getting opinionated, that's what they want from us and our ability to kind of show them best recommended practices and the customer being confused about, well, what does software config within heat? And at the end of the day, we just wanted to deploy our application. There's an initiative within the Ops stack community for Project Cutsolom that wants to take application code and convert that to a managed running app. Well, one of the sort of insights that we've had at Rackspace is when we think about sort of a Heroku-like system where you can go from app code to running app. That Project Cutsolom, you may have heard Adrian Otto earlier today say that we don't tend to try to characterize the Project Cutsolom as a pass itself. Instead, we tend to think of it as a framework by which you can enable a pass. Well, one of the things we hear over and over from customers is that as you have an application deployment pipeline, there's many different steps that need to happen. Everything from get the code out of GitHub, test it, build it. Maybe you want to create an artifact out of that like a Docker container. Maybe you want to deploy that environment. Maybe you need to run some additional tests, et cetera. Is that customers are getting confused about heat and what provisioning is versus deployments. And whether they can or cannot use heat and heat update calls to actually manage the life cycle of their application. One of the goals and visions that Rackspace has for Project Coulom is that we hear over and over again from customers coming from the likes of Heroku or other systems that end up coming to Rackspace is that they hit a point where they want to be able to open up the covers and look underneath, see what's there and then start tweaking and configuring the various parts of that deployment pipeline to meet their specific needs and to have more control over that process. So one of my sort of intuitions or feedback that we've gotten from customers is that with heat, if we can get sort of a, not more opinionated, but if we can help enable some of that deployment pipeline pieces that projects like Solom and Murano and other projects that are evolving. I would really like to see that done in a way that we make it really easy that a customer can go from the, I don't see anything, it's all black box. I just give you application code and I get a running app to then opening that box up and being able to see and understand the pieces underneath. And with that becomes making application deployment really, really easy. And whether that belongs in heat or somewhere else, I don't really know. But one of the things I get asked over and over again from customers is using heat, how can I make my application deployment really easy in terms of revving to my next version of my app? Does that belong in heat? Not sure, I think that's a great topic for discussion if we have time after. Anyway, those are some of our learnings. With that, I would like to hand things over to Randall to talk a bit about what it's actually taken to operationalize the heat service. Cool, thanks Heath. So yeah, I'd just like to preface a lot of this with, this is probably not gonna be very surprising to a lot of folks, but that's also kind of the lesson I think in the end. So the first thing I wanted to talk about is basically our CICD pipeline and how we leverage that and why we leverage it. We're told over and over again, it's always beat into our heads all the time, automate all the things. Well, yeah, that's true, but in our particular case, we've had some very, very strong motivations to actually stick to that very strenuously and invest a lot of time and a lot of effort into some fairly sophisticated automation. At its core, it's nothing, again, terribly surprising. We use Jenkins to kind of drive our build package, deploy pipeline, but we also trigger those builds in several different ways. First of all, we're actually hooked in to the OpenStack, Zool, Garretz, Build, Deploy, Verify pipeline so that when Garret actually, when Jenkins actually verifies a patch that was submitted to heat, we actually pull that down, apply all of our special config and stuff we do to get it to run on our cloud, bundle that up and at least run some very preliminary sort of gut-check tests on it. And that's a signal to us because everybody in our team are also contributors to heat to say, hey, here's a patch you might wanna look at or may have some impact on a config is changing or somebody moves something and you need to be aware of it or oh, look, there's a bug that used to be tolerated in your config file that's no longer tolerated and you were just kind of getting away with it unexpectedly. And it gives us the opportunity then to either correct our configuration or what we're doing in our builds or say, oh, by the way, this breaks heat under these conditions and we can kind of address the patch or file a bug. The other side of that is that anytime that Garret says, oh, this patch has been merged, we actually run through our entire CI CD pipeline, we pull heat down, stock code from upstream, we make adjustments in our own configuration file for our particular environments, apply whatever sort of small patches that we need to do to account for some differences in rack space and some of the ways that we do stuff. And we do that entire build, we run all of, of course, the unit tests and everything and then we actually step it all the way through automatically through our development testing and pre-production life cycle with various levels of testing along the way. So that at any time we kind of know, all right, well, okay, so the latest version of heat is good. If we wanted to, we could actually flip this on and release it if it's including a feature that we're really wanting to get out the door. So this all sounds well good, but that's kind of, the point is the motivation behind that in that heat has, especially in the time we've been watching it, is evolving tremendously. New features are available faster than we can keep track of them. We've got software config now working and all of these other things and there's such a tremendous amount of contribution to heat. There are things that we don't wanna wait six months to take advantage of. So we've actually kind of invested all this time because we're currently, I guess, we try to stay no less than a week behind master depending on the how the tests go out or any sort of feature we're actually blocked on. So I guess in the end, it's about, there's a huge benefit to us. And in fact, the benefit was so great that it was well worth the time and effort that we put it sunk into, literally automating to this degree. It gives us a lot of confidence in releasing our API, even if we're actually tracking very, very close to the bleeding edge heat. And we don't actually lose our minds trying to patch chase as well. Also, we spend a lot of time and effort in this. We have developers that watch this process and keep our eye on it. It's constantly improving, constantly evolving. So it's actually not, it's not static. We don't just set up the jobs in Jenkins and go. We're constantly coming up with ways to kind of make it more automated, more give us more intelligence so that we can actually spend more time contributing rather than writing herd on our deployment pipeline. For testing and validation, I will say there's like two things that aren't automated and it's like not, so not all the things are automated, but they're very small, like gut check stuff we do right before we flip production. But everything else is automated up to that point, including, again, graduating it through the various environments. We actually, this is something again, I keep meaning to try to contribute to upstream tempest test, but we actually have this template we wrote called the kitchen sink. And it has in it every single resource that we support in Rackspace Cloud, all connected together in very ridiculous but non-trivial ways. And we push that out to our endpoints on a regular basis saying, hey, is everything still working? Do we have downstream failures in Nova or Sender or anything like that? So that's kind of our canary in the coal mine is our kitchen sink template that says all the things should be working. And we also run select templates that we know are ones that we provide through our catalog or that we know have some resources in them that we've had issues with in the past. We run those constantly against our production environments to make sure that they're still behaving as expected. A little bit about our architecture in general is, again, no giant surprise, I think, to anyone, which, again, I think is also important with some various tweaks. I mean, making things HA are pretty well known and it's pretty basic. I will note that we actually deploy heat on our public cloud. So we kind of dog food our cloud too, as well. Everything is isolated behind our cloud networks. We also manage the entire environment with Chef. We have Galera as our sort of backend database cluster. We also, I mean, we do blue-green deployment, right? So we push stuff up to offline production nodes and validate and test them. And then when everybody's happy, we just flip vips around. So we're no downtime when it's time to do the next upgrade. And we get to, of course, if something goes horribly, horribly wrong, we just flip back and we're back to the old running version. We also deploy heat using the multi-engine feature that the initial pass-out was actually developed by, largely by one of our Rackers, Jason Dunn's, were. And that's worked out really, really well. So it's one of those things that you really need to explore and hammer on for expected workloads when you want to deploy heat at scale. There are better, stronger, faster methods of doing this, I think, on the horizon. Right now, it works pretty well. We have not hit any sort of odd contention or limits around it yet. So that's one of the things that definitely you need to kind of play with and hammer at before you push it out to production. These are, we do have a few little sticky bits. This is more actually a shout-out. Because of our blue-green deployment, we do run a single database, a single cluster for both, for both blue and green. And I just got a shout-out to the heat community that there's so much care and time taken that database changes and migrations do not break anything. It's like never, ever, ever, ever been a problem. So that's just a kudos to the heat community. So Keith had mentioned earlier that we had some requirements that were about application best practices. Heat, or rather, Rackspace has a lot of very experienced DevOps and application architects. And we'd like a way, and customers would like a way for us to kind of codify that knowledge in something that they can consume and repeatedly deploy and rely on. So we started discussing this sort of catalog capability to heat and said, hey, maybe if heat could keep a list of templates. And somebody could like upload them or store them and that way they could be vetted and considered good by an operator. And it was like, well, yeah, the sort of the concept sounds okay, but heat just really doesn't sound like the right place to do that. And after a lot of discussion, we're like, yeah, you're probably right. So we've been in discussion with the glance community to actually implement that sort of stuff into glance. However, in the interim at Rackspace, we still had that requirement, right? And we had to actually deliver on that for some folks. So what we decided to do, and this was again, a very conscious and labor decision, was we said, you know what we're gonna do? We're gonna have a very thin shim over the top of the heat API. It's not going to actually interfere or change or modify the heat API. But all of these extra things that we need to actually provide, we'll just have that service those requests and pass everything else onto heat. And the reason for that too is having had lots and lots and lots and lots of experience running OpenStack code at scale as publicly available services, one of the things we wanted to avoid more than anything else was patch, having to patch upstream code. Now, I'm not saying we don't do that. We do, but it's very, very, very, very minimal and mostly around configuration. And also like, we did not have time to test every single resource that heat provides natively on our cloud. So part of our build deploy process actually kind of turns a bunch of stuff off, right? So when I say patch hell, I mean making very disruptive changes to upstream code and we wanted to avoid that completely. And so that's why we implemented this fusion, shim on top of the heat API. And largely its job is to provide a template catalog so that customers can actually query our orchestration endpoint and say, oh, by the way, give me a list of rack space vetted best practice templates and let me send, you know, and it'll give you a list with IDs and stuff and let me send a call to stack create, but I'm gonna pass it this template ID. Now, that can be troublesome, right? Because that functionality is going to wind up somewhere and hopefully it's going to wind up in glance or it will wind up in glance. And that's kind of one of the, part of the reasons we did it in the shim is because now it would make it very easy for us to win template catalog comes to glance and creating stacks off of glance template IDs comes to heat. We don't have to change much and our APIs don't change and our customer experiences don't change. So, yeah. So those are the sort of the technical or technical-ish basics. Just a little bit about the heat team at Rackspace. We have about six developers that are dedicated to doing heat contributions upstream in some form. At least four of us are dedicated solely to heat. We have a couple of folks that split their time between heat and tooling and CI CD and that kind of stuff. And then on our, we also have two of the heat core reviewers that are on our team as well. We have two QE that are at work tirelessly and they're way overburdened with the things I try to break. But interestingly enough, the entire infrastructure, all of this, the automation, the chef environment, we deploy heat in every region that Rackspace supports so it's all regionalized and that's all managed and run by one DevOps engineer. Runs that every day. So, yeah, so we are a legion of people that make his life hard. And that's it. So Keith, if you guys have questions, there's a mic here in the center of the room or just shout. Not, we can let you go early. If not, oh yes, please. It's Bash and Sherry Pig. Make it random, make sure to repeat the question. Oh, I'm sorry. What tools are we using to do patch merges? And it was the uninspired, the disappointing answer of Bash and Sherry Pig. We were trying to do polls and things like that, but getting specific commits was really problematic. Heat alarming in terms of have we tested it on our cloud? No, I have not. I've used it on DevStack. I haven't used it on our cloud. I haven't tested it yet. You got a question? The question is, is the glance catalog thing going to be in Juneau or is it under consideration for Juneau? Yes, I have a design session to that very end tomorrow. So not yet determined? Not yet determined, but we are intent on at least starting the process and providing some resources to do that development. Cool. Were you getting up or are you going to ask me a question? Okay, were you gonna just make fun of me? Okay. Oh, yeah. The question, Keith, is in the templates we're seeing deployed on average, how many instance, how many servers? Is it big? Is it a lot of little deployments? Yeah, so the number of deployments that we're seeing where in a single template people are spending up hundreds of servers from end customers is smaller. Where that gets interesting is with auto scale. So we have some folks that are not part of any of the team numbers that Randall showed that are actually also working on heat. So we've actually got more folks working on heat. Some of them are working on helping with the auto scale solution. Internally, Rackspace has a public auto scale offering on our website and so we're looking at how to use heat and in those use cases we're testing to the limits of within single templates, spending up hundreds and hundreds of servers. The predominant use case from our end customers today are one to six type servers. It's the smaller within a single template. Oh, yes, sorry. Chef Solo. Yeah, so the question was the Chef Solo resource for heat. Where can you find out more information about that and where is it documented? So right now we've got pull requests that are in review. It's in a Garrett review right now. It's on Garrett and right now we have been trialing that internal within Rackspace and with some select customers with plans to have that actually publicly documented on our version of heat that we're running as a service very, very shortly. It is running on our version of heat. It is. We make that resource available. I don't believe it's in the docs. It'll show up though if you query our heat service and you say what resources you support the Chef Solo resource will pull up in heat sort of self-documenting and that you can ask heat questions about okay, well what things does that resource support? Looks like we've got two minutes so we have time for maybe one more question. Otherwise we'll wrap up and then make sure the other presenters can start doing their thing and Randall and I'll be kind of around the side and out the back if we have more questions. Any one last question in the minute 45 that we have? Nope, okay. Thank you very much. Appreciate it. Thank you.