 Lou, we're going to talk about rolling your own Kubernetes clusters on OpenStack utilizing today. We're going to talk about the background of deployments on Kubernetes, or I should say on OpenStack with. And today we're going to talk about Ansible and Terraform. But first, just want to quick introductions. This is Spencer Smith, the cloud engineer at Selenia on the consulting side. I'm Luke Heidecke. I am an engineer director at Selenia also on the consulting side. Background Selenia, been in existence for about three years. We focus on open infrastructure, including OpenStack, Kubernetes, and other solutions for adopting those technologies within large global 500 enterprises. We have offices in San Francisco, New York, Asia Pacific, including Tokyo, and Seoul, South Korea. We do sort of three prongs work, consulting, training, and a product side called Goldstone. But both Spencer and I work a lot with clients to figure out, during especially initial phases, to figure out the adoption and sort of prove a concept, early pilot phase of some of these technologies. And some of our work within the past year has been helping several clients figure out the space of Kubernetes, figure out how to best deploy Kubernetes on top of their existing infrastructure. And the clients we're really working with right now have very large OpenStack environments in some of them thousands of nodes across the world. And they're looking at how to adopt containers, how to adopt some of these new architectures on top of their existing infrastructure, but also look at sort of the overall playing field of Kubernetes and what the best practices might be for deploying these systems. So today we're gonna look at some of the options for deploying Kubernetes, some of the sort of production quality or quality of these solutions when looking at them in regards to deploying on top of OpenStack. We're gonna look at those available tools and some of the tools that you can use outside of what is within the normal contrib directories within, or contrib repos within Kubernetes. And then we're gonna do a great demo of the Kubernetes deployment on OpenStack that's gonna work flawlessly according to Spencer. So first I'm gonna go ahead and hand it over to Spencer. He's gonna talk a little bit about what he's gonna kick off and kick it off and we'll kind of go from there. Thanks Luke. So is this working? I fell out of my pocket when I first got up here so that was good. Yeah so just to give a quick introduction of what we're gonna do, we're gonna try to launch an end-to-end deployment of Kubernetes on OpenStack. I'm using Terraform to do the provisioning of the infrastructure and then kicking off some ansible playbooks to actually do the rollout of Kubernetes. I've kind of wrapped these things and I'm gonna show you before we start here. If it'll work, I think I've gotta go this way. All right so I've wrapped it up in a shell script. It's drop dead simple, right? It's a Terraform apply. I'm gonna do a really lazy way to wait on CloudNet. I'm gonna sleep for a minute and a half and then I'm going to run the ansible playbooks that we're gonna use. Like I said, we'll go into the actual playbooks. We'll talk about, we'll see the Terraform scripts in a little bit but I'm gonna kick it off now so that we've got some time. It takes about 10 or 12 minutes. All right, Luke. I think you're up. So throughout this conference we kind of, having some client meetings and also some discussions with some of the Kubernetes community members and quite a few of those have been focused on, quite a few of those discussions have been focused on the challenges, the current challenges in deploying Kubernetes on top of OpenStack and really getting a good picture of the basic options and the variety of options that are available within a Kubernetes deployment, including network and storage and things like that. So there's some initial challenges right now and I'll talk about the first of the options. So the first piece is that if you wanted to today figure out, not even looking at OpenStack, if you're just looking to deploy Kubernetes, you have some options of local installs via Docker or Vagrant. You have these turnkey, great turnkey cloud solutions on Google, on Amazon, on Azure that really one button, maybe a few pieces of configuration of how large you want your cluster to be, you can do a one button deploy and get a working Kubernetes environment on top of that cloud provider. Next piece is some of those, the custom cloud solutions on things like AWS on Rackspace and those are also great and a little more customizable. There might be an options between several OSes or a single or multi-master solution and it kind of opens up and the flexibility opens up as you start looking at these pieces. And all these pieces, by the way, are available documented on Kubernetes.io and they're all available within either the mainline or the contrib repose within the Kubernetes Oregon GitHub. The next piece that you can look at is some of these custom bare metal solutions with Fedora, CentOS, Ubuntu and CoreOS that allow you to have a lot more flexibility in how you deploy Kubernetes but that are still fairly opinionated on how you might deploy Kubernetes within that environment and to be honest, I'll go into the next piece but the next piece is scratch and then I'll go into the problem is as you look at these solutions for what's available on OpenStack, you sort of have to cross out some of these things and you really have these highly customized, sometimes very opinionated solutions or you have to build everything from scratch. Especially as clients are looking for the first time at installing and testing out and really seeing what's available within the very powerful platform of Kubernetes, it's not recommended to do it from scratch. You want a solution that is easily deployed so that you don't have to spend the next days or weeks depending on your luck to install Kubernetes for the first time. There's this tool within the Kubernetes repo called CubeUp which is really great and there's some work being done within the community to improve these solutions but it's not quite the OpenStack, I guess, option for deploying Kubernetes is not there yet but there are PolarCrests and if you are interested, I would urge you to try it out, to try the crimp PR and your eHor is also working on and I sure would love some help on a, he works from Rantis and would love some help also on looking at a non-heat solution for deploying OpenStack in sort of a turnkey manner for that initial POC effort. In the future, we're also looking at as a community more production grade installs of Kubernetes with CubeUp but at this time, it's really looking at the POC level. So what do we start doing as we need to look at sort of the beyond the POC phase to a pilot install of Kubernetes on OpenStack that's in a production-like scenario with the technologies that you might want as an enterprise. So things like tying in with Keystone, maybe changing some storage options, adding or not using Swift, depending on what your drivers are and how your networking is within, can you use bridge mode with Newton Networking? If you're using cloud, I should say provider networks, that might not be an option. So there's a lot of different questions you have to start asking and the current solutions just aren't there. So as a community working a lot, highly recommend you take a look at those but for our clients, none of what was really out there worked perfectly. And we had a lot of fortune with CubeSpray, which allowed us to kind of, and which he'll talk about later, of why that allowed us to give the flexibility of some of these deployment options to our customers with trying to deploy Kubernetes on OpenStack. So some more background about what the challenges are for our customers is that, so the good news is that a lot of people looking at containers, a lot of people wanna look at Kubernetes as their orchestration management platform. So you see a lot of interest in containers, as you probably all know. The unfortunate part is that, especially with our clients running, definitely not Liberty, the Ice House or Kilo, unfortunately, they either can't, won't, or both support Magnum. And looking at the latest survey numbers, you see Magnum on the bottom there, as far as production installs, especially of being able to, there's not a lot of likelihood if you have an existing large enterprise OpenStack installation that you can use Magnum. Also, you look at heat, and not a lot of people either utilize or have chosen to enable heat within their environment, especially some of the older installs to support heat. So there needs to be some solution that doesn't utilize the sort of the best of, the most flexible for the largest sort of audience to be able to at least try out Kubernetes before they decide to start changing their underlying infrastructure. So the next piece I'll hand over to Spencer, but our choice was a sort of a combination of a, taking the existing tool and doing some wrappers around it to kind of give the best flexibility for a customer, and trying to be able to just get their hands on a production-like install of Kubernetes. Yeah, so I mean it's kind of one of the things where, okay, there's not a good option. Our clients still want to include a Kubernetes cluster, right? So we chose to take a look at kind of rolling something ourselves in such a way that we felt it could be supportable, it could be production grade for them. And so we did, like we were talking about, did a little bit of digging around the community to see what was there. And here's kind of what we came up with as, well, what we're showing you today for one, but also it seems to work pretty well. We're using ish, this same stack with some of our clients, and it seemed to deploy pretty well across a pretty large amount of Kubernetes nodes. So we're using Terraform as a way to define our infrastructure. If you guys don't know about Terraform, you may or may not, but it's developed by Haschicorp, the people that do vagrant and all those good tools. It's just a way to define my infrastructure across several different clouds. They've got different providers. So I define my infrastructure against the open stack, and I say, give me this, and it does it. And then we're rolling into Kube Spray cargo. So they just changed the name. So we're having a hard time remembering to call it cargo, but it's Kube Spray, and then the package inside of it, that is the collection of playbooks, it's called cargo. It's all Ansible playbooks. The goal here is to create a production level install, regardless of the host that you're gonna run against, right? We're also using Ubuntu 14.04 for the demo. Kubernetes 1.2, the latest Flano, which I think is 0.55. And then an open stack Liberty Cloud that we've got in our test lab. So that's kind of the stack we're working with here. The Terraform templates, let me make sure. Let's see, okay, yeah, I got everything. The Terraform templates that we're dealing with, there's only two. You guys are gonna find that this is really pretty easy as far as creating the infrastructure, and it's really not that bad. I was kind of impressed by it, but one template simply creates the open stack infrastructure. The other is gonna drop an Ansible inventory file into the right place in the cargo directory. And then we're just gonna launch cargo, and it's gonna do its thing against those hosts. As far as Ansible Playbook options, if you're an Ansible fan, and I should say that we chose Ansible mainly because our clients are already using Ansible. They develop expertise internally, and so we didn't wanna compromise that by giving them yet another way to deploy Kubernetes, right? So as far as Ansible Playbooks is a couple of different ways that we've done it. The first one is the community contrib repos, and so those are under the Kubernetes GitHub repo. We found those to be, they work, but they're very Red Hat specific. Seems like there's mostly Red Hat guys maintaining it. It doesn't work with Ubuntu. It's Single Master, NoHA, all that good stuff. And then there's QSpray cargo. So those are kind of the two that we were able to identify that seemed like they were well supported. They were actively being worked on. There's some community around them. Cargo supports, like I said, it aims to be kind of a prod-ready deployment. Supports multi-master, credent, etcd cluster, all that good stuff. And it supports Ubuntu, supports the rail family of OSs, and will also support CoreOS if you wanna deploy CoreOS with these Playbooks as well. So this is kind of what happens. I'm not sure how good you guys in the back can see, TV's a long way, but so there's two Terraform files. Here I'm just calling them 00 and 01, but 00 is gonna create my infrastructure in our OpenStack environment. It's three VMs, a controller, two nodes. I keep calling it a master, so if I do, sorry, but and then 01, like I said, is gonna take the infrastructure we created. It's gonna create an Ansible inventory file out of that, and then it's gonna drop it into the right place in the cargo directory. So that said, we can see if it worked. I'll be honest, I've been having some issues with it today. And if it doesn't work, it's gonna be a perfect opportunity for us to encourage you guys to get involved with the community. So I will show you the code first though so we can see what's happening. Here is what it looks like to define infrastructure, and that looks probably really small for you guys, inside of Terraform. So there's several variables at the top, number of nodes that I wanna create. I'm doing a single master only for the demo, but number of nodes, internal floating, or internal IPs, floating IP pool, what image I wanna use, the flavor, all your standard OpenStack stuff. An availability zone that is faster than their other one, so I'm not having to specify that guy. So I'm gonna go through, it creates a single, a single floating IP for the master, and I'm doing that by itself just for some naming reasons, but the single master IP, the single master node, I said, use all the variables we defined above in order to create that guy. So then two floating, or two node IPs, it's using the count variable here, you guys can see. So I've specified two in my configs, so it's gonna create two floating IPs for that, and then it will create two nodes for that as well. The variables that we're seeing at the top, they're empty, so I'm requiring you to define those, and like I said, it's pretty straightforward, it's the same thing you would see if you're doing a nova boot. Number of nodes, internal IP pool, all that good stuff. Nothing too crazy. So the second file, the second Terraform file, create inventory, so this one's a little uglier, but it works. I'm using the Terraform local exec provisioner, and so what that's gonna do is it's gonna take the output from the first definition and the creation of those OpenStack resources, and it's gonna put them into an Ansible config file, or an Ansible inventory file that allows Ansible to talk to the proper servers. I know what the file needs to look like, so I'm basically doing some echo hackery here to just write it in where it needs to go, and then it looks like this at the end of it, right? So this is the example one I didn't wanna give you our IPs, really, but so node name, the SSH host, and the IP of how to get to it, this would be what a multi-master inventory would look like, so if I wanted to create two masters, three STD cluster, this is what it looks like. So that said, we are not done. We are still provisioning our Kubernetes environment. It looks like sometimes it tends to take a while. It kind of appears that one of our guys failed here, so it's probably not gonna work anyways. So that said, I can at least show you what it looks like in OpenStack. So it's three nodes, they're named the right way, they've got the floating IPs, the image name we want them to have. Like I said, it's pretty dead simple when it works, but yet here we are. And so let's figure out what comes next. So we'll roll back at the end and see if it works. Like I said, I think it's gonna fail, but we'll double check kind of at the end. So the question you may ask, given it not working especially, is why would you want to deploy this way? And the answer is you wouldn't, don't, unless you have to, right? So there are a few reasons from our clients that we're seeing that they have to deploy this way. It's things like they're running some really old cloud, something Juno earlier, with no upgrade path, even on the horizon. So it's just super old, it's not gonna support it, they can't use something like Magnum, they don't want to install Magnum for whatever reason. Upgrading the clouds in Nightmare, they've written custom network plugins, they've had hired Cisco to write a custom network plugin or something or some other network provider. And the upgrade is just crazy town, right? The other thing is long testing cycles. Yeah, we want to do Magnum, that's fine, but it's gonna take us six months to get it installed on our cloud. So what do you do in the meantime? We still wanna do Kubernetes, we wanna build expertise there. The other last one, and this one is actually surprising when we're common, is clients just an end user of the cloud, right? He has no control over what the cloud admins are gonna install, what they're gonna do, or their cloud admins a big meeting and he won't install it for them. So we gotta figure work arounds, right? And so we chose, like I said, we chose Terraform and Ansible for our clients basically because of familiarity. They're not comfortable using heat, they are comfortable using Terraform, they're guys in the language. Minimizing the roadblocks to deploying Kubernetes is good for them. And so that's where we've been. But beware, right? So there's, as you guys can see, is there are bugs that pop up here. I've been having some flannel issue all day trying to get this thing going. So there's gonna be bugs, be prepared for that, be prepared to track down some really funky issues, be prepared to know what's coming down the pipe from Kubernetes. If things are gonna fundamentally change when you between 1.1 and 1.2 or something like that and you wanna deploy it quickly, know that's coming down the pipe. So one of the things to think about is there's redemption on the horizon here, right? So there's a very active community that's working towards good Kubernetes solutions. We met, when was that? What day was that? Wednesday? I don't know, I'm losing track of days here, but we met with the Kubernetes OpenStack SIG. We had a really great meeting around talking about the Kube-up scripts we were touching on a minute ago. Spent an hour with those guys. It's a good community that's starting to develop around having OpenStack run Kubernetes well. So encourage you guys to become a part of that and reach out there. So, as far as lessons learned that we've had trying to do this for our clients is the community needs a lot of help. Like we're talking about the OpenStack SIG, they're just now getting started. The past month or two, we've just kinda just kicked things off and defined a vision. Lots of code commits to make, right? There's lots of issues in the playbooks that we've used. Lots of things that can, lots of ways for people to contribute. And the project moves quickly, right? So it's, if you don't call in to the Kubernetes Meetup, it seems like it's impossible to keep up with what's coming down the pipe. It's like 1.2 is a ton of huge changes. Deployment API, Ingress API and all that stuff comes in. It's just hard to keep up. So be prepared to listen and keep up with what's going on. The other thing is be prepared to either maintain a fork of the playbooks that you're gonna use or generalize your environment in such a way that you don't have to. So depending on what your environment looks like, it can be very hard to get the playbooks to work and you wind up hacking a bunch of crap together. And then your task was supporting that, right? So pick your battles, I guess. If you want the newest stuff, the newest playbooks, the newest Kubernetes deployment methods, try to keep your environment general enough to support that. And then lastly, you know, the things we're talking about today may not apply six months from now, right? So talking again about a fast-moving environment, if Magnum all of a sudden is deploying production-ready clusters, it no longer makes sense to try to do it some other way unless you have a really good reason. There's some work going on around that as far as defining some reference architectures. What a production-ready Kubernetes cluster looks like on OpenStack, which is another way to contribute back. We'd love to hear more about what people are running internally and what's working for them. And so that said, Luke, let's talk community. Do you wanna check on the... I can check, but I'm telling you it's not gonna work. Miracles happen. You can do some live troubleshooting too, right? Yeah, we're still running. Okay. So one of the wrap-up with a little bit of just idea of the community resources available, a little bit more on the community. The ideal is that, especially during an initial POC, initial, especially pilot-based stuff, that a community-supported resource like Kube-Up is available and is a seamless experience for users of whether it's Amazon, Google, or OpenStack or Azure, that you're able to try it out to see a working instance of Kubernetes and then hopefully that there's also some level customization available. There's a lot of work being done currently on that and also on the reference architectures for what you would want to include in both something like the Kubernetes maintained Kube-Up, but also what does a production Kubernetes cluster look like on OpenStack so that projects like Magnum or Murano can utilize those architectures and that people in a generic community sense are happy with what they end up with and that you actually utilize those tools. So to get help as you start looking at these things, the Kubernetes Slack channel at slack.k8s.io is a great resource. I'm on there a lot. Spencer, there's a lot of other great people that are looking at Slack channel and monitoring it. There's a specific one for the SIG OpenStack group that is monitored for specific OpenStack integration issues for Kubernetes. The next piece is the Kubernetes community hangouts. The Sarah from Google has a great calendar on the community site that lists all of the various community hangouts, upcoming events and things like that, that Google and other people will be talking about Kubernetes at. And then there's also the Contrib modules and open source from Kube Spray that are available for there. Speaking of SIG, the next meeting is May 3rd, every other Tuesday at 1 p.m. Pacific, there's a Google hangout or I think, yes, still hangout. And you think, yeah. If you go to the Google groups, you can sign up for calendar updates and things like that. And then if you're in Southern California in the LA area in El Segundo, AT&T Entertainment is hosting at their space right near the airport, a meetup for where Google's gonna talk about continuous delivery using Jenkins on Kubernetes. We're gonna talk about another one of our installs using with the target install of Kubernetes in the hundreds or thousands of nodes, so large scale Kubernetes clusters on OpenStack and some of the challenges of the current challenges of getting that done for a client. And then the third speaker from Amgen is gonna talk about the challenges and lessons learned of integrating a large scale pass for the enterprise there. And I believe there's going to be food and things like that also provided by Apprenda. So that should be a fun time and free food and beer. So it's always a good call for me to go to anything. Definitely give a call, give a reach out on Twitter. We're always hiring and join us in Los Angeles for sure. We'll be there, guys from quite a lot of the different Kubernetes user groups within the LA area will also be there to kind of chat and should be some good stuff there. Do you wanna take one more look? We can see right now. If we can see the output product there. It's false hope, man. I'm full of it, yeah. Yeah, it's flannel all day long. It's been crazy town. All right, yeah. So anyway, we'll open it up for questions. Anybody got anything to ask? There's mics, by the way, if you don't mind using them. The guy's in the back like it when you do. What's that? What was the problem with flannel earlier today? There's a step where it generates the subnet E and V file and it just hangs for no reason. It's pulling an image from key. It's pulling the flannel image from key and it just sometimes doesn't seem to work and there's a 10 minute timeout in the playbooks and it's just like, I don't know. I haven't figured it out yet. I haven't figured out what's taking so long, why it's breaking but. And then it's like it'll work for a couple of them but not like the third one and so it won't even be dead as far as Ansible's concerned. Thanks a lot, guys. Thanks.