 Yes, okay Hi, everybody. My name is Matt Ray. I'm a senior technical evangelist with ops code This is my I don't know was this the seventh open-stack summit. This is my seventh I'm here to talk to you today about chef for open-stack and The ecosystem we've got going on around that the first thing I'm here to tell you about is why chef is everybody already familiar with chef All right, most of you good enough When you're dealing with chef we're chef is based on the idea of infrastructure's code This is the idea that everything you do with your infrastructure the way you set up your servers the the way your hardware's provision the way the applications get deployed Everything that you do is tracked in source control And you version it you fork it you make pull requests everything in your infrastructure is backed Like any other code base and the the reason you want to do this is because if you have that You can reconstruct your business from source backups and backups of your data and access to new bare metal or a new cloud You know, that's kind of kind of important But the way chef works is it gives you a declarative interface to your infrastructure. You're saying This is what I want. You're not saying how you want it You're not saying apt-get install this this and this you just say hey I want the nova compute package, you know and chef will go off and happily enforce that policy for you You're you're pushing Rather than declaring Chef is written in ruby the recipes that we write are written in ruby because it's a nice third generation language It allows you the ability to interact with other libraries to calculate things dynamically to do things You do with programming languages and extend it if necessary and you get all sorts of great Libraries and all that fun stuff When you write these these recipes you put them in a cookbook cookbook is just a way of sharing infrastructure You know pretty pretty straightforward stuff. It's got you know, the recipes templates custom resources Whatever you need, but these become libraries for your infrastructure that you can reuse and You know version and share on our community site. There are hundreds of cookbooks already available on our community site and Community is very important to ops code, you know, it's very important to chef itself Chef has an Apache license, which means you are free to do pretty much whatever you want just like open stack You know as long as you're not applying patents and You're good to go Because of this we have over 1,300 individuals who contributed code to chef or some of this other Applications in the chef ecosystem. We have over 200 Corporate contributors companies like Dell or dream host HP rackspace and people on the halls and over 900 community cookbooks How what does this have to do with open stack? So we've learned a lot of things in our time in the open stack community, you know, there's Chef has been fairly popular this is This is because chef can tile this infrastructure together automatically we use search rather than Upfront saying how your infrastructure is going to be laid out when you deploy a new node. It says hey, I'm a new compute node Where's the API? You know, I'm a new You know glance where is swift to store my stuff in All these things get tied together automatically. So as you add more infrastructure it automatically reconfigures itself How you when you start to scale up these recipes will automatically account for those changes or they can be written You know to account for different sizes events for structure The components become interchangeable because they're libraries They're functional building blocks if you want a different database. Well, you just use a different database cookbook And this the code is the documentation for your infrastructure tells how everything is put together and since Almost everything is under the Apache license is available for you to do whatever you want So let's take a quick overview of what's actually out there for opens stack Here's a quick list of some of the companies that are using chef And we've been working with to some extent or another, you know, some of these are customers Some of them are community members. I guess they're all community members and some of them are partners So why what is chef for open stack? It's a project About three summits ago I got up gave a similar talk to this and I talked about the great work that Dell was doing the great work that rack Space was doing and dream host and HP and none of them were on the same code base None of them were sharing now them really cared to share I mean they'd like to share but that's not you know HP's businesses that make you sure the rack space moves open Stack effectively So to reduce fragmentation and encourage collaboration, we said let's call let's make a project around this Let's try to encourage collaboration chef for open stack is a project. It's not a product Ops code is not an open stack vendor like, you know piston or an Abula or rack space And everything we do is under the Apache to license. We're trying to get people to deploy open stack it's not secret sauce, you know, there's a lot of documentation gets better all the time and you know, it's Your business's advantage shouldn't be the fact that you can stand up open stack. You know, it's what you do with it afterwards So what is chef for open stack? It's a code repository for deploying open stacks pretty straightforward It's documentation and it's cookbooks cookbooks for the current Seven major components. We'll start adding more stuff eventually and knife open stack And so we'll talk about knife open stack here in a minute There's the wares. We have an IRC channel We have a whole bunch of code on github a whole bunch of forks on github and You know, we have a Google group that is very lightly trafficked and and a Twitter handle So what is there today? We have a chef repo. It's currently for Essex I started working on fulsome and Grizzly came out and Grizzly's done. So I have pull requests to get Grizzly. So I'll probably backfill fulsome Community member told me he wanted fulsome because they have customers who are deploying on on fulsome So he will backport the Grizzly stuff to fulsome Currently it's a boon to 1204. There are plenty of forks out there that are supporting other operating systems We'll start pulling those into the mainline version. It's KVM and LXC. I will be happy to add other stuff It's my sequel. It's still using Nova Network There is a nice era plugin that we wrote that works with quantum on a slightly different code base from AT&T It also integrates into test kitchen, which is our test framework. So I'll talk about that some more in a little bit So tomorrow, you know chef for OpenStack tomorrow in about two weeks We have a development sprint scheduled and we will be working on Grizzly merging in patches from AT&T dream host HubSpot and Rackspace primarily as well as a lot of other community pull requests Those repos are all publicly available So if you want to jump on somebody else's branches or forks right now You can go nuts with it and we'll be documenting more of that about how these pieces all tie together the roadmap looking ahead one of the things that We've we've been really looking at as a community of deployers is is deploying from source The districts are nice, but they tend to have opinions that conflict with configuration management If you get a you know a package and it creates a bunch of users and turns on a bunch of services and writes a bunch of schema Maybe that's not how you set up your day your infrastructure So there will be some threads about building packages from source and once we start doing that Hypervisors one of the forks already has bare metal Nova working for Grizzly We'll be probably merging that just out of the box Hyper-V is a very viable thing to do with chef because we treat windows as a first-class citizen. We have quite a few very large windows deployments out there and Hyper-V requires mixing Linux and windows and chef does that very well Databases a lot of people like Postgres. We're gonna support Postgres as well. It's pretty much a drop-in thing Cinder Ink Tank has good code books for Seth already and we can just drop those in as back-ends for Cinder Quantum meet occur has chef cookbooks for their plug-ins. I'm gonna work with them to make sure that they're well supported in this framework as well Red Hat Debian and Sousa. So Sousa has cookbooks available with the crowbar project for deploying OpenStack Red Hat cookbooks are available through rack spaces repo and Debian has made pull requests into One of the branches has has Debian pull requests as well So if you have a favorite distro, it'll get mainlined eventually And of course documentation all the stuff has to get documented as we've heard some of the other documentation sessions It's a thankless task, but it needs to be done And of course we'll try to support the HA configuration as outlined in the in the HA guide So the ecosystem that's just kind of the goal of mainline chef for OpenStack Which is kind of a lagging indicator of what's going on in the ecosystem, you know I'm I'm chef for OpenStack with like two or three other co-workers and You know the ecosystem is exploring the space, you know, they're they're doing grizzly, you know weeks before it comes out They're doing you know red hat. They're doing Sousa. They're you know They're exploring the space and then we try to merge back in as much of it as we can First up AT&T. I've been working with AT&T a bit All their cookbooks are online on GitHub Hopefully you've heard about AT&T successes. They are running a real production loads in a number of data centers Fully automated infrastructure. They drop Cobbler in into their infrastructure and deploys all the nodes via pixie booting chef has put on all the boxes They're I think they have 15 data centers and with very heterogeneous hardware It's pretty interesting stuff. All the cookbooks are online Over an AT&T cloud They are the primary source for the Folsom work. They have very Well-documented cookbooks that are continuous integration and release frequently They have a open-stack common library that is providing a lot of the core infrastructure That'll get merged into the mainline stuff. They're also the source for sender LVM Netapp and RBD and lots of other support technology cookbooks because the nice thing about OpenStack or the bad thing about OpenStack Is it touches so much technology, you know, so many other things besides open-stack get touched by it So it has a kind of a halo effect on the cookbooks that it touches Dell's crowbar So if you haven't heard of crowbar, it's a hardware provisioning application management platform It's got chef embedded inside it You know it exists over on crowbar github.com Dell and Sousa are actively engaged in it a lot of other community people are working on it They're cookbooks that it uses a lot of the same cookbooks Patches are going back and forth between some of the other branches. It is the likely source of Swift Once I actually get around to writing to pulling in swift cookbooks into, you know, the official chef for OpenStack Dream host all their cookbooks are public. They have been really good about, you know Integrating Seth and what they're doing. They have salameter and quantum cookbooks as well They're the source of the quantum cookbooks that that we're using Nicera we engage with Nicera to write cookbooks for MVP We also have an open v-switch cookbook in progress and it is up on github You know, so if you are currently deploying Nicera on Folsom, that's available for you Rackspace private cloud. This was the original source of the Essex cookbooks that we forked off. I'd worked with Rackspace previously and The cookbooks will be talking about Rackspace's cookbooks in a session on Thursday So if you're looking at the workshop, we'll be going through Rackspace's cookbooks These are all on github nice refrain all open source and You know likely they'll be the source of the red hat support because they're doing a lot of testing with red hat You know chef is front and center and embedded in their product as well So the halo effect that chef Ropenstech has it touches so many other cookbooks You know whether it's rabbit or you know my sequel or other things like that Test Kitchen is our testing framework for testing infrastructure if you have a a cookbook For something say like my sequel, you know what test kitchen will do is you define which platforms you want to test it on? like Ubuntu 10.04 12.04 windows, you know 2008 R3 and It will run VMs for each of the OSes that you specify and whatever tests you have whether they're unit test functional test You know behavioral test. That's what test kitchen does. It has a knife. Oh, it has an open stack runner So you can just specify the open stack instances that you want to run with and so your infrastructure automatically gets tested We'll talk more about that in a little bit Knife racks based in HP borrow a lot of code from knife open stack Hopefully someday we can get those off of separate code bases. That's a different thread Crowbar pixie dust razor. These are our bare metal provisioning tools people use All sorts of different tools to get the hardware up to put chef on it All these get a lot of work and a lot of love and if you're not using open stack You're still probably going to have this problem. You need to solve Arista EOS Arista makes Programmable switches, you know software to find networking the Arista has a chef agent that can run on the actual switches themselves and people are starting to use this in their open stack deployments So no longer. Do you have to manually manage your routers and switches? There are a number of other vendors if you're interested we can talk about that offline Berkshelf librarian spice weasels by Nick these are These are tools in the in the greater chef community that because of their exposure in the open stack community They get a lot more patches just because people are using these to deploy their infrastructure and They're having a bigger effect on the greater chef community So knife open stack, it's great We've gone and deployed open stack or maybe we have you know a vendor for providing open stack for us You know someone like piston and nebula that they have you know an appliance or something like that that is Open step is deploying open stack for you or maybe you just have access to the stack cloud that you don't have anything to do Knife open stack is how you're going to work with a knife is the command line tool for chef It's how we talk to API. It's how we talk to our own API, but it's also how we talk to cloud APIs So what we can do with knife open stack is we can query the the open stack API for things like the flavors the groups the images and the servers that are up on the on the Up on the open stack deployment And once we run a command like knife open stack flavor list we can say hey, these are the flavors that are available to me Then I'll run knife open stack image list, you know get the images that are available to me And then I can go and say knife open stack server create. I can actually contact the open stack API Requested an instance to get spun up Chef will then ssh into the box bootstrap it put the chef agent on it and then run whatever infrastructure you want on top of that So if I need to deploy Hadoop workers if I need to deploy lamp stacks Whatever sort of infrastructure you may be running on top of open stack you do it in one shot You don't have to you know go click in a UI to deploy a node and then come and run another chef command to put Stuff on it. It's all on one Knife open stack server create shows up in the dashboard Go and ssh into the box, you know 30 seconds later your box is up and ready Knife open stack has been tested with all of these It works with Diablo through grizzly I'm trunk it uses the open stack API. So we're not using the EC2 API if you need to use that We have nice knife EC2 And then we've done a pretty good job of adding new features to it It's by far the most complex of the open of the Cloud plugins because open stack supports so many heterogeneous types of environments You know whether you have private networks public networks multiple networks name networks You know if you're running Windows if you're running Linux, we're you know on top of all that And you can bootstrap Windows nodes as well The roadmap for knife open stack not super exciting. We have a bunch of tickets You can go and look at them. You can go and read the documentation Next thing that'll probably drop in the next month is managing floating IPs and networks on quantum So you'll be able to create and destroy Networks and when you boot up a VM you'll be able to say hey, I've got seven networks. I want to connect to these three That's that's the short-term roadmap Probably in time will add support for more open stack features It's kind of a question of what do you want to support? What do you want to have chef do as opposed to what do you want to do? Through the the open stack dashboard, you know, how often are you actually going to create new X? You know whether it's networks volumes, whatever things that you do a lot. We're gonna automate So, yeah, that's great. I can talk to open stack. Why would I move stuff to the cloud? You know, we actually do want to do work. Open stack is not just you know, open stack for the sake of open stack You know the promise of the cloud is things like instant infrastructure unlimited capacity auto scaling No commitment. That's really important and immediate replacement. If something's wrong with the node kill it, you know replace it That's the beauty of So of infrastructure as code is you know exactly everything that's on that node if it's having a problem get rid of it So why open stack it's real open source anyone can play You know, it's real open source because if you might not have noticed there like 2,500 people out there And everyone has a different solution and they're all exploring different parts of the ecosystem. That's a good thing It's a little messy at times, but it gives you a choice It gives you real choice and the features that you want in your cloud You know the features are either achieving parity or way accelerating anything else that's out there That's what's nice about open stack and what's important is When you start dealing with different cloud vendors, you know, I'm not talking specifically about open stack here But most of them have something that's going to tie you to them You know, you're going to have a hard time getting your company off of You know vendor X because all your data is there or you're locked into their proprietary tools So when you start thinking about this you need to understand Maybe I don't want to use all the you know fancy widgets that they have if it's if someday I want to get off of their cloud But if you use an open stack cloud chances are good that there's feature parity across different open stack deployments So chef is good for infrastructure portability. We have plugins for I believe current count is 36 clouds That's most of them And you know not all of them are equally supported But chances are good that you're going to be moving your company your developers are going to be working between different You know hypervisors or different desktops solutions, you know, whether you're on vagrant or vmware Maybe you're using cloud stack. Maybe you're using eucalyptus bare metal your infrastructure is going to go between You know desktop to data center to the cloud You know, hey, we're going you know from one way to the other but really stuff's going every way And it's going to go back and forth And the reason this is important is you want to treat it all the same You don't want to have to care that open stack only does this rack space only does that You know ec2 only does this you want to you know generalize across all of it So you can deploy your application wherever you want And so what you really want to do is go on the fat the path to full automation full infrastructure automation Over on the left is the basics of configuration management You know common automation tasks a little bit of scripting making sure you understand how your machines are set up That's the table stakes for configuration management. Everything should be doing that When you start getting into version control, uh actual, you know Uh And and once you have you know your operating system and the applications The basic services manage you're going to want to start deploying applications on top of it So now your developers are in the same tool chain, you know, there's no reason your developers and your operator should be using different tools Uh, once you have your developers using it, they're already using continuous integration You know, they've been using cruise control for like decades Uh, maybe a decade, but uh once you get to that you want to move to continuous deployment You don't want to just have you know developers drop the code on and it works You want to be able to redeploy everything from soup to nuts from bare metal to the cloud Uh, and you want to get to the point of full infrastructure automation This is A continuous deployment workflow. Uh, this is kind of like how open stack works, you know Code goes in on the left. It goes through some testing. It goes into garret It gets reviewed it gets accepted it gets tested some more gets promoted into mainline It gets you know promoted into staging validated each step and then it shows up in trunk Uh, that's one form of continuous deployment But really this is what you want your business to look like you want your business to Developer has a new feature you want to get it to market as fast as possible You don't want to have to wait for a change window for in for from your developers You want to have a clear documented workflow for new features to go in and source and show up on the web Or you know inside your infrastructure And so what we've been building for a number of customers is a continuous deployment toolchain A single ubiquitous process for building your infrastructure starting at bare metal You know, we're not we're fairly agnostic about what you use to provision your hardware And then you're going to manage the entire technology toolchain with chef Hopefully using get subversion works great mercurial is good Garrett works really well for code reviews. It's what open stack uses And then you know a tool like Jenkins or perhaps You know any any other tool most of them can use chef And so once you have built your workflow You have standardized tooling for your developers to your operations and features get out fast And open stack can be used for your private cloud It can be used for your public cloud It can be used both ends with some virtualization in the middle. It doesn't matter It's all seamless and that's where you really want to get to you want to do real work with open stack So Chef for open stack too long. I didn't listen. It's a project not a product It's an ecosystem of people doing work There are lots of contributors with real deployments and a wide variety of ecosystems Essex is working Folsom works in a lot of branches grizzly works in a lot of branches grizzly will be coming to mainline Folsom will probably get backported and the features that Show up are driven by demand. So if there's something you want Send patches and it'll show up But really the point is we want to do real work with open stack. So, uh, that is I blew through those slides fast So any questions Yes, uh, is that on the wiki? Yeah, we're in the process of of pulling everything on the wiki. Um I'll publish these slides in there was a link to my stuff and ops codes source repos It's all in there. Um, and probably in about, you know, three weeks we'll have, uh I'll make sure that cactus page goes away. I thought it was gone, but apparently it came back Right The the canonical site will become uh docs ops ops code.com. Okay, which one? Okay Uh, that is a totally loaded question that, uh, I get asked every time I give a talk. Um Chef and puppet have different, uh ideas about how your infrastructure works Chef has a very strong model that you know, this is they have a dsl for modeling your infrastructure and You know, they say, you know, this this represents how things are going to work. Um Chef, uh, is based on the idea that, um It's more more like code. I think dan bodie from puppet He and I gave a talk last summit and that was the last question and dan said, you know Puppet is is about a model chef is about programming is about source So we treat your infrastructure as source code, uh, where you know, so do they but They're trying to make you fit into a model of how the infrastructure works. They do this with the dsl We try to be a little more flexible. Sorry a little Um, I think our approach is a little more flexible because uh chances are good Something is going to not fit in that model You're going to need to call an api. You're going to need to calculate something dynamically You're going to need to uh You know Search, you know, that was that used to be a distinguishing feature. I think they recently added that You know, so we can tie things together automatically just by asking the chef server. Where is everything? Puppet has a very strong centralized server that is going to calculate your Dependency tool your dependency graph for you and send that down the wire Chef is decentralized the the central Decentralized the central server is a search engine. It's a search engine So when a node configures itself it pushes data up to the the the server and says here's everything about me And so when the next node comes along and says hey, I need to find the uh, the nova api it says Chef server. Where's the nova api and as opposed to in advance? I say you're going to talk to this nova api Let me calculate that for you The reason this is important is when you start to have large deployments like an open stack deployment could be um You might need to uh as the as you start to scale up that number Anecdotally, I've heard 500 nodes is about where a central where a single puppet master maxes out We had somebody do 10 000 with a single box With their open source chef And facebook's a customer so I could tell you 10 000 is not the highest number That's quick we could talk over drinks or something but I can you know dive into it all day long if you want You know there's a there's a very viable ecosystem around both products um So that that's something that we don't We don't feel that most people who are deploying hardware have already chosen a bare metal deployer um there I could name 10 for you right now, uh and For us to try to impose one I had like five of them in my slides already So I mean for us to try to impose a choice in advance for you seems a little presumptuous because at the end You're going to get your hardware provision and chef is going to be on it and that is a Nasty sticky problem of provisioning hardware So if you like xcat if you like, you know crowbar if you like razor we're going to just try to support all of them well um and The various large public deployments that are out there each one of them has something different You know hp has something dream host has something dull has crowbar rack space has razor AT&T has cobbler. I have pixie dust You know What am I up to seven IBM has xcat Those are all open stack deployments with chef So for us to choose one probably not gonna happen. Yes You have one of those ops code books Uh The sx ones have been up there for a while The the grizzly stuff is in my repos. Yeah, it'll get promoted in about three weeks after it's docked Open stack common. All right. We at the end of the grizzly sprint, we will be on open stack common. No, no, no, this is So, okay quick genealogy here Everyone came together at sx AT&T forked off of rack space and I forked off of rack space You forked off of rack space IBM forked off of rack space And AT&T rewrote some of the core libraries And it's called open stack common as opposed to oasops utils The order and so I am going to switch to that The order of the merge integration is going to be A employee of HubSpot Name Craig Tracy has moved everything to grizzly. He took the sx stuff moved it to grizzly and it works bare metal nova That's what he's doing with it. I'm going to move to that Then I'm going to pull the AT&T stuff in on top of that Then I'm going to pull the dream host Quantum stuff in and then I'm going to compare it against rack space to see if there's anything we're missing Yes, and then he's going to backport it on fulsome and make sure meet occur as well supported Absolutely. Yes, uh if Should we have a Whiteboard should I sign up for one of the un sessions? Okay, I'll try to find Is Fletcher here I'm doing the razor talk too. Yeah, that's thursday 320 I'm doing that with rack space. So it's their cookbooks. I'm I'm giving this talk on top of someone else's cookbooks I worked on those too That's the the beauty of it Yes With heat. So what's interesting about heat is it started as a And this is my biased view As kind of a clone of of cloud formation of aws cloud formation And I didn't find cloud formation particularly interesting, you know, because People who are using chef are usually a little more past that, you know It's simply like a script to stand up a node and we're you know, I like to think we're past that So I didn't pay a lot of attention to what heat was doing And then amazon released ops works, which is based on chef. They're very public about, you know, amazon ops works It's using chef They trumpeted the fact that it had compatibility with, you know, the the ops code chef community Even though they're two releases back So we're working with them to get them up to date That muddies the waters a little bit around heat because heat has also taken off Dramatically and I need to pay more attention to what's going on there because folks like rack space and IBM are both involved in heat a lot They're also both huge chef shops. So I need to I need to follow up more in heat. I don't have a well-formed opinion But I'll be attending a couple of heat sessions Yes Yes You're on question Right Chances are good. You're going to disagree with something that is in the the the packages provided by the district And you're at the whim of the distro to update those packages for you. Um, it's not that we're You know antagonistic with uh with the distros it's just that What we found is people have opinions that differ from the distros and they end up writing cookbooks that undo those You know in the recipes are actually undoing the behavior. They're deleting users and changing the schema and dropping tables and So what we'd rather have is just vanilla packages that put the the you know the actual Artifact, you know the actual, you know binaries and libraries and stuff on the file system and we'll go from there um We'll see how well that progresses. You know, we'll keep supporting the the district packages as well But once we have that model in place, we're also not tied to the distros release schedules So we don't have to wait for you know anybody to update a certain package What if I want to build off of some git branch somewhere because I back ported a vendor brand, you know a vendor patch into my tree I can do that. And so that's kind of the flexibility that You know our user base has been asking for you know, we're not My goal is you know kind of what's funny is is my goal is to support You know the guy who wants to set up open stack in their lap You know, it's like hey, we got an open stack cluster boom. It's up That's kind of what I want to support because that's most I mean that's a lot of that's most users But the people who are doing a lot of the day to day work are people who are deploying Real large deployments of open stack who have very you know, hey, they want to run trunk, you know, they want to run Yeah, they had they have their own patches You know, they have stuff that it hasn't merged upstream yet, but it will So they're building off of all sorts of different branches and forks all the time Chef can support that use case. So I have to support kind of both at the same time, you know, which is why I'm always behind everybody So any I saw some hands and yes so The the process that opscode has Is we since we're in a patchy license project. We do require cla So, you know, I listed it off of you know, 100 1300 company people who who've done the cla 200 some companies After that after that That's where it gets kind of nebulous and hand wavy because there are a bunch of outstanding pull requests that haven't gotten merged quickly We're trying to get more Engineering behind it. We're building up a continuous deployment That continuous integration toolchain around our knife plugins And then once we have all the night all the different cloud plugins working Well, we're going to need to do the same thing with open stack. So we will build up a continuous integration Framework for open stack and then when people start saying well, I needed to support this and this and this I'll go to fries by some more hardware and you know, I'll have a line of you These is the susa chain and this is this and you know, my office will be really hot but um We're working towards better engineering around it. I you know right now it's In my head, I know all the the pull requests that are out there and I've got like 40 tabs open Then I need to merge and we need to have better engineering around it Just that's all I can really do is you know be public about the fact that we're we're slow Because it's like me and a half, you know person But everyone else seems to move fast anyone else I'll be here all week. So if uh, uh, if you have any questions feel free to plug me up Matt at ops code Thanks a lot