 Okay guys, so we're here to discuss DevOps on OpenStack. So we got a distinguished panel here. I'd say one thing, the first question, the first discussion, we'll kind of run through a list, but this is really to stimulate the discussion. So by all means, raise your hand and ask questions. The panel is really here to answer your questions and not mine. We'll start with a quick introduction, we'll start with from here. So Koby, you're the first. Hi, so my name is Koby Ortser and I lead DevOps groups in LivePerson. We are an OpenStack users, started in LivePerson two years ago, and since then building the DevOps team and trying in its work in progress to bring the DevOps culture into LivePerson. I'd like to add that Koby is also the only one that is a customer, and the vendor here on this panel is awesome. Rackspace in a way is also a user. I'm saying Rackspace, even though it's a vendor, it's also a user. So my name is Sebastian, I'm the founder of Scatter, which is an open source project to manage applications that are deployed on the cloud, specifically managing them in a DevOps way that conforms to IT policies for the enterprise. Hello, I'm Marc Lue, I'm the head of the DevOps and Automation Advisory team at Rackspace. This team is international and it covers both Rackspace Private Cloud and DevOps Methodology. So we try to embrace these and tell these stories about DevOps to customers so they can embrace this as well. Hi, I'm Marc Ram. A couple of years ago I worked at a media company and we had a lot of, let's say, DevOps issues, not success stories at the beginning. We would, developers would develop an application, spend a lot of time working on it, and then right before we tried to deploy it we would hit this wall that was ops. And we solved a lot of those problems in ways that were very specific to our own little world and built tools because we needed them. And when I got the chance to come to Canonical and work on Juju, which I thought was a tool that would help lots and lots of people do DevOps better, I jumped at that chance. So. Hi there, I'm Marc Mohamed. I run products at Cloud Scaling. We help customers build private clouds based on OpenStack. And I'm involved with DevOps in two ways. One is that's how we stand up customers infrastructure is using Chef. So we spend a lot of time with that. We wouldn't be in business if we weren't for DevOps. And the second way is we get a lot of customers either coming off of Amazon, so DevOps is the way of life for them. Or we have enterprises coming to us and trying to figure out how to deal with OpenStack. We advise them in terms of how they would use DevOps on OpenStack. Good. So I think we have a lot of good perspective. One problem is that two marks, Marc with a C and Marc with a K are sitting together. So when I'll say Marc, I'll need to somehow say Marc with a C or a K. So Marc, the first question is for you, what is DevOps? A lot of people define DevOps as using this tool or that tool or using this process or that process. But I define it as development and operations working together through the entire life cycle of a project from project initiation all the way to deployment and doing that in a structured and meaningful way. And I could talk a lot about what structured and meaningful means, but probably who here is a developer? Who here is the operations person? Okay, we've got a few operations person, people who are also developers. Because I know them. You know, software teams collaborate and they have tools to do that. They have version control tools, they have IRC, they have ticket trackers. Collaboration and software development teams are not version control tools and ticket trackers. That is the tools that we use to get the job done. And there's DevOps tools and we'll talk about those. But the fundamental thing is getting operations and developers to work together in a meaningful way. Sebastian, I think that you wanted to say something about that. Yeah, I was just trying to find out an angle that was diplomatic here. Yeah, sure, it's about collaboration and all. But I also think there's, so traditionally in the enterprise, there's this wall that Mark talked about and that's very much true. But there's also this other cultural change where developers get operational burden and people in operations have to start developing code. So it's not just the two departments just talking to each other. It's also about a responsibility change in the roles in the different parts of the organization. So, Koby, I think that you had some experience about the difference between culture and tools and how they play together. Do they help each other or is it mostly culture, mostly tools? How was your experience in life first? I want to refer before that, for me as a user, as a customer, DevOps is a little different because we have the operations, the traditional operations, and we have R&D, very two big groups, and the gap between them is very, very big. And you can't take all operations, traditional operations and make DevOps out of them. So there was a very big gap and we needed to talk with the dev, needed to talk with the operations and see how to close this gap because R&D wants their services in production and operations need to understand what R&D wants and take it to production. So we needed to bring the tools, the right tools, the orchestration, the automation, the deployment, the provisioning. So we closed the gap with the tools and with the ability to speak dev and operations. So is that mostly culture or mostly tools in your view? Does it work? Yeah, there you go. So I would say that DevOps is both about culture and tools, but for me the important part is culture. So everyone thinks it's about tooling, makes things faster by tooling. But if you don't break that wall, Sebastian said, it's very important that it's a culture change. So what you're doing here is that you're getting the best abilities of software, lifecycle and software development that have been practiced during the years and you apply that to operations as well. And then you get the best practice of operations. So it's analytics and load capacity and all these things and you pass all this data to the developers because this way developers don't feel the disconnection of their code. They don't feel like they're not responsible anymore of their code unless it's in production. So they feel the continuous lifecycle of their codes and they can see where the code is failing and where the code is suffering in order to make it better. So how much the developers and the operations are using the same tools, for example, thinking about breaking the wall? Well, so, I mean, earlier I said it's all about culture and I think that's true, but software development teams or software development collaborations are all about culture. I mean, software development projects are cultural entities, but I cannot imagine going back to a world where we developed Linux by passing around patches on mailing lists. Version control and Git has changed the way that Linux software development worked. It changed the culture. It fundamentally reshaped the way that we think about software development collaboration and I think deployment automation, while not absolutely necessary, is a fundamental tool of development operations. If things are not automated and if the operations and development teams are not collaborating around that automation, then there's a problem. And I think the next step is equally important if there are not analytics about the wellness and functionality of code. If, for instance, you're trying to get as far as continuous delivery and pushing multiple times a day, you have to have metrics. You have to have tools that say, hey, this is not working. At Sourceforge, we knew within a couple of minutes of pushing code whether we had a decrease in ad click-through rate. We knew within a couple of minutes. And that's not something you've been test for because that is sort of a random user behavior. So basically what you're saying is that culture is important, but without tools it's much harder to change the culture if both groups are using completely different tools. You almost can't. And I think having the right tools is really important and that's why I took a job working on DevOps tools in service orchestration. So how many of the developers in your organization, for example, are using monitoring and the desktop in the same way that they're using monitoring on their production? Speaking of tools. Is it, I mean... You know, so I'm not a developer. And thank goodness I'm not. But we have our Jenkins board on a big 42-inch screen and I go in every day and I look for the green light. And because we deploy about 50 times a day on our infrastructure to make sure the code, right? And so for me, without that, we wouldn't be in business, right? So we use that. The developer uses it. We have a very, very small QA footprint. So everything's about automation and without these tools it's not going to work. So, you know, I think, again, I agree with you. I can't see a world where we go back, where you're passing stuff around. We would never release code. Yeah, so I'll repeat my questions on the tooling side, maybe for Mark and for Sebastian, is that how many of the developers today are using the same tools on the desktop than the one that the operation guys would use in production? Well, what's the gap there? In my world, I run the Juju project. And Juju includes the Juju GUI. The GUI is deployed in production as a service that our ops people run. The charm around that is developed by the GUI team in collaboration with the operations team. They are deploying the charm in LXC containers every day as part of their development process with the exact same scripts that we use to deploy it to production. So that is maybe... And there's other teams that are doing similar things. So maybe half the Juju process itself, if we don't deploy to production, we release as packaged software. And so that has a different set of requirements and is less DevOps-y. Let me tell the story of a talk from one of the operations guys at Twitter. He gave this talk, I think, in 2010. And I think it illustrates really well the DevOps mindset. So at Twitter, they have, you know, teams are at West today for all sorts of different things. And they push code out all the time. And inside the Twitter offices, they have these boards that instrument a whole bunch of metrics. Some of the metrics they look very closely at is the number of error rates, like 500 and things like that. So when... They're able to correlate that to code pushes. So you have this timeline where you have code pushes and you have different events, a new switch was down. There's a whole bunch of things that are correlated on that same timeline. And when they see variations of average memory utilization or average CPU utilization or increased error rates, they're able to correlate that back to a certain event that happened on the development side. And what's interesting is how they tackle that problem of trying to identify what caused that involves a lot of people in the organization and not just the person who pushed the code. It could be something that happened upstream a couple hours ago, but had this effect that kind of snowballed and only appeared two hours later. So it's this collaborative effort around instrumenting everything that the entire organization is doing to be able to pinpoint the root cause of problems. Good. Now there was a point before I asked the audience for questions, there was a point that I think you mentioned about the fact that now come back to OpenStack, the OpenStack angle. How critical is DevOps into OpenStack? And you mentioned, I think, two stories as far as I can. One of them is users who wants to move from Amazon to OpenStack, and the other one is people who moves from VMware to OpenStack. How critical is DevOps in OpenStack? Yeah, so for AWS, that's just the way of the world, right? There was no ops guy. So the developer needed to do everything him or herself. So that's an easy conversation for me to have. We also get a certain number of people from VMware coming, and for lack of a better term, they're asking for open source VMware in OpenStack. And we shut that down very quickly, right? That's not the way to do things. And so we have a message that says you cap and grow. If you've got an existing application that runs well on VMware, keep it there. You want to build your new applications using DevOps on the OpenStack infrastructure. And you can do really cool things with it, right? And you don't want to complicate it, too, because we spent a significant amount of time upfront explaining to people why is it, right? Because ultimately, for them, from a human standpoint, somebody's going to lose a job, right? What used to take 10 people to go do DevOps now can take one, maybe less person to do, or the developer can do it themselves. So it's a very controversial message, but that's progress for you. So basically saying that if you move from VMware to OpenStack you need to change also some of the practices and way of doing things and not just move and expect it to behave like in VMware, right? You're experiencing the same thing yourself, Koby? Because you have the VMware environment and you introduced OpenStack as a way to reduce the VMware cost? I think that as an OpenStack user, it was not possible if we didn't have the DevOps skills. We started with the OpenStack very early in Diablo and it took for us a lot of effort getting into the code, see the problems, see what else needs to be done and they fixed it. And you can't have an OpenStack cloud without all the tools. But you had a VMware environment before, right? We had a VMware environment, but it has nothing to do with it because we decided that we were going to go virtualized as much as possible so we decided to drop the VMware and start with OpenStack. So we didn't have to move things for the VMware because it was in a very small footprint and we started small with the OpenStack, built a confidence app and got where we are today with 1,400 instances. But I think that being a DevOps skills are a must for an OpenStack. Maybe less today because it's becoming more easier and there are more vendors and help that you can get. But still, OpenStack... Why is it critical to be a DevOps in OpenStack? Why is it critical to be in DevOps or have a DevOps skills? Because again, OpenStack alone is not enough. You need to have all the tools for the change configuration, for the automation, for the auto scaling, for the monitoring, okay? So you can't do it without DevOps skills. So, I mean, we've just said, you said DevOps skills like five or six times and what the hell does that mean? Okay, that's a very good question. Like you said at the beginning, DevOps for everyone is something that's how we see it. For me, DevOps is being able, of course, to see and write code but more importantly, come from the operations world. Yeah, so there is a question from the audience first, yeah. So that's an excellent question, how do you build OpenStack? I'll start actually with Marc, because he's from Rockspace and they're building OpenStack at scale. So what are we doing at Rockspace? And this is a good question. The important thing is that OpenStack is such a vibrant project in such a speed that at Rockspace, why we're trying to skip as close to trunk as possible. So for example, right now at Rockspace, we have seven of the OpenStack companies that we're just two weeks behind trunk. And this is fairly important. We have a very, I would say, DevOps team in a way that culturally, there's a different kind of attitudes and methodologies towards that, but at the same time, the tools are very important. So it's both a question of having the right people with the right to link around them. And with that, keeping two weeks behind trunk is very hard. It is not an easy task. So we're trying to do our best on that. It doesn't come without some track devices, but that helps us being as up to speed as possible with some bug fixes and new features possible. Yeah, just let's see if anyone else... I was going to talk about the difference between below the cloud and above the cloud. Obviously, the cloud itself is an application that runs on some servers, and you can think of the developers and the operations of the cloud collaborating to make that happen, make that work. They described a model where they have sort of continuous delivery that's slightly lagging behind trunk. That is the sort of nirvana of DevOps is to have continuous delivery and have it work well, but it's hard. And in some ways, depends on your application, depends on your workload, whether that is worth doing. But I want to talk about above the cloud for a second. I think a lot of interest in cloud and in my experience in enterprises, a lot of developers in enterprises go to Amazon because they can get around operations. It's not DevOps, it's no ops. It's screw the ops. I don't need you. I can do it all myself. That is a different thing. That is not DevOps skills. That is Cowboy. I think most cloud deployments are Cowboy. Whether that's a good thing or a bad thing is yet to be determined. I think there's a place for Cowboys. Did that answer your question? Yeah. One interrupt here just before you answer. How many of you here are building OpenStack or building or consuming cloud? So we'll start with building. Raise your hand. We'll know how to write the questions mostly into that. How many of you are using OpenStack or consumer OpenStack? Obviously all the rest. Okay. Let's continue. I think the fundamental things that are important for above the cloud are important below the cloud. First of all we need to recognize that operations people while they are risk averse and while they have been perceived as a wall and the handoff between development and operations has been painful. That is a structural problem because there are different teams that are very painful. We can just look at Olympic they pass the torch, it falls. I want to have more questions from the audience. I just want to say that there are two fundamental skills in my mind that apply both above and below the cloud and that is repeatable, testable deployment and monitoring and testing of things in a test environment in as close to production as possible of a simulated production environment as close as possible. If you have those two things and you're doing them regularly you're doing DevOps. Yes. Could you repeat the point that you mentioned with the mic? Okay, borrow this one. I just wanted to add that I agree with these two points. You're from Sriram? I work with ThoughtWorks. I'm a continuous delivery company and I consult on continuous delivery. I just wanted to add to your two points and say that it's also important that you add to that list infrastructure as code as well because with a view to having self-provision environments because if you don't do that you may have all the deployment automation and all the great monitoring but you still have to go through the regular IT channels to get your environments up. The open-stock in your view is ready for that rolling upgrade of the infrastructure itself. I'm sure that you experienced different... It's a very big chance for us to upgrade the open-stock every time. In fact, when we started with the Essex in production we had to use the Nova Network of course but when we wanted to move to Quantum we had to build the infrastructure from scratch and then migrate from the Essex to Folsom. So what we actually did was build the parallel environment with the new version. We had to buy a parallel environment. We had to build the parallel environment because we did a lot of tweaks and a lot of rocker rounds with the Nova Network to put it inside our Layer 2 infrastructure so we couldn't just upgrade and use Quantum and the network was a very painful thing in Essex. Mark, you're not using Quantum right now in the rack space, right? We're using parts of Quantum. We wrote a plugin for Quantum called MVP which ties with our hardware. The thing for me about Cloud and about how DevOps relates to the Cloud though is that Cloud helps have very high iterations so it commoditizes the server as a service, let's say. Then you stop having the sacred server that you shouldn't touch because otherwise everything rakes and everything becomes repeatable. So anytime you face a problem normally the best case is just to redeploy, redo, get a clean site, what we call solid-state deployment. So be able to create new platforms on demand with these resources that you have available. Got it. I wanted to give Sebastian some time to answer that question as well. Do you have anything to add to that? I wanted to get Kobe to express his opinion on this. Okay, got it. Mark, I'll get to this. I want to get more questions from the audience. Sure. Yes. Get closer because we don't hear you and we'll give you a mic. I think that would be better. Okay, so I have this friend of mine and for lack of a better term he's working in an environment where they're building a product. In a year's time there's going to be shrink-grap products. Is that operations as we have an operational website type of a thing? But he has a boss saying hey, this DevOps thing is cool. Everyone go out and buy the Phoenix project and read it. Okay, now go. Any advice for my friend? Mark, I think that's a question for you, man. Well, right. So we write a product. Well, we have several products. We have Ubuntu. We have an open stack distribution that's a product we release every six months. And we have Juju. That's a product that we release on a regular basis. You can still do the two things that I said are really important. Continuously redeploying the service, continuously testing it. You can do that whether you have customers using it or not. You can do that internally. And having that stuff all automated in a repeatable, buildable way that is something you can do. And then the third thing you can do is pull in operations people to talk about this is something that actually Juju can do better to talk about operations guides. The kind of things that operations people care about over time. Because while we like to think that Juju is a cloud and everything is throw away, sometimes it really doesn't matter if your service goes down. So I just wanted to make sure that there is more tools, not just Juju that does those type of things. I think that So I wasn't even talking about tools. We don't use Juju to do DevOps for Juju. Yeah. Just to make sure that we're not coming here with a conclusion that if you want to do DevOps, do Juju. I do. Sebastian. Maybe a good way of seeing this. I wanted to interject a question to you that I think you mentioned. One is what is the minimum requirements for implementing DevOps? Is there such a thing as minimum requirements to implement DevOps in OpenStack? And if you can also relate to the questions that I think was laid out, if there are between the versions of OpenStack, like the versions, progressions, what are the tools that actually makes OpenStack better? If there are any recommendations to use heat or use some of the new features that are coming with OpenStack that will make DevOps simpler. So if you could talk about those two points, please. I guess the overarching method is how does operations and development talk to each other? Sometimes it can be over Skype, sometimes over IM, but it can also be over an API. And in this case here, when your operations team is providing an API to your developers to do their stuff, then that's a method of communicating that's well understood, documented, assuming you do all that stuff. Now if you have an OpenStack cluster where some of the services have a dependency on stuff that does not have an API, then you have this communication issue where for some things you make API calls to get more instances or more storage, but for some other things, then you need to walk down the hall and go to service an hour and file a ticket to get something instantiated. So I just wanted to outline that there's this communication problem. Can you give an example for those? Like you mentioned, I think a few of them before. Let's say that you are not using Cinder for some reason and you're using some network-attached storage to launch an instance, you want to have some network storage to it, and so then you file a ticket in service now and you then wait a few weeks to be able to get that volume that you're then going to be able to attach to your instance. That's very unantial. Any recommendation for the new tools that are just coming with the new releases of OpenStack? Are they making the life like as maybe you want to touch on that about heat or about any of the framework? Yeah. I want to go back to the under the cloud because we've built them. Engineering does the deploy to our four labs every day and we turned the corner because we actually stand up private clouds for customers. We actually had our deployment guys who are effectively ops people use the same engineering tools. The reason why was because it was reduced human error, got it up and running. We have a week to go in and stand up and turn around but it took six months for us to get there. I think how you get people to adopt is really to share to drive it forward but ultimately threat of threat or something which is you're going to have to get your stuff done in a week or you're on your own. In terms of tooling we use Chef. We used Chef three years ago and it works just fine for us. I'm not saying that it's the only tool out there but that's what we use and we're comfortable with it. People are also using automation for applications. The one thing that we see a lot on the developer side is people wanting to build these apps that the nirvana of self-heal. You don't need VMware HA anymore. You have the application do the availability part and then autoscale whether it's private cloud or public. That's when we don't really prescribe what it is that they use but effective requires some sort of modern tooling to make that happen. Before that there is more questions. Let me... Before the question please. I'll just pass the mic while you do that. How many Chef users we have here? Raise your hand. How many puppet users? I think I see more puppet. So I can set it for us in live person. Salt. Salt. How many people using Ansible here? Quite a bit. How many people using salt? Just to complete that. Sure. So puppet for us was a very, very big thing. We talked about the gap between operations and R&D. So puppet was the thing that helped us a lot to bring those two together. Because we introduced puppet and we had to teach our R&D to write in puppet. So today every deployment, every R&D know how to write in puppet and deliver is a new upgrade whatever to operations that also know how to use puppet and deploy with it. So I think you and Sebastian are saying the same thing. I think all of you would agree that tools helps to create that communication between the groups. It's all about the tools. Let's not get too much into that. Yeah. So I was wondering your name again? Your name? I am yours. I was just wondering when we're talking about DevOps how the investment of doing DevOps pays off and what the scale of the project needs to be before it starts paying off and whether that is something that you could build a kind of repeatable process to write. Continuous integration? Well sometimes we have very small projects too right that only require two man weeks of work just to integrate two existing systems and what might still be something that is very important and will be used by a very large user base but in terms of code it's not something that is very complicated or hard to do. That is an important question. I forgot the name of that guy who wrote the book Taste of Taste but he was actually referring to that in his recent blog about the fact that if you don't get your kingdom in order and you try to build DevOps you're actually going to get the emission return and what he was referring for example is testing feedback. So if you don't have for example if the testing for example takes a long time until you actually get the feedback loop then you're doing continuous deployment that's not going to work because then the time it would take you to actually get a feedback for that wouldn't work so you need to get your kingdom in order before you can actually implement that and the parallel that he was using is building a production line where you have all the goods in place but the shipment doesn't actually get and at the end of the day what you get is a lot of warehouse where you need to put all the goods that you're producing. I answered part of it but I think the question is really for the panel here do you want to comment about what do you need to actually implement DevOps as a substrate? I would say that it depends on the size of the team but the sooner you implement DevOps the better right because there's things that you design in the beginning when you're small and you're not too pressured to grow up and those things can carry technical depth so do you guys anybody who doesn't know here what technical depth is? So yeah okay perfect so you need to make sure that you take the right decisions and then you can start implementing the tooling and you can start implementing especially the communication methodologies between your team as the team grows up because as the team grows up normally in all startups the economy pressure grows up as well financial pressure to the results so you have less time to react and this is where DevOps is very good it helps you react faster to these kind of things So is there minimum requirements to start DevOps? What do you need to have in your organization or in your infrastructure to actually start DevOps Mark? I would assume that if we do understand DevOps to be a cultural thing then culture is much harder to change at 50 people than it is at 5 so I would do that as soon as you possibly can because it's only going to get harder unless you're down to sizing. So basically saying bring it until it will come or do it until it will come There are a couple of considerations I think the size of your team like the bigger it is the more established your processes are the more important it is to have started DevOps early if you're you know that it's never going to grow beyond one person working on this application for one week there's no such thing as DevOps because there's no operations there's no handoffs DevOps is how you like handoffs are evil the more handoffs you're going to have the more important DevOps is and then the second thing is it costs time it costs effort it pays back over time if you have a project where you're going to continuously deploy we're going to deploy multiple times that is when it's worth investing in it like sometimes it's worth writing software and writing libraries and doing all that sometimes you just write a bash script and throw it away and then the other thing that is sort of ancillary to all of this is sometimes you want to do DevOps you just want to get a product you want to have someone have solved this problem for you and just use something that works but that is a whole separate story like you don't have to solve every problem you don't have to do DevOps for everything and one of the things that OpenStack I think has not done is become a product it is still very much a set of tools that you have to integrate you have to do this stuff there are people who are trying to make it a product but by default if you just do it yourself you got to do DevOps so Mark how much heat for example in your view is changing a lot of the challenges that I think Mark was the other mark was referring to is that a solution is that a promise that is not yet fulfilled what do you think is that something that if I'm trying to implement DevOps on OpenStack I should start with I would say that one is consequential of the other one DevOps is all about the challenge it's about the velocity of your development how fast can you get your product to the market how early can you get that into the face of your customers so having cloud helps you potentiate that it helps you turn faster and make things faster and don't worry too much about having to order the new server have to wait a couple of weeks to rock it up so you can immediately get all that the render spike so you want to own your base you want to control that but as soon as you have a spike you need to react to that faster than having to get more computing power so should I use heat if I want to implement DevOps or should I use some other tools OpenStack we advocate for all tools we advocate that heat is an amazing tool for orchestration but we also advocate that it's also important to keep the state of the service but we normally recommend is to use heat and on the back end of that use something like Puppet or Chef as well to maintain the state so we need to wrap up so I'll finish with a round question for the panelist about each one of you one recommendation for those who wants to implement DevOps on OpenStack for building OpenStack or for using OpenStack so choose one recommendation and we'll finish with that you've all died you've got to do it now it's the that's how you develop things in the 21st century yeah automate and repeat that's it I would say embrace the culture that your people in your company have and at the same time make it faster so that's the main message use scalar again automation as much as possible and of course auto scaling, auto healing everything without is good we'll finish with the fact that I think with OpenStack and Cloud in general there is no option by implementing DevOps so I think with that we'll end thank you very much guys thank you very much