 All right, our AV people are all situated and sat down. So thank you all very much for coming to this last session on our second day of OpenStack Summit. We're going to be talking to some really amazing people about your OpenStack DevOps team. And it's really going to be a conversation that we'll start with our panel and some prepared questions about what it is to have a development and or an operations team that works in the OpenStack community when you are part of an actual company. So here is our moderator and panelists. I am Rania Moser. I am a software development manager in the Deploy and Release section at Rack Space Hosting. I have been known to burst into Broadway showtunes whenever a lyric popped into my head. So if I hear cue, I might start singing. And I like to call myself an innovation evangelist. Just started coming up with that idea of how do you go faster, better, more? Next to me here is actually Mike Wilson, the scaling master at Morantis, also known as a systems architect. And he plans on going to Mars Colony one day. Next to him is Tofer White, the ops guy from HP. And I was very interested to know that in some circles, not only is he a very snazzy dresser, but in some circles he is better known for his catering than his technology. And he's actually officially the architect of the public cloud operations and infrastructure at Hewlett Packard. And then next to him is Amy Trong. She is the chief cat herder of Rack Space, also known as the dev manager for cloud databases. So she works extensively with Trove. She was on a TV show. So ask her and she'll tell you what it was. And on the end is Jesse Keating, the chief automator at Rack Space, also known as the senior engineer. And wrote a script to automate his entire job for a week while he was in a training class. He also works remote on my team. So now I'm like, hmm. So I'll leave these up during the course of the presentation. So if you want to jot down people's names and their contact information, save a tree from a business card, that would be great. Just to kind of set the tone and the context of where we are, I've been involved in OpenStack for two years as the SDLC manager for Rack Space before I got into the development management project. I've worked with Nova, Glantz, some with Neutron, as well as OpenStack Infrastructure. Mike here has worked with OpenStack Nova also in Neutron. And in the past has contributed as a developer to the Linux kernel, as well as may they rest in peace, small dead open source projects. We all kind of have those in our history, I believe. Topher was reticent on what his OpenStack history or open source history was, but he has watched HP go through the evolution of using OpenStack from kind of we're going to do our own thing all the way to their very highly participatory attitude that they take now. And so has some great lessons and stories to share from that. Amy really was part of the team that grew the Red Dwarf project, which was the original OpenStack database service concept into Trove, which is now a community in and of itself. And so she's had some great experience there. And then finally, Jesse works in OpenStack Infra Team as well, is a heavy contributor to the Ansible project, which we use for orchestration, and also was on the Fedora project for many years in release engineering and the Anaconda service. So we have some good open source experience up here. And that's really what we're going to talk about. So to set the context beyond the OpenStack involvement or the open source involvement, I'd like to ask each of our panelists, starting with Mike, what development and operations look like within your teams, within your company? Sure. So the places I've been at and involved in OpenStack, DevOps to me means that we have a team that is responsible for the development of the products, new features. And they're also very concerned about the operational problems. They're not divorced from that. They're not able to hand that off. That's also their responsibility. I think naturally, you still have people on that DevOps team that are more Opsie or more Devy. The key thing to me is really the shared responsibility and the team goal of making Ops the end all of it has to work. It can't be a concept that is pretty in a software engineer's mind. It's been interesting watching HP evolve over the last couple of years. And when I started there two years ago, it was a very, very strong culture of the development and the operations of a particular service were together reporting to the same manager. And the last few months, they've made a significant pivot culminating in the release of our new Helion product, in which they really wanted to consolidate the development environment, the developers, to be oriented toward producing shrinkwrap software and take that approach. And for public cloud operations, we are to be one of the premier receivers of that and operating that. So we've made this pivot to say that operations is going to be an organization that runs shrinkwrap software that is produced out of a development organization. And that has left us with some gaps that we're grappling with right now to figure out how we fill in some of those pieces. Little spackle. Little spackle. Little spackle. Maybe a lot of spackle. So I'm going to answer the question a little bit differently. So the question is, how does development and operations look like in our team? And so I'm the software development manager for Cloud Databases at Rackspace. And while I have a team of developers, we work very closely with our operations team. So it's actually two different teams, but it's in the same organization. And I don't think it really, for me, it doesn't really matter as much which team it's on as long as we're actively working together. So our daily stand-ups, we're talking with them every day. We're constantly interacting with them on IRC. So really, I guess I'm not really answering the question of how I define it, but just more of how we do it. And I just think it's just important that we work together as one team, whether you're reporting to a different manager or not. So much like Mike and Amy, our group, we approach development and operations in DevOps not as a position, but as a culture. And while our developers do focus a lot on developing upstream and writing their code upstream, and that's kind of their day job, they also have a role in how that software gets deployed and configured out in our production environment. And they are typically the ones that write the configuration stuff and write that. And we go even a step beyond that. And as we go to do deployments, we often use a hot seat approach and we take a developer from one project or another. We have them sit there and actually hit the buttons to make the code go from one point to another. With a lot of support behind them of people telling them what's going on and why it's doing this, but it helps grow that knowledge and grow that understanding of what's going on in the operational side of what their software is doing. And that just brings everybody to the table with an expressed interest of making it perfect from, or at least good, from the very beginning to the very end. Thank you. I would like to also add that those developers that get in the hot seat, they volunteer. We don't just pick somebody at random and say you have no operational experience. You're now part of that. That's so nice. It really is. It really is. And we have a waiting list, actually, to go do it. So one of the reasons when I propose this panel, and we've talked and just gotten to know each other, a lot of people just met each other yesterday for the first time, was to actually bring together a few companies that are doing this open source development and open source operations to actually talk about. And so we have three great companies, three great examples, Mirantis, Hewlett Packard, and Rackspace. And so as we set the context, and we're gonna spend the next couple of minutes and the next couple of questions, really talking about engagement and how we engage with the OpenStack community within your teams or within your organization. And so this question, and let's start with Tofer, actually. What are some of the ways your team's OpenStack participation manifests itself? Again, this has gone through some transition for us. And I think it's been really interesting to watch HP and specifically from public cloud operations. As operations folks, we're kind of on the end on the receiving side. And the thought about what does contributing and participating in open source mean was really a very little consideration to us. And there's been some things that have come along that have been big, bright flashing signs that said, wait a minute, this is gonna have a lot more impact on us than the code change in Nova that I really don't care about. And what's going on with Triple O and Ironic were real highlights for us to say, wow, this is something where this is gonna have huge impacts on how we do our jobs. We're gonna need to participate. So we've gone through a growth period where we started out, let's go, 14 months, a year and a half ago looking at feeding stories into our development organization and saying, hey guys, here's things you need to think about. Here's what really happens in a data center that you need to anticipate. Through several iterations to the point now where we've got teams, we have one team in our operations organization that is primarily developing tools for us and are looking at how do we drive those into OpenStack and let our voices be known there. And even then into individual operations folks who, the great thing about operations folks is they're always writing tools, they're doing little automation pieces and saying, hey, wait a minute, if you're gonna solve that, let's start thinking about how we solve that in the community together. And so what was once something that was very much at arm's length and saying, just deliver a software, we'll run it, that's fine, has now become something that we really realize the importance of participating in and we're driving that idea of participation down through our entire organization. Awesome, Ms. Amy. So the project that my team is focused on is Trove and Trove is actually a relatively new project in the whole OpenStack community. So just a few years ago it was just a handful of rackers and an HP joined and we've grown the community to now over 15 companies, over 50 contributors to the Ice House release. And so now what we do, what I guess our day-to-day is like is so my team, we have one PtL, so well actually one former PtL, so Michael Bazinite was the Trove PtL and he's handed that off to Akil from HP. And so we have a PtL who's focused full-time on Trove and that's something I want to highlight too, so if you're working with an OpenStack team and you just, everyone needs to be aware that it's an investment. So there's a lot of requirements, a lot of expectations from the community and so you need to make sure that you have the support from your executive teams, from your company to make sure that your team can be successful to focus on the community priorities. So my team, we have one former Trove PtL, two core, in addition to that two core contributors, so three total with the former PtL. My team, we meet weekly, we have a weekly Trove meeting and something I just wanted to share is like, if you're thinking about contributing to the community, what's helpful for us is we do an internal sync up prior to our meeting. The purpose of this is to prioritize and just to make sure we're on the same page, the type of things that we want to bring up. So if you guys have been in RSC meetings, you know that it can get kind of a little busy. So we find that that's been helpful and we actually use TeamSpeak, so I'm getting into some of the mechanics of how we do this, but hopefully this can be helpful for you guys. But we actually use TeamSpeak and if you guys have never used TeamSpeak before, it's like a lot of gamers use this as voice over IP. And so we're on, this helps us kind of align, we kind of, this helps our rackers. We have 10 rackers who are contributing to Trove kind of on here just like kind of chatting about, like we'll hear the things that I want to bring up and it gives us an opportunity to ask questions beforehand so that you can spend the hour with the Trove community with some other important questions. And the other thing that we do is we kind of hang out and we just kind of stay on TeamSpeak while the public meeting goes on in IRC and that's an opportunity for people to kind of talk among with each other, just to kind of like get people like feel more comfortable because sometimes, especially for newcomers, we have people who aren't really sure, like is this the type of question I want to bring up or maybe another teammate already has an answer so I can just ask him or her real quick and then we can focus the public meeting on something else. So anyway, just wanted to share that. Also, the other thing is how our team participates in OpenSack is we actually, so we had a Trove mid-cycle meetup. So for Ice House, that has become a really popular thing especially as the OpenSack summits have gotten really large and the scope has gotten a lot broader. The mid-cycle meetups allow each project to focus on its specific topics. So we had one in Austin this spring, it was for three days, Rackspace sponsored this. We had four different companies come out, there were like 30 participants, so it was really an awesome thing. So, and I know that other projects have been doing this as well, like Nova, Glance, Neutron, they've done mid-cycle meetups. So anyway, so that's the stuff that we've done. Excellent, thank you. So Mr. Jesse, I actually wanna switch gears a little bit and so we've heard some of how the participation manifests itself in an operations environment and in a dev environment. I would like to hear from you and Mike on what are the frustrations that you've had to deal with. So from our team's aspect, we're not really open stack developers, we're not in Nova, we're not in Glance, but we're users of these and pretty significant users of these, but it can be hard as somebody from the outside of that specific development community to come in and say, look, we've got this big problem that we're facing as a user, can you help us figure this out? And a lot of times the initial responses will throw some code at us and we'll review it. And that's great, I mean that's something we could do, but what we're also looking for is more of a collaboration type thing. And so trying to bring that our wants and desires, not as just as something we're throwing over the wall and say, hey, you need to do this, but as a way of creating a collaborative session, that can be pretty difficult, particularly for somebody who's new to how the open stack upstream ecosystem works, just getting in, getting all their things together so that they can actually do a submission if they have code and just finding the right people to talk to to get that to happen. That's one of the issues. Thank you. How, can you expand on that? Sure, I mean, I think I'm gonna follow some of the same things that Jesse talked about. When I initially got involved in the open stack community two years ago, I didn't know any Python. I really didn't have any cloud background other than like. Ditto. Other than some random experience with whatever, some consulting experience. So, getting introduced to the community was confusing. There's a lot of information out there. And like Jesse was saying, a lot of people are gonna say, well, submit some code and that's how you can help. I don't think that's necessarily true. There is a lot of ways that you can contribute to the community. I think one of those ways that's maybe less accessible is coming to the summits and participating into discussions here and pulling aside key people in the community. Letting them know about your use case, participating in the user committee. Letting them know about your pain, your troubles. Some more accessible online ways to do this is reporting bugs, reviewing bugs. There's a neat little thing that you can do that people actually pay attention to. You can go report a bug and if someone else has reported or if someone else encounters the same bug, they can go find your bug and they can say, yeah, this bug affects me too. And it kind of raises the heat level of the bug. And when the types of people who go through and review bugs and prioritize them see that, they're gonna take that into account. So another thing that I would encourage is really, when we talk about the non-code aspects of the community, you really need to talk, you really need to be vocal. You need to get out of your comfort zone a little bit. You need to do weird things like contact people over email and invite them to a Google Hangout and say, I wanna talk about this problem because I don't know where I can help, but I have a problem. And I wanna talk about it with someone who can help me get this into open stack. So I mean, these are really the ways that I feel more involved in the community. I'm not like some awesome coder. I feel like I'm just a guy with some use case experience and I've been able to share that a little bit and I feel like that's had a positive impact. And not only that, but I like doing that. Excellent. I wanted to open it up to the floor and see if there are any questions from anyone right now. I see someone walking up. If you could just come up to the mic and if you'd like to ask a question, we're gonna do an exercise and getting out of your comfort zone and talking. So you will have to use the mic. Okay, this question is more DevOps in general, not necessarily the open source, open stack or anything related, but we're looking at moving from a shrink wrap software model that we sell to providing a service that's hosted in our own private cloud. And when we talk about DevOps and we're trying to figure it out, what's the granularity of these changes that you push in on a daily or hourly or weekly basis? How do you figure out what that chunk of code is that somebody's fixed a bug? Does that mean I replace the entire web app or the entire native app? Or do you know that if you build an RPM that contains that piece of code that that RPM can install and you don't need to restart any services or reboot the application? Or how do you kind of just go about in general dealing with that? So one of the interesting things when you talk about granularity is that the amount of balancing the volume of change with how long it takes to propagate that change out. So there are certain classes of changes that we have in our environment that I know it takes me a minimum of three weeks to roll that across everything that it needs to be rolled across done in a safe way. I do not want to do that with every check-in, right? So you have to start to get some grip around how long does it take me to roll out? How much time do I want to spend deploying? What does it take to deploy safely? And then that starts to give you some guidance about how big and how do I want to manage these chunks? You will find that that is not one answer. There are certain services that I can roll out in three or four hours. And I'm okay doing those more frequently. The general feeling that I've gotten in working with both operators and developers in this space is more frequently is better because then the amount of things you're changing are better. The ability to isolate whether or not a particular change broke something is better. But you can't kill yourself rolling out small changes if it takes a very, very long time. So you got to find balance is what I found, is that what's the effort to deploy it? Versus how often do I want to do that? I have rolled, I kind of fall on the three to one. So if it takes me three weeks to deploy something that I don't want to do that any more than once every nine weeks. If it takes me a few hours, I can do that much more frequently. But you got to find that balance point and how much effort it is. I would add to that as well, that another factor you want to bring into this is what is the impact on your consumers for those changes? And again, there's going to be different types of changes, but you have to look at the impact of each particular type of change and then measure how frequently you can have that impact upon your consumers. If there's no impact, then do it as frequently as you can because again, that brings your change that you're making into so small that it's easy to isolate any type of issues. But if your change is going to have an impact on your customer and your application at the time, then that's really something you have to reflect upon and figure out how frequently you want to inflict that. And that will drive how frequently you do your changes. Yeah, I would even add to that. We kind of had an agreement with our customers how often we could take things down and be super disruptive. And yeah, I feel like it was pretty successful. We had a model where we would deploy smaller changes. Hope we would try to do that weekly. And then the game-breaking changes, we had a cycle where once a month, we would kind of roll those out and everybody was expecting it and planning on it. Does that help there? Yeah, and Topher, you mentioned that you were kind of moving back and pivoting towards a shrink-wrap model and a release model to where the developers hand that off to the ops folks. Is that, can you elaborate a little more on that and what drove you to make that change? Well, what drove HP to make that change is they looked at, they actually had a number of different OpenStack projects going on. Public Cloud was one and that's the side that I represent and we had developers producing code that was very specific to public cloud and doing a lot of patching and some things were moving up. But they had lots of other projects going on. They took a look at that and the cool thing about HP is that they were able to say, we're gonna have multiple groups working even in the same space because we don't know how the space is gonna evolve. We don't know yet what our customers are gonna want. And after a period of time, they're able to top and take a look at that and say, based on the experience in public cloud and wow, we learned some lessons there and our experience in this team over here and these conversations that we're having with our customers, now that we've gathered all this data, we think this is the right pivot for us in our market. And then what does that mean for how we're gonna deliver to public cloud? So it was very much a product and marketing decision driven by experience for a particular segment. And we have a particular segment, Fortune 1000 that we're going for and that's what was right for us. So we have next up here. I have a couple of questions. So my first question is, I noticed that nobody talked about QA in your DevOps process. So how did you guys include the QA in your whole process? Question number one. And question number two is, the cloud technology is enabling the continuous deployment. And that's what we go and preach all our customers and community, everybody's saying that organization should go agile and then follow CD. And as leaders in this area, what is your vision and what is your take on the CD? Is it a successful model that can be practiced and shown to the world or is it just theory and practically it is not going to work? So let me just make sure we have the first part is the QA, is how we're integrating QA. Thank you for that because we always forget to say it and it's so important. And then the second part is just the CD, the continuous deployment model that we're kind of preaching in the DevOps world in particular or in the development and operations. Is that actually sustainable and do we see it as a viable option? What we see in the internet world is DevOps is for CD. DevOps is a means to achieve the CD. So as leaders in this domain, what is our experience and what are the problems we are seeing? What should we go and tell the community in the world? Who wants to take the QA? I can take the QA one. So I think QA is a really key part of your DevOps process. The only way that I've been able to make DevOps actually work, I can't do what some of these other guys do. I can't actually get my developers, my operations guys in the same room and interested in the same problems. But I can put them at an agreement point, right? Which is that they both have to convince QA that this is going to work. So I think a core part of this is some of the CI, CD stuff, but a code review process, a formalized code review process, where QA makes sure that someone from Ops is signed off. They've given their okay. Someone from Dev has signed off and they think it's legit. So really this two check system between Dev and Ops never worked until we gave QA the final say. So I'll let somebody else handle the other part. I'll add on to that. Can we do a release? So we try to follow a, currently we're trying to follow a 10 day cadence. So every 10 days we cut a release base and this is the latest and greatest code and cut the release, test it. But then on the day before we can release we have a sign off and it's a sign off between development, operations and the QA team. So that three has to, we have to agree before we push that out. We see a lot of patterns in threes and QA really is the third leg of the stool. Without it, the stool falls over, right? So continuous integration, continuous delivery also means continuous validation and that's the only real way that you can get from something landed to something in production is by way of validation. So it's absolutely key. If I can talk about the continuous integration question. I think that- It was actually about continuous deployment. Which is a slightly different, different thing, yeah. It's, you know, but yeah. Okay, well, I'll talk about the continuous deployment question. The, I think that, is it real? The question comes down to at what scale and in one environment. Absolutely, is it real to be able to have something coming in, getting out of a validated, getting deployed out somewhere to some environment in a continuous way? Yes. Is that how you will deploy it out at scale in production? I think that's got a long way to go and I'll tell you one of the reasons is we have seen some really fantastic bugs that do not manifest themselves until you get to a certain size or a certain time. And that is something where we really believe in not only having good long bake times and bake times at scale, but when I said before, when I talked about how long it takes to deploy something safely, that's about being able to segment deployments so that you limit your risk of and something that you did not detect in QA, you did not detect in bake, hitting your environment. And so that is something that we have felt that when you start getting to very significant scales, there's just really interesting bugs that emerge at that level that you want to protect yourself from by being a little bit more slow, a little bit more deliberate. And we may have continuous deployment going into an environment all the time and at some point say, great, we're now gonna take this thing that's all up to date and move it to another segment. The fun part about continuous delivery is that continuous is never defined. Continuous can mean every single day, continuous can mean every other week, but generally continuous doesn't mean every six months. A lot of the strategies are, you don't wait for a massive upstream major release, what you're doing is you're pulling fairly continuously, you're validating fairly continuously, and then you're deploying that bit as it makes sense in your organization, but you're not waiting for that, for the big things to happen before you. So continuous can happen, it's just continuous at whatever pace continuous means for you. Whatever regularity. And so the English nerd inside of me really wants to say it's supposed to be continual, not continuous. I can't hear a snazzy dresser and an English nerd. I also want to add one more thing, like for this to be continuous, you need to support from the rest of your team, your whole organization. So a lot of times I think it's really easy for people to want to say, oh wait, we need to hope the release because I need this one bug in or I need this one feature in. Well, how we look at it is like, it's kind of like a train station. You know what? It's gonna, another train's coming up later. So you shouldn't hold things back, you just need to keep going. But there are some checks and balances, like I mentioned the sign off process that we have with Dev Ops and the QA team, but we have to think of it like a continuous train station schedule as I just keep going. So I think that kind of to summarize this particular segment was that QA is really the glue that holds the Dev and the Ops together and kind of gives them a place to come together and actually be successful. And then that continuous whatever is very much a real thing and it just very greatly depends on how you define that, how you use it, what is the right thing. So as customers come up and start using the cloud, start adopting the cloud and saying, I'm gonna create a pipeline where I can do continual delivery and you're gonna ruin me forever now. Thank you so much Topher. I will never be able to say this again. It's really gonna depend on what their use case is, where they are in their maturity level and how complicated their application is. So yeah. I could just add something to that summary. One takeaway, you hear a lot of CICD talk around process and really I think we've talked more about culture than process and I think that's where the model is awesome and is really gonna make our lives better. I saw a hand over here. The buzzword is continuous integration and continuous deployment CICD. Correct English apparently is continual. On both of those and that's just the English lesson for the day. It means you don't stop. It doesn't actually mean that you're doing it constantly. Okay so when you implemented your continual delivery transition or there was like we're gonna have a whole new movement come out of this one session. I just add one question which is did you actually shift away the responsibility of the uptime of your environment away from the ops group to the development units or how is that transition because in our experience one of the biggest let's say impediments into the developed transition is ops group incentivized to keep the uptime as high as possible and therefore being very resistant to transitioning that responsibility to towards the development units. So I can take this one. What's been successful for us has been to try and align the teams so that your ops team and your devs team are all reporting up the same ladder eventually so that they're as a whole judged on the product itself not whether ops stayed up for a certain number of days before there was an outage or whether certain features got in but as a whole they're all judged together and when they're all working towards the same goal then it becomes less of I'm not gonna take your change because it might break my uptime it's more about let's get this in so we can grow the product usage and we'll do it in a way that prevents outage so it keeps customers coming back. Okay. It's about shared incentive, shared responsibility. If you are incenting the two sides differently and holding them responsible for different things then you have a lot a lot of resistance. If dev is incented to deliver on a particular date and has no responsibility for the uptime of what they deliver yeah, ops is gonna be resistant to that. And then you're gonna remember to actually get your keyways to check it and then you're gonna be like, oh maybe, oh okay yeah. So that's usually how that goes. We're getting better, I think as an industry we are getting better at remembering the three legs of the stool. Earlier in the process. So are there any other questions right now? Cause I have one final one. Here, go ahead. Okay, yeah. So we're very new in the moving to the cloud architecture. We're doing a standard shrink wrap app and CI CD seems to be like the pipeline for all the new fixes and everything goes in. Is there a best practices you guys have seen as far as our traditional apps are based on RPMs and we do in-place upgrades or do we destroy your VMs and just if we're not horribly scalable yet but I can imagine where in the future when things are re-architected you can just sluffer cattle and bring up new ones with the right stuff but kind of the older transitioning from kind of the legacy model as companies move to cloud architectures. Whether you guys have any best practices or have you guys experienced best ways to handle those continuous, continual deployment kind of challenges. So Mike, did you wanna take that one? Sure, I think I can provide kind of a different perspective on this. I don't know, maybe I'm taking too much credit but I come from the background where we have a lot of pets and my use cases weren't interested in getting rid of their pets. But CICD was kind of the standard. That's what we wanted to get to to accelerate development and to have a better customer experience and not incur all the expense of some long release cycle. Right, so where was I getting at with this? I don't know because you talked about pets and I got really confused. Well, maybe I misunderstood you but I feel like a lot of people are coming from this. They have tons of pets. They understand that okay, maybe pets aren't the greatest, we need to move to cattle but there's this transition phase, right? So what we did in kind of this transitive phase is we actually did deploy with RPMs and I would say we were like full, any changes that got merged into our dev branch were in everybody's development environments that second. That got deployed continuously. So that was kind of a different packaging and distribution method than what we had for our production environments. For that, it was more of a deployment problem and we didn't want to hurt people's pets. We wanted to package it up very nicely. We wanted to make sure that all the distro rules were followed and then we would roll that out in a slow fashion, you know, whatever, 500, 1,000, 2,000, 3,000 up to our full environment until we felt really confident. And like of course, some of that is gonna hold you back but I feel like that's what worked for us and our customers. So I have to cut the conversation short because we are at time. I do encourage you to stay behind if you have other questions and you wanna continue talking with us. Reach out to us on email or Twitter as well and also find us around. We'll be here for the rest of the week and thank you very much to my panelists, my special guests and thank you all for coming and being part of the conversation.