 Good afternoon, everyone. Welcome to our presentation. Before we actually kickstart this, we are going to use an application. It's called Poll Everywhere because we want to interact with you guys. So if you could download the application, that would be perfect. If not, you could use your mobile phones to actually go to the website that is hosting the polls and contribute to the conversation. Cool. All right, so welcome to Don't Fail at Scale. So Julio and I are here to talk about all the mistakes we've made as architects at Red Hat. So I'm Dave Costakis. I'm responsible for OpenStack Solution Architecture in North America for Red Hat. I'm Julio Villarreal. I am the technical lead for the OpenStack site in the cloud practice of Red Hat. So between the two of us, we're literally responsible for hundreds of deployments across North America. So we have a lot of things we've learned, and we're going to try to show those today. This is a little bit about our agenda. So we're going to talk about kind of, from the beginning, how do you build your team to support OpenStack? How do you plan for your deployment? After you've done a deployment, how do you continue to operate that thing? And once it's up, how do you make it a success inside your company? All right. So we're going to start just to kind of frame the conversation up with the scale that we're talking about. And so according to 451, about almost three quarters of all OpenStack deployments start with 1,000 cores. And they go up to 10,000 cores. So we're talking about a scale that's big. And we want to be able to plan for that and support that. So let's do the third poll of the day. We want to know how large is your OpenStack deployment? Everyone here that has an OpenStack deployment, could you please help us out and give us some answers? Are you under 1,000 cores between 1,000 and 10,000? Or do the deployment has over 1,000 cores? We see the answers coming in. 451 research was wrong. Well, at least for this group, right? For this sample group. So just to kind of frame that up into servers, the latest Intel servers have about 20 cores per socket. So if you have, that's roughly 25 sockets for 1,000 cores, or 12, 13 servers. So we don't have anyone in the room with over 10,000 cores. And actually, no people in the room with over 10,000 cores. They're just not working. They're not voting. I think it's a lot. Guys, we're in the last summit session in this room. So perfect. But we are able now to frame the conversation, right? Like everybody here is mostly under the 10,000 core mark. That's right. So a lot of you are kind of starting out with smaller clouds. And so this may be a good talk for you, hopefully. Right? So everybody has quotes in their presentations. We're going to start with a quote that we ask ourselves every day is, what do I need to operate this cloud at a customer site? And then we have another poll. And this is the phone one, you know? Can you describe with one word what is the most important thing you consider while building your OpenStack cloud? What is that important component that you are considering while you are building your OpenStack cloud? So any word? We have two, two, three, three, OK? Some coding message in here. Do we have any spies on the room? That's right. Julio is a good word to think of if you want immediate support. I vote for Julio. So stability, scalability, agility. Fear. Somebody put fear in there, man. Efficiency. So we have a bunch of words popping up, two, two, three, three. Yeah, so scalability and stability are kind of the two ones that I see as number one and two. So far, with cost is a third. But that kind of makes sense. People want to build a big cloud. Multitenant. So we have a few answers. So turkey. Oh, turkey. Now people are just messing around. I think they need to move on. So we have turkey as a service already? That's a new thing. Thanksgiving. All right. All right, thanks for that. So we're going to move on from the fun stuff and talk about building your team. So again, I'm responsible for the OpenStack solution architecture team in North America. And I'd like to talk about some of the things I learned. So we have hired a lot of people Red Hat. And not every person that we've hired and identified to work on the team has worked out. So we're going to talk about some ways you can hopefully avoid the mistakes that we made when you build your team. So identifying people. This is where we've made some mistakes. But OpenStack talent out there is very difficult to find. The foundation puts these numbers up. OpenStack engineers make 30% more or whatever than regular engineers. And so finding these people is hard. They get paid more because there aren't many of them and the ones who are in demand. So if you're going to look for these people and you can't necessarily hire someone with OpenStack in their resume because they're not available or you just want to look for someone internally, these are some of the kinds of things that you're looking for. The number one thing, this kind of goes in order of what we think is important, but self-starter, self-learner. OpenStack changes every release drastically. So if you don't, if you're not a self-starter and you're not going to learn on your own, you're not going to make it in this business. You've got to have a passion and not be afraid to go in there and hack up the code. I do it more than I'd like to admit. You've got to care about troubleshooting issues, be able to get in there and figure out what's wrong and pull all these disparate parts together. The fourth thing is really important for me. It's a willingness to prove that you know what you're talking about. So I talk to a lot of people. We try to hire them. They talk a great game, but there's a difference between people who can talk a game and use the buzzwords in this room that you hear at the conference and people can actually do stuff. So I only hired people who took and passed a test before Ann, so keep that in mind. There's a lot of buzzwords out here. Don't be taken in by hearing Kubernetes or K8s 10 times in an interview. And then these people that we're talking about are not afraid to ask questions. There's no one I know who knows everything about OpenStack. So if you don't know how to ask, this person is not going to be right for the team. So as far as technical skills, again, we kind of listed these in order, right? And I talked to people about this and they seemed surprised that I put OpenStack at the end. But again, the fact is it changes a lot. And the other fact is, is that OpenStack is designed to run on Linux. And if you can't get in there and create an OVS bridge manually, if you can't set up link bonding manually, if you can't write shell scripts, you are absolutely not gonna be successful as an OpenStack administrator. Sure, you might be able to go into Horizon and click some buttons, but that's not what this is about, right? Virtualization, networking and storage. So ultimately with OpenStack, we're trying to virtualize and manage our data center. So you need to be able to understand KVM or the hypervisor that you choose to run. A lot of people, especially telcos, have advanced networking requirements. So if you don't understand link bonding, 40 gig, full duplex, load balancers, high level, you need to understand what those are and how to run them. You know, storage, I don't know how many people I talked to who don't know what multi-pathing is, and that surprised me. I was stunned by that. So as we move down the stack, I think these are things that we think are important, but they're also things that we think we can train people on. So we'd be great to hire people who know about Python or Go. That's cool stuff, but I can train people on that. The same thing for configuration management. Ansible's on here. I learned how to do Ansible in an afternoon. Most people can. So being able to build that skill set into your portfolio is very important, but something you can train people on. The concepts around continuous delivery and integration and DevOps, important to know about, but we can work through those too. And then OpenStack is the last thing we put on here, just because it's so hard to find these people. We're gonna talk about an enablement next, but we bring them on board and we train them. So enablement is what we kind of alluded to last. It's more than taking a class. Everyone who works on my team has been through the enablement. Julio and I have helped teach the class before. So what we kind of did was we identified our top talent originally, and we had them create this enablement class for us internally. The assumption we made this was that anyone who walked in the room needed to know everything about the environment, everything about OpenStack, and everything about how to install and run it. There's a hands-on course, a boot camp, we call it, that people go into and they actually deploy OpenStack on real hardware with real-life scenarios. After the class, people have an assignment. So what I've noticed is if you give people a class and they sit through it, they tell you they learned a lot and then they leave in a week later and they forgot everything you just talked about. So people who leave our class have to take a test and they have to pass the test and if they don't pass the test, it's not good. You can't take another class from us if you don't pass. And then don't forget vendors. If you are starting about the process of building your own training and your own class, you'll look to your vendors like Red Hat or the OpenStack Foundation and their certification offerings today. So an environment for success. After you've trained somebody, you can't just walk away and think that they're gonna be enabled for two or four years. Each time OpenStack comes out there are new features in it. There's new pieces of technology that people are talking about. You talk about something core like replacing OVS with OVN. Those are changes and people need to know how to troubleshoot and understand those. So the expectation we have is we give people a sandbox and they have to continue to test and learn in the sandbox, have these different lifecycle environments to prepare themselves. And then retention. So again, we've talked about this. OpenStack is a hot skill set. It's easy to get hired if you can put OpenStack on your resume still. So you wanna do what you can to maintain these people that you've enabled. So recognition, internal to your company. I know that some companies have different rules about how you can recognize externally, but at least recognize people internally. If you can allow for external recognition, allow people to go out and speak at OpenStack Summit. People should speak at meetups, do blogs, go to conferences, whatever it is. People like that, right? They like to build some notoriety for themselves in the public. And then reward them, right? I mean, at some point you gotta pay these people. They become experts. Now we know what steps to take to build a team. But we're going to go now into the planning phases. You know, like planning for OpenStack deployment is the super important part of a journey. So do you know the effect you intend to create? This is paraphrasing sci-fi author, Melissa Maffil. Who knows that? You know, what kind of impact are you going to create in your organization with the process of cloud auction? Planning for OpenStack, the first thing that you need to do is you need to identify what is driving your OpenStack adoption. I keep talking to customers and the first question of the meeting is like, why are you adopting OpenStack? Why do you need OpenStack? Like, over the last couple years, that answer been changing, you know? Like, on the beginning it was like, oh, it's the new thing, you know? Like, my boss read about that on CIO.com. And it's what is hot, you know? We want to adopt OpenStack. Like, while the time pass, we see a lot of the answers actually changing, you know? Like, people are adopting OpenStack because they actually have a business need to adopt OpenStack. There is a business reason behind. Like, some customers of us are adopting OpenStack because they want to go faster to market because they want to reduce TCO. So those are really specific business reasons. If you don't have a business reason to adopt OpenStack or any technology, you should ask yourself the question like, why am I doing this? To myself and to my team. The other part, and it's the second question on this slide is what are the technology reasons what I am adopting OpenStack? How OpenStack fits my technology roadmap? You know, like, what kind of problems I am fixing by adopting OpenStack? Like, do I need them as a service platform? I'm gonna take use of the SDN incorporating into OpenStack. Oh, I want to do SWIF for my object store. I want to integrate with my existing SEF. Those are valid things, but you need to have valid reasons to adopt OpenStack. The other part and it's super important is what will be the impact of OpenStack in my organization? Like, you are adopting a new technology. How that new technology will impact your organization? Now we are talking about here about not just software. It's a collection of software that have legs in pretty much every department of your IT infrastructure right now. You know, you will have to collaborate more with your networking team, with your storage team. Some things that need to be delegated to your OpenStack team. You will need to create new teams in some cases to actually go and do OpenStack. You know, like, take a dev approach. Those things are really valid reasons that could impact your adoption of OpenStack and need to be included in your planning side for OpenStack. The other important part is like, you need to communicate with your team. What are you doing and why are you taking this direction? Like, if you start communicating early the reasons from the business and the technical side to your team, you could get people on board of the idea of doing OpenStack and adopting OpenStack as a cloud technology in your enterprise. The other part is like, super important day one and Reha, we do this every week with our customers. We sit with our customers and we go through a discovery. You know, like, why? How are we going to do things? We go and we define the use cases. We map the use cases to OpenStack one by one. You know, like this use case, map to this project in OpenStack and this is how we're going to do this. You need to do that within your enterprise. The other part, and that's a thing that I do a lot is you need to create your architecture. And I have multiple customers here. Thank you for coming, by the way. And you need to go to a point that your architecture meet your use cases. You need to go and do the pairing. And you need to define everything during your architecture planning. The number of nodes that your cloud is gonna have on the phase one. The number of services that you are going to make available. You need to go and take a most likely phase approach. Do you want to bring all the services that OpenStack contains into your organization at once? That may not be the answer, you know? How are you going to incorporate gradually those services and provide those services to your customer, to your internal teams? That's a super important part that should be defined during the architecture session that you are having in your enterprise. And after that is how I'm going to execute this. You need to create your execution plan. Please, plan for it. Like, I've been to too many places where we talk to customers and they say, well, we tried OpenStack six months ago or a year ago and it didn't work. And when we start digging and asking questions, the problem was planning. They didn't plan for it. You need to plan how you are going to execute your OpenStack deployment. That's important. It may sound simple, but most people don't do it. Plan for it. How will the deployment happen? Who will be responsible for each step of the deployment? Okay, this team is responsible for carving the villains. The storage team is gonna be responsible for providing me my NAS or my son or my safe backend. So, identify who is doing each part of your deployment. And the other part is create a test plan. Like, create a test plan from the beginning. What are you testing for? What your cloud needs to perform for? You know, how highly available need to be? You need to identify this on the planning phase of your deployment. It's not afterwards. You know, it's not, oh, I have the cloud. I architect it, I deploy it. Now, what I'm gonna test for? You need to have your testing plan defined from the beginning. Now, rolling into some of the other logistics, is what kind of hardware I'm gonna use? Who is my server vendor? My storage vendor? My switching vendor? What kind of support do they have for OpenStack? Do they have a reference architecture? Those are really important parts. Do the hardware that I am choosing meet actual, my actual architecture? You know, I need SRIOV, because I'm a telco, or I need a 4G pipeline because I'm doing high performance networking, or I need NVMe storage for my IOP requirement. Those things need to be answered and planned for during this stage. And then, at the last but not the least, software, you know, software is super important. It's not because we're a software vendor, but please, you need to choose your software, not as a vendor, but as a partner. How are you going to bring your software to work with you? What kind of support are they having? What kind of experience do they have? I'm going to go for a vendor. I'm gonna go directly to a stream. I have the talent in-house. I'm gonna risk it. I'm gonna do that. You could do that, but you need to plan for it. Of course, if you go with Rehat, we're gonna be super happy. Now, support. Support is a good component of your software selection. Like, how many operators do we have in the room? Like, how many of you are actually in charge of OpenStack? We have one, two, we have a couple of people, right? How many of you like to sleep on the weekends and do actually fun things? So, everyone, we love to do that, too, right? You need to be aware that when something happens, you could grab a phone and you could make a call and you could get somebody to help you go to that problem. Like, we see this many times when people go and they start Googling things. OpenStack changes so fast that you could Google something and you could get like 20 different answers for what you're Googling for. We've been there, done that. So, having a vendor that you could rely on to support you when you need it, that's a really important part and need to be defined in the planning exercise. The voting part, and I did that for several years, I've run operation for one of the largest clouds in the country, is you need to understand what is your data center constraints? Do I have a space for my OpenStack cloud? Like, how much space do you wanna take out of my data center or my data centers? Do I have power to power the thing? What kind of availability I'm going to get out of my data center and out of my OpenStack? That will determine your SLA response and then what kind of connectivity I need to have into my cloud? All these are part of the planning phase of OpenStack in order to actually, but based on our experience to do it in the proper way. I might now give it to Dave. All right, so operating OpenStack is a little like being in war, so Leo added this SunSu quote in there. But so, you've built a team, right? You need to complete that. You need to go back to the code and you need to ask how many things that operating OpenStack is like being in war? Because I feel that, you know? There's been a few times, right? And so, after you build an awesome team, right? You trained them up. You plan for your deployment. Just like Julio said, you're ready to go. You've got your deployment and it's up, right? The deployment at this point is an afterthought because you planned and you're ready to go. Now that it's up, you need to think about how to keep the lights on, right? Most enterprises that we work with already have a monitoring infrastructure in place, right? I can't think of an enterprise I walked into who's not monitoring something already. And so, what you wanna think about is how you're gonna incorporate the monitoring that you have to monitor the services in your OpenStack cloud to meet the SLA that your customers have, right? And so, most people are running whatever, Sitescope or whatever the thing is that they're gonna use to do web API checking and that's a really good thing to check the status of your cloud, right? Do synthetic transactions against the cloud periodically. Ensure that everything is up. Hardware monitoring should be in place. You should link that all together. Remediation procedures. We just talked about being up on the weekends. I hate being up on the weekends. It's the worst. So create these procedures and you can pass off remediation to people whose job it is to be in the knock and be on call. Centralize your logging. Learn what the errors mean. Red Hat can help you with this. What I've noticed, one of the most frustrating things for me about OpenStack is that there are errors in the logs that aren't errors and there are errors in the logs that are errors, but the message says something that's not what the error is. So learning about all that, the ins and outs takes experience. Red Hat and vendors and the upstream community can help you do that. Managing, manage your internal expectations, right? If you don't manage expectations, people are gonna think the service is gonna be up all the time and if it ever goes down, it's always gonna be your fault. So manage the expectation for your service internally. Post an SLA, whatever it is. Leverage the community, leverage Red Hat. But whatever you post, your SLA has to be two nines, seven nines, whatever it is, make sure that you respect it. So, life cycle. It is, so many customers think that they're gonna build OpenStack and they're gonna build one cloud and that's gonna be their one production cloud. And that's cool. But what happens when you need to test something? What happens when you need to add a new piece of storage in? What happens when you need to plan for your in-place upgrade and salometer change to something else, which is sort of in the middle of happening in the upstream community? You might care about testing that stuff and so we highly recommend treating this environment like you would treat any software deployment. Have a dev and a test area. Have a staging area. Let your people use their internal sandbox environments that you built and made available to your awesome team to plan and stage and automate all this stuff. So when you go do it in a production environment, you're ready to go. It's been done before. Again, some of these are maybe they're obvious, but so many people just, they kinda wanna just stick it out there and then maybe go back to managing a Windows machine somewhere. So think about this and plan for it. So take an infrastructure as code approach when you go to operate your cloud. Check your configuration into CVS or Git. It's so easy. It's free. Use GitLab or whatever and you don't have to pay anything. Automate your changes. Red Hat provides a framework to be able to automate upgrades, for example, but it's kind of up to you to look at that framework and use it and put it into inside automation. Again, use your lifecycle environments. And this is mine. One thing I strongly recommend. No manual changes in production. This will enforce automation. If you don't have users who are able to SSH into your environment, then you're gonna have to go through automation to get anything done. Now we cover how to build a team, how to plan for the deployment. The deployment is done. We know how to put some practices and how to operate OpenStack. Now we have all that. Now how we succeed with OpenStack. And this is what your manager is gonna ask about. How are you translating technology into business? So let's do another quick poll. And I love these things. What is the biggest driver for OpenStack adoption in your enterprise? I want to know what is actually driving your OpenStack adoption right now. Cost, service, money. I've been getting money in all my presentations. We have some people access with money here. Which is good. API booking, sufficiency, agility. Holio, oh, thank you very much. CACD. Paranoia. Some paranoid people also sell service. Make me a sandwich. But the biggest one. Oh, make me a sandwich. Sudo makes me a sandwich. I like it because you have Sudo from all the way. I'm gonna go for Brent Holden on that. Yeah. He's over there laughing at his own joke. Yeah. So when we look at the ones that actually we're typing seriously, cost, agility, sell service. So these are one of the biggest drivers for organization adopting, not OpenStack, but cloud as a service components. So at the end of the day, William James said that toughs become per section, per sections become reality. And when you are working in any enterprise, this is a really important thing to consider. How what you are doing is perceived by everyone else in your organization. And this will help you to do the cloud adoption process. You need to identify what is working against you. This is super important. The number one that we see with some of our customers is status quo. Status quo works a lot against the cloud adoption or anything that modernize. You have your coworkers that have been doing this thing on the mainframe for four years. And that's the only thing that he does and he doesn't want to change. He want to keep that happening. And that's a reality on the enterprise. And we talk to big enterprises, that's a reality. Then the other part that drives people away from OpenStack is the perceived complexity. We are not saying that OpenStack is super easy, but it's not as complex as people put it. Like every technology that you will implement in your enterprise will have a learning curve. OpenStack has a learning curve. You're gonna start with it and it's gonna be rough and then you're gonna start making sense of the software. You're gonna start knowing how to handle it, how to operate it, how to make things out of it. So the perceived complexity is a thing that goes against OpenStack adoption. The other part is like the lack of innovative drive. Some people don't want to innovate. They say like, you know, things are working. It takes me 42 days to get the virtual machine, but things are working. After 42 days my developer is getting a virtual machine. That kind of lack of innovative drive will be against your OpenStack adoption. So if you are trying to push people in your enterprise to adopt OpenStack as a technology that could freeze things that are broken, you are gonna have people on the other side thinking this. I'm telling you that in your face, which is really funny. The other part is the lack of internal customer champion. Please get a champion. If you are not the champion within your company, or you may be the champion, but you are not high enough to actually promote change, you need to have a champion to influence the right people. You need to have somebody within your company that is championing OpenStack with you. That's a really important thing. The other important part is don't build it with the idea that I will build this and people will come and use it. That never works. Never, never works. I've been to so many places and say like, yeah, I have an OpenStack cloud. Is there idle? Nobody uses it. I use it right now. I'm running WordPress on it. It's like, why are you doing that? You need to have a business purpose. Build it because people are gonna use it. Communicate. The other part is what will work for you? And this is based on our experience on the field. Like as Dave said, we are responsible for hundreds of OpenStack deployment across the world. So we see this every day. Build a big head with your champion. You need to have that champion that will go on voice and support you no matter what during your OpenStack adoption. The other part is you need to start controlling the internal narrative. What people say about your cloud that need to be under due control. And how do you control that? You provide a lot of status updates. You provide a lot of reports and you use a bunch of metrics. And I know that these words are really bad words on the engineering side, on IT, but you need to have that control. After that, you need to champion your internal success. You need to show value. And we do that in a really funny, I did a program with one of my customers and he said like, how do I show value? And I was like, you know, let's implement ChargeBack. So we brought one of our Rehab portafolio tools, CloudForms and we hook it up to OpenStack and we start doing ChargeBack. And it was fake money. And he was changing, he was charging Amazon rates to his IT department and to his customers. But at the end of the month, he was able to go on and say like, you know, this is the money that I am saving with my cloud. You need to be able to show value, to justify your existence and why OpenStack is an important thing within your organization. The other part is, please, as they were saying on the session about building your team, build notoriety for your OpenStack engineers. These people are super difficult to get. By the way, we are hiring. Pay me if you need it, if you do OpenStack. We struggle getting the right people every day and you will struggle, you will go to struggle maintaining the right people. Praise these people, you know, recognize what they are doing. That's an important thing that will help you to your cloud auction. It's not easy to go and build a cloud and then like two months later, lost half of your engineering team that built it. The other part is, allow your people to check what they are doing. Come to summit, you know, this is how I'm doing OpenStack. That's an important part because it's not only sharing your experience, it's establishing communication with the community. So you could get some feedback. If what you are doing is right or wrong. We're gonna open it up for questions and while we have questions, we're gonna take a note that from Red Hat Jew Lim is here doing some usability testing. So if you drop by the booth, ask for Jew Lim. She would love your input on usability of OpenStack products for Red Hat. With that, we're gonna, I don't know if there are questions, but if you have any questions, please go ahead and walk up to the mic. The session's being recorded. So if you do have a question, please be sure to speak into the mic so people who watch the recording can hear it. But I don't see any questions out there. We put the wrong slide. Yeah, we have a little out of order there. So any questions? If you, we have one over here. A question I wanted to ask is a question that I get asked pretty frequently, which is what do you think the size of a typical OpenStack team is when a company's looking to do an OpenStack deployment, are they building out a three-person team or a 25-person team? And what are the different factors that would change the size of that team? Well, so I think it's at least two and as many as maybe five or 10, depending on the number of clouds you plan to deploy, how big they are and what their SLAs are. There are a number of factors that kind of go into that, right? But certainly it's gonna be how much time these people have. I mean, you want these people to be probably not 100% dedicated to OpenStack. They probably have a rest of their job to do. So I would think about those things, but I would say it starts with two so someone can take a vacation. Yeah, and I think really large teams also on the field. Like some telcos have like large teams, like multiple teams working on OpenStack. So you could end up with that. At the end of the day, what is the size of your cloud and how important is your cloud for you? Any other questions? No more questions? All right. By the way, we are gonna be hanging around when we finish. Julio will be hanging around. I will. He'll probably have for like a thousand meetings, but. Thank you for coming. Thanks. If there is no anymore questions. Have a great summit. Have a great summit.