 Good afternoon. My name is Steven Smith. I'm the Director of IT Infrastructure at MCNA and welcome to OpenStack Summit in Austin, Texas. I'm going to be talking to you about getting OpenStack out of the corporate basement. So there's a lot of elements that you'll have to go through depending on whether what your current infrastructure is and where what your starting point So we're going to talk about components of a successful project proposal, which is where all projects start from Discovering and satisfying the corporate infrastructure requirements Addressing the upper management concerns and any hesitations that they might have promoting and presenting your OpenStack project team training Some no-cost proof of concept strategies and We'll review a little bit of a couple elements of some infrastructure design and finally alleviating risk so as all projects start you'll need to start with a proposal so the most important part of the start of starting such a project is Best to clearly state your business goals and objectives. So what are your objectives? Are you reducing your overall operating costs for servers? You want to run some additional applications? Will your investment pay off and what will your total cost of ownership be from your current solution? There are other elements that of OpenStack that you want to Also deploy do you want to enhance DevOps or deployment services deployments and other services? So What is the reason for going moving over to OpenStack? So You want an open flexible reliable technology base for the for the future? So MCNA is a dental insurance company. We have to be HIPAA compliant and In our in a private cloud you'll have vastly easier compliance since you have control over all your your infrastructure and as a Cases in some of the public cloud systems, they will charge you up lit costs For dedicated instances or resources So what are your current pain points? What is it you're trying to to solve? If you're in a public cloud system all the hardware is shared And if you'll notice on top, you'll see If not occasionally most of the time Stolen CPU at the top that means that those are CPU cycles that are being taken from your instance and given somebody else If one tend to just having a problem or an issue it affects you as well If you have support from your Cloud cloud provider We've been on hold for hours at a time during during major outages with no resolution Lot most private public cloud Providers you have absolutely no access to the console on boot up And which vastly reduces your Your your time to solve issues Tier one or tier three support a lot of Cloud providers, they're all the same Matter of fact most of our techs We don't even bother calling because we know more than they do and calling them again. They just Generally have no answers We have instance maintenance in the middle of the business day, which has never made sense to us whatsoever we had You know a reboot server or a stop in two o'clock in the afternoon For business continue a continuity and disaster recover An infrastructure in one site doesn't necessarily mean Doesn't necessarily mean that you have fail over in a second site and generally you'll have to double your costs to have fail over in another site In the case of your private cloud, it's just the cost of a second set of hardware Any Upgrades that they do hardware schedules or any decisions they make you're at their mercy and you have no control over them So those are our pain pain points, but what you have to do is in your proposal is list your find out what your pain points are and and Resolve them You'll want to hit on all the service improvements that that you have Total hardware control gives you faster response to resolve your issues Higher level of security you have way more control over the security and again if you're have to comply with some regulatory laws It makes it a lot easier You have a greater scope of instance and service customization to suit your business needs. You're not stuck with whatever resources they your cloud provider provides as a as a For resources for for CPU and memory You can design your computing environment for any level of performance you you want you can have different hardware for different parts of the infrastructure And your hardware network performance can be specified and customized so Your implementation phases are going to be very important Mostly you'll be starting off on a sandbox where you will have no resources where Maybe your boss isn't even even aware that you're you're going to propose this so You can use miscellaneous hardware you use dev stack Some of the open stack all-in-ones installations Virtual box and as we did we start off with a bunch of workstations that we just grabbed and and installed open stack on So your next proof of concept once you've gone through the sandbox phase your proof of concept There's other Resources that you can use there's some free trials out there that you can sign up for And you'll want to bring it up to the next level of a separate controller compute and and storage in your pilot a lot of hardware vendors will lend you Hardware once you're getting more serious about it Either in their own Facilities Or they unlikely they'll they'll ship you some but There's also some vendors that will do allow you to do hardware trying buys Where you purchase the hardware and if you don't like it you just return it And at this level you should purchase a subset of the hardware that you're going to use for your final production design and then Then you move on to your production environment where you'll actually Deploy the hardware your final production hardware and Start migrating your production instances So in each one of the phases You probably want to you want to be finding out does open stack play well is it going to work for your company? Is it going to solve? Your pain points and These are usually isolated Implementation tests and you want to prepare for your your your proof of concepts In the proof of concept This is where you will find out a little bit more if it'll really work and You'll actually Try to be testing the implementation of open stack on the separated separate compute storage and and Hardware Pilot and getting your feet wet You're getting the execution of the the the implementation framework you want to be exercising How you're actually going to be working with open stack and by this point you should have the whole production environment Designed Most important take your time. This will not happen overnight Could take four to six to eight months depending on approval processes and It's important to get each phase approved and accepted at each level So The first step towards a successful open stack deployment setting is realistic expectations. I Under promise an over deliver Cloud components will fail and you'll need to design and high availability and it will take longer than you think it will So set up a schedule up front and stick to it from from the very very first sandbox Try and get a schedule set up. It's all it goes all the way out to a production deployment Like I said, it will take longer than you think so be very conservative When you're choosing your dates and how much time it's going to take to do each step so You're going through your sandbox part way through the proof of concept and You're going to have to decide what are your infrastructure? Requirements so you want to determine the services that your current infrastructure provides whatever that may be and then You want to map them to the open-stack components. So these are the basic open-stack components That every cloud should should have and Which a lot of public cloud providers already provide so Whatever you decide or whatever your infrastructure currently is map them to The corresponding open-stack components What are you not using in a current infrastructure? What are you not using with a current? cloud provider If they were free to use would you start using them? So These also mapped a lot of cloud providers of offerings But perhaps you're not using them because You're not there. They cost too much or they don't fit within your budget. Well an open-stack. These are would all Be free to use as long as you have the hardware to support it which a lot of these will Basically work on a controller You may need to buy some extra hardware, but you can plan and budget for that. So One of the toughest parts is not the technical part. It's actually getting management approval and managers read Probably more than tech guys do and they know of they've read all the articles and They've read reports so We'll just go over some of the Some of the concerns and they might have so their first concern is the pool of candidates Open-stack is Pretty small at this point it's growing but They aren't standing out on the street corner with signs will work for four open-stack. So Um Security is another huge concern if you're building your own infrastructure where it's a security. How are you mitigating it? What are you doing about it? Your deployment, how are you going to deploy open-stack? Are you going to deploy it from by yourself? Are you going to get help? Are you? Wow, are you going to get help? Are you going to use some other tools? And Then why should we be using open-stack? Well, yeah, who paypal and some other large players are using it. Yeah, but we're a medium-sized company. So What can we do? We don't have all the resources of these? Well, I'll show you how we can find them later on What's the system stability of open-stack I've read it's not stable Well, I'm going to through my proof of concept. I'm going to show you that it is stable and I'm going to document everything to show you what what's possible and with open-stack and how stable it is and What if you don't have the capex this year for hardware well, there's other leasing options and other options that you can work out with a lot of the hardware vendors and Just don't Go telling your manager or your boss everything up a management everything. That's great about open-stack. Well Everything is great about open-stack. There are some Issues and there are some things that you won't be be able to do so make sure they understand both the good and the bad so you've got part way through your project and You Want to accelerate it? What can you do about it? Well, you can start seeking out stakeholders and collaborators and collaborators and innovators in your in your company Top one on my list was the dev team They wanted the ease of deployment for test servers They want to start moving in the docker. They wanted to Be able to launch farms at a click of a mouse Which is all possible with open-stack So you can get them on your side and seek out others then you can get some Supporters of your project So when you're promoting and keep concepts simple don't be too techie. Yes, you understand all the technical Requirements behind it, but your manager may not understand what you're talking about so Lot in the way you may get arguments as well Well, open-stack can't do this and open-stack can't do that or we can't you know, we're also we're looking for something a little different So present them all counter that with solutions well, yes, but we can do this or We have we can get around it some other way or do we really need it and again? I'll go back to what is your current infrastructure and what are you using? You may have all Kinds of paperwork and design drafts and and everything else But somebody's gonna read it at some time. That's an executive. He's gonna say what is this? I so at the beginning of every plan I wrote executive briefs two three pages long that Basically outline all the technical details Before below and that was in the document We gave many demos We had one demo go bad and it actually did more damage than anything else so make sure you practice them and and Make sure that they go go smoothly because it's It's one of the worst things that can happen if they actually see that something's broken or doesn't work properly If you don't talk in the correct terms of management, they'll hear clinginese. They Won't even know what's what's going on and stay away. So let me make sure you stay away from the technical jargon And Speak in terms of business and goals and metrics That's what they understand. They don't understand That it's really cool to have new hardware or you're running you run 80 cores on a single chassis or anything like that and As it doesn't really have a return on investment think more of TCO toll cost of Ownership you're gonna. Are you gonna reduce that hopefully and? That'll be something that they'll be very interested in If you're gonna start stating savings and calculations in your project plans make sure That they are accurate be very conservative of them because if you get into one phase or you get into Production and they're not realizing the savings that you said that you were gonna get it'll be viewed as a failure and Don't make promises that opens that can't keep I've had guys in meetings and demos. Yeah. Yeah Opens that can do that No, it can't really and then you get in trouble because it could be a component that one of the managers will Latch on to and so where is that? I don't see that so part of Part of the process you're going to have to have people to support it design it and building it build it so How are you gonna do that? Well, they can start reading the documentation which is a concept. I know foreign to most techie people, but The Open-stack docs have gotten much better over over the years and they're very good now a Lot of the installers deployment tools have documentation on their website that you can get a lot of information from from and Build a build a lab for you guys. Let them let them fool around in it It's nothing better than typing away on the CLI and understanding, you know, what does what? It's also inexpensive training materials books ebooks at Linux Academy also has some Training materials that they start offering. There's lots of expensive training to You can either bring in a trainer or fly him out to a Training facility Make sure you allocate time for system ins to play Make sure they're not busy all the time so that they can actually progress in their learning and Get to know and as more that they learn the more that they'll be eager to to start implementing And for that matter you can create a training schedule as well summit videos for past Open-stack summits awesome source of information they can watch them at lunch or even let them watch them during the day and You can also explain them Open-stack would be good for your career. You're not getting replaced or getting enhanced and and Hopefully that will satisfy their suspicions Okay, at this point I would tell you an Open-stack joke But you'd have to build missing your part missing parts yourself to make it funny. I Think it's the only Open-stack joke in existence No cost proof of concept strategies So the obvious one dev stack I can start install it on Texts can the system ins can install them on their machines. I Give them some extra machines to install it on If you got some raspberry pies hanging around you can make a cluster out of them We grabbed spare desktops that were either waiting to be deployed or might be a little bit older and Started and the proof of concept started separating out all the components onto individual machines There's a lot of free trials that are out there rack space Moranis tri-stack they'll give you some resources storage space VCPUs and You can use that either to Build your help design your architecture or or for training in Lot of hardware vendors. They'll give you a couple of chassis Which is will be a larger set of of hardware that that would be available in the previous suggestions They'll give you a blank slate. You can actually practice installing the Open-stack on it and And testing the components out So go over some Some architectural I mean there's lots of papers out there and architectural design But there was some key points that I found Let a lot of talking to people that they they they missed out on or they didn't quite catch so When you're designing a file Production infrastructure you should spend more up front in hardware to To capture to for future capacity So if this is what we're using today, and this is what we think we're gonna be using in six months from from now Buy a little bit more Because then you won't have to be spending time adding the hardware in in later. I Cannot stress enough about high availability so instead of One storage node one compute node and one controller node We want a lot more You look at some of the design documents architectural Recommendations and you'll see it's a minimum of three controllers Whatever compute you have and you spread your storage out over as many nodes as you possibly can Best security is a stack-separated network where all the admin storage Networks are all separated from each other again. There's lots of documents out there that will explain and show how to set set this up, but it's Absolutely essential to to security you don't want one of the VMs Accessing one of your directly accessing one of your storage nodes and Very important in architectural design is your team so you want to Build the right team we divided them in a three layers where we have our core architecture architects system architects that were actually the brain trust for Open stack that Taken a lot of training spent a lot of time learning it and actually went through each level of the process and Does not help design and build and and test and test the systems so as you're doing that and if Your projects progressing well you can start Training some of your Linux sys amends They could have become architects down the road but you want to show these guys depending on the infrastructure you're coming from that how did How these system works how are they going to operate and how are they going to utilize? Open stack and Then the knock You'll want somebody you don't want to stay up 24 hours a day. You want to train them in some of the basics of Issues that can go on and how to fix and correct them and And we'll get into some other pieces of that in a bit Very important at each level of the Sandbox proof-of-concept pilot and production is to do testing At each level Don't wait to the end to do all your testing You will Find out you've made a lot of mistakes things don't work And you'll have to go step back on phases and explain the management why you're now going to take two more months to deploy this So there's some very good tools out there for testing operational use cases Tempest is one of the best rally is even better It actually uses tempest and has a UI and produces some better grass and some text output So you want to you know exercise all the API's that are possible on on the system and Try to break open stack We've not had a lot of success of that actually Now you're going to own the system the over the infrastructure is going to belong to you You're going to be supporting it now So one of the items that I've seen people miss out on is monitoring and alerting so A lot of these things in cloud providers happen in the background You'll get your alerts at your VM is down, but the underlying infrastructure. They're looking after well now you're looking after it So make sure that you find a tool that's going to monitor every single aspect Of of your open-stack infrastructure from the hardware up to neutron to the different components Of this is a Zenos by the way. You actually has a Zenpack. It does an awesome job already of Monitoring and alerting your system Wow, so alleviating risk Don't need to do this right from the beginning But as you go along you're going to realize what risks are involved some are obvious Like it you can own your own infrastructure now You're going to have to be worried about the power on on the bus on in the data center The networking into the data center you're gonna have to start worrying about now. So our risk register was about 22 pages long of items like this and Some of them a major some of the reminder, but they all still had to be Mitigated items such as Well, there's a who's gonna monitor who's gonna look after any of the security? Holes that are discovered in open stack or a component of open stack Who's going to be upgrading it? So and then what's involved in that process of upgrading? Well, do you have to bring the system down? Can you live migrate what's going on? So This should be identified you start this right from the sandbox, but it needs to be completed by the pilot To alleviate for design for high availability Don't even think about anything else if you cannot design for a high availability and in your components then don't even bother moving to open stack Because it's going to components are gonna fail hardware is gonna fail Somebody's gonna do something on the system. That's gonna bring something down So As far as the guideline goes have more than if it's possible to have more in one region that your Disaster recovery either as a live system or as a failover site Have more than well, obviously have more than one compute node. There's an awesome feature called live Migrate words of a If a compute node fails you can bring them up on a different compute node If you're gonna design it so you can have more than one compute node put them in zones have One zone per Tassie But no more than four zones and then place one controller on each one of the AZs data Durability if you One of the worst things is going to be you lose all your data all your volumes We went with cinder on sef and we went with a replication factor of three I mean there's three copies of All the data on the entire system that allowed us to do upgrades the systems If one got corrupted or one failed We had we had at least two other copies We used glance which is on on Swift as well And we also used a maids Swift multi-regional. So it would automatically copy it over to a different region Two regions for zones with a replication factor of four for a Swift Build a flat and fast ethernet fabric layer two There's The hardware is now becoming Less expensive and it's easier to get into a four-year or 10g equipment now and Bonded nicks off of some of the nodes and Low degree of over description. I put the sign up in The system in area no hacking at any time. So if you start hacking in the system and Changing out too much of the code or any of the code for that matter. You're gonna have some difficulty upgrading in the future I have a friend who works for a company and they're stuck I've been stuck on Folsom because they modified the open stack code so much that the features that they need they they built into it They absolutely need and they can't upgrade I told them if it's not a repository you cannot use it if you Want to add another feature or do something then you need to push the code up and have it accepted then you can start using it But otherwise There's a lot of architectural guidelines out there. Stay within the architectural guidelines whether it's either from the hardware vendor or Moranis or or Rackspace or whoever They're all widely documented. They've widely tested and they're all widely used. So just All the work's already done for you stay within them. You'll you'll be safe There's another hardware vendors out there that have open stack Deployments and I consider that a risk the whole opens part The whole advantage of open stack is getting away from vendor lock-in Having a vendor that insists that they're gonna you have to install their version of open stack Locks you in you've just locked in both the hardware and that open stack deployment and whether they Will add in features to it or you're basically back in the same situation you were when you first started out Partner with an agnostic support provider when I say agnostic I mean somebody is not going to push you towards a certain set of hardware or or a certain solution Yes, then stay within their architectural guidelines But somebody that's going to support you All the way help you install it help you to design it. It's going to Be a cost. Yes, but every dollar you spend is going to be well worth well worth it There are some proprietary open stack component solutions out there for different pieces either for management of a certain or deployment of a certain component Try to avoid them if not Then you know think think very very carefully again, that's going to lock you into their solution Set realistic expectations with open stack as you're moving through the pilots and the proof of concepts if you overstate or describe a Component or or a feature of open stack that Is not really there or doesn't really work as advertised You're going to have to you know after deployment What can you do? It's it's So I mentioned all those saw those other different features of of open stack that perhaps You don't have a use for today Well when you do your initial deployment don't deploy those they can be added in later So you want to reduce your complexity footprint? When you're doing your very first deployment first of all at your first deployment second of all You don't want to make it as complicated with with the other components you can add them in later and The learning curve is going to be steep So make sure that everybody understands in your team Understands exactly what's going on. They're not overstepping their their knowledge level if they are stop and identify the part that they might be a little weak on and Have them brought up to speed either themselves or through some other training and then and then continue on any questions Well, I didn't have that much I didn't have that heart of a time they Open stack is cool so It was I started off as a director very hands-on director so I started off with it myself and showing them on Putting a Dev stack on my laptop and showing them, you know, we're out for Lunge time or whatever and I'd show them exactly what the capabilities are are Tech guys like new technologies and this is very new And like I said, you can explain to them. It's very good for their career it's One of the worst things that that that Systeming can do is is stop what he's learning. So this is something new for them to learn Intellectual curiosity it it kind of just happened. So it's just really exposure And knowing that they have the support for it as well So the applications that we Everything revolves around a database, right? So the application that we moved first at one point We inherited our Infrastructure for the most part for the most part I did and then of course the guys that came after me did These systems were just Simply installed no documentation. We started building documentation for this as soon as we realized we want to move to open stack at some point we created runbooks and Decided on some technologies that we'd use to start the deployment Ansible For instance, so even if a new server was going into the current Infrastructure we would also in open stack in the lab write and build an equivalent So we had about 25 of our infrastructure that boom just moved over the rest we had to actually go through and like I said, we didn't build it we just supported at that point and and decipher exactly how applications were installed Some it was easy some it was very difficult The first ones we moved over Were the ones that were not dependent on the entire database infrastructure system. There were some supporting servers there were some servers that infrastructure used and The last part was the actual database applications and web servers and whatever How we moved it over we couldn't copy the images over how we moved it over Was rebuilding everything in open stack again, however, there's a vanish to that We actually got to rebuild everything document everything and build it correctly and more up to date Thank you very much