 enterprise session There's lots of open stack to the enterprise sounding sessions. So this is the Dell and Red Hat one. Hopefully you're in the right one Let me get it kicked off right now So I'm Steve Croci. I work for Dell that long title basically just means I own You know our open-stack roadmap what products we bring to market with open stack how we how we productize it and how we Sell and work with the open stack community Presenting the kind of second half of this is Sean Cohen My counterpart on the Red Hat side product manager principle product manager for open stack So what I'm going to talk about today is First of all, you know just look at how you know discussions about Enterprise and open stack and mentions of enterprise have changed over time Who are these guys? How do we define it between us and Red Hat? Who is an enterprise customer? What kind of things that they run a real short intro there? And then why is this kind of a good thing for the community? So, you know short intro on just you know what we're doing who these folks are why you know why we're working in this space and then really start to dig into You know how we work together Dell and Red Hat, you know What things we think are important and some of the lessons we've learned things that we you know Hopefully you can leverage and learn from yourselves And then where we're going next the things that we're starting to develop on now The areas where we're going to work into work on upstreaming and then some takeaways So, you know just drew a little chart of you know kind of the you know And a open stack getting more and more enterprisey over time So looking way back at the beginning, you know Austin Diablo, you know It was kind of a you know very few enterprise mentions not really big focus at the time It was kind of a you know, don't even think about it type of space, you know, it's not ready But once we hit, you know Essex and you know through Folsom it got into what you know Pretty much I'd call like a gathering requirements phase. So people started talking about okay What would an enterprise need, you know, you know, how can the enterprise benefit from this technology? And so it started to come together and it was a little bit more of well It's getting there. It's almost there and you know, but still, you know, you still had the you know Standard, you know standard open stack summit dress code You had t-shirts and jeans and a lot of the presentations But then you know things started to get you know a lot more corporate entities coming in so the golf shirts came in Grizzly through ice house and the talk was then you know, how can vendor X, you know enable the enterprise, right? So a lot of vendors started coming in and saying well Here's what we're doing to make this make this technology make open stack more palatable to the enterprise Here's that you know here the areas where you can benefit and that's where it was like it's you know It's almost ready. It's almost there We started to see some you know adopters in the enterprise start to start to pick up and run with this thing and some really good early Examples of how to use open stack in the enterprise and now you know, we hit Juno and You know, it's it's it's we're starting to actually hear here's how Enterprise X has benefited from open stack. We're seeing more and more talks about you know success stories or at least stories of people You know using it having some best, you know coming out with some best practices coming out with some some of their Learnings and really we're finding the core is ready now There's still some you know other products on the you know projects on the on the peripheral periphery that you know need potentially a little bit More work and may not be enterprise ready, but really the core is ready and you know Given I work on the on the product management side I look to you know Gartner every now and again and they made a statement recently that Juno is open stack 1.0 for the enterprise which I kind of like that statement I think I think it kind of makes sense and it's kind of directionally correct that right around Juno Things started to get you know kind of stable enough that it was a real You know 1.0 kind of product for the enterprise and so people could start using it People could really start going get some good reps on it and you know take it from there But it's still you know 1.0 gets a part that you know It gets gets to the point of view that there's still some you know You know still some places to go with this and so you know I talk about enterprise and like who are these guys, right? You know who's this enterprise that we talk about and it's like you know If I look at the you know Dell general picture asset catalog You know these are the guys that kind of come up and you know now that's that's that's not it So how do we characterize you know enterprise IT and the folks that we're looking to address and the folks We're looking to help with our solutions and so you know first of all there's no One definition it's not if you do this you're an enterprise if you don't do this your enterprise IT It's it's not that clear-cut so it's kind of you know It's it's it's a loose definition and it's more around how folks are using it the types of things. They're trying to manage and run and Also, you know fortune 500 doesn't equal enterprise so just because you're a big company that's not you know enterprise IT Very large companies operate their IT in very very advanced ways that don't have you know Some of the traditional challenges we think of when we you know when we talk about enterprise IT So there's really no you know hard and fast definition there at least in our words when we say we're making an enterprise grade open stack solution So what are they running? Well, we start to see and this is one of the areas where it really starts to get a little bit clear We see a larger responsibility to support legacy and proprietary, right? So there's you know, you've got some old applications that have been running for a very very long time may not you know Have an opportunity or really the ability to you know upgrade those or you know Use something that's newer or that's something that's you know potentially scale it out You still have some scale-up types of workloads You've also got some proprietary things either legacy proprietary things or just new-style proprietary things There's a lot of enterprise virtualization and the features that go along with it So you know these are the types of things to comprehend the other thing is There are bound to be pets so you have this pets and cattle conversation There are going to be some pets there now the question is and you know It's there's lots of ways to go with the conversation is which of these things should be an open stack Which of these things shouldn't be an open stack should you control them from open stack should you keep them separate? That's you know completely separate conversation, but it's the type of conversation that you have in this space And so you know what do they do well? There's you know you've got a whole organization to support you've got multiple business units and really you're turning it You know turning into a service provider for all these different business units So there's you know people are running their business and ultimately you are the you know You are the provider that for that and by the way you usually have some finite resources We see a lot more resource constraints in this space and therefore a lot more of an openness to leverage you know Vendor support and support resources so that's not that's not to say this is everybody But ultimately the challenge of you know running mission critical supporting the business with a you know Very tight budget and the pressure to expand and to support more and more You know more and more technologies from different LOB's with the same budget of the same folks That's kind of the challenge we see and so there's you know reliance on vendor support but now kind of a push to to You know try and do things a different way and so What is the community done so far and and you know Breezing through slightly quickly in the first half of this because this is really just you know a little bit of history a Little bit of you know storytelling on where we are today And so the first piece of it is just the expansion of the design summit, you know anybody who's been through the last Through the last summits, you know, you look at the pretty much number of people here is doubling every year So that's more people getting involved more people getting interested more and more talks as I showed in a couple slides ago on Enterprise and how to use these things There are working groups specifically dedicated to this group There's an enterprise, you know win the enterprise working group which met once earlier today They're gonna meet throughout the week. There's also I just left before this the product manager working group Which is how do we add, you know, how do we try and add, you know, some product management rigor to open stack without killing the You know the agility and the and the the way in which open stacks been able to grow as quickly as it has More drivers more hardware more devices. So that proprietary and legacy stuff I talked about in the previous slides. We're getting more and more support so that the you know The the question is no longer well go buy a whole bunch of new stuff to run open stack on it You can run it on the stuff you've already got especially in the enterprise. That's something, you know For example, that we you know, we hit on the Dell side. We have you know, equal logic Which is one of our enterprise storage, you know storage products. We support that codes upstream So hey, if you have that and you want to tie it into open stack, that's cool Don't worry about having to rip them replace everything and that's happening throughout almost every vendors come in to make sure When you enter the data center almost every technology that's already in there. Yeah. Yeah, we that'll work with open stack The feature set so adding things in some cases that are actually a little bit contrary to the you know kind of the original goal the original you know direction of open stack like Tenant, you know tenant ha and VM ha which Sean's going to talk about in a little bit These are things that people are working on to bring into open stack to be able to support You know just a little bit of an ease a lower hurdle to get you know The enterprise users working on open stack and using it You know if it take advantage of the operational model without necessarily having to change your entire application model And then there's also commercial distributions for with support once again running mission critical You know applications with a finite group of engineers Well, you've got companies for example red hat behind you to be able to you know kind of fill the gap should anything go wrong Okay, so now I'm going to dig into you know given this definition and and this enterprise group that you know We're trying to make it you know open-stack enterprise grade four Let's I want to talk about what Dell and red had have been doing together You know both of us have been involved in open stack for for quite a while But it was a little over a year goes by year and a half ago The two of us came together to really start working on integrated solutions to kind of address this space and so You know kind of the example that came to mind is I you know I've got my karate kit example there because really the things we've been focusing on the past You know year and a half is really you know doing the quote-unquote the boring things that matter most and you know the real foundational core You know repetitive things that open stack needs to just be stable and and and and able to you know run Get closer to production and mission critical workloads right out of the gate without having to stub your toe a bunch of times to get there So that's you know getting the core functionality right and I'll talk about some details there in the coming slides Making open stack repeatable, you know if we cannot repeat what we've done in our lab at the You know at the customer site or in your data center. We can't support it It's a nightmare if we can't see the same thing you're seeing so trying to find ways to make open stack repeatable So that not only between our data center and your data center But between your data center and your other data center you're able to get some consistency out of open stack Making open stack testable So there's you know a number of different ways to get from you know Bare-metal to a deployed open stack cloud. How do we test to make sure that things are you know? Just because it's up and running doesn't mean everything's up and running the exact way that you thought it would be running So working on making open stack testable best practices on configurations And I'll talk about this you know this in some detail in a slide or two is that things get interesting when you put it on Real hardware testing things in you know in a virtual environment or in a limited You know CI infrastructure doesn't really ring out everything so best practices what we've learned on configurations And then the fewer snowflakes, you know statement is not in any way any kind of statement around an appliance or trying to you know Really limit what you're able to run an open stack But it's just being able to drive some consistency being able to make sure that once again when you deploy in one place and you deploy In another place it's not going to change drastically from deployment to deployment so that you can expand easily So you can connect easily so that you can do all these things and so you know really the statement at the bottom is open Stack that just just works and you don't need the you know ninja or your mr. Miyagi to come fix it for you So full stack validation on real hardware This is one of the things we do and I point to the things kind of weird doing but I you know I want I don't want this to just be hey Here's how we did it just these are some examples of areas where we've you know We've had to do things because we've either run into bugs or issues or you know things that need work on so You're doing it yourself. These are things to think about so one of the things we've done is we integrate the you know Kind of the product or the configurations that we sell into the red hat ci so every every everything that gets you know Update on the red hat side goes through our ci Which is the actual hardware that we sell in the actual hardware in the actual configuration we sell now That's significant because you know I have the note here upstream doesn't test the full matrix Upstream tests everything and there's a ci that tests all the features But it doesn't test every single feature with every single other feature So every single you know cinder driver with every single hypervisor with every single piece of hardware with every single nick Etc. Etc. So we're testing the exact configurations that you know and and and validation suite that we've got With the exact hardware it configured in the exact way that we Configure it so once again ringing out those issues up front. You're going to see things as you get to real hardware Tempestum rally for validation. So there's a talk Tomorrow afternoon at 2.50 on this you know very topic and there's a couple a couple more during the week But we've been doing some some work with tempestum rally to build a validation suite with an open stack And that's another area where I mentioned earlier just because you've got it up and running Doesn't mean everything's running the exact way you expected it to so building a real comprehensive test suite with tempestum rally We do still have some tests outside of tempest right now But we're working to you know get more and more of that functionality in and today It's just a val it's just a deployment validator longer term goal is for it to be a continuous validator So you run it on deployment you continue to run it and this is very much in line with you know The def core work we saw a big announcement on that today Really getting it to that level where you can continue to run continue to run tempest and get to the point where you've you know You're validating your your cloud regularly And then finally ha and fault injection another one of those areas where you know Testing the capability of dealing with a failure is different than being in a lab pulling a cable You know Having having a hardware failure pulling a note out of service killing a service You know doing all those things on the real gear on real hardware is area where we've actually found failures things like you know Even below open stack pacemaker for example We've run into pacemaker issues as we you know that have had to go and get fixed as we as we really test all the failure modes in the ha system so Validating the full stack on real hardware is one of the areas that it becomes really important The other piece is you know hardware selection and configuration now That's not to say that this is anything that nobody in the world could go do But once again, it's the homework we've done up front to be able to provide you at least you know For someone looking to get get started quickly a good point of view on things known working in areas where we've seen problems and areas We haven't so for hardware guidance like you know There's there's some things that just matter more than than others when we hit you know Incompatibilities and bugs and issues when we try to deploy one area that we've you know seen it happen a number of times Is is nicks, you know Nick firmware Nick vendors all different types of things We've had deployments where you go and say oh, let me just switch out the next because this is my vendor of choice What could go wrong? It's you know things break things don't come up correctly. So, you know, we provide guidance around Okay, we're we standardized on nicks, but what server model it is less important, you know What how you how you set up your storage how you set up your raid? Maybe a little less important how much memory which processor you use less important So there's guidance around that and I just have you know some little pictures of the docs that we provide Which once again is you know I talked about a little bit before an RA that's more than an RA and we call what we do what we create with red hat a reference architecture But reference architectures are usually just you know sims is sort of a single configuration You know single way to set it up and ours is a little bit more modular We we have with it deployment guides and all the tooling and scripting and everything that goes with it So it's more of a flexible framework for how we're able to deploy these things or at least to give you a head start on how you can deploy them Second piece is hardware configuration. There's a number of deployment tools out there not all of them Configure raid configure bios configure all the low-level things that you need to get started once again to drive some consistency throughout the cluster We have some tool set at Dell tool sets at Dell that we've created some automation around to make sure when we deploy this thing It's consistent every time, but we're also pushing that support up into ironic So, you know for as things like triple O You know start to gain more traction will have it in there But also just for running bare metal instances that support is getting pushed up as well And then finally, you know just the network configuration not, you know anything that's that's you know Rocket science here, but once again having tested it having run through and giving you a starting point in a template That's known to work and pretty much, you know Focuses on balancing performance and scalability and resiliency and isolation and it's not one of those You know like you know you can see in storage where you know you got three three aspects you could pick two We've really been able to balance all of these without having to sacrifice in any major way one over the other so, you know, we run For ten gig connections per node We run dual ten gig bonds an example there gives you the resiliency. You've got failover. You've got high availability But you've also got the throughput. You're not you know You're not having a passive connection and you're not wasting that extra connection We use VLAN type for neutron. We found that it's you know, that's the that gives us the best You know kind of balance of performance of scalability, you know once you get really big and you run out of VLAN Okay, if you've got a large number of tenants, you know Then we start to push into other things like the excellent but for for the you know for someone going into production and Without a you know huge large number of tenant networks VLAN works quite well, and it's actually pretty speedy and then finally You know the way we spread our services. We have eight dedicated VLANs. I could show the picture here This is in our reference architecture So you know you can go to there's a link at the end of this to go pick it up But ultimately we've just architected it all out to balance across all those factors. It's scalable It's easier to take these VLANs and pop them on to new nicks as you get bigger and bigger And once again since we're aggregating on multiple connections, we get the bandwidth improvements We get the redundancy improvements, but we also get the isolation of having everything VLAN separated and so, you know didn't want to make a Product pitch so but this is just a quick, you know what we how we sell this today We you know for kind of getting rolling We've got you know you can just get quoted immediately a half rack to three racks beyond three racks We usually like to talk to you because there are network considerations There are a lot more considerations once you start to go big but half rack to three racks It's pretty standard architecture running the latest red hat open stack platform latest Dell gear We've recently just Moved a neutron we moved a neutron a little bit later But once again, it was another one of those scenarios where we didn't you know We stuck with nova network up until Juno wasn't until Juno that we really felt that that that neutron was it was you know stable enough and I offered exact and Had ha capabilities and all the things we needed to be able to stick by our ha story and our enterprise grade story So now we're on neutron once again fully active active ha the last component to make us fully active active Came in Juno with VRRP so Sean will talk about that in a second So now we're fully active active and then one of the other areas that we spent some resources in in you know Kind of up streaming is around multi-storage concurrent What we mean there is having multiple sender back ends at the same time So given that our default storage in this is set the default storage back end is set for for our our configuration We still support Dell has equal logic which we've upstreamed the drivers for we also have Dell's Component products and you know which we've upstreamed the drivers for in kilo and other vendors You know enterprise storage so once again moving into an environment where you may have Various different storage technologies one of the areas we thought that would be very important is to be able to connect all those Into open stack at once and not have to not have them all you know Have them be able to use with QoS and other types of features So at this point I'm going to kick it over to Sean who'll kind of work up the stack a little bit more and talk about What we're doing next thank you Steve all right, so we talked about the integration with the hardware and Steve actually walked us up to pretty much the software and this is where I want to zoom in and talk about The work we've done in the core engineering and integration port, but before we start I want to actually touch upon the open stack side and the importance and I would actually would start with a Question how many you focus in the crowd drive cars Raise your hand all right most of the crowd how many car owners Are your cars are operated actually by computers? Most of them right what happened if your computer your car computer goes? Not working right like a failure breaks something Who would you go to fix that brake failure which is computer related when you stop any any shop on your way out From work or you would go to the manufacturer that actually knows the compute the car computer to actually to fix it And when we look at open stack just to stand up open stack You probably need something like nine services to talk to each other just so we have a consistent deployment This is just open stack layer But if we don't dug deeper into open stack open stack is pretty much a pluggable Infrastructure that talks to other layers below to do the work So if I'm going to create my volume in open stack I'm gonna act open stack actually goes and ask liver to do the dirty work and create the volume for me And if we look down the stack Open stack has a very large dependency of the operating system. It relies on top and The reason I bring it up in the last years We have customer cases where we started up finding an issue and when we dug it's an open stack bug It's an open a set cakes we started to dug into to the layers We found the problems in underlying layers all the way to kernel. So if I have a kernel problem, right? as part of my cloud deployment, who would I go to and Right at this being one of the leaders of open stack In terms of code contribution is it's not something that we see yeah We would like to stay as a top contributor. This is part of what we do We need to stay behind the code in order to support it and we support it for few years Right now and the support actually goes if you look at the stack It's throughout the whole stack. So it's stuck with actually the guest OS That's runs on top of open stack throughout the open stack itself with all the different services and components all the way to the operating system and The hypervisor so in our cases KVM, which is the most common hypervisor We see today in open-stack implementations, but it's not doesn't end there because we as being or prison working on providing rel We also have a very tight integration with the hardware community So we are have a lot the largest set of hardware vendors integrated and certified Against what this brings us through the table if we work our bottom up is Whatever I plug into open stack have a quiet at night that it's going to work because it's certified We actually make sure that whenever you connect to Your rail KVM open stack all the way up to the guests actually works in terms of hardware, right? So that's that's something but you also are able to impact on each one of these layers And I think the strength of our distribution just in a word is the ability to actually do modifications That's just supporting and bugs and issues But we actually be able to do performance assessments for example at the hypervisor lever To benefit the cloud infrastructure for example, so we are able to do Impacted different layers of the stack to actually to bring more value So if we connect that take for example as a Linux Which is the security? Militia great support security that we have built into rail and it's part of of open stack And we talk about enterprise right so if you're an enterprise customer What are your security requirements to deploy open stack? So you have a different set of knees that needs to be addressed and when you took about Connecting it all to the hardware we have a full picture What next so we identify three different areas that we would like to focus on so as Steve mentioned The blueprint we have right the cloud solution that is validated is already available today, but We believe there's still things missing and there's work to be done in order to get us to that enterprise grade Level that we want to give you as customer to have the state of mind Quiet state of mind and there's three topics are high availability rolling upgrades and deployment and we'll start with high availability Which is breaking into two Which actually deals with both the tenants high availability and the open stack services themselves have availability and if Active active mode, etc Rolling upgrades that's probably the number one biggest headache we have today in open sack, right? It doesn't matter. Where did you jump on the open-stack train in what version right all the way to kilo release? How do you upgrade right so I have let's say I've jumped in key Juno and or Diablo whatever I want to upgrade now to the latest and greatest release How does my upgrade gonna look like right? I have to stand the whole cloud environment again and regrade my Would I be able to do it in a more CI level continuous integration? That's all right So that's a very problematic area that we are focused on With minimum data, of course deployment Open-stack deployments are very complex and as you can imagine it can be spread around the world in different locations It's era, but we would like to get to a point where we can automate the process to almost like an easy bottom deployment tool and That's the goal, right? We're doing everything that automated and what about deployment? That's something that is currently missing from open-stack, and we're seeing as the next thing that we want to focus on So we'll start with high availability of services and what have already availability of services means It means if I shut down my compute node. No vibe. Just kill it Everything is still working my VMs that was hosted using that VM Controller actually being able to meet it automatically to that and no similar to what we have today in traditional virtualization But it also mean that I need to my cloud implementation need to survive any poor and fully if I Steve mentioned that pulling Cables out right that we're doing as part of the validation of the cloud infrastructure Your whole site can go down. It's not just one node Which this is why we have built in AJ, but the whole cloud implementation go that can go down And you need to be able to do a disaster recovery So when I AJ is the first step for disaster recovery and and today we have a lot of active active already Built into open-stack services, but we still have some holes as I'm going to mention in a second And I'm going to give you some example how we utilize for example pacemaker As well as AJ proxy and our horizon VIP So I'm not sure how many of you seen this slide before But today that's pretty much what's happened under the covers when I'm getting a service using fine my horizon UI dashboard and guess what one of my Controls goes down so so I have basically a controller pairs being in a j-mo where we leverage AJ proxy And the way we do it we actually leverage a virtual IP as well for arising So we basically are able to route all the traffic from one fit one pacemaker close to two and a If we look at the high availability This is where we had gaps until The junior release if I had a cloud Network agent going down and that used to give service to specific tenants guess what these tenants cannot get its service Because the know was done so we needed to fix that problem and the fix was came in a virtual Router we've done the C protocol of your RP So it's already supported in relatively six and it pretty much provides us the aging network Built in already per tenant and we have a keep-alike process Per virtual routes and we are able to actually route The traffic upon failure That's pretty much how it looks so we have virtual master and then we have backups on the other nodes and When when one of the nodes goes down it can be automatically served by another backup virtual router All right, so we talked about the service high availability Let's zoom into the tenants high availability and I would Just gave one example because we have limited time and we do I do encourage you There's a lot of sessions other sessions that go in much more detail to deep dive to each one of this But just to give you some highlight so you can feel where we are so we started with Juno So I talked about the VRP. That was a big must and was added because that was a single point of failure In kilo which was just released We're now focus Focusing on on providing instance high availability using pacemaker. So if we have we need to migrate our VMs if our whole I-provisor went down we need a way to do it and we're gonna leverage pacemaker and of course there's a whole fencing mechanism happening to detect the failure and to do the The trigger all the way up to Nova so we can actually perform the operation Moving to the Liberty release some areas that still need hardening is like migration So I just mentioned hypervisor going down right and the unit can grow all the way to as I said to a site fellow We need a way to make sure that automatic evacuations of my instance happen in all states Including non-active states and believe it or not. We're still we're still not there. So There's a way for there's a still way from a way for us to go to actually make sure that all the cases of VM migration are there and and sadly it's not we have something like six blueprint just on hardening the Ability to live my great instances If we look at cinder Cinder is until this release is pretty much the only core service that still doesn't work in active active it's still active passive and So that there's a lot of work to be done There's in this release. I won't go into all the features that being Addressed, but there's just to give you an example if I have a volume I'm not going to create a volume using the cinder volume and I'm gonna shoot the cinder no during the process I would be stuck in that state if I'm gonna go and Create a change of quota and sender and I will shoot the cinder by so node boom absolutely stuck in that states forever so we still have Few states that are not there and dealt with and the way to deal with it. This is actually task-close So there's work right now been done on volume persistent support to actually mitigate it so it will allow to operate is simply shut down the cinder API and For maintenance, right? So we talked about different kind of fellows from different reasons But this is something that is useful from operational. I need to be able to do this just to maintenance, right? I need to do upgrades it's there updates, etc I need that to work and this is something that we're working on so we can actually resume our work after In the block storage service moving on the third topic I mentioned is Deployments and upgrades the two and third and I will start with deployment and when we look at deployment Typically fear. Yeah, we need to install open stack. That's our goal. Guess what? That's our starting point now. You need to manage your open stack environment And the focus is on is basically provide tools That are undatified by open stack operators in production to control in debug So as an operator, I need a way to know first of all a Failure in my side one of the nodes went down. How do I know about it, right? To how do I debug the problems what log a great aggregation lock search tool? I have available out of the box to actually support me maintain my open stack So we again just to zoom out. We are discussing enterprise grade open stack How many of you run production environment today not even in cloud if something goes down You need to know where and if we go back to the analogy the analogy I gave with car if I have any problem with my brakes, I would get an indicator on my dashboard This is what I'm talking about. We need that indicators. We need that in dashboard indicators to know what's going on in our cloud environment When we talk about out of management tools, so I'm running an enterprise environment I have different tools and being able to standardize open stack APIs to allow us to manage and use other management tools as part of a deployment. That's key, right? We don't we're not open stack as a scenario is all about being a pluggable infrastructure management and deployment management needs to be part of that a open API I Want to add or remove capacity, right? I need tools to allow me to do it on the fly without bringing out any service downtown and actually allow me to Bring to life the elasticity promise that opens like rings, right? If I am not able to do it in a consistent sway and easy to do away What am I doing? Providing control API to extend what I mentioned, but the last thing is actually updates and when I talk about updates I'm not just talking about All right, we have a software of update for open stack, but also patch management, right? I'm as I mentioned earlier. We have a very strong dependencies to the hypervisor OS, etc. I need to be able to run Security patches. Let's say there's a severity one security Bug that needs to be addressed. How do I roll that? But bug fix which is critical in zero day, right? Throughout my cloud I need consistency tools to do that to allow me to do it on a regular basis so To capture the focus we're doing is on the default development side is actually 366 year life cycle that includes both updates and upgrades as well as capacity adjustment And this is where I actually want to give you some hint in the next real OSP version That's going to come up this summer. We're going to introduce what we call the real OSP director, which is a new deployment tool it's based on triple O which is the operated Way to deploy and manage your Underclaw and opera cloud as well as inspired by other tool Which is more see ICD tool It's called spinal stack and this is coming up in the next release and it will allow you to deal not just with the deployment but also bring a lot of management tools so you can manage better your cloud and what's what's going on in my cloud as well as Allow me a way to do updates patches management, etc. and of course trying to hit the bigger problem, which is upgrades and Allow me to stay on a Consistent place where I am in open cycle. It's our now Juno release I want to be able to stay that for a year get all my patches and still Have my production stability But when I do want to upgrade to the next reason by the way the fact that open sector is a release every six months Doesn't mean I need to move all my investment every six months, right? I still want to have the stability get my Backports it some of the feature are being back to that actually we're doing it for customers per release when it's available available But it also allows us when we do want to move to the Liberty release, right do it in an easy way without Taking out the whole investment in setting another cloud and migrating all my instance into the second cloud and with that I want to actually Summarize The solution and what we've done just to zoom back to the thing that Steve actually was describing earlier in the work We're doing so we are taking open stack As well as the hardware and we're doing a lot of work just to integrate it to Throughout the cycle we found a lot of issues that believe it or not are not open stack issues Steve mentioned the the pacemaker box for example bonding in networking was another one so it allows us to actually harden open stack and get it to be a more production ready and If you want to hear more just go ahead and scan the barcode the reference architecture is there And it's very detailed as Steve mentioned. There's it's not just one configuration It's different configuration that allows you to grow and we have different network topology different storage topologies everything is outlined in the reference architecture and of course we're here and now it's time pretty much to open The question Q&A. All right, Steve, you want to join me back? All right, if you have any questions, please use the mic at the frontier so everybody else can hear Everybody's taking picture. It's just good. No, this guy So any question technical Any level? All right, it means you did a good job So thank you very much