 Yeah makes this guy look tiny Okay, how are we on time? Are we ready to start? Okay? All right. Thank you very much Thank you for coming out today. We're going to be talking tonight today about planning and enterprise open-stack deployment My co-presenter today is gonna be mr. Jeff Applewhite from NetApp over here. We're gonna start off I'm gonna talk about just some general Things that we've we've been you know Susan's been doing Open stack for what three and a half years now at this point something like that and we we've had lots of conversations with customers and We've seen that there's a couple of common themes that have come out in terms of what people need to do to be able to prepare for an open-stack Deployment, so we're gonna talk about some of those kind of general things from my standpoint I will talk about some specific details about how what decisions you make now how that's going to impact you down the line But but it will be somewhat general and then Jeff is gonna actually talk about Some work that he has done on reference architecture with Susan cloud and Susan opens that cloud and NetApp and some of his real-world experiences and and how he can help how how he managed to to do things so Should be fun should be exciting Right everybody's excited Now just for the record I had this a lot more fun I had like all these cool images in my deck last night that that I had like little comic book characters For each of the different things I was going to go through you know like you know when you're you ask yourself a question There's a picture of the Riddler and stuff like that and then I chickened out I decided that I didn't want to risk getting in trouble for you know copyright infringement or something like that so It's that I do have one image, but but but the most of them are guy took out There we go. Yeah. Okay. Got turn it on first. Okay, so we talked a little bit about About the agenda already, so let's go ahead and dive right into it So first question. Why are we even bothering to do an open stack deployment? Right? I? like this little this little cartoon right here because you know This is kind of the typically what happens in an enterprise right the the grand mantra for for for large enterprises Don't get fired right and there's the nobody got fired for by buying IBM I'm gonna buy windows because that's what everybody else is doing that even happens in the open source world where where People will say I'm going to buy go go with this particular Linux distribution because that's what everybody else is doing But at some point in time you had to figure out how you're going to differentiate from your cut from your competitors What are you going to do differently and in that point? You can't just this expect some magical thing to happen. It's going to take some time and effort now in the In it right now it is really kind of be seized kind like our superheroes right here By the people around them right IT's kind of under attack You've got the the CFO people the finance people are saying cut all your costs I'd wanted how to cut your budget in half, but I want you to deploy 40% more services Next year you have the line of business people that are saying you know you guys it takes me six months to provision a VM I can just go run my credit card out on Amazon and spin something up right now And if it breaks, and I'll just make you fix it later Even though you knew nothing about it in the first place anybody else had that happen Okay. Yep. So So the IT is really coming under siege. You know that kill them all All right, so let's talk about just some of these some of these questions you need to think about We in my experience in talking to customers and successful people 99% of your Time for law for implementing your cloud is going to be spent in meeting rooms Right the actual and you know for those of you who are here for the for the for the set the previous session when Cameron had his his appliance we have for speeding up the process of deploying an open-stat cloud and It's you can we can we can literally deploy a cloud out. We consistently can do it in and a day Maybe two or three The longest ones that we've done have been like two or three because customers keep saying oh well You did this we'll let's try and do it with this and shift the requirements a little bit But we can do that quite quickly So most of the time that you're going to be doing and what's going to determine your success is all going to be in the meeting rooms You have to be talking to a lot of different groups You have to involve all your storage guys your networking guys your end users your finance people all these people have states And what's going to happen in your cloud and so you need to make sure that you are talking to each and every one of Them so that when you get to the end of the process There are no surprises for anybody All right, so the first basic question you have to answer for yourself before you can go out and deploy your cloud is What problem are you trying to solve is it? I need to sound relevant in the press because I'm tired of analysts asking me what I'm doing and I don't have an answer around cloud That's one possible reason for your goal That's not necessarily the best reason to invest and do a multi-million dollar cloud deployment But it has happened before Usually you want to say okay our problem is that our developers and our QA people they need to be able to iterate through things faster And so we need to be able to provide these services for them. That's awesome Another one is like we talked about we've got the guys over in the line of business that keep going out to Amazon and Spin up instances and we need to make sure that we provide them the services that they need to be able to do the self-service Stuff so they don't have to wait around on it anymore And that can be a very different cloud from what you're doing if you're doing something that's targeted towards developers and QA Right because it's different levels of people And different skill sets So your your storage decisions might be different your performance Optimizations may be different so you really need to understand what problem it is that you're going to be trying to solve with your cloud Otherwise you're going to be starting to run into problems. I need to make sure I've got a timer up here So I don't I can talk for hours on this stuff. I'm Okay All right, and I've already used up 10 out of my 15 minutes. We were only on the first question Alright, how much of your current infrastructure do you want to keep in your current workflows? And what I mean is Implementing cloud is a lot of change, right? There's the whole you know, we've got the pets versus cattle mentality And we want to go to a cattle mentality so that we can just throw out any given server at any given time and nothing's Going to happen and then it's all going to be unicorns and lollipops for everybody But in reality when you're trying to mitigate risk, which is what most large enterprises are trying to do You need to be able to leverage your existing investments I just spent five million dollars getting my VMware stuff all paid for for the next couple years I got spent a hundred thousand dollars in training all of my staff so that they can be up and running If you go to your CFO and say I want to throw that all off the door and let's go go get KVM set up And retrain all of our staff and reduce their productivity for the first year while they're trying to figure out how to adapt to the new hypervisor in the new infrastructure Can you cut us a check for that? That's going to be a much more difficult conversation than saying okay We've got our VMware stuff right here. We're going to go ahead and leverage that We may add in some additional KVM on top of that so that you can do things like You know let's run our production stuff out on our tour of VMware But we can give the devs and the QA we don't want to pay the VMware tax for those guys So let's go ahead and put them up on Xener KVM Maybe you want to have some hyper V out there as well So that if you want to have so that if you have windows workloads, you don't have any incremental costs for spending up additional Windows workloads on those all those things all those different hypervisors you can run concurrently under open stack and it's all supported by Sousa and that enables you to to be able to To be able to leverage your existing investments and simply add on to it which which mitigates your risk Now the nice thing is because you can do and say your KVM and your VMware at the same time You can gradually spin up more KVM and take down some of the VMware as you want to transition slowly over and a natural progression and through your natural attrition You can do that, but it also gives you the escape hatch that says if I'm doing the KVM stuff And it's not giving me what I want I need to have my DRS on my VM where I need to be able to integrate this into my existing disaster recovery scenarios Having that escape hatch gives you the ability with open stack You can just simply transition more of your hypervisors back over to VMware And it's not going to change anything on the end user experience for your for your your line of business All right, so then then we get into the what are the business and technical requirements of process project? So this is things like okay. What are my SLAs right? What are what are my end users expecting in terms of downtime? I'm not running are my e-commerce platform on here like Walmart was has been talking about In which case, you know if I'm down for two minutes then with them. It's probably like a billion dollars that they lose I don't know what it is. I probably should they're my customer Any case But yeah, you know is there you can have that level of uptime You're gonna require for your cloud or is this gonna be more of again a more of a If it's down for 30 minutes because we're rebooting control nodes Then then that's not a problem. These are these are the kind of things that you want there Well that I'm talking about here in terms of business. What are your SLAs? What are your in it? What are they looking to demand from you? constraints for the project Generally, you're you know, there's the you can have it what you can have it fast You can have it quality or you can have it cheap and you can pick two of those three at the most and usually the Usually somebody one from the finance wants it to pick all three But at some point you have to figure out what what are the actual constraints on here? I know do we have a specific deadline that we have today hit, you know going back to the Walmart thing They were talked about there, you know, their business is cyclical They have a they have a shutdown that happens in November where everything freezes and you don't touch anything on the it Until after January, right if you have a if you have a hard and fast date like that That's going to change your approach compared to if it's okay It can take as long as it needs to but we just have to keep the cost down as low as possible So that you have to weigh those competing priorities because that is going to impact how things come out What additional resources you need to have to have a successful appointment, this is kind of my generic catch-all but This is something online along the lines of Do I need to make sure that I have? Do I need to add in temporary man count do I am I gonna need to have consulting resources? Do I have the expertise in house? To be able to do this or do I need to farm this out to somebody else or do I need to hire? additional people Understanding what resources you're gonna want to have is again going to make a large even directional Destinational change and in terms of your cloud deployment, and I know that these are very kind of broad and overarching Things that I'm asking you to consider but in all honesty all these questions have come up and all of the People that I have talked to you about clouds in one form or another But in the in terms of answering those questions the why the answers have been so wildly different Between all of them that it's really hard to say okay Here's a list of five things that you need to make sure that you do in order to have a successful And make sure you have you know your net app storage or your self deployment or make sure you have this plug-in for neutron The all these things are going to be different depending on what you vote depending on your specific location Not location Situation yeah, there we go Okay, so this right here is just kind of a diagram. I put together on an airplane a couple weeks ago Just kind of laying out. This is kind of this the service Architecture of what a cloud deployment is going to look like right so you got your your control plane here in the middle You've got your your your height your VMware Your VM hosts down here in the orange or yellow And then you've got your external stuff that you're going to be integrating with whether that be tools in The Susa world that's a Susa studio or Susan manager that you're going to integrate with that Maybe you have an external v v sphere cluster that you're integrating with All that kind of stuff is out there Okay, so how does it all come together so let's let's from a technical perspective Let's start looking at some of the specifics here of how do you want to group the control plane services? So typically You're gonna have as Cameron was saying in the in the in the previous session Usually what you say is you want to break out specific services such as your database and your Your networking and you want to be able to have those on their own independent clusters and Cluster usually needs to be three nodes so you can have quorum and stuff like that That means that if you have if you've broken out that the the database and the networking and then you have this stack for the remaining services You're talking about nine nodes just for the control plane. That may not be financially feasible So you can do some stuff for example And this right here. I have a four node cluster where they're all clustered together But you can use affinity groups to pull some of these know all these services by default out onto separate nodes And then if a node goes down it will start to combine things to get you up to keep you running while you fix the other nodes so you can do some stuff like this to kind of Save your costs if you don't want if you can't invest the infrastructure to have the full You know that dedicated service for each thing you can use affinity groups inside of your Inside of your cluster to be able to to take care of that And that that does bring that bring up the point of is a highly available infrastructure important You know, do you want to invest the time and effort in this? We have as soon as we've actually taken a lot of that time out with our admin appliance for our crowbar appliance for for deploying it's That is this drag-and-drop my my stupid little marketing thing that I say is if you can operate a power button You can operate a mouse then you can deploy your cloud and you build your cluster It's not that great, okay all right, so You'll have to do things like to find out your shared storage. What do you how do you want to do that? What's stoneth mechanism anybody know what stoneth is in the ha parlance? Okay, stoneth stands for he's he's raising his hand. He should he's actually done this Shoot the other node in the head. This is what it stands for. It's a fencing mechanism So if you have a cloud if you have a cluster and you have three nodes If one of them starts acting crazy the other two turn around shoot it in the head Put her down because you don't want that one that one node to start interfering with the with the operations and the services running on the on the the rest of the cluster So you there are a number of different ways you can do that You can tie into say the IPMI device and actually pull the power from the IPMI You can actually some of them you can tie into the the actual powers the power controls for the APC's and whatnot and you can pull the power that way you can do and Via SSH you can do there's one called SBD a split brain detector that enables you to do There's there's lots of different methods for doing this If you have questions about this, please feel free to come talk to us and we have some Suggestions, we may be able to help you work out How am I doing on time? Not very well. Okay, what how much time do we have we were going to I need to give you some time, okay, so Capture the side what hypervisors you want to support we support as KVM's and VMware and hyper V you can do all those concurrently I gave one scenario earlier where you would want to have those kind of things How do you want to handle storage if you're doing something that is very high-end? it has to have guaranteed execution times with low latency and Good performance you're going to want to have a NAS backing in backing up that backing up that infrastructure Whereas if you're going to be just doing just you know standard dev QA. That's not necessarily performance So that doesn't have those performance requirements Maybe you can do something like a sep or a swift as your as your back end use local disks. What not not him He's from NetApp he wanted He wants you can use both of them together you can have you can have multiple storage back-end so you could have You can have your you can have your high-end stuff rolled out on onto your NetApp or whatever And then you can have the lower and stuff roll out onto your onto yourself. Okay What neutron plug-in do you want to use? This is going into impact spiff various things in terms of your hardware interactivity. There's you know, there's direct If you're going to be using VMware for example, you have to use the VMware plug-in in order to be supported by VMware So you would need to do that and then also if you're gonna do that You need to make sure you use the VLAN encapsulation tool not GRE because vSphere does not understand GRE and you will not be able to communicate effectively or at all with your VMware hosts. That's that's kind of a bad thing I would think so Okay networking so These are the networks as it's commonly set up using the the SUSE setup and so these are all we have defaults That are in there and these are all really just kind of based around You know getting it out the door kind of stuff. So I want to walk through each of these networks really fast and And talk about how your decisions here will impact you down the road so that you can plan appropriately for your networking So the admin network, this is the network that you use for doing the actual pixie deployment of your your bare metal systems And whatnot because it is uses pixie pixie does not understand VLAN tags Right, so it cannot be a tag network It either has to be a flat physical network that can be isolated or has to be untagged to the ports You have to decide how you want to do it, but you have to do it The fixed network what this is is the internal communications layer for the cloud is it's the default network that every Guest workload by default is going to come up on you can tell it not to go up on there But for most part everybody's on this one So your subnet range on your fixed network right here that is going to directly impact how many VMs you can deploy on your cloud So if you pick a class scene network, there's a 24-bit subnet mask How many how many instances can you run in your cloud? 253 because you have to have one for the router. That's a trick question But yeah, but you've given yourself a hard upper limit You don't want to make networking choices that are going to limit you down the future So you need to take that into account The STN this is for the neutron stuff again, how many Overall network connections do you want to be able to use within your in your system? You need to make sure you give it an appropriate thing public. This is the the This can be the smallest one this the public network is used to expose the API in service endpoint services to to the outside world so at the very most You were talking about 1315 IP addresses if you have a single You know if you have a single IP address for each the different services You break them all apart. You could have as many as 1315 Different ones in in practice. You'll probably have about three or four So it's not it does not have to be very big the floating network However, this is how you expose the guest VMs to the outside world and this one is important Now the floating and public they both have to be within the same subnet range Okay, so you'd reserve the first X number of addresses for your public network and then all the rest of these over here over for your for your floating network the number of addresses that are in your floating range are going to directly impact the number the number of Guest workloads you can expose the outside world expose outside of the cloud. Yes Yes Yes, no they have to overlap they have to overlap so So you would have for example by the default The public is like 192 168 126 dot two through ten or fifteen or something like that and then the floating starts at like 130 or 150 or something along that and has a range of like 50 or 70 IP addresses in there But they're both on the same subnet because they have to be able to communicate with each other And they both have to be exposed to the outside world and you want to do that through a single continuous So you have to make sure that you have enough space in whatever net mask that you're using to be able to handle both of those networks concurrently And then storage if you're Swift, this is going to directly impact how many Swift nodes you're gonna have you know if you have again if you have a to a 24 bit sediment mask You're limited to 254 Swift nodes that you can use so All right, so coming back to our Diagram and honestly I cannot remember now because I did these slides a couple weeks ago And I can't remember why I came back to this diagram, but I'm sure it was important at the time Just a reminder that we've got you know the blue right here these guys of the networks So again, this is just kind of illustrating the fact that the admin network is going to determine how many of these hosts You can have software to find network you have a little bit more flexibility on here Then you've got your your flexible network and Fixed network is in here as well in the public floating network to expose out to the outside world. All right Additional considerations. We're gonna move fast identify patching strategy. Are you gonna just in and admin server? We have a bar clamp, which is a plug-in that says I'm gonna automatic I'm gonna run out patches automatically you can do that you can do Susan manager You can do a manual process you can do the cross your fingers and hope you get away without it Which is what most people end up doing for the most part Then you have to think about you know if I'm gonna do this as a proof of concept versus a production What are going to be the differences between those those? deployments and am I am I okay with those differences from an evaluation standpoint, you know your networking you don't necessarily have to be as as You have a lot more room for error on your networking when you're doing the POC versus a production You may do just a single control plane or put all the have a single cluster and have all the services on one for the POC, but then for production you spread them out You may wait to do your active directory integration until after you're doing production those kind of things Additional additional operate if you do all these steps Fist pump boy is gonna think you were awesome, and he's gonna tell the world about it This is gonna be your CFO because he or CTO or what whichever one that you roll up to who's going to Thank you for putting together this cloud Probably not gonna the C level people don't usually do that to IT people I've been the I've been the IT guy where the CEO decides to pull to to move up the moving from moving offices from two weeks from now to today and Then while we're doing that Doing that move for them He threw a party to celebrate the move that none of the people who were actually doing the move could go to because they were actually Doing the move so we're sitting there racking up servers and screwing stuff in and bolting stuff in while they were eating Sip and margaritas and stuff like that. Yay All right, you're awesome. Okay. We do have documentation Susa comm slash documentation Susa cloud 5 or if you just go to the comm slash documentation You'll be a you'll be in shape there to be able to find all the different stuff that you need All right, so that's it for me. I'm gonna turn the time now over to Jeff who's going to speak really quick See if I get some sound here. Yeah, okay, that's good. Well, I'm getting hooked up here. Let's see do I have Briefly tell us tell a quick story here while I'm while I'm getting set up. I thought I was trying to think you know how would I go about Explaining, you know what why net at with with Susa with Susa cloud 5 and Wow There's a kind of a repeated thing that we do at net app and our senior leaders training and I wanted to quickly tell the story I do have mostly technical content here, but I will I'll move on So we do this thing where they bring in all the engineers and the senior leaders to get them together They they they separate me into groups. They send down they bring in boxes and in the boxes are bicycles And they give them a tool set but the tool set doesn't have all the tools They need to actually to make the bike so this creates a situation where the groups have to work together They got to go around and you know borrow a bag and steal to get what they need to get get it done Thank you. They You know so and then they don't have very much time So they don't have they don't have the tools they need they don't have enough time They got to put these bikes together and a limited set they're there with all the high-level execs You know Tom Dorgon's our CEO is there and they finally get a point where okay? They call time some of the guys are the guys that win you know or get their bikes together or very happy and proud of themselves and Okay at that point, you know and some people just have a pile of parts because they don't know how to put bikes together So it's a big range, you know at the end of that they come in they take all the bikes or the parts or whatever way and they Say great job But what you didn't know is that we're giving these bikes to school children So they income to school children and bring the school children in these are the bikes you're gonna get and so there's a big Wide range of reactions. You can imagine the guys that did a good job or thinking Well, we put that bike together really well, you know the school kids gonna be riding this thing The guys that didn't do such a good job or you know cut corners. Maybe they didn't put their screws in tight enough They're sitting there. They're thinking to themselves Man wish I'd have spent a little bit wish I had a better bikes building skills So the the reason I tell this story is that's how we look at building our products in net up And so why would you pick net up and you can build, you know, obviously with a lot of other things? So our value is that we this is what we've been doing for, you know over 20 years Day down tapas a very long history We I mean so when we're building our our storage appliances, we're thinking about you know These appliances are gonna be running medical workloads. They're gonna be running, you know People's SAP installations. They're gonna be running payroll, you know It's not acceptable to say Sorry, you lost data. It's just not it's not something that we're built for so that's the value prop And now I want to dive into the actual so now I've kind of explained Why would you pick net up to use with this Susie up in cloud if you have important workloads? You want to run in a cloud? This is an architecture that will support, you know, two to two hundred compute notes compute notes because as As we were discussing earlier, we have a three cluster design where you have a data cluster composed of three nodes a Services cluster of three nodes and a network cluster of three nodes So obviously a lot of a lot of horsepower a lot of capability and bandwidth there to to support a large growing number of compute notes down here at the bottom and Then obviously mediated by the Susie up in stack cloud five installer, you know It does all the pixie booting discovering and all that so let me move ahead here Just a brief overview of net at architecture. So basically, you know, you think of a faz net faz unit It's a physical box. It has diss in it and moving from the bottom up as far as our architecture We have aggregates, which are basically, you know groups of disc raid groups you could or groups of raid groups You could think of we have what are called storage virtual machines Which are kind of a grouping of IPs and storage that can be moved between these Physical units so they're actually virtualized. They can be migrated on the fly without without an outage You can so it's non disruptive operations of what we're all about as I was saying we're building these things to be up If we need to do maintenance and know we can move actually move resources off of it, okay? at the higher level you have flexible volumes that sit on the aggregates here and then Finally you get to the where the actual services, you know hit the network Or you're either serving NFS or ischia which we're using both of those protocols in this design. So We move it, you know, we also have the capability have mixed disc type on the back end So if you need flash you need spinning media, you know, SATA discs for archival, whatever we can support anything flash to sasta SATA whatever you need Well, a lot of capabilities including deduplication, you know instant snapshots without performance hits what flex clone technology which is basically a flex clone is a Rightable snapshot so you can create a snapshot immediately start writing it and then the data that gets written is just the difference between The snapshot and what you actually need to store so As far as the deployment process here, there's some preparatory steps I linked at the bottom there if you actually if you want to if you those of you who are curious want to see the actual architecture of the deployment guide that we worked on that Susie it engineers and I worked on together You can go to that that Google link there and pull it up There's a lot of detail. Obviously. I don't have a lot enough time to go in in the full detail over all the details But so basically in a nutshell the preparatory steps are you want to set up your switch ports? For link aggregation, so you have you know high availability failover at the link link layer You're going to create or modify a storage virtual machine What I was talking about in the diagram the the the middle piece of the architecture that can actually move between nodes That's where you're going to be serving your NFS and your ice-cozy data from You need to create the and set up your ice-cozy disc for the share block device fencing So we're talking about stoneth earlier. They shoot the other node in the head in this design We're actually using ice-cozy disc as the stoneth fencing device So those are done through multi-path HA so there's high availability there and basically it's kind of like a voting disc Where the stoneth nodes communicate through to each other through those discs? And then moving up the stack you have your databases postgres sequel Rabbit MQ, you know for the Q sender glance Flex vols for where you're going to actually store your sender data and your glance images on the glance images We recommend deduplication you might get up to 90% space savings with that You think about you know, you have a lot of Linux VMs are all based on the same image You can actually deduplicate those blocks at the 4k block level and reduce your data storage down to greatly You can use it in the in production as well on the center volumes And then also there's an optional Section of the deployment guide for if you want to actually support live migration and what this means is in a case Where you have two hypervisors You want to be able to move that a VM from one hypervisor eight over to hypervisor B You need to have shared storage to mediate that so That's done through NFS mouse And then when you get in the actual deployment you build and configure we went through that you build and configure the Suzie opens that cloud 5 host you modify the network I'm going to show you that in just a minute how you modify the network for the store particularly for the storage network You run in Suzie install cloud important discovery, you know pixie boot your host they come up there seem they show up in the gooey as we saw earlier You install all the nodes with batch edit Once you've got your nodes up then you can hook them up to the block devices I talked about the ice scuzzy block devices so you can do your stone-thincing amongst your three hoes and Then you deploy your HA controllers and the open-sex services. So that's a very hell of a bit I'm not going to be to obviously go into everything But I'll show you sort of key parts and you can if you I'm gonna I'm gonna pull up the link to the Reference architecture again at the end you can pull that up and see the actual details of how to set up the stone-thince devices and fencing and all that stuff So basically what I did in this case So pretty much went with the sort of the recommended the Suzie networking scheme with the exception of the the storage network I had to modify that to get out to my actual storage network and the lab So this is how you would do that in the in the Yast interface So we're you know, so you can see that you're setting a range of addresses here The minimum IP address the max IP address and you do that by clicking down on the The edit ranges button down here at the bottom right. Yeah And then this is what it looks like if you've discovered all the hosts The obviously the the compute nodes could vary greatly that could be three nodes or it could be 300 nodes But what you're gonna see is you're gonna have you know You basically your foundation is gonna be your your data services. That's where you know, your your databases live Your controllers Where all of your control services keystone glance sender all those things live on the services cluster And the cluster is the target that we deploy these things to we're not deploying to a single node When you drag and drop in the interface, it's actually to the cluster. So and All of the obviously all the the network services run on the network cluster Neutron This is where you would actually go in and you could select, you know That your batch edit mode and you can see how I've select I've grouped these in the batch In this with you know, their intended role whether their controllers compute network What have you you can go through and set them all you boot all your all your nodes up They come up you recognize them you select what role they have within the within the open stack deployment And then when you click, you know, you these are actually completed here If they they weren't you'd see a over here. You would actually see checkboxes But once you do that your boot and they come up they're complete. This is how it looks Okay, so this is a little bit of a kind of a two-screen shots This is down here. You can see well first of all pacemaker is the service that mediates the high availability Amongst the nodes in the three the three nodes in its cluster And as I said, we're using stoneth with the share block devices So in the guide, you'll see it talks about how to how to create the lines on the net app side how to map them on the host side And then once you've done that in the actual you have that sort of plumbing in place Then you come to your pacemaker proposal here you put in the block devices that you've set up You hit save you hit apply and boom it does all the magic There's no manual configuration of anything and I have to say What a pleasure it was working with since dollar really good job on it tell your your engineers. It's really it's really a good tool It's a it's a quality quality piece of work and And so once you have all these these clusters built you'll see you'll see your three clusters showing up You know, they're they're ready. They're available and then see you're on to the next next process Postgres SQL here, we're using shared nfs storage as you can see here I'm giving it a amount and all the details for this are in the guide as well But I'm just kind of giving you the highlights. So basically your your databases are now going to go in nfs storage You got block devices for clustering of the fencing And then shared shared nfs storage for your database As you can see down here, uh, we actually select the data cluster as the target for our postgres equal database So it's running on a cluster of three nodes if that makes sense glance We set up to you know to use nfs here version 4.1 which is pnfs. So it's highly available. It's almost like A lua or multi path for those of you who are familiar with storage protocols But a way to sort of dynamically figure out where can I get my data from without having to worry about fail Over the client is smart enough to know how to fail to a different target Which has about a big advantage is in a large cloud Uh Here we're starting to install our services. So in this example our keystone server gets dragged to the Services cluster that we've set up in the same process. Um as as in the other clusters Okay And then you get down to actually pulling the the netapp sender driver Which when you we know when you go to do a sender create for your block devices for your nodes There's a very nice spark lamp here. You can select either seven mode on tap in seven mode or cluster data on tap You can select your protocol. You want to use ice scuzzy or if you want to use nfs the The management address of the netapp faz that you have We recommend hgtps for security obviously 443 give it an at you know your username and your password here Um And you have to specify the name of the v server in this case the name of my v server my svm My storage virtual machine was susie cloud pie And then you have to give it uh at least one you can give it multiple exports here If you have if you want to have more than one export you can have One that's thin provision one that's thick provision one that has deduplication one that's mirrored You can have all kinds of different options on the back end when that app So you can create what we call uh, we call it our storage service catalog But basically it's a sort of list of services that you can depending on your needs So when you go and just say and you're in a horizon you want to click the drop down and select a disk type You can say well, I want this is an archival. So I'm going to put it on dedupe Or you know, whatever your need is, you know, if you need high speed flash disk You have a disk type for that you select that in your cloud So it's very flexible. We'll give you a lot of uh capabilities uh for different use cases Yes We're done. So this is basically it. Um sender And this is my last slide. So if you want to go and get the uh, the reference architecture There there it is at that link. Thank you everybody