 Thank you. Thank you very much So we're here to talk to you today about Dreamhost and our story with OpenStack First introductions. My name is Jonathan LaCour. I'm the vice president of cloud at Dreamhost Been a dream host for about five years been in the Python community for over 15 years Been a programmer and engineer for a long time as well And I'm Justin Lund. I'm the product manager for our cloud services at Dreamhost My career has always been focused on customers and helping them to use the technology So let's talk a little bit about Dreamhost just so you can understand who we are We were founded in 1997 in Southern, California by four college buddies Just to host people's websites and we've grown today to be 400,000 customers Over 1.5 million hosted domains and applications. We do manage to web hosting web presence services and cloud services And we're here today to talk to you about those cloud services And we have a pretty global reach so while we're us based and that's where our infrastructure is over You know about 25 percent of our customers come from outside of the United States So let's introduce dream compute. Yeah So dream compute is our open-stack powered public clouds It's it helps in our goal of helping customers own their digital presence That's our noble cause that we fight for it dream compute or a dream host We debuted in beta at the Folsom Summit in San Diego in 2012 But the design started in 2011 With some basic design tenants. We took a slow approach to curate the cloud and to ensure its stability so debuted in beta in 2012 and Moved into general availability in April of this year So dream compute and a nutshell today Powers everything like Jonathan said from simple websites to dream host internal services Most of our new products will debut on top of dream compute, which is really great Today, it's super fast as boot SSH times in less than a minute So we're we're very proud of that. It's super speedy and it's evolved a lot from that 2011 design We learned a lot from the early days of open stack and we'll share a lot of that with you today So as Justin mentioned We kind of debuted dream compute in 2012 at the open-stack summit in San Diego and Our cloud architect at the time gave a presentation there that had a lot of our core principles and our decisions And we made some waves about that which was really fun And we're gonna look at those decisions and some of them were good Some of them weren't so good and revisit and look at all the the pain points So let's go back to 2011 and get inside the head of our cloud architects and think what we were thinking at the time So first we had a core principle about storage we we wanted to make sure that we could scale up and out and That our performance would be adequate for general purpose use cases, right? We weren't looking to create a high-performance computing environment We wanted to create a general purpose cloud allow people to kind of utilize block storage We wanted to be resilient right and resilient to a lot of different types of failure mode So drives full nodes or even a whole racks going away With minimal customer disruption We also wanted to be able to do this on top of commodity hardware and software and we wanted it to be cost-effective At dream host we'd been running storage for years and years on our hosting platform and we done it on everything from local storage to You know expensive proprietary nas environments as well So we had a lot of storage experience under our belt We really wanted to do something new. There was a little bit more resilient and a lot more cost-effective On the network side We also you know if you view public cloud or sorry web hosting as kind of the Wild West which we do People kind of signing up randomly and doing all sorts of crazy things installing whatever application they want It's it's very much the Wild West if if shared hosting is the Wild West Public cloud is the Wild Wild West like giant spiders will Smith. We're talking like really crazy, right? So we knew we wanted to isolate our tenants as much as possible from each other And we wanted to do so at L2 if possible so that meant fancy new virtual networking technologies and and so that was exciting and We also knew that the world was going to be running out of IPv4 addresses and while we are a player in the space We don't own massive blocks of IP addresses. So we wanted IPv6 support out of the box We also wanted to support private networks with that and floating IPs No single points of failure if possible and we wanted to provide as much bandwidth as possible from every VM to every other VM no matter where they resided in the cloud, right? So that customers wouldn't really have to care about where their VMs were located and we wouldn't either Sure so another one of our core principles for the compute side was we thought in 2011 that the world was going to be About concurrency Lots of parallel cores the speed of the cores didn't matter It was all about having as many as possible to apply to as many tasks to get the job done We thought we thought we thought Or someone thought yeah, I don't know if we thought it right can we cover ourselves for that not our fault We wanted the also we wanted the operational flexibility to move VMs around We wanted to be able to do Upgrades easily it should be something that's kind of constant and easy to do and not a big event We also wanted to To manage hardware failures Easily because hardware fails. It's that's something we've learned a lot in the shared hosting world We know discs die. We know Hypervisors fail. It's computers are bad software is bad. Everything is bad. So that's life So let's talk about how we actually built and designed our beta based on those core tenants So for the the storage we decided to use Seth Seth was created by sage wild one of the dream host co-founders It has the similar benefits to a traditional nas vendor So it's it's very reliable. It's very scalable It's but it's not a black box. It's not very expensive It uses commodity hardware, which is very nice There's no expensive service contracts that go along with so it's easier on the wallet The trade-offs were similar also to traditional nas vendors where it was speed But that was okay because we were looking for general-purpose storage speeds around a hundred megabytes per second right times So that's that's what we were targeting and that's what we were able to see Back in 2011 Seth wasn't very well known yet The community hadn't adopted Seth like it has today in the last few years But we realized the potential and and we were excited about it. We knew it was going to be a good fit for us We could use it for both block storage and image storage Ephemeral storage I think was questionable at the time it was And it's fault tolerant we could lose We could lose hardware like we know we would and customer data would remain safe So that was that was big for us. I'll also add we had some operational experience We had been building dream objects, which is our object storage service as three compatible on top of Seth as well So we kind of had a lot of Seth brains in the house. So we thought might as well put them to good use, right? The the compute architecture we selected KVM. It's simple. It's free. It's open-source. It's built into the Linux kernel There's no special modifications to the guest OS required, which is nice It's pretty easy to monitor. It's scalable and fast without having to make any specific changes to to be able to scale it supports live migration, which is important for us and Since it's built into the Linux kernel, it's easy to deploy. It's easy to patch and we know it runs on commodity hardware And so we chose some commodity Dell hardware. We chose AMD processors with lots of cores It kind of fits into what we thought was Concurrency easy for me to say and and lots of memory enabling lots of VMs so That limited our provisioning over provisioning on vCPU, but we did not over provision on memory at all So we'll talk about where we made a lot of bets was on the network And this is one of the things we'll learn about a little bit when we address some of the things that we maybe shouldn't have done So one of the big decisions we made was to use white box switching with cumulus networks Linux on on those white box switches at the time when we announced this cumulus wasn't even a public publicly known entity They were still in stealth and and so this one got a lot of Jaws dropping in 2012 This has actually been a great and and wonderful decision that we've made and we've used that all over the place now But it was a big deal at the time and we went with the spine leaf network architecture using 10 gig and 40 gig and We also made a big bet on network virtualization and that was a very difficult decision to make There were only a couple of ways to do it none of them were open source And we went with NYSERA before they were acquired by VMWare We were again one of the very first customers of them We used them for L2 isolation and then at the time they didn't even provide L3 at all So we had to build our own and that was a project that we ended up creating called project to Stara which is an open-stack project and It essentially created service VMs to act as routers for every single tenant in the environment and this served us really well We'll talk a little bit later about what we're doing today and kind of you know why we did it Why we maybe should or shouldn't have But this was the decision we made at the time because it was really the only path forward that would tick all the boxes of our design tenants So what did we learn? What about mistakes we made? Yeah, how do we do? How do we do? Let's do our report card on the beta So Seth was the right choice I think History showed us that Seth and the community Seth usage in the community's got rocketed. It's become very popular We saw constant improvements in integration with the open-stack and Seth. So we feel like that's been very good As we knew hardware fails And having redundant storage and a fast working live migration Having the storage on Seth on the network a little bit slower as the trade-off The being able to live migrate and be resilient to data failures was a big win for us and our customers One thing is we bet big on storage usage and lost Cuz we we created a big storage cluster thinking based on our our hosting customers They'd store a lot of data. They put a bunch of stuff on there Turns out they were just fine with the amount of storage that that came in the flavors Looking back it makes sense, but at the time we thought they would use a lot more storage And the customers that did want to use storage wanted faster storage So that 100 megabyte per second kind of target we were going for wasn't as fast as what people wanted And while spinning discs are cheap having them step a separate storage nodes is costly So that was something we learned also one of the unique quirks of Seth is that it wants raw images to help facilitate Facilitate copy-on-write while customers want to use smaller images like q-cow So that's been kind of a challenge as well So let's talk about what we learned on the network side So good like I said cumulus has been a fantastic experience for us as but white box switch white box switching It's given us a lot of ability to kind of manage the switches on our network just like any other node in the environment Right, it's we don't need a specialized networking team or our operational engineers. They can handle everything right we can use chef We can use all the same monitoring tools. We love it The architecture itself has been good The only thing we've really done there differently as we've gone forward which we'll address in a bit It's just to beef it up a little bit, but but overall I think our physical network design was was good We did well so a plus We also bet big on quantum which is now neutron and I'm saying quantum because at the time That's what it was. It was still called quantum and it was really really early. It was not ready for prime time We thankfully managed to get one of our engineers to become the PTL for for quantum at the time And he shepherded it through the the name change to neutron And we were able to help move it forward a lot to make it actually start to you know Be ready for our need in our use case Back then everybody wanted to use no vannette. Yeah, everybody was using over network at the time There was scary and there were people fighting about this which was frustrating but Needless to say, you know quantum now Neutron has made a huge amount of progress and at this point everybody's deploying it things are working great But this was not the world we've lived in at the time And having to build our own layer three services. It was very much one of those how hard could it be? Right, it's just a VM. No big deal turns out an enormous amount of effort. We're very proud of what we created with it It's an awesome project. I encourage you if you have that use case You need to manage service network VMs to do virtualized network services inside of VM It's a great solution to that problem, but it did take a lot of effort for us And we're a very small team. So, you know, that was not a very good thing and they also do Have some downsides, right? So if you have a service VM if that service VM goes down if you don't have an HA pair It's a single point of failure and they also require memory, right? So for every tenant whether they're using a VM or not you have to actually Have a VM ready for them. That's the service VM providing the networking services So that was the downside again these things could all be improved and made to work in a star But a lot of effort, right community wasn't really Forming as much as we'd like we also had a lot of issues with the early days of SDN, right? and this is not a criticism of The vendor okay to be very clear it was early We were trying to do a big big thing public cloud with this very new technology And open v-switch was not ready. We had a bunch of nightmares with that And but you know again the world has changed OBS is much better now a lot of the SDNs out there are much better But this was a big difficult magical thing for us at the time if we could go back Not so sure we would have done it this way Also customers find private networking confusing the vast majority we found that our customers They just wanted to plug into the public network They just wanted a v4 address now my v6 address now I don't care about you know isolation as much as you know, maybe we thought they did and this is our customers, right? Not all customers our customers On compute what else to be learned? Well KVM is great, right? This is we still use it today awesome decision My migration is very well supported and especially with using seph for the storage We can move workloads all over the place whenever we need to When we have a security issue for example And we need to say upgrade the kernels of all of our hypervisors Makes it really convenient to be able to just take a VM move it around Evacuate the hypervisor without anybody really necessarily needing to know and being able to do upgrades and things like that So we have a hardware failure or some other problem. It's it's hugely helpful And you know frankly KVM has been really stable and has had very few security issues or problems in the Five four four years since we started doing this The bad side. Well, we were wrong about the future. Well, we weren't wrong about the future necessarily We just we thought it would come sooner And it turns out, you know what people want Really fast cores and they don't care how many like two three maybe four They don't want 16 cores, you know, maybe they do but they want them all to be really really fast so most software as I mentioned earlier is terrible and It isn't very concurrence doll it wants one very fast core and it will will operate better in that environment So we got that wrong And the CPU and RAM mix we learned a lot from our beta customers What what kinds of flavors they wanted and and when you start to see the actual usage patterns? It helps you to change a little bit. So we'll talk about those adjustments in a moment Yep, so as we started building our next cluster, we took into account these These learnings and we made some adjustments adjustments based on what we learned So the first thing we did is we moved the pendulum to the other side and we went hyperconverged our storage and hypervisors are all on the same hardware Which was great because it reduced our overall CapEx bend you take away all the CPU and memory overhead from separate machines running on storage nodes and put them all into one machine It's much more cost-effective. There's one global skew that we can order. It's easier to provision easy to order So that that part is much much nicer But there are some downsides it's important to know that your Resources for VMs in your storage on the same machine means there could be resource contention there So you just have to be aware of that and make sure you know how to handle it And then we're also looking at During usage spikes one can almost take out the other again. There's ways to kind of manage that We've gotten we're learning a lot about how to do that properly learnings for another summit or two Yeah, right, but tunable step tunable, you know Typical type of operational things you have to do to make sure that one process can't own your system right if an OSD spikes It is it is a new problem to have and your ratios are fixed You know how many CPUs memory you have and how much storage you have and if those start to change with your customer usage One goes up and you know you have to change the hardware you order because those are kind of fixed as you add more Machines that that ratio stays the same so good news is we had a bucket of customers who had showed us what those averages We're going to be so now it's it's been a lot simpler, right? Because as long as the averages don't move they move we have to adjust and that's fine, but yeah So like Jonathan mentioned we still believe the future is concurrent, but today our customers are not Our customers want really fast cores and Intel provides those So we we made that switch listen to our customers Having faster cores also makes it easier to over over provision on CPU So we do a little bit more over provisioning But we still don't over provision on memory. We still think that's the right call there And even with the over provisioning performance is over double of what we had before so It's pretty important. Yep So let's talk about storage As we mentioned in our beta cluster we had separate storage nodes with spinning discs in our new cluster our production cluster We have all SSD storage. How is this possible? Well a couple things happened one the world evolved and SSDs became more popular And therefore they manufactured more and therefore economies of scale they got cheaper Still not nearly as cheap as a spinning disc, right? But it started to make more sense also when we went converged We didn't have to buy separate storage nodes and when you can combine all of that into a single, you know A single piece of hardware you get some savings and so that helped us to pay for that So what's the good? Well, there's obvious good here, right? Performance is over double again so we've over double performance on Compute over doubled it on net on storage. We'll get to network in a second Seth recovery also is a lot faster. So if you aren't familiar with how Seth works when When you lose a drive or a machine or you're expanding capacity or contracting data needs to redistribute and That recovery when you have spinning discs is very slow, right? But when you have SSDs, it's really really fast Especially when all of their machines are connected with 10 gig 20 gig actually we have dual linked 10 gig out of each hypervisor now So you end up with 20 gig effective between every node And that's great actually makes recoveries really fast the downside to that is it gets so fast that again This resource contention problem can bite you so you have to be careful if you're if if you do Change the the crush map effectively change the distribution of the data the data will Seth is a little bit of a control freak a little bit OCD right it wants everything to be just so right and so it makes everything happen as quickly as possible and when you have Everything according to the crush map has to be so and it wants to make it that way as fast as possible Yeah, it's it's like a it's it's a great roommate, you know keeps the whole place clean for you But sometimes when you have a party it's just like you won't stop dude relax So that's that's lots of fun So I'll also talk about One quick if we have time for our a little story. We we did go multi-vendor on our SSD Drives and that was a really good decision as it turns out when we were first going through our burn-in process on This hardware everything was looking great, you know, we were you know throwing lots and lots of IO at it We were You know running different workloads simulating things everything was looking really good And then we started to run into problems right when we went to you know more customer-facing testing where SSDs were just getting kicked out They were just you know the the hypervisor to say up. I don't like that disk anymore. You're gone, right? It was very odd and we ended up having to get work with our one of our vendors the SSD manufacturer They actually sent out an electrical engineer and they installed voltage regulators on the PCB for the SSD like it's crazy So I'll say that we did have some nightmares on this but this was still a very good decision for us performance is awesome And we have several different SSDs we can choose from now Which is always a good thing and everything is working great, but it was a little bit of a nightmare The other adjustment we made was on the network in our new production cluster actually no longer using the service VMs our beta cluster still does and We are using distributed routing we went more vanilla with neutron and why do we do that? Well, as I mentioned before we're a small team We were spending a lot of cycles right trying to do things differently And we still like a lot about that approach the problem is we just don't have the time right so we decided to Try to go in a little bit more lockstep with community see if we can contribute to Whatever one else was you know kind of trying to deploy and So it's good. We less time spent on our homegrown stuff We're a bit more resilient to failure now because you don't lose a service VM taking out a whole tenant You may lose an agent on a hypervisor and just lose some of the VMs connectivity on that machine Which is much better if you you know you're a more isolated Failures, and you know we're also able to get improvements from the community a little bit quicker more people working on DVR More people working on vanilla neutron than on what we were using before trade-offs though The bad part about distributed things is the fact that they're distributed Right when we had an issue with the network before we could just go into the service VM and look at the IP tables rules Really easy now if you have an issue with connectivity between two VMs on two different hypervisors Oh dear you have to name spaces on this hypervisor figuring out where they are It's a bit of a rabbit hole you have to climb down, but again not harder. It's different. It's different It's a different discipline And we're still some finding some warts in the neutron DVR implementation for our particular Way that we've deployed it right You know some we are actually using VXLan, which we'll talk about in a moment And there are some we found a couple bucks, but we're working with the community on fixing those things so And I think you were supposed to do that last one, but I'm gonna continue on go for it No speaker notes so Yay, no more proprietary SDN. We're really happy about this. This has been a great move for us VXLan at the time we in 2011 was very early, right? Now it's everywhere right every switch you buy every nick you buy well most of them have Offloading for VXLan in hardware, right? So it's very very fast And we're taking advantage of that and we worked with our partners at Cumulus They have some technology. They've kind of worked on called LNV lightweight network virtualization. It's very simple We're using this bum flooding suite called VXFLD you can find on github And effectively what it allows us to do is just using Neutron and a couple of agents that run in in our cloud We get you know automatically provisioned VXLan tunnels for everything And so all of our L2 is virtualized and it's all hardware accelerated very fast very lightweight no OBS anywhere, which makes me very very happy and It's very open no magic. So the downside is Trade-off not really widely deployed with one of the few people using this I think there are others which is good and Cumulus is a great partner in helping us find issues when there are Issues to resolve but it is definitely not a hugely deployed thing, but it's it's it works great for us And the next learning that we made an adjustment for is the public network So by default in our new cluster whenever you spin up a VM you get a public IPv4 address by default That's what customers want. They want They don't want to fiddle with private networks. They don't want routers for the most part. They want a machine They want to plug it into the internet and that's Making them happy Customers using deployment tools most of these deployment tools now are just looking for launching a VM and expecting a public network So it's made integration with their party tool is a whole lot easier And so instead of having a lot of customization to make Those tools work. It's it's now just working out of the box I know Ansible is very very popular with our customers as a version to with our new cluster. It just works So that's been really great. Yeah Absolutely Got something in here about other learnings. These are more product related learnings. I would say One of the things that we we tried to do is that maintenance is the norm and not the exception Maintenance should not be a big event. It shouldn't be a customer downtime event. It's just something that's that's going on And live migration has been immensely helpful with this as well Customers a lot of the customers that we're seeing are not ready for cloud yet They're really just looking to offload their infrastructure reduce their capex And move it to op-ex and they're still looking for a hundred percent availability And reliability so that's what we're seeing. I know that in the past I've heard some of that And I just I still think that's the case today with a lot of our customers We've tried several different billing methods Originally we tried prepaid plans For a set quota, so you pay a small amount and you would just get a as much or set quota And you could just spin up and down as many machines as you want. I was kind of confusing for customers They felt like they didn't get what they were paying for We tried some pure meter billing and customers were scared. They didn't know how much their bill was going to be So we kind of went with a mixed approach of what we call predictable bill So now it's you Spin up a machine and if it runs for 600 hours, which is the equivalent of 25 days It's a flat price to a meter up to a flat price You know what it will be over the month, which is really nice No matter how many days are on the month and it kind of rewards you for long-running VMs, which is exactly mostly typical for especially our customer like you said not everyone is you know Ready for cloud a lot of these applications are not cloud native. They're not using configuration management necessarily They're not with a hosting background We have a lot of customers that are used to paying you know yearly or monthly and We've tried a few different things, but this has worked really well for us. So we're happy about this And user experience matters a lot We've found that we want to avoid information overload We try to pre-configure everything as much as we can And that's helped a lot when we launched with horizon and our beta cluster Customers had to create a network. They had to create a router. They had to allocate floating IPs to be able to assign We did a quick start type of thing where it automatically created all of that and allocated IP So a customer could just log in and launch an instance having to go through all that process Was too much and so we're trying to simplify that as much as possible And I think customers really appreciate that And then one of the other things we found is our customers anyway find horizon confusing There's a lot of ways to shoot yourself in the foot You can launch an instance without a key and can't log in you can Delete your network and kind of get it into a weird stuck state So we've been working on a project work internally calling sunrise But it's a way to simplify things make it more Focused on servers and not a whole infrastructure platform So that's kind of something we've got that deployed in preview right now at cloud dream host calm So if you're a dream host customer, you can check it out making good progress on it new stuff coming But and and also I'll point out this is not a criticism of horizon, right? Horizons great. It but it is what it is, right? It's it's very much a Swiss Army knife, right? It's it's designed for all of the use cases and Master of none. So you just need a pair of scissors. Yeah, exactly. We just need the scissors for the little toothpick. I'm not sure So let's bring it all together and then we'll open up for questions. So this is what? Things look like in our new dream compute deployment So you can see that we essentially have two top of rack switches Every node in the in every rack is dual Connected so one to each tour and then the tours are connected to each spine and then the spines are connected to each other Right, so you end up with? Very fast networking, right the VX line overlay as we mentioned cumulus networks VX FLD and neutron just ml2 And all of the end cap and decap is hardware accelerated at the host and in the spines And you get 20 gig effective between every node in the cloud, right? So if you have a VM anywhere in the cloud, you can get 20 gig effective to any other If if we give it to you, which we don't generally but we can And we were using L3 services via open stack of Stara as I mentioned we've moved to DVR now Just kind of an overview of the storage and compute the the converge nodes So we're using eight nine hundred and sixty gig SSDs, and then we have the boot SSDs in raid one We're using Intel chips like we said Zeon e5 26 30s that 2.4 gigahertz Our customers love that they are fast 192 gigs of RAM in each VM Roughly about 26 of that is is used for stuff and the remaining are used for VM I think I've got that number right off the top of my head and Then we do get more over subscription available with those faster cores But it's still pretty minimal overall and no over provisioning on memory again All right, it looks like we actually do have some time for questions I think I'm not sure what the time in three minutes three minutes Okay, come up to the mic if you can so that people can hear your question And then we'll we'll try to finish on time so that the people after us don't get in trouble like we did at the beginning And then we'll be over here and it can answer more questions. Yes You mentioned that when you moved to solid state devices you actually saw an increase in resource contention And that seems totally counterintuitive to me, so I'm wondering if you could just kind of speak to that a little more So in Seth you have these processes these OSD processes that run on on the hardware, right? And their job is effectively to move data around right and so if you have like a large spike in IO, right? Or for example, let's say we lose a hypervisor and we lose all those OSDs That means all the data has to move in the cluster because everything has to move around And what happens is Seth really wants things to get back to normal as quickly as possible And if you don't have your tunable set right, you're not careful about how you manage those processes The fact that you have such a massive amount of IOPS that you can drive Means that that process can really take up a lot of memory It can take up a lot of CPU and just go nuts, right? And that's where you get that resource contention because it's co-located with the customer's virtual machines All right. Thank you. Yeah. I think it's related to her question It's aside from the SSDs What else you guys use to to to get to get set settled down when things are not running fine meaning in order to not Punish the VMs running on the same note. How do you guys use it? It's a container. How do you do it? So smart team? Yeah, we have a great team I think that is the key thing is you know, Seth is is a wonderful piece of technology It's not magic though and you can't just it is not is absolutely not a shrink-wrapped piece of software, right? It doesn't it's not like a NAS where you just like drop it in. It's a black box or whatever You really need to understand the tunables you need to You know really have a lot of operational experience before you go to production Because if you do have problems it can run away from you and it still runs away from us sometimes and we have you know Five six years under our belts doing this so having a great team using the tunables being involved with the seph community Who is fantastic? They're very helpful And and for a lot of organizations the best thing to do is just to pay somebody to help them with that Right so that that changes the the game a little bit right because then you have to pay red hat But they're great at what they do so you know worth it for some teams at some it's you know Like those tunables would be like max backfills you can limit the number of Of backfills that are happening to repopulate all that data you can I also recommend having the creator of Seth be on your board. That's really helpful if you can do that. Yeah, that's how What you guys Supposedly if I know it's not running out of CPU if Seth wants the whole CPU He's gonna have it or If you can you know, obviously there's things you can do right managing processes and you know setting limits And so on and so forth that you can do right? We haven't done as many of them as we should And we're moving towards that but yeah, I'd recommend Seth can consume it all if you don't throttle it And I think we have time for let's say one more question, and then I will be available over here I guess yeah, hi. Thanks a lot for your nice presentation many of the things you mentioned actually Very similar to the experiences we have made at CERN with our cloud right one bad experience that we have made And that you didn't mention though is how much memory of the 192 gigabytes that you have on the compute nodes do you actually hand out to the instances? How much do you reserve? How do you do this technically? So we have a hundred ninety two gigs of memory on our cloud nodes. I believe we reserve 64 gigs for Seth OSD processes. Is that right? I thought it was more, but I can't remember yeah I'm pretty sure that's it and then 128 is available for yeah, my team is nodding for for customer VMs That's how we do that and then we use the standard Nova schedulers And we configure them so that we make sure we don't over provision on memory We make lots of extra space, but there's actually nothing that you Reserve for the for the processes itself so something that can only be used by the hypervisors all the services running you hand out all the memory Is that right? Actually, don't I don't know the answer to that question. I have to ask my team Okay, thank you everybody's questions will be over here. Thank you