 OK. Good afternoon. There we go. Microphone working, yes it is. All the other speakers have been saying how bright the lights are on the presenters, and it's absolutely true. It just cannot see a thing with these blinding lights. So I'm Neil Levine, Director of Product Management for Red Hat. I work in the storage BU. I've been working on Cef for a few years now on the product side. And I'm going to try and get through my slides as quickly as possible, which are very much the sort of Cef one-on-one. And hand over to Will Bogey from Target, who's going to give you the experience of a real-life Cef deployer, and some of the experiences he's had getting Cef to run in an open-stack environment. So very quickly, how many people are running Cef already in your audience? About half, and how many are thinking about running Cef but not running it currently? So not many. OK, so I'm going to bore a lot of you for the first 10 slides because you guys are running Cef already. But my marketing team like me to go through this stuff. So I'm just going to go through a little bit of introduction to Cef for those of you who don't know it, how it fits within the Red Hat portfolio. And there's my screensaver kicking in too early. And then hand over to Will, then I'm going to cover a little bit of the roadmap at the end, which I suspect will get a lot of questions as Cef dual, the latest version, has just come out with lots of good stuff. OK, so very quickly, let's go through this. What is software-defined storage and how does Cef fit into this ecosystem? So there's a lot of buzzwords. Software-defined storage doesn't really mean anything. It's a marketing phrase, but we still have to use it. It's a part of a general change. And we like to slide a lot. It sort of explains what's going on throughout the data center, both in the application development, moving to from waterfall to agile to DevOps. You can see containers as being part of this move to sort of make things go faster, get better velocity. And we really see storage as fitting within that software defined storage, specifically fitting into that same change that's happening throughout the data center. So we had to scale up with our appliances. We then had to scale out, sometimes still with hardware appliances. And we're now moving to a software-defined storage world. And again, this is all about velocity. OK, so what does this actually mean? Well, the old world was big silos of storage. You would provision them for individual projects, individual applications, sometimes individual users. And it was hard to scale, because when you ran out of space, you had to buy more stuff and move data around. You had a new project. You had to go through a whole procurement process again. With software-defined storage, much simpler. You buy standardized computers and disks. Layer-on just software on top of this commodity hardware, present to author a nice API. And suddenly, the users could provision their own storage, decide the performance they want, the hardware characteristics. And ultimately, everything scales, both the software and the hardware layer underneath. So what does this matter? Well, you can start buying your hardware from a single company. If you're already buying Dell or Supermicro for your compute, you can also use those vendors to buy your storage as well. So standardizing a supply chain gives you some pretty big economies of scale. And you can also really pick the hardware you want specific to your app to your environment. Obviously, we also get the scale out. Architecture, shared nothing, distributed, no point of failure, no single point of failure. Architectures, which are really sort of the intrinsic goodness of Seth in particular, is around the robustness of that architecture to really allow you to get up to petabytes of storage. And importantly, you move all of the intelligence up into the software layer after the hardware. And so you can really have high innovation in software much faster than you can when you're tied to the hardware. And of course, being a Red Hat person, we've preached the open source way. And you get the open development process with that, too. So with an open stack, this matters, because if you're scaling open stack, which hopefully many of you are, you want to be able to scale the storage in the same way you're scaling the compute. You're using provisioning extra machines because you need the extra VMs. But if you're buying storage appliances, you're going to be buying these big monolithic blocks and big capex up front, where when you're doing it with a software-defined storage world, you can scale it exactly the same level. So you get that agility. You can tie it together both in the actual buying process, but also in the deployment cycle as well. OK, so let's get out of the way, software-defined storage. OK, so what is SEF for the three people in the room who are not familiar with it? It's a unified storage system providing object block and now, as well, a stable file system, too. At its heart, it is an object storage system, which we refer to as RADOS. And this is where the 10 years of R&D and peer review have been going on in a SEF project were really at that RADOS level to get a highly resilient, robust architecture for storing data. If you want to interact with that object store, we have a nice little set of development libraries called Liberados, which allows you to talk directly to the object storage interface if you're obsessed about performance and you really want to get rid of as much abstraction overhead as possible. That's what you're going to be using. Most people, they want to be using that. They want to consume more common APIs. So we have the RADOS gateway, or RGW, which is the component, which allows you to talk S3 or Swift. We also have RADOS BlockDevice, or RBD, which allows you to consume SEF as essentially a virtual disk image, which is what the majority of OpenStack users use SEF for. We now also have the file system, which allows you to interact with RADOS as if it were a POSIX compliant file system. So that's the one-on-one of what SEF is. It's a incredibly vibrant community. In fact, we have many of the SEF committers here, not just from Red Hat, but from other vendors and even users of SEF. And the SEF development community has actually had a very similar trajectory to that of OpenStack. The interest and excitement, sort of the curves have been going up in the same way. So if you're interested in participating in SEF, we have some great mailing lists at sef.com, ILC channels. We invite you to come and participate. OK, now the heavy sales pitch, which I have to do in one slide. We have a distinct product called Red Hat SEF Storage, which is what Red Hat offers. This is where we take the upstream community addition. We apply our own QE, our own integration, add some extra bits and pieces on there, additional patches, some extra tooling, and we distribute it out. So Red Hat SEF Storage is based typically on one of the stable releases with a little bit of lag to actually allow us to do a little bit of extra QA. And then that's what we offer, particularly with Red Hat OpenStack, the Red Hat platform. OK, sales pitch over. OK, why OpenStack? And why does SEF fit into this? So we use this slide a lot, because it's very cool. And it shows us winning. We like this. So SEF RBD is the leading block solution with an OpenStack. And it actually has been for quite a while. It's a user survey, I think, from April? Yeah, so a few weeks ago. But actually, SEF RBD has been leading as a deployment technology, a storage technology within OpenStack now, I think, for about three or four years. And indeed, it sort of came out around the Cinder project sort of evolved alongside RBD becoming almost the de facto block implementation for OpenStack. OK, very cool graph, but why? And what does this mean? So the obvious thing here is it's unified storage. You get object and block. And now you're going to get file too. So this has some very obvious benefits in terms of buying hardware. You only have to buy one class of hardware for a SEF cluster and tune it and so on. It also means you only have to train up your team on one storage technology. Supply-driven storage is complex distributed systems. You have to train people how to use it. But if you're going to need a variety of different storage services for your OpenStack platform, you want to train them up on just one technology. And crucially, it's tightly integrated with all of the OpenStack services. A lot of the drivers you see for things like Cinder and so on, they offer a basic iSCSI interface and things can be connected through. Whereas we actually tightly integrate directly with the Cinder and Nova and so on services directly. So to explain what this actually means in practical architectural detail here is we have a library called LabRBD, which is something that's actually running inside the control services of OpenStack themselves on the block and VM side. So on the right-hand side here, you can see both with Cinder and Glance and actually through the hypervisor with KVM. This is where you run this little lightweight LabRBD library, loads into the process. And effectively, it turns the OpenStack control service into a native surf client. What that means is you get a one-to-many IO relationship. It's not a single point-to-point relationship with traditional storage technologies. You get a massive parallel client cluster protocol as opposed to a client server one. Again, this is where the RADOS cluster has got lots of individual components. That's what allows the scale out. That's what that's a cool architecture. These clients can access all of those nodes at the same time to get access to the data. Been using a RADOS gateway. We obviously present the Swift as well as the S3 API and connect that into Keystone for authentication too. So it really is tightly married deep inside the OpenStack architecture. So why does this matter? Well, from a practical benefit, you're able to boot lots of images really, really quickly. Whether it's one or 100 VMs, they're pretty much booting up simultaneously. And this is mainly because of the sort of the copy and write benefit we provide down in the SEF service. So when images are being, say, booted from glance or you're snapshotting something from either, say, from Cinder or even from Nova, it's all happening down at the RADOS layer, down in the SEF layer. There's no data movement happening up through the OpenStack control services. So you can boot things very quickly, back up and snapshot images very, very quickly. So that, again, it comes back to that velocity. The speed of operation is much, much faster with SEF. And you get all the benefits of live migration and sort of all the important things you need from a network storage service. Also, within a SEF cluster, as you've mentioned, you can pick and choose your hardware. So if you want to have a hot tier, running, say, all SSDs for your block, maybe you want to have a slow tier running 7,200 RPM HDDs or hard disks for, say, your object storage, you can provision them on different hardware, allocate them to different pools. But it's all a single logical SEF cluster, so you're only having to provision and manage one system. And then also now we're starting to have some very good multi-site replication services. We've had sort of fairly OK-ish snapshotting and exporting of the backup that we've now got with the latest version of Dual, which I'll cover later. We've actually got some pretty automated backup capabilities, which allow you to push images between clusters or to snapshot and export things off to remote clusters even running in public clouds. It's very cool. I have no idea what this clip art is doing. Daniel, the marketing guy, I've always wanted to know why you've got somebody being chased, and I'm totally sure that is. But I've left it in there anyway. OK, so quickly to finish off here, SEF is a complex system. It has a lot of knobs and levers and buttons and tunings and everything else here. And getting the most out of a SEF cluster is something that a lot of people have developed expertise for over many, many years. And we tried to distill it down to just sort of a couple of basic sort of rules of thumb, and then hand over to Will to sort of go into the details here. The first one is select the correct hardware. I think Will's going to talk about this in detail. This is the most important thing you have to go through when building your SEF cluster. A lot of people think, great commodity hardware, I'm going to buy the cheapest server I can find. I'm going to pack as many disks in it, and I'm going to have myself an exabyte server sand. Super cheap. You will, but it will be unusable. There's only a certain amount of CPU you have in these boxes. There's only a certain amount of IOPS you can push through these machines. So you need to make sure that you're picking the right blend of processor, disk, disk type for the workload that you have. If it's IOPS optimized, you probably want to go for the SSD approach. If it's real about cost and capacity, you want to go for spindle, it has a huge impact on a number of disks, even a networking here. So we publish a lot of reference architecture guides, and we've got more coming out. We've had, I think, one with SuperMicro. We've got one coming out of another vendor soon. We've got a bunch of these. And readies, these are 40-page documents, lots of amazing detail, which really helps you understand what kind of hardware you should be purchasing. Because once you bought it, you're not going to be able to swap it. And you want to make sure that you really are going to get the performance you need out of your SEF cluster. So this is just an overview slide again on the site. We've got this broken down. If you want for an individual square in a cell, you can see sort of the specific SKUs that we have. Many vendors now have dedicated SEF SKUs that you can buy. OK, so hardware, very, very important. The second thing is topology. Do you want to go hyperconverge or dedicated tier? That is, do you want to co-locate the storage with the compute, which may make sense in very small sort of pops, or sort of limited deployments, where you really are sort of constrained by space? Or if you're running a large public cloud service, you want to go dedicated tier. Again, it has different networking implementations, implications, sorry. So make sure you know which one you're picking for the growth and the use case that you have. And the final one, and probably one of the most important one, but also one of the hardest to distill in the PowerPoint slide, so I didn't, is the tuning. And again, Will will talk about this in a second here. There are a lot of settings in SEF. There are a lot of settings which will squeeze performance out of your SSDs, out of your hard disks, out of your network. Knowing which settings to make based on your environment is a little bit of a dark art. This is definitely an area where Red Hat help and can help. So I think even one of our solutions architect was saying, customer border, all SSD array, they installed it, complained the performance sucked. The solution architect went in, made four settings, and bang, the thing was sort of doing forex on a profile. Just do some basic kernel settings and some SEF settings. So tuning is all. And with that, I shall hand over to Will to go into some details. Will Bogie, Hi everyone. My name is Will Bogie. I'm an architect at Target Corporation. I work for our enterprise private cloud engineering team. And we are basically the private cloud team at Target. We deal with mostly open stack. We do responsible for object store and as well as scale out block storage, which we use SEF for. We're a full stack ownership team at Target, which means we kind of untraditionally for an enterprise we deal with pretty much every facet of our service that we own. So just to give you a little history, our first SEF environment and Target went live in October of 2014. It was based on Firefly. And SEF was the storage backing for our first official open stack release. And basically we use SEF for RBD for our instances and volumes within open stack. We use RADOS gateway for our object storage. And that's instead of Swift. Eliminating that component was kind of added some simplicity to our implementation. And also we use kernel RBD for a few different things. But the main thing is backing salameter for our Mongo database volumes. And when we implemented SEF, we replaced the traditional array based storage solution that was in our prototype Havana deployment. And we kind of discovered that especially back then a lot of the storage vendors didn't really have those integration pieces to open stack. You desk them, open stack, what's that? We don't integrate with that. If you can make it work, great. But what did work really tight was SEF. I mean SEF has a really tight integration with open stack. Always has. They've kind of grown up together, so to speak. So what do we do with our open stack cluster and our SEF? Our development environment is probably our largest. We use it for small internal workloads. A lot of Chef cookbook development goes on in there. A lot of people like to use it for just kind of scratch infrastructure. Like if you're going to test a new piece of software application, it allows them to get infrastructure very rapidly and reliably. And also a lot of our CI CD pipelines are hosted in there. Jenkins, Drone. We have a lot of different pipelines for a lot of different teams, so they're typically hosted there. We also do have two production regions. They host a lot of internal marketing workloads, business intelligence workloads, as well as some customer facing applications. Our initial SEF rollout, we had our development plus our two production regions. The requirements were to build out a object slash block storage solution to support our open stack environment. Pretty standard. Three monitor nodes. We used 12 core blades. We had 12 OSD nodes in each site. And they used the large form factor SATA drives. They were four terabytes. We had 10 OSDs per server. The journals were co-located on each OSD, which yielded us about 400 terabytes per location. We had a pretty basic LSI mega-rade, whatever they call it in, whatever your server vendor. It's probably the same card. And 10 times RAID 0. We had 10 RAID 0 devices on each OSD. After we rolled that out, it became evident pretty quick that the response times were too damn high. So we'd get a lot of anecdotal evidence, young transactions taking forever. Instances would boot. Sometimes cloud and it would time out. That general user griping, hey, it's slow. What would he mean it's slow? I don't know, it's just slow. But if we would add some instrumentation to our deployment, we'd see things like, and in the graph down below there, we'd be getting about 40 milliseconds of IO weighted, around 350 IOPS, which is not terrific. Also, just doing some benchmarking inside of the instances themselves, we would be getting pretty poor performance, especially right performance. So what could we have done differently? There's a few different tasks. And Neil kind of lose some of them. Hardware selection, monitoring, tuning, and understanding what our users are going to do with our environment. So first, you've got to understand the hardware requirements for your workloads. Just like Neil said, fewer disks that are better almost always will outclass more cheaper disks. That depends on your use case. So if your intent is to build a large object solution with hosting images and things like that, maybe the slow SATA disk will work for you. It all depends on how you're architecting your solution and what it's going to be used for. So don't be greedy if you're building to have a performance block solution for OpenStack. 4 terabyte SATA disks may be not going to cut it for you. So in our case, when we were designing our environment, we kind of overestimated the customer desire for object storage solution. And we underestimated their need for block, which kind of skewed the use case more towards having a more performance storage solution. So what we did was we ended up redeploying our OSD nodes, 24 core 2U chassis, the 24 small form factor. In our OSD nodes, we have 20 10 case F spinning disks, as well as six SSD journals in each box. I'm sorry, four. That's the type of that wrong, sorry. Because that equals 26, not 24. So a couple of tips on SSD journals. You need to pay attention to the SSD vendor that your hardware provider will give you. There is a great deal of disparity in SSD performance in the industry right now. So definitely do your research. Understand what CEPH requires for an SSD and make sure you're being proactive with your hardware vendor and not just accepting what they give you based on a SKU that says SSD disk. Talk to them, understand what that disk is, do some research about it. Make sure what you're getting is going to work for CEPH. We also had some additional improvements. We added a Mega RAID controller that had actually a write back cache as well. The next thing we could have done better is have better monitoring our CEPH cluster right from the outset. We kind of relied on using traditional monitoring tactics to understand what was going on there. And it's kind of a unique animal. CEPH has a wealth of data about what state the cluster is in. But a lot of it you just can't see from if you type CEPH-S. I mean, it says health OK. But the pro tip that I found is if you form, export it in JSON, it gives you way more verbosity about what's going on inside your cluster in a very easily parsable format that you can parse and ship up to monitoring solution, be it Nagios or Patrol or Datadog or a solution like that. It makes it so you really don't need to. There's no excuse to say my hardware or my monitoring vendor doesn't have a SAP integration because you can pretty much write it yourself in about two or three lines of Ruby. Monitor the user experience within the instances themselves is not good enough just to look from the outside in. One thing I actually do is periodically stand up a heat stack that runs FIO within the instances and pipes the output to a dashboard. So I understand the type of IO the user is seeing and not just what it looks like from the outside to the bubble. We initially stood up Calamari. It was a decent start, but we kind of outgrew it pretty fast. It provides some rudimentary metrics. But one of the things that we didn't like so much about it was it can only get really a week of history in the Calamari interface, which we wanted to do a lot longer trending over time for capacity and performance and things like that. One other additional thing, Seth, like Neil mentioned, you really need to tune your workload. And it's not something I can really stand up here until you hit these switches and buttons. Seth has lots of buttons and parameters. As far as which ones to engage, Red Hat Support would be a good tool to engage for that. As well as your x86 hardware vendor, I would say most, if not all of the major hardware vendors have some sort of Seth competency now, and they can give you some best practices around, hey, you need to set this to that, and you should set this to that based on what your intended workload is. But in general, there's a few things to be mindful of around tuning Seth. Default recovery priorities, especially when in our old Seth environment with SSDs, any recovery would basically just destroy client I.O. It would just grind to a halt. So you really need to understand how to tune those recovery priorities, balancing speedy recovery with the client experience. Deep scrubbing, there's the maintenance deep scrubbing tasks at Seth performs. Just by chance, occasionally, they can converge on the same node and cause a kind of noisy neighbor type effect, ensure that you're tuning that to minimize that impact. Your journal sizing and your flush intervals will have a big impact on your I.O. A lot of times what you'll see in a Seth workload, maybe that's poorly tuned, is you'll see spikes of I.O. as the journal fills up, and then it crashes while it flushes a journal and spikes that sawtooth I.O. pattern. And that really tells you that you need to really pay attention to your journal sizing and your flushing. Also, this is more on the open stack side, but there are a lot of settings for LibRBD and Nova for caching. Maximized performance, you want to ensure that you have all the proper caching settings set for your compute nodes as well as your Seth nodes as well. The results, after redoing our design, we have improved latency in an extreme way. We understood the impact of our changes because we had the instrumentation in place to understand that as we go, we could see, is this having a positive impact? Is this having a positive impact? That's not OK. Take that setting out, take that hardware out. We were able to see as we go and understand. Because a lot of times, anecdotally, I don't know about you guys, but we could try things in a sandbox environment, but until we really start putting it out in the wild with actual user workloads, actual user patterns, you don't always see the problems. So you have to have an approach where you can monitor what you're doing as you go. We've improved our IOPS significantly. User feedback, extremely positive. We kind of got hailed as a hero a little bit for a lot of our open stack users because their storage performance was so poor that, and now it's quite excellent, actually. And that's the example of our IO weight, and you can kind of guess when we implemented the new hardware. So just wrapping up here, some of our next steps as part of the target cloud team around stuff, we need to start being proactive about our physical world, specifically our data centers. And typically, till now, our data center strategy has been kind of, we're going to put the server in the next available slot, which doesn't always work well when you're trying to optimize like availability through crush. So what we're trying to do is create a whole rack unit of cloud expansion. It includes the compute nodes, the monitor nodes, the CIF OSD. So when we fill up, we basically just wheel in another aggregate unit of our open stack slash CIF storage. That allows us to use crush for better availability, at least on a per rack basis. One of the things that I kind of see as a misstep when we put it in that I'll be working to rectify is, when we spin up an instance, we don't really limit the IO. So basically, it's all you can eat IO buffet. So your performance in your instance is basically whatever CIF can give you at that particular moment. So we're going to be working to implement some limits around that or reasonable limits. And we're going to, the approach I think I'm going to take is be using KVM, QEMU settings, and set that in the flavor. So in a particular instance, a flavor will only give you so many IOs. And also that will allow us to leverage sender to add different tiers of IO. Normalizing that will help with the fact that if you don't do that, typically you're going to run out of IO in your cluster way before you run out of capacity. So you need to kind of understand that curve and size your instance appropriately and give users a proper amount of IO. Experimenting with different drive types, SATA, SSD, just trying to maximize that economy within each OSD node so we can get the most capacity, yet still being performant to our user's needs. And also just working to trying to broaden the sef beyond the cloud use case. So in conclusion, form barking, have a good idea what your objectives are. If it's capacity, object, you can go for those higher capacity drives. But if you want performance, you have to really understand how to architect that. If you make decisions poorly from the outset, what it can do is it can negatively impact anything that relies on it. So in our case, our OpenStack offering was kind of got a black eye just because of the storage performance. So you need to understand what's going to be leveraging sef and ensure you're meeting your customer's needs as a storage solution. Unless you're architecting for raw capacity, use SSDs for your journal. That's pretty much without exception. But if you have to for some reason, then I would recommend using a write back cash card within your OSD notes. Research, experiment, break stuff, let red hat help you when you break it, cry. Monitor, monitor everything. Monitoring is super important with sef. That's kind of one of my main themes. If you know what's going on for your customers do, chances are they won't notice it. But if something does happen where the customers notice, make sure that they have a short feedback loop to talk to you. A lot of times, sef errors can be sort of a snowball. Like a pebble rolling downhill, gathering up snow. So if you catch it quick, then a lot of times, you save yourself a lot of headache. What was that? Pass it back to Neil. So I'm just going to finish off quickly. We have a little bit of the roadmap. And then we've got five minutes of questions and answers. So the sef upstream community released dual or version 10.2.0 a couple of days ago, which is the latest stable long term support release by the community. So we'll be taking that into the Red Hat sef storage product and releasing that in the coming months or two. The main features we're going to have here, so going from the stack from the bottom up on the Rados level, this is a tech preview feature only. It's one of the biggest changes to hit sef actually in quite a number of years. Sef was written before SSDs were really very common. And so it assumed you're going to have a hard disk. You have a file system on top, XFS, and so on. And that's what it was optimized for. That's what it was designed for. Obviously, nowadays we have SSDs. And the file system itself can be a bottleneck. So we've now got a new back end or backing store called Blue Store, which replaces the old file store. There's no file system anymore. It's subtracted out. It does raw block writes. And as you'd imagine, that improves performance hugely. So indicative testing is showing orders of magnitude. We're looking 2 to 3x on different types of workload, but pretty much all of them, sequential, random, small, and large. So you'll particularly notice this on SSDs, but it's true of a hard disk as well. So this needs some time to mature. When you come out with something this fundamental and low level, you need to give it time to bake and draw out potential issues. So we'll have it in tech preview. We'll see the feedback we get, and then hopefully we'll be making out the default pretty soon. But this is beginning to open up use case opportunities for Cef to really back you a sort of hot or a sort of database-centric workloads. And actually we had a Pocono Live last week. We did a presentation of Cef running MySQL or backing MySQL, an open stack environment with some very impressive performance numbers. So we should be publishing. I think we may have already published a paper on that soon. The other thing, and we'll mention this as well, which is that traditionally it's been, I think it was at the IO Buffet. You said Cef has sort of been first come first served in terms of IO. And you can tune it, you can say give, don't do so much on backfilling and scrubbing and rebalancing, try and do more on a client. And, but we've also now implemented a sort of a unified queue. So as these actions are happening, we can sort of really ensure that client IO is really always getting the primary sort of CPU attention. So you should now see the technical way of explaining the 99th percentile latency should be much lower, right? The issue is not necessarily that the screaming IOPS, we have that now, it's about getting the consistency and we've made a lot of changes is to ensure that your client consistency is better over time. For a block user, the biggest one that a lot of people have been waiting for is what we're calling RBD mirroring, which is essentially volume replication. That's the ability to take copies of all of your RBD images, your disks and copy them over to another site for disaster recovery. We'll be wiring this up into the Cinderuplication API in as of now basically, but the first capability, even if you're not running OpenStack, you can still as an admin go and set this up and just say, you know, back up everything somewhere else, don't worry about the user sort of triggering that through the API. I just want to back up all things everywhere. We also have an iSCSI implementation now, not particularly useful for OpenStack because as you saw earlier, we sort of have our own native protocol wired up into these services. But for people who have non OpenStack workloads, you still want to use Cep as their storage system. iSCSI is obviously the link with Franca. So if you're running Windows or something like that, you can start to use this. VMware, we sort of do a little bit more testing, but we should have support for that eventually too. On the object side, lots of features, completely re-architected multi-side experience. You can do global clusters, have active active and sort of read and write affinity everywhere. So, you know, three distinct clusters, users reading and writing to any of them, single namespace, everything syncs up. So, very simple to use compared to previous implementations. Lots of Swift API fidelity around Keystone V3, we've got Swift Object Exploration, static websites, and native LDAP and active directory if you're frightened of using Keystone. Finally, from, then this is specific to our product, we're moving to an Ansible based deployment technology. So, a lot of time, you know, setting up Cep is the initial sort of investment of time and effort. And obviously Red Hat has a strong interest in stake in Ansible now. So, we're moving our deployment tools to that. And we're also gonna be having a new UI tool sitting on top, which we should be announcing soon, which will allow you to do full end-to-end lifecycle management of your Cep cluster, both to provisioning and the management and monitoring as well. And then the big one, and this is, again, this will be a tech preview feature. Similar to BlueStore, it's a big new thing, you need to give it time to bake, but it is now stable, that it is, it stays up, won't lose your data, is the Cep file system. So, we're gonna be working on a minute integration for that. So, you can use that as a shared POSIX store within OpenStack. And we should have the kernel driver available if you're running a REL guest inside OpenStack to be able to access that. Hopefully, an update to REL 7.2 in the coming weeks, in fact. And that was it. So, we have three minutes for questions, and I've been asked to invite people to their microphones on either side, if there are any. I see a lot of your presentations, I always talk about the SSD to HDD ratio, or the OSD to journal ratio. What I never really see talked about is the reasoning for that, and the failure domains that you are creating by doing these, you know, if you do a really high OSD to journal ratio. I never see that talked about. Even though you create this huge recovery failure domain, if that SSD dies. So, you're talking specifically around the journaling where you're using the SSDs as a journal rate. Yeah, I mean, most people, and he said it too, SSDs used for journals without exception. So, the rough guideline is use one SSD journal for every four OSDs, for whether those OSDs are backed typically by HDD. But is that a recovery statement or a performance statement? That's a performance statement. It's also, the challenge is a lot of people say, well, I'm gonna raid my SSD journals, because I wanna get some resiliency there. The problem is SSDs fail because of wear and tear. And so, if you're doing the same right to both SSDs all the time, which you would be even as the statistically, you're likely to lose both SSDs more or less around the same time. So, you don't actually get that much improved resiliency by raiding the journal, as well as a result like performance implication as well. So, yeah, you do create a certain failure domain risk, but as I said, our guidance, I mean, it's a failure domain thing as well. You really wanna limit it to no more than four. You can do more, but it's also a performance thing. So, it's a bit of both. But losing, if you've got a most normal-sized Ceph clusters from two, five, six up, losing four OSDs shouldn't have a huge impact on your cluster. I can help you too with the reliability side, so. I think we have time for one more question. Yeah, just a quick one. Just wanted to have your view between Swift and Ceph for objects. What's your positioning there? So, very quickly, difference between Swift and Ceph is one is architectural. Ceph uses a strong consistency rather than an eventual consistency model down at its lowest level. So, we only confirm a successful write-back up to the client once we've written it to say three copies or do you raise your coding if that's what you're doing? Whereas Swift is lazy, it's eventual. It'll do two and then confirm back. So, that's one. The second one is that we actually do a lot of work around the S3. It's actually almost in a way our primary protocol, whereas I think the S3 support in Swift is, I mean, there is some there, but I think it's fairly rudimentary whereas we're trying to keep good parity with the S3 protocol, which for a lot of customers is actually more important. They've got tools that are written for S3 and so on. I mean, the big thing, of course, is unified storage. We do the blockers and the file as well as the object. And so, if you're looking to simplify your deployment and you've got a limited amount of space, you don't wanna be using too many storage technologies and so that's really the big win that we have with Ceph. Thank you. Okay, all right, well thank you very much.