 Okay, welcome everybody, please sit down. I hope everybody's ready to drink from fire hose. Give it up for Sage. Hopefully not a fire hose. All right. Hi everyone, I'm Sage. I'm going to talk about what's coming up in Ceph Nautilus which is the next release due out at the end of February. All goes well. All right. Ceph, as probably most of you know, is a unified storage platform providing object block and file storage within the same cluster. We do upstream Ceph releases every nine months these days. So we're just about to the Nautilus release here in February. The next release after this is going to be called Octopus and will be due out November. We're doing a strategy now where you can upgrade every release or every other release. So you'll be able to go luminous to Nautilus or mimic to Octopus but not luminous all the way Octopus. So it gives you an 18 month jump if you do sort of the maximum. Those are the upgrades that we'll be testing and supporting. Okay. So at a high level, yes. There's no mic for the room. Sorry. No PA. Sorry. It's for the video. Yes. All right. So we have four priorities that we were focusing on as far as upstream is concerned. One is usability and management. Ceph has developed a reputation over the years of being hard to understand and hard to use. So a lot of effort has been going in over the past two years to address that to make stuff simpler and easier to manage. A lot of the things I talk about will be around usability. Performance is important. Everybody's moving to all flash. We have to go faster. The new flash NBME cards are ridiculous. So there's a lot of work in performance optimization. There's a whole project called Crimson, or that's essentially rewriting the OSD to go much faster. Work around container ecosystem, so mainly supporting Kubernetes and CSI and so on. Then a lot of features around multi-cluster capabilities for multi-cloud and hybrid cloud. So those are the four main themes that we focus on. All right. So talk a bit about the ease of use thing. So the biggest thing that's happening for Ceph ease of use is the dashboard. So there was an early prototype of the dashboard in Luminous. That was a proof of concept in Mimic. We brought in the open attic project, and that's much better. Then we continue to expanding that with Nautilus. So lots of new features there. So we finally for the first time have community convergence across multiple vendors and lots of users on a single management dashboard for Ceph, and it's actually built into Ceph. So it's part of the Ceph manager dashboard. It's there when you install Ceph. By default, you just have to turn it on and set up an SSL key and so on. And then currently, most of the stuff in the dashboard is around monitoring and management of your RGWs and your buckets and your RBDs and so on, managing Ceph itself. And we're sort of growing into adding more features around managing the cluster. So you'll be able to add nodes and replace those keys and so on, but still work in progress, but lots of progress. Lens is going to be talking about this later today, so we'll get a great demo of that. One of the new exciting pieces in Nautilus and that's just being added is the orchestrator abstraction that I like to call the orchestrator sandwich. And it's essentially an abstract API in the manager that lets Ceph call out to whatever tools being used to deploy Ceph. So there are four plugins that are being worked on currently. One is Rook, which is the operator for Kubernetes that deploys and manages Ceph. There's Ansible, Deepsea, which is the salt-based one that Suza works on. And there's an SSH1 that's going to be sort of a bare-bones-trivial thing. The idea here is that Ceph can call out to the orchestrator and ask it to go provision an OSD on a device, and then the orchestrator will do whatever it needs to do to run the right commands on the right host to do that work. And this is the piece that's going to allow us to have a generic CLI interface or dashboard interface to do these sort of lifecycle management functions. So things like fetching node inventory, creating or destroying demons, blinking device LEDs, which has been a sort of a frequently requested feature. And there'll be sort of a single CLI that will be the same for Ceph, regardless of what tool you use to deploy it. So that's the goal. And eventually, the dashboard will be able to do all this stuff too. So Nautilus includes the framework. And we have sort of in-progress implementations of these four, exactly how far and what features each of those implementations are going to have. And Nautilus remains to be seen. It depends on how much gets done over the next month. But this is all new functionality. So a lot of it's going to get back ported also to Nautilus as well. So we're very excited about finally doing the last mile so to be able to manage the full Ceph cluster from the dashboard and CLI. One of the big new RADOS features in Nautilus is PG autoscaling. So picking a PG num for a RADOS pool has always been sort of a black art. It's sort of the hardest thing to explain to users and have them understand and pick a number that isn't just wrong and going to be problematic. It was always confusing that documentation was hard. So in Nautilus, there are sort of two key things. Previously, if you picked a PG num, you could always increase it so you could increase the level of sharding within a pool, but you couldn't decrease it. So in Nautilus, you can now decrease it and it'll actually merge PG's into fewer. And so if you make a pick a number that's too big, you can fix your mistake. So that's the big key capability. And then on top of that, we have a module in the manager that will auto tune the PG num for you. And so it basically looks at all the RADOS pools in your cluster, looks at how much data they're storing, and how many total PG's you have. And it figures out that, oh, this one should have fewer and this one should have more. And it can either issue a health warning telling you that, that your PG's are too big or too small. So you can do it, or you can just flip the switch and it'll automatically do it for you in the background. And there's a, you might not be able to read this, but there's an example of the command that will just tell you the status. So it'll tell you what pools you have, how much data they have. They might be empty because you just created the cluster, but as an administrator you can tell the plugin how much data you expect to store there. So it can pick an appropriate PG num. Or you can tell it the fraction of the cluster that you expected to be, like this pool will be 20%. And that will feed into the same algorithm. And then it compares how many PG's you currently have versus how many it thinks you should have. And if it's off by more than a factor of three, then it'll basically either do the action or tell you to make the change. And it can either warn, do a health warning, or it can just do it for you. So hopefully this is gonna be able to let most users just not think about this. In sort of the worst case scenario where you deploy every PG with one PG to begin with, and then the pool fills up over time to be a petabyte, it turns out that if you sort of asymptotically look at the total amount of data movement, you basically end up writing everything twice. You'll end up writing the data once, and then it'll move one more time before it ends up in its file location. So sort of the worst case scenario isn't too bad. But if you tell the cluster what's gonna happen, then it'll do better, obviously. All right, device health metrics are sort of another key thing that we've taught stuff about. So the OSDs and monitors now look at what the underlying physical storage devices that they're consuming. Underneath all the layers of LVM and DM and whatever else, they look at the device model and serial number. And they report all that back up to the manager, and the manager maintains this. So they have a set of CIF device commands that will list your physical devices and what demons are using them. And then there is a set of capabilities in the OSD and monitor to scrape all your smart health metrics and store them in rados. And there's a module in the manager that can operate in two modes. One of them has a pre-trained model that will try to predict the device life expectancy based on the smart metrics. And that's the local mode. And you can see that reflected here. And then there's also a cloud mode that is contributed by a company called ProfitStore that will call out to their SAF service and either use their free prediction service that's pretty accurate or their paid service which is super accurate to figure out what the life expectancy is. And then based on an information, CIF can either just issue health warnings saying that this device is gonna fail soon or you have a lot of devices that are gonna fail all at the same time and that's gonna be a problem. Or you can flip the switch and it will automatically mark those devices out so that your cluster will automatically replicate to new replicas before the device fails so you don't have that loss of redundancy. So we're pretty excited about this. Even just having this visibility into what devices are attached to what demons is nice. And this will be tied into the, once we have the orchestrators knowing how to blink LEDs, which I think just deep sea is gonna be the first one to do it. But the others will follow, then you'll be able to tell it to blink this device's LED or you'll be able to tell it to blink this OSD's devices, one or more of them to give you that sort of full closing loop experience. And then one of the last sort of nice bits and pieces that we added to the manager is what we're calling crash reports. So previously if SF demon crashed you would get a splat and a log file on some host somewhere and then system D would restart it and you probably wouldn't even notice. Now whenever a demon crashes it writes a small record to Varlub crash with the process ID of the time stamp and a stack trace and then those are regularly scraped and reported to the manager. So the manager has a database of all the crashes that have happened recently in your cluster. And so you can do SF crash LS, Delissa's crashes and you can get info about them. And then additionally there is a, this is actually new in Mimic. There's a telemetry module that's in SF manager that is opt in. Obviously you have to turn it on but it will regularly phone home just very basic information about your cluster to SF developers saying this cluster exists, it has this many OSD's, it's running this version, it's this big, these are the services I'm using so that we have a sense of what people are deploying. So in Mimic that telemetry module also has the ability to phone home these crash reports. So if you opt in which I encourage everyone to do but you obviously don't have to do then we'll find out what crashes are actually being seen by real users in the wild on what versions and what those stack traces are so we can tell what's broken, what's broken the most and how we should prioritize our work and so on. So we're really looking forward to this actually getting deployed and being used in the wild. So those are sort of some of the things we're doing around management and usability. Other things that are happening in Rados that are a little bit more under the covers that are exciting. So Messenger 2 is a new specification and implementation of the on-wire protocol for Ceph. There was a talk yesterday that Ricardo did talking about this in a bit more detail. If you caught that there's also a recording. The main sort of user-visible feature that this is gonna deliver is for the first time we'll finally have encryption on the wire. So all the traffic that Ceph is doing between demons and between clients can be encrypted if you turn that on. But there's a bunch of other stuff too. There's improved feature negotiation. The encapsulation of authentication protocols is cleaner and better. Sort of paving the way for us to add Kerberos support hopefully by Octopus. And there's also all the infrastructure to do a dual stack so that all your Ceph demons can be using both IPV4 and V6 and both clients will be able to connect. Today you can do either V4 or V6 only one or the other. This will let you do both, which some people will probably like. We're also moving to our official IANA assigned port number for the monitors 3,300, which has been sort of a long time coming. So messenger two is this is actually the last thing that we're trying to finish up for the Nautilus release before we get out the door. So hopefully we'll get that sorted soon. A few other things, odds and ends on the Rado side. In the past it was hard to predict how much memory an OSD demon would consume and to control that by adjusting cache sizes. There's a new setting now called OSD memory target. That's just one setting, one number. You say, I want my OSD to use three gigabytes. And then the OSD will figure out all the stuff it has to do internally to fit within that memory envelope. Whether it's adjusting the size of the blue store cache or whatever else in it. Internally it's monitoring the RSS size and doing sort of a dynamic controller to make sure that happens. So that simplifies configuration management considerably. There's also a set of commands around NUMA. So the OSDs are looking at what NUMA node their network interface controller is attached to. And they also, if it's an SSD or NVMe, they'll look at what NUMA node their SSD is attached to and it reports that to the monitor. And there's a new command, Ceph OSD NUMA status that tells you the OSD and what NUMA node they're on. And then there's a single setting you can set that will pin an OSD to a particular NUMA node. And you can see that also in that output. So this simplifies the process of, for these high performance nodes of pinning OSDs to certain nodes. Hopefully if you have a balanced machine, the network and the flash are on the same NUMA node you can just pin the OSD there and you'll get better performance. Previously this was a super tedious process. You had to do your own bascripts to sort of manage all that, it was gross. But much simpler now. In Mimic we added centralized configuration management. So instead of having a Ceph comp file that was spread across the Zillion nodes it's all stored in the monitor. That's obviously still a Nautilus and we've improved it particularly around the manager modules. So you don't have to, you can set settings for manager modules and you'll have to restart the daemon. And there's better reporting for help, help for the descriptions for the options and what the legal values are and so on. So a lot of that just got cleaned up but works much better in Nautilus. There's also a new command that will spit out sort of a minimal Ceph.conf that you can put on a new node that has just enough information to contact the monitors and then everything else you can stuff in the monitor configuration database. So lots of stuff that just uses the configuration. There's also a new manager module called the progress module that attempts to essentially establish some state around long running events. So for example, if an OSD fails and there's a recovery operation that has to happen, it'll establish a progress event for that recovery and it'll actually tell you you're 50% recovered or 70% with the new TA and so on. So there's a new command, Ceph progress that you can see those events. Eventually we're gonna actually incorporate that into Ceph-S that didn't quite happen for Nautilus but hopefully by the next release. And this is a little thing but it's annoyed a lot of people when you have misplaced data that the cluster is simply rebalancing and moving somewhere else but the data isn't re-degraded. That's no longer a health warning. It shouldn't make your pager go off. You can go. You can enjoy your weekend. So it's a little thing. There's a setting you can adjust this but by default you won't get health worn when we're just moving data around which I'm sure some of you will appreciate. On the blue store side. So blue store was introduced in Luminus. It's the new default. It's been, it's good. People are happy. In Nautilus there are a few new things. There's a new implementation of the allocator that's used internally. So it's a bitmap and memory instead of a red-black tree essentially. So it's faster. It has predictable memory usage that's independent of fragmentation levels and there are certain behaviors of the old allocator that could lead to a sort of pathological fragmentation that this one doesn't have. So it's just better. So that's there. You won't notice for the most part. And it's just the in-memory allocator so you just restart the OSD and you'll get the new allocator behavior. So nothing on disk. There's also more intelligent cache management. It turns out that one of the hardest bits in or one of the harder bits in blue store was managing the size of the ROXDB cache for metadata versus the cache that blue store implements for all the data and onodes. Because you wanna, there's certain things in ROXDB that are really important to cache like the indexes and all the SST files. And so blue store is now smart enough to make sure that that higher priority stuff is being cached depending on how much memory you have and how big your store is and all the other stuff. So it's doing internally monitoring all that and making sure that it's doing the right thing. So that's much, much better and generally just get better performance. We also modified the on-disk format. So new OSDs will get this and you can do an FS check repair to convert old ones. But it basically, we are tracking more fine grained utilization metrics on a per pool basis. So for example, if a particular pool has compression enabled, we're tracking the amount of user data stored versus the compressed size afterwards. And also the amount of data stored for metadata internally to blue store or the OMAP metadata or the file data. Just much more granular understanding of where your disk usage is going. And that all bubbles up into CFDF and the detail view so you can actually see what's going on. And lots of performance improvements, just optimizations here and there. A few other rados things. In Luminous, we introduced the device class concept. So you could tag OSDs as either HDD or an SSD or any other type actually. And you could easily write crush rules that would only target one class or another. But prior to that, you sort of had to manually craft a crush map to do this and you ended up with this manually maintained hierarchy like shadow hierarchy with different nodes. It was sort of tedious, but lots of cluster did this prior to Luminous. And so there's now a feature in the crush tool to basically take that old style manually managed hierarchy and convert it to the new style without reshuffling all your data. So certain large installations will appreciate that. There's also a limit on the PG length. There was sort of a few corner cases where the memory utilization of the OSD could grow in strange recovery situations. That's fixed. So there's no hard limit, hard cap on the amount of memory that we use for recovery metadata. And one of the nice examples, sort of academic research translating into the open source project. There's a new erasure code style called clay code, coupled layered something. I can't remember exactly. That's basically a more optimal balance of IO and recovery bandwidth. So if you have like an eight plus three code, but you only lose one replica, then you have to read less metadata in order to do the recovery than if you had lost three of the additional parity or three of the nodes or whatever. So that's new. And it's in there. It's marked experimental because it's a new erasure code. You wanna be kind of paranoid about that stuff, but you should try it out. It should be pretty good. On the Rados gateway side. So this is the S3 gateway. We have a few new things, mostly around the Federation stuff. The first one and I think the most exciting is a new pub sub Federation capability. So you can create a zone that's essentially generating events when you have things like putting into a bucket or creating a bucket or deleting a bucket. It'll generate an event stream that you can subscribe to. So there's a polling interface where you can just ask it for events. I think this is the interface that we was used for a demo with Knative, which is a Kubernetes function as a service serverless thing. That was, there was a talk about this at KubeCon in Seattle a couple months back. And then we're also working on sort of push model where we push the event stream into AMQ or into Kafka, which some of our enterprise customers want. So this is exciting. You can sort of glue Rados gateway to serverless. So when you do a put of an object, it'll trigger a function as a service event that'll go do something to that image. It's pretty neat. There's also an archive zone addition. So you can define a zone within your RGW Federation that just gets basically a complete copy of everything in the other zones. And it turns on object versioning. So you'll have all versions of all objects. It's a relatively simple thing, but something that some people ask for just for compliance or backup or whatever else. There is an S3 API that implements life cycle policy, I believe is the correct name for it. Essentially saying that when you put an object, it initially is put in one tier and one set of Rados pools or whatever, over time it'll get migrated automatically to another tier and then at some other point in time it might get expired. There's a whole specification around what policies you can find based on S3. So we're implementing that now and it can control the tiering within a particular stuff cluster among Rados pools. So you've always been able to specify different tiers for RGW buckets or individual objects. Maybe a Risher coded Rados pool or replicated or whatever and this lets you automate that life cycle management across those tiers. And then on the performance side, a long time ago we used usually Apache and FastCGI to talk to RGW, then we used Sivit Web which is sort of an embedded simple web server. That's what we currently use by default now. In Nautilus we've switched to a new web front end called Beast which is part of the Boost project. It uses a Boost Azure, a more synchronous programming model. It scales better, it's faster. So that's new in Nautilus. And then there's also some new features around STS which is sort of the security authentication framework integration. I'll be honest, it's all a blur to me because there's so many standards and I can never keep track of which ones they've implemented but there's something new in STS in Nautilus that you can go ask the RGW folks about and they'll be happy to tell you. All right, on RBD, block device. The first big thing on RBD is live image migration. So you've always been able to have different rados pools with different performance characteristics or placement or whatever and have images mapped to those pools. With live migration you can have an in use RBD image that's mapped by a current, in use by a VM, actively reading and writing data and you can move it between performance tiers, between rados pools while it's being used and everything just works. So that's new, that's good. Excuse me. Another new feature that we're pretty excited about is RBD Top. This has sort of been an oft asked for feature. So there's a bunch of rados infrastructure now where the manager demon will essentially tell all the OSDs to start sampling their request streams. You know, either all of them or restricted by a pool or by object prefix and report that information back to the manager. So we have sort of a cluster wide view of what IO is happening. And then there's an RBD CLI that uses that service to basically subscribe to IO for RBD images and tell you what the most active images are, how much IO they're doing, that sort of thing. Excuse me. I'll be honest, I haven't actually used this feature yet so I don't know exactly what it looks like. But I keep asking about it and they keep telling me it's there so I'm really excited. Mostly I'm excited actually about the rados infrastructure so we have a cluster wide view not just for RBD of everything. So you can tell what the top clients are and who's using up all your bandwidth. A few other things on RBD. RBD mirror is the demon that does the asynchronous replication feature across clusters. For RBD, it's been there since luminous but it's always been kind of tedious to set up and configure. Excuse me. So a lot of work in just simplifying RGW management so that you can have any of them and they'll scale out. And also the configuration for connecting to remote clusters is now stored in rados so it's just simpler to use. Name space support is new. So rados has long supported name spaces. This idea that within a pool you can sort of carve it up into different security domains and you can restrict clients access to a particular sub namespace of that pool and then have that security enforced. RBD now supports that so you can have RBD users that are locked into a namespace. They only see their images and they can only and they're sort of walked in that way with Cephex. So that's all supported through the RBD CLI complete user experience that's been asked for for a while. Some things with the config overrides in RBD. So you've always been able to specify overrides on the configuration for the client on a per image basis. So for example, whether the caching is enabled for this particular RBD image without having to go configured on the client. You can now do that on RBD pools and the overall experience is simpler. It's the same CLI interface to control it all. Just cleaning it all up to make it simpler. And apparently now we're maintaining time stamps on images so when you do an RBD LS detail you can see information about what images are being used by when and when. All right, Cephex. So Cephex is awesome. The first big thing that happened is that the multiFS volume support is gonna be stable now and not less, it's actually been there since luminous but it was marked as experimental just because we didn't have a lot of testing around it. But Patrick's been cleaning up a lot of the code here and documentation and edge cases or whatever. And so it's gonna be pretty sure it's gonna be stable and not less. This means that you can create multiple Cephex file systems within the same Cephe cluster and each of those file system will have an independent set of MDS servers. So they're totally independent but sharing the same radius. We also have now a first class concept of sub volumes. So previously with Manila, we wrote this library called Ceph volume client, confusingly named, that basically just creates subdirectories within CephFS and sets a quota on them. It sets up Cephex capability so you can only access that sub directory and then it passed those to Manila as a share. So it's sort of a lightweight volume concept with quota. That sort of concept has been brought directly into the manager now. So you have a full CLI management experience for creating volumes, which are complete CephFS file systems and these sub volumes which are just lightweight directories within that file system. And the goal now is that everything is going to be consuming the same abstraction. So Manila will use it now. Kubernetes, RWX volumes with the new Cephseoside driver are going to use the same sub volume concept and you'll be able to, it'll be the same one that you access via the CLI and view on the dashboard. So we're happy about just sort of simplifying and unifying the experience across the board. On top of that, one of the big new things for Cephseoside is managed NFS gateways. So this is clustered NFS-Kanesha. You've been able to run NFS-Kanesha on top of Cephseoside forever and lots of people do it. What's really new is that Jeff Layton spent a ton of time making the scale out aspect of that and the NFS grace periods, making those work correctly so that if you had a single MDS fail, the grace period would be enforced across all NFS servers in a coherent way so all the delegations in lock state could be reclaimed. All the corner cases around the recovery that people can easily ignore or not even realize are problems are all sort of managed correctly. And so the NFS-Kanesha demons are now storing their configuration in RATOS. They have an object in RATOS that manages the grace. They're sort of doing this in a coherent way. And now those Ganesha demons are also managed by Rook at least when you're using Kubernetes. So Cephse manager using that new orchestrator API will call out to Rook to create the NFS demons or however many you want to do. And then they'll all be working together to do the grace period. So it's sort of a complete coherent experience and without a lot of tedious manual configuration needed. So we're excited about that. I'm a couple of other things on Cephse. There is an outreach intern project that created a Cephse shell. This is just a sort of a wrapper around Libcephse, a CLI that you can run and do like Maked or LS, things like that. And you can write scripting around. This is really mostly convenient for scripting around Cephse. So for example, if you wanted to set a quota on a directory, previously you would have to go mount Cephse and do a set adder and then unmount it. Now you can just do a CLI command to the shell to sort of do it all for you in a quick lightweight sort of way. And then there's just been a lot of work around bug fixes and performance fixes. Most of this is around situations where you have nodes with lots and lots of memory and really big Ceph metadata servers that with huge caches. There were a lot of sort of robustness and stability issues that we fixed with those types of environments. Or not less. Okay, container ecosystem. So lots of stuff with containers, especially around Kubernetes. When we, in the Ceph project, when we think about Kubernetes, it's really in two sorts of ways. The first way is that we want to provide storage to Kubernetes. We want to be the storage underneath all your containers because any scale out infrastructure is going to need scale out storage to go with it. And then the other is running Ceph clusters inside of Kubernetes, which might seem weird, but actually has some advantages because it simplifies and hides the OS dependencies. You have finer controller or upgrades, and you can automate and schedule a lot of the annoying tedious parts of managing a Ceph cluster with Kubernetes that you couldn't do before. So we have OSDs and monitors that have state. They tend to be tied to storage devices, but then you have all these other demons, the managers, the MDS servers, the rate of skateways, and so on, that can run kind of anywhere. And not having to think about it and just letting the container platform decide where to run them based on how much memory and network and CPU and so on is available, simplifies things. And so in this view, you're really thinking of Kubernetes as sort of a distributed operating system. Instead of thinking about individual nodes and managing them, you just say, I have a whole cluster, go run this stuff, and it'll figure out where to put them. So that's sort of where we think of things. Rook is a newish project in Kubernetes. It's a storage orchestrator, and that's sort of our, as a community, we're essentially all in on Rook as being the preferred or primary focus of how we run Ceph and Kubernetes. You can sort of do it manually with home charts and so on, but Rook automates a lot of the process, and it's great. And so we're focusing all our efforts on just making Rook work really well to run Ceph. And it does intelligent management of the Ceph demons. So it'll make sure that if you're adding monitors or removing monitors, it doesn't break quorum, it knows how to place them, it knows how to map OSDs to devices, it knows and understands Ceph well enough to sort of do all the right things when you're automatically managing it. I mean, it also gives you a Kubernetes style of specifying what your cluster is and how it should be deployed. So there's a CRD where you just tell Kubernetes, you should have a Ceph cluster that looks like this, and the Rook operator will go and do it, and you interact with it in sort of the Kubernetes way using KubeCuttle and YAML or all that stuff. And there's a talk later about Rook, yes. So make sure you don't miss that. But yeah, we're very excited about Rook, that's great. But not everyone wants Kubernetes, not everyone wants Rook necessarily. And so we're also thinking about sort of the old school style of deploying things. So with the new container orchestrator, we have all these plugins, right? We have the Rook one, the Deep-C, Salt one, Ansible, and there's also an SSH implementation that's just sort of like a minimal bare-bones implementation. The idea there is that you just give the Ceph manager essentially a root SSH key that knows how to talk to your nodes, and it can do all the basics. And that'll sort of capture the rest of us and drag us into the world where Ceph can manage itself. Similarly, Ceph Ansible recently learned how to run all the Ceph demons in containers instead of just running them the normal way with SystemD. And it turns out all it really does is create a SystemD unit file for each demon that does a Docker run and runs a container, right? So there's no real magic there, but you have the advantage that your demon is in a container, it can be upgraded independently and so on. And so the plan is to teach the SSH orchestrator how to do that too. And so that the sort of the default bare-bones implementation of the orchestrator API that Ceph will include, will be able to run Ceph in containers. And the advantages here are installation is easier, right, all this logic around, how do we install on Debian versus CentOS? What Apple repos do we need? All this stuff that was just super gross and annoying before is simplified because all you have to do is say, use this container repo and use this particular container version. And you can now upgrade demons independently one at a time. It's actually pretty nice. So I was pretty skeptical about containers a couple of years ago, like what's the value of containers to something that's low level like Ceph? But it really does simplify some of these operations once you sort of do it right and you figure out how to not lose your logs and so on. So anyway, that's coming. Just a couple of things on the community side. There's a new Ceph Foundation that we launched back in November through the Linux Foundation. We're very excited about it. Essentially it's a way to get all these corporate participants to pool money together that we can spend on the community. So don't tell anyone, but it's really just a way to get money to spend. But we have 31 founding member organizations, which is pretty exciting. Three more members have joined since launch. We have a couple other conversations going too. So we're excited about that. We also had our first Cephalocon conference in Beijing in last March, which was awesome. It was two days, four tracks over two days, like a thousand people showed up. It was bigger than I thought. It was really fun. So that happened and we're doing a new Cephalocon in Barcelona in a couple of months. So marker calendars, this is going to be the two days right before KubeCon. It's in Barcelona, which is great. The CFP just closed, so you missed the boat on submitting a talk if you haven't already. But early bird registration is open. So I encourage you if you're a Ceph fan to come. And there's also a reduced hobbyist rate too. So if you don't have an employer and you're just self-funding or whatever, you can apply for the reduced rate to attend. That's it. I'm happy to take any questions about Ceph Nautilus or anything else. Thanks. How far is the reuse cycle of post-movement began? And I was wondering if you were thinking about reconsidering it and I think what we want to mean is that we can move from this area. Yeah. Stretch us through and just turn the way it was reduced because we're not. Yeah, so the question is around a nine-month cycle and not being aligned with distribution cycles. Yes. The timing. Yeah, so the question is can we do three instead of two? The challenge is just that there's a lot of investment and overhead into testing and maintaining the ability to do those upgrades. It means that we have to keep around old behavior for that much longer and it gets harder to maintain and so on. And so two is sort of a compromise that we settled at. And so I'm not super excited about expending it to three, especially if two is enough to get you from point A to point B even if it's not as convenient. That said, people are always complaining about their release cycle. There's talk of expanding it to a year. There's talk of shortening it again. And so it's not that it's going to be that way forever. So if you have a specific request or whatever on that, I would e-mail the list and we'll bring it up. Thanks. Yes, the benefit is. So the question is around the NUMA configuration stuff and can I quantify how much it improves? It totally depends on your hardware. So there have been cases where we've had people doing high-performance reference architectures and they fix the NUMA stuff and they've gotten I think like 30%, 40% or more. But in other systems, it probably will make no difference. It totally depends on what the balance of the device attachments are across the two sockets on your system. It turns out there are actually very few machines that are actually balanced. Most of them just hang everything off one or they hang all the storage off one node and all the network off the other or something. That's what I have at home. So your mileage will vary. But Quanta apparently has a balanced NUMA node that has half the SSDs and network on both. It's one of the few that you can get. That we all have what? The question is how realistic is it that we'll have Rook and Kubernetes for Nautilus? So the Rook stuff is sort of a little bit decoupled from the Nautilus release. So Rook is in the 0.9 release. Lots of people use Rook today and have it in production even. We're working towards a 1.0 release. I'm holding out for the 1.0 release before I sort of pull on grab the megaphone because there are a couple of things with the way that OSU deploys that I want to clean up a bit. But that's independent of Nautilus. So 1.0 will use the Nautilus release but Nautilus supports Rook today. The only real tie-in on the step side is the orchestrator plug-in and it sort of calling back out to Rook to tell it to do stuff. And that's only if you're using the new CLI or eventually the dashboard to do it, which is sort of brand new anyway so you don't really need it. Hopefully that answers the question.