 All right. Hello, everyone. We're having some problems with the projector. The thingamajigs aren't working, so we have no video. So I guess I'm just going to talk through my slides, and they're going to get a new thingamajig, and hopefully we'll be able to see a little bit later on. So welcome, everyone. Thanks for coming. I'm sorry we don't have anything to look at. You're just going to have to listen to me. Today I was going to talk about Ceph data services in a multi- and hybrid cloud world. Talking really about two things. First, what the multi-cluster and multi-site capabilities are that exist in Ceph today, and then also paint a picture of where we're going in the future, and try to sort of put it all in context. So I was going to talk a little bit about Ceph itself, talk about what I mean by data services, and then go through File, Block, and Object, and then talk a little bit about the future roadmap. So as most of you probably know, Ceph is a unified storage platform. It's a single system that takes over all your storage devices and can store File, Object, and Block all within the same system. All of that's handled on the lower, on the bottom of the stack by something called Rados, which runs OSDs that manage all of your storage devices and handle replication, failover, high availability, and so forth, so that you have a single system managing all of your storage. Ceph is obviously an open-source project, it's completely free to use, lots of different vendors offer Ceph solutions. We do upstream releases every nine months, so the last release was Mimic, which was out in May. The next release is going to be called Nautilus and is out in February in another couple months, so we're sort of just coming up on the end here. We release every nine months, and we maintain stable backports for the last two releases, so that you can stay on the stable release for up to 18 months, and then you can upgrade up to two releases at a time, so you can go Mimic to Nautilus, or you can go Mimic all the way to the next one, which is going to be Octopus. At a high level, the four key priorities that I have been talking about for the last couple of years in Ceph Project, the key things that we focus on are usability and management, making Ceph easier to use, understand, and consume, performance, because the storage landscape is changing, people are increasingly using SSDs and very, very fast NVMe drives, and Ceph can use all these devices, but in order to capture the full capabilities of these devices, there's more work that we need to do, so there's a lot of work on performance, and then the last two are container platforms, making sure that Ceph works well with platforms like Kubernetes, and making Ceph work in sort of the new world, the hybrid and multi-cloud world, and that's really what the focus of this talk is about. And the motivation here is that IT organizations today aren't managing computers in a single data center. They usually have multiple private data centers, and increasingly, they also have part of their IT footprint that's living in the public cloud, and so they're managing multiple sites, multiple clouds, lots of data spread around these different pieces of infrastructure, and they need, ideally, some way to make it all work together and keep a handle on it, and as we sort of look into the future, it's increasingly becoming more cloudy, so when we previously would talk about on-premise infrastructure, those infrastructures are increasingly running private cloud platforms like OpenStack, and developers, when they build applications, are increasingly making use of self-service APIs in order to provision all their applications, so they're using provisioning virtual machines in a programmatic way. And finally, the next generation of applications that people build and deploy are going to be, these next generation of cloud-native applications are increasingly going to span clouds so that they'll have, they're not just going to live in one data center, but they're going to live in multiple data centers and multiple cloud footprints, and so despite all the talk about new application architectures using stateless micro-services, all of these services actually have lots of state, they're just sort of trying to put it in one part of the stack, and when you have lots of state managing that state across multiple data centers and multiple clouds becomes a challenge, so that's really what data services is in this context, so hopefully we're going to get some slides here soon. I apologize about that, so when I talk about data services, there are really three things that that means. The first component is about data placement and portability. Where am I storing my data and how can I move it between different sites and locations, and how do I do that in a seamless way without interrupting applications so that I can move my data between data centers and the application is still up and available and users don't notice. The second component is introspection. I have lots of data stored across lots of different sites. What am I storing? For whom? How much? Why? Can I get rid of it? How much does it cost? Things like search and indexing and giving metrics and insights into what you're storing. That's sort of the second component, that's sort of, again, a layer. Abstracting up a level above blocks on the hard disk, what am I storing? And then the last piece of data services is around policy-driven data management. So what are the policies I need to enforce as far as all that data? Life cycle management, do I need to store it for a certain number of years to comply with conformance and so on? Automation as far as automatically tearing it to slower, more cheaper storage after it's certain age, that sort of thing. So these three components, data placement, introspection and policy management are sort of the three main things in the context of data services. But I think the... Oh, that's encouraging. We'll see if that works. The key thing to keep in mind is that data service is about more than just data. You can't just add a bunch of features to your storage system and expect everything to work. What really is about how those data sets are tied to the applications that consume them. For example, if I'm going to magically move my data between data centers, I also have to move the application that's consuming that data in a way that doesn't compromise availability and so on form. And for this reason, container platforms and application provisioning platforms are key. When we move things between data centers, we also need to reprevision the applications on top of them. And this is why for the first... We hear a lot about new platforms like Kubernetes because they allow you to programmatically declare what your application should look like and what data it consumed. And so a lot of this work is not just features in the storage layer, but also features within an application orchestration layer like Kubernetes. So I'll mention that several times. So when we talk about features in the storage layer, things that Seth can do, there's sort of five key scenarios that we talk about in terms of data use scenarios. The first one is multiple tiers. We have different kinds of storage that are appropriate for different applications, fast storage, slow storage and so on. People are pretty comfortable with that generally. The second is mobility. So we want to be able to have data sets that are active in one tier and move them to another, or that are being used in one site and move them to another site. The third is disaster recovery. So you might have all of your applications, your entire infrastructure running in one data center, and you want to be able to survive a site outage so that if everything is a flood or whatever, you can fail over to another site. Does this kind of work? Hopefully. All right. Cross your fingers. Almost there. Maybe. I'll keep going. So multiple tiers. This first scenario, second scenario would be mobility, being able to move your data between sites. The third would be disaster recovery. So you're using it in one site, but if it fails, you need to fail over to another. The fourth would be what I call a stretch use case, where your data coexists in multiple sites at the same time, and you want to be able to consume it from applications in both sites at the same time. And the fifth is what I call edge, which is sort of a bundle of scenarios where you have smaller disconnected sites that have a subset of your data and are operating on it. So how do we actually make use of this storage? We'll talk a little bit more about specifics. So starting with block storage. As we know, block storage devices, they emulate a virtual disk. They're exclusive access by nature. You usually have a, aha, that's looking better. You usually have, what's that? Can I use your laptop? I guess I'll do the sitting down. All right. These are all the slides that you missed. Five ways to do it. All right. We're going to talk about block storage. So we can probably even skip this. All right. So the first thing that you can do with RBD is what we call tiering. So you can have multiple RADOS pools in your stuff cluster. They're backed by different storage. You can have images that are backed by those. So different applications at different requirements. Most of you are probably familiar with this. One of the new features that's coming in Nautilus is live migration. So if you have an application whose needs change, while that VM or client is in use and using the storage, you can live migrate it between different storage pools within the same cluster. So moving it from slower storage to fast storage, or back again. This is new in the Nautilus release coming up in the next month. Another scenario that you can do is stretch. So you can take a single stuff cluster and span it across multiple geographic sites. That means that all of your data, assuming you set this up properly, is going to actually exist in both sites. If one of those sites goes down, your cluster is still available if you have a third monitor. So this satisfies both the disaster recovery case and the stretch case where your data is available in two locations at the same time. The challenge is that there are some sort of tricky restrictions because performance is going to be limited because of the latency of the links between those two sites. But if you add in tiers, you have a lot of possibilities. So you might have RATOS pools that are localized to a single data center, one pool that spans multiple data centers, and then you'd have the ability to live migrate between those pools to move an application from site A to the stretch pool then to site B, giving you the sort of app portability. You might even be able to migrate the VM depending on what your sort of upper layer application infrastructure layer can do. So several more possibilities. The challenge is that building stretch clusters isn't necessarily a good idea. So the network latency, as I mentioned, is critical. You need very low latency links to get performance. And you also need bandwidth because you need to be able to sustain high IO throughput when you have failure and stuff is doing recovery and repair. It's also pretty inflexible because you have a single cluster spanning all locations, you can't join existing clusters, and you have this high level of coupling. And it's only going to work in certain scenarios where you have data centers that are geographically very close to each other, which is sort of a subset of what most users need. So for that reason, we have what we call async mirroring, which lets you make stuff clusters across data centers talk to each other. So this is the image exists in one stuff cluster and then has sort of a write journal. And then the Arbidy mirror demon will asynchronously mirror all the rights to a second data center where you have a backup image. And that you can set a configurable time delay, but usually it can be a few seconds behind perhaps. So if you lose the first cluster, you can fail over to the second cluster. You have a point in time consistent backup image, but it's only a little bit old. You've only lost the last few seconds of rights. That means that you can solve this disaster recovery use case so that if the primary site goes down, you fail over your VM to the second site. And there are options to re-synchronize. If the site comes back up, you can fail it back. So sort of the full feature set that you would expect there. Incender, this feature is enabled. Various parts of it have been added in Okada, Queens, and Rocky. At this point, all the machinery is there to drive it. The main gaps are around deployment and configuration tooling. It's hard to set up. You can't do this with multi-attached volumes, but that's generally not an issue because those are a pretty small subset of the overall images that you have in a cluster. The real limitation is that once this failover happens, all the images are there intact on the backup cluster, but you lost all your Nova attachments. You don't know what the images are or what they're for. And this really highlights sort of the key missing link in all of this. And that's you can't just solve these problems at the storage layer. You have to have some coordination with the tools and processes that are deploying your applications. It's impossible for an infrastructure as a layer, as a service layer, to reprovision in your application a new site. You need something that's sitting above that. You can't solve it all around. And that's, again, why there's a lot of talk about Kubernetes and why we're very interested in that space, because that's becoming sort of the de facto way that people manage and deploy these applications. So that's block. Let's talk a little bit about file. So just a quick status update. CephFS and Ceph has been stable since crack-in. The multi metadata servers have been stable since Luminus. Snapshots are stable since Mimic. You can support multiple Rados pools. So you have this basic tiering capability that you would come to expect. You have automated provisioning with OpenStack Manila. In fact, it's the most popular Manila driver by a lot right now. And in short, CephFS is fully awesome. So we can stretch a CephFS cluster the same way we can stretch an RBD cluster. It has all the same capabilities and all the same limitations. Again, latency is critical. You have this big failure domain across multiple sites. You've got to be pretty careful. Sort of the one kink is that CephFS has this additional metadata server daemon that you're communicating for file metadata operations. And it turns out that for file workloads, in general, in particular, metadata performance is critical. And those metadata servers are going to be running either in one data server or another. And so the applications, if you have an additional latency, that performance is going to suffer. So again, stretch clusters might solve some use cases, but they're really not a general-purpose solution. So what can we do with CephFS across multiple clusters? So one scenario or one strategy we can use is what we call snap mirroring. So in CephFS, we have snapshots on any arbitrary directory that give you point-in-time consistency and sort of arbitrary granularity. You can snap whatever part of the system that you want. CephFS also has this RStats capability that gives you a recursive roll-up of the most recent modification within the hierarchy or how much data is stored in the hierarchy. And RStats gives you sort of efficient data transfer. And so if you pull all these things together, you can, on your master cluster, take snapshots at regular intervals, synchronize those across to your backup cluster, and then take the same snapshot and sort of stage all this data from your primary site to your secondary site. The key gap in sort of putting the solution together is that there's one missing feature in CephFS that will flush those RStats after a snapshot so that they're the most consistent and most up-to-date view. And there's a patch in flight that should land before Nautilus, hopefully that will provide that missing link, and then we can sort of put this whole solution together. The second, once that's there, then we can patch our sync to use these special attributes on files and add some scripting and tooling around it. So this is coming hopefully soon, not exactly sure when. This is one way to deal with it. But it's very much the traditional disaster recovery case where your primary site is where all your applications are running and the backup site is really there for backup and you'll lose the last few minutes of changes if you fail over. One of the questions that comes up though is do we really need this point in time strong consistency in order to do a disaster recovery for file? The easy answer is yes, because just for the same reasons that file systems care about the consistency of the data on a disk, applications care about the ordering of their rights to files, especially if they're things like databases. But actually it isn't always the case. There are a lot of applications that don't care. They're not that particular. And in fact, the fact that some georeplication features in other file systems, notably Gluster, are actually just built on our sync. They're really sort of in a best effort, eventually consistent fashion syncing files across and not giving you a point in time consistent view of the entire file system. The fact that these solutions exist and they're used and they work in many scenarios tells you something about the applications that require them that sometimes the application consistency isn't such that you actually need this strong, strong consistency guarantee. I'm gonna skip that. So let's look a little bit about how we deal with migration in a couple of different ways that you can deal with application migration in general. So the simplest scenario, you might have an application consuming some files and storage on site A. You stop the application, copy all the data over to site B, and then you start the same application on site B. That's what anyone would do manually. It works. Application maintains exclusive access, but you have this long service disruption. So we can improve on that by copying most of the data over while the application is running and then stopping it for a shorter period of time and only doing sort of that last synchronization to get everything in sync. And then you take that availability gap and you make it much shorter. That's an improvement, but you still have an availability gap. What we really would like in an ideal world is to be able to have no availability gap. So you can seamlessly move an application from site A to site B without compromising availability. And in order to do that, you need to have this in-between period where the application is sort of running on both sites as available at both sites and you're bi-directionally replicating that data set across the link. That gives you no loss in availability, but it does mean that you have this concurrent access to the data and that somehow needs to work. And in fact, you might run that scenario indefinitely. It might be that's what you always want to do. You always want to have your application hosted in multiple clouds concurrently accessing the data and that's just your normal model, not just sort of a special case of migration. So can we achieve this bi-directional file of replication? We don't have it today, obviously, in CFFS. We don't have a general purpose solution here. And the reason why is it's hard to resolve conflicts for any POSIX operation. So for example, if you have an application in both sites that modify the same file in different ways, which version wins? How do you resolve that conflict? Maybe they overwrite data. Another one truncates the size. Can you sort of combine those two effects? Or are they orthogonal or are they not? But so it's hard to make a general purpose file solution that just works in all scenarios. But when we're talking about applications running in multiple sites, they only do this if they're already written to be cooperative in some way. They're written in a way that they avoid those scenarios where conflicts would arise in the first place. And the way that they do that is by using sort of usually a mostly static data structure. They're careful about how they modify files, and so that conflicts don't arise. Or if they do, they're resolved in a trivial way, like renaming over an existing file to replace it, that sort of thing. And so they have sort of a simplified data model. But if you think about it, that sounds a lot like another type of storage, object storage, that also has a simplified flat data model and a simplified conflict resolution. So let's talk about that for a minute. Object storage. Why is object storage so great, particularly in sort of a cloud era? It's based on HTTP, so you can insert caches in the middle, use traverses, proxies, and CDNs, all that sort of thing. In the terms of the context of this discussion, it's great because it gives you atomic object replacement. So if you upload a huge object, it's only when you finally finish that upload that it atomically replaces an old version of that object and becomes available and becomes the most recent version of the object. So you have sort of a trivial conflict resolution. The last writer wins. And the fact that you don't have sort of partial overwrites makes things like erasure coding easy, so you can efficiently store it. It has a flat name space. There's no rename. Rename is very complicated and can lead to all kinds of conflicts. It's sort of scalable, efficient, all that good stuff. And my conjecture is that the future is going to be very object-y for lack of another term. So file storage isn't going away. It's going to be a critical part of your infrastructure. Lots of applications use it, and it's actually useful. The things that the file systems provide are very useful. Similarly to block, block is a key part of your infrastructure. It's not going to go away. We can't ignore it. It's well suited for a lot of applications. But my conjecture is that most of the new data that's produced over the next decade is going to land in an object store. So all the videos that are being captured, all the images, all the telemetry, all the medical imaging, all the genome data, all that data when we talk about all the zettabytes or whatever, it's mostly going to land in object stores. And so the next generation of applications are going to be written around this data model, and this is where we should focus most of our efforts. So I'm going to talk a bit about RGW Federation and CEPF. So this is the object gateway, how we do federation in multiple clusters. So the base piece is a zone. This is sort of what you traditionally think of as an RGW cluster. It's some rados, pools, and a bunch of gateways that serve up its data. Zone groups are simply logical collections of those zones that have a replication relationship, either active active or active passive. A namespace is sort of a container around the whole thing that encompasses a namespace of S3 style users or Swift users and the bucket namespace. So similar to how Amazon S3 has one single global namespace, a bucket in S3 exists all over the world, that sort of maps onto a single RGW namespace. You can create multiple namespaces if you like. So today, you might have multiple CEPF clusters. Each cluster might have multiple zones. In this case, each one has two. And you would take some of those zones and you'd group them together logically into a zone group, and they would have a replication relationship, either active passive or active active. And you can have multiple zones. It's not just pairs, but you could have up to N zones in a single zone group replicating each other. The real gap in this scenario here is that currently, those replication relationships are managed on a per zone level. So it's sort of like a site-wide or zone-wide thing that the operator sets up. You can't say that this one bucket I want to replicate into another cluster. This is coming in the future. And so sort of going back to our active active use case, where we wanted to have two applications consuming file and have some bidirectional application, I think the way to actually think about this is the data consistency model that you actually want is object storage. Simplified namespace, simplified conflict resolution. And so the way to achieve this is to use the NFS gateway that's part of the RATOS gateway that gives you file protocol access to objects, as long as you don't get too ambitious with it and try overwriting existing files and such, and then use the object storage layer to do the replication and the conflict resolution and all that good stuff. The way that the Federation infrastructure is built in RGW, so you can have different replication plug-ins. The regular one just actually replicates all the objects and all the data. There's also one for Elasticsearch that will replicate only metadata. It'll put it all in Elasticsearch, and it has a whole API for querying and doing searches across metadata about objects and user-provided metadata. There's a Cloud Sync one for Mimic that will replicate buckets to an external object storage system, like S3. There's an Archive one coming in Nautilus that will keep all versions of all objects that you've uploaded, so you can go back and retain them all for backup or compliance purposes. And there's a new PubSub feature coming in Nautilus that you can wire into a serverless framework so that when you put an object in RGW, you get a serverless event, and you can trigger something using K event or whatever. There's a talk at KubeCon in Seattle talking about that. But when we think about the big picture again, we have IT organizations that span private clouds and public clouds. You can actually include the public cloud in this mesh as well. And so the idea here is that you'd have a minimal stuff cluster that's sitting in a public cloud that is acting as a gateway and storing just the metadata about replication and so on, but then storing all the actual object data in S3. So today, we can replicate data that's stored in the RGW buckets into S3, and you can get them either via RGW or by talking directly to the S3 bucket using Amazon CDN or whatever it is. In the future, we'll want to reverse that as well. So when you put an object into the linked S3 bucket, it would also replicate back in RGW. That doesn't exist yet, but it's coming in the future. What is coming in Nautilus is a new set of tiering capabilities for RATOS Gateway. So today, of course, we have all the intra cluster tiering. You can have multiple RATOS pools. A bucket can be backed by multiple pools that are linked to your RATOS Gateway zone when you set policies on buckets and so on. In Nautilus, the ability that's coming is to take individual objects that are put into an S3 bucket and push them out and tier them to an external object store, like another object storage cluster or something like S3 or some other public object storage service, a match or blob or whatever it is. That's sort of the first thing that's coming in Nautilus. Later, there'll be additional set of capabilities coming. Things like encrypting the data before you push it into the external cloud. So you might have gateways that hold all your encryption keys and all your data is actually stored in public cloud storage services, where the prices are really low, but it's encrypted. Yes. But stepping back, I just want to talk about the big picture of what we're looking at with RATOS Gateway. So today, we think of RATOS Gateway literally as a RATOS Gateway, a gateway to your RATOS cluster as a single-step cluster. And we sort of have some geo-replication features tacked on for disaster recovery and replication and so on. And when you sort of try to get an object out of a gateway, if it's not stored locally, if it's stored in another site, the Gateway gives you a location redirect and points you at the right gateway. So it's a seamless experience, but it's really about islands of storage and sort of bouncing you between them as necessary. In the future, we'd like to sort of turn that situation around. And so instead, we think of RATOS Gateway as a gateway to a whole mesh of sites. You have this whole network collection of object storage sites where all your infrastructure is placed. And the Gateway is really a gateway to all of that and encompasses all of these tiering capabilities, replication, policy, and placement, all those sorts of things. So the Gateway might redirect you. It might actually proxy you. It might be encrypting your access before it pushes you out to a different tier, all those sorts of things. So it's really just a different way of thinking about the Gateway. And again, today, we replicated zone granularity. So it's really targeting an operator who's running the entire cloud service. And in the future, we want to be able to set that policy at bucket granularity. So as a developer or an application owner, one tenant of a cloud service, I can say that I want this data set to be replicated across these sites. I want this data set to be encrypted. I want this data set to have the GDPR policies applied to it. It can only land in these data centers and those sorts of capabilities. So just briefly, I'm going to talk about how sort of edge computing fits into this picture. So when we talk about edge and set at the edge, examples might be a quarter rack or at a telco point of presence. You might have autonomous vehicles with their own little networks locally that are semi-attached to a central site. Lots of different edge scenarios come up. Several other talks about that at this conference. As far as set is concerned, there's sort of a number of things that we've been doing to make sure that set is suitable for these environments and we can tie them into the overall picture. We need to scale down set clusters so they run efficiently and small sizes. We need to support multiple architectures. We build releases for ARM today. Power is coming in the future. There are some builds that have been done at OSU that work, but they're not sort of integrated into our upstream processes. And we need to make set more hands off so the system is more self-managing and can run without anybody looking at it for longer periods of time and automate the management and enable the ability for someone to operate thousands of set clusters spread across all these edge sites in an efficient and manageable way. And when you think about what data is actually going to live at these edge sites, there are a couple of different scenarios. The simplest might just be that all the volumes in the edge sites would be replicated back to a central site. So if the edge site disappears, you sort of have a disaster recovery backup. That's something you can do today. In the context of objects, I think these sites kind of fall into two main categories. One would be data producers. So they might be gathering telemetry and gathering all of this data, staging it locally, maybe doing some processing. And then when they're connected to a central site, uploading it. So you can imagine an autonomous vehicle that's gathering all of this video and telemetry and then it's eventually sending it back to Waymo or whoever so that they can train their models to do self-driving cars. Or you might imagine sort of the other end where you have edge sites that are operating on a small piece of a very large data set. They might sort of come down and cache. They have access to the whole data set, but they cache the bits that they need locally. But the key thing in our minds is that we want to try to solve these problems in the context of object storage and in the context of this view as federated self-clusters and a mesh of object storage. Because the object storage paradigm and data model makes it much easier to think about these problems. So I'll finish by talking just a little bit about Kubernetes because everybody likes to talk about Kubernetes. So why? Why does everybody talk about Kubernetes? I think the key thing is that enabling true mobility of an application and data between sites is a partnership both between the tools that you use to orchestrate and deploy your applications and the storage system that's actually storing the data. The storage system can't solve those problems on its own. And if the orchestrator tries to solve us on its own, it's going to be slow and inefficient. So we need to work together. With CEP we're primarily focused on enabling those capabilities at the storage layer, but then working with Kubernetes teams to make sure that we can expose the higher level user experience. So right now, that data falls into two categories. We have persistent volumes. There are drivers in Kubernetes. Multiple sets of drivers, in fact, that let you use self-storage with your container platforms. We're very excited about the Rook project, which is a tool for automating the deployment and management of your self-cluster. So there's lots of work going there. And the future really is around automating the process of taking containers and pods that are consuming persistent volumes and moving them over to a different cloud site. So replicating the persistent volumes over, reprovisioning those applications in a way that they're all sort of linked together, and you have a coherent view of what happened. And then the second type of data, again, is object data. Today we have sort of trivial provisioning of RGW clusters with Rook. Super easy to do. But what we're working on today is the ability to sort of as not as a cluster operator, but as an application user, as a developer, the ability to on demand have dynamic provisioning of bucket storage resources. So you're saying, I'm deploying my application. I need these buckets to store data and automate that process of provisioning that storage, similar to how you do it with persistent volumes today, but using slightly different interfaces. And the goal then is to have a consistent sort of the next step goal is to have a consistent experience so that if I'm deploying my application on a Kubernetes cluster that's running on OpenStack or on-premise, those object storage resources will be provisioned from Ceph, but the application doesn't know or care. And if I happen to be provisioning in Kubernetes running in like GKE or some other on-cloud Kubernetes, those resources might be provisions using the cloud's native object storage service, like S3 or Google Storage or whatever. But again, having a consistent application experience is the goal so that you just specify what the storage class or policy for that object storage is and the platform takes care of the details of which cluster it ends up landing in, that sort of thing. So bringing it all together, when we talk about data services, we're talking about these three key things, mobility, moving our data, introspection, what have I stored, and automating policies for how we're going to do it with computers instead of people. But in order to solve these problems, particularly mobility, there needs to be a partnership between that storage layer and the application layer. Ceph already has many key multi-cluster capabilities. So we have mirroring for the block storage. In the object space, we have federation, replication, cloud sync, tiering and archiving are coming soon in the Nautilus release. And so the set of capabilities covers components of the tiering, disaster recovery, mobility, stretch use cases. And we also have some of the introspection covered as well with the RGW indexing of your object storage so you can do queries on what data you have. But our future investment, as we look forward, is primarily around the object storage services. How do we do this with object stores? And in that future, Kubernetes is going to play a key role. So expect to hear more about that from us soon. Thank you. And thank you for your patience. I'm really sorry about the whole projection situation. I'm not probably over time, but maybe we can take some questions. I'm not really sure.