 I'd like to thank everyone for who is joining us today. Welcome to today's CNCF webinar, Data Services for Cloud Native Workloads. My name is Ariel Jitib. I'm a Business Development Manager at NetApp and a CNCF Ambassador. I'll be moderating today's webinar. We'd like to welcome our presenter today, Abe Singh, Founding Engineer and Architect at Diamante, Shilpa Mayana, member of the technical staff and Narendra, Director of Product Marketing. Before we get started, just a few housekeeping items. During the webinar, you're not gonna be able to talk as an attendee. There's a Q&A box that you're gonna see at the bottom of your screen. Please feel free to drop your question in there and we'll get to as many as we can at the end. This is an official webinar at the CNCF and as such is subject to the CNCF Code of Conduct. Please do not say anything to the chat or the questions that would be in violation of that Code of Conduct. Basically, please be respectful while you are fellow participants and presenters. Also, please note that a recording and slides will be posted today later on the CNCF webinar page at CNCF.io forward slash webinars. And with that, I'll hand it over to the team from Diamante and kick off today's presentation. All right, thank you, Ariel. And good morning, everyone. Thank you all for joining us in these busy times. I hope everyone is staying safe. Let's get started. Just a quick intro. I'm Abhay Singh, Founding Engineer, Architect at Diamante. We have Narain from Product Marketing and Shilpa also from Technical Team, Engineering Team. I'll quickly go over the agenda. So in today's webinar, I'm going to start with some cloud native fundamentals before getting into data services and recovery point or recovery time objectives these services provide. Data services that I'm going to talk about today are the services that protect data against any kind of failure. It could be a software failure, hardware failure, node failure, network failure, data center and so on, data center failure and so on. And we'll also see why it's critical for storage architecture to adhere to cloud native principles for these data services. After that, we'll go with few demos that Shilpa is going to help me with and followed by Q&A, okay? These are the cloud native storage fundamentals that I'm going to briefly talk about, scaling, resiliency, isolation, tiering and mobility and how storage architecture should take care of these fundamentals at architectural level. So one of the fundamentals are scaling, right? So because the need application may require the need for resources as well as IOPS and bandwidth can go up and down depending on the need of application. So underlying storage infrastructure should be able to provide that support. So when we talk about scaling, a lot of times people talk about creating multiple replicas and it should basically scale in other words, scale out. However, that doesn't complete the whole story. The specifically for a stateful application, story completes if underlying storage infrastructure can also scale up the IOPS or bandwidth needed for the application whenever it is needed. So when we talk about scaling, it's both in terms of capacity as well as performance, provisioned IOPS, we should be able to scale up and down depending on the need. The next is resiliency. Microservices can fail and restart. So the restart could be on a different node, could be in a different zone, different data centers. So there has to be some data service at the back that provides data access whenever and wherever application or microservice comes up. Anything that can fail will fail at some point of time. So our infrastructure should be able to have those things built in to provide the required resiliency. It's not only microservice, but any of the components can fail, a node can fail, drive can fail. So in order to have the data available, we need these data services at services to support resiliency for microservices. The next is isolation. So the whole premise of cloud native is we share the common infrastructure and variety of applications can run on the same infrastructure, which means applications with different kind of higher loads, different requirements will be running on the same infrastructure. They would be sharing the same node, same CPUs, same PCI lanes, all the way to maybe same set of drives. So noisy neighbor problem is really complex to address and it's very important to address for such environments. It may, for security, we may require per volume encryption because there can be multiple tenants running on the same infrastructure. So you may need to secure each and every tenant or for that matter, each and every volume independently with their own keys to the next. Tearing. Some applications may require high access, high level of access. Another application may require medium or low. There are multiple ways to look at this problem. One, a lot of people solve this problem by media type. So anything that is a high priority or requires a low latency access would be located from, at least from the storage perspective, it would be located on flash. Secondary data could be located on hard drives or could be located far. What we believe is in standardizing the infrastructure and virtualizing the tiers. So the same infrastructure is used for all kinds of applications and what can be done is tiers can be virtualized. So high tier application can get, let's say 40K, 50K, 100K IOPS. There can be a mid tier, which may get say 10 to 20K IOPS. Low tier can get 1K to 2K IOPS. But everything is running on the same infrastructure. The underlying the storage layer takes care of the tiering. So we believe that NVMe is the new setup. Earlier, the quality of drives were defined by, let's say RPM, 15K RPM, 10K, 5K. And now that has kind of transitions to a number of drive rights per day. So for example, 10 drive rights per day, 5, 1, and so on. So it can be virtualized up to that layer too, up to that level too. So for one application, a volume can be provided that supports 10 rights a day. For example, for another application, a virtualized volume can provide one right today. And storage classes have been designed to do same. So these things can be configured using our storage classes in Kubernetes environments. So the last one that I'm gonna touch on is mobility. So application mobility requires efficient data mobility as well. In case of multi-zone clusters, the data could be sitting in two zones. Could be in two different rooms in the same data center or completely two different data centers. In case of failure or whenever an application needs to move to another data center, the data has to be available there. So it requires efficient migration and replication solutions or services at storage layer. So another way to look at it is for some of the secondary applications like test and dev or analytics, a user may require to move the data to a different cluster and run their work route. However, if underlying infrastructure itself provides isolation and tearing that I just talked about, then maybe those secondary applications can run directly on the same infrastructure without affecting primary applications and their performance, right? So in that case, it may not be needed to copy the data to another place for these applications or these workloads moving forward. So these are the data services that I'm gonna talk about today. Mirroring, snapshot, backup, replication. They all have their specific use cases and it's not that I'm not gonna advertise that you need mirroring and it solves all your problems and the same for other services too. We need all and they all serve different purposes. Let's take a look at what kind of RPU RTO these services provide and what actually it means. So all these services have a different schedule. Traditional tape backup where you move tape off premises. The schedule is done in days or weeks. Backup are typically done, let's say once a day or so. Snapshots could have a lower schedule. For example, a few snapshots every day, every four hours or every six hours. Replication is in general. When I say replication, I mean asynchronous replication. For synchronous replication, I'm using the term mirroring here. So replication runs behind typically a few minutes, could be 10, 15, 20 minutes. And mirroring is synchronous. So application rights are not acknowledged to the host until it gets written at all the places. So in case of two-way mirror, it needs to be written at both the places. Three-way mirror would require the data to be written at all three places before application IOS are acknowledged. It has latency implications. So we do not recommend mirrors to be separated by long distances. The schedule of these data services defines what kind of recovery point you can achieve. So if you need to recover data from the tape backup, the data that you're gonna get is gonna be days behind, depending on when was the last schedule tape backup. The same is for backup, snapshot, and replication. So going to, when you go to the last backup, that basically kind of defines what is the RPO going to be provided by these services. So how far back you need to go to recover your data? RTO defines how long does it take to recover your data from these data services? In case of traditional tape backup, it typically take days before you can get your data and have your application up and running. Disk backup, cloud backup also takes hours before you can populate your data and application can be brought up. Snapshot, snapshot is interesting because it totally depends on the architecture, how much RTO you can achieve. Schedule is in general configurable every four hours, six hours, so that defines the RPO. But how long it takes to recover, it totally depends on the snapshot architecture. If it involves data copy, then again, it can take hours before you can bring up your application and recover the data. However, an architecture can provide instant restore, then within minutes you can be up and running. The same goes with replication because data is getting replicated to a different site and data is available. It will probably be a few minutes behind, but it can be brought up, application can be brought up within minutes once the failure is detected. Finally, mirroring. Mirroring would provide near zero RPO and RTO. Near zero RPO because the data is synchronously getting returned to both the places. And once the failure is detected on one site or one zone, it can be brought up immediately on the second zone where data is available. So for most of these data services that I talked about except mirroring, Snapshot technology basically forms the base or core of the data service. Backups are typically done on top of Snapshot in order to be consistent. Replication can also be done on top of Snapshots. There are continuous replication solutions too, but if RPO requirement is beyond 10 minutes or 15 minutes for replication, then it's typically Snapshot based periodic replication is more efficient. So Snapshot architecture matters for all these data services. And whether we adhere to the cloud native principles is also important. For example, it's Snapshot space optimized that basically defines how much you can scale. Does it require data copy? If it requires data copy, then you may be affecting your primary application as well. And it also limits the scaling. How fast can data be restored? Can it provide instant restore? So that basically defines the resiliency of the system. If it takes hours before you can recover your application that's not resilient enough. Can Snapshot access be a different tier? So application is running in a high tier, a virtualized high tier can another secondary workload, whether it's taking the backup or running some kind of test and dev or analytics application, which can run on Snapshot. Can it be provided on a different tier, medium or low, so that it doesn't affect the primary applications running on the same infrastructure? Which means basically, is your storage infrastructure is providing enough isolation or tiering? Does addition of Snapshots affect parent volumes performance? Which basically clearly defined by isolation. Then both the workloads are actually not isolated from each other. So with that, I would hand over to Shilpa for the demos. Let me stop sharing. Thanks Abhay. Hi everyone, this is Shilpa Mayanna. Today I'll be demonstrating some of the key data service features that are available on DM&D platform. First let me share my screen. Okay, first I'll go with the mirroring feature support that we have on DM&D platform. So with this, you're gonna see how a stateful applications can seamlessly failover from one node to another node within the DM&D cluster whenever there is a node failure. So you're gonna see how applications can seamlessly failover without experiencing any data loss because the data is highly available on DM&D platform. For this demo, we have a topology setup this way. There is a cluster that's already created with multiple nodes. There's an application that is running on one of the nodes in the cluster. And this application is using a persistent storage on a DM&D volume. The volume itself is actually created on two different nodes in the cluster. It has two mirrors. So whenever the application is performing rights to the volume, the storage subsystem ensures that the data is synchronously replicated across both the available mirrors in the cluster. So with this, in case if there is any node loss in the cluster, if a cluster happens to lose a node on which the application is running, the application can seamlessly failover to another node where the mirror exists. You're gonna see that the application won't experience any data loss and it can start resuming from where it left off. So now I'm gonna start sharing the demo screen here. As I mentioned earlier, there is already a cluster created. This cluster has three nodes here and I've also deployed an application. It's a WordPress application that is using MySQL database in the backend for its persistent storage. These volumes are associated with the given PVCs here. And let's take a look at the volume that corresponds to MySQL PVC. You can see that there are two mirrors for this volume that are created on two different nodes in the cluster. Application itself is running on AppServe 86 and both the mirrors that are available for this volume is kept in sync, which means when application is performing writes on this volume, the storage subsystem is making sure that the data is kept in sync on both the available copies. Now let's verify what we have within this application. Currently, I have the WordPress application running on this particular IP address. So through the browser, I'm gonna just verify what the application has. There is a simple DM&T website created here, which is pre-populated with some blogs. As you can see, there are two blogs created. I'm gonna create one more blog here, saying that I'm gonna test application lower. Publish the blog. Let's verify this blog on the website. As you can see, the new blog that I posted appears on the website. The data for this is persistently stored on MySQL database. So now I'm gonna simulate a node failure. So in order to bring down the node, I'm just gonna cordon the node on which the application is running and then delete the pods manually. AppServe 86 is cordoned and both the pods that are running on this node has been deleted. Since these pods were created with Kubernetes deployment, the moment the pods are deleted, the deployment controller will automatically create another instance of these pods and this will cause the application to fail over on any of the available nodes in the cluster. Now let's verify the status of the new pods that are created. As you can see here, the new pod, both of these pods got failed over to AppServe 87, which is where the other mirror existed on the system and there is the same IP address associated with that pod. Let's verify the content of this application one more time. Look at all the blocks. As you can see here, although the application failed over, it didn't have any impact because it has up to date data. There is no time spent in the recovery of the data. So this is how the applications can get an RTO and RPO of zero when there is a mirroring feature enabled on the system. So now let's verify the status of the volume that corresponds to the application. As you can see here, one of the mirror for this volume is in detached state and the reason for that is AppServe 86 was previously cordoned. Now, the moment I uncoordined this node, the backing system will start the synchronization of this mirror and it will bring this, both the mirrors back in sync. Let's take a look at the volume status one more time. And you can see now both the mirrors are active and both are in sync. So this completes the demo that I had for synchronous mirroring feature. Let's move on to the next demo. So here I'm gonna demonstrate how you can instantly restore a volume from a given snapshot. Snapshot is a way of providing data protection for volumes within the cluster. So when you create a snapshot, it actually captures a state of a volume at a specific point in time. So this is how the topology looks like for the demo. There's an application running, which is using a Diamante volume for its persistent storage. You can actually set up a volume to take periodic snapshots in the cluster at regular intervals. And during this time, if application accidentally overrides any data within the volume, you can still choose to restore this volume from any of the available snapshots in the system. So we do have support for something called Instance Snapshots Restore. Here there is no time consuming data copy. It just happens within few seconds. The moment you specify a snapshot for a volume, all the data from the snapshot data will be available on the volume. So now let's move on to the demo here and I'll be walking through the steps on how to create a snapshot of a volume and how do we restore that volume from a snapshot. Again, for this demo, I am gonna use the same cluster and I'm gonna take a snapshot of a volume that corresponds to one of this application here. In order to create the snapshot, I'm gonna log on to Diamante UI. This is the cluster that I'm trying to operate on. So when you look at the list of volumes that it has, it reflects both the volumes that is associated with the app. Here I wanna create a snapshot of a volume that corresponds to my SQL PVC. So there's an option to create snapshot from the UI here and I can choose on which node I wanna create the snapshot on. So now you can see there is one snapshot created for this volume. Let's take a look at the snapshot page here. There's a snapshot that's an available state. At this point in time, this snapshot has all the information that was there within the volume for this application. So now I'm gonna do some writes on, modify this application to delete some of the blogs. Let me also delete this blog. Let's go back to the website and you can see right now there is only one block that exists which is called DMNT vision. So now I wanna now restore my application's data from the snapshot to recover all the blog that has been deleted. So how do I do that? I can choose to restore the original volume itself from the snapshot, but in order to do this, I'll have to first bring down the application. So since it is created as a Kubernetes deployment, I'm gonna scale down both of these applications to have the replica count zero. So at this point, the volumes is actually detached and it's an available status. Now you can choose to restore this volume from a given snapshot in the system. So this is the snapshot that was created prior to modifying the application. I'm gonna check this box to say I wanna proceed. Restore. So the moment I restore the snapshot, you can see the volume immediately goes into available state. So now this volume can be used to bring up the application. So all I have to do is again scale up the application here. I'm gonna reset the replica count back to one. Let's look at the status of this volume on the UI. Both of the volumes should transition to attach state. Re-verify the application to see if it has all the blogs that were previously deleted. So as you saw, like prior to creating the snapshot, I had all of these blog information. Later, I modified my app to delete these blogs. Now once I restored this application, all the blogs are back within the volume. This shows that the volume that we had was instantly restored from the snapshot within the cluster. So with this, I'm gonna move on to the next demo. So here, I'll be demonstrating how to set up a replication of a set of volumes across DM anti-cluster. With the last demo, you saw that we enable data protection for volumes within the cluster. Here, we're gonna extend this across clusters. This is basically done to handle disaster recovery. So in case of losing the entire primary cluster, the application can still be brought up on the DR site with the replicated data. However, when the application is brought up on the DR site, the volume that it's gonna use is based on when the data was last replicated from the primary cluster. So for this demo, I have two clusters created here. One is the primary cluster, which is also called active cluster. And the other one on the right side is my DR cluster. There is an application that's running on the primary cluster, which is using DM anti-volume. So whenever you set up a replication across clusters, there is a driver that runs within the cluster to facilitate this operation. On the primary cluster, there is a snapshot of a volume that's taken. And this snapshot at volume is used by the replication driver to copy the data onto the DR cluster. On the DR cluster, now again, a snapshot of a volume is taken. And this is basically done to keep the snapshots consistent across both the primary and the DR cluster. So for this, I think, let's go back to the demo screen. So I'm gonna use the same cluster here for the primary site, the one that I had for mirroring and snapshot. And there is a DR cluster that's created on the secondary site. Let's verify what we have on the DR cluster. There are no applications running. And there are no PVs or PVCs. On the primary site, I have WordPress application and MySQL database running. And both of these are associated with two PVCs here. Now I'm gonna set up a replication for both of these PVCs to copy the data from primary cluster to the DR cluster. So for this, let me go back to the UI here. First, I need to set up the volumes on the DR cluster. Currently, there are no volumes on the DR site. I'm gonna create a PVC here. And this PVC is created with the exact same name as it appears on the primary cluster. Let's wait for the PVC to be created. Now I'm gonna create the second PVC. So now both the PVCs are created on the DR cluster. Now we can actually go ahead and set up a replication for these two volumes on the DR site. So for this, we're gonna create a replication object. So since this is the target, I'm gonna name the replication object as target and also set up the role for this as a target endpoint. And here, there are multiple volumes that is replicated within a given replication object. So I'm gonna create a PVC group for it and add both the PVCs within this group. So now we have everything set up on the DR cluster for the replication. Let's go back to the primary site. On the primary site, the volume already exists. We just have to go and create a replication object for this volume. I'm gonna name the replication object as source. Set up the role as source here. I'm gonna specify the remote endpoint where the replicated data needs to be copied onto. So this is the IP address associated with the replication driver that comes up on the DR site. So here you can specify the interval, how often you want the data to be replicated between these clusters. Again, here we wanna group the volumes that we have here into a PVC group and save the replication object. So at this point in time, we have replication objects created both on the source cluster as well as the DR cluster. Let's see what objects got created on both of these cluster as a result of setting up the replication. The first step I created two PVCs to match the PVC that exists on the source cluster, which is MySQL PVC and WP PVC. There's also a PV that corresponds to these PVCs here and there is a DMT volume that got created as a result of this PVC creation. These are the volume that corresponds to both of these PVCs. And the replication object is created as a Kubernetes custom resource. The describe on this replication object will have all the parameters that were specified through the UI. This includes the information about the PVC map and the frequency for the replication and the role for this cluster. Similarly, on the primary site, we have the exact same replication object created. Let's take a deep look at this replication object on the source site here. So we have a list of PVCs provided to this replication object. There's a destination endpoint that is specified to copy the data on to. And there's an RPO interval here, which says this replication has to be done every fifth minute, nine hour. Currently, the replication is not running. It's in the detached state. Whenever the replication starts, you can see the admin state of this object going into attached state and the connection will also reflect accordingly. And once the replication completes, the status of this object gets updated to indicate when did the replication start, how long it took, and also the number of blocks that got transferred. As a result of creating this replication object, the replication driver creates a Kubernetes cron job and this cron job will be set up to run on every fifth minute, which is the replication interval that we specified. Currently, the replication is active. It started like few seconds back. Let's verify the status of this cron job one more time. Replication is also completed. At this point in time, you can actually do a describe on the replications and see the status of this object. Here it says the number of blocks that was transferred during this replication is 409,000 and the status of this replication says it is completed successfully. Now we know the volume that on the DR cluster has been populated with the data from the primary cluster. How do we verify what exists on the volume on the DR site? For this, I'm going to run a fire drill operation and in order to run this fire drill operation first I need to go and create a volume from the snapshot on the DR site. There are two snapshots that got created on the DR cluster after the replication process. So here I'm going to say, I want to create a volume from the snapshot for this fire drill application and this fire drill application expects the volume to be with a certain name. So I have to create the name, the volume with the exact same name and similarly for this as well. So both of these snapshots now has volumes created. So with this, let's go back to the DR cluster and try to spin up the application here. So this application is exactly same as it appears on the source side. Let's verify the status of this application. WordPress is running and this is the IP address that is associated with the WordPress on the DR cluster. Let's verify what this application has on the DR site. As you can see here, the DR clusters volume has exactly same blogs that were posted from the primary site. So this shows that our fire drill operation completed successfully. The data that we have on the DR site reflects the exact same data that we had on the primary site. This pretty much concludes what I had for the demo today. I'm now going to hand it over to Nareen. I'll stop sharing. All right. Thank you, Shilpa. Can you folks see my screen? Can you hear me well? Yes. Thank you so much. All right. So Abhay talked about the fundamentals of data services for cloud native workloads, as well as some of the attributes regarding the types of data services that we could have as well as what do we get out of each of those types of services, as well as the key demos of those requirements and features. But overall, CSI container storage interface is the conduit or the gateway for any data services and Kubernetes. So just to sort of cover CSI at a very high level, container storage interface is actually based on FlexVolume. And FlexVolume is something that DMRT actually worked on along with the rest of the community members, such as Google, to make it open source and contributed into the community back in the days, which finally evolved into container storage interface, which we use today for all storage services in Kubernetes. And this was the beginning where, this was the point of time when we could start running stateful applications on Kubernetes. And it was a very critical point to make Kubernetes the infrastructure for any application, whether it's stateful or stateless. At the same time, another important aspect here is we can have different types of data services. We can have different types of requirements for each of our applications, but at the same time, doing a bottom-up view of infrastructure up to the application, one of the key things is also scheduler extensions. So if we have an application, we should be able to deploy those cloud-native workloads using specific well-defined attributes that match the best infrastructure and resource choices. This enables us to utilize the best choice of infrastructure as well as efficiently utilize all of the options available to us. And that is driven through scheduler extensions, which again, Diamante was a key contributor to in the early days. So we heard about data services overall. So any storage system, any data system that we have, the key pieces that it needs to offer are mirroring, snapshot, backup, and replication. And we have heard about this. At the same time, okay, so those are the key services that we need to have, but what are some of the key requirements for data services infrastructure in a cloud-native environment, right? So just to recap from what Abhay and Shilpa covered here, scalability is very important. Scalability with respect to being able to increase or decrease in capacity, as well as according to the demands of the applications being able to cater to their performance requirements. Resiliency is very critical and resiliency on a per volume basis, not only as a full-fledged system, but being able to granularly provide that at a volume level. Isolation, yes, we can have the ultrafast storage system, but at the same time, it needs to be able to isolate noisy neighbors so that we don't end up over-provisioning the infrastructure because one application at some point in a very uncontrollable fashion is gonna take over the entire system because there are no bounds, there are no limitations of the amount of resources that it can use at any point in the system. And then, of course, isolation is not only for performance attributes, it's also for security reasons so that one application is completely separated from another one, control plane is separated from tenants, and give us those fundamental bounds within which applications can operate. Tearing, so end of the day, we will have one infrastructure, so we should be able to deploy many applications with different types of requirements. And the data services within that infrastructure needs to be able to provide this, right? And not only that, it also needs to be able to provide some sort of a policy-based mechanism so that scalability, resiliency, isolation, all of these could be achieved, enforced, and implemented in a very seamless and consistent manner. Last but not the least, in terms of the overall data attributes is mobility, right? So, and again, mobility is not just data, it's application and data mobility on demand or by choice. And being able to migrate applications, migrate data as well as to replicate data in a very efficient as well as timely manner. And of course, all of these things land into the key requirements that an application will have or an organization will have with respect to the RPO-RTO targets, recovery point objective, and recovery time objective targets. Another key point in this checklist is form factor. Are we looking for storage or data services system that is dedicated only to storage? Are we looking at a hybrid convert system here as well as or is it a cloud or hybrid cloud architecture or design that you would like to take on? Of course, cost is very important. So what is the cost per unit of storage with respect to replication, with respect to scalability, with respect to overall storage, all that matters. And also, while we are seeing exodus of data, exodus of information being out there, it also comes down to resource efficiencies. Are there ways that we can leverage advancements in silicon to actually be able to have a very optimal cost per unit of storage and hardware resource utilization to achieve all of these requirements? So with that, here is more information if you would like to learn about us. You can go to dmrt.com to see what's coming up from DMRT Next. As we talked about data services and stateful applications in a timely manner on May 21st, we are hosting a webinar on supercharging your stateful applications like your database applications, for example, Postgres, MariaDB, MongoDB, Microsoft SQL. So be sure to join us on May 21st. Of course, dmrt.com has a lot of very useful resources. Go to dmrt.com slash resources, there's a bunch of white papers, as well as key customer case studies. One of the key ones here, noted as indices on follow who fast-track their digital transformation with DMRT. Of course, at any time, you can follow us on LinkedIn, Twitter, and Facebook. All right, with that, let's transition to Q&A. Hey, this is Naira. I think there was a couple answers and folks responded, but we'll capture it here. We have three minutes left. Maybe the group can also respond for those who haven't looked at the Q&A box below. What happens if there's an interruption in interruption to the connection during the replication? Yeah, Irvind replied to the question, but let me repeat. So in case of synchronous murdering, if there is an interruption application would move forward with only one mirrors, but whenever the connection restores, it quickly re-synchronizes the mirror based on the chain block. So it maintains the differences and it only syncs those blocks. And the same goes for replication. Replication is also based on changed blocks. So if there is an interruption, it will, whenever the replication cycle restores, it will synchronize those blocks. Looks like we have another question that just came in. Anonymous, how many nodes are possible in a cluster? Is there a storage limit? I think those are two separate kind of concerns there. Do you want to try to tackle that one? I'll let Gopal answer this one. I mean. Sure. So how many nodes are possible in a cluster is basically a Kubernetes cluster. So Kubernetes basically with every release says and tested around 5,000 nodes. However, for us, what we have tested so far is 96 nodes within a cluster. So hopefully that answers that question. The second question was about the storage. So Diamante being an XCI platform, as Noreen pointed out, there are different conflicts of nodes. They can go from four terabytes per node to 32 terabytes per node. And so if you do the math and multiply that out with the number of nodes, you will get an idea of the scale. If you're talking petabyte scale, easily possible. I hope that answers the question. Back to you guys. Yeah, it did for me. I have actually one final question. A lot of this functionality is there in that storage, that Diamante storage stack and the diagrams and the layers. Is there any open source technology in there? I know things like FDO Arc, now Tanzu Valero or examples of open source punch tooling that provide at least for some of the functionality that you described. Is there anything that you guys are consuming at that layer that comes from projects and out of the ecosystem? That's a great question. Imagine when we say cloud native storage, really what we mean is that it should be consumed like a service, right? So take the case of public cloud providers. EBS can be consumed via Kubernetes cluster via the CSI interfaces, which are again all open source, right? But the way EBS itself and the storage services, which basically Abhay alluded to, they are all basically our own implementation. Now, you ask a question, what standard do we follow? So the standard which we follow is NVME. So in this implementation, we are virtualizing NVME to each pod, so to speak. The NVME is an open standard and it already has drivers available natively in Linux and Windows kernels, right? So from that perspective, yes, we leverage open standards like CSI, NVME and moving forward NVME over fabrics. And what Shilpa showed the examples of how replication objects are created and tied together is all within the six storage group. Six storage group, as you may know, within Kubernetes is one of the most active groups and they are us together in the community are trying to standardize on these storage features as to how they are consumed. So the CRDs and the objects basically are all based on what the six storage group is advocating. Did I answer that question? You did, thank you. And I appreciate the answer in Christie's patience as we ran a couple of minutes over, but I wanna thank all of you for a great presentation today. That is all the questions we have for us today and the webinar recording slides will be available later on today. And we look forward to seeing you at a future CNCF webinar. Have a great day, y'all. Thank you. Fantastic, thank you so much. Have a great day.