 Okay, good morning, everyone. So we'll get started now. Thank you to everyone that is joining us today. Welcome to today's CNCF webinar. Everything you need to know about storage for Kubernetes. My name is Sanjeev Rampal. I'm a principal engineer at Cisco and a CNCF ambassador. I'll be moderating today's webinar. We would like to welcome our presenter today Alex Chakop, founder and CEO at storage OS. Before we get started, a few housekeeping items. During the webinar, you will not be able to talk as an attendee. There is a Q&A box at the bottom of your screen. Please feel free to drop your questions in there. And we'll get to as many as we can through the call and at the end. Note the Q&A box is different from the chat box. Please use the Q&A box. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that code of conduct. Basically, please be respectful of your fellow participants and recent presenters. With that, I'm going to hand it over to Alex to kick off today's presentation. Please go ahead, Alex. Good morning, good afternoon or good evening, wherever you are. I hope you're all staying safe and are all well. Just a little bit of introduction about myself. I'm one of the founders and the CEO of Storage OS. We've been focusing on building a software-defined cloud-native storage platform. We'll talk a little bit about what a cloud-native storage platform means. I'm also the co-chair of the CNCF Storage SIG. And in the Storage SIG, we've been working on providing facilities like educational content, like landscape documents, as well as working with the TOC on some of the items, on some of the project reviews for projects that are looking to join the CNCF or to graduate through the CNCF process. Before embarking on this startup adventure, I've spent 25 years in infrastructure engineering, trying to build some of the larger infrastructure environments. I'm going to get a little plug-in for the CNCF SIG storage. All the calls are fully open and we meet twice a month. Sometimes it's to cover project reviews and project presentations from the community, and sometimes it's to work together on different contents that we share through the SIG repo. It would be really great to see some more members attend, so feel free to sign up. And we'd love to talk to you about the projects that we're working on in the CNCF. What I'm going to do in terms of agenda today is I'm going to talk a little bit about some of the challenges of storage in cloud-native environments and discuss some of the principles of what to look for in cloud-native storage environments. I'm then going to cover some of the aspects of the storage landscape, which are covered from the CNCF landscape white paper document that we've created. I'm also going to talk about some of the internals of how Kubernetes uses APIs like CSI to work with different storage providers to manage storage within their environment. And then finally, assuming my little prayer to the demo gods went, was heard, we'll move ahead and have a little demo where I can show the use of some of the Kubernetes technologies like storage classes and PVCs and PVs to run a stateful workload like a database in Kubernetes. So first off, we want to discuss why storage important. So I'm going to say something fairly controversial here. Despite everything about containers being stateless, there is no such thing as a stateless architecture, right? All applications are looking to store state somewhere, whether it is consuming, whether those applications are consuming services that exist outside of the Kubernetes cluster or whether they're using services or storage functionality that lives within the Kubernetes cluster. Now cloud-native storage is a very different beast. When we talk about Kubernetes, we're very familiar with the concept of pets versus cattle, right? So the concept that nodes are specially curated and have special resources or special services is a complete anti-pattern to the way Kubernetes clusters are deployed. What we really want is we want an environment where your Kubernetes cluster is treated like cattle, so every node is homogeneous and can be used to deploy applications. Now, in much the same way that today a developer can use a bit of YAML to specify, hey, this is my application, this is what it looks like, it needs this amount of CPU, this amount of memory, and maybe some network connectivity, Kubernetes then goes away and takes that declaration or that piece of YAML and kind of does a really good job at playing Tetris with your infrastructure. So it abstracts away all of the services that are available in the infrastructure and does the best to fit the applications in the most efficient way to that deployment and makes sure that we maintain rules to maintain scalability or health. So based on the fact that applications within Kubernetes use this sort of functionality, you should also look for a cloud-native storage solution where the solution is both declarative and composable, right? And in order to do this, you're looking for a storage system that has a rich API-driven process that Kubernetes can interact with. So in much the same way that you specify CPU, memory, and network requirements, you should also be able to specify something like I need 100 gigabytes and it needs to have these sorts of storage attributes like replication or encryption applied. The second challenge we see with cloud-native storage is that data needs to be able to follow applications, right? So in a traditional storage architecture, you're very often presenting volumes or consuming storage on specific servers or nodes or VMs. And in cloud-native storage, we have a major difference here. The storage isn't being consumed by the node, but rather it's being consumed by the application that's running on the node. And a lot of effort, obviously, has gone into containerizing the application and making the application portable so this application can now run on any nodes in a cluster. So you need to make sure that the storage subsystem can also apply the same rules and effectively be able to follow the application wherever it moves within the data center or within the cluster. The other thing is, obviously, since Kubernetes does such a great job at abstracting away the infrastructure, allowing you to effectively deploy an application in much the same way, whether you're running bare metal on-prem or in VMs or in cloud instances, you're also looking for a storage system that can be platform agnostic and work across all of those different platforms, too. And finally, and I know this is a little cliched, but I'll go as far as saying, you know, you want to look at the agility of the storage system, too. So it's one of the things that needs to be considered because Kubernetes environments tends to be more dynamic. They are designed to be dynamically scaled and dynamically upgraded and therefore nodes come and go and nodes are scaled up and scaled down. So you want to store a system that can adapt to the changing nature of the Kubernetes environment. And then finally, this is kind of obvious, right? The people are deploying the automation and using Kubernetes to automate everything with their infrastructure, right, and make it predictable and recreatable. So this kind of feeds on the concept of cattle versus pets, too. That said, because we're looking to automate everything within the infrastructure, we also want to make sure that the automation applies to the storage environment and that the storage environment is consistently available and sort of protects the data within your Kubernetes cluster and can also provide the appropriate levels of performance and security that interact with the Kubernetes environment, right? So, you know, predictable deterministic performance and security that natively works with Kubernetes access controls and Kubernetes namespaces, for example, are strong attributes that you should be looking at for. So I'm going to move into talking next about the CNCF storage landscape white paper. The landscape, the storage landscape white paper is a document that we started working on now probably about dating 24 months back. And we've had contributions and lots of reviews on the document. So what we've tried to do here is we've tried to explain the different components and different attributes of a storage system, covering where a storage system can be deployed, the various attributes of the storage system, the layers and the topology within a storage system, as well as how you access storage and the management interfaces that are used on the control plane for those storage systems. The reason why this is important is because storage is a very broad topic, right? So we tend to think of storage traditionally as volumes, but in a cloud native world, storage is any way you can persist data. So in the cloud native world, you kind of have two types of methods of doing that. One is with traditional volumes and one is sort of API based methods, which includes things like object stores and databases and key value stores, for example. And more than ever before, it's becoming really important for developers to understand the storage attributes of their system and to understand how a storage system works, because they need to understand, they need to ensure that those attributes match to what their application requires. So we'll talk a little bit about the attributes in a second. The first point is there are lots of different ways of deploying storage within an environment, right? The more traditional hardware solutions are typically deployed in a data center, but obviously some of those solutions have limitations on the portability and the use of those solutions within cloud environments. So software solutions tends to be more platform agnostic and some software solutions and some of the projects that we're looking at in both commercial vendors and CNCF projects, you have software solutions that can be deployed as a container and the deployment can actually be automated through an orchestrated too. And finally, of course, there are storage services which can be consumed from the public cloud providers. So public cloud providers provide services like object stores or disk volumes too. So we talked a little bit about the storage attributes, but why is it important to understand what these different storage attributes mean? So when we dropped the landscape white paper, we classified the attributes into five different main types. Availability, scalability, performance, consistency and durability. What we found was as we were discussing these different attributes is that each of these attributes are important to different aspects of an application. So for example, most applications have architecture patterns that depend on a storage subsystem to provide a layer of availability. So if an application needs to move between nodes or failover, it needs to be able to continue to access the same data, for example. Similarly, another lever or another aspect that is very important for some applications is scalability. Now, scalability again can be measured in a number of different ways. It could be the scalability of the number of operations or the throughput of that environment, or it could be things like the scalability of the number of concurrent clients, for example, that can connect into the system, or indeed the number of components. So does the system scale horizontally or does it scale vertically, for example? Similarly, we talked about deterministic performance and often performance is one of the key measures of a storage system that people want to understand. But again, most storage systems apologies have a number of compromises and they are typically optimized for some combination of different performance parameters. So what we see is we see some storage systems, for example, that are highly optimized for low latency and are suitable, for example, for transactional systems, which require that low latency per operation. Whereas other storage systems are optimized for throughput, for example, for things like data analytics or data warehousing type operations. When we talk about compromises and optimizations of the different storage systems, the other aspect to keep in mind is the consistency. So most systems will have some level of, you know, different levers on the consistency attributes that dramatically affect availability, scalability and performance. And consistency is measured in two vectors. One is the amount of time that it takes to commit the data to non-volatile storage once it's being committed. And the second is the delay once that data has been committed to be able to access that consistently across all the different nodes in the cluster. And you'll find that some applications can tolerate eventual consistency and some applications need very strong synchronous consistency. And then finally, we look at durability. And durability sometimes gets confused with availability, but they are fundamentally different things. So availability is talking about the ability of storage to failover and to move between nodes and to provide some level of redundancy. The durability is a measure of the way that the data is protected on the back end in the system, and that includes redundancy. But it also includes additional measures like, for example, the way the storage system protects against corruption on the underlying disk media, for example. So we move on then to the storage layers within a storage system. So in a storage system, it's now more common than ever that a typical environment will be composed of a number of different layers. And some of the layers can be intermingled from different systems. So it's not uncommon to find different storage systems, for example, layered on top of each other to provide different services or different functionality. At the very top, you have the work that the orchestrator is doing. Things like virtualizing the mount points and making sure that mount points are available on different nodes within a cluster. And the integration into things like namespaces and the container bind irons, for example. Another factor to consider within the storage layers is the topology that the storage systems use. There are storage systems that have a very centralized topology, perhaps because they have some proprietary hardware interconnects or they're designed for scaling vertically. And those sort of systems tend to be optimized for performance and low latency, but obviously they have the complexity of only being able to scale vertically. You then move into distributed systems, which tends to provide much better scaling capabilities by scaling horizontally rather than vertically. But distributed systems then have additional complexity and additional latency requirements that need to be taken into consideration because the data is spread out across more nodes and therefore across more network connectivity is important. And then there are also technologies which often apply to databases called sharding, for example, where scaling and distribution of data is done by filtering the data into different buckets and placing different buckets on different systems. And the last topology I'd like to talk about is hyper converged, which is something that's becoming a bit more common nowadays. And what we're talking about here is an environment where nodes are simultaneously providing storage to a storage subsystem and used to provide storage capacity whilst also running applications and compute capabilities. The next layer down talks about the way the data is protected within those systems. We have things like traditional systems that might use some form of raids, which effectively creates parity or mirrors for the data. More commonly in distributed systems we see a ratio coding, which again is a way of splitting the data into multiple components and creating multiple fragments of parity in data that can be used to recover the data if any individual nodes goes down. And then we have the concept of replicas, where data is replicated in its entirety to a number of different nodes. And often this is a big sort of play between performance data protection and availability and durability. So things like a ratio coding provides amazing durability, but typically at a latency cost and things like replicas provide a lower latency solution, but typically at the cost of using additional capacity. Now, one thing that mustn't be forgotten is the concept that every storage system also provides some layer of data services. So things like replication of data or snapshot cyclones, which are point in time copies of data, which are, which are typically important for different workflows, whether it's, you know, providing disaster recovery or business continuity, or providing backup functionality in your system. And we can't, we can't forget the physical or the non volatile layer right so you know from traditional spinning media to fast solid state devices and moving forward to to non volatile, you know memory class type components, each of those components offer a variety of different storage attributes when it comes to, you know, obviously performance, but also more importantly things like durability to One of the, one of the things that's that's quite interesting is we try to put a, we try to put a table in place to compare the different storage topologies between local remote and distributed systems, and we kind of see that you know the topology comparison affects each of those storage attributes by, by quite a quite an amount right so local systems, for example, are limited and availability whereas remote and distributed systems have better availability and better scalability. But of course, performance tends to be tends to be lower in local systems and you know we'll talk a little bit about state of locality in a minute. But obviously remote and distributed systems tend to have slightly higher latencies. So, we move on then to the data access interfaces. And we talked a little bit earlier on about the concept of volumes and API is right. So it's, it's really important to distinguish between the data access interface which is, which is what a container or an application uses to actually access the data and the control playing that that the orchestrator uses to do things like dynamic provisioning of, of, of volumes or management of the storage system. So, it's probably fair to say that the most mature API's that are available today with with Kubernetes integration are volumes. So things like block devices file systems and shared file systems. And, and, you know, a lot of databases or, or, or message cues or instrumentation like primitives, for example, will depend on being able to use these sort of volumes. But likewise, of course, there are a whole suite of storage systems that provides object store interfaces or key value stores or, or, or databases that are accessed over, over an API. And those sorts of systems, for example, will, will, will typically be using that API over, over a network. Each of the different data access interfaces are typically suited to, to different, different sets of attributes. So, what I'm going to do, what I'm going to let that settle in for a second. And you can, you can see some of the comparisons on the screen. So, for example, you know, you typically expect block systems to perhaps have a lower latency or file systems to be able to provide the data, the ability to, to share workloads across multiple nodes, for example, and an object stores are well known for, for being able to scale to very large capacities, for example. But that said, one word of caution that caveats all of this is that, you know, we go back to the storage topologies and the layering. We often have to try and understand what is happening with the storage system right so sometimes, if you have block devices which are typically, you know, linked to, for example, a physical low latency device on the server. Block systems can now block devices can can now work remotely and can work on distributed storage systems and therefore they, they inherit the storage attributes of that distributed system. Similarly, for example, there are many file systems that are built on an object store backend, and therefore they have the, you know, they inherit the latency and the availability and durability of the object store rather than just, rather than just the attributes of the of the file system so understanding the layering is, is, is fairly important when we're trying to look at these comparisons. And then we move on to the orchestration and management interfaces. So this is how things like dynamic provisioning work between Kubernetes and the storage system. So the container orchestrator or Kubernetes has a control plane interface. There are a number of these interfaces, and they talk to the control plane in the storage system. Now, in some cases, the storage system supports the control plane API is natively. And in some cases, the orchestrator is talking via a framework or a set of tools to provide the bridge between the orchestrator API and the storage system API. Of course, as we discussed before, the workloads talk to the, talk to the data plane via the data access interface, which is, which is very separate from the control plane interfaces. So you might ask, okay, we've listed a number of different control plane interfaces, but what they all mean. So the key interface here is the container storage interface. The container storage interface is, you know, similar to the, to CNI for networking or CRI for, for the runtime is a standard API, which, which was adopted between orchestrators like Kubernetes and different storage vendors. So, so currently there are, I actually lost track, but there's easily 50 or more storage systems that support the CSI interface that integrate with Kubernetes. It's, I'm highlighting CSI because this is the, this is the fact that the factor standard for interfacing Kubernetes to different storage systems and different storage services. In the past, we had drivers that were mainlined into the Kubernetes source. And we had the concept of, of Kubernetes flex volumes, but the, the, we, it's if you're trying to implement a new system nowadays, we should, it's probably worth focusing entirely on on CSI because that's the standard where we're all the development and and advancements are happening today. So we'll talk a little bit now about how storage is configured in Kubernetes. So, so now that we talked about them, we kind of set the scene of the storage system with all those different layers and different data access methods and different control plane access methods. What does this actually translate to in, in, in real life and how are these, how do these systems interact with Kubernetes. So, we'll talk a little bit first about the concept of dynamic provisioning. So you remember when we said storage needs to be application centric. And therefore that enables, you know, self service and declarative and composable storage. How is this actually achieved. So the main abstraction layer here is, is, is called a storage class. So, so effectively a storage class is, is a definition that that specifies a driver interface typically through CSI that that driver interface will be will be used by the storage system to provide and manage the storage right. And this is in relation to dynamically provisioning the storage, but also things like attaching storage to physical nodes or to nodes within the cluster and mounting that storage and managing the storage life cycle. So, so that's all well and good. But what does a storage class really look like. So, so a storage class is actually typically something really, really simple like this. The storage class defines a name. So in this class we in this case we're calling it fast because you can create different storage classes for for different types of environments. It might specify some, some labels or flags that are specific to the storage system. And it defines the provisioner in this case where we've simplified it and just and just call it storage or S but if you were if we were using CSI the provisioner would be would be a CSI provisioner. What this translates to then is how do how do developers who are who want to use that composable storage how do they actually register what they need. So the constant here is a persistent volume claim. Right. So, within the definition of your application or your pods, you can make a persistent volume claim from the pool of storage that's linked to that storage class. Again, what does a persistent volume claim look like. Well, fundamentally, it's really, really simple. You give it a name. In this case, we're calling it a database volume. You tell it which storage class to use in this case we're specifying the fast and you specify the size of the storage. I'll talk a little bit about the access modes in a little bit. But fundamentally, all all it is is you say I'm going to give a name to my persistent volume claim. I'm going to use a particular storage class and it's this amount of capacity. It is also possible to pass to pass things like labels which which might affect the provisioning capabilities of which might be specific to a particular storage system too. What happens then behind the scenes is that Kubernetes talks over the storage interface to the storage system. The storage system creates a persistent volume. So, so we said that the developer makes a claim and the storage system replies by saying hey, here you are. This is your persistent volume. You then reference that claim in the pods and when your application in the pod is started that persistent volume is connected into the pod. So, moving forward a little bit. What does that actually look like so so this is this is an example of of a simple, you know, an empty pods that just just runs a sleep command. And uses a PVC one claim. So what we see here is when the PVC is requested. In the pod, the storage system will create the persistent volume like we just described, and then that persistent volume is attached to a note and is mounted typically in violet cubelit and along a long path name and violet cubelit. Once that happens and the container is then started that that part in the note is bind mounted into the specified man points, which in this case the slash mnt within the container name space. So, so the application starts and now has access to a file system or that volume under under in this case the slash mnt mount point. If that application then moves to different nodes within within the cluster. The reverse happens the persistent volume is detached and reattached on another note and then remounted and can be reused by an application on the other notes. So, if you recall we know we talked a little bit about volume access modes to so in the volume access modes. We have two typical modes one is read write once and one is read write many. So read write once means that the volume is mounted and accessed exclusively only by one note. This is this is typically the type of storage that you'd see from say block storage services. We also have the read write many and this will typically be used to access a shared file system so so effectively this gives the ability to mount a file system on multiple nodes simultaneously and those and you know that can be used for for different different storage patterns perhaps where a common file system needs to be available. To provide you know a consistent level of config or a consistent level of reference data to multiple to multiple nodes within a system. Okay. So what I'm going to do is I'm going to show a very very quick example assuming everything works of provisioning a stateful workloads and moving a stateful workload from from one notes to another in a Kubernetes cluster. For reference, I'm just using one of the simple examples that's available on the on the storage OS website. But obviously, you know you can you can run any application with any number of different storage systems. With in this particular example, I'm going to start up a my SQL database and I'm using the free storage OS developer edition which is which is which is free forever and available and available on our website. The stateful sets definition effectively tells Kubernetes to make a claim for my SQL database and to start it up with with file of my SQL in the mount but so I've already done this. So what I'm going to do now is I will briefly briefly on share my screen, and I will now share my demo. Sanjeev, can you just confirm that the demo screen is loaded. Yes, Alex looks good. Awesome. All right. So I'm going to be using, I'm going to be using a little tool called K9S to provide visibility of what's going on in the cluster K9S is a is an open source tool, but it's it's really great at visualizing what's going on in the Kubernetes cluster. If you could maybe magnify the font a little bit that might help. Right. Is that any better. That's better. Okay, great stuff. So, so this is this is a running cluster. We see the storage OS source software is running here as well as the CSI software and it's a, you know, a typical is a typical implementation of a Kubernetes cluster. We'll focus on the application that's running. So this is the, the MySQL pods. So before we, before we move forward, what I just want to show is I want to, I want to show the pods and the YAML for that pod, which is, which is running successfully here. And I want to be able to also, and I also want to show you the storage class. So we have the fast storage class here, and we can describe the storage class, which gives you the very simple information to say that this is using the CSI API to talk to, to talk to the storage system. We can look at the PVC. So we have the PVC called data MySQL zero. And it's, and it's connected to this volume. So we can also see that volume. We can see the persistent volume here. So this is the persistent volume. That's that claim is using. And what this means is that we now have, we now have MySQL running, running with a, in a stateful set on our cluster on a persistent volume. So as an example, I will show you the, we can connect to MySQL and quickly show the databases that are currently defined. This is a blank install. So this is just the stuff that you get on a native on a startup. We will, we will run a little bit of SQL to create a database called shop and create the table called books and create a fictional book called CNTF webinar into that into that database. And we can now query the database and we can see that there is the, the, the table was created and the book CNTF webinar was, was, was inserted into the database. So now what I'm just going to show you is this cluster is running on, on three nodes. And the MySQL, the MySQL pod is running on the, the node name that ends in 3U J9R. So what I'm going to, what I'm going to do now is I'm going to use the Kubernetes Corden command to tell Kubernetes that I don't want any new workloads to be scheduled. On that node. And what I am going to do now is I'm going to kill the stateful set that's running on that node. So there we go. I've just deleted the pod for that stateful set. Kubernetes is going to see that the pod was killed and needs to reconcile desired states versus actual states. So it's, it's now going in and creating a new instance on another node that ends in 3U J9B. Hopefully that container will, will start up in a second. Of course, this is where normally it takes two seconds and today it's decided to take a little longer. Alex, a few questions that piled up in the Q&A chat. So if you feel like you want to pick, want to do at any point. Okay. Actually, that's, that's a good thing. I'll, I'll just finish off this demo now and, and we can, and we can do that. So what we can see here is that the, the MySQL instance actually has now started on, on another node. And what we should be able to do is we should be able to query the data today's again. And what we see is we get the same, the same information back. So, so effectively what's, what's, what's happened here is the pod was terminated. Kubernetes restarted the stateful set on another node reconnected the PVC and that persistent data was, was, was provided there. We, we, we have a system where it using the orchestration capabilities of Kubernetes and the, and the functionality of cloud native storage. It's now possible to run stateful services like databases within your Kubernetes cluster and incorporate them into, into your environment. And it, and it also means that, that you can now quite easily build anything as a service, right. So, so whether it's a database and message queue, primitive services, elastic search Kafka, whatever that is. You can now run them as a service dynamically within, within your Kubernetes cluster with that persistent storage backend. So I'll stop sharing that demo and go back to go back to the slides very quickly. Slides closed. Nevermind. I'll just talk. I was only going to have a tanky slide up. So, so we can go through an answer and answer some of the questions. So, so hopefully this provided some useful background. I tried to make it, you know, I only tried to use storage OS as an example but, you know, hopefully this gave you a broader understanding of the of the storage landscape would would love you to have a look at the at the landscape white paper. And it would be great to to to get feedback on that white paper because we're about to release the second version that's further expanded and updated. The, the, there's a question about the meetups for for the CNCF sake and for storage OS for the CNCF sake we meet twice a month. The meetings are in the CNCF public calendar. It's the second Tuesday and fourth, sorry, second Wednesday and the fourth Wednesday of every month. We had another question to say, can I limit containers to access to spec files or inherits the volume settings. So, I'm going to try and interpret that question so so effectively the the containers will be using will be using the volumes and the volumes, just like the containers and the pods are created within the same concept of of the Kubernetes access controls so so things like namespaces and access to those namespaces and the filters etc that apply to those namespaces also apply to the to the to the volume abstractions to there is a question about Ceph and the Rook operator. So, so that's a very good example of, you know, the concept of having frameworks and tools that that are helping or that integrate with the storage system to provide the Kubernetes integration. So, so Rook is an operator and it's a framework for for actually more than one storage system. It's going through CNCF as a graduation process right now in the storage sake and and Rook can deploy a Ceph based storage system, Ceph being a storage system which is fundamentally based on an underlying object store but can provide block and and file system semantics to can multiple containers in a single pod share the same PVC. So, the answer is yes. There are two ways that this can happen if if there are if multiple containers within a pod can access the same read write once volume if they are on the same nodes typically. So this this is something that that some storage system supports. However, if if you want to if you want to follow the the storage patterns properly it's best to use the read write many or shared file system if if many containers need to be able to access the same the same volume. Provide. We had a question to set to ask if we can provide a little explanation on for on prem storage solutions for Kubernetes. So, so on prem solutions are quite varied because of course, you know, as we discussed, we have some of the traditional storage vendors so maybe things like traditional physical hardware storage array type solutions, some of which have very well integrated CSI drivers that can integrate with Kubernetes. But likewise, there are a whole, you know, there are suites of software defined storage systems, you know, like storage OS or or seven for example, that can that can be deployed on premise to And, you know, there isn't there isn't an answer where I can say, you know, this is this is definitely the storage solution that you should use for on prem. The reason being, you know, as we discussed is that there is a whole suite of, you know, different attributes that you need to look at and perhaps some decisions that might factor into the process might be might be cost based or they might be platform compatibility based. So for example, you might find it easier to deploy a software solution if you're running if you're running environments on prem and in the cloud or some sort of hybrid. But but it's it's something that you need to that you need to understand from from the back end from your back end system to There's a question to ask is storage OS the shared file system. How does it handle read writes from multiple nodes and do you handle file looks. So what you'll find is that storage OS can provide both read write once and read write many solutions and similar to some of the read write many solutions they are provided using using a shared file system. So in the case of something like stuff, for example, it might use the stuff fs file system. Other storage systems might be using might be using something like NFS, for example, as as the way of providing the shared file system across across multiple nodes. So, so in terms of how they handle things like file looks will be very dependent on the NFS implementation of the underlying storage system if they're using, if they're using, you know, a modern NFS implementation and support things like NFS version for then you should be able to handle state and lock failovers to So we're being done to provide a more user friendly platform or not to share it file system approach. I find it difficult to provide a solution that will work on many platforms and clouds. Right. So, so that's, that's a good question. I think if you're looking to find a solution that will work on many platforms and different clouds. You should be looking at some of the software defined storage solutions that are out there. There are a number of CNCF projects that provides software solutions and there are a number of commercial vendors, you know, like storage ones that provide those those solutions to software defined solutions typically are typically our platform agnostics, although, you know, you may find that different solutions might might be optimized or might have some caveats on on different services or different cloud providers that they're optimized for that they run. Can I depend on servers internal SSD was another question. So that's a very, very good question. And this ties into again the software based solutions. So typically software based solutions like storage or I sort of wrote, for example, can use storage that's available on on each nodes to to provide a storage pool that effectively spans the different nodes within a cluster. So yes, you can use the internal SSD and effectively the software defined storage system provides kind of like a storage persona to those nodes and allows and allows you to turn those those commodity disks or commodity systems into into a storage service with, you know, different data services based on based on the software solution you're using. Is there a CSI driver to allow accessing an object store via PVC. So the answer is yes and no. So, specifically a PVC. No, there, you know, the CSI API is is focused on providing functionality to access volumes within a system and those volumes typically mean block, you know, file or shared file. There are, however, a number of discussions which are happening in the Kubernetes storage sake right now that are talking about providing an abstraction to define access to to object stores and buckets and things like that through a Kubernetes abstraction layer. So, you know, as I mentioned earlier, volumes are probably the most mature abstraction within Kubernetes when it comes to storage. Things like object stores are happening right now so you should follow watch this space. And if you're interested in finding out details join the Kubernetes storage sake mailing lists. And with that, we are one minute from time, and I believe I've covered. I've covered all of the questions in my list here. Sinjeev, were there any other questions that maybe I missed out on. All right. Well, in that case, I think, I think we're done so I'd like to say thank you to everybody that joins the webinar and will be sharing the information and the slides later on. Yep. Thank you, Alex. Thanks for a great presentation. This was very useful. I learned a lot. Thank you to everyone for joining us. The webinar recording and the slides will be online dated today. And we look forward to seeing you all at a future CNCF webinar. Stay safe and have a great day. Thank you. Thank you.