 Yep. Yep. Yeah Excellent We have a pretty light agenda today first of all Alex sudden if Alex is here today, but he sent apologies he has Been very very busy in the past couple of weeks and hasn't made any progress on the document yet But he has every intention of doing so before the next meeting We have a presentation scheduled for day today from Kieran from open EBS So if you'd like Quentin, can you see my screen? Yes, yes, I can thank you. I assume everyone else can too. So when you're ready Take it away. Cool. So this is going to be a quick one so the background of this is We discussed about this component called not this manager for a project in the Kubernetes face-to-face storage meeting that happened last week and Clint suggested that, you know, maybe it's better to present it to a wider audience So I'll just quickly walk through this one the intent is to kind of share what we are doing here and then also get some feedback and Inputs in terms of collaborations or ideas for this open source project All right Just a bit of a background about me. I work for Maya data I work on the open EBS project Open EBS project was one of the projects that we presented in the CNC of storage work back in February So one of the offshoots of opening peers project was there were several storage related problems that are pretty common for storage solutions So we're trying to handle this as multiple projects. So some of them are like no disk manager is one of the project Litmus is another e2e testing project that's coming up and then we host user which is about how do we Create containers that use spdk to directly access disks, right? So today, I'll probably be talking about the not disk manager so the Little bit of a background is that we have primarily three types of persistent volumes that we can create in Kubernetes today one is the network attached mode, which is primarily you know, either an External sand NAS or cloud disks the actual storage is outside of the Kubernetes cluster But then there are like two other modes that are coming up. One is a direct attached storage this is where a lot of work is going on with the local TVs and the other one is a hyper-conversed solutions like open EPS or cluster FS or root All these orchestrate storage engines that run within Kubernetes clusters So the common thing about the NAS or CAS is we need some kind of a mechanism to manage the disks that are attached to the Kubernetes clusters so that Some kind of a high-level operators can be written to create local PVs or in the case of container-attached storage solutions. It's about Providing these disks different types of disks to a CAS pod so that they can Provide some storage controller functionality and the volumes can be shared by different applications in local PV typically you kind of go from a Application to one of the disks Okay, so one regular pattern or like you know one Common implementation pattern that we see is each of these kth nodes can be linked with local disks or this could be coming from external targets like iskasi or FC or It could be cloud like GPDR EPS So all these disks will be taken and a concept called pool will be created and using this pool multiple TVs can be created for applications Pools can be of it could be as simple as creating a ext4 of a LVM-based host directory or it could be Portworx Volumes or cluster pool volumes are set Right What are some of the common challenges or Disc-related things that we need to handle or these caspots are typically long-running and they are on the critical IVO path. So it's not Advisable to kind of restart them very frequently and how do we handle the cases of disk failures, right? So if it disk fails then we should be able to Get notifications so that we can replace that with some other spare disks or take some corrective actions before the actual disks go bad So the motivation for doing this So though it started off as a Sub-project of OpenEBS it kind of started making sense to do it in a generic way so that multiple projects can use so one of the Places where this can be used is local PVs, right? So local PVs today is about Statically provisioned PVs that can be used by apps and there are there is a need for some high-level operators to be written which I think in the last few can be heard from Ian and Michelle about the node prep Kind of a containers that have to be launched that will actually discover the disks do some kind of clean up work or Formatting work and then provided to the local PVs. So these kind of operations can be generally done by the disk manager And then additional things like you know, how do we monitor the usage or like in errors? How do we handle the cases of dynamically? detecting that is attached attached kind of situations and then Notifying the operators to take corrective actions Okay, so While it complements the PV then they Within Kubernetes there are as when you're operating that at scale You would need some kind of Storage operation operators that you will eventually develop to manage the failed disks of the failing disks This is for that purpose and also how do we kind of Say the disks move from one node that node is gone into a state where it cannot be recovered. So that especially if it's a hard disk or if it's a Externally attached disk you can kind of detach it from that mode and attach to some other node or do you detect that? Okay, it's actually a disk that was used to store some data Now I can recover it from a different node and use it for a local PV that kind of operators can be built All this can be done if the disk information is Available with some unique identifiers and some kind of attributes. So this node disk manager Proposes a design and there's actually a prototype that we're building that can show how we can use Kubernetes custom resources to maintain a disk inventory with a lot of attributes in a uniform way that the Storage operators can use to build their functionalities So for CAS though this Becomes even more necessary because local PVs or position volumes itself cannot be used Since you cannot dynamically attach detach disks and Most often CAS will have that requirement of making sure there are multiple disks attached to that And you should replace a failed disk with a new disk kind of stuff and also ability to Predict failures before they happen so that's got some kind of a migration work broken be taken or let's say for performance reasons you want to Pull together different types of disks to kind of better utilize the cache or the expensive resources that are available Any high level questions so far or I can probably just get into the next I have like just couple of more slides Yeah, I Had one or two questions one is there seems to be a blurry line here between file systems and block stores and I just in your Volume types you you had both file system types and Block store types so ZFE and the XT for for example our file systems, but EBS and LBM are block stores So the blocks are really there are discs that are coming from the that's correct So the underlying discs that come are usually blocks But on top of that you can create the pools that could be either a file based or a block-based pools Sorry, you broke up a bit. I'm not I'm not sure if the problem is on my end or your end So did you explain? Do we do we Deal with block stores and file stores as if they're the same thing or do we make a distinction between them? good I just pause the sharing so that Maybe it's the internet on my end. That's causing the problem. Can you hear me? Okay now? I've been struggling to hear you, but I don't know if it's just my Problem maybe do a sound check now that you've stopped your sharing Quintan I just stopped the sharing. Maybe it's the problem with the bandwidth Yeah, it was working fine earlier on the call. It just became problematic in the last minute or two Okay, let's continue. I Guess you'll have to reshare We can hear you fine Karen. Go ahead. Oh, sorry for that So just to answer the question I think Quintan was asking whether We are using both file and block based file systems That's correct Quintan Though the notice manager manages the block disks from the underlying Storage of the southbound on the northbound Karen, are you still there? Yeah, I couldn't hear me there, but I can hear you find us on Yeah, maybe he dropped off. I'm not sure. Let's give him a few minutes. I Think zoom is pretty good at reconnecting automatically actually Okay, maybe do use the time wisely in the meantime, so I think you're pretty familiar with With some of this stuff at least Could you give us an answer to the previous question about whether there's a hard or no distinction between file systems and block stores in the volume interface So today There is a way to expose both block and file in the Kubernetes volume subsystem You have a way to request it in the persistent volume claim object to say whether you want block or file and then under the covers The plug-in can can decide how it wants to implement either of those And then you also have the ability to implement file on block Implicitly if you're a block Support file as well In the future. We're considering making file on block a first class field in or says first class Within the Kubernetes API This will help us with secure containers But that's that's basically it Okay, thanks. So I guess in summary it's a sort of a blurry line that they're both volume claims and they're both volumes But you get a subtype file or block Yep. Okay. Cool. Thank you. Thanks, Art Sorry, my connection is bad indeed. It's going I it's probably because I'm trying to Share this Your sounds very bad Bassa most least for me. Can you speak less to the mic? Yeah, I don't think he was talking to us. Maybe just join them Mike enabled Okay, I carry on Karen Yes, like we might have lost here I Shared the Google presentation on the chat window if someone could just help me Present that looks like my boundary problem is when I share it and then talk at the same time I tried to share it, but I don't have permission to the doc If you can give me for you I Just sent your request for that doing that right now. There we go. Someone's sharing it for you. Just while he's bringing that up This Georgia Erickson is a bit concerned about equating file system and volume. I Would equate maybe the storage of a file with a volume, but not a file system, which is a namespace Josh I Don't think I fully understood the question So the I can reach out to you offline and then understand that question or if somebody else understands the question From the Kubernetes perspective Yeah, maybe a question of terminology and we can deal with it offline. That's fine Cool, so the next slide if you can Thanks for sharing next one and the next one one more please This is the last one. Okay So the way this notice manager is going to work is it's going to be a demon set that's running on the storage nodes in the Kubernetes it's going to use different discovery mechanisms to Identify the block disks that are attached to a node and then it's going to put them into the Kubernetes as the custom resources, right? I have an example of how the custom resource looks in the next slide But using those custom resources the storage control plane operators like let's say the local PV or open EBS or cluster it is can use them to Create objects that they need for example, it could be like a local persistent volume or in case of open EBS we could be creating a Storage pool pod and similarly a cluster FS cluster Taemin or The cluster Taemin set can use these disc resources resources to identify which nodes should have the Pool created There's also a monitoring piece that's getting built into node disk manager that can monitor this disks for errors as well as The metrics in terms of higher of latency throughput, etc that will be Exposed to Prometheus that can be actually configured to be exported to Prometheus or there could be some alert setup or some events that can be Sent to the storage control plane so that they can handle that appropriately so some of the events that We have prototypes using you you'd have to kind of get the attached detach events for the disks and then Adding those resources and also detect some kind of a disk errors and then convert that into events and send it up So one of the future items for a node disk manager is also to be able to Take something cause it called as a disk claim so that it can provision the disks on the external storage provisioners for example on an sand or cloud Next slide please Okay, so this just explains some of the objects that I explained earlier The interest of time. Let's just go to the next one Okay, so this is a example disk resource object that will be created. It's a custom resource It will have information of the topology of the disk here. I've just taken from the Kubernetes GKE cluster. So it just shows the host name from which it's available What's the path and the capacity and then some details that we kind of got from LSBLK But there are Additional information that we can get in if it's a block disk or if it's a SSD or if it's NVMe connected or If you want to get to the topology at the CPU level those things can be obtained from this one So I'll stop here. I think the main intention was to kind of say that We have this project going on and we have Open EBS and then also Humble, I think is also on the call from Red Hat looking at how to use this for writing the Using this with the cluster FS and Open EBS We are really looking for contributions in terms of design ideas and then Maybe take it forward to see how to make it generic so that we can build some high-level storage operators using this Thanks very much Kiran. That was very useful Are you at a point where you want to open up for questions? Yes, I think now is a good time to take questions So one thing Yeah, one one other thing I wanted to still look at I haven't I need to get in touch with woman It's on the call to see how What they are doing in room. There is a similar functionality there To see how that kind of relates to this and if there is any collaboration that's possible with this one Sorry I was just saying I Question but I also I'm conscious of dominating conversation So let me step back and find out if anyone else has anything that would like to talk about or ask questions specifically about the presentation Okay, if not, if not, I'll ask my question. So it was around Storage networks. So most of the public cloud providers don't really have the distinction that I'm aware of between multiple works whereas in Enterprise data centers, it's not uncommon to have multiple storage networks attached or certainly within the data center and possibly also attached to a particular node If anyone or if you have given much thought to the concept of storage networks and how do you Distinguish between them and how do you advertise? You know, it's not Volume to be Advertised on multiple networks and sometimes at different performance characteristics, etc. Have you given any thought to that? I Haven't included that in the Currently stuff design, but that's a good one Quinton. I think it can it easily ties into the Things that Nordisk manager can provide in terms of topology So the net storage networks can be treated as a topology item that can tie in to the operators Yeah, I'll add that as one of the items here. Thanks. And I see one question from on the chat about How do we determine the Types of the disks So we currently do it using the type that is exposed Using the attributes using LSBLK and using the tail-based society attributes that I get But there's also plan to use the benchmarking to kind of set up the types Yeah, that's an interesting question. I guess one could distinguish between, you know at provisioning time Performance and so some kind of static snapshot of performance taken at some point in time Versus Performance over time so you can imagine when storage networks or the general network perhaps get congested or the Device that the storage volume is stored on gets uploaded. The performance might vary over time Which of those two are you considering? So the one that's already considered is the runtime performance benchmarking that we collect and then push it via alerts or into Prometheus The one the static Benchmarking is not yet done We basically use some kind of a map to save which type of the disk it is We plan to do the static benchmarking as well before assigning it to a Persistent volume or a pool Okay, that makes sense Just to get back to my previous question Is there anyone on on the call who can comment on whether or not they have any strong requirements for Even exposing the concept of a storage network to start with and then secondly Multiple storage networks in a data center or per node. Is that I I've seen requirement Some number of times, but I'm not sure how general the requirement is. So curious if anyone else has experience Around that point So I mean we I've seen this requirement When you're setting up the private data centers and then cloud I've not worked on the Kubernetes cluster with this requirement But non-cuban it is cluster. Definitely. This was a requirement Okay, good What is it that you're trying to manage? When you talk about the storage network, are you trying to manage the Fiber channel and DME or whatever aspects of the network or Or is it simply just connectivity between Storage and clients Well the question that I was so the use cases that I've seen in the past are one You may have multiple storage networks as you say the kinds of Mentioned and some of them are connected to all the nodes and some of them are only connected to some of the notes and so having If a given node is connected to a given Storage network, you know volume is is exposed on that network Then then you can schedule a pod that needs that volume onto that node, but not onto all nodes and then similarly there are different performance characteristics and perhaps even Permission and so you might have Storage network that's limited to you know certain tasks or certain users and then you may have a lower Maybe even on the on the general land where you can access the same storage but across the different lower performing network So those are the kinds of use cases I've seen in the past those should all exist and those are those are more about discovering the topology of Network connectivity Yeah, exactly, and it sounds like we don't have that at the moment So it wouldn't be possible for the schedule it up whether it was contactable enough Hey Quentin, can you hear me? Yeah, I can so there's one of the problems is No multi homing networks on container is still not supported in Kubernetes I've seen people do this and we've looked at this For Rook scenarios the special CNI plug-in that enables you to switch between between networks So they there's a body work I'll try to I'll post an email on the working group with with some details, but But that's one area that I think me You know we need to think through how a container can choose which path it goes out or uses When when you know money to use the network Yeah, yes, I'm well aware of the fact that that we don't provide multiple networks per part at the moment It's quite problematic in some application areas specifically building in a network infrastructure like And also telco applications which have multiple control network and data play network, etc There is one CNI plug-in that I think the folks are Intel started That does multiplexing But it's it's I say it's early work Yeah, yeah, we also have we have a thing called see an idea me. I think which does something similar I Guess a pertinent question in this space is is do we want to treat storage networks as Generic networks and sort of leave it up to CNI and the networking sig and these kind of people To to solve that or do we think that a storage network a sand basically one as is Is a special enough thing and distinct from general networks that Want to have for those and solve that in a separate storage centric way So I'd suggest that there's two levels to that answer At one level you're discovering the network connectivity between essentially ports on one side with ports on the other side and And switches in the middle At the second level you want to know what's underneath the port so you want to be able to To go through a port You know as you and say Okay, what can I see? And do discovery there Sorry, you broke up a bit there George I think I think my question was a different one Do we want to treat storage networks as distinct from data networks or do we want to treat them as a converged thing? It seems to me like there are arguments in both directions. I mean in many cases general purpose TCP networks are used to attend Bridge Public clouds work for example But there are other cases where you have completely physically separate and and the protocols that run over them are completely different and those are, you know unlikely to be solved by the networking sig unless we go and push on them to solve that I Would imagine it would be difficult in practice to solve the two problems separately because they're such a big overlap But we should just be aware of that If there is a strong requirement to support sands and in particular multiple sands per node or per data center, you know per cluster we need to probably engage closely with with this networking people to Make sure that gets solved. Otherwise, it's likely not to be Yes, as I was saying the topology between endpoints It certainly needs to be covered by the networking guys, but more broadly It's also the same problem if you're talking about an IPM I network as an Ethernet network or You know variants of that or a fiber channel network Second level okay any other questions Sorry, I've got a very bad audio connection. I keep talking over you George my apologies That's okay. What I said before was that there's a second level to that problem Once you can get to an endpoint on on some target Then there's a need to do discovery underneath that target What file systems or volumes or dis or whatever you can see there Yeah, indeed and and one could argue both ways whether that Belongs within a general-purpose platform Kubernetes or the kinds of things CNCF provides or whether that is more Proprietary whether whether each storage Provider might have different interfaces for those discoveries and things and maybe we push that outside of the standard definition Yeah, I don't know an answer to that, but yes, I agree that that has to happen somewhere It's not clear that applications need to do that. It seems more like the job of a storage controller of some sort Okay, any more questions for Kiran Britain I would like to just add one more last comment so one of the things that we wanted to do with the node disk manager is form a separate Group that meets bi-weekly to kind of make progress on that one So if anybody from this group is interested in joining that Please bring me a comment on the slides. I'll just open that up for comments and then we'll take it from them Thank you everyone for your time Thanks Kiran exciting project That's all we have on the agenda for this week. Does anyone else have anything they want to discuss while we're all here Otherwise we can just wrap up Going twice All right. Thanks everyone. We'll see you again in two weeks