 Thank you, and I want to make sure to point out that James bought a really expensive suit just for this occasion I've never seen him dressed like this before Thanks, Steve So neither not even at work No, I haven't seen him dressed like that at his office But thanks once again, I'm with the code team Which is a group of developers who work on community open source projects like Apache mazos Sponsored by Dell technologies and I guess you can go ahead and embellish that great introduction if you if you've got more to add That was great. Alex. Thanks. Yeah, I've been with mazos for a couple of years and like Alex said Spent some time working on the Kubernetes mazos framework and marathon some other mazos related projects, so So I'm G so if you're not in the previous talk, so I'm a mazos commander and PNC member things 2013 and working on containerization now working on storage and I want to make I want to point out a person who's missing in action here we gave this talk at mazos for your North America about a month ago and At that time there was a fourth presenter Chakri Who works for Diamante and we had a represent we basically wanted to represent a Couple of storage providers because a key message here isn't that is that this is industry-wide this effort to Standardize on a container storage interface isn't just one vendor or a couple vendors and in LA We we had a couple representatives, but he couldn't he couldn't make it here for this event But his slides are still here and I'm going to deliver his slides But I want to make sure that everyone's aware that he was a contributor to this deck so Here's an overview with the agenda. We're going to Cover the state of storage in container orchestrators today then move on to the benefits of Standardization and there's a couple there's benefits from a user perspective How many users of mazos or DCOS do we have out there today? I could show of hands quite so most of the audience falls into that category but from the Authors of container orchestrators not just mazos, but DC OS kubernetes that cloud foundry Docker swarm there are benefits to this then finally from the storage providers perspective there are benefits from Standardization we're going to give a quick overview of CSI Moving on to mazos integration with CSI We've got plans for releasing this late this year, but there's a roadmap that goes on with further enhancements beyond that We're making effort to get this out quickly Rather than you know take a lot of time to do the full featured version day one finally we're going to close with Message as to how you can get involved and maybe Help contribute to the decisions on what features we might want to add to that roadmap and maybe even help out building the things over the next year so the background from the User perspective is that Users need or want support for persistent storage For to run stateful apps under container orchestrators They want this abstraction because it allows portability Potentially across container orchestrators across the boundary between public clouds, you know They want you want portability from the Amazon cloud to the Google cloud and then even on to an on-prem cloud You'd also like to have something that would enable portability across storage providers So that you could write and implement applications that aren't even aware of what the underlying storage provider is And then change that over time Migrate your app from an on-prem cloud to a public cloud and not have to change it Some of these stateful apps that could benefit from this are shown here by logo sequel databases no sequel databases a whole category of things that Need to retain state somewhere that don't have Alzheimer's when you shut down the container that holds them So the the background from the perspective of Container orchestrators is that we have a number of container orchestrators shown by the logos here DCOS kubernetes warm cloud foundry Apache mazos that have evolved independently with their own Independent implementation of a storage interface now in reality, of course Apache mazos and DCOS maybe those two do have a common storage interface but the rest all in their first generation came up with their own implementation and When when they've implemented this they've typically even integrated in their own versions of storage plugins that live in the source tree of that container orchestrator, so That is kind of bad too because with many of these orchestrators They're on fairly time-consuming release cycles and the underlying storage provider that we're we're dealing with with these storage plugins Kind of is it on its own independent release cycle and if you come across a scenario where you've got a security patch a bug fix that Originates down in the provider of the underlying storage You might have to wait to get that into the source tree of the container orchestrator and that's a bad thing I mean a security patch is Often critical and you'd like to get that in as soon as possible But if you if there was just a release of Apache mazos and the next one isn't likely to occur for another five weeks You widen this exposure gap and we don't want that to happen From the background of a storage provider, and I'm kind of in the shoes of one of these being an open source group affiliated with Dell You're dealing with a current world and it's not just me But these other storage providers the amante port works Amazon Azure the public clouds like Google cloud We don't really want to implement these plugins Four times for four different container orchestrators. It would be great for us if we could do it One time and then reuse that it allows us to More efficiently use resources Maybe have these resources left over to invest in a heavier degree of testing and validation and you know Even if there is a scenario of a security patch if we have to go Apply that and test it across four or five different container orchestrators. It's that much longer for these things to get out so The summary is that a storage provider wants to write and maintain one plug-in that covers everything From the the state of the world today just to show you how bad this is out there in the world I Went out there and did some research using Google to find out how many storage plugins exist just for Amazon EBS and I found five of them You know that it's sad that in an open-source world you'd have five independent and I have to have five in implementations of plugins for AWS EBS To visually summarize this what we're trying to avoid is this that's a picture of worldwide Electric outlet standards. I came here from Los Angeles and that means when I bring my laptop I've got to get some adapter and this scenario is what that storage interface is with container orchestrators today And the downside of this is users have difficulty using devices portably If you're an appliance maker the you know the The analogy to a storage provider you have reduced economies of scale You've got to do your engineering in multiple passes and often their certification involved in an encouraged testing delays Vendors who sell these or bring them to market or distribute them have to Inventory larger inventories of different versions and that might result in you for going entry into some markets I mean maybe someday they'll be a sixth container orchestrator and I as a storage provider would just say no I'm not it's too much work. I'm not going to do that but if we get a standard in place it it it opens it up for you know evolution and Creativity from different people at this point It's a multi Multi-presenter presentation today, and I'm going to turn this over to James. Okay. Thanks Steve Yeah, so I'm going to talk about CSI a little bit the primary goal CSI is To present a neutral standard protocol for container orchestrators And their interaction with proprietary storage systems CSI presents a consistent set of behaviors and expectations for both container orchestrators and vendor implementations of CSI For vendors like Steve was saying there's less work involved to support, you know X number of CO's Which allows vendors to enable more CO's more quickly for container orchestrators? Gives access to a broader storage ecosystem Orchestrators can leverage open API's and they have looser coupling with back-end storage systems Some other goals of CSI Especially with respect to this version that we're iterating on we're trying to keep the set of API's relatively small But still enable many use cases kind of aiming for a lowest common denominator API And what this will do is lower the barrier of entry for the initial round of CSI plug-in writers Right so from a high level CSI presents a control plane interface that is largely focused on volume life cycle It is a service oriented interface as opposed to a command line interface This allows plugins to easily co-locate with other long-running processes. Think about fused Eman or Gluster FS services or NFS services Those are all long-running services that that co-locate well with with plugins Services are exposed to be a GRPC Some advantages to this There's well-understood mechanisms for proxying and load balancing GRPC calls GRPC supports streaming responses, which although we're not taking advantage of those in the first version of CSI The doors open for later versions GRPC also scales well, it's an open specification and there's great community support with respect to configuration and operation CSI allows plug-in supervisors, which means a container orchestrator or something else That's that's supervising the life cycle of the plug-in to decide how to deploy and or isolate plugins CSI does not specify security protocols an operator would protect a CSI Unix socket just like they would protect other file system objects CSI with respect to packaging there's no mandated container image format the spec suggests that Providers try to use something that's cross co compatible Again, there's minimal expectations with respect to supervision Generally speaking plug-in should terminate upon request for example like a sick term Isolation is not guaranteed, but it is very likely so for example Pretty sure Kubernetes is planning isolation. We're planning isolation with mesos But it's not guaranteed other COs may not isolate and even across COs that isolation may not be the same so CSI consists of three GRPC services These services may be composed in different ways depending on the deployment requirements That's determined by the plug-in vendor For example, a headless deployment would bundle all three of these services together To further illustrate this idea is this slide so On the left It's more of a centralized model Where you've got a controller service Running on a master node and then you've got all the CSI node services running on the agent nodes so the controller centralized the node services are distributed in the Diagram on the right is what I called headless before where you've got all three services Well in this case, I've only shown the controller in the node But identity isn't there too. So you've got all the plug-in services running on all the agents in the cluster And there's nothing specifically running In the center on the master nodes Oops, there are a couple more points here I go back Yeah, the last point I wanted to make is that it's up to an operator to configure a container orchestrator With respect to with respect to deployment And it's up to a plug-in vendor to provide the documentation regarding deployment. None of that's specified in CSI itself Great So the lifetime of volume From creation to deletion is illustrated on the right There's different volume states That that volumes go through as CSI RPCs are in vote and what's important to understand here is that it's the co That's really driving the provisioning process so a co is going to Invoke the controller publish volume, which might mean Attach some volume or some device to a node The co will then invoke a node publish volume, which probably means hey go mount this volume in some container Plugins advertise support for lifecycle operations via capability RPCs An example of some of these on the slide there create and delete is a capability and the controller publish and unpublish your capabilities Next couple of slides. I'm just gonna briefly walk through the API. This API is intended for consumption by container orchestrators It's not something that may so send users will see that said it's still useful to understand what's happening under the hood and It's also useful for you if you want to take a stab at writing a CSI plug-in So first up is the identity service. This is important for version negotiation It allows a container orchestrator to select a supported version of The API for use with future RPC indications All CSI endpoints must support this service. It is not optional the controller service This maybe runs in a central location or else it could run on the nodes themselves across the cluster The RPCs here kind of split up the the top two RPCs are the required RPCs The bottom four RPCs are optional and the optional ones are selected through the capabilities API Which is the first call so they get capabilities RPC just reports which controller RPCs are implemented by the plug-in. So that's discoverable by the orchestrator Validate volume capabilities allows an orchestrator to ask does volume X support some Y set of capabilities And that's important because a plug-in may not provide the create and delete calls Volumes may be pre-created Maybe a CEO gets a list of volumes from a plug-in and it needs to validate that hey Does this volume actually support this thing that I want to do with it? The next four RPCs like I said are optional they create and delete Calls create is what you think it is and allows a container orchestrator to tell plug-in I'd like to create a volume given some names some size and some set of capabilities And that's some point. I want to delete that volume There's a controller publish and the controller unpublished volume This used to be called attach and detach those names really didn't fit all the workflows that we envisioned It's really useful for centralized deployments where you have a central plug-in controller that's executing these RPCs It could also be useful for helpless deployments I think the example that came up in one of the discussions was I scuzzy somebody wanted to implement specific eye scuzzy commands One controller publish and unpublished were invoked The next call list volumes The intent here is to show pre-created volumes. It doesn't require create and delete but plugins are certainly free to implement any subset of these these calls and The get capacity call reports the available space on the back end Presumably a CEO would invoke this before calling create so that it knows how much space is there before it tries to create a volume that Maybe takes up too much space Lastly is the node service this service must run on the nodes upon which volumes are mounted The first call probe node allows a CEO to instruct the plug-in. Hey, go check your configuration Check for any required software or devices that are needed on that node if the plug-in fails this call That's a signal to the orchestrator that the plug-ins not ready and it should not try to do any kind of volume life Suckle management the node publish and unpublish calls you can think about these like mount and unmount volumes They're invoked on a per workload basis To mount a volume or unmount a volume to or from a container The get node ID call presents a consistent identifier for the node From the perspective of a plug-in instance and the last call get capabilities Used to just be a placeholder. There's a PR in flight right now to add a capability To kind of separate out some of the responsibility that right now lives within node publish and unpublish that new capability is publish an unpublished device And I think that's it. I'd like to turn the rest of this over to G Who's going to talk about CSI and and how it's going to be integrated into mesos? She right. Thanks James. Yeah, so I'm gonna talk about like CSI is great that we have this vendor-neutral Interface that across all the container orchestration system and then the question is how mesos is gonna leverage that interface and how mesos Are gonna build features on top of those interface So so for mesos we introduce a new concept called resource provider So think about right now inside mesos the only resource provider you have right now is the agent So agent provide resources like CPU memory disk and the master keep track of those resources and send to the framework using an offer But but this is not very flexible like there's no way currently like I mean well people can use custom resources but you do you cannot Right now you don't have a way to Customize the handling of operation on those resources for example If you want to define a custom operation you want to perform on that resources trying to convert the resources There's no way you can do that in mesos right now So as part of the storage work we try to introduce this general concept called resource provider so that We allow developers to develop on their own resource provider that providing resources to mesos and mesos master will collect those resources and apply quota reservation all these resource Primitive and then send those resources to the the allocator the framework and Resource provider can be either local or External so by local. I mean that the resource that this resource provider provides are tied to a particular agent note External means like the resource you provide don't tie to a particular note things like IP addresses EBS volume those are kind of external resources that don't tie to a particular note So we introduce both local resource provider and the external resource provider and if you think about that agent can be think of a Combination of task management plus local resource provider So why introduce RP as I mentioned like we allow customization and extension on resources and and the one the biggest reason we introduce this Concept is because we want to support external resources aka like global resources That mesos use don't support previously But I think we do see a lot of use cases that global resource definitely useful So we definitely want to introduce that concept. So and we introduce global resource through resource provider concept Okay Okay, so so we're gonna introduce a first-class storage resource provider Which is tied to on the storage work. We want to do with CSI We're gonna introduce a storage store resource provider that talked to CSI plugins And this storage resource provider will expose disk resources to mesos And it will be responsible for handling operations things like volume provisioning and volume mounting as well And the goal we want to achieve inside mesos is in the end and the storage vendor just need to give that CSI plugging Docker image or a binary to mesos and mesos will just take care of the rest and And give you resources to frameworks. So that's the ultimate goal We don't want the user or the operator to be and like I mean We don't want to you we definitely don't want the user to be aware of all these internal CSI stuff to users It's all just resources the same resource you get right now inside your framework So this is kind of the high-level architecture of how this is gonna work So as I mentioned we have you can see here So we have a master node and agent node. So we as I mentioned earlier, we will introduce external resource providers So for storage, we're gonna introduce a first class Storage external resource provider that talked to the mesos master directly because the resource it provides does not tie to any particular Agent it does not make sense To render on a particular agent, but essentially you can run this provider on any agent And then for example scheduled by marathon to run that plugin on any agent And that will talk to mesos master through HTTP API to provide those resources to master and that storage external resource provider will in the end Talk to a EBS for example like talk to a storage vendors Controller plugging as James mentioned in CSI we have this controller service and no service Controller service can be run anywhere So the external resource provider will talk to controller service to do things like provisioning and deprovisioning a given volume And then on the agent side as you can see here We have a we need to do a storage local resource provider for on local disk resources And it's it talks to a CSI plugin as well in this case I use an LVM and example I say you have a bunch of disk You want to treat that as a single pool and send resources to mesos master So and and we can build an LVM plugin for that and LVM is responsible for handling all these provisioning and deprovisioning volume requests Coming from the CSI interface And also as I mentioned as I as James mentioned earlier that you have to run some no plug-in on the node that you want to use the volume Think about the EBS case once you provision the EBS volume Eventually you want to you want to attach the EBS volunteer a given node and actually mounted the volume and that operation can only Happen on the node you want to use the volume so we need to deploy an EBS no plug-in on the node as well And that would be handled by the agent itself to to launch that no plug-in container and agent will talk to that on no plug-in container using CSI protocol as well So that it can actually like make sure that volume actually show up and mounted on a given location so the container can use those Volumes so we're gonna support both local and external storage providers and support all the CSI plugins So this is kind of real map in mesos for storage support So we're gonna support local resource provider first because it's more tied to the current model and And then we're gonna build a local storage resource provider with CSI And then we're gonna move to external resource provider integration and then build a CSI integration there So there's an epic you can track the progress there So LRP is a local resource provider for storage isn't target for next release Which is one five and the external storage External resource provider it a support is target for one six that the next release after one five All right, so I'm gonna turn over to Steve Thanks So I'm here representing a storage provider and The storage provider that my team works on the code team is something called rex ray That has been around for a while already in the mesos environment But it was implementing the mesos proprietary storage interface But we've spent the a good portion of this year Enhancing this to drive rex ray to become a CSI compatible storage plug-in and The benefit of course here is that we'll be able to use the same code to interoperate across multiple container orchestrators like mesos, but others as well Now the the key point I want to make today that is that rex ray Is a CSI Compatible provider at mesos con North America on the Sunday before it started We announced Implementation in rex ray version 0.10 for CSI support The the overview you can see here We've been delivering support for persistent volumes for mesos for around two years now, so this is nothing new in the mesos environment But our 0.10 released on September 12th brought an implementation of the CSI proposed specification The orchestrator vendors behind this are still declaring this CSI spec to be a pre-release at this point But I think as soon as the first two build them in build implementations This will be declared a release. They just wanted to retain The option to potentially change something should somebody spot a mistake I mean if you can't prove that it's interoperable until At least two instances have managed to pull it off But somebody had to go first and rather than wait for the orchestrators to do their end and have nothing to attach to and nothing to test with the code team Decided that we wanted to get out there and do that groundwork to provide the ability to Hook up with something if you're a container orchestrator and actually do your development and testing So this should be compatible with mesos kubernetes and docker swarm on the road map We're planning the 0.11 release October 16. I don't know if We're already past that I believe that happened right Clint, okay So that got out there to the the late ad there was this deck dates back to to mesos con North America, so I didn't update the deck But it is out there the feature added in the 11 release was support for Azure unmanaged disks This is an architectural overview of what rex ray looks like as a CSI storage provider as Mentioned before by James the primary interface to the container orchestrator is a g rpc interface so the Rex ray plug-in implements that it's written in the go language We have implementations for a number of different flavors of storage I'll get to that in a minute There are components involved here that operate at both a client and server level and we are working on conformance Conformance libraries so that we have a test suite that we can use to validate that these things are compliant with the specification What we've got today. This is a quick wrap up On the right side you can see the support for public cloud forms of storage, so we've got All the popular forms of storage on Amazon. We've got persistent disks in the Google cloud block storage and digital ocean and The latest ad was the support for the Microsoft Azure unmanaged disk over on the unprint on-prem side We've got support for CFRBD We've got a whole class of a classification of local storage These would be things that would be Mountable on the local cluster nodes. We've got coverage for block device NFS VFS for Hardware or software based storage in the Dell family We've got support for scale I o and I salon and on open stack. We've got a sender interface supported At this point even though I'm going to keep talking in the original presentation The second storage vendor Diamante was supposed to come on stage So pretend I'm Chakri now So from the perspective of Chakri as a Diamante storage provider they already dealt with Support for multiple container orchestrators and they were actually one of the pioneers over on the kubernetes side of this thing called a flex volume interface that was a Generic storage interface that called out to a command line or an exec form of mount-on-mount for volumes and It got a fair amount of traction over on kubernetes I don't flex really didn't have that much of an equivalent over on the mazo side at any time But it was an example of a storage provider Maybe trading a little water harder than they should have to in that they're you know doing work on multiple platforms that We're hoping will go away From the perspective of Diamante, they're now happy to be working with the CSI community to bring these Container orchestrator platforms together So finally Diamante like the fact that CSI means that there's one Storage plug-in interface that delivers it to multiple platforms Get brings the benefit of reducing development time and increasing the time left over to do testing so If you're blowing less resource on duplicated effort You can excel or you can reallocate that to doing innovation and work on collaboration. So Diamante is a backer of this as well there are others These are People who've announced that they're Participating in the CSI project. There's logos there for both storage providers and orchestrators So the roadmap right now we've got a Pre-release spec and the orchestrators are planning to support this I Can't really speak on their behalf, but I believe it's fair to say that they anticipate this being by end of year for the Kubernetes and mazos platforms If they might miss that they probably shouldn't miss it by much There are some things that didn't get in that first cut of the spec But these things are on the roadmap So, you know, there's always that trade-off of put everything But the kitchen sink and release one and then it's going to take longer and these roadmap items that will get in to release and You know one plus and our support for snapshot volume resizing quotas Windows the wind container support for the windows OS and User ID and credential pass through to the storage provider There's one is one thing that's deemed out of source simply because the different container orchestrator Providers mazos Kubernetes Really had different architectural ideas or wanted Wanted to use this as distinguishing features and that was storage class something that's often referred to as profiles in the virtualization world and those those are not Destined to become standardized through the container storage interface. It's kind of a higher level of Managing and mapping classes of storage to containers that are controlled by a scheduler I'm gonna close here with a Call to action to the community for people to get involved right now a number of orchestrator Developers are heavily involved with this project as well as a number of storage providers But we could use some more increased involvement by the user community. How would you get involved? Well? We have a github repository that retains the CSI specification and You can go look at it and you could create issues against that specification If you want to get a background we have recurring meetings that are done under zoom There hit the link here the link is there and this deck has been uploaded to the Linux foundation So I believe you can already download this deck If it didn't get there yet this deck is Largely unchanged since the original presentation in Mesa's con North America, so I know for a fact it's there already Anyway, you can go to that zoom link get the schedule from from the links here and Just join that call You've missed some of the calls, but the a running log of the notes from those calls is kept at the the link you see there and These zoom meetings are recorded So you if you'd prefer to digest that in the actual audio rather than reading the notes that's available as well There's a Google mailing list that is actually quite active in this community from both a perspective of orchestrators and storage providers I guess I I'd like you to get on board because this is one of the real benefits of open source as opposed to closed source commercial products where you get that step where you as a customer would go through a process of Going to a vendor Maybe getting their product manager to put your request on the feature list Here you can get right up in front of the developer's faces and there's a unique opportunity now because we're still Building this for the first time that is perhaps easy if you spot an omission or you have a Real need for a unique request that maybe we haven't thought of now is the time to get in there The other reason for getting in there now is if you're a large organization that maybe anticipates pushing the envelope for performance or scale on this By getting involved now where we're having active discussions on the design You'll inherently get that deep dive view So you have a real understanding of the fundamentals of why things were built the way They were how things are glued together where things live and sometimes after a project is a couple years old Is it isn't all that easy to pick up on those kinds of things? So even if you want to get involved with this community as a lurker just to listen Listen in on what's going on in these design decisions in my mind That's a great opportunity that you don't always get so I really encourage you to get involved here So at that point, that's the wrap-up for the presentation. I think we've got a Few minutes left to handle questions if anybody's got any Yeah, we have 15 minutes for questions any questions Well people thinking about questions. Could you please tell me the story of your logo who designed it? Where does it come from? Oh the CSI logo is CSI logo. We actually Commissioned a number of implementations of a logo and put it out for a vote and It was hand the microphone over to Clint there in the first row because he's the one who organized that effort So the the logo's background is that we Open it up to the CSI community to Participate and to vote on the different logos So sod and all the co's and you know the team up here on the stage We're able to participate and vote on what they liked So there were about 80 different logos So I don't know there's a specific story to tell other than everybody liked this one and so it's the most popular logo Yeah, so there it is oops. It's the oh I Went back too far. I want to bring it up. So we know what we're talking about here. I Mean it. It's the orange. Oh one more So it's the orange thing. Yeah, so the The description that we asked for from the artist was to present something that was connecting two things together So hence you have the dot in the middle which represents the interface and then you've got on the top Maybe the co and on the bottom of storage platform. So I didn't vote for this I actually called called this I kind of dissed it calling it You know CSI really only works at the control plane of storage So it does the mount on mount the attachive devices to nodes So it's a control plane. It isn't involved in the data plane So there's no latency added by using this and I called this logo snakes on a control plane Or better the snake to me it looks like the snake that ate all your data And if you like the logo in the code booth, we've got some embroidered patches with it So come by the booth after this breaks up and get yourself one. We've got stickers with it, too Okay, any real questions We got one right there have question to the mesosphere guys Is there thinking about how to design that storage class concept the profile concept where? May be linked to the quotas on both capacity and IOPS How would the implementation look like? Yeah, so There's protobufs that are have even either landed or will land in mesos To support like a profile name that can be associated with a volume There's some ongoing work in DC OS to add Support for dead like an underlying implementation for profiles So it is something we're thinking about you know in terms of you know, what is a profile? How do you how do you associate parameters of the profile things like that? So maybe I'm thinking that there might be some new adopters of mesos or DC OS in the audience So it might be worthwhile to explain the value proposition of why you want to have these classes or profiles If I'd be happy to answer it because it's a storage for I don't wear of it But you can have in it sure so that the whole idea of these classes is that you as an application writer a dev Really don't want to link your storage needs coming out of the application to specific Identities of a volume or a type of storage because that stuff changes over time and it changes as you move from an on-prem Situation to a public cloud or from one public cloud to a different one. They have different types of storage there so the last thing you want to do is Say something in your app like I want My relational database my post-grace dress database to link up to an SSD provided by Scale I owe because if you were to ever move that to AWS it doesn't exist there What you'd prefer is a scenario where you define these Classes with names that are managed by an administrator So a class might be the fast class meaning high performance, but expensive storage and a Slow-class or a cheap class, which is stuff where you'd prefer to save money and if you take the example of a Relational database they might have things like index tables that would greatly benefit from Utilizing that fast class of storage, but they could have other things like Log files that you'd really like to save the money because they don't really have performance demands And if you abstract those out with these broad general descriptions of fast and slow or fast and cheap Those tend to be consistent across clouds So if I defined a fast class in Amazon that might map to high performance Storage in Amazon that cost me more and the slow to slower and that same thing could it be mapped to other categories in Google's cloud or to your on-prem Storage it also can be consistent across time So if you put yourself in a time machine went back five years with that relational database Fast might have been 15,000 rpm rotating drives and slow would have been the slower drives, but it's all rotating Today fast would clearly be SSDs and slow might be still rotating But those fast and slow labels would still apply you don't you could Change in one place for your whole data center the mapping of fast to some class and slow to a different one And at the app level no you don't have to touch anything So it saves you operational expense you the goal here is that your app definition and configuration lives for a long time without administrative intervention and That storage class thing is a great feature to have because it can lower your operational cost Yeah, so so to translate to mesos API will add an interaction in the resource object called profile name And that will be exposed to the framework So from to scheduling decisions based on the name of the profile rather than like around parameters of a disc like SSD Or spino disc or IOPS so they base they do the decision to pattern matching on those profile names So it's kind of a level of interaction So that name can be mapped to different things on different in different times But I mean operator will be responsible for config those mappings But frame of make decision both based on those profile name user make their selection like say I want to use a Profile blue or profile fast disk user make those decision based on those names. It's just like a level of interaction Okay, I think we kill this question There's another one More questions here is a hand. How does open SDS relate to CSI? Sorry, what's the question? I think it was open SDS open the software defined store storage. Uh-huh. It was a yesterday a lot of Noise about it. So And how does it relate to CSI? How does that related to as CSI open SDS? Okay, so the question is like how does open SDS there's I'm not sure everyone knows about open SDS There's a kind of platform from Huawei, right that wants to to do the pretty similar things trying to Provide the platform a lot of different vendors to plug it into that platform and the open SDS also has this northbound API to Talk to the container orchestrators and the question is what's the relationship between CSI and open SDS? So we start the CS CSI project around the same time as open SDS if I remember correctly That was last November or last December at that time I Think we just I think we pretty much start the project Simultaneously, I think one of the things that we want to achieve in CSI particularly is trying to be storage vendor neutral So that we all the container orchestration system get together and trying to define a standard And then we don't want to actually if you see the doc right now There's a community doc which we explicitly say that we want the maintainers to be off from all container orchestration systems so that we can ensure the standard is storage vendor neutral I think that's a key things to drive the standard to be successful. And I think Open SDS is from Huawei who is a storage vendor self So that's kind of the one thing that I see might be problematic for a standard but I think I don't see really a huge difference just like two APIs and CSI just the one that we agree on Among all these container orchestration system and then we start about the same time We're not aware of SDS at a time when we start a project Yes, we do have some chat with OBS DS people actually, I think open SDS is more similar to live storage. Yeah, or perhaps Rick's right I'm not a spokesman for what their roadmap is Perhaps they intend to implement a CSI interface. Yeah, they have open SDS, but I can't speak for that Yeah, I think I chat with open SDS folks a few times I think they're like two different ways they can integrate with CSI one way is you can build a CSI plug-in In their northbound API. So that's one way. So so basically treat open SDS as a meta CSI plug-in that They have their own interface that other storage vendor plugging into their interface and When container orchestration system wants to talk to open SDS is through the CSI interface So that's one way. The other way is I think I talked to open SDS guys is because they also want to handle the provisioning like like sending resources to mesos to like kind of Render storage itself on two mace around on the container orchestration system So another way to do that is they build a mesos framework that providing resource like the resource provider API that I mentioned earlier that they build a resource provider That talk to mesos to provide resources to mesos and they handle all these morning provisioning Requests and then send those requests to the corresponding backhand drivers in open SDS. So that's one way So we chat about these two possible ways more questions We've got one right here Okay two questions the first one is is ESI you relate to to to replace persistent volumes in mesos Okay, so the question is well CSI replace persistent volume inside mesos the answer is no they're gonna be co-exist These are two orthogonal concept Just think about persistent volume is just another layer because not all the volumes are persistent Some volume can be ephemeral. Well, but what I mean by ephemeral here is like the life cycle of the volume does not Have to be persistent when container terminates that long and get GC right away the reason for that For example, I just want an SSD disk for a scratch space. I don't care the persistence I just want the speed things like this. So these two concept are not not exactly the same. They're orthogonal So you can have a persistent volume on a CSI CSI disk we all you can have a persistent volume on the traditional local disk because we need to be backwards compatible But eventually like everyone should move to CSI So it's backed by a CSI plug-in that CSI plug-in can be a dumb CSI plug-in Just like having directly under having sub directories under directory to pretend to be a volume things like this So that would be exactly the same behavior as it is right now for the local persistent volume that we have right now So I think it's fair to say what you just said is for a while they'll both be available in parallel Yeah, I mean these are first of all these are two orthogonal concept because CSI volume It's just really the volume and you can be persistent volume and there can be ephemeral volume Which is on top of those primitives these are told like orthogonal concept And then I'll say if you're using the existing persistent volume support in mesos with Rex ray I can tell you that those volumes will be portable in other words if you Commission a postgres server right now today in the current release and go through Rex ray to get to it You won't have to blow that away or back it up and restore it You will be able to simply remount that volume using the Rex ray CSI plug-in. So There's there's not an issue of rip and replace It's simply going to be on mount and remount. It should take seconds really to flip over If you're a framework writer then then the API you're going to create persistent volume will be exact the same for local local Like that the traditional like legacy volume and CSI volume so the front frameworks perspective It will be exactly the same The only difference is the profile that we just mentioned that it provide an interaction allowing to do make decisions based on Some name that maps to a bunch of disc parameters So that that would be the only difference there should be no difference from frameworks perspective All right, so the second question is related to the to the to the frameworks. So it's How is the future of this year's commons relate to CSI standard? Sorry, can you repeat the question the future of the fios commons relate to the CSI standard? it should be We include Features to support. Oh, yeah. So the question is whether DCS common is going to support CSI the answer is yes It's on the road map. It's definitely going to be happening after DCS 111 It's on the road map Okay, we have two more minutes You can get more questions Okay, you mentioned LVM as an implementation for local volumes Can you tell us a bit about that? Is there any plans to do something in messes around that or on the LVM? Yeah, okay, so the question if you want to take that question or you want me to it So it's it's LVM and what the support plans are is that is there any plans who? It's a concrete implementation around LVM So it's an implementation of CSI Because you mentioned that in the talk. Yeah, so we are building an implementation internally a mesosphere To use LVM There are plans to build a reference implementation that will be open source It is on the roadmap. I Cannot promise a delivery date, but it is something that we do want to open source Yeah, I mean building an LVM plug-in should be pretty straightforward I don't think it's too complicated. It's like you have a bunch of volume Like physical volume and group into a volume group and then whenever you receive a request to provision of volume Just create a volume group using LVM command or like any library like LibLVM. It should be pretty straightforward I don't think any There's any secret source there I mean in your Your feeling is that the user will have to prepare this volume or the Implementation is supposed to take care about everything Like yeah, so user I mean if you if you're if you're talking about user like marathon user They should not be aware of whether it's from LVM or not They are just pick the volume from profile. I mean that case I Usure means operator. Oh operator. Do they need to be aware of LVM or not? Is that a question? Yeah At which point the Implementation is supposed to prepare the LVM config. That was my point. Oh, okay. Yeah, so maybe not defined So sorry about that. Okay, so I think if I interpret your question correctly Are you saying that the LVM implementation should be open source and should be contribute back to the So so everyone can use that. Is that the intention of the question? Not that much. It was you're having your You're feeling about what would be the current implementation in LVM. It's maybe not defined So I would be okay with this answer But is it defined or not like do you have any plan concrete plan for that or not of the LVM implementation? So maybe I can help here. So maybe the question is To what extent operator should be involved to take your implementation the LVM implementation and deploy it in the cluster Is it the question? So what work what expertise is expected from the operator? Oh, yeah I think as a man. Okay. Sorry. I am missing interpret your question So I think our goal as I mentioned in one of the slides the goal is The operator just need to hand over to me So a single container image name or a single binary for the CSI plug in which is the LVM plug in and misses We'll just take care of the rest to deploy those plug ins if it's not launched yet and talk to those plug in to To communicate through CSI interface to provision volume So the only thing the operator needs to do is provide that container image name and most likely is a version You can also upgrade a version if you want to redeploy that plug in using a new version So that's the only thing that the operator needs to do our whole goal is trying to simplify the operator as well to not to worry about Internal details just the container will be running on the mesos agent for those CSI plug ins the mesos agent will talk to those CSI plug ins For LVM at least for the LVM case for external case It's a little tricky because you have to run the controller plug in somewhere and you probably need to involve something like marathon Or Aurora to schedule those containers somewhere, but at the very least you don't need to worry about building I mean Like you don't need to worry about the implementation detail of the LVM plug in You just need to specify some container image name in some app config. That's it I think it's also interesting to mention from a framework perspective. There's some new resource offer operations That are going to be available within mesos and those resource offer operations will map Directly semi-indirectly to CSI Plug-in operations so they're frightened. It's actually possible to write a framework That can interact with plugins kind of through mesos in the offer operation API. Yeah, so I think yeah That's a good point So one thing you can do with LVM I like which we cannot do before is you can just model a giant a volume group as a giant pool of disk space and Then the framework can make a decision to carve a volume from that volume group So maybe some framework say hey, I'm opinion. I want to control the size of each volume I don't want the operator to pre-pick a size for each volume So right with the CSI where you can do that basically you can say hey These are a storage space. It's like a thousand gig like a terabyte and I want I only want a 10 gig volume Just create that 10 gig volume for me And then once you issue that operation to mesos you will receive a feedback later saying hey You see the offer later with a 10 gig disk in your offer stream Then you can start to use in this volume so that will be also like impossible with the CSI work the storage work Yeah, so it looks like for a parade is very easy You take the binary put into your cluster and wait for a page in the middle of the night when the snake is all your data Any other questions? There is one more week can probably take the last one Yeah, I mean if you have a more question, I'm here and these guys are here, too So we can yeah, we're at the code booth just across the hall from the door here I think that there's a town hall tonight addressing story. Yeah, so there's a there's a town hall tonight We will be there. So they're like four town hall topics warm topics approaching me. So it's so I'll be there You guys probably gonna be I'll be there too So and if you have more questions you can join the town hall and then we can chat over the town hall The whole purpose of town hall is trying to give you guys Opportunity I mean they're like multiple purpose one purpose I think it's like people get together like we always chat over slack and we don't know each other This is a good opportunity for people to know each other in person rather than over slack. So it's a good opportunity If you want to join Yeah, so can I ask last question? Yes. Yeah, first of all, thanks a lot to very nice initiative here And on one of the slides it was depicted that methods agent talks over HTTP and gRPC so our HTTP through some mediator like storage sleep driver to storage services and Directly through jrpc in another in another case So the question is what is behind this architecture and why we need the HTTP when we already have jrpc at place What sorry the question is what why do we still need HTTP API when we already have jrpc? Yeah, so that's a good question So so jrpc has two side the client side and the server side And I think we all we build the client side inside mesos already the server side is slightly tricky because the way So yeah, I mean, I think mesos has this goal trying to move to jrpc as a The API so once we get there we just switch that I think it's very easy to switch to the jrpc once we get there So the goal I mean for API maybe not when we want maybe v2 is gonna be jrpc But I cannot say that for the whole project there I think if you if you're interested go to the town hall tonight, I think Mino will be there He's the one that kind of maintain the API side and we definitely have some conversation to switch to jrpc for our API You will it will massively simplify the client of mesos like the scheduler, right? You can use Any language that jrpc support to write a scheduler, which is a good wing for us I will vote for jrpc for all the apis and once we get there it will be jrpc. Perfect. Thanks. Yeah Okay, thanks. Once again, we'll be in our booths and at the town hall So if we missed your question or you wanted to keep it private find us there All right, thanks