 The music turned off. Is that any time to talk? All right. So welcome to storage vendors are killing Cinder. Don't tell anybody what's kind of a misnomer. So who am I and why should you listen to me? I can answer the first question. I'm Corey Stone. I work at Rackspace and have worked on this cloud block storage thing since it's been a project. And I'm a self-admitted rabble rouser in the Cinder community. I like to hang out in Pound's open stack Cinder. I'm guitarzan. And I like to answer questions. But even more, I really like to play devil's advocate when people start introducing interesting conversations. And I'm always on the side of simplicity, even at the cost of maybe something being less than ideal. And I found out the titles are super important. I put my name on three summit proposals, hoping that one of them would get in. Hoping that more than one, but it turns out. The first one was going to give details on the back story and the current story of Lunar, which is Rackspace's own homegrown LVM back in for Cinder. People have been asking that. So I figured, you know, everybody likes that. And they've been asking about it for a long time, not accepted. Also, a co-worker of mine submitted a talk about how we deploy our environment. Salt stack's all hot and stuff like that. And our transition from working straight out of the Ubuntu archive to our own deployments. But I found out that if you propose an antagonistic sounding topic and you put a BuzzFeed title on it, you are in for sure. Here's a few examples if you need help with your own sessions next summit. And they do say, like, vendor X and vendor Y. If you've been around Cinder long enough, you may know who the inspirations were for a couple of the things that will show up. So it's interesting, but I've heard half of my talk being given already by other people at the summit. So apparently this is a big deal. People are all about interoperability. And these questions I find are very important from Cinder, Cinder's perspective. So I'm going to ask those questions. They may not have good answers. And then just some crazy tangential stuff at the end. This is the example of a terrible PowerPoint slide, because it's just a wall of text. It turns out that this wall of text is actually Cinder's mission statement from the Wiki. If we want to digest that a little bit, we say that we access block storage resources. I put that in quotes, because in our mission statement, we don't define what that is. And we use the term that everybody likes, software defined storage. Nobody agrees with what that is either. So this is a marketing speak. Somebody came up with a pretty good, put pretty good words together for our mission statement. Bad slide number two. This is the open stack mission statement from the Wiki. I can leave it up there if you want to read it, but this one's pretty easy. And this is an amazing mission statement, actually. OpenStack really wants to be the ubiquitous cloud. It wants to be everything for everyone. But it also wants to be simple to operate and implement. And it wants to be scalable, regardless of if you're a public cloud down to just a little company that wants to build a lab. I'm immediately reminded of the project management saying, you can have something fast, good, or cheap, pick any two. So I find that mission very inspiring, but it seems very difficult. And last, this is some text off of Def Core's Wiki. It's actual mission statement was very recursive, so I didn't think that was very useful, but it's very simple. Def Core's entire goal is interoperability in these three areas. So we have OpenStack and Cinder wanting to be everything for everyone, but we also have Def Core that wants to battle that a little bit and have everyone conform to some set of standards. And as a Cinder community, because we have 50 drivers or whatever the huge number is now, I find this directly important to us and we need to have a really clear perspective for our own reasons. So the first question is pretty obvious. How much interoperability makes sense for us? It's a loaded question. Everyone has their own opinion and it usually involves whatever I want, that means interoperability. We've had several discussions in OpenStack, not just in Cinder, but where someone will come up with some case where one cloud or one install works slightly different than the other, and the statement is, well, they need to fix their cloud. Like there's some objective source of truth that says, this is what it should be. And thinking like that, I don't find helpful or useful in any conversation. Well, let's see, but so we have, then we have this little battle between Def Core versus vendors versus OpenStack. Vendors obviously want Cinder to do whatever they, their stuff can do. OpenStack wants to be everything and still be uniform. So I think for Cinder, it's pretty obvious that the answer isn't gonna be 100% or 0%. Everything, there will always be a difference and I think we need to just embrace that attitude. Your cloud is not gonna be exactly the same as my cloud. My storage backend will absolutely have different characteristics in yours. You will have some features that I don't have. So we need to decide what does it mean to be interoperable. So the next question is pretty simple, but the answer is not. What should Cinder do? Well, at its core, it's just another OpenStack API. So it does everything that everything else, every other OpenStack API does. It presents you this set of objects and it has a DB schema to control that and all that crazy RPC stuff. But the question really is, which parts of the interface between Cinder and those back-ins, like what should that be and what parts of that interface should we care about? This is hilarious. John had almost this exact same idea earlier. One idea is we can have a very minimal set of things that are part of the interoperability discussion. I don't wanna put, I could make up a statistic on the spot, but I would just say most of our users, at least in the public cloud, they just want a volume. They just want storage. They would be almost fine with, hey, Cinder, create me a volume and tell me how I can attach that volume to my instance. And really, all of those guys would just go away. I'm done, that's all I need. We have a few more really common operations and some just default implementations in the volume manager that I don't think there's gonna be a whole lot of disagreement about. And then on the other side is there's the attitude of, well, I can fix this problem in software so I'm gonna do it. We are devs and we like to solve these edge case problems that we imagine that never actually happen. So we can take the other approach and take the, you have to do absolutely everything that we can support or we'll kick you out of the tree. I love that argument, it's so useful. So the problem with that idea, one of the problems with that idea is as we start to implement more and more advanced features, our reference implementation is completely left out in the cold. We have LVM and what that means in most cases, it's just local LVM on a small Linux box. There are people that I assume are doing LVM on big back ends where they have lots of space but LVM really falls apart on small storage nodes. From Rackspace's perspective, I would say that the LVM driver, you can't even really use it. LVM has snapshots and clones but in any real environment, those boxes are gonna get full and just blow up. So that's kind of, that would be bad for us, we wouldn't really like that. In fact, the main difference between our driver and just the basic driver is we just don't let that happen. We're not gonna have snapshots blow up our storage nodes. That would be, I would get lots more phone calls. But we've added these new features like replication and consistency groups that are completely foreign to that implementation. The next slide is the best slide ever. People know what this is? All right. So the minute that I proposed this idea, I knew that I would have a slide with the center compatibility matrix on it. And it took me like three or four browsers to actually get one that would let me zoom it out far enough to fit it on one slide. So there it is. It does show that there is a wide swath of features that are very widely implemented and a newer swath of features that are very, very sparsely implemented. The problem with this discussion is all these vendors can do more stuff than sender exposes. The whole point of vendors is to sell themselves. This is my value add. This is the thing that I do better than someone else. That's why you need to buy my product. So for them, they obviously want, they want to have that thing in sender. The problem is, even though those features will be comparable, like so my vendor X version of this feature, I'll still say why it's better than vendor Y's version of this feature and here's the differences. Whenever someone proposes one of these new features, we just get lost in the weeds. We immediately go down to details and terminology and everybody disagrees about what those things mean and the discussion is just usually die out. So how do we reconcile that idea of the ubiquitous cloud? I want to do everything for everyone but still conform to some level of interoperability. I wonder if the fact that we do get into all of these ridiculous arguments every time that a new feature gets proposed. I wonder if that in itself is a good excuse to say that those features probably shouldn't be first class citizens in sender. I wrote my outline long before I wrote the slides out and planned out what I was gonna say. So I had this question, what shouldn't sender do? And that's actually a really hard question to have any sort of concrete answer. Again, the vendors would absolutely say, well, why wouldn't you implement shiny feature X? I don't have any good arguments for them. I mean, that's their whole point of existence, right? So the question is, is there a way for us to make everybody happy? Probably not but implement some of these things without making sender just horrible and huge and impossible to understand and catch on to. Switching gears a little bit. We like to talk about configuration all the time and I will admit that the docs around all of the configuration options aren't amazing. This is compounded by the fact that OpenStack loves to deprecate things. OpenStack loves to rename configuration options and move them into their own sections and do all the stuff at will which makes it really difficult to keep up with that stuff. If it happens to be a topic that you're passionate about, I know the docs team would absolutely love any help trying to keep this stuff straight. So I have a trick question, obviously. People complain about the number of config options. Anybody know what my answer's gonna be? Damn, Duncan, thank you. Not enough config options. I am strongly on the side of operator choice. Maybe that's not obvious but for a long time we made a focused effort to ask the center community to have independent snapshots, for example. We wanted to be able to say if your back end can let you have a snapshot without having a volume, let go for it. So that was a fight where we were asking for a config option that would let us do what we want. We lost that fight, whatever. And then there's also this discoverability dream. The idea that I would go out and purchase some $100,000 pieces of hardware and plug it into my cloud and just flip a switch on and expect it to work the way I want it to work out of the box seems a little misguided. I think it should probably be a big deal when I add an entirely new storage back into my cloud and it's in my best interest to know how that thing works and to know how to twist any knobs that I'm concerned about twisting. And I think the vendors should have that same mindset. You want your customers to know how your product works and help them to do the things that they want to do with that. And leading into my next topic, it's not really that difficult to set up a volume type in Cinder. My favorite topic by far, volume types. What's a volume type? In Cinder it's just a label, nothing more. All a volume type is is, hey customers, here's a list of things that you can try to create. We keep trying to apply more semantic meaning to these things and I'm not convinced that it's actually making our problems any better. As I said before, most of our customers just want a volume. They just want space for their little cloudy app to run and that's all they care about. In the second case, you have customers that absolutely do care. They want feature X, they want QOS specific, they want these characteristics, they want encryption because they think that your DCOps texts are gonna rip the drives out of the boxes and run. All of these features can be solved with just volume types. We don't actually have to do anything for them. Things started getting really complicated when we decided that in some universe we were gonna have a bunch of different back-ins in our cloud and we were gonna try to expose these comparable but very different back-in features and we were gonna expose those through the same volume type. Maybe someone's doing that, but if so I think you're kind of crazy. Those customers that actually want those very specific features, they're probably not gonna be convinced by you saying what we have is good enough. They're gonna say what is the thing and trying to map those things to multiple back-ins just seems like you're asking yourself for a lot of trouble. And about volume types, I'm of the opinion that Cinder should care a heck of a lot less about those things. Quick story time, we were running Cloudlock Storage on Grizzly for a long time. Like I said, it's basically just a simple LVM back-in so there's not a whole lot of features that we needed. However, Grizzly had a whole bunch of terrible performance problems once you got over a lot of volumes. So thank you to whatever Devs helped us out by fixing some of the API N plus ones and the DB N plus ones because upgrading from G to H to I made a huge difference. However, when we upgraded from G to H, we found out that you could no longer even attempt to create a volume from a snapshot that came from a different type. It just, the API is like, no, go away. So before upgrade, everything's all happy, has all passed, after upgrade, the world blew up because the API is just like, no, and I'm like, why? So we did the API hijacking to work around this but then we upgraded from H to I. Before H to I, clone from one type to another type. After H to I, bam, the world blows up. Whatever, I don't know why we want to care about things that are not necessarily, I don't know, these are features, these are API validations that just came out of nowhere. So the thing about volume types is, as I said before, all of these things can be handled simply by keys and values on your volume type. Encryption is already an option in a bunch of drivers. I think most of those are actually just back end turned encryption on for everything but you could actually do that on a per volume basis if your back end would let you do that. QOS is one of the huge rat holes that we get into every six months or so. We had a great discussion a couple years ago when capabilities first came up, where we were arguing about what an IOP was. We were arguing about throughput metrics, what they were. We were arguing about what is a full read. It was absolutely bizarre. John's been putting QOS on volume types forever and there didn't need to be any code and centered it, do any of this stuff. Replication is another case where I think you could simply set up your back end to be replicated. Your customer could buy that replicated volume type and they'd know that if a meteor hit your building, their data would be safe and that's really what's most important. Any of these volume characteristics can just be configured. And lastly, there's this other push which has kind of slowed down but why would we want to expose these maintenance operations on your back end and sender? Is it that hard to use your product that you don't want your customer to ever touch your storage device? It seems like, again, you as a vendor want to help your customer out and help them learn. Help them know how your product works. I do have some sympathy for this replication failover thing that seems pretty common but the problem that I see with that is simply that if something really bad happens, your customer's guest VM already finds, like it already crashes because it's not magic, right? The guest, your customers already know you have to wake somebody up in the middle of the night anyway and they're the one that has to perform the action and sender. It's one of those features that sounds nice but when you get down to what it actually means to implement, I don't know what the win is. The guests already crashed because they're read-only. It's just, I'm not sure. It seems like we should, we as a center community should want to push back just a little and help the vendors understand that it's actually good to have knowledgeable customers and it's also good to write products that your customers can use. So this is where things get crazy or I thought they were gonna get crazy until I heard a comparable comparable suggestion about this whole experimental API thing. So my immediate, my version of this before the summit was what if instead of starting at the top and doing the whole design by committee thing and trying to get everybody to agree on what things mean, what if we did the opposite and gave a playground for vendors to be able to experiment, let them try things out, see what sticks, let vendors learn from each other's experiments. I wonder if we could get some better, maybe just a better opinion if people see what actually has been implemented, let code talk a little bit and just see what happens if we let people play and we just have less fear of people doing things that we may not understand or agree with. I ripped this quote out of OpenSec Cinder a couple months ago. I don't remember who it was, so I can't attribute it. Is this a person here? Anyway, I just wonder what would happen if this wasn't the case, right? What if people felt like they actually could make changes and do cool stuff on any reasonable timeframe? And then there's some last thoughts about vendor stuff. Code reviews are, code reviews suck. Nobody likes code reviews. I think the pain is probably felt by the vendors more than anyone else. We always have this argument about when does your driver have to be in in order to be considered for that release? How do I get someone to actually review my driver? I mean, these questions are not new and they are clearly a pain point because they keep coming up. And I do recommend reading that spec, make plugins, external webs. I'm not convinced it's a good idea. I probably don't think it's a good idea at all, but I do recommend reading it. CI, CI people gripe about it, but really it's the only way that any of us know that any driver code actually executes. There are some cinder devs that are really good at reading code. I'm not one of those. But none of us can actually execute any of that code. So just mad props to all the in for people and all the focus on the CI stuff. So in closing, OpenStack as we keep hearing is trying to become this big enterprise thing. I still don't know what that means and I'm not convinced that I like it. But as it becomes more and more and it grows up more and more, I think that us as a center community need to have a very clear picture. I think we need to have it out in the open and we need to know what it means to be interoperable for us. I think we should be able to help make that, drive that discussion for our part in the deaf core stuff. We could just continue to be more designed by committee and just say no all the time, but I really hope that we don't and that we make that interoperability set as small as we can. As small as is reasonable and let people experiment and try to do awesome things with the rest of their stuff. Thank you. And I definitely, yeah, definitely want feedback and yeah, he's got one right here. So have you tried buying Monty Taylor three whiskies and they're telling me you want clouds to have different API behaviors like can move between two types, you're not with the clone. Because I guarantee that we're in explosion. Explosion of vehement disagreement as Infra already having a nightmare with all the tiny differences in APIs. They want the lockdown and fixed, not more of them. Like as Infra one of the biggest cross cloud users, cross cloud, cloud design, co-design now over the moment and they're already with the limited differences we've got having an absolute nightmare. They're saying if difference is spiral they just won't be able to write cross cloud apps. Now there's lots of, lots of customers who want to go cross cloud and we're gonna tell them basically no. Wait, wait, how did you make that leap? Just saying no. Well, because they won't be able to do it. The people who are trying to do it are saying if more difference is creeping you won't be able to write the code. It'll be too difficult to keep it up to date and et cetera. That's only if they want to use that feature that only works on the backend. But how do you tell what features work on what backend and what don't? I mean, the idea of locking down things like cross type clone in Cinder was that it didn't work on most backends. 90% of the time it failed, 90% of configurations it failed totally. Well, I'm not gonna argue with that. I mean, I'm just saying the things that I think we should care about is a smaller set than everything, right? And if volume type cloning between types if people really think that that's number one, a huge deal and really common if it's a really common thing, then why not make it work? If it's an uncommon thing, why would you care if it's not a standard or if it's not enforced? And like I said, there's no, I claim that there are no objective truths here but I will fight for, I will always be on the side of vendor choice or operator choice in this discussion. So the answer from Infra and Monty had been jumping up and down quite literally saying this in a couple of talks already in the last two days is the little things do matter because if somebody writes for them, all of a sudden that code doesn't work in all the other places. I will absolutely let people have their opinions on that. I mean, I'm not gonna say that they're right or wrong. I'm going to say cloning between types is reasonable and if it doesn't work, what's the pain of it not working? It means you have to work around that or it's a shortcut, right? You can either have the shortcut or not have the shortcut. And if you don't want to have that option, that's fine. Again, I'm not saying it's true. It's like, I'm not saying this is absolute truth. I'm saying I tend to be on the side of operator choice and configurability. People like Infra are not on that side. They're very much loudly. Correct. Monty is actually where I got that quote from go fix your cloud. I don't want to answer for you. I want to say that I think you did a great job and I agreed with a lot of the statements that you made. You know, the cloud compatibility thing. One thing that I think is often forgotten in that is the whole reason that we use types and that we used to recommend the use of types to do that sort of thing is because that's left to the operator and the admin to actually configure that properly. And it means that you don't have the compatibility problems that Duncan is saying and stuff like that. In my opinion, if you have multiple types that aren't compatible and that doesn't work, then you screwed up. You screwed up your cloud and that's your problem. That is not an open stack problem or a cloud problem. And it certainly would never show up in Infra anyway because Infra doesn't even do anything with types. It only creates arbitrary types that don't do anything. So it doesn't really matter. But anyway, I thought you did a great job and I think you're right. Thanks, John. You heard it here, guys. He was on camera, on microphone. Hello, oh, sorry. Okay, thank you for your presentation. I have two questions. Sure. Do you know what happened with shared volumes on Cinder? I think there was a blueprint. The multi-attached stuff? Yes. Talks to you, oh, where is he? Oh, he's right there. So the short answer is it's really hard to make that drastic of a change in Nova and it's taken a lot longer than we expected. Is it planned to work in any time? Yeah, I mean. Yes, I'll be back in the morning tomorrow. Okay. It has to be discussed with the Nova. Okay, and one last question. Sure. Okay, do we have a do nothing driver because I see a lot of customers saying that, well, Cinder, it's not gonna work on our environment. We're just gonna do it outside of Cinder, just attach it with our regular, I don't know, maybe web GUIs. And if we do that, now OpenStackDB doesn't know about this drive and when it tries to delete, it says, oh, I failed to delete because there's something attached that I don't know. Oh, I see. So you're still gonna use Nova Attach and Detach? And but do nothing. Just. Gotcha. Yes. Yeah, such a thing doesn't exist. That's actually interesting. I've never thought about. I've never thought about doing everything, like doing everything outside of Cinder except the Attach and Detach. I mean, I see where it's coming from, but I don't think anyone's too interested in implementing that. Oh, I want that. Hey. But if it's a do nothing driver, it will be extremely easy to write. Yes. Yeah, that would actually be extremely easy to. But seriously, I do see a lot of customers saying like, for example, Hyper-V, it uses iSCSI for LVN, and a lot of customers say, no, iSCSI, I want SAM fabric. Gotcha. And then we have to say, well, sorry, then we have to do it outside of Cinder. So no one's implemented the fabric stuff in the Hyper-V? Yes, so now we have a problem saying that, okay, so let's do it outside of Cinder, and now when we say NovaDilly, oh, it's gonna. Can't detach because it crashes. Yes, I got you. So I really want that. Well, so there's two answers, right? So there's, you could write a very simple do nothing driver, or you could try to talk to the Hyper-V folks to make the other fabrics work. As long as you write CI for it. What? Yeah, CSI. True. Okay, so. Oh, there you go. What? He promised it. Oh, okay. Thank you. Oh, thank you very much. Sure. So the one thing I totally, not to agree with you on is we need a way for experimental APIs and playing and trying things out. Sure. A couple of people have proposed that you even make that really explicit and you have to put a header in your client request. I know this is experimental and it might go away. And if I'm relying on this and I come and cry at the summit, then you're allowed to laugh at me. Yeah, yeah, yeah, sure. There is somebody else saying, well, as soon as you put a feature up, somebody will build something on it. So I very much think those people are wrong. And I agree with the safety margin of putting experimental and I'm not allowed to complain in a header kind of thing. But I think we really do need a way of doing experimental stuff and that would have made things like replication. We could have iterated that in the experimental space for as long as it needs until people can agree. I think if the general case of executing random code in your driver had come up before replication or consistency groups or a lot of this stuff, it would have been really interesting to watch the ride. I don't know if there is a next one planned. Unfortunately, replication was the big one that is now going through its second thing. But yeah, I totally agree. I think if we would have solved that problem and allowed that initially, we could have saved ourselves a lot of pain. And like I said, I'm not really a storage guy. I don't know what the next one is, but hopefully we can use it. So I was excited about the title of the presentation. Yep. I had maybe something else in mind. I think a lot of the times when someone proposes a new feature, they're coming from a vendor, they speak from experience, you kind of have your own storage in mind and maybe not really focused on what the API should look like, on what the behavior should be. And you kind of make it work for your own thing because that's what you know. And then every other vendor will come and say, well, that doesn't, my storage doesn't work like that, so let's change the API. And then my storage doesn't work like that. Let's change the API. And it would be great to just do it reverse. Like let's decide what the API should be. And then if your storage doesn't work like that, we'll make your storage work like that. The problem is that's exactly how it works. That's exactly how traditional open stack stuff works. And we get things that, so the problem is that doesn't allow you to, that doesn't allow you to quickly change things, right? So I'd like to say that we're really good at that, but we're terrible at it. Like we just don't do it well. We've never been able to agree on anything, right? So to then go and say, you need to change the way your storage works is also pretty unfair. Maybe that thing is just something that isn't easily abstractable. While you can do anything in software, right? That doesn't mean it's gonna look good. And since we've never tried it, I just think it would be very interesting to see what happens if we try the other way around it. Because that has been the way that it's gone all the time. We implemented this QoS specs thing after two months, two years of talking about what it looks like. Instead of just, well personally that's an example of something that I don't think does anything, but that's a case where we argue and try to come up with a solution. And I don't know who's happy with the solution ever. So I'm just suggesting trying it the other way around. I think given the fact that we haven't had the ability to do experimental APIs, the fact that it's sometimes taken quite a while to get features implemented is not necessarily a bad thing either though. Especially since now the TC is saying that we can't ever change the APIs. We can't ever remove things from APIs. So if we had the experimental APIs back in the day, then I think we would have been able to do things a little bit more quickly and experiment and try to get things in. But to simply say that we've never been able to agree on something and never been able to get something done is not actually accurate. I mean, it's obviously an extremist statement, right? So yeah, that's true. I know. We're still committing code. Right, it's an evolving process. It's an evolving process. And with things that are as complicated as replication, a lot of us, including John and others in the room, we've spent a lot of time trying to make it work for all of us. And it's vendors obviously have a motivation to get those things in, right? But I don't think we've ever come at it from a perspective like, hey, I'm gonna really put it to solid fire because our array is so much better or whatnot. At least the developers in the community really wanna make Cinder work. Sure. Yeah, I'm not attributing anything to malice, right? But replication is a great example. There's a replication V1, right? How many drivers implemented it? One? So I just wonder if we're just not really honest when we say that we do cooperate well and come up with something that we all like because there's only a couple of good examples and unfortunately that's one of them. Sure. Totally agree. Since he doesn't have a mic, he, Duncan pointed out that an amazing example of a counter example where just recently backups from attached volume just like got in. And it was, yeah, like you said, it was actually an amazing example of where a feature could get added that was pretty much universal and with like, because we have Cinder doing all the work, the generic implementation for that just works and everybody wins. So I mean, that's a great counter example. Another pretty good example of that is the work that Pure Storage recently did with the image caching capability. I did a lot of testing of it. It didn't work for our driver, but I wasn't gonna block it. I filed a bug and I committed a fix to our driver and we're still working on that, but it definitely helps the entire community to add something like that. I think that's probably true in these cases where Cinder's kind of doing automation, workflow-y type stuff where it is absolutely not necessarily a driver thing. I think you did have some sort of image cache thing, right? But the general solution, the general purpose implementation for the image cache thing, yeah, is another good win. Yeah, that's pretty spectacular, I would say. Yeah, yeah, right, yeah. The copy and write image stuff, yeah. One more comments to example, for example, replication. You just mentioned in replication view one, only one driver implement this version, right? That is why we have a version two that simplifies the replication. I believe that more and more drivers will support replication v2. Yeah, so no, that's kind of my point though, right? So iteration works. I mean, I don't think there's any argument that iteration doesn't work. The question is, could it have been quicker or maybe I'll just say more interesting if multiple people had tried to implement replication and we could have seen which solution people preferred? Yeah, oh, I totally agree. So one thing I'd be curious what you think, especially from an operator's perspective. My belief is that one of the things that makes this difficult is the insistence that we have all of these APIs and all of these API calls. There's actually a replication v0.1 that I did back in Essex. Sure. That is in production, that is nothing but volume types. It doesn't use any API calls or anything. No, not that. And one of the things that I question is, do you focus more on putting those custom things in a type and encapsulating them as opposed to requiring that there are all these APIs added all the time until you're ready? Replication is a great example. That's why I spoke about it. I see where it comes from, but I question the need for this failover thing in Cinder. I mean, replication failover is already human intensive and customer impacting. So maybe we make it slightly easier by having a Cinder call, but how much easier really is it? I don't know the answer to that question. So one thing I can say with absolute certainty, having spent the last three weeks and meetings with them, is that our customers absolutely want this abstracting and that is a huge sale point for OpenStack. There are customers who are moving to OpenStack because of all this abstraction. They don't want to go and train up on... The replication failover? Replication is one of them. They really like to see replication work, but in general, no, they don't want to log into their storage front ends and web GUIs and do things there. They absolutely don't, because that locks them in with training, it locks them in with all sorts of things. And it's a sales point for them. I mean, it's okay saying, well, they're not saying tough, that's an option, but it's definitely a value proposition for certain customers and there are some fairly large, slow IT crowds, you know, civil service, et cetera, who are looking at OpenStack now. And the value proposition of OpenStack is the fact that they can move between solutions in a couple of years' time or on the next cloud or the parallel cloud and require far less training and far less... Not zero, but far less... Yeah, I do have sympathy for that case. I understand it. It sounds really good. And if people are happy with replication V2, hey, more power to us. Way to go, go team. I'd love to sit in an experimental API till we're happy with it, because it makes iteration faster and easier, but there is a definite value proposition to things like replication being somewhat baked in December rather than having to go and poke a web console that's specific to NetApp, specific to SolidFire, specific to EMC. So there are customers who love that, and we'll hand over Bigbox to OpenStack vendors to do that. Who doesn't like Bigbox? So in defense of the replication API, and not the API itself, but its existence, I think it's necessary for a user who wants DR, a full DR solution, to be able to go to the destination cloud and be able to clone their volume and to run a VM off it and to see, okay, my DR really works. And sometimes in the real world, that takes 10 or 15 or 20 tries until you configure everything right on your VM to get it working. And you can't rely on the admin at the destination site because your storage is replicating your data and there's no Cinder on the other side and there's nothing else on the other side. You can't rely on the admin to test your DR for you if it's a cloud. Sure, but that's still, it's not something that a customer can do, right? That's something the admin does. No, a customer can do it. A customer can do the failover? That would be... Okay, well maybe then that's a flow on V2 and we need V3. Yeah, I can't believe we're still talking about replication. But, you know, so I, oh geez, but I'm, let me say something here. I'm a little confused through the process of the presentation here. I think I see what you're trying to get at and you're concerned about vendors that have functionality that everybody can support. We're talking about the importance of replication, but that's an example of one that doesn't work in the reference platform. So it kinda, it seemed like you were saying two different things there. But what I wanna get at, let's move beyond replication, let's talk about the next thing, whatever that may be. We have customers who want to get into the cloud who have been using storage for years and who have had functionality that goes far beyond what Cinder is doing right now. And I don't think we can say, no, we're not going to do it for them if we're gonna continue to have open stack to be relevant and move forward. So what we have to do is we have to figure out a way to work out together as vendors, as developers, as cross-company people. I don't care whether I'm from IBM, whether I'm from NetApp or HP. We gotta figure out a way to solve this. I think maybe that's what you're trying to get at in this presentation, yes? So more clearly, what my real point was was simply that the interoperability discussions should tend to shy away from those complicated abstract, complicated abstractions and those features that are not global. That was the main point. So what that means, and then the idea that we can have this set of experimental APIs where that vendor that wants to implement that thing, they can just do it, right? But then how do we actually, so we've got to experimental, we still have to get beyond the point where we agree that that's gonna become part of the product, or are we saying it never does? I'm saying that vendors may have their own things. Whatever that thing is, right? I'm saying your vendor may not agree with this feature at all. I'm saying it is okay for there to be features that are outside of the interoperability discussion completely. Yeah, I think that touches a little bit on what John was getting at earlier today with the ability of doing plugins that are external. Yeah, yeah, the whole contrip thing, and the experimental APIs. And the contrip thing, I think that works really well with the perspective. That's why I was kind of both happy because I thought they would be more well-received because a couple other people have very well thought out opinions about this already, and mine was just yet another, you know. And without any, without actually implementing a plugin mechanism within Cinder, there's no real way for a vendor to actually start doing those things without some major, you know, calisthenics within the code base, which is kind of painful, you know. And then. I'll look at my Gifos repo. I was gonna say, so if you want my absolute crazy idea, and this may be a terrible idea, is to have one generic API RPC driver call that the vendors can do dynamic dispatch. You can call, you can say whatever you want, and you can pass it whatever you want, and you can do whatever you want in that call. So if you wanna perform an operation on a volume, that way we could have even Cinder client have a generic command, right? Just pass down whatever the heck you want to your driver. Send it JSON. Yeah, just do stuff. Here's a package, do stuff. And then you could pass it down to the driver, the do stuff driver for that volume, and just. You've got to do nothing driver in the do stuff. Yes, I mean, so I mean, I don't know if that's a good idea, but that's one way where, you know, free rein drivers, or vendors could have free rein to play around to try this stuff. And that's fine, but that doesn't necessarily get Cinder off the hook here, because then you've got. Off what hook? Well, I'm curious because, you know, we just had a, you said there's all these wonderful features and everything that Cinder doesn't do or implement. Well, a couple hours ago, I said, what are the features that we're not doing? Because from my perspective, I agree with Corey's slide, create, delete, attach. It's really kind of the key, right? So I asked that question and nobody answered, including you. Yeah, I have a second example. Hold on. We have a static connection, but you know, maybe we have several parts. So some parts the customer want to use by OpenStack. They want to just set a restriction about the part range. So this is maybe a feature. I'm not sure this is a general thing. No, so I agree with that. That's a big difference from saying we're missing all of these great features that they've had for years. There's a big difference. I agree with that, but I want to focus on the fact that if we have some great advanced feature that we need to put in there, it should still be outside of the interoperability discussions. That's my whole point. It doesn't need to be agreed on. So the only one downside that I could see to this, which is kind of a little bit of a red herring, but if you upfront allow people to do what they want, they want in the experimental, which I think has a lot of value and I definitely support that. The downside to that is that you don't upfront require you to actually talk to other people to see if it makes sense to do it a certain way. Agreed. And that there is value in the community there by forcing yourself to talk with others in the community to do those things. Sure, I mean, so you will still get, I mean, you still have to go through Garrett, right? So people will still have to look at your code. But it is totally true. Maybe we would or wouldn't require blueprints for stuff like this, where people would actually say, you know, you're completely insane or, oh, that's a great idea. Why haven't we already done this? I mean, it's totally true that if we let it become more Wild West, it will become more Wild West. You were complaining about that feature chart and it's actually a very simple chart. You know, it's fairly linear progress. Drive is implement things later and later. So I don't think that's avoidable, given we've got 50 drivers and some things require driver changes. We can't change 50 drivers to get a feature in, I think. Otherwise, features would take forever to get in. I don't, I don't follow. You put up the slide with the feature matrix. To avoid the need for that feature matrix, you have to make everything standard. Otherwise, your feature matrix is going to explode. As soon as you allow vendors to support or not support something, that feature matrix is going to explode and probably become indimensional. And, you know, what releases the support on what driver is going to get more, that support matrix is way more complicated than it already is. I will say that's true to some extent. But I think it's slightly different when instead of having a standard feature and the people that implement it, you have a list of, this is my driver and this is the crazy thing that I do with that driver. It kind of changes the way, the meaning of the matrix. Does that make sense? You have this idea of here's all the stuff that Cinder really cares about. This is the interoperability stuff. This is the stuff that matters. And then you've got this stuff. Oh, here's the crazy stuff that vendor X and Y are doing. And with these drivers, you can make this crazy call. So. But what's crazy today starts to become more standard. I mean, that's what I'd hope is that. Yeah, yeah, this is actually standard, Cinder design session stuff here now. Yeah. At nine tomorrow. I thought we're doing something else then, Dan. So yeah, if we've just devolved into yet another Cinder normal dev design session, I'll just say thank you guys for coming. Hopefully we just open the conversation. That's really the goal. We get lost in the weeds in, this is meta weeds.