 All right, I think we're ready to go that's like That's good. Okay. We can keep this slide up the whole time All right Don't worry if it goes up All right. Well, this is great. I think this is the first session of the day The ask the experts challenges for open-stack storage We're gonna be recording this session as part of a speaking in tech podcast Which is distributed by the register every Wednesday. I'll try to get on maybe this week or next week a Lot of the folks that are on the stage have all been guests on the podcast And we might as well start off with it with the introductions first up Alex McDonald Alex is with net up. What's your title Alex? Good question. I guess I can read on the screen What is it this week? No, I actually work in the office of the CTO for net up and do work with industry associations like SNEA IETF and Organizations like open-stack and you're available for birthday parties too. Yes, I am also available if you want a vote fixed We do referenda where I come Next up Aaron delt from solid fire Aaron, what's your title? Bonjour. Good morning. Aaron delt director of technology solutions at solid fire Neil Levine Now director of product management for red hats looking after staff in particular outstanding Monju I work for office of technology and planning at HDS I lead a small team focused on a network solution innovation as well as open-source innovation including open-stack And last but not least mark to me some of you know miss stored Zilla last and least Not at all my ego will expand to fill this room any moment now My name is mark to me. I work in the core technologies division in the office of CTO for EMC So unlike the rest of them I do not have a team because they'd never allow me to manage anyone but eventually everyone works for me even if they Don't know it very nice We're gonna start off the discussion the title is challenges for open-stack storage and Alex One of the things we've had conversations with in the past is that open-stack storage might be open But it's not neutral. What do you mean by that? So this is a question that I posed myself about six months ago when starting to get involved with open-stack in Atlanta And this is a critis this is this is gonna come across as a criticism I have a very good French colleague who was asking me why I was so down about Paris because I'd mentioned the fact that every decade In Paris since about 1740 there's been some major riots That actually is what makes Paris really interesting In fact a lot of the architecture of Paris the white boulevards are based on the fact that you can march an army into Paris Really quickly And I'd like to see the same thing with open-stack storage in particular I think open-stack is open, but it's not necessarily neutral. What do I mean by that? I think that we've got to the point where in the compute space in particular We're almost looking at a monoculture. So being neutral is simply doing your best in the storage environment Doing stuff that's slightly out of the ordinary is a bit difficult and having a Uniform monoculture way of dealing with storage is incredibly awkward and neutrality gets lost in the process It was still open but not neutral and I don't know how to fix the problem And I don't actually know whether anyone else even agrees that if there is a problem with neutrality as opposed to openness Neil do you have an opinion about that? I mean I would I would look to you as kind of the Yes, we want some from Switzerland if you want real and neutrality I wanted to ask a follow-up question because when Alex sent email I was sat there reading it thinking to I'm not sure I understand What he means by neutral. I mean are you are you complaining it's open? But we're just having a lowest common denominator of functionality and as opposed to allowing some real innovation from different vendors Well, let me give you an example How many ways are they're doing storage replication and the answer is only two if you're working with open-stack That's a bit of a shame because actually there's about half a dozen ways of doing replete if not more I mean you wouldn't invent new ways and the problem is there's no way of expressing that in open-stack I am not sure if it's so much neutrality just as I mean It's inevitable that when you have multiple vendors trying to collaborate on something, but I they want to be seen to be playing Nicely as part of a community But I also need their own differentiation that you're always going to get a somewhat of a lowest common denominator of features Well, let's all the stuff where there is no disagreement and let's do that Which I think has certainly been a case in Cinder and then okay We've been our goal of all of our differentiating kind of cool stuff We don't want to commit it, but we do it feels to me. That's that's the tension we have now Okay, so I'm gonna put you in the eye Open-stack swift is not safe But there are people in probably in this room who would actually understand open-stack swift to be safe That to me is not neutral That's more of a marketing problem than a technical one I'd say I mean so the swift is unusual within the storage space of Open-stack in that it's an API and a project and an implementation all in one As opposed to other projects, which are far more sort of disaggregated And I mean that's been a more of a political problem within within Within open-stack here. So, you know, I think it's Perhaps certainly with an object storage pushing to sort of just have it within set We've always wanted to have it more as just an API where it can be a choice to plug-ins on the back end And there is some there is some movement towards that within swift But I mean Cinder in particular has, you know, I think has got a reasonably good level of abstraction out So you have solid fire able to Qs things have us able to do our own sort of copy and write things and so on But again, I mean, this is Great feel free to sit down whilst Alex and I take care of the session No Actually, I've got a question and this we open this up to anybody on the panel No, Alex brought up a good point, but this in Neil You also said it doesn't necessarily preclude the individual vendors from adding in those features into it. So, I mean let's start mark What's what's to preclude even though you've got the basic functionality instead of open-stack? What's to preclude a vendor from adding a certain replication technology on top of it? Probably nothing and our view especially when it comes to open-stack when it comes to Cinder when it comes to Manila is make these things better Right, that's that's pretty much the the internal strategy is that yes, they are lowest common denominators and I could you're not the de facto I'm not going to pick on you as the de facto Representation of open-stack. I am going to blame you for yeah Yeah, yeah, I am but I am going to blame you for system D, right? So But the so the thing is is that but making Cinder better and improving these things and adding these new models is What we're actively doing inside EMC? We don't have a lot of people working on this by EMC standards You consider we both 15,000 software developers for 300 people working on hardware I think we have a single digit number of people working on the the Cinder and Manila Implementations, but we're ramping that up over time But the the thing is there is the way we see it is that yes You can assume that in a standard you are going to end up with the lowest common denominator actually exposing value what we consider to be value-added functionality and Turning the storage array or the storage system taking preventing that from becoming just gray goo Which is abstracted under all all the layers of abstraction in the Structure-of-the-Service model such as OpenStack That that's down to us. So that's how I see it It's down to us to actually do that level of work and don't assume it's going to be there out of the box Because you can be dealing with people in a project who don't understand the whizziness of your product. That's right That's right here. Do you have anything to add to that? Yeah, I was just going to say yet really to Mark's point The big challenge here is how do you add vendor specific functionality? Underneath that that lowest can nominate lowest common denominator of that cinder abstraction layer if you will how do you effectively build? product-specific features into a project that is open and a lot of people involved in and a lot of Code commits into it. So how do you really develop that and that is a always going to be a challenge because you're basically taking a Standard, you know glob and trying to make it vendor specific under the covers and add some some functionality to it It's gonna be a constant challenge going forward Alex. Did I answer your question or satisfy? I love being judged in advance Just just just to give you an indication of where I think neutrality can go on Wednesday morning I've just seen Mark Carlson with Toshiba walking at the back. He's no But Mark and I are going to be running a session where we're going to try and convince the development People in open stack and hopefully some of you in this room as well that come along that it is possible to have a For instance object-based environment that covers not only safe, but covers things like you know swift in general s3 CDMI we you can actually build open and Neutral Infrastructures and the problem. I've got is I don't think we're doing that very well in other areas And I know that's a strange thing for a vendor to say but to be quite frank I think with open stack you either embrace the idea of neutrality and you don't go for the lowest common denominator And we all pull together in a given direction or we just pretend that you know That the open will do and neutrality is not an issue I'll poking you in the eye for just a second. Don't worry about what usually convincing is code, right? So it's when it comes to open source projects code, right? That's that's much more convincing I mean, I don't think when it comes to open source the kiss of death is I have this power point No, that's not gonna work So actually actually building it and people seeing the value there is the convincing thing But yeah, what you're what you're doing. I consider it to be good effort You said that you're gonna say Different question minding to start with you We'll go through the whole panel here. What do you think is missing in open stack storage today? features or functionality Adoption, what is it? You know the way I look at it is it is an evolution It's sort of like building on the first question you started really, you know, we are at a point where there is a set of And that set keeps on increasing And it will continue to improve right there is still, you know Basic functionality Replication is a very good example actually There was a project called the workshop that started and I don't really know where it went There's not a lot of activities going on with it And I think that's of the problem is a lot of these high-end features These guys they really work hard to build something Differentiated features And what's really happening is take that and Try to bring some sort of like So that you can start using it across You know different hardware vendors platform that's going to take some time, you know, I think What I'm seeing is many of the vendors are Let's get my clothes many of vendors are getting to a point where they're not defensive, you know, like just like Alex said I think we're seeing across the board with all this storage vendors willing to open up their technology in terms of how they are differentiated, you know features work That's a good change that's happening and the next change will be what's the Best way of doing it, you know, in a more like commonly agreed upon way of doing this high-end functionalities I think in terms of where the gap in the center functionality I think that's what will be the evolution or taking this high-end differentiated functionalities making it available across anybody's Neil I want to go to you next see where's so the gaps that you've identified I Can't do it from a slightly different angle. I think I mean certainly from our customer feedback There's always another feature. They want to send it as always, you know, some extra part of the long tail They want but I think the bigger challenge is actually integrating storage as With many of the other components going on with an open stack specifically look at the policy stuff Where this is I mean the feet would be doing feature development forever in a day But actually trying to get an open stack to work as a coherent whole So somebody sets a policy that policy gets equally applied to storage and it working compute and those three things are then also talking to each Other to ensure that that policy is adhere to I think is actually where most of our customers seem to be driving us And so I think you know, this is this is a coordination problem, and it's not even necessarily a technical one It's just defining where the boundaries are responsibilities about who does what do you mean my coordination problem? I'm not sure I follow you then well There's multiple projects around policy stuff Congress is one but there's other companies spinning up their own You know who's responsible for that foot for the different components of that? Where does it? Where does it get police? Where does it get applied? I think a lot of that is again defining the technical boundaries and responsibilities of individual projects and getting a common Interface between them, which we don't have right now. Well that usually involves for that to happen It usually involves someone else surrendering right and actually becoming part of a project. It's not about surrendering I think it's You know, maybe this is more for the technical committee with an open stack to actually, you know There's this huge explosion of projects and actually say like it's fine to let's let's have a you know A thousand flowers blooming, but let's make sure we get interoperability between them well established through, you know, whether it's I don't know. I hate to use the word standards But it's just some clear sort of reference so we know if you want to start your own project another object storage project or another Policy back to your law's common denominator Again, but it's very possibly but that might be fine We don't even have an lowest common common denominator for that actual sort of policy framework quite right right now I think I think even when that's in place. I think especially when you talk about storage vendors and Let our name in just a second. No stale You you talk about storage vendors all them have their strategy, but Open stack. It's so large and so expensive It's going to be tactical engagements with each of these projects to nudge them forward in your direction So I think the fact of the matter is that I think there are very few people especially if you're if you're in the storage business are going to take on the the behemoth that is the multiple open-stack projects and try and Weal to those and shape them in direction I think you're going to see a lot more tactical involvement with individual Projects so cinder where we understand the the storage layer and how that will advance but when you talk about things like glance and telemetry and other other things going on there like that I think where you're going to start seeing much more tactical Interactions and investments there. Yeah, and I was just at a macro level completely agree with with Neil on The idea of we need some kind of policy-based management across projects because you know even within cinder Doing volume types for management isn't going to cut it long term. I just don't believe that but at a micro level something else I would also bring up though is Having a cinder driver just isn't enough. There needs to be more transparency of you know What does your driver do and what does what works and what doesn't? Because the drivers are very different in case anyone hasn't noticed yet. I know shh don't tell anybody So that is definitely I would really like to see it, you know Being up in Github makes them all the same You know that lowest cannot lowest common denominator isn't always a lowest common denominator So I see that as potentially being issue another area to to kind of go along with with what you were saying earlier of how to really offload some of these features is there's two ways obviously is to build it into cinder and kind of raise up that common denominator or Honestly making a call from cinder and then offloading you know to the to the storage array in some way a common call that offloads in a unique way Behind the scenes which I don't think we're seeing enough of today as well from an architecture point of view I think it's going to be the common call. Otherwise cinder will start looking like some hair ball Alex yeah, I just I just want to make sure I'm speaking of hair balls. Yeah Fuzzball they just to make sure I wasn't understood. I don't believe and I'm not claiming the neutrality is lowest common denominator I'm having to look they are tickling the difference But if you want to have a fight with me afterwards about it, I think I'm up for it The I want to get back to the policy issue though. So how many of you are as old as I am Just you just it's just me no one raised their hand for anyone listening to this no one raised their hand This is a warning about doing policy based stuff because the bill joy of Sun came out with what we call the seven fallacies of network computing Some of which goes follows bandwidth is infinite latency is zero Shortage of resources is somebody else's problem There are a number of issues you need to face with policy management one being that it's very difficult to express Policy for storage. It is not a simple task at all Although it might be pretty easy to or you might think it's pretty easy to express policy for instance for a network Well for compute doing it for storage adds in a whole layer of problems And the one thing that you know I can agree with everyone up here And I think they'll agree with me one of the things that's different about storage is when you give us a bite We guarantee to give you the same bite back at a later date And that is actually a really really really really tough thing to do Especially when you layer over the top of it things like policy in terms of performance in terms of cost in terms of scale in terms of location We solve and I can't emphasize this enough people who work with story We solve some of the most difficult problems that are known in computer This is rocket science dealing with storage is rocket science the networking guys say it's not that no that The networking guys are right. This stuff is rocket science. They just have to be small cheap and readily available rockets We're doing stuff that's really quite interesting and quite exciting and I think storage is the last great Frontier in terms of computing, you know, how many network vendors are there? How many virtualization vendors are there? Nortel bit How many storage vendors are there a lot and I think that's actually indicative of the vibrancy of the industry But before you start doing policy based stuff You really need to think about what you mean by policy and I really recommend looking at the seven fallacies a network Computing it still rings true and very much so would to do a storage, but I think Alex I do agree with you on the complexity of you know, really Defining the minute details of the policies for storage management, but I think you are also gonna agree that as It stands today We don't have really good even you know higher level policy management for open No, in fact, the only policy management we appear to have is I'm either gonna do this really cheaply because I'm gonna go By a bunch of discs and string them together in a line Or I'm gonna do this really expensively by buying emc net up or somebody else's yeah Yeah, that's my point. I think like it goes back to what Neil was saying I think you know on one side there are feature enhancements that we could do on the Cinder but the other storage related enhancement will come from projects adjacent to Cinder and You know key one is a cylinder the cylinder I think in the Atlanta summit we discussed quite a bit Cilometer is really good It gets a lot of the server information and you can do some monitoring take some actions based on the information You get from Nova, but there's not a lot of information fed from Cinder back to Cilometer I think there's there is some enhancement work going on. I know within Hitachi's team itself We're submitting a couple of blueprints try to enhance Cilometer as well as enhance Cinder so that it's sending the You know useful metrics back to Cilometer. I think that's another area definitely You know I do see you know some decent evolution happening in the near future Neil before you go If I could just say anything any questions in the audience. We've got two microphones here please feel free to Step up to the microphone and we'll get to your question. We got about 15 minutes left here Yeah, I just want to come back to To Alex's point. I think a lot of the policy stuff isn't actually even storage stuff It's just it's networking and perhaps I have a very self-centric view on this where everything is is you know commodity hardware It's all a software process just making sure is the network Well, we can have a fight afterwards, but me and my friends first of all But we you know the just ensuring how we got the right VMs running and how we got the right networking He's almost likely did the first part of what we need certainly from a self-perspective So this is not even about worrying about the storage pieces, which I agree There's all sorts of challenges, but this is where I'm talking about a coordination It's actually even before we get to the storage Just have we got the right infrastructure laid out and the rest of the open stack projects for us to do our job yet And right now, you know, there's no communication really. It's all very isolated You know one of the things I want to bring up, you know Alex kind of hit Oh, we do have a question go step right up. Yes. Thank you Eddie Schwarzman IBM I have a question for the esteemed panel in keeping with the theme of neutrality. Nobody said esteemed So I would say and I think most would agree that the sort of NAS ice-cozzy story predominates cinder Cinder drivers, but recently We've seen a lot of progress in Sort of the sand fiber channel NP IV story So in in in reference to this discussion about neutrality, do you see sand NP IV as an equal citizen to To ice-cozzy NAS and and the other attachment type what a great question go ahead and the answer is really simple I think it's because the project technical lead isn't broad enough I think it's too narrow and that's why we suffer from Narrowness of solutions. That's basically it Well, we need broader governance of the projects we need broader engagement And and sometimes a lot broader scope for some of these projects than we currently have But that doesn't come back to the early point, which is I mean just turn up with the code This is making source project. I mean from a from a tentative point of view I mean Seth again, it's been very selfish. We don't use ice-cozzy tool We've got our own implementation for doing sort of client cluster based Communication, you know, so Cinder hasn't limited us in terms of what we can do We've not been restricted by just having to use ice-cozzy or something else So we've not been restricted, you know, we've just turned up the code and there are challenges getting code in Which is a whole not a topic here, but I don't I don't even know if it's a scope I mean, I think the scope of Cinder is there by is driven by the committers and I'm thinking of fiber channel, which everybody does Yeah, well, and I was just gonna add The way I look at it as well is is actually take it above storage for a second and really look at workloads and The infrastructure that supports them a lot of times, you know When we're doing open stack at least when we started it It was really a workload that is honestly more of an ice-cozzy based workload of really scale out doing a lot of Really cutting-edge newer web-scaling kind of things, right? But when we're we're starting to see fiber channel brought back in it is quite frankly to embrace the enterprise and a lot of times it's because there is no way ice-cozzy is going to get in that environment and it is really more of green Field versus brownfield and different application workloads and a different existing infrastructure is what I see it Yeah, I I couldn't agree more with it. We started off with ice-cozzy support for our storage platforms and The the same customer who had asked us to support the ice-cozzy platform came back to us to support Fiber channel as well as fiber channel or ethernet on there on our highest and storage platform So I think the change is definitely happening, you know to your point You know it had to do a lot with the workload, you know the kind of workload That started adopting open stack, you know ice-cozzy was good enough But now real enterprise applications started deploying on open stack FC is coming back I was really surprised about FCOE. I'm actually not sure other than HDS anybody else had to work We had to do work because of one of our a large customer So it's not even just pure FC. It's an FCOE is coming back Just on that our view and it's because we support fiber channel in our drivers for our platforms What we're doing is that we're working from top to bottom right through our things So whether it's not support ice-cozzy support FC We're doing that but our view is that if we want that work done the same thing with consistency groups, right? We have to actually push the work so because It's it's not going to come from nowhere. We can't this isn't the the Linux model It well kind of but it's not really the Linux model where there was the idea start out Well, I'm going to build I'm going to build an open source version of Windows 95, right? If you remember way back when that's how it started This isn't what we're doing in the enterprise market with open stack So you got to got to get there and actually start begin cracking out the code if you want to do that And that takes time it takes investment you have to commit Is it satisfied? Yes. Thank you very much and just one other thing. I wanted to mention There's also the area of zone management right where we've seen we've seen significant progress working on that too Yeah, you have to assume you're dealing and this sounds terrible But you have to assume you're dealing with people who don't have the 25 years of experience that you might have in a particular field So it's just up to you to actually lift that weight. I agree. Thank you. Got another question Hi guys, I'm Aaron from in advance and we were speaking about the Amount of work that remains on the storage and you spoke about the network Vendors and there is a use trend around the open site is software define something and a lot of network vendors You try and there we go. That's their new try on is the implementation We have Cinder which works and then there's new Trump. Sorry go ahead Many network vendors have switched to software defined networking and around this panel I see only one which is doing software defined storage Which is reddit with self and my question would be what is the thinking about the legacy vendors around this So if I define something around the network, I love when I get I love when I get call legacy What that means is that you make billions and billions and billions per year and you're competing against a guy who's made no money So the software defined storage isn't necessarily you believe that red I would disagree that red at or the only people up here doing software defined storage We might disagree on that. Yeah, you see no, I couldn't agree more and I don't often agree with Mark Yeah, we don't we just don't so you've done something very unique here and you know, he said two warring nations No, it's a but let me ask you couldn't just come down to what your definition of software defined storage is What is it about red hat that that make obviously you've got the belief that that that they're shaping Software defined. What is it about red hat for you that that makes that definition the distinction to what the other vendors are doing? Mostly today if I want to set up some Distributed storage on an open stack system. My first thing is using set Well, because you've chosen to use that that doesn't mean it's doing for example emcee offer scale at all Which is purely let me let me do you and what what you probably mean by that is it is a software package Yeah, that you can download and put on commodity hardware. Yep, and Yeah, and so so what I'll what I would look at that is as I would say actually Although it you are buying nodes with solid fire I would say it is if you notice what the nodes look like they look a lot like an industry standard server I won't say the name but super micro Actually, no the super micro no Quanta are our six twenties so But but again at the end of the day what we're looking at is Really, we could very easy for easily flip that hardware package It is just a distribution medium for us, right? And so we could actually take that and really flip it over to another manufacturer if we wanted to it's just for us It's it's a way of packaging and on packaging I mean we do deliver emcee deliver software as appliances, but also a viper object storage It's just pure software. You download install and come out a commodity soft hard commodity hardware scale I oh same thing pure software Download it install it's just an installable binary So the idea that you have to buy a server and a whole bunch of discs from the vendors up here isn't true I'll even go to My colleague at the far end Alex. I mean they have their cloud and tap System it's software as well running on top of public cloud So the legacy idea, I think there is the perception we sell big tin right with lots of disk drives But there's it's just software and we have our own software on the options Okay, let's give Neil a chance here to start throwing some darts the other way I'm far too diplomatic to do that. I mean so how exactly did you kill Swift? Tell us it's what we haven't killed Swift before we we support the protocol and active members red-hacked up to Contribute us to the project the other listen It's I really hope this panel doesn't turn into let's define software to find storage where which you know the objective definition of software to find Storage is whatever it means to help your business cell. That's what it means But I mean your point is there is it's it's all software you pick the hardware you want to use I mean the you know the red hat attitude is it's not just about control plane It's actually the data services should be open as well I mean that's one of the big differentiators between red hats and some of my peers up here Which it you know is a differentiator in itself, you know Cinder has been a fantastic project at the control plane But stuff has been popular because it's open at the data service as well and that's very important to distinguish Well, my view is that everyone else needs to be more open and you're too open That's great. Is that answer your question? Yeah, thank you No, I'd like just to amplify one point I'd like to go back a little history and explain the reason why I think software to find storage has become slightly overused phrase the first thing to recognize is that when Let's take net up as a company because I understand the history of it So when we started out we started by just delivering software you bought your own hardware and you actually run our so run our software on Somebody else's hardware the issues with that are enormous because you're then into hardware variability and hardware 25 years ago wasn't exactly Great stuff. It was pretty unreliable. It broke regularly and we spent an awful lot of time fixing hardware problems rather than software ones So we standardized on a set of hardware Guess what? We're now in the position where the industry is actually standardized for us on a set of hardware The number of hardware manufacturers has dropped like a brick. How many chipsets are that out there one one? Well, two with arm Two with arm, but one for all practical intents and purposes for this audience currently I mean arms going to make progress So the point is the variability has gone from the industry. We don't have to support decal for any more We don't have to support HP risk. We don't have to support Intel chips with flaws on them because believe it or not these things come out of the box That and they don't necessarily run as you might think so And disc drives in particular. Do you know a complex of disc drivers a Flash drive is even as complex. I mean these things take an awful lot of engineering And you can't cover all the bumps with with software software to find storage depends on having a reliable and regular Set of hardware underneath it and going to use the well-worn American phrase to fries or PC world is the UK Well, I don't know what you do in France or Germany You go off to some your local shop where you can buy disc drives and put together these things is only an approach The companies like Google and Facebook can take why because it requires vast amounts of engineering skill and vast amounts of money to do it No, yeah, that's probably a different topic for a different panel, but thank you very much. Jay. You have a question Yeah, I believe it and I have a question My name is Jay Mets. I I work at Cisco, but I speak only for myself Is it about knowing where you are because we can we can help you with that one immediately? But anything beyond that that's what my phone is for it gives me the GPS I think one of the the major issues that I have with the challenges for OpenStack in Tying in some of the things that have been talked about with neutrality and tactical elements and and then lately the fiber channel introduction into into the conversation is That when we do storage and storage networks, we have two different types of approaches Effective we have the lossy approach, which is currently the denominator the dominant theme inside of OpenStack And then you have the loss less approach, which is the dominant theme inside of fiber channel over ethernet loss of scuzzy FCOE They don't really match very well one is a top-down approach and one is a bottom-up approach and What I'm concerned about is that the panning off of the the tasks to these other groups like the policy group And the the neutron groups and the networking and so on they don't care about loss of storage and And when you start building up the cinder files and a cinder block and you want access them by a fiber channel For example or fiber channel over ethernet. What's the feedback to know your over subscription ratio? What is the elements that tell the network? Hey, I've gone too far because you can't drop frames You can't break the storage network in those environments Jay dropped a question on us with three minutes left I was waiting He was patient getting into that three minutes. I mean it's The concept here with OpenStack is sometimes the question becomes you talk to customers What they want from OpenStack and they say I want a cheaper VMware, and that's an absolute lack of ambition Right, it's a lack of them. It's an absolute lack of ambition. It goes back to the hole I want to clone Windows 95, but do it with open source software You know that stupid idea that then evolved into a multitude of approaches And I'm just going to take this wrong to what I said earlier It's up to any any of the vendors who provide the last list of to do that work because it's not going to emerge From people who have no interest but that goes back to the neutrality point Because you know Neil kind of brushed it off and said well getting into the code base is a different story No, that is the story You know because we've had we've had a very public PTL who said oh, I think Cinder is software-defined storage and he is not a fan of fiber channel Is it any wonder that we haven't really seen or because this is the perception now mind you I'm not saying that there's necessarily a conspiracy. I don't disagree, but the perception is that no fiber channel is welcome here So if that isn't a challenge for OpenStack storage, I don't know what is and that goes directly to his point about neutrality Thank you I'm gonna sit down now and Transcripts in my speech are available I just want to emphasize that I just want to emphasize that it is actually changing. I know the exact conversation you're referring to But I do see things changing, you know, I think I and you sort of touched on one of the topic Which was sort of like the the new age applications and The vendors who are helping to solve the demands of the new age applications, right? They don't necessarily have the knowledge know how of what are the challenges of really making it work in a fiber channel type of environment So there's definitely I think that you know the ignorance issue that has made some of the challenges that you mentioned Become significant, but I do think that like, you know, Mark said and I'm seeing from Hitachi as well, you know I'm seeing like vendors like EMC Hitachi NetApp. They do start they have started Pitching in and they do have the kind of expertise that you're specifically talking about right? It's changing. All right. We're gonna have to wrap it up. We're right at okay. You'll get one word in I would I think you're crediting too much to the to the PTL sort of power here I think we've been open-stack. It's it's like any other open-source community You have to win friends curry influence and commit code and you can change things. It's not about one person blocking It really isn't and I think so, you know, it is not a technical challenge. This is community challenge. It's open source And that's a great way to end this session. We appreciate Neil Alex Aaron Neil Monju mark. Thank you very much. Thank you for attending appreciate it