 With the insiders get moving My god, man Let's let's kick this off. My name is Greg near a man of technology evangelist for Hitachi data systems I'm also the co-host of speaking in tech and this session is going to be recorded as a podcast for download For subscribers so right off the bat the session is called to ask the experts designing storage for the enterprise We did a similar panel to this in Paris in November with the same panelists. Everybody has survived since Paris You'll glad to see you're still at Red Hat You haven't you haven't bounced yet for me tank So let's go while you haven't Gloucestered Seth yet, so you're making progress. There you go Let's begin The doors all right, let's go right down the line mark introduce yourself and your company and title mark to me Technology director CTO team emcee Yeah, need the mic Aaron Delp director of solutions at solid fire Neil Levine director of product management at Red Hat storage Manju Ramanakura CTO for intelligent platforms. I'm responsible for some of the open-source Related incubation strategy and planning as well as network solutions. We don't need the life story title Who's the moderator Alex to you Alex we're all from NetApp member of the office of the CTO doing standards and industry associations of which this Open-stack is one of them Fantastic and Alex we're gonna start off with you this week Thank you last year or last keep saying last year six months ago. Yeah, MPI feels that the whole session felt like a year Let's talk about the maturity of Enterprise storage in open-stack since the Paris summit and just what you've seen here at the show this week Where are we at? So, you know, I'd like to thank everybody for coming along to the session in the first place It is quite interesting saying people coming along to ask experts When what I've seen this week still I'm still a little disappointed pleased in some respects We seem to have made some progress Disappointed in other respects and the respect I think I'm most disappointed in is the Understanding that people have of storage within the open-stack community is not perhaps as good as it should be Not could be should be it needs to be a lot better What's the context for that? Yeah, obviously we all understand story. I'm still seeing presentations where people are talking about Vendor lock-in. I mean come on guys grew up. This isn't about Vendor lock-in. This is about providing business solutions with enterprise class products Out to the businesses that you serve in so that's the first thing and the second thing is some of the technology Issues I've heard described here this week. We solved in the 1990s These are 20 year old problems. You're digging up again. Now. I'm an old guy. I've got gray hair I've been around the block more than once twice Four times a lady There's a song in there somewhere as a village when he started not a block So that the point I'm making is that we appear to be in the open-stack community Reinventing problems. We've already solved and one I heard of yesterday for instance was Scrubbing data scrubbing on drives. There's another, you know, we're looking at Implementations of storage. I'm gonna know names. No pack drill with large numbers of SSDs that performs like a dog I mean, how can we how can we build systems like this in that? You know in the 21st century in the year 2015 it's it's crazy guys We really really gotta grow up with the storage stuff stop thinking of the vendors as the enemy They're not that your friends. We've done all this before we can actually provide you a solutions as well That are cheaper per gigabyte per month than some of the advertised figures I've seen this week from people screwing together stuff that to be quite frank My ex CTO great guy. It's got a real phrase. I use all the time, which is There's a difference between Computer science and software engineering and what I'm seeing is a lot of computer science Okay, we need to do more software engineering. We need to be a bit more robust In contrast to that Neil, I kind of kind of looking at you and wondering what your reflections are That's actually from the perspective of Seth, right and we've already had specifically Are you seeing some of those same recycled questions? Or is it different now because we're more in a software oriented era as opposed to a hardware infrastructure era? I think well, we're going to speak directly to Alex's point when I have a different perspective here I mean, I agree of a lot of what he's saying that there is a lot of sort of fundamental stuff being addressed Especially when you look in some of the open stack services I mean there's things that should be implemented which should be easy and still don't work and are pretty essential, but Yeah, the computer science is a software engineering. I mean, this is open source open source is is not a you know It's a anarchistic kind of community effort. It doesn't produce perfection first time it sucks, but it's it's brings a lot of other benefits From from the from my perspective when I see with self customers. Yes, things have definitely matured with Deployments now. I mean we have customers not just kicking POC's and doing tests and there they have real production workloads with Not just single storage back ends. There's a variety of storage back ends. I see actually it's not exclusive anymore Which is great. This is what we wanted. We want to be able people to be able to plug in different things into different drivers and You know, these are mature systems. So are they perfect? No is there more to do? Yes But it is adjusting business problems people are running, you know, we've seen some of it from the keynotes There are some pretty big systems running with a variety of storage systems underneath that that's that's a success I think Either before you I just want to remind everybody that we've got a microphone here right in the middle If you've got a question, this is a session for you ask the experts just like welcome to the microphone And we'll get to you right away. Go ahead, Eric. I can talk now Yeah, so no all I want to do is kind of maybe tie those two points together and just kind of say You know again the title of this is is enterprises in the title there, right? What is the expectations of the enterprise the the enterprise has a much higher set of expectations? I think that then really some of the open-stack way over the years has has been providing, right? But I think we're getting there without a doubt and that's where I think Neil's perspective comes in if How do you take open-stack and really make it up to the enterprise expectations? I think I think we're getting there I do but at the end of the day to what is the expectations of Cinder specifically within that and You know at the end of the day for most of the conversations I've had with folks is they just want it to be a clean abstraction layer of the hardware a You know interface that they can kind of use just like they've been using previously with storage and it's more of getting those base features in there and Getting the drivers in there for all the different platforms They're using and having them actually stable and 100 compliant into the Cinder API and fully functional and That is probably the biggest challenge right now still and that's something you know I think we talked about the last time we did this. Yes, that's true And it's you know, it's six months later and I you know They're getting better But those issues are still there and that's one of the biggest challenges I've seen talking to enterprise customer specifically Monju we've talked last time about some of the gaps In enterprise storage for the open stack environments. What are your observations between now and where we last had this panel six months ago? I think I'd agree with Both of the gentlemen who spoke about some of the gaps that's no fun. Yeah, you guys get reasonable not a gentleman Okay, there we go. So this dude and this gentleman So basically if you look at it right last a last panel discussion We talked about Cinder is ready in terms of doing the basic functionalities, you know Provisioning the storage replicating storage snapshotting these type of basic functionalities Cinder is ready That was never the forte from enterprise storage is perspective. Those were table stakes, right? What you do from security what you do from multi-tenancy perspective and what you do from monitoring what you do from troubleshooting Those were the things that were always the value added functionalities And this was the reason always enterprises went with vendors, right? Anybody can do the basic table stakes stuff. I Think if you look at where we are and compare with where we were we are making progress But we still are not there, you know I have an engineer here in the audience who's been working with the Cilometer working group, right? Cilometer is one of the you know the component Especially from a Cinder perspective There's not a lot of metrics that it can actually gather make it available from a monitoring point of view or you know fine-tuning perspective I think things are Progressing, but we are still not there. We are better than where we were about six months back Obviously people are working very hard, but I think you know if I look from our customers perspective Customers are looking at Cinder to do the basic work that table stakes work and beyond that They are still falling back on traditional features that enterprises have provided. I think yesterday. There was a Presentation I could not join but some of our guys joined AVG I believe was the Customer they're using Hitachi storage today in production with Cinder and Believe it or not. They're running other third-party storage. I won't name them. They're behind here Hitachi storage And there's a reason why they're doing it. We have a technology UVM that people like it, right? And this customer happens to like the technology. Similarly every vendor has their differentiated functionalities that are in their storage and Unfortunately what I'm really seeing from our vendor. I mean the customer adoption perspective They're still falling back on these value-added functionalities that are really really needed. They're not really optional features So that's what's happening right now. Cinder adoption is growing, but at the same time The the enterprise features that vendors have always had They continue to be add-on features on top of it, which becomes really the key piece of it Cinder becomes more of a you know standard table-stack features We gotta get to mark here because I know you've been waiting to Chomping at the bit because everything's changed in six months. No, yeah, you weren't there. It's you're all new So we turn around I look at the mark of what's going on in in our case is In the emcee case should I say is that yes people are still doing it with the storage array of platforms They're open stack and one those tend to be people who are just adopting open stack Anyone going big is doing that usually the software to find storage routes like scale I or something like that going big But when we talk about enterprise features and moving forward and something I picked up in this show The only way we're going to move storage forward, right? Everyone up in this stage and the other vendors are out in marketing We're gonna have to get together and start pushing some of that experience that would that we have If we don't if we don't put the rocks down and stop battering each other And kicking each other in the face just for the laugh mostly Much bettering going on though. I would say I think generally in the storage market was I don't vendors like to come hold on Was I I'm sorry No, but I mean generally isn't in the open stack environment. It's a little bit more friendly I mean look at just look at us here on this panel. It's it's I don't know if it's as confrontational as it is I think it's I think anytime any time you have a sales guy coming in with another sales guy It's going to be they're going to beat each other with the rock because we're here It's a bunch of engineers sitting on the stage I think it's just the problem of democracy things will evolve But when you put multiple people with different interest, you know as a you know most of the people who are contributing to open source today, right? there is a vendor backing behind it, right and From vendors perspective, they want to be a good corporate citizen to open source But at the same time they have to keep their job, right? So that's where the difference in priorities what my priorities are would be different from To me's priorities are so I think that's where you know, that's kind of more of a democracy things are evolving up Yeah, I can I can just hand you my microphone and you know, you can finish my part I was just gonna say you know that that I used to work for a manager used to say, you know We live in a democracy, but you don't get a vote and I think that's that that's a way a lot of a lot of this Needs to necessarily work the point I want to come back to mark on about the the cooperation between vendors at the education aspect of it as well I think we have been remiss. I think we as a as an industry have been remiss it not educating people about How difficult storage is I? Can't really explain how much work. It's an enormous amount that we collectively Do to make SSDs and disc drives actually look like SSDs and disc drives They are the ugliest pieces a kit under the covers you've ever seen we spend enormous amounts of effort Making them look nice The other thing that we do and we try to do really well is Virtualized stuff so that you end up with you know, we don't have puns anymore. Do we have lones? We went through that phase 25 years ago. It's all logical But the virtualization I'm seeing here. You must be really careful. There are two things You cannot virtualize away from storage you can virtualize a certain amount of reliability into the solution so we can always make discs look more reliable than they are But we cannot virtualize more bandwidth into a system and we cannot virtualize more or less Latency into a system you have to remember the underlying hardware always pokes through in the end Bandwidth and latency are two things. You cannot avoid. They are physical in nature And that I've been seeing again this week where people believing that by having Some form of abstraction layer you can abstract the way the issues of bandwidth or you can abstract the way the issues of Latency and then we get presentations about how people have gotten seriously bitten in the bum when the real world imposes its You know it who is it? There was a presentation. I'm again. I don't mention any They said they tested a whole bunch of SSDs that didn't perform as they thought they should You know how I mean we spend a huge amount somebody making sure SSDs perform as they should because we chuck away The ones that don't we don't use manufacturers where we know they won't work You guys are in exactly the same situation. Yeah, and I was that's what we've done really badly is fail to educate you It's a really good. Yeah that to give you a for instance to get take that the SSDs for instance in in my example of We do a tremendous amount of yeah How do you take not just the same batch of SSDs or a bin of it? How do you take them from different manufacturers and account for all the underlying? discrepancies in the specs of all the different drives and then potentially maybe if you don't want to depend on That for I don't call garbage collection right how do you disable all of that? And then write your own so that you can basically remove hardware dependency I agree I completely agree and you had mentioned something actually last panel that I want to bring up again of you made The point that I thought was really great of again at the end of the day storage is really the only one out of like a lot of the projects here of At at the end of the day, how do you store something now? Right and be able to retrieve it at a later date Right how do you retrieve it next week next year and be able to you know? The vast majority of the time get it back in a reliable fashion the reliability is is so key to all of this without a doubt I'm trying to work out what this has to do with open stack I mean I think nobody's going to disagree with the comments. Do you're making here? But let me explain very quickly what it's going to do with open stack I believe that we are in a situation where we're very comfortable with Networking we're very comfortable compute and we've been through a cycle of understanding I don't know you know you're not going to the neutron sessions if you think that's to taste them Let's pick on neutral One thing all the storage people can agree on that we we don't like this But you know I mean that I don't disagree with anything you're saying here I mean he's a generic storage problems I mean I'm wondering whether the thing you're picking up on is like who's who's doing this now something from the the self perspective We definitely do the storage teams, but often we're dealing with cloud builder teams. These are not storage teams These are people have different expertise. They are Linux people that DevOps people. They are running fast They're not they're taking 18 24 months to build a solution. They're doing this in three to six months I'm wondering here. What is just a persona issue that? You know this was coming back to my point that I got chopped off on Maybe great can finish it for you. Yeah, exactly. So do you want to handle that? What what it comes down to especially that what I was going to say is that we around Not only on education, but around pushing things forward is that we as vendors need to come together Otherwise something will emerge inside open stack. It'll be top-down. It'll be all written in Python It'll be some all of these things that are going on We'll find it absolutely immutable and we're going to lose that experience because if we don't do the legwork It's not going to happen. And as Neil where Neil said it's cloud builders. They don't care, right? They're going to use they're going to use what's considered to be best of breed and if we don't define what's best of breed as Vendors, we're going to lose that. Well, I don't know. You can't force what what is best of breed on people That's got to emerge, right? Well, we're already forcing neutron and people I know it's turned into a new transaction, but the way I see it there are projects We know what we mentioned neutron. We mentioned salameter. It's like these were invented here You must use them. It's like no are you trying to give a fantastic lead to the big tense come question here This is the fantastic way. I was laying all this You should be in radio. You I should I just sit down over the side here. Just let you go You're supposed to be the moderator. This is good. This is the talent Neil Do you want to ask a question? We're not giving you any airtime here at all now I mean this this seems to be leading up to we you know, we actually do prepare for this panel It sounds odd. I mean it does consist of three emails, but it still counts as preparation here About I read one of them It was in the middle. Yeah, just picking up on it. So, you know, there's there's a there's a we were discussing whether how to Big-tent philosophy and governance model that's moving into open stack is going to affect storage here Which I think it's kind of where you're going with this which is pretty much where I was going with it You know from a storage point of view, it's probably well I think storage and computer network networking. Those are the big fundamental things, right? Those are probably things which you want to mess with the least because people have got the most invested in them They've they have the most impact if you change anything here And I think this is the big tent probably is a great idea for ancillary projects which are not dependencies Which are not critical. I think I think absolutely makes sense. Let's survive under the fittest But yeah, we have an issue here where there's there's a there is a tendency in open source to say It's not invented here. I don't want to maintain old kind I'm going to do something new and you know the question is is that going to happen to the storage layers and That's a risk. Yeah, and and to add to that a little bit too of the biggest Issue I guess you could say that I see with with the big tent philosophy as I completely agree with the philosophy behind it of its competition survival of the fittest I get that but is that in the best interest of the user base and growing the users right now it seems to me that is it is Big tent is focused especially in a storage context is is focused more on the vendors It's more focused on the developer isn't it mostly navel-gazing though. I mean how many a customer point of view it totally is yeah Yeah, outside looking in I believe it is and and the reason why I say that is because you know if you have if tomorrow, right a You know that another cinder project came up. That's not cinder and started to compete with cinder Which is entirely possible now with big tent You know, do you have a fragmentation of the ecosystem does that confuse the users do I you know It does now your storage platform have a cinder driver or a cinder dot next driver It's it further makes The outside looking in and an adoption of new users into the open-stack ecosystem It makes it very confusing you have this this dichotomy of big tent on one side and then Defining a core on the other end one of them is very vendor focused and developer focused And one of them I feel it should be very user focused and use case and application focused And that is where I'm really struggling with with big tent of at the end of the end of the day Will it will it be best case or best use case short-term versus long-term? mind you Yeah, I mean there is a good side and a bad side to it. That's your point Alex, right? So In the near term my opinion is that yes, it's gonna cause some confusion, you know, you've got basic components Nova Cinder Swift Neutron Manila now But anything beyond that right it's anybody's game today like as you go higher up the stack there's a clear overlap between what cloud foundry is doing and what open-stack wants to and what Kubernetes is doing what Apache Mesos is doing. I think Containers conversation all of a sudden the room will explore the people that's the whole Containers. Well, that's the whole point of the big tent, right? We want to be a one big happy family Let the best solution win that fits the overall objective of the providing your next-gen cloud solutions, right? I don't have an opinion on big tent, but how can you be one big happy family where somebody wins? Oh, that's the point So so that's really the point, right? Who's gonna win, right? If you really look at how open-stack really came to power It's the developers at the end of the day They put in their sweat and made the initial components work, right? The making it work is the key thing right right now as you go higher up the stack It doesn't work very well across all the pieces that you want to integrate So what's going to work, you know, I think it's only going to time is going to tell that and Will it be one vendor driving it? Will it be like Kubernetes driving something or EMC driving something through the Snaky product, whatever, right? So But the point is you had to you had to light the fuse But the point is but the point is it's very important to encourage Whoever has the passion to bring the technology in to the community, right? If if your customers are passionately Asking for that type of solution you're going to support that kind of solution, right? And it's going to thrive if more than one customer bites it it's going to try The problem here is this happens in open source Nobody has passion for doing re-architecture and maintaining technical debt and this this is that's the challenge, right? That I think that big intent is inevitable. It's inevitable networking. It's inevitable and compute It's inevitable in storage. So I understand all of the reasons people are saying we shouldn't it's going to So real quick again, just reminder if anybody's got a question, please come up to the microphone Jay, you had plenty of time to walk to the microphone. No last-minute grenades Oh, we got someone really would like some oh, no, he's using the bathroom Go give us your name company and question test. Okay. I'm Hubbard Smith. I'm with Samsung My question the panel is Getting there, but you're flirting around the main issue the main issue with all of you guys is Quality you guys have engineered spent decades engineering emc net up. I used to work for net up red hat You built your business around quality. Why don't you address the quality question here? The quality of what? How do we as a community? engineer quality into our platform Engineer I mean open stack open stack. Yes Sorry beyond one project or in individual projects. Yes. Yeah, you know, that's yeah, right? So there's the I think the thing here about the delivering and developing quality it requires that I Actually believe the big tent Approach is necessary but not necessarily going to give us any better quality. That's the first thing Although although competition always does at the end It's a very long game that you play when you when you're in a competitive environment and expecting quality to come out the back end of it that the issue for me is the Governance of the projects needs to be much more rigorous and I really am a big believer and yes This is a democracy. Yes. This is a big tent. No, you don't get a vote I was quite serious about it I do believe that we need to have a bit more structure in terms of the way that the projects are organized managed and The mechanisms by which they conduct their business Having a very open and free mechanism for conducting your business You know using tools like IRC is one thing But then having some kind of collective consensus on which you can all agree to disagree or Disagree to agree, but at the end of the day, you're all going to move in the same direction Is another thing and I don't think we're seeing enough of that No, I think last year you even talked about how that kind of boiled us down to the lowest common denominator. Yeah, that's right We were in danger of delivering real quality for something that we already had which was LCD You know it literally was the lowest common denominator You wanted disk space and it came in blocks and that was all you got. That's all it did You know, I think where you're going is something that's come up before which is that the PTLs and not BDLs If I can pile in the acronyms there at the project technical leads They get you know look at Cinder Mike started saying no this session, right? There's not many PTLs who are brave enough to start saying to big vendors. No, you're bringing bad quality. You're breaking things It's not going to ship and that that's difficult because the PTLs right now. They're volunteers They are often working for vendors themselves. They're in an awkward position I mean this is you know if you look at Linux Linus and in free to language says no a lot And that's what made has made Linux such a high quality, you know in the upstream before even any downstream gets to it I think this is a challenge on OpenStack that you you know You need to say no more often but that's very difficult when you're trying to build a big tent and get community and get Adoption that there's a balance and maybe the balance isn't in the wrong way. I think when you reach a specific level I won't call it critical mass because there are things you can't buy time as one of them, right? So once you reach a specific critical mass you can get to the point where you start saying no to people because it's no It's a real thing, right? You say yes all the time when the project's beginning because you need to build it out But now we're reaching that point in OpenStack where we can tell people this is a bad move to make I'm not going to do it and life carries on tomorrow after you say no And I think that's where we are on that. So our own quality We'll get the quality in there. We can't buy time If need everything needs more time or at the point now where the most important word is going to be no We're not doing that. Well, and I think one of the big big points up to that is you look at what has happened in the last You know cinder cycle right what has happened in the last cinder cycle is first of all you had a lot more drivers again I mean it is up to I don't know 40 40 50. What's it up to? Yeah, it's I mean It's a lot of drivers right but at the same time that the the passing CI testing and pushing that actually back on the vendors Helps a lot with the quality, right? It's going to help a lot with the long-term That is still relatively new right and everyone's still trying to get through that gate But that is something that helps with the quality, but to your point earlier It is it is hard to motivate Developers that they may be paid for it, but the end of the day they they typically want to go after sexy not boring and Getting the quality at times is boring Having a ptl willing to say say no is maybe maybe boring at times and sometimes the quality is just the boring work unfortunately the moment this This gets this gets into enterprise which as we know is boring, right? But the thing is once you start having enterprise customers they have to live with this all the time So this this I'm going to go off and I'm going to conquer new worlds that goes out the window because someone saying to you I got upgrades coming. I need things going on over here. What what are you doing for me? You're not doing anything for me then it becomes an issue So I think the the rocket Putting the rocket boost the rocket booster on top of the car and shooting it down the runway That's been all nice and fun But now it's a case of we need to put this thing in the road And you need to put the kids in the back and run to the shops and that's where we're getting in the life So one one comment. I want to make is that you know, I was just Speaking with one of our engineers yesterday. He mentioned about the new QA initiative that has actually started Which is going to do the integration test and they're seeking for volunteers to help Test the interoperability between multiple components. So I see You know, there is more and more Investment being made to enhance the quality, you know, I think it's a good time I think it's just a QA initiative that started You know There are lots of meeting hope and happening this week in the design summits around who's going to be contributing How are we going to improve the overall quality of open stack as opposed to individual components? We got another question here Scop right. Well the emcee Wanted to ask you mentioned neutron. You're starting to allude to What I think they did is a decomposition of drivers and plugins out of upstream out of the main neutron and give that Most wholly to the vendors, right? That's how they manage their beginning to manage now this Big swath of drivers that they have to deal with and I'm wondering your thoughts about not necessarily big tent and competing with sender but Putting smaller shims in to sender and allowing more people to compete sort of within the driver space and have feature rich drivers That are extensible and can do more application logic that's driven by the value. I'm going to have Scott tag me out here So I'm just gonna go up and high-five. Yeah, so my question would be back to you What are the consumers of this one? Because there's that you know having having some neat and cool features and the ability to shim things if there's no demand for the stuff That we're shimming or the stuff that we're enabling by shimming then Where's the advantage? So I you know, I need it's hard to talk about demand for things that aren't available So once we make feature more feature Function available and possible underneath sender then perhaps we can start talking about demand But it's unfair to say where's the demand for something that can't be done today. Yeah, but Effectively we're abstracting it a layer so that there's already a demand there in place out in You know consumer land for certain features that storage has so if we you know There's if we're not meeting that demand and that's one thing, but I'm wondering what kind of demand you're talking about Or non-demand. I mean I need an example a replication oh Boy, I need to create a I need to create a sender. Can we talk about this in Paris? I don't know. Actually replication is a really good pick because if there's one thing I've learned this week is that we still can't agree what anybody means by replicating So give me the control to be able to do replications That's pretty much like if there's if there's a nail to be hammered with shims. It's replication. Yeah Yeah, I mean this is that this the stuff that needs to have the quality improved is what we have now, right? I think we can There's the essential stuff which has gone in there I mean that you know we've back up migration replication for whatever definition it is That's what that is what the demand is right now, and it doesn't work very well now if that requires Re-architecture if it requires if it's just a QA issue whatever it is, and that's what needs to be addressed right now I did you know, I don't think we want to be re-architecting just for the sake of it It has to be driven of it. I think it's it is quality. It's just getting the things to work now That's what needs to be done That's great replication was a really cruel one you couldn't have picked the worst one But but it is a great example Pick up on that a bit Alex, you know we talked about you know what we've what changes that we've seen in the last six months Let's talk about replication. So Okay, I'm not speaking from experience here I'm only speaking from a sort that osmotic feel you get by talking to people who know things more things than I do That's quite a lot people by the way apart from one person in this room right How did I become the heavy all of a sudden I'm making this yeah The world is chasing you out of the last summit all right the wheel turns my friend apparently so the thing about replication is we We've got some fundamental disagreements about what we mean by replication what it is to do replication and how replication is done There are several ways of doing it in the industry and the word is unfortunately become a catch-all for a set of procedures and processes That we need to further break down we need to actually kill the word replication talk about other kinds of stuff We need other words for it because there are about I know of at least five different things that people would class as Replication that are all different. They fundamentally work differently, and they're not abstractible in the same way So, you know, that's a that's a that's a big issue replication is a good example of Where where the markets going right now the reason this stuff is becoming important It's because people are running this and of course the first thing they see is crap I need backups I need DR I need you know This is this has got to go into multiple places here I mean it's like it's a kind of obvious thing when you think one step ahead of the development But you know that the challenge here again This is the tension between things that should be an architect if a cloud where it's this is not enterprise backup It's your application handles this that's the cloud way. That's the Amazon way This is what it started off as but now you've got these enterprise demands come in going I'm an admin. I can't rely on my users to do that. I gotta back the stuff up for them I can't trust them. So I think it's a tension between the perhaps the original design goals of you know pure play sort of cloud Amazon style architecture and now what enterprises the demanding which is now what we're used to and I think that's where the tension is is Producing bad, you know bad quality outcomes and I think previously replication you talk about replication people say to you Especially around these halls. Well, if I've our sink in one side and our sink in the other side No, no But yeah, we already that there's a backup project from what I understand rolling along replication is the next big frontier and it's It's it's a huge mountain to claim. So there's gonna be a lot of work about this in Paris Isn't this way? We don't have to rely on open stack to provide every single feature That's where the vendors should be contributing in their own value add The manageability aspect to that. It's not just the underlying technology. So there is It's not the top note of creating a replication service from scratch It is a case of how do I kick all this off or all the individuals who might be using the open stack thing But it's also it's like who's it for is it is it the user level is it the admin level? There's different again from profiles of people and how they're using it it varies and this this is why it sits It's a hold on but again, it has to be addressed I'd rather we just sat and focused on this for the next 12 months and didn't do any other sort of sexy features You know if there's these are these are the kind of things which we've got to just nail in the head And we can't be discussing them 12 months from now If we're doing that for 12 months. I'm retiring. So I Kind of I kind of think like, you know, I Agree with Greg and Greg is the smart guy for this topic now But Basically, I think you know, there are a lot of lower level problems that open stack still needs to solve I would rather leave replication type of a solution for vendors to solve right now on top of the you know Lower level core, you know components of open stack There's so many things when you look at the replication, you know mark very well as well Every vendor solution is different And it's different because the customers deployments are different How you do a replication underlying the technology, you know, some people have patterns Some people have some unique technologies that they they you know Hitachi for instance looks at how do we speed up the data transfer as part of that? Replication not just do the basic replication because we have that technology. Similarly. I'm sure you see has something solid fire something I think it's a very complicated problem to solve I would rather have open stack focus on getting the bare-bone core components There's a difference here. We yeah, the vendors are gonna address this but as it's it's um, you know, it's what Mark said It's the it's the orchestration. It's all the control playing stuff, you know, and as you can't separate one from the other This is the inherent problem, right? You get we bring all these great underlying things that we have to express them together in a unified way Which works for the end user and that's proving hard, you know, that's that's that's difficult So guys we have to cut you off. We got two minutes left. I want to get No, no, no, I wanted to get some last thoughts. We'll go right down the road Alex The weather's been lovely Quick show of hands how many of you are going to Tokyo? Okay, may not see you there. Oh Um, yes, I would agree Remember every time you put a workload in a public cloud storage vendor fire someone All right, we got your I love you man I know I can't follow follow with that one, you know, I live in Seattle. The weather is nice here So it's in Seattle come over to Seattle when you get chance. No I we've been bitching complaining a lot, but I I also want to say thank you because it's just it's it's also a great Community to be part of it. There is a lot of excitement. We may get frustrated with it But it's I want to thank all the developers to contribute to the cycle and they're gonna contribute to the Next one here. I don't want I don't want to feel there's a lack of love for the what is it's an amazing project, right? And it's easy to pick and that's what we're paid to do, but it's well done everyone who contributed Well, yeah, you took mine a bit and yeah, I just wanted to say also. Thank you I mean it is a completely thankless job without a doubt It is amazing to me at the end of the day though to to watch everything evolve and Also really be able to openly talk about the points replication quality all these things and Raise them up to the community and watch them get solved over time is is truly amazing that I really like that portion of this Mark take us home the Bottom line and it's all here. We do great when we do complain I heard yesterday of people are saying that networking or is difficult storage is quite easy because there are only three types of abstractions Nonsense right and everyone up in this stage everyone up in the stage anyone involved in the silver The cinder project and Manila and everything else is going on wants to make this stuff better, right? We wake up in the morning and when we get up here and we complain about this that the other the fact that matters that when we leave This room. It's right. How do we fix those? That's what we do. That's why we're here and we thank you for your time That's great. Thank you very much and thank you for joining us. I really appreciate it