 Live from San Diego, California, it's theCUBE. Covering KubeCon and CloudNativeCon. Brought to you by Red Hat, the CloudNative Computing Foundation and its ecosystem partners. Welcome to the third day of wall-to-wall coverage here at KubeCon CloudNativeCon 2019 in San Diego. I am your host for the three days of coverage. Stu Miniman joining me this morning is Justin Warren and happy to welcome back to the program Aaron Boyd who's a senior principal software engineer at Red Hat. Aaron, thanks so much for joining us. Thanks for having me. All right, so we had a chance to catch up in Barcelona on theCUBE there. Storage is definitely one of the faster moving areas of this ecosystem over the last two years. Why don't we start with really the event? So, as I said, we're in day three, but day zero, there were a whole lot of things. We had some of your peers at Red Hat talked about OpenShift Commons, but storage, my understanding, had a couple of things going on. Why don't you share with our audience a little bit of that? Sure, so we had a SIG face-to-face for Kubernetes. It was probably one of the best attended. We had to cap the number of attendees, so about 60 different people came to talk about the future of Kubernetes in storage and what we need to be doing to meet our customers' needs. In conjunction with that, there was a parallel session called CNS Days, which is Container-Native Storage Days. That event is very customer-focused, so I really enjoyed bouncing between the two of them to go from the hypothetical programming architecture view straight to what customers in the enterprise are looking at and doing and what their real needs are. Yeah, so from that SIG, can you actually share a little bit of where we are, where some of the requests are? We know storage is never one way to fix it. There's been some debates, there's a couple different ways to do. I mean, traditional storage, you got block file and object, cloud storage, there are more options in cloud storage today than there was if I was to configure a server or buy a storage array in my own data centers. So where are we, what are those asks? What's on the roadmap there? Right, so I think for the past five years, we've been really focused on being mindful of what APIs are common across all the vendors. I think we want to ensure that we're not excluding any vendors from being part of this ecosystem. And so with that, we've created the basis of things like persistent volumes, persistent volumes claim, storage classes to automate that, storage quotas to be able to have management and control over it. So I think now we're looking to the next evolution of people as the models maturing and people are actually running stateful applications on Kubernetes, we need to be addressing their needs. So things like snapshotting, eventually volume cloning, which has just gone in and migrating. All these type of things that exist within the data plane are going to be the next evolution of things we look at in the SIG. So one criticism that's been mentioned about Kubernetes a few times, and one that's a bit complicated, but also it didn't really deal that well with stateful sets. Stateful data management has always been, it's been a little bit lagging. That seems to have pretty much been sorted out now as you mentioned, there's a lot more work been done on storage operators. But you're talking about some of these more, these data management features that operators from other paradigms are kind of used to being there. When you're thinking about moving workloads to Kubernetes or in putting in new workloads on Kubernetes, if you're unsure about, well, will I be able to operate this in the same way that I did things before? How do you think people should be thinking about those kind of data services in Kubernetes? So I think it's great that you mentioned operators because that was one of the key things when Rope came into the landscape to be able to lower the complexity of taking something that requires physical storage and compute, geography, node selection, all those things that help people who were used to just the cloud model. I create a PVC, it's a request for storage, Amazon magically fulfills it, I don't know what's backing it, to be able to take these more complex storage systems and deploy them within the ecosystem. It also does a good job of supporting our brownfield customers because not every customer that's coming to Kubernetes is green. So it's important that we understand that some customers want to keep their data on-prem, maybe burst to the cloud to leverage those services but then keep their data close to home so operators help facilitate that. Yeah, Erin, I hesitate a little bit to ask this but I'm wondering if you could do a little compare contrast for us for what the industry had done back in open stack days. When I looked at storage, it was every traditional storage company certified their environment for open stack on a storage standpoint, it feels like a different story to me when I hear about the ecosystem of operators in open stack. So I know you know this space so maybe you can give us a little bit of what we learned in the past, what's similar, what's different. Right, well I think one of the benefits is we have a lot of the same key players. As you may know, OpenShift has pivoted from Gluster to Seth, Seth being the major backer of OpenStack. So we're able to take some of that technical debt and learn our lessons from things we could improve and apply those things within Kubernetes. I just think that it's a little slower migration because we have, in OpenStack we had, like you said, we had certification, you know, there were different drivers and we're trying to learn from maybe, I wouldn't even call those mistakes but how can we better automate this? What can we do from an operational perspective to make it easier? Well I think it's one of the, it felt like we were kind of taking some older models and well I'm testing it, I'm adding it. The ecosystem for operators here is different. Many of these, we're talking very much software driven solutions. It's built for container architectures so it's understandable that it might take a little bit longer because it's a different paradigm. Right, well and I think, you know, the certification kind of, it wasn't an inhibitor but it certainly took a lot of time and I think our take was on, you know, we used to have all the storage providers be entry providers within Kubernetes and with CSI we have since, you know, started to redo the plugins in the sidecars and move that out of core. So then the certification kind of falls, you know, outside of that instead of being tightly, more tightly wound into the platform and I think it will allow us to have a lot more flexibility. Instead of waiting on each release, vendors can create operators, certify them themselves, have them in their own CSI driver and move it, you know, the pace that they need to move. So how do you balance that need for Kubernetes to kind of be a common operating platform that people can build on with each vendor's desire to provide their own unique capabilities that they think that they do particularly well? That's why they charge the money that they do because they think that theirs is the best storage ever. How do you balance that sort of tension between the need for a standard platform to make it interoperable, but still allowing the flexibility for people to have their own kind of innovation in there? So when we created the storage class, for instance, to be able to create kind of a service level over storage to be able to provide the provisioner that we're going to use, we made the specification of that section completely opaque and what that allowed us to do is that when vendors wrote their provisioners and now their CSI drivers allowed them to feed in different attributes of the storage that they want to leverage that don't necessarily have to be in core Kubernetes. So it provided a huge amount of flexibility on that. The other side of that, though, is the feedback we get from real users is I need backup and recovery and I need DR and I need that across the platform. So I really think as we look to scale this out, we have to be looking at the commonalities between all storage and bringing those APIs into Kubernetes. One of the things I've really liked to see in this ecosystem over the last year or so and really highlighted this show, we're talking a lot more about workloads and applications and how those, what works today and where we're growing. Can you speak a little bit from your world as to kind of where we are, what's working great, what customers are deploying and a little bit the roadmap of where we still need to go? Sure, I think workloads are key. I mean, I think that we have to focus on the actual end-to-end delivery of that and so we have to figure out a way that we can make the data more agile and create interfaces to really enable that because it's very unlikely that an enterprise company is going to rely on OneCloud or stay with OneCloud or want their data in OneCloud, they're going to want to have the flexibility to leverage that. So as we enable those workloads, some are very complex. We've started with, hey, I just want to containerize my application and get it running. Now I want to have some sort of state which is persistent storage and now I want to be able to scale that out across any number of clusters. That's where the workloads become really important and long-term where we need policy to automate that. My pod goes down, I restart it. It needs to know that because of maybe the data that that workload's producing, it can only stay in this geographical region, so. Yeah, when we talk about multi-cloud, you mentioned data protection. Data protection is something I need to do across the board. Security is something I need to do across the board. My automation needs to take all that into account. How's Red Hat helping customers get their arms around that challenge? Yeah, so I think Red Hat really does take a holistic view in making sure that we provide a very consistent, secure platform. I think that's one of the things that you see when you come on to OpenShift, for instance, or OKR, that you're seeing security tightened a little bit more to ensure that you're running in the best possible way that you can to protect your data. And then the use of RookSeph, for instance, Seph provides that universal backplane where if you're going to have encryption or anything like that, you know it's going to be the same across that. And it sounds like there's an opportunity here for people new to Kubernetes who have been doing things in a previous way. There's a little bit of reticence from this community to understand enterprise. Well, actually you're kind of doing it wrong. It's slow and inflexible. There's actually a lot of lessons that we've learned in enterprise, particularly around these workloads, having security, having backup and DR. In the keynote this morning, there was a lot of discussion about the security that is either in Kubernetes and some parts it's kind of lacking. I think there's a lot that both of these communities can learn from each other. So I'm seeing a lot of moves of late to be a little bit more welcoming to some people who are coming to Kubernetes from other ecosystems, to be able to bring the ideas that they have, that you've already learned these lessons before. We can take some of that knowledge and bring it into Kubernetes to help us to do that better. Do you see Red Hat bringing a lot of that experience in its work? Red Hat has been around for quite some time now, so you've done a lot of this already. Are you bringing all of that knowledge into Kubernetes and sharing it with the ecosystem? Absolutely, and just like Stu pointed out, OpenStack was a big part of our evolution and in security with NREL, and I think we absolutely should take those lessons learned and look to how we do protect our customers' data and make sure that the platform, Kubernetes itself, as we evolve OpenShift, can provide that in ways that we can certify that. Aaron, you're meeting with a lot of customers, you were talking about the day zero thing. What's top of mind for your customers? You know, we talk about that Kubernetes has crossed the chasm, but to get the vast majority, there's still lots of work to do. We need to, as an industry, make things simpler. What's working well and what are some of the challenges from the customers that you talk to? So I think, you know, if you walk in across the hall and you see how many vendors are there, it's trying to get a handle on what I should even be doing. And as the co-lead of the CNCF Storage SIG, I think that's one of the initiatives that we take very seriously. So in addition to a storage white paper, we've been working on use cases that define when should I use a data store? When should I use object? Why would I want to use file? And then really taking these real-world examples, creating use cases and actual implementations so someone can, oh, that's similar to my workload. Here are some tools to accelerate understanding how to get that set up and also creating kind of those guardrails from an architectural standpoint. Like, you don't want to go down this path. That's not right for your workload. So we're hoping to at least provide an education around containerized storage that'll help customers. Yeah, I'm just curious. You know, I think back 10 years ago, I was working for a large storage company. We were having some of these same conversations. So is it very different now in the containerized, multi-cloud world or are some of the basic decision tree discussions around block file and object and application the same as we might have been having a decade ago? I think we're starting to just touch on those. And I'm glad that you brought up object. That was one of the things I talked about in Barcelona and we actually talked about it, the face-to-face. To me, it's kind of the missing piece of storage today in Kubernetes. And I think we're finally starting to see that more customers are asking for that and realizing that's an important workload to be able to support at its core. So I think, yes, we're having the same conversations again but certainly in a different context. Yeah, I mean, back in the day, it was the futurist object but we don't know how we get there. If you look behind the scenes in most public clouds, objects running a lot of what's there. Absolutely. So, all right, Erin, I want to give you the final word. KubeCon 2019, from that storage perspective, what should people watching take away? That we're only beginning with storage. Yeah, we still have a lot of work to do but I think it's a wonderful community and vibrant and I think there'll be a lot of changes in the coming years. All right, well, definitely a vibrant ecosystem. Erin, thank you so much for all the updates. We'll be back with more coverage here for Justin Warren, I'm Stu Miniman. Thank you for watching theCUBE.