 Okay, welcome back to SiliconANGLE and Wikibon's theCUBE, our flagship program. We go out to the events and extract the signal from the noise. This is our OpenStack Enterprise events. SiliconANGLE, Wikibon in conjunction with SolidFire, live broadcast here. Enterprise is hot, OpenStack is hot, OpenCompute is hot. Everything open is hot, but more importantly, the cloud. Dave, I'm with Dave Vellante, my co-host. Our next guest is Rodney Peck, OpenStack Storage Architect with PayPal. PayPal's in the news lately. Obviously, that's like the hot product within eBay and a lot of the news, but we want to talk more about the tech going on. eBay, huge hyperscale environment. You guys are not foreign to the notion of a lot of stuff going on in terms of data, compute, networking challenges. So I got to ask you the first question. A lot of commentary in the crowd chat about the storage architecture. Block, storage, object store. How do you handle all this with OpenStack? What do people need to worry about, about OpenStack Storage? How do you use OpenStack Storage? Well, one of the big things with OpenStack is it's related more to Amazon EC2 and object storage than to more traditional storage like POSIX NFS. So a lot of the stuff we're running into is educating developers about how to use Amazon's features and OpenStack's features more effectively. It hasn't been that difficult. Once the architects find out the features, they're quick on board. The agility and the speed of OpenStack being able to create new volumes and attach them makes any sort of downsides of change really not that big an issue. In the past, to use storage, you'd have to work with a storage vendor and have volumes allocated and targets created and it's a much more physical thing, probably creating tickets to your site. But with OpenStack, it's just click, click, or API calls. So those are the sort of changes. So maybe you could explain a little bit. So you're using a little bit of AWS, a little bit of OpenStack, a little bit of other stuff. Can you just sort of describe that environment for us a little bit? Just paint a picture. As far as Amazon and... Yeah, you're mentioning Amazon. So you're using some Amazon for what? Test and Dev or? Our M&A people, the companies we purchase, a lot of them have piloted their things in Amazon. So they have something that's working with S3 and working with AWS and Elastic Block Store. So you bring them into an OpenStack environment? Right, the OpenStack lets you set up an environment where you can create almost the exact same thing. Maybe even use the OpenStack EC2 API but move towards the OpenStack API for additional features. And so presumably they're using a lot of S3 and maybe even some EBS. So the analog of Elastic Block Storage in OpenStack is Cinder. What's the sort of status? What's your experience with that? How's it going? Cinder, Cinder is going very well. Cinder and SolidFire is really tight. So you use SolidFire underneath Cinder? We have many different vendors. But yeah, we use SolidFire for some of our Flash Storage. We're looking at Cep for low-cost block storage. We have NetApps, we have EMC, we have Atachi, we're a very large company. We've got one of all, one of everything. A lot of legacy hardware. Like a lot of cloud service providers actually. And really that might be something to talk about is you don't have to throw away all your hardware just because you're moving to this OpenStack. A lot of these companies have developed Cinder drivers. Some of them aren't as feature rich as others. Like some provide fast cloning to create volume. Some you make a new volume and you have to copy all the data by hand. So some are more effective. But if you've already purchased a piece of hardware and have a lease on it for several years, you don't have to throw it away. You can use it with Cinder. And for some people that's a better solution than by now. So in general, you're going to, I would presume, increasingly demand high quality drivers and connections from legacy storage into whether it's Swift or Cinder or Cep. Or Manila, the NFS project. So yeah, anybody who's working with eBay or PayPal now, we make that a clear point that these are the APIs we're working with. Where anybody who works with Cinder or Neutron or any of these other APIs, we're willing to talk to them. And it could be a big opportunity for a small company. So John was just talking about, we were just at the OCP summit yesterday and with theCUBE and today for a little bit. How are you addressing different hardware configurations for different workloads? How closely are you looking at that? Are you thinking about OCP? I wonder if you could talk about that a little bit. Yeah, OCP, the biggest problem we would have when using that for storage is, who do you call when there's a problem? What, there's no vendor to go that's going to dispatch somebody to help you fix this. To say if you're a million dollar account. So PayPal, we're very high reliability. We're very conservative in redundancy and things like that. We'll spend extra money to make sure that we don't end up in a situation where everybody's up for two nights with an outage. There's not a lot of outages of PayPal you read about in the paper. So Flash, obviously big disruption in the storage business. For years the spinning disk has been the bottleneck. We all kind of know that Flash comes along. You got this persistent medium as either a memory extension or stuff into existing boxes. So many different use cases across the spectrum. I wonder if we could talk about Flash and sort of eBay's philosophy on Flash. What are your personal opinions? What you're seeing as a storage expert? That storage used to be boring. Used to be how many gigabytes do you need? And that's that. We used to call it storage. Yeah, so now with all the new things coming out iSCSI 10 gig networking is cheap. Flash, suddenly storage is really complex and the best solution isn't really obvious. And what's happening is that the price performance curves are changing. I think right now it's a little early to be buying all Flash unless you can afford it. But if you go out and buy really large drives next year they're boat anchors. So I think in two years it'll be really, really obvious what to do right now. It really depends on your use case, on your application. And you have to analyze that if you're going to buy a lot of something. On the other hand, things are relatively cheap so you can kind of get a common denominator. A lot of vendors are working on spinning disks, prices at Flash speeds. That doesn't fit our case because our data centers are full. So we don't have any room. We need the Flash speed. We don't have room to be putting in spinning disks. So we want as much Flash in the rack as we can. You talked before, Ron. Different sort of problems. Yeah, sorry, okay. So you talked before about all the different legacy platforms that you have and the nice thing about OpenStack is you can actually leverage those assets. You don't have to throw them out. Now of course you see all this, you hear all this discussion about software defined. And you have a number of companies that are talking about separating the control plane from the data plane. And essentially that data plane, I guess that backend storage gets commoditized and all the value goes to sort of the control piece. What do you want your control plane to be? Is that OpenStack? Is that 100% right? Yeah, in fact the vendors a lot of times will have a big GUI or some sort of, or the worst is when they have a Windows appliance that you have to run to manage your storage system and it has all these point and click things. We don't want any of that. We want to just control it from OpenStack APIs. Otherwise you can't script it. You want to consolidate all those different points of control so you can automate, presumably. Yeah, single pane of glass reporting, single pane of control. But I feel like the single pane of glass I feel is like it's a single version of the truth. Do you see the light at the end of the tunnel from a storage perspective on the single pane of glass? Yes, I mean it's one of those project bullet points that's like next quarter we're going to do that. And so we try, but every, yes. But you know what I'm saying? You've been hearing that from vendor pitches for decades, haven't you? Everybody talks about single pane of glass. The higher you go up in the management level, the more important that is. Right, okay, but so OpenStack is you think can deliver on that promise? Sure, you know, what's kind of how it's like a snowball. It's sort of as you have an API for networking, you have an API for storage and an API for compute. If you have something else, say backup, why wouldn't you have a similar, you know, so it's sort of just if I can do this and this and this, it just provides a framework and a model. Rodney, if you had to look at thinking about your life before OpenStack and your life after OpenStack, and you had to, let's say you had to build a little business case, you know, after the fact, you know, hindsight being 2020. What would that business case look like? You know, roughly, and even in, you know, we've got field terms, percentage terms, however you want to communicate it. How much money did you save? How much, you know, faster were you able to deploy things? I wonder if you could just share any metrics you can share or any subjective comments around that. Sure, before I was with PayPal, I was at Yahoo, and I was the operations architect for Yahoo Mail. So it's like an enormous mail system, and we at Yahoo are deploying, you know, thousands of machines a quarter, and we would find when we shut down a data center, 10 to 20% of the machines in the data center were idle for the entire time they were in the data center. It's crazy, because you have to go to David Filo and justify them every week. It's like, we need 12, we'll give you eight. Not only underutilized, you're saying idle. Idle, like the machines were installed by the operations team. Nobody ever logged in, nobody ever installed any packages. We only found out when we shut them down to close the colo. So then, of course, the management's like, we can't continue to do this. We need to find out, so they got strict during the allocations. You need 12 machines, we'll give you eight, because 20% of what you asked for, you're not going to use, and that's what they did. And somehow that didn't work either. But OpenStack and Virtualization gives you this way to say you need 100 machines, okay, we'll get you that capacity. If you only use 80 of them, that other 20 is available for another project, and you don't have to keep track of it. So we literally save millions of dollars a quarter, just an unused allocation of resources. And I think that's the big business thing for the senior level of management. If we could save that 10, 20% loss, we get more value out of the data center, and that's multi-million dollar, because the power and consumption is huge. Last question I have for you. Any big shortcomings in storage that you see within the OpenStack framework that you'd like to see addressed by the community? Yeah, there's always new features coming out. So things like backup, things like inter-vendor, inter-backend migration, to say move a volume from a SEF to a solid fire or something like that, those sort of things. There's features, but right now, creative volume, user volume, the delete, all that management sort of stuff, it's functional as it is. So we've been really, really happy with it all in all. Excellent. Okay, Rodney, thanks for coming on. We really appreciate it. I know you guys are really short in time. No, no, we'd like to get, but I want to get to Dave, wanted to have some specific questions I had some, but we couldn't wait to laugh to the camera. This is theCUBE, we'll be right back. We're going to go into rapid fire. We want to get the toughest questions we can find. If you see him, go to crowdchat.net slash OE forum. That's our public open timeline across LinkedIn, Facebook, and Twitter. That's a public chat on the hashtag OE forum. That's a crowd chat. Put your questions there. We'll be right back with our next guest. Have a short break.