 Okay we're back, this is theCUBE and I'm Dave Vellante of Wikibon.org and with my co-host David Floyer who's the CTO of Wikibon.org, and we're digging into backup. We've been talking at Wikibon for quite some time, backup needs to evolve, it needs to change, we've often said backup is broken and data protection as a service is really the fix. Vijay Banga is here, he is a technical fellow within IT at Federal Express, Vijay welcome to theCUBE. Thank you, pleased to be here. Yes, nice to be here, we talked a little bit on the phone and know a little bit about your story but we want to share it with our audience, so you heard some of our narrative here about, do you feel sometimes like your backup is broken over the last ten years and has evolved? Well, it was, we've come a long way, we've had challenges with backup but I think we've come a long way from where we were to where we are. Even though we used Data Domain ten years back in our development and test environment, I think it's only the past few years that we've been successful with the advent of TenGig in a data center where we've been able to use it for our database and OS and we've been able to replace most of the clarions that we had on the floor with Data Domain, so we've had a good story for the past few years. Yeah, so we're going to dig into that a little bit, so before we get too deep into it, I want to set it up a little bit. So you're a fellow, a technical fellow, that's a great title, we love talking to the fellows, so tell us a little bit about your role and what you do at FedEx. Okay, I'm an enterprise architect and I work for the VP of IT and my role is to help guide, take products from development, test to production for various areas that are required. So last few years I've been involved with the migration of our legacy data center, so I also have multiple hats, so one of the roles was from a storage reference architecture to try and get the methodologies for migration of our legacy data center. So we did inventory for all our storage and then figure out ways to migrate all that. Of course, Data Domain paid a big help in that as well, but currently what I'm doing now is trying to help work on the DR infrastructure, provide DR as a service. We've been doing DR for several years, but at a larger scale that's what I'm trying to do now. So talk about the infrastructure that you're involved in, FedEx is a huge company, obviously, we can talk for an hour about what this infrastructure looks like, but specifically the areas of your responsibility, what's that look like, what kind of applications are you running, just paint a picture for us. So we have, you know, actually FedEx is a great example of a whole homegrown application, it's all tied together, it's a spiderweb of applications and at the scale that you don't find products out in the market, so it was all developed in-house and we are distributed across the globe, but we have a major few data centers in the US itself. So we deal with primarily all the infrastructure within the data center. Prior to a couple of years back, we used to do this very much on a project by project basis, but we moved from that to do everything as an infrastructure. So infrastructure as a service, infrastructure model is the way we've gone to. Okay, so when you got there, so how did you get there, what was it like before and during the transition and let's talk about what it's like now, what's the state you were coming from? Well, on the storage side, you know, it's an easy example to give. We had distributed islands of sand, so we were never able to capitalize on all our assets, so we had assets that were dispersed and we were not able to fully utilize that. So when we went to a legacy data center, we moved from a legacy data center to our modern data center, we had a green field opportunity and we wanted, we actually had reduced floor, we had constraints for power and cooling, so we really had to figure this out, design this and then start implementing it. So we had to think through what it means to do everything infrastructure-based, to be able to use our assets across the floors on a data center so any host can get to any storage. So we did that on the sand side and we are working on that towards the IP side as well. So in our conversation that you told me, you've got Oracle RMAN going, you're using SRDF, Time Finder, you've got Data Domain, you've got Recover Point installed. How do those pieces all fit together to form a backup strategy? Okay, so on the backup side, we used Data Domain primarily, both for our ROAS backups with NetBackup and on the database side, RMAN backups to Data Domain. So once we got on the 10-gig infrastructure, we get out of the way by providing the infrastructure to the DBAs so that they can have the Data Domain infrastructure and they can do the backups, they figure out the scheduling part of it and then they can have at it on the Data Domain. Okay. Besides that, we've also done things from a DR perspective where we used SRDF and then we've used Recover Point as well for various aspects, for migration, for DR and off-site backups, for a larger databases. So, you know, based on the service level that is needed, we have the solution that we provide for it. So talk a little bit more about that. What makes it backup or data protection as a service? How do you sort of define that and how do you turn that back to the users and the businesses? Right. So prior to this, it was really a challenge for figuring out on a capacity plan to figure out how do we get the data domains because we were doing this spread out on islands of sand but once we got the data domains, the way we defined and designed it is we have two security zones, back office and customer facing. So we provided the data domain infrastructure, we provided the 10-gig interface for them to come to the backups and then we have, we turn over to the DBAs, you know, we work with the development and design DBAs so they understand and turn over to the production area as well and then they have the data domain infrastructure and then they're able to do a self-serve. They do the backups, they manage everything but we assist from providing the infrastructure. They can provision on their own and do you do chargebacks or no? No, we don't do chargebacks yet. Do you want to do chargebacks or is it, I mean, a lot of customers that I talked to don't want for their private clouds, they don't want to do chargebacks. It's a political nightmare, what's the point? It is, on a smaller scale, we are looking at it, just like you said, the political nightmare but I think if we have to scale this out and figure out the right service level for the right area in this time of economy, it makes sense to at least show the business as to what it costs. It might, you know, and if we provide a different level of service, they can understand what is going to cost the business. Yeah, or even a showback. Showback. Yeah, it's not a chargeback, so really it's really a showback that is what we're focusing on. So that I think is very worthwhile. Yeah, absolutely. And then you can have some healthy debates about what am I really getting for this. If it's not really hard dollars, because they're paying for it anyway. Yeah, exactly. So okay, and then where does quota management fit into this whole thing? How do you deal with that? So, prior to this current version of the data domain OS, it was not available. So, you know, the challenge we had is we provide the infrastructure and the way we had designed data domain was really to use for Ironman backups, for OS backups. But if you provide the infrastructure, it's available for the DBAs and they could use it for other areas. So that's when it became a really a challenge as to, you know, why do we have these concerns and issues that we see? And so what we did is try and address that by, you know, having more discussion with them and finding out what is the real goal and the usage for data domain versus what other scratch areas they need. So, where we are going with that is trying to move to DD Boost so that we can give on the database side. We've already done that on the OS net backup side. But on the database side, we'll be able to give the data domain as the infrastructure for backups. At the same time, provide NASA solutions with Icelon so that they will have that space with quotas on there. So that's where we're heading. So David Flair, explain briefly what DD Boost is for our audience. So DD Boost is a speed up, isn't it? Yes, exactly. And I'm interested in it. Previously, it was only available for the OS and for FiberChannel. So you've taken the new version of you, which is available for FiberChannel or is that your... Well, no, on the database side. So we use the DD Boost as on the client on the net backup side. But now it's available on the database side as well. So there'll be the agent on the database side. I'll be able to do that work for you and then send less data to go to the... Less data to go on to the data domain. So it really helps us scale out our backups and help utilize our network bottlenecks that we had in the infrastructure. Right. Are you combining that? Do you use ARM on in another next way? Or do you just keep a simple... No, it's really simple. ARM and with DD Boost. So how much do you think that's reduced down your amount of time that it takes? Well, I think we've just done the POC for it. So we are headed down that path. So we will find out in a couple of months. But I think the key that we're seeing is the gain from a network bandwidth usage. So it allows more backups to be scheduled so that it's not using the network hogs, because there's still other resource constraints in the environment. So we're trying to reduce them along the way. So on the backup time perspective, we know it's not going to reduce that much, but it's really going to help reduce the constraints in the environment. Right. Are you using any sort of snapshot technology to take a point in time and then... So we are using snapshots on the data domain on the remote side to make sure that we don't... Because we have data domain that's set up with replication, so we have used that as our off-site backups. But then that's how we also used it for migration and other reasons. But we are using snapshots, timefinder copies for some other solutions with SRDF as well. And how do you deal with access control? Did you talk about that a little bit? Yeah, so with the newer version, we are actually connecting that to Active Directory so that the users will have access to whatever we want to control them with. So that's how we're going to do access control. Paviji, you mentioned you improved the concept with this, but I still want to talk about the business impact. What's the expectation? How did you develop the business case to move in this direction of data protection as a service? Just with the DDPoost or in general? In general. Oh, so in general, I think the drive to the newer data center drove us because we had, and when we did the inventory of the storage, we had tier one storage, tier two storage, which was being used primarily for backups. And we didn't have a great solution from that perspective, so instead of having dispersed islands, it was an easy case to show that introducing data domain as an infrastructure, we sweep the floor with all the other tier two storage, and at the same time, we can do much more with that. Yeah, so you were underutilizing assets, or utilizing assets in a way that wasn't productive and efficient. Correct, yes. So it's a hardcore TCO saving? Yes, exactly, yes. Okay, yeah. So do you have, looking ahead now. Yes. Where else do you think you're going? Do you've succeeded, or do you think you'll succeed in this particular part of it? Yeah. Where else do you think you're going to take, do you want to be able to use this on the archiving side as well, or do you want to be able to extract some data from that, where do you think you need to go? We did look at the data domain from the archive side as well, and I think we have internal challenges yet to figure out, because sometimes the retention that we have is much longer, and on the wrong tier of storage. So we're going to try and see if we can use the data domain archiver itself, and use that for our tier two, tier three storage as well. That'll reduce the footprint on the tier one storage. I wonder if I can come back to the business case. So hardcore TCO, do you have any metrics you can share, or any sort of percent change that you can share? Hard numbers, and it's not hard numbers, even sort of gut feels as to how much more efficient you are as a result of this. Sure, so I think I can shed some light in just numbers of backups. Traditionally, we were restricted by only up to 800 gigabytes that we could do, because in the one gig environment, we were only able to scale up to that, but... So your window just wouldn't allow you to? Window would not, and we had 24 hour backups going on, but now with the data domain and with on the 10 gig infrastructure, we have scaled up to at least 3.3 terabytes in our four hour SLA. We can do the restores, and then we can play the logs. So in the six hour time that we have, we've scaled up to 3.3 terabytes, but we know we are trying to take it forward so that we can go even much further to five or six terabytes that we can do in a timeframe. And the experience on the restores? Talk about that a little bit. Okay, so challenges are not normally on the backup side, but it's on the restore, but with the ease on the data domain side, the DBAs love our man, and it's really an ease for them. We provide the infrastructure and get out of the way, so we've done a lot of testing, and when they need it in the environment, they have the flexibility and they have the infrastructure. So we're out of time, but I want to, last word, advice to your practitioner peers. Somebody who wants to get on this journey, what would you suggest that they do, that they don't do? What kind of advice would you give there? I think the key thing that we have to do is in the environment, we have to plan, do your POCs, but plan from the bottlenecks, because the bottlenecks are not just on the IP side, on the fiber channel side. If you run our man, it'll grab as much horsepower as needed on the storage side, it'll drive your storage hard. So look through all the paths in the data path for backup and restore, and know how you can scale. So capacity planning is a major effort. You have to work with the capacity planners, you have to do your schedules correctly, and the amount of retention, so you don't do multiple fools all day, and do fools and incrementals and blend that all up. So a lot of planning ahead of time. VJ Vanga, thanks very much for coming on theCUBE. Appreciate you sharing with us the Federal Express story, and it was a real pleasure meeting you. Okay, thanks. And thank you, David Floyd, for helping me out with this one. Keep it right there, everybody. We'll be right back with our next guest. We're going to look at a service provider and how they are delivering backup as a service. This is theCUBE. I'm Dave Vellante with David Floyd. We'll be right back live from EMC World 2013 from Las Vegas.