 Hi everybody, we're back. This is Dave Vellante of Wikibon.org and this is SiliconAngle.tv's continuous coverage of Oracle Open World. We're live here at the Moscone Center in San Francisco and this is the spotlight on Oracle Backup and Recovery. We've been drilling down with experts and practitioners and analysts on the topic of backup and recovery. It's a complicated situation for a lot of practitioners out there in our audience. You make a mistake, you lose your production database. Not only are you out of a job, your company is out of a lot of money and that's a bad situation. So backup and recovery is a very important topic and we're here with Darryl Smith who's the chief Oracle architect within EMC's IT organization. We've had a number of practitioners on from EMC's IT organization in the past, from their CIO, Sanjay Mershandani to a number of other folks and we really appreciate Darryl, you're taking your time out to come on theCUBE. Welcome. Thank you very much, I appreciate being here. Yeah, so as I said, this is a complicated topic for a lot of people. It's one that doesn't necessarily get a lot of the headlines. You haven't heard a ton about it in the keynotes and things like that. But so let's start with what you do at EMC IT. You're involved in backup and recovery and you're involved in Oracle. Talk about your role as the chief Oracle architect. Sure, well as you mentioned, I am the chief Oracle architect for EMC IT. I'm responsible for the successful run of all the databases and protecting them is a critical component. As you mentioned, if you lose your database, if you lose your data, you're out of business and that's a lot of pressure. And unfortunately in the same respect, nobody thinks about backup and recovery until you actually need it. And that's one of the big challenges. There's no heroes in backup and recovery. There's only villains. Only goats, right? So obviously EMC is a big Oracle shop. You guys have made no secret about that. It's one of the largest on the planet. And you basically run your business on Oracle. So tell us about protecting Oracle data. What's different about it? What's unique about it? What's so challenging about backup? Sure. So we have well over 800 databases approaching 900 databases that we're responsible for and need to backup. And that's pretty much about 100 terabytes of data to backup a day. So the biggest challenge is that's about 100 terabytes of data to backup a day, right? And that's growing all the time. And we talk about the explosion of data and that's in Oracle databases. It's just growing and growing and growing and we can't get rid of it. So it becomes more of a challenge as these backup windows start to get bigger and bigger and your backup infrastructure doesn't keep pace. So you're backing up 100 terabytes a day over 100 terabytes a day and you're doing an incremental backup? Is that right or you're doing daily fulls? We used to do incremental backups and that can really help you save on space on your tape drives. If you're still using tape drives, hope not. Or if you're using disk arrays to do your backups too, much faster response times. But the problem is with an incremental backup in order to recover that, recovery time is very important. In order to recover the incremental, I first have to recover my last full. Then I have to roll it forward with each of these increments and that adds time to recovery. So our goal is really to backup full whenever we can. And with the advent of data domain and the deduplication, I can now do full backups but in the same space the incremental backups would have taken. Okay, so your RPO is low for your Oracle apps. Zero. You have zero RPO. No data loss. That's the requirement. Okay, but you're also saying that your RTO is a main driver that affects your methodology. Correct. If you've got a mission critical system and when it's down you're losing revenue, how long can I take to recover that database? Not long. Well let's say it's 10 terabytes or 20 terabytes. Can I really afford to back up or to recover at let's say a terabyte an hour? That puts me at about a day. A day of lost revenue. We can't do that. So how do you achieve RPO zero? You have a three site data center or are you? So if you're talking about disaster recovery, that's a little bit of a different story but we do replication from our main data center which is in Massachusetts down to our secondary data center which is down in Durham, Massachusetts at Durham, North Carolina. Okay. And we do that through multiple ways. In some cases we do full database replication through our storage replication technologies like Recovery Point and SRDF. In other cases where we've got super mission critical databases, super large databases, we do a standard roll forward where we just replicate the redo logs and the archive logs and we just keep that database rolling forward. So we know it's always ready for us at a moment's notice. Okay so locally you can achieve RPO zero independent of a disaster, a disaster is a different situation. Okay. So locally we do that because in an Oracle database after you take your full backup you then back up all the archive redo logs, right? So all those transactions are saved in there so you can roll your database forward to any point in time. And use an R-man or what methodology is it? Absolutely, R-man is the only way to go, right? Yeah but there's still a lot of people who don't use R-man in the Oracle. Right, well I can remember back in the day even before R-man where you had to back up the data files and back up the archive logs individually and you had to basically put them out somewhere and then have some backup thing come along and take them later. It's a manual process. Oh you write a script, very error prone. Very error prone, especially if you really don't know what you're doing. With R-man it's very simple, you just tell R-man to recover until this point or recover until current and you just sit back and watch it do its work. Now how do the products that you're using, obviously using EMC products, right? Absolutely. How do they work and integrate with R-man and the whole Oracle environment? The entire EMC set of products are really designed to work with R-man. So we have backups of strangers where we offload them. So we take a storage clone, mount it up on a completely different server and back it up from there using R-man. But R-man understands that and understands that even though we're backing it up from this proxy server, the storage clone, the storage node, the archive logs that we back up from the production database are going in the same area and it knows they're the same. Networker adds an integration. That's another product from EMC so that when I back up through R-man it uses Networker as its virtual tape device to back it up to whatever virtual tape array you have. For us, that's now data domain because with the deduplication again, as I mentioned, I can do full backups all the time. The nice advantage to offloading this backup to a proxy host is that if I need that data back, I simply reverse the clone and I'm rolling it forward immediately. That helps me with my recovery time objective. Okay, so paint a picture of your backup architecture. What are the piece parts and help us understand what it all looks like? Sure, so today we've got databases that are backed up with these storage clones. We've got databases that are backed up straight through Networker to data domain. We've got some databases, like our databases service offering, where we just do daily exports, again to data domain. And so we use a combination of all three of those methods in order to back up each or 800 and some odd databases. Okay, and you mentioned before, I think off camera, that you were 100% virtualized, is that right? Well, in areas, so from a server virtualization perspective, we're at about 98% virtualized. When it comes to databases in general, we're around 84% virtualized. But our new system that we just launched, which is Propel, which is based off of SAP, running on Oracle databases, is 100% virtualized and it's running extremely well. Okay, so it's pretty high levels of virtualization. I think Joe Tucci several years ago in an endless meeting I asked him, when you're going to be over or close to 90% and said we're going to do it within five years and so you're tracking to that. For overall servers, we've hit it. Yeah, and servers, I was talking about the applications, which is really the hard part. Right? It is. And so what makes that hard? What are the barriers to getting that up? Do you have an intention of getting that up? Is it performance concerns? Is it the application heads? Is it process? What's the issue there? So from a database perspective, we were concerned about performance and we spent a lot of time trying to figure out how can we get our databases virtualized and achieve the same performance that we had, or better, than we had in physical world. And so we spent a lot of time, we worked with VMware engineering, we worked with a lot of performance experts and did a lot of benchmark in our labs and we've come up with a bunch of best practices which help you to implement that. And so now we're getting, as good in some cases, better performance, right? Moving to newer, faster servers. And so now it's a matter of resources. How many people can we do as quickly as possible to get to that virtualization? So where does the cloud fit in? I don't know if you had a chance to see any of the keynotes this week. Big dose of cloud. What's your take on cloud as a backup target and what are your thoughts there? So adding cloud in for backups, that's a problem. Your backup infrastructure typically is going to be held back by that movement to cloud, or I'm sorry, the cloud's going to be held back by your backups because your backups are defined to go to one place. And so if you start moving your cloud from here to there to somewhere else, how are your backups going to follow it, right? That's where we came up with DDBoost. So DDBoost is essentially abstracting or taking away from your hardened infrastructure. And so you can back, it doesn't matter where you are, you'll backup through DDBoost, full arm and integration to your backup target, regardless of where you move to. And so that's really helping us to accelerate into our cloud. Okay, so public cloud as part of the strategy or not yet? So we have very limited public clouds. We have essentially applications that are running outside of our data center. Some of it is in our business managed units. Some of it is, for instance, our sales, our sales contacts, I can probably say at salesforce.com, right? So we've migrated some of it out there. We have deep integration with them so we've got that same copy of the data internally so that we're able to very easily and quickly move anywhere we need to go. Going forward, we're looking at actually extending ourselves into other people's data centers because at end of quarter we have a huge surge, not just in infrastructure and people needing to get their work done, but that's when it becomes critical to get it backed up and critical to make sure that we can restore that very quickly. So that's an issue if you just don't want to buy the extra hardware and software assets if you can just do an end of quarter surge. Well, yes, and I'm sorry, no. So there's the over capacity that you can get for end of quarter surges, but there's also, as the business needs to be agile, we may not be able to get that infrastructure in quickly enough. So it may be a temporary surge out into the public plan. So it's not just an expenditure issue, it's a speed, flexibility, we've got the agility. Agility is really the name of the game. So, Darrell, my last question for you is, talk to about what you would advise your fellow Oracle practitioners with regard to how they're architecting backup. What are the things that they should pay attention to, some of the gotchas that they should try to avoid? What's your best advice? So my best advice would be plan ahead. Your backup infrastructure's not going to keep up with you unless you make it. That's been our biggest problem. It's slowed down a lot of our plans because our infrastructure couldn't keep up. You've got to get to some sort of a de-dupe or compressed version on disk. Tapes really can't support you anymore, not with the size of the way the data's going. So you've got to plan ahead, you've got to get those guys on board before you can get launched off. Excellent. All right, Darrell Smith, thanks very much for coming on theCUBE. It was a pleasure meeting you. It's been fun. Take care. All right, keep it right there. We'll be right back with our next guest. We're live. This is theCUBE from Moscone in San Francisco, Oracle Open World. Keep it right there.