 The most valuable applications in the world run on Oracle databases. The amount of money spent on Oracle and the processes and the people around Oracle databases is in the tens, if not hundreds of billions of dollars annually. The mission critical applications that are running on Oracle need data protection. Hi, everybody. This is Dave Vellante of wikibon.org. Daryl Smith is here. He's the chief database architect at EMC IT. And we're going to unpack best practices in protecting Oracle databases. Daryl, welcome to theCUBE. Thank you very much. Good to be here. Appreciate you coming on. You're within EMC IT. We always love on theCUBE talking to the folks within EMC IT, because you guys have a hands-on perspective. Yet at the same time, you're pretty savvy about what's going on in the marketplace. So first of all, tell us a little bit about your role within EMC IT. Sure. I'm the chief database architect for EMC IT. I'm responsible for all of our Oracle databases. As a matter of fact, I'm responsible for all of the databases in the company. And that includes keeping them highly available, keeping them safe, secure, as well as architecting for what the next wave is going to bring. So Oracle is the leading database in the market. And as I was saying in my introduction, the most mission-critical applications are running on Oracle. I mean, why is that? How did Oracle get to its prominence? Well, Oracle's been around for a long time. I actually started in Oracle back in version 4 in 1988. So I've been with it for a long time. And it's really put a lot of its effort and money into R&D. And so they pretty much, from the get-go, as far as relational databases go, have always been the market leader. And they really focus on the ability to handle high transactional workloads, which is really their sweet spot. Yes, so I mentioned as well, the value of these applications is very high. And the people and processes built up around the technology is substantial. So presumably, protecting the data around Oracle databases is critical. But what's unique and different about protecting data within Oracle environments? So I think you've covered a bit of that already. The data that you put into an Oracle database is generally your most mission-critical, your highest transaction volumes, which means you've got your most users there. And your uptimes are usually absolutely critical. Otherwise, why would you pay that premium? Yeah, so OK. So it's really, I guess it's the impact of downtime is just greater. The impact of lost data is greater. And that's a board-level impact. You might read about it in the Wall Street Journal. So it makes the stakes much, much higher. Talk a little bit about how one goes about protecting Oracle databases and the alternatives they have, but before you do, let's assume, are we assuming a virtualized context? I mean, is that something that is increasingly prominent within Oracle environments that you're virtualizing the database and the applications? Well, that's certainly a component. More and more people, more and more DBAs, are virtualizing their databases. And they're doing it for a lot of reasons. One is just to get control of the costs. Licensed costs for Oracle are extremely high. And so the more cores that you're running your database on, the more you're going to pay. Well, there's inherent efficiencies in running it in a virtualized infrastructure. And I would take hours to try to explain why that is. But essentially, I can get a lot more processing power out of my servers and out of my cores when I virtualize. It also adds in high availability that doesn't exist in a physical server. So because of those benefits, and obviously many more like agility, being able to stand up a database server in minutes versus days, months, weeks, years, are all benefits as to why people are starting to virtualize. So it's part of the story. And it's much, much bigger than that. So what are the choices that a customer has in terms of protecting Oracle data? Right, so there's all kinds of levels of protection. One is obvious, I just need to do backups because I may need to go back to a prior version, either because I've hit some data corruption bug within the kernel or some bad code got out there and just made a bad sense of the data. I may need to do a restore. I also may need to do a restore if I don't have a business continuity plan and I lose my data center. And or I lose my server or the storage that it's residing on. I may need to do a restore. So one of them is basic backup and restore capabilities. And this is really starting to become a challenge for people as big data really hits. Our databases were big back a couple of years ago when they were two or three terabytes. Well, that's a small database today. Big databases now are 20, 30 terabytes and bigger. How do you continue to back that up using the same old technologies you've been using for years? So usually there's the backup window, right? Backup windows are getting bigger and bigger and so DBAs are having to put more and more resources at it which has a much bigger impact on the database. So if my database is already busy and now I'm trying to back up a 20 terabyte database that's highly active and users feel the pain. So in terms of choices that customers have to address that pain, I mean, our man is obviously the standard in a lot of high end shops. Talk about that a little bit and maybe some of the choices that people face. Yeah, so back in the day, back before our man, we would have to do either cold backups or hot backups but manually, right? Write the files off to either another disk area or tape directly. That's just not even possible anymore. So a cold backup would be quite as the database. Take a backup. Well actually, cold is shut it down, right? Then copy the files off and start it up again. So that's pretty much not even an option these days. A hot backup, I can do without our man by just putting the database in the hot backup mode and copying the files off. We've kind of taken that one and put a little twist on it. We actually can either put the database in the hot backup mode or not but with Time Finder Snaps on both our VMAX and VNX series storage arrays, I can take a clone of that, those data files instantly. So whether I put the database in the hot backup mode or not, I actually am not required to, I can take a backup in about 30 seconds and then write that off to a completely different set of disks and the database is right back up and running. So that completely different set of disks could be on-site or even off-site, right? That's ideal. Well, ideally, if I'm doing a snapshot or a clone and I'm doing it within the storage arrays, I'm in the same data center. But literally I could do the same functionality with RecoverPoint, which could be in the data center or could be replicating elsewhere. You could also do it with any of our replication technologies like D-Plex or SRDF and actually do your backup on the DR side. So the hot backup that you described is still manual, R-Man automates that, right? Talk about that a little bit. Right, so R-Man is really kind of the glue that holds it all together and makes things much easier for us. R-Man has features like compression, multi-threading, which allows us to backup a database with many more threads running and not have to worry about which data files I've gotten and trying to balance that load across different threads. R-Man handles all that for you and gives you a catalog, which is intelligent in those exactly when your backups were, which archived files are required to restore it up to what point in time. So it's a very automated way of doing it. Most third-party backup applications actually tie into R-Man. And so while you've got a backup repository in your third-party tool like Networker, for instance, is one we have, you also have the R-Man catalog. So when we talk to practitioners within the Wikibon community, they tell us, and it depends who you talk to. The CIOs, they wanna align with their IT, they wanna drive business value, they wanna manage their risk. When we talk to the infrastructure people, they wanna obviously cut their costs, they wanna make sure that they're leveraging infrastructure across as many applications as possible in the portfolio. When you talk to the application heads, they just wanna make sure they're meeting service levels and they're getting the right performance. There's this kind of inherent tension between the objectives that some of those different roles have, particularly as it relates to storage management and backup. Can you talk about some of the DBAs and the storage admins and some of the organizational nuances that are involved there and how you guys are addressing some of those? Sure, in backup space specifically, Data Domain has a storage appliance which has deduplication built into it as well as compression. And so, typically when we're trying to back up to another disk array, what happens is the owners of the storage admins that own that disk array are trying to always push back and say, hey, don't take as many backups, I don't have that much storage. And so it leads to us having to do like our main incremental backups, which can be painful or could actually be catastrophic when you try and restore those back, right? I have to restore back a backup that I did a week ago and then I have to restore a bunch of incrementals to do an incremental merge. And that doesn't always work and sometimes I actually can't get a backup done. With Data Domain, I can take full backups every single day and the only difference in storage is the difference of the data rate change for that day. But it also includes compression on the fly. And so I actually have backed up a 13 terabyte database and it took three terabytes on the first backup and then added another 300 gig on the backup the next day. I mean, literally you're reducing that storage down to a bare minimum. Okay, so talk a little bit about, you're starting to discuss some of the best practice. So what are some of the best practice considerations that you guys have applied within EMC IT that you might share with some of your fellow practitioners? Sure, one of the ones, one of the practices that we do on our most mission critical, highly active databases is one that a lot of people, once they hear it, are actually go out and start implementing it. And I talked about how I can take a backup by simply taking a time finder clone. Well, I can actually mount that clone up on another proxy server. That could be a physical server or just another VM. And then I can back that up using our man to my data domain or wherever my backup storage is. And I can do that without any impact at all of the production database. So even if I've got a highly active 20 terabyte database, 30 seconds to take my backup and then I can take as long as I want to actually back that up to my backup media, up to and including running it as fast as like eight terabyte an hour. Which as you said could be on site, it could be off site. And how do you guys actually manage that? I mean, obviously you're doing some stuff on site, but off site as well. We'll talk about that a little bit. So I like to keep my backups local. Obviously compliance people as well as the business likes to make sure their backups get off site. Now, why do you like to keep them local just because they're there and it's faster to recover? Well, if I ever need to recover them, it's a heck of a lot faster if it's local than if I have to either drag it over the wire that's 600 miles away or call a Byron Mountain or something to have them bring the tapes. And yeah, and most of your recoveries is just presumably correct is for reasonably fresh data? Yes. Yeah, generally if you're doing a recovery, it's something that happened in the last 24 hours. It's rare that you would have to restore something that's older than that, although that is certainly possible. And generally that's a compliance question or I need to do some surgical restore. I've accidentally deleted data that was from last quarter. But all of those are fairly simple, especially when you add in that proxy server. But let me get back to first getting the data off site. I like it local, but I need it off site. I use the data domain replication that actually replicates that data behind the scenes to my business continuity site. So I've got my backup now in two different locations. And if I use a product like DD Boost, the Armand catalog is actually the one that controls that replication. So my Armand catalog knows about both locations. So I could restore from my DR site or I could restore from my production site. And then you mentioned tape before. At that point, do you go to tape? Do you offload, you know, and stick it in an Iron Mountain somewhere or just just in just in case? So you could, I suppose. But given the DD implication and the fact that I'm only storing the data rate changes, I can easily store like a year's worth of backups in that data domain without taking up very much space at all. Doesn't really help me to get rid of that old, old version. Now, obviously we need to do yearly saves for compliance reasons. That we do backup to tape and send that off site, but I don't ever expect to have to retrieve that. So that's deep archive that you never want to see again, probably. Correct. If you do, then you're pulling some all-nighters. A lot of all-nighters and I don't like those very much. Okay, how about maybe describe the environment a little bit more inside of EMC-IT in terms of what you guys are doing, how you're applying Oracle and protecting Oracle. Sure. Paint a picture for us. Yeah, so we've talked a lot about backup. What we really haven't talked about is availability, which is another way that you need to protect your database. Can you afford to be down or have that database down for four hours or eight hours or 48 hours? That's just not possible and that's not protecting your Oracle data. So you need some sort of high availability which is why people move to Rack. Oracle Rack is a very, very good technology of keeping an active, active copy of your database up and running. But in general, people don't need Oracle Rack to have something that's always available. That's really kind of a fallacy. Most of our big Oracle Rack databases have two to three hour outages because by Oracle Rack. What most of the databases really need is they need that ability to come back within three to five minutes. Database goes down, the server blows up, I have a fire in the data center. I want to get that database up as quickly as possible and five minutes is pretty darn quick. And so that's where virtualization comes in. That lets me have high availability of my single instance database. So I'm protected out of the box with just a VMware virtualization solution without having to go into that very expensive, very complicated product of Oracle Rack. So I want to change topics a little bit and talk about external customers. I mean, EMC, like many IT companies, but EMC is particularly astute at this, will trot out its best IT practitioners and talk to other IT practitioners, ostensibly customers of EMC or potential customers of EMC. I presume you're one of the spokespeople to do that. Talk about those activities, what you're hearing from customers, where does the EMC Salesforce use Daryl Smith and leverage your knowledge and what kind of discussions are you having with customers? Yeah, unfortunately, they leverage me just a little bit too much. But I do speak to a lot of customers and these are not only our big customers, but also our small customers. They not only like to know what EMC internally is doing, how we're using our products, they also like to know just how a large Oracle shop does it. So I speak to customers in what's called our Executive Briefing Center and I also speak with customers over like Webex meetings, as well as just regular conference call meetings. Sometimes they tell me other problems and I give them how we solve those. So I speak with customers all the time and they have several complaints and not necessarily with the EMC. They're all trying to solve the same problem that we are with Oracle. How do I protect my data? How do I reduce my licensing costs? How do I get an efficient use out of my infrastructure, whether that be storage or CPU or you name it? How do I efficiently use that storage, push it to the limit and yet protect my business? Yeah, and they're resource constrained. I mean, you pointed out. I mean, we've quantified the impact of Oracle licenses. It's expensive and licenses and maintenance. We've also done some work in terms of just evaluating how to balance out the system to actually reduce the number of cores that you have so that you can actually reduce your licenses, maybe spend a little bit more on storage, maybe even add some flash to balance the system, reduce the impact of the number of cores that you need and enhance the ripple through effect to your license costs. Have you guys played around with that? Absolutely, and that was a great blog article. You even mentioned memory. The simple addition of memory can reduce my CPU consumption, thereby also reducing the number of cores that I need. It's all about trying to offload that kind of work away from the CPUs that you have to license into something else. So that was a great article. And we play with that all the time, which is one of the reasons why we have extensively tested VMware and found all the different knobs and buttons that you can push or twist or pull to get you that native performance that you would get without the virtualization. This in-memory trend, since you mentioned it, is kind of interesting. But in-memory databases have been around for a long time. At the same time, prices are coming down, memories are getting larger, but you still gotta be able to protect that data that's in-memory. So what are you guys, what are you seeing in terms of that in-memory trend? How are you exploiting it, and how are you protecting that data? Yeah, well, there's a lot of new in-memory database players, right? SAP put out HANA. They're actually looking at virtualizing HANA, which I think will be very interesting. VMware also has SQL, I'm sorry, it used to be VMware, and now it's Pivotal Labs has SQL Fire, which is an in-memory database, which then persists to some storage. That storage could be a flat file system, not really a great idea, or it could be another database back end. We actually do our integration layer that we have. We have a common integration layer throughout all of the EMC applications, and that runs with a SQL Fire, front end, which persists to an Oracle back end. And the reason that we persisted to an Oracle back end is really all about that data protection, right? We are able to replicate that database 600 miles away with a zero data loss very easily, and it's something that we feel very comfortable with. And so a lot of times, and I really think it's best practice, is use that in-memory database as a front end to some other database. And literally, if that memory database is doing all the work, then maybe your back end database doesn't actually need to be that hefty. So I'm starting to play around with doing that same kind of workload with a SQL Fire, the in-memory database, persisting to a Postgres database. I'm getting great results. So a lighter weight back end that's maybe less expensive, more efficient. Why do you see Flash playing all this? Do you see sort of Flash as adding another layer, or do you see over time those layers collapsing, sort of high performance, tier zero, and then a bit bucket? What do you expect? Yeah, bit bucket. So Flash is definitely making a huge play here. I mean, there's Flash products from all Flash arrays. Extreme I.O. is one of the MCs, which is coming out, I believe, in Q4 of this year. November, I think, yeah. November, right, yeah. Which is a phenomenal product. I've been playing around with it in my lab. I'm migrating into my data center. I'm gonna get it in there before we even go GA with it. It's extremely fast, and it's got built-in deduplication, right? So Flash is expensive. Regardless, if you're trying to build an all Flash array, you're gonna spend some big bucks. But if I can utilize that deduplication ability, I can have my production, my development, my test, all in the same array, still get great performance, yet basically only use the amount of space that production would use, right? Dev and test are kind of free. But when you're talking about Flash, again, you're offloading resources that would be utilizing CPU resources that would be utilized in weight states for I.O., you're removing those, right? So you're gonna use a lot less CPU, as well as get your data back faster. But again, as I said, all Flash arrays are very expensive. It's much cheaper to put stuff in a bit bucket. So if I can have multi-tears within a single storage array, I can really help to reduce the cost, right? That's one of the products that we have, which is called FastVP, which automatically migrates that data behind the scenes into the appropriate tier. If it's data that doesn't get used often, it goes down to that bit bucket running on SATA drives. If it gets used sometimes, it'll go up into the fiber channel, which is our traditionally high-performance disk, magnetic disk. And then if it's really, really random and is used a lot, then it's gonna go up into the Flash area. So it lets you make a more efficient use of that Flash. All right, Darryl, let's bring it back. I'll give you the last word to data protection in Oracle. So piece of advice to practitioners. They're struggling with backup windows. They're trying to keep their costs down. They're drowning in Oracle license and maintenance costs. What's the one thing that you would advise your fellow practitioners in this area that are struggling with this problem? So when it comes to backups and backup windows that just keep growing, you need to do something about that, you could go the traditional route and look at trying to do like RMAN incremental backups, but your better bet is really go look at data domain. Data domain has a product called DD Boost, which only pushes across the wire those bits that have changed. And it can really make your database backup run a lot, lot faster with a lot fewer resources, both on the production server, as well as in the network, as well as the backend media. And then there's also what I talked about earlier where you offload those backups. I mean, we have, like I said, 20 terabyte databases, which we back up in 30 seconds. No CPU utilized on that production server. So those are both good options. Anything you would do differently if you had to take this journey again? Ask for more money. Yeah. Well, we've certainly learned a lot along the way. You know, we took a lot of lumps in the beginning. It would have been great to know what these best practices were before I started or even shortly after we started. And so in order to prevent other people from having to experience that pain, we've published all those best practices. And you can find them up on a site that EMC has, which is called Communities. The Communities at EMC. And you can just get to it, you can Google everything Oracle at EMC, or you could go to Communities.emc.com, and there's an everything Oracle at EMC group there. And we publish a lot of our best practices there, including the backup stuff we talked about, as well as how to virtualize. I've written plenty of blogs on all of the best practices and virtualization, how to get native performance. So there's nothing I probably that I could have done differently, but I would certainly have liked to have learned from me back when I started. Yeah, there's some good resources definitely on EMC.com. You guys do a really good job, I think of testing out various solutions and configurations. It's a great freebie. I always tell customers, take advantage of it. Daryl Smith, thanks very much for coming on theCUBE. It's great to see you. All right, everybody, thanks for watching. This is Dave Vellante from wikibon.org. This is theCUBE, SiliconANGLE's continuous production, where we go out to the events, we bring the experts in, we extract the signal from the noise. Thanks for watching, everybody. We'll see you next-