 Okay, we're back, this is Dave Vellante, and this is the Spotlight VMware Backup and Recovery. We're here with Mike Crystal, who is the Systems and Enterprise Architect for Server and Storage at St. Charles Health. Mike, welcome to theCUBE. Thanks, Dave, for having me. Good to see you. We're here at VMworld, we're talking off camera about all the action that's going on here. First of all, what are your impressions? Excellent show, excellent show, just to... It is the show now, isn't it? It's pretty cool, if you're into virtualization, it's the place to be, so. Yeah, who's not into virtualization? Come on, it is a practitioner, it's totally changed your life, hasn't it? It has, I mean, I don't think we would be able to continue to grow our business if we didn't have virtualization, staff's expensive, machines are expensive. You got to do something about it, and virtualization is definitely the way to do it. So, talk a little bit about St. Charles Health, and then we'll get into what you guys are doing in storage. Sure, St. Charles Health is a regional healthcare provider in Central Oregon. We have four hospitals that were responsible for a slew of doctor's offices and clinics. We're the largest employer in Central Oregon with about 3,000 employees. We are on the road to virtualization, like everyone else. We have about 700 servers, 450, so a little over half of those are virtualized. And in terms of the applications, heavily virtualized, working your way up this... We're working our way up, we took the low hanging fruit when we got started, that legacy applications that were easy, but over the last two years, we've really started to concentrate on those business critical applications. A good deal of our current EHR and our ambulatory care applications are virtualized. Are you on a vSphere 5? We are on a 4.1 right now. Are you on a 4.1 and planning to go to 5? Especially after we heard some interesting news yesterday in the keynote about VRAM. So we are excited to hear that. Or VRAM pricing. Yes, yeah. I had to make you happy. Oh, definitely, definitely. I think that makes everyone happy. Yeah, okay, so that'll be a catalyst. Big catalyst, and then we are taking one of our main databases that we were unable to virtualize due to sort of vendor specs over the last couple of years, and that we're going to do it this year. And we need those specs that 5.0 and I guess 5.1 are going to support. So we need to be able to put more than eight virtual CPUs. So it's a big deal for you. I mean, you guys are in the business of saving lives. So, if you could spend more money on care versus infrastructure, that's a good thing for everybody, right? Oh, it's a great thing for everybody. So, talk about backup. We've talked a lot on theCUBE about how virtualization stresses storage generally and backup specifically. What was your experience when you started to get heavily into virtualization in terms of the impact that it had on your backup? Sure, we just really weren't prepared. I mean, it's a whole different paradigm backing up in the virtual world. We're taking the traditional have a client on a device, back it up on a certain window and call it good. And we were doing that to tape. And what we found, and I think what most people find in a situation like that is we had to build our ESX servers in such a way as to support our peak backup times. Caused lots of problems in the organization. We're going over our backup windows and we needed something to address that. And so, we went to the marketplace and we found Avamar and Data Domain from EMC. So, we've been super happy to have that. So, you essentially were not hitting your backup windows. So, you had your traditional backup system that you had had for how many years? I mean, was it? 10 years. So, I've seen some data. We have Jason Buffington on a little later from ESG. And I've seen some data that a number of companies have refreshed their backup and they've actually re-looked at it. I presume virtualization was a big catalyst that clearly was for you all, right? It was, it wasn't. And we also had a data center optimization program that was going on where we moved data centers and it really kind of exposed just what our problems were. We were replicating data with our old backup system and it was taking two days to replicate the changes from one data center to the other. We couldn't meet any of our RPOs or RTOs if we'd lost a data center. So, you expose that to the board and people get a little tense and they say get out there and fix it. Yeah, so, now, okay, so you chose a solution that includes both Avamar and Data Domain. Can you talk about the use case for each and why you deployed that way? Sure, you know, from a front end perspective, to us, it's seamless. So, you know, we use the Avamar as sort of the UI, you know, and the Data Domain sort of sits behind that. So, that's how it kind of came into the organization but once we got in and we started to deploy, what we found was we could have the Data Domain as a target for backups. And so, what we ended up doing is instead of changing the way our Oracle backups were running, we just swapped it, you know, the Data Domain in. So, you know, we were backing up basically to an NFS mount and then scraping that data off and off onto virtual tape. So, what we do now is just back up directly to the Data Domain. And what happened is those backup times shrunk and then the restore times. It's a phenomenal difference on restore. I was just talking in the segment before this just about, you know, why that is. But, you know, we were having jobs that took 12 hours to do restore that now maybe take us an hour and a half. So, it's made a big impact on the business. Can you elaborate on why that is? Why are the restore times so much superior? I think there's a couple of things. I think Stephen spoke to quite a lot about, you know, how the data is actually put down on the device. But, you know, from our perspective, we don't have to send that stream compressed anymore and then uncompress it as it goes back down. And as far as our man's concerned, it's just a straight backup. So, you don't have to hydrate, rehydrate. Exactly. Rehydrate. Exactly. Transparent. And then it's all on spinning discs, so, you know, that. Yeah, it's not serial. So, are you virtualizing Oracle? As an aside? We are. We are. Join the torpedoes and we virtualize Oracle. Good move, we've advised that that the benefits outweigh the risk. Have you had any issues with that or? We are only in some test dev right now with what we're doing, but the database that we talked about this year is the one we're going to concentrate on getting moved to production. So, excited about it. So, talk about the project, if you will, where you sort of deployed this backup solution. Can you take us through, kind of briefly, the sort of, you know, the planning and design phase and then the implementation? And I want to get to, you know, what you do differently if you had to do it again. What advice you'd give to practitioners? But take us through, paint a picture of what actually happened there. Well, we had a problem. You know, we described that. So, we identified. Got people's attention. Got people's attention. Developed a series of requirements. Went to the marketplace. Started doing due diligence. Talked to people out there. Hey, what else are you doing? Talked to folks like Gartner. Said, hey, what, you know, here's our problem. What should we do? We basically narrowed it down to four separate vendors. One of which was incumbent. You know, the company we were using already. And looked at, you know, the dollars and cents of the situation. So really ended up coming down to incumbent technology and to Avamar data domain. So if we left our incumbent technology in place, we were going to spend $1.3 million over a four-year period to upgrade the system and keep it running steady state. So we make an acquisition or make an acquisition with Avamar data domain. We could do that for under $1 million or like $940,000 saving $300,000. So essentially, the system paid for itself. We went to the board and said, hey, I have an ROI of less than three years. You don't have to take money out of your pocket. It was already in the budget. And it solves all the problems we talked about. So that was pretty much a slam dunk. They picked it up and went from there. So then as far as the actual project itself, we brought in EMC professional services. And they assisted us with doing the install, which was very smooth. I mean, we did 700 agents and didn't have a single downtime. So it was a great experience for us. So 700 agents. 700 agents, yeah. OK, so the ROI was basically ROI, right? Benefit over the cost of the benefit. If you lower the denominator, you're going to get a better ROI, right? Exactly. So that was pretty straightforward. How has it impacted your business? You talked a little bit about meeting back up windows and recovery times. Can you give us some color on that? Maybe some metrics that you'd use? And maybe talk about the recoveries that you've gotten. I mentioned a little bit about the replication going from taking over two days down to just a few hours. I think 10 to 12 hours is what it's taking now. We essentially got an FTE back by deploying the solution. So instead of spending time figuring out what failed, why it failed, rerunning that back up again, locating tapes, doing reclamation, all that work is gone. So big, big change there. So that's really where the concrete savings are. And the FTE was not in the business case, is that right? We didn't include that in the business case. We did not. Because if you do, then you have to get rid of the FTE. Exactly. OK, but so ultimately that led to greater productivity and did other things with that person. And then in that nearly a million to $900,000 plus dollars, that included the deployment cost, the professional services piece? It did. It did. So it was a huge, huge, huge savings for us and really a no-brainer. Any learnings that you can share with your peers or things that you might do differently or advice that you might give them? Sure. I think one of the biggest difficulties we had and I think it's probably more relational than anything else was that Avamar and Data Domain were relatively recent acquisitions to EMC. And so the communication pipeline wasn't necessarily there to begin with as far as we had some difficulties with just a sheer number of VMs that we had and trying to get them backed up in that right window and get them replicated in the right window. So we were able to work with engineering and get all that solved. But it just took a little bit of relationship help. And we had a great sales team who was willing to do that. So it was a good thing. So just good service, attentive. Any advice that you'd give to other practitioners looking to try to solve their backup problems? Do your due diligence. Figure out what your use case is, go to the market, and do your homework. But I think if you do, our solution is probably going to work for you. Excellent. All right. Well, Mike, thank you very much. Great story. Appreciate your insights. Thanks for coming on theCUBE and sharing them with our audience. And good to meet you. Great. It was a great meeting you. All right. Take care. All right. Thanks, everybody. Keep it right there. We'll be right back with Jason Buffington from ESG. We're live. Silicon Angles coverage from VMworld 2012. We'll be right back.