 Hi everybody, this is Dave Vellante of Wikibon.org and welcome to CUBE Conversations. SiliconANGLE Wikibon's discussions with IT practitioners and technologists. Over the last several months, we've been talking to EMC customers about their journey toward backup as a service, something that the Wikibon community has been talking about for several years now. Tom Jackson is here. He is an IT practitioner, the infrastructure head of operations at McHladry. McHladry is an assurance tax consulting firm. Welcome Tom, good to see you. Nice to be here. So let's start off, tell us a little bit about McHladry and your role there. McHladry is the leading US provider of assurance tax consulting that focuses almost exclusively on the middle market companies. We do a lot of work with companies that have grown beyond the mom and pop shop and are not quite up to the top tier of their particular industries. We employ around 6,700 people a year around that goes up to as many as 8,000 during the peak seasons. And we have offices stretching all over the United States. So a little bit of background on that is you've got distributed offices, so that makes backing up data a challenge. Now, of course, your purview is more than just data protection. You run all infrastructure operations, but talk about some of the challenges you faced historically with backup. You've been there at McHladry for several years now. What was keeping you up at night with regard to data protection and backup? Well, McHladry has a centralized data center environment for a lot of data, but there's a lot of work that happens locally in the offices and there's applications that in our industry just need to be physically there with the people that are actually doing the work. So we had a very distributed model of backup including, as in a lot of companies, tape backups at every site, managed by people locally. And for some of those offices, that was actually being done by office managers or whoever was coming in that day to clean the floor or pay who'd mind swapping a backup tape. So across those offices, for people to properly maintain and keep the data protected, there was some inconsistency in performance of those duties. There were inconsistencies of people actually checking to make sure the backups were running on site daily. And just that uncertainty is what probably kept me up at night the most. Okay, yeah, so you had a situation where it wasn't IT pros and lab coats doing the backups, but it could have been a professional or somebody who's an accountant. And as you say, they may not even been doing the backups or doing them properly, is that right? That's correct. We had many, many times where we would go, check logs, do a review of what's happening and find offices where the backups were not being performed as frequently or as consistently as we would like. So what did you do about that? Talk about the project to change. Well, we started looking at it in a traditional method of, okay, how do we put some sort of centralized control around the process? But we ran into other things that got brought to us as problems as well that we hadn't even thought about initially, primarily from our litigation services group, around things like deletion of data. So once you create a backup tape, yes, the backup itself may expire, but how do you make sure that tape itself has been purged of all data so that you don't find data from 10 years ago on a tape somewhere when you get involved in a lawsuit? We ourselves don't get involved in many lawsuits for ourselves, but as an assurance and tax and consulting company, there's a lot of times where we're advising or providing information for our customers. Okay, so talk about the project itself. The infrastructure that you brought in, what changed in your environment? Well, we ended up reviewing a couple different vendors that provided similar types of products or at least claimed they did. And what we initially looked at wasn't going to work, we determined. We would have to replace too much of that actual tape media hardware. We found just the method of using tape in general to be inconsistent and it's just basic design. So we said, okay, let's go to something where we can pull the data disk to disk centrally if we can and manage and maintain it in a simpler manner. So we actually went with EMC Avamar as our end solution and it's been a great product for us so far. So why did you go with EMC Avamar? Maybe take us back to the decision point. Well, the primary drivers, obviously, we were looking at doing a disk-based backup system. And in some of these office, we were mainly talking about office type files. So Excel, Word documents, PDFs, all sorts of files of that nature. And the backing up that much data somewhere in neighborhood of 70 or 80 terabytes of data we're protecting out in the field, back to a central location, would have been really difficult to do with a lot of normal type of backup solutions. EMC Avamar was one of the solutions that did a client-side deduplication that allowed us to do the backups without having to increase WAN capacity. That was one of the driving factors. So what role did EMC play in this whole project? Were they sort of a consultant, trusted advisor, just a technology supplier? Maybe you could talk about that a little bit. EMC initially brought them in. We do with EMC on the storage side, have for many years. So we brought EMC in to look at their solution initially as just getting a sample of what the market looked like. I mean, Avamar was one of the few names in that space at the time we started looking. And so we brought in EMC and their top two competitors in that space and said, okay, show us what you got and why we should go with you versus your other competitors. And some of the deciding factors of EMC's solution was we really liked the way that EMC, we liked the way that EMC's deduplication was more of a variable block deduplication instead of a standard type block. So the deduplication rates were better. We liked the licensing model that they had at the time. We also really liked that when we bought the EMC solution we were all getting a hardware platform as well as a software platform. Because some of the competitor products we would have had to build out a storage platform to act as those repositories. And we really kind of wanted a turnkey solution. And that was really what we got with the Avamar. So a lot of product considerations there, the technology, the product of, you mentioned licensing as well, but it really was a product-centric decision. Is that right? Yes. Okay. Now, I understand you also subsequently brought in a data domain system. Can you talk about that? Right, so we use, we currently, and still today for the data, for our central data center actually use a competitor product for a tape solution there. The history of why that is is probably too long to go into now. But we were, at the time that we brought in data domain we were running a very large, probably four frame tape library. We had a requirement to maintain copy onsite, copy off site daily. And we were looking to reduce the floor space necessary in our hosted data center facility for that purpose. So we again said, all right, well we liked the way the deduplication space played out with Avamar. We need some, we need a solution that we can plug into our existing tape provider. So what can we go with? And again we looked at a couple different vendors but had gone back to EMC initially and my account team there and said, okay, show us what else you got. And they showed us the data domain. We liked what it looked like. And we also liked that in the future they were talking about Avamar and data domain kind of merging space. So that was kind of a deciding factor there. Yeah, the integration obviously, we talked to a lot of customers about that and that's appealing when you buy two separate companies that take some time to put them together. But that's something that I know you're excited to see happen. And we talked to their executives about it considerably and it seems to be moving in that direction. Talk about the outcome. So what was the before and the after like in terms of things like time it took to complete the backup, any cost factors, other business impacts that you achieved. So we were doing a traditional tape backup solution before. So a standard office, it would take three or four hours for a small office to run through and do a incremental tape backup on site. Today a full backup and allow those sites happens in under 15 minutes because since it's just gathering the changed blocks it takes almost no time for the data to move between for the system to figure out what needs to be sent to the central data center. We also had around 60 people roughly in the field probably taking care of backups. And if they were doing that job correctly it would have taken probably an hour a day for each of those 60 people to actually check the logs, change the tape, correct any issues that may have occurred possibly rerun a backup. And that's gone to be done within four hours by one person a day, just verifying that things are happening and correcting the stakes that occurred over the night. But time in thinking about the project you obviously made some big changes to processes. I'm not sure if it affected skill sets it probably did and it might have even touched upon your disaster recovery. If you had to do it again, what might you do differently? Or what might you advise your peers if they're moving down a similar path? I think one thing that caught us initially was even though FMR does a great job at backup once the central node actually has a backup getting that first backup on offsite is a time consuming process. We initially had tried out a few smaller offices maybe only two or 300 gig on site and it was taking weeks to do a first time backup. So we kept changing and saying, all right, let's buy some physical devices, encrypted devices, copy the data there, ship them to our central data center location and actually seed the data that way. So we actually did our first pass of not really a real backup. We were just getting a copy of the data into the Avamar so that when the on site node did kind of a check in check out process it actually could verify that the central node had most of the data already. Yeah, seeding is always a challenge. We talked to a lot of Wikibon practitioners about that and you've got to plan for it and you chose to use what I call the CTAM method, the Chevy Truck Access method actually faster than the pipe. So that's a planning consideration and something that you should actually build in to the project. Very much so. Go ahead, sorry. No, definitely, that was a big thing. And then after that there was understanding how the restore process is gonna happen is a big change too. That's something that everybody needs to really take into consideration. I mean, we talk about it all the time. The backup is really unimportant. The restore is the important part. So making sure that you get that nailed down is huge. Backup is one thing, recovery is everything, as they say. All right, what about your journey, Tom? Maybe talk a little bit about futures. Going beyond 2014, we talked to a lot of practitioners that are moving toward data protection as a service, as part of their IT as a service initiative, as part of their IT service catalog. Is that where you're headed? Talk about futures a bit. We've looked at it. Obviously, now that we've moved more to a central platform, kind of the data protection as a service is kind of a next step that's in consideration. We, being an accounting firm that we do a lot of analysis when we do pricing, and we've kind of kept our eye on the protection as a service and a lot of added service projects because we find this actually with a lot of our mid-market customers, kind of interesting is to a certain point, protection as a service, infrastructure as a service makes a great, makes a lot of great sense. And as time goes on, the prices get better and better and better, so larger and larger companies can make that investment. But right up until a couple years ago, the data protection as a service frontier was much more expensive than us doing what we've been doing in-house. That's finally starting to turn around, which I'm really happy to see. I'm expecting probably in the next three or four years we'll be seriously looking at that again. Or even replicating that in-house is really sort of what I was referring to. Essentially duplicating what you might get from an external sort of cloud provider internally. Is that something that you think you can achieve? We currently do have a replica of the Avamar off-sites and duplicating what they're providing. It's, again, we don't have necessarily the capital to build out that big of an infrastructure, so that we're kind of in that middle space again where we really have to figure out what's going to make the most financial sense and what the company wants to take on for risk and time. So what's next for you guys, sort of improving your processes and holding that for the remote offices? Well, next for us, we're looking at, obviously now that we are, we are about four or five years in now to owning the Avamar product and about three years into owning Data Domain. So we're actually looking at that integration phase and actually trying to merge together our, what we're doing from our office and what we're doing in the data center and see where those two can link up. So in our next, probably year, we're going to look at moving to that merged Data Domain and Avamar product. And do away with tape as much as possible. Because it's much easier, as I said earlier, litigation is a huge consideration. It's much easier to prove that we don't have the data when it's on a tape system. It's on a disc system, I apologize. We'll have to leave it there. Thanks very much for coming on theCUBE. It was great to hear your insights and your advice to peers and good luck with things going forward. Thank you. All right, thanks for watching everybody. This is Dave Vellante and this is theCUBE. We'll see you next time.