 The Cube at EMC World 2014 is brought to you by EMC. Redefine VCE. Innovating the world's first converged infrastructure solution for private cloud computing. Brocade. Say goodbye to the status quo and hello to Brocade. Welcome back everybody, we're live here at EMC World 2014. It's a big event, over 12,000 people here. We haven't had the official number yet, but I'm here with Steve Keniston. This is the data protection and availability, drill down spotlight that we do each year. We go deep at EMC World, bring on technologists like Steven Manley and practitioners really to find out what's happening at the front-lining. Greg Würzberger is here. He's a systems engineer at Sub-Zero. Greg, welcome to the Cube. Thank you for having me. Great to see you. Sub-Zero, those cool appliances that everybody wants. But tell us a little bit about Sub-Zero, what your role is there. Sub-Zero is obviously a manufacturer of very, very high-end both cooking and refrigeration appliances. Been around since the 1940s, and we actually brought in the wolf appliance and the cooking line in the year 2000. My role there is a systems engineer, so I'm basically in charge of everything involving servers, storage, backup, data protection. Been there for 14 years now, so I've got my feet pretty wet in that area. So how would you describe the evolution of Sub-Zero's IT environment in the past 14 years? It's actually pretty large. I mean, originally the size of the entire IT department was about eight people. We now have a staff of over 40. I think we're at about 45 now. That includes not only the systems and network people, but also our application developments and our system I, or I'm sorry, our I-V-I-M, I-V-M-I area. So what are the big systems or applications that you're running? Well, we run, we're a Microsoft shop, so we run a lot of SQL, we run a lot of Microsoft Exchange, that kind of stuff. We do, our ERP system is a system I, or an I-V-M-I, so we're running that in 4XA to do our ERP and our item inventory. Okay, so talk a little bit about how your data protection environment has evolved over the past, probably not 14 years, we don't have to go back that far, but let's say in the past five to seven years. Yeah, about five years ago we were running just a semantic net backup environment running on an SL500 tape library, 150 tape capacity. Extremely effective at the time. However, with the increase in storage and the increase in things that we had added to the systems over the years, it suddenly went from adequate to completely inadequate in a matter of a few years probably. So we ended up getting into a situation where our backup system was not able to handle what our current load was as far as data protection and we had to come up with a solution to find a way to fix it. Why was it inadequate? Was it, you're not hitting backup windows or just not allowable? Or basically you would say it was semantic, so writing all that to tape. Right, it was a little bit of everything actually. The environment had expanded so rapidly. I think we went from somewhere in the vicinity of two to five terabytes of space to 20 terabytes of space in a matter of two years. So a lot of data just came flying in. Not to mention we had implemented virtualization with VMware during that time. So we went from having, I'd say, 20 to 30 physical servers up to close to 100 virtual servers in a matter of years. So once again, we just completely ran out of space. The other problem was that we were unable to anticipate what we were gonna have from a file system standpoint. So that also increased in space. The SIFS backup that we were trying to do, it just couldn't handle it. We had a situation where if I was lucky enough to have a 72-hour window, I might get a full backup. So it was just, it made me never get a 72-hour backup. No, it was, yeah, maybe Memorial Day got lucky. So if I asked you what was keeping you up at night, it would be doing backups. Yeah, it was literally. Yeah, it was, did I get one last night or not? The other thing kept me up at night is knowing that the next morning I was going to be putting in anywhere between 20 and 40 tapes, just to handle maybe a day or two's worth of backups and having to rip out the other ones, put new ones in. It was using up, I think at the time we saw, it was using up eight to 10 hours of my week. Oh my gosh. And you were, now, talk about virtualization at the time. What was the virtualization strategy and how fast were you virtualizing? We actually, we implemented our virtualization, I believe, in 2008. And it went from one or two systems to we were at 75% virtualized in a year and a half. We actually, the ROI on our virtualization project, we hit it in less than three months. So it was a huge flip. We had a whole bunch of servers that were ready to just die and we just flipped them over. So did you transform your backup coincident with that sort of virtualization ascendancy? No, we ended up just kind of using what we had for a while. We didn't finally get around to implementing the Avamar system into 2012, but that was my, I believe, third attempt to get that system brought in-house. That must have really, I mean, that level, that rapid pace of virtualization must have put even more stress on your backup. It did, yeah, absolutely it did. Because now, from our point of view, we didn't really have a virtual backup strategy. It was more of a physical turned virtual. We had, we treated them as if they were physical. Without as much physical. Exactly, exactly, so just couldn't do it. Okay, so then you bring in Avamar. What was that like? Talk about that whole process. We brought in Avamar in March of 2012 and it was probably one of the most successful implementations that we've done. Probably right up with VMware. The system was brought in-house in February. It was brought up on March and by the end of March we were in full production, full backups and I was already starting to turn down our net backup solution. You were at Avamar at the time, right? Yeah, yeah, so I have a question. So you had said that this transformation took place over quite a period of time. You said Avamar actually replaced data domain, some data domain that you had in there. No, no, I'm sorry, the Avamar came first, the data domain was brought in later. Data domain was brought in later? Yeah, we brought in the data domain mainly as a virtual tape library substitution because we were just doing, again, tapes through our IBMI. So once I brought in Avamar and removed tapes from that environment, my poor system engineer that handled all the IBMI stuff, all of a sudden he's the one that's dealing with tapes left and right and he started to get a little cranky so we did find out the data domain would do the VTLs and we're able to bring that in and eliminate tapes from that as well. So now you have a hybrid environment? Yes. Yeah, we use the Avamar as our main backup. However, our virtual tape libraries as well as a number of our SQL backups go on to the data domain. And then we also have matching grids in our main facility is in Fitchburg, Wisconsin. We also have a facility in Goodyear, Arizona and we have matching grids on both sides replicating back and forth. So post bringing in Avamar, talk of give us the before and after. I mean, your RTO was a mess. Right. Your RTO was a point toss. Pretty much not existed. So what kind of business results did you see? Well, within, the biggest one that we can actually show is that our NDP based backup or SIFS based backup was the one that would never finish. We just couldn't do it. I went from not being able to finish that at all to obviously beyond the first backup which takes a while. Once that came through, I now see those things back up in a matter of an hour, maybe two hours on a bad day. So I'm getting full backups and some of the upgrades that the Avamar's done with the newer versions of the OS and that and the ability to proxy within the virtual environment. I'm now seeing my backup window shrink even further. We got it down to under 12 hours. We then shrunk it to eight and I'm down below six. So that's an incredible change. And I heard you guys talking about data. Then subsequently you brought in data domain? Yeah, we brought in the data domain a little bit later. Mainly because the focus of the data domain was for that virtual table library ability. Yeah, okay. And then the SQL was a nice little add-on. Okay, so that's a different infrastructure. Are you bringing those together? Are you trying to bring them together? Are you trying to push EMC to bring them together? Bits and pieces we are. The one thing that we did push over to the data domain is our exchange backups. So we were trying to do those on the Avamar and it just, it worked fine, but we were able to actually decrease the backup window by another hour by moving those backups over to the data domain through the DDBoost. So, but for the most part, yeah, the data domain and the Avamar kind of do their own separate realms, but it's worked very well. What would you like to see, so as Dave points out, right, and I think we've seen this through a number of different folks coming up here and talking to us about their infrastructure and how it's transformed and how they've had different products to solve different problems. Anything you'd like to see from EMC to be able to either pull these together or make life simple? I mean, what's missing from that overall picture that will make your life even better? Well, I mean, some of the things I can think of is that you're seeing some of the really cool additional abilities added to the data domain that aren't necessarily added onto the Avamar side. I'd like to see some of that come along. I know that one of the neat things they're working with the data domain is the ability to actually bring up a virtual backup, bring it up online, and be motioned it over, or be storage motioned it over live. Just stuff like that, just being able to meld those two together, it would be really cool if it actually became a single environment with a single pane of glass to look at both. Right now, they've kind of got there, you've got the single dashboard now with an Avamar, that's a much, it's vast improvement, but nice to see that get to mesh a little bit better. So what would be the one thing you would give advice to your fellow co-hosts, your fellow practitioners about if you're going to make this type of a transformation? Hey, heads up, you should probably pay attention to this. I guess the best advice I could give them is that you got to know it's going to be a little bit slow at the beginning. I mean, obviously the initial backups take quite a while. If you try to implement replication, the one unfortunate side effect of our replication is that we weren't able to see the device prior, so we're sending across the country, so that took a little while. That took a long while, but it did actually eventually start working, so it's just stuff like that, just know that it's not going to be immediate, but you're going to see some immediate results that you wouldn't be able to do with your current system. Once that initial seeding is over, you're kind of on your way. Right, right, and now the best part I have is you talked about not being able to sleep at night, I sleep like a baby now. Every morning I get in and I look and everything's gone through. If there wasn't a minor problem, it's usually, I have to say it's usually user end, they shut down a system when they weren't supposed to, or anything like that, but for the most part I get solid backups every night. All right, we got to run, but I'll give you the last word, EMC world for you. You know, what's your takeaway? This is my first one ever, so it's actually kind of cool to be here. It's neat to actually see what all you guys are doing, some of the new technologies that haven't even been released yet that they're seeing, especially in the data protection and the storage migration realms, some really cool stuff, and I'm looking forward to seeing it in the real world. Excellent, Craig, well thanks very much for coming on theCUBE, great to see you, and we'll hopefully see you next year at EMC world, or VM world, or wherever theCUBE is next. All right, keep it right there everybody, we'll be back with our next guest right after this. This is theCUBE, we're live from EMC world 2014.