 I met Barry last night. I have not met Kai. All right, have fun boys. All right, thank you. So we're going to do a little case study on Mazda. Come on in, Barry and Kai. Barry, nice to see you again. It was great to meet you last night, hang with you a little bit. It was fun, wasn't it? I said to Callie, it's kind of hard to be in here without the kids, but we had fun at Disney anyway and at Epcot. Kai, how you doing? Dave Vellante? Dave Vellante, nice to meet you. So Barry, last night we were talking a little bit about your organization and so what I'd like to do is set it up and we're here at the Dell Storage Forum. This is DaveVellanteWikibon.org and this is our continuous coverage, Silicon Angles coverage of the Dell Storage Forum, which is why you guys are here. So Dell, Barry, why don't we start with you. Talk a little bit about what you do at Mazda and the organization, the IT organization specifically and then we'll bring Kai in and hear from him. Well, I'm the infrastructure architect. I report to Kai, he's my manager. I have a team that handles the infrastructure of four guys, me and three other guys that handle the virtual infrastructure, the storage, Active Directory, things like that. We have a networking team that handles all of the network and we have application groups, programmers, things like that. How many total in our IT department at this point? Well, the total IT is about 80 people and we support about 1100 users internal and about 15,000 users at the dealerships. So within our organization, Barry is part of my team. We are what we call the infrastructure services. So basically what we do is we actually work with the business application team in architecting the environment for any new business application projects. So you guys are across North America? Is it U.S. responsibility? Yeah, we support U.S., Canada and Mexico. U.S., Canada, Mexico, so the Americas? North America. It's Mazda North American operations. So North American operations includes Mexico? Mexico and Canada, yes. Okay, cool. So talk about a little bit about, actually Kai, I wonder if you can start with what is driving your business? When you're talking to the business heads or the application heads, what are they pushing you guys to do in IT? Zoom, zoom. Zoom, zoom, zoom. Hurry up before we go. Yeah. Faster, faster, faster. Since we are part of the global company, our parent company is Mazda Motor Corporation. So some of the business initiatives is actually driven by Mazda Japan and so those were considered global projects. And some of them are North American initiatives, how we would do business better, how we consolidate, and sort of provide the same systems and application to both Canada, US, and also Mexico. So other projects will be more US specific, as far as if there's any target market in certain area, there could be some application that drive that. And then of course, we also have our own technology projects where we have to keep up with the technology change and we have to be flexible and be nimble. We're not very big company, so the idea for us is really to be flexible enough to change with the business requirement change. Of course, in a company, we are very much aligned with the business unit, so we have an IT manager, application manager that's servicing their business partner and then from that they do through the business planning that become our projects and then they will come to us and then we put together infrastructures to support that. So it's very project driven with like a lot of IT organizations and it's business project driven, not IT project driven necessarily. Some of them are, some of them are technology project, for example, when we initiate the virtualization, those are IT projects. Okay, and we'll talk about some of those. Before we do, what's the biggest change in the automobile industry that you've seen in the past decade? Is it the globalization? Is it the move toward cost cutting? Is it energy efficiency, all of those? I mean, what is a bit of the big changes in your industry that are driving? Well, obviously in the past few years, with the gas price rising and with the downturn of economy, all the car companies going through a little bit of transformation and as a master, we know to make smaller cars and with fuel efficiency, fun to drive and we actually have technology that coming out that can help both the performance and also greater mileage, so we're really excited about that. So from the business transformation, it's really for us to actually kind of like uplift the brand image to be like, we're not like the Toyota or Honda of the world, but you know, we want it to be one of the choices that people would like to drive cars. Must just get cool cars, right? Right, fun to drive. So we, like I said, we're not as big as those big guys, so we know that and we're not competing with them in many respects, right? So the transformation was really how we strengthen the brand and we have been really focusing on the customer loyalty and as long as it's just like any other company, if we keep our customers happy, they'll come back for service and buy more cars. Right, right. Okay, Barry, let's talk about, let's get into the whole IT infrastructure, you're an architect, right? Spent a lot of time on storage, probably virtual infrastructure as well. Talk a little bit about your shop, what it looks like and paint a picture for us, if you will. Well, we've done a lot of consolidation down and into virtualization. So to get greater efficiency, greater return on investment, it's been a very good model for us as it is for a lot of people. We needed to simplify because as I said, my infrastructure team is four guys to handle it all. So we had a very complex physical environment at one point, like a lot of companies did, trying to maintain all that environment, trying to back it up, restore it during testing and stuff like that. Disaster recovery test was becoming very, very difficult. Also, all of our equipment was leased. So we had this lease return schedule. So the benefit was we always had new machines, we had new Dell servers coming in all the time. But for the business initiative, they wanted more and more applications so it meant more and more servers. So every time the lease cycle came around, it was time to replace, our team was scrambling to replace dozens of servers. And what was the storage at this time? What were we talking about, three or four years ago or five years ago? What's the time frame? 2005. 2005, so what was the storage on the servers? Was it the direct detach storage? We had iSCSI, we were an early adopter of iSCSI. A lot of our databases were on iSCSI Lens. So we tried to consolidate and that was an effort to make it easier to maintain. So we would have the centralized sand storage, iSCSI connections, we'd be able to maintain the storage that way rather than having distributed completely into all the local servers and stuff. The reason I'm asking was a lot of customers I talked to when things come off lease, it's like, oh no, we got to migrate data. It was becoming so difficult that we were not meeting those, we were leasing servers long beyond their lease expiration. Okay, so that's not really the ideal financial strategy. And obviously it's not unique to us all the other companies are going through the same thing. So as you, not only for lease replacement, but as far as how do you recover those systems, right? The backup recovery of it and how do you do DR? And so all things combined, it just makes good sense for us. And the idea of doing more with less, which has always been the key. And so we just have no choice. We couldn't handle all those physical servers. So we set the strategy to go with the virtualization and we're just moving forward. So at that time, you said, okay, we're going to go server virtualization. There's a lot of customers do as you said, Mary. But at that point, did you say, all right, we're going to go storage virtualization and synchronize that or would that come later? That came a little bit later. A lot of storage vendors were promising virtualization, but some parts were virtualized somewhat. We're here at the Dell Storage Forum to talk about compelling because we implemented compelling. We feel that is truly virtualized storage. We do too. We've written about this extensively. We use that as a poster child. Others, I mean, three-par, compelling. I mean, those are sort of the clear, built from the ground up, virtualized architectures. So when we began virtualization, we ran into a lot of the same problems that other people did as we migrated more and more machines from physical to virtual. We began to run into certain performance bottlenecks, both on storage, networking, servers, the backup window, a lot of IO going through the ESX host, stuff like that. So the compelling storage, we looked at that as a way to solve some of those storage problems. We basically re-architected our entire virtual infrastructure. Our chosen platform is being VMware. We use the compelling storage and we upgrade our network to 10G. Because of the nature of how big of our organization is, we wanted to keep it very simple so that we could manage it. So we didn't want to do anything fancy with the raw device mappings or anything like that. We wanted VMFS volumes and just VMDKs. VMware was promising that the performance for even tier one databases would be fine on VMDKs and we took them at their word and we implemented it that way and we found that the performance was actually better than our physical environment that we had before. Really? Yes, yes. Oh, that's interesting. And this is for critical application like SAP? Yes. So you guys are virtualizing SAP? That's correct. We've already virtualized it and depending on the type of transactions we're doing in our SQL servers for the SAP, we have performance gains of anywhere from 80 to 400% in our virtual environment over our physical environment that we have. Have you virtualized Oracle yet? We don't really have any Oracle. You don't have Oracle Shop? No. We have a lot of clients out there who want to virtualize Oracle but it's not necessarily the easiest thing to do because Oracle doesn't want you to. You don't have that problem. So that's cool. SAP is very supportive of VMware. Yeah, they were very supportive. We actually talked to SAP representatives who gave us a thumbs up for virtualization. So yeah, there were quite a bit of concern especially for application, critical application like SAP. So obviously we had to reach out to partner like SAP VMware and to try to get them to kind of give our management a little warm and fuzzy feeling that say yes, they will support SAP under virtualization under VMware. Yeah, so in Dell is a big part of that too because you have the hardware platform and they can, Dell can attract in in SAP or VMware and get the attention, you know, but it must have been harder at the time for a compelling to do that. Now compelling obviously wasn't part of Dell when you chose compelling. So you kind of stuck your neck out there, didn't you? The new company, small company of Minnesota, they're not in Silicon Valley or Boston. That's true. Actually, the transition hasn't been that difficult, okay? I mean, we've been on a different storage system for the past seven years. And obviously one of the concern I have for various team, because since he only has like four guys, it's really the learning curve of how you manage a new storage subsystem. But, you know, the compelling is very different. First of all, the architecture itself and the software that they have to manage their system is very user-friendly. And so, but what really kind of help us decide or sell it to our management is really their co-pilot. Okay, so. Talk about that a little bit. People, people have been talking about co-pilot this whole week. I mean, that, that what's, I can't say enough good about that. You can't say enough good things about co-pilot, because, you know, we did, we did an EBC visit their headquarter in Minnesota, and we actually met some of the co-pilot folks and we actually see how they handle the calls and the fact that it's great that when you call support, you always get a live person on the phone. But it's a 24-7 program, right? And they're not just dispatch, they're engineers that actually know our system. In some cases, more than we do. And we've noticed that even late at night, if we had to call, the graveyard shift support technician was just as knowledgeable as the daytime shift. The quality of knowledge doesn't degrade. Doesn't degrade. At midnight 3 a.m., really. And that's really, you know, it's like, I guess, bar none, number one, the best. Yeah, so. It's a unique program, isn't it? It is. I mean, you know, everybody's got, these days, some kind of support 24 by seven, but not super high quality engineers, you know, waiting for you to call. That's right. And they do stay with, you know, if you do need some help, they do stay with us from the start to the completion. I wonder if we could get some of our consumer companies to take that kind of service. You know, like when you call a bank and you get put on hold. You call in the airline and you can't get a live person. Well, co-pilot would be a great method. Yes, yes. Okay, so, you know, Kai, you were talking about, you were concerned about the whole learning of a new system because a decade ago, if you went from, let's say, an IBM to an EMC, it was a whole new learning experience, wasn't it? And so you didn't have that issue with Compellent. It was sort of, I mean, there had to be some learning curve, right? I mean. There was a learning curve, but it was very quick. I've heard people say that even the help desk could run it, but don't, don't call me. Fantastic. I was gonna say, it's so simple. The manager can do it, but he doesn't. Yeah, no, don't touch my sand. Now, you guys do an iSCSI, right? You're not doing any FCOE. Yeah, we were at iSCSI shop prior and in the physical world. So, as we moved into the virtual world, we were very comfortable with that technology. We decided we didn't need Fiber Channel, any of the complexities there. We decided to keep it simple and utilize our existing physical network for that. And what were you doing before? Was it, you had a, what was the storage before, a compelling? The model or the? Was it a sand? It was a sand. It was sand, yes. iSCSI sand. iSCSI sand. Okay. So we were early adopter of the iSCSI Sand Network and it's been working well for us and simplified a lot of things. There's no Fiber Channel or HBA that we need to bring in for every server and also there's no sand switch and, you know, all the management that goes with it. So simple, right? Exactly. We kept it simple. So when we went to the virtual infrastructure, we kept it simple again. We didn't use any iSCSI initiators. We just used the software-based built-in to ESX. Kept it very simple. There was a lot of people out there saying, oh, well, your performance might not be as good and we have not. Well, that's a big concern about iSCSI, right? I mean, I talked to a lot of guys in FC land and they say, hey, if you're going to put mission critical data on the storage array, it's got to be on FC. I'm not going to do anything else. What would you say to that? Well, we have not experienced any problems at all. For example, our SAP environment, very large databases, terabytes databases and a lot of transactions going on and we don't experience any, like I said, when we went from physical to virtual, we had, depending on the transaction type, we increased our performance 80 to 400%. Yeah, so I mean, this really, it's almost like politics when you talk about, you know, protocols. So that's good. They were paid on the OSI model. Now, did you quantify the impact of the move to Compellent? Did your management want to see that? Do you use like the, you know, the hero report inside the Compellent? If you use that at all, the one that shows you the allocated versus written and, you know. We use some of that. We use some of that, but the greatest impact was actually the users themselves. The user experience was better as they would do operate transactions in their SAP GUI. They would, they were reporting a much better experience themselves. Because of performance or? Because of the performance was there. Right, yeah. And then their batch job, the nightly batch that process a huge amount of transaction or the accounting process that goes through that system. They actually, you know, that's where we see the biggest performance improvement. Did it affect your backup? You know, did you change the way in which you back up? I mean, both the virtualization of the storage and the servers? We re-architected the entire thing. Soup to nuts. Can we talk about that a little bit? Because backup is a real pain. Well, let's talk about what our backup was prior. In the physical environment, we were very traditional. We had the physical server. We did have the iSCSI lines. We used some certain storage technologies to back up at the storage level, but we also did agent-based backups, you know, streaming backups, things like that. So it was very, and as the data grew, the backup times increased. And pretty soon, the backup times were overlapping with production hours. And your backup window, I'm presuming, was nodding. Creasing. Yeah, so it wasn't. No, in fact, as those processes increased, we had more need for more batch jobs to occur at night. And the batch jobs were interfering with the backup schedules, so they were competing for time. So with virtualization, we decided to look from scratch and we decided, let's virtualize all of these databases, these applications. Let's look for a backup solution that is geared specifically for VMware. So we did that. So we chose Veeam as our backup solution. We wanted a backup solution that would... What software were you using before this? Was it a traditional? Just traditional agent-based backups. Yeah, right. Yeah. So it's an agent-based Veeam. And you're backing up the tape. Backing up the tape. Yeah. So we changed it. We backed up the tape. Yeah, Veeam does a good job in VMware. I've actually... I haven't talked to them. Are they at this event? I haven't seen them, but I saw them somewhere recently. I think they might be here. Maybe it was a Veeamug. We've been very busy. We haven't been able to see much. You guys go to Veeamugs? You go to the Veeamugs? I've been to a couple, yeah. Yeah, they're good. And Veeam has a good presence. Okay, so they've got... So you're using Veeam to do the backup? So, again, when we simplified our storage and our VMware, we just used Veeamuf's volumes and VMDK's, that's what Veeam requires. So that all works serendipitously together. So we back up to disk now to Dell NX3100 NAS device. And that's D-duped? Which is... Is that D-duped? Is native D-duped in the Veeam software? Is it client-side D-duped? Yes. So it's client-side D-duped, right? Yeah, Veeam does it, yeah. Client-side. Yeah, so you're... Because presumably your IOs explode when you go to server virtualization, right? Yeah, exactly. And so you're reducing the load and the network by D-duping at the client-side. Is that correct? You see that effect? Oh, we're getting performance numbers that Veeam is excited about. So with the backup with the Veeam, the way it works is it utilizes the VMware snapshot capability. It queues the IOs on the SQL database so that we have consistent database backups and that was critical for us, especially for our SAP critical data. The IOs queues very quickly, the snapshot's taken, IO resumes. The system is immediately back online to do any nightly batch processing or production or whatever. In the meantime, the snapshot is being backed up by Veeam to disk and at high compression rates and at very, very high speeds. We have stellar performance numbers. That's all local so far, or no. Yes, it's all local. So far, we're talking just local. Right, back up to disk. And then you shoot it off-site? That's plans for the future. We're planning on taking the NX3100 and maybe using VFS or something to replicate that to an off-site location. So just to give you some of the performance number that we were seeing, our SAP database, it's about 3.4 terabytes in size. And with the incremental backup, Veeam takes the snapshot and that takes what, 35 seconds. And then the incremental backup itself take about between, depending on how many changes that happened throughout the between cycles, the backup, the real backup actually took from between 20 to an hour and a half. And that was it. So we're here with Mazda, we're talking about a case study and we've entered the backup realm, which is always, to me, the backup is the biggest pain, right? It's always been the biggest pain point in our industry. So have you been able to dramatically change the whole backup window pain? I mean, it sounds like, but you haven't eliminated the backup window concept, right? I mean, let me ask it this way. So you're taking, are you taking continuous snapshots? No, we're still maintaining the single. So it's daily incremental and you're doing weekly full? Weekly full, yes. And do you do anything else special for your mission critical stuff? Like do you take a daily full for SAP or for instance? No, we haven't had the need to do that. No, okay, so it's daily incremental, weekly full. Are you doing any archiving of data? That's at the application level. Yeah, we do. Which might reduce the amount of data that you have to pack up on the weekly full. Now, as we move towards the co-location, then we will also use a beam replicator and the instant recovery that beam provides. Yeah, we'll be replicating our tier one applications to another VMware cluster at another location. What do you think about this notion of taking CDP, taking snapshots every 15 minutes? Is it just overkill for you guys or do you think that has potential or what do you think? I think that has potential. I think we might be implementing that in the future for our more mission critical applications. So, do you think that you can start to, because most of the customers I talk to, and you kind of fall in this category, I think, but actually, I should test that before I say this. But basically, most customers take a one-size-fits-all approach to backup. This is our backup, this is the service level that you're going to get, and we're going to apply that to all the data. Is that what you do? Currently, that's what we do. Because there's not really a better way. Right, right now we still have a mix of old technologies and new technologies, but as we move everything over to VMware, we are beginning to use the Veeam product for that. So, the one-size-fits-all, as far as that's concerned, we want to get down to a single backup solution as we virtualize 100%. But based on the application, the criticality of the application, or tiering or stuff like that, we might employ different service levels. So in theory, you could use the a compelling system to take space-efficient snap shots, if I could use that term, right? And then do some kind of CDP, whatever, 15 minutes, half hour, and then take a copy, shoot it off-site at your convenience. Am I thinking about that the right way? I think the future will include some of that. Right now, like I said, we're not using RDMs or anything like that. We're keeping it very simple. We're using just Veeam as our backup solution. We're not using the snap, the replay technology and compelling for that. Replay is what they call it, right? We do, we have used it. It was very critical for us in upgrading our SAP environment. So we had- Oh, when you're doing the migration. Right, when we did the migration, we had 4.6, and we went to ECC 6. So, or ECC- 4.7. 4.7. ECC 6, right, sorry. So, to get from one to the other, it's not just build a new one and migrate your data over. There was various stages to upgrade both the operating system, the versions of SQL and SAP. So to do that without risking our data, we used the compelling replay technology to create replays and then create views where we can mount those views in the new servers and perform the upgrade of the databases and things like that. So that was critical for us to get from SAP old version to the new version. So from an architect's perspective, to do what I'm talking about, there'd have to be some kind of integration between the Veeam and the Component, and you really want to rely on the Veeam to do that because it's sort of controlling the whole backup. Because of our simplistic model that we're trying to maintain, we're using the Veeam. We chose Veeam as a software-based backup solution as opposed to a hardware-based backup solution. It allows us the flexibility to use different hardware at different locations. However, with Compelent, we're so happy with the product, we're going to be using Compelent at our remote location as well. But the idea early on was, the Veeam would allow us to back up and replicate that data to other storage if we wanted to. So some of the thoughts there were, as when we first brought in the Compelent and when we put together the Veeam backup solution, and today we have the Colesite subscription for DR at the Colesite facility. And so what we were kind of talking through was the what-if scenario. What if they don't have Compelent on the other end? So that's why we chose to use the software-based replication so that we could, even without the Colesite facility, we could do the Veeam replicator to the hot sites. Yeah, it's going to give you more flexibility. Exactly. Regardless of what they have on the other end. Yeah, you love Compelent, but you don't want to get locked in, and it's always good to keep your options open. Our ultimate goal is to have our own Colesite facility in another one of our offices, and our plan is to put Compelent in there. Okay, so that's going to extend into your DR strategy? Yes. All right. All right, good. I've been thinking about Dell and watching their moves of bringing in the file company Compelent. Obviously they're equal logic, and backup is an area that I think is the next untapped area for these companies to really solve. I think Dell could really add some value there. I don't know, what advice would you give Michael Dell in that regard? Well, I think he's taking the right approach, and I see his acquisitions. We were very happy to find him acquiring Compelent, because we thought that was a great match. We were already at Dell's shop, we had Dell servers and stuff. Then we already had the Compelent and the marriage of the two, and officially like that was great for us. We were really pleased with that. Yeah, I mean, it's hard, right? These small companies, they come out, they innovate. We say we need more startups, but it's hard for IT to get, adopt some of these technologies sometimes, right? Right, because you know how long, in some cases when economy goes bad, I don't, these little startup could go away. So sometimes they do have the great technology, but they don't have the big company funding to support what they're planning to do as far as technology vision. Yeah, it's a real trade-off for IT, because you want to innovate, you want to simplify, and it sounds like it's had a big impact in your business, particularly in terms of the complexity of managing the systems. Absolutely. And you grow, are you growing fast? I mean, are you going data like crazy, like everybody else, or is it? Yes. So that means if you don't do something about it, Kyle, you were saying do more with less, you're going to have to hire people and brute force it, and that's not a good solution. That's going to make you less competitive as an organization. Right. You know, and you're in a very competitive industry. Exactly. Good. I'd like to keep ourselves lean and mean. It's all about Zoom, Zoom, and Monster, so. So yeah, Zoom's out of the blue. I've got to throw that here. So I wonder if you guys could, my final question is folks out there looking to whether it's virtualized servers, virtualized storage, simplify their infrastructure. What advice would you give them from the standpoint of an architect and then from the standpoint of an executive trying to sell this concept to his organization? Well, as an architect, I try to look forward. I try to see what the future is going to be and I try to look for solutions now that will play into that future role. So moving forward, I want to keep the key thing, simplification for our environment. The complexity that we had before was very hard to manage. And as we simplified it, it becomes easier and easier to manage. So I could get sleep at night. I don't get calls in the middle of the night. My guys get sleep at night. One customer said, I got my weekends back. Yeah, exactly. So that's from an architect's perspective. Yes, keep it simple and look at your acquisitions and technology with an eye towards the future of how it's going to play down the road. Okay, we'll give you the last word. Any closing thoughts? Okay, well, from my point of view, I think the very key to our success is actually to gain the management support at the very early on in the project. Okay, so we've been very fortunate that we have our CIO that's very much in tune in what we plan to do. And he's actually attending our planning session and the strategy sessions. And he agreed with the idea and he fully supported. And that is, to me, that's like 75% of the battle, right? So because once he endorsed that and then all we have to do is just set the strategy, put the plan together and just start executing a little step toward that plan. Do your jobs, do what you're good at. That's it, that's right. And then our CIO will take care of the rest when he'll go to the executive committee and get the funding to support what we want to do. And we make it worthwhile and we can prove that we can do more with less. You guys seem very relaxed, you know. You see, all the time you see stressed out IT people. It's easy to talk about these products. Good, they're great. Kai Sukwongzi, am I saying that properly? And Barry Blakely from Mazda. Thanks very much for coming on theCUBE. It was great having you guys. Thanks for having me. Good meeting you last night and nice to meet you today, Kai. Thank you. Take care, we'll see you guys later.