 Okay, welcome everybody. Today is September 4th. We're here in the Wikibon offices and we're going to be talking today about continuous, long-distance data replication for continuous availability. Before we get started with today's peer insight, I want to remind everyone that you can tweet to us at Wikibon. We'll try to monitor that and answer questions. You can send us questions that way. We'll also pause throughout the call today to poll people for questions. Sometimes there's background noise, so please smute your line by pressing star 6 if you're not asking a question. That will dramatically improve the audio for this call. Thank you, everyone who's doing that now. And as a reminder, we are recording this call. It is streaming live now on SiliconANGLE.TV with a slight probably one or two second delay. If you do watch on SiliconANGLE.TV, please, please turn the, please mute your line so that we don't end up with a loop. We'll get an audio loop. So with that, let me, I'm really excited about today's peer insight. We have the CIO of Resolve, who is this Robert Reeder and he's joining us here in the Wikibon office. So thank you very much for being with us today. Thank you, John. I'm excited to be here. Thank you. So the Resolve Group is in the educational services business, but why don't you tell the audience a little bit about Resolve Group and exactly what you're doing? Sure. Resolve Group is really a company that's dedicated to two things, helping families plan and find a way to afford, plan for and find a way to afford college. We help them with filling out forms, in particular the FAFSA form, as well as identifying deadlines and getting information about colleges, how much they're going to cost, how they can afford what their payments are going to be. And then on the flip side, we work with institutions to demonstrate affordability and accessibility to their college. So in some ways, we're like a matchmaker. We'll put colleges and students together and make sure they're right fit for each other. But the idea is that when you have your college experience, you don't get out of college and say, wow, I didn't know it was going to cost that much. I didn't know it was going to have this kind of payment. You know all that up front and you figure out what you can afford. And then we're also going into the area with businesses that have tuition assistance programs. Those shouldn't be left out there, too. We're also working to bring those into the fold so that everybody can make informed decisions. And the Resolve Group delivers its services as an online, is it mostly online services? It's actually pretty much split between online and a phone service. We do have a website where people can go and either fill out the form online or just sort through all the information we have and do research. But we also have a telephone call center, a very large call center that really counts for about half our business, where people can call up. A lot of people feel a lot more comfortable about this talking to somebody and saying, well, I've got this unique circumstance. Does that matter? And our trained advisors can say, yes, that does matter. No, it doesn't. Well, let me get a little more information on that. And then they can walk you through the whole process or just give you other advice as well. So these trained advisors are working off of systems that you're responsible for maintaining. That's true. We built our own system in terms of data entry and information that they work off of. And they go through about a six-week, full-day, six-week training program, and then a period where they're observed when they're on the phone. So it's an extensive training. We've got Stu Miniman here with us, who is a Wikibon senior analyst. And I know we're both very much interested in sort of the IT infrastructure that supports your services here. Can you give us a little idea of the infrastructure you've got? Sure. We actually have, I like to separate it in terms of our external and our internal infrastructure. Because on the internal side, we actually have four offices in the U.S. Boston, Indianapolis, Lawrence, Kansas, and Sacramento, California. And two of those offices in Lawrence, Kansas, and Sacramento, California are actual call centers of comparable size. And yet one of the requirements in our operations is that those two call centers operate as one so that no matter who's calling, we always can have somebody available to help you. And sometimes that might be somebody in California, somebody in Kansas. So the network infrastructure to connect all those four offices is quite a challenge. Because as you know, as if I'm a user in Boston, I will expect to get my file service, my email service, my phone service, just as well as somebody who may be sitting right next to the hosting facility where all the servers are. So that's one infrastructure. And those offices are all connected through 100 megabit fiber. Okay. And we're actually putting in place redundant 100 meg fiber. I mean, we're all benefiting from the reduced cost in telecommunication for bandwidth. And so we decided that we've now put those in thinking, wow, that's great bandwidth. But then when you look at the utilization, and you're using like 30, 40 megabits daily between all the video conference and, you know, we have video conference in the offices, you realize I got to have something in backup. Because if that goes down, that office is really dependent on it. What happens if you go down? Well, right now we actually have fat pipe units, they're very sophisticated routers that support multiple WAN ports into your single WAN in each office. So every office has their own VLAN. And the fat pipes can actually establish VPNs across the internet between all the other units. So if say we had a 100 megabit go down in California, all the traffic would be seamlessly switched over to our backup circuits, which right now are MPLS, but we're switching to the 100 megabit fiber. Anyway, it was switched to that. So that basically, since we've had the fat pipes in place four years now, we've never had a network outage between the offices. Now, clearly if an office totally loses power and, you know, the office is offline, well then it's offline. And then the other office, let's say it's a call center, would start taking all the calls and have all the... And they would have all of the data up to date? They have all the data. Well, that's the data is all stored in our hosting facility. So now we have two hosting facilities. Okay. Okay, the data is not stored. The only data that's stored in the local offices would be local backups of laptops, you know, your own personal files, things like that. All right. So you went to, and you went recently to an active, active sort of scenario for disaster recovery. Is that right? Right. Or for failover? I would say in all honesty, we're in the process of... Okay. We're still in the sense testing. We're not live and production the active. Okay. But we're actually doing testing and working on that, which sometimes the testing is gives you a lot more information than going live. But yeah, so we have a hosting facility on the West Coast, which is where all of our activity, if you will, happens, all our email, all our database activity, all our... That's the only entry point on the West Coast where our consumers can come into the website. From all over the country, that's where they go. Right now, that's where they go. That's our primary facility. Okay. And that's sitting in a major hosting facility. It's actually in on the network. So, you know, we have a large amount of bandwidth, large amount of burstable bandwidth. So, we can take as many calls as we need or as many hits on our webpage. So, they all come in there. But then we have a disaster site on the East Coast. And in the past, that's really been where you store all your backups, where you could bring up another site if you needed to switch the DNS over there. But there would be some downtime in doing that. What kind of downtime do you think you would have? I think at best, you'd probably have a few hours, you know, three or four hours. Best case. Best case. At worst case, you'd probably have 24 hours. Worst case, meaning that the disaster was such that you had some issues transferring the latest data to that site as part of the disaster. Right. And therefore, you may have to do some work to get that site back up and realize, well, what's current? What's accurate? What's not? We had, we had a guest on from Animal Health International back in April, I think it was. And he said, if I lost data, I basically couldn't get it back up. Just trying to find all the transactions. Getting application consistent snapshot was one thing, but then trying to restore the changes and things like that was almost impossible for him. What's your experience with that? It is impossible. All of our data, I mean, when you think about it, all the data we do is in the database. The advisors when they take calls are working in a database in, you know, an application that is instantly storing everything in the database. The users come online, everything's in the database. So we do not, if you lose even an hour's worth of database, you have lost the work that happens in that hour. Yeah. And how many transactions or updates would you do in an hour? At the transaction level, that's a little hard to speak. So it's happening. But we can take, we've had as many as 5,000 unique applications in one day. We take, you know, probably 350,000 applications a year just on the FAFSA alone. And then you look at some of the other data that we're doing in terms of reporting and generation and stuff we're doing. You know, right now we're testing, for example, the replication between the two sites, SQL Anywhere, we're testing it like the minimum load test would be, we blast in 100,000 transactions as fast as the database will take it and then look at the delay on the other end. And we're very excited because the delay is only five seconds. That's cool. So Robert, you have a setup that's, you know, before pretty typical to most customers. So you've got two locations, you've got 3,000 miles between them. You know, that's a lot of latency to be able to have, you know, it's not going to be synchronous ever. But you've got active and passive. So you've got equipment that's not being utilized. And, you know, there's a lot of reasons why people didn't just have two active sites. I mean, you know, cost, distance, you know, the amount of bandwidth I need. Can you walk us through kind of what was the decision to be able to kind of move from your old environment to where you're looking to go? And, you know, what technologies and what things were you looking at that now you believe you're going to be able to move to that kind of active, active environment? Sure. Sure. Great question. It was a, it was a, and I want to say it's not a, this is not something that happens over a few months. This is a multi-year path that we walk down. Although the technologies are there now, we could have probably done it a lot quicker. But we realized very early on when we started growing rapidly. And we've had, you know, 30, 40, 50% growth for the past few years. And that puts a lot of challenges on the IT infrastructure. You start with some bare metal machines that you think, oh, these are, these are pretty bad machines. These have got a lot of power to them. And then in a few years it's like, well, that's not near enough for what we need. But one of the things we realized early on is as our database size grew and the amount of data grew, the time to recover that data, to reload it into another database, to move it to another server, just the time to move it, was also growing at the same rate, if not faster. And that was making recovery times, you know, like your RPO. It was really stretching it out to where you had to start saying, well, if this machine, this bare metal machine just died, you know, and we needed to reload the database somewhere else, there would be several hours. And so you started looking at mirroring databases or, you know, some people would look at database farms. But all of them had their issues until we had to move into a virtual environment. Because one, when you're not working with a sand as we started small, that was an issue. And so as we said, okay, the first thing we have to do is get into a virtual environment with a sand, so that we can now move between machines and not be locked into a single failure. But then in terms of networking, we said our disaster site really is still offline. It's just a backup vault in a sense to bring it up. What we'd like to be able to do is push that data over in real time to the backup site and be able to just switch our DNS to the backup site and have people come up if we had a major failure in our primary hosting facility. And as we started to look at that, to be honest with you, making it, if you will, a warm disaster site had a lot of challenges because, okay, how do you cut over to the database? And when you cut over, well, now that needs to be your primary site. And your primary site now needs to be your backup and you need to get it in sync. And how do you make sure all the files go back and forth? And once we realized, I guess I should say that we sit on, and our hosting facilities are on two major pops. And so we were able to get at a reasonable price, a one gigabit connection between layer two connection between the two sites. And we said with that kind of performance, why couldn't we have two active sites in a sense, independent sites that are communicating with each other and simply informing the other site of what they're doing so that if you ever had to stop one site and only use the other site, it would keep working. And then when the other site came back up, the site that was had been continually running would inform that site, well, this is what I did while you were gone. And that's where looking at some of the technology we have like the Actifio, the infinite of WAN accelerator with this, with our connection, we were able to say, yeah, we can do that. We can actually have two databases and just updating each other. Yeah. Speaking of the WAN link, I know you've got the WAN optimization. It's not just about throwing BAN with that at a solution. I saw some interesting notes in one of the case studies. What was your finding on your applications and your distance? Yeah, yeah. When we first put that BAN with then and just tried to do a simple file transfer, it was very disappointing. And after doing a little research, I guess, I actually have a little spreadsheet that depending on the millisecond ping time round trip or the latency, it will absolutely limit your transfer rate no matter how fast your circuit is. Because when you think about it, a millisecond in computer time is a long time. And so if I have, you know, one tenth of my BAN with is taken up just in terms of communication, unless I put very large packets in there, it's going to slow it down. And so what we realized right away is that without either a specialized device for WAN acceleration or specialized packages along the way, you know, there's some, I guess, file transfer programs that say they overcome latency because they package things in very big blocks. And so they eliminate latency. But you can't always do that. Sometimes you'll have users just wanting to do something normally. So you looked at the packaging option for file transfer. But you chose the Infinita solution. Tell me, tell us why. I'm a person, I guess, one of my favorite movies is Contact. And I love the line about all things being equal, the simplest solution is probably the right Occam's razor. Occam's razor. Occam's razor. Okay. To tell the people who use the servers, because you have some developers, you have IT people, you might even have some user servers that Wikis where people are working on the server in this location or that location to tell them that there's now the specialized programs. Oh, you're used to transferring data one way, but if you transfer from a server from this site to that site, you've got to go through these different processes. That just, that seemed complicated. That doesn't work. Kind of reminds me of a supplier that I used to call on me who said, if you'd planned better, I'd get it there on time. Yeah. Users don't necessarily plan exactly. And then there were some areas like, you know, whether it be email or database or some applications where they had to connect to each other. That's the only way they could keep themselves in sync, because they wanted to do it behind the scenes. And if they didn't support, if they were small packets or they didn't support the concept of a WAN as opposed to a LAN, there's nothing you could do about it. And I'm guessing based upon all the different traffic that you're sending over this, you've got, you've got a fair bit of burstiness to your, to your traffic. Is that right? Absolutely. And also given the educational market and sort of it has a natural flow to it, right? The educational market, yes. I'll tell you, this was an underlying issue whenever we design systems. We have on about six days a year, and in particular one day, February 28th, we have literally 10 times the traffic, the volume, the activity. This is a major deadline to the FASFA. I got kids that are coming up to college age. So what's February 28th? Oh, it's a major deadline for filing your FASFA for a lot of states and colleges. The deadline starts, the FASFA is released on January 1st for fall term. Okay. And so a few states have January 15th is a really early deadline, some have February 1st, but they all figure by about February 28th, 29th, April 1st or, I'm sorry, March 1st, right in there, you should be able to have your FASFA submitted, which is interesting. That's actually before April 15th, which is your tax deadline. So that's a little bit of a misconception and why people get confused. What's the difference between a FASFA and a tax return? And why am I being asked to do the FASFA so much earlier? But anyway. And for those who don't have kids coming up, that FASFA is the fight. Free application for federal student aid. Basically, if you're going to get any type of grants and grant me money that you're given, you don't have to pay back, or any type of student loans, and that could be federally subsidized loans. It could be loans from like MIFA here in Massachusetts, you know, that state loan or any lot of states have in California has a lot. And even schools that offer additional scholarship that's need-based, everybody who participates in that has to fill out a FASFA. It basically levels the playing field and tells them what your true need is. And your hosting provider takes care of that burstiness that you have so that you didn't have to build your infrastructure specifically for, you know, the one day of the year. That's right. Because one of the discussions we were having off-camera beforehand is, you know, you've got, as you said, half of your stuff is online and you've got your call center. You know, why don't you just kind of move to the cloud, if you will? Well, that's a good question. I will say, in terms of planning, when we started thinking about moving to virtual two or three years ago, the cloud was not as sophisticated. There's a couple issues. One, as a company that's growing as fast as we are, we've been able to do that by being very nimble and very flexible. And I know a lot of people feel that if you're in the cloud, you let somebody else take care of it, you can be nimble and flexible. But you're always working for us within their constraints. I can be a lot more nimble and flexible in my own data center. And also, quite frankly, the volume of data we move around, bandwidth has always been an issue. And part of our concern was in working with the cloud, providing enough bandwidth to the cloud for what we wanted to do internally. For example, when we, you know, we do a backup of a database, you know, you might create, in a matter of a few minutes, an extra 300 or 400 gigabyte file sitting out there that you need to move around or deal with or someone needs to load it into a development environment. So that's one reason. And the other reason, I guess, and we don't have to get into this a lot, but it was security, we really needed to be absolutely certain that the security in the cloud was going to be as tight as our security is. And that's a huge issue for us because of the type of data that we're collecting. You know, I want to pause here. I've got everyone on mute. I want to unmute the lines and see if there are any questions here. I'm trying to find the unmute all button, but I'm not finding it. So I'm just unmuting the lines real quickly here. If anyone has a question, why don't you go ahead and shout it out. Any questions from anyone on the call today? It's a good opportunity to get details on how to optimize your network, how to get continuous availability. Okay, let's turn it back to you still. So Robert, you said this is, you look at this as at least a year process. What have you learned along the way? You know, how would you advise other CIOs to approach this? And are there things they can do to maybe shrink this process? Yeah, I guess what we've learned is that one of the things is you really need to have a very clear understanding of what's in your data center. And when we're talking about connecting to data center, what's in data center, what runs in your data center, and what the servers are being used for, especially on the development side. It's not, I mean, development you kind of in a way give people a sandbox sometimes to play around, try things to test things out. And you need to understand what's in that environment. For example, some things that I can suggest that really helped us a lot. Go through all your software and remove IP and server calls and use like a DNS name in your domain. Because we have something like production database or production DB. And if we need to move a database server, if we need to change its IP address, if we need to put it into a server form, whatever we want to do, it doesn't matter. Because when we do that hot cut, it's production DB. And little things like that just make it so much easier for you to make changes in the environment, to grow the environment, to move somebody from a bare metal to a virtual. Because it's moving from that bare metal to that virtual too. Never letting production going down, making sure the virtual is tested, and then doing a hot cut. And basically everything we do, we look at and say, why can't I do a hot cut? And the answer may not be you can't, but why not? Database, well, we need to update this database. We need to do this. We need to do that. How long is the downtime? Well, we need a couple hours. Why? Why? I can do a backup in three minutes. Why do you need a couple hours for downtime? And you start working through it. And the developer is great, because they ask themselves this question now too. So if any of them are listening, a lot of this comes from them. They do a great job. They really do. But they look at it and say, okay, we can do all of this set up ahead of time. And so when we actually have to do an update or we have to cut over, we only need two minutes, three minutes. Because literally it's just applying these few stored procedures, bringing it up, testing and run their scripts, everything's automated and we test it. So really understanding how things happen and then asking yourself, why can't I do this as a hot cut really forces you to think of ideas and out of that will come the methodology that you need. Don't accept, you know, downtime is a given. I have to have lots of it to do the cut over. Sounds like virtualization is a large part of your environment. Not only do you have server virtualization, but you've got the IBM SVC storage virtualization. Can you walk us through kind of what percentage of your environment virtualized, what's virtualized, what's not? Sure, I would say right now that we probably have 80 to 85% of our environment virtualized from application perspective or server perspective, server perspective. Yeah, server and application pretty much, I mean, at this point, the rest of it will be done probably by the end of October. So you think you'll be 100% virtualized? 100% virtualized by the end of October, absolutely. Some of the stuff we were waiting for, what I call our slow period, September, October, you know, that's the period where now everybody right now is going to college, they're buying stuff for their dorm rooms, they're going to college, either paying for it or you're not, if you haven't figured out how you're going to pay for it, you're going to put it off a few months. Okay. And then in end of October, November, December, it really starts picking up as people think about the next school year. So this is kind of a period of time where it's less traffic. So we actually have bigger windows of low volume where we can do things. But so that's why we'll be done by the end of October. That played really big role in what we were doing because part of dealing with this much data and being flexible, and this is probably not news to anybody, is we really needed everything talking to a sand because you wouldn't know if a product changed or a product took off. And suddenly that virtual server or that server needed to access a larger database or needed more storage. And you didn't want to run out and, you know, buy more disk to put into there if you had storage somewhere else. So the typical reasons that somebody uses a sand, we grew into. When we started, we just didn't have the volume to, you know, we grew into it. But also as we started looking into it, I think it would be hard to create an active, active site without a virtualization, virtualized in a sand. One of the things we did is bought a product called Actifio, which is in stream with the communication between VMware and the sand. And it basically listens to block changes, the delta changes in your data. So what it's able to do is it's able to create through its interface, snapshots, milestones, checkpoints, whatever you want to call them, to go back and see what the data looked at at a specific point in time. It's able to keep a backup if you say, suddenly, oh, this file, and not necessarily database, any file, any file. Oh, this became corrupted. I need to go back and retrieve another one. All of that is done in stream, in line. And so what's really interesting is, again, you eliminate those windows where people typically say, oh, we have to have a backup window here. You know, this has to be unavailable so we can do these backups. And they're gone. So the benefit of being in stream is what? Well, the benefit of being in stream, to my way of thinking, is it's the simplistic approach. As data is changing, the Actifio on its little stand is recording those changes. And then with that, as it's own sand off to the side, then you can apply whatever you want. You can ship those changes to our site in Somerville, so you have an off-site backup, if you will. It can record in a database, in a metadata, different milestones in SLAs. For example, the one thing the Actifio can do is, you can say for this data in this area, these files, I want you to keep a snapshot every hour, but I only need to keep that for two weeks. That might be a database or something like that. But these files, which are more permanent, I want a snapshot every 24 hours, but I want to keep any changes for six months. So it allows you to manage your information. I think information life management is a big buzzword. This basically does it because it captures the changes and then allows you to set up the rules for managing it. And then you use the network infrastructure to pipe the data off to the other site? Well, that's the key part. You talked about kind of the setup of what you needed. Once we had that, now we said, okay, we have all the tools we need. We have complete control over our data and our changes. We have complete control over our servers. So we can now create another Actif site that mirrors that, that works in tandem with that site because we have control over all those pieces. They aren't separated on bare metal machines. But to do that, effectively, you really want, in a sense, to have land speed between them, not land speed. And we had a one gigabit connection, which, by the way, can be ratcheted up to like eight or 10 gigs just with a software change. It's actually on a gigabit port. For a price. For a price. Absolutely. Everything's for a price. By the way, disclaimer, everything we're doing costs money. Not as much as I would have thought, though. You shop around, you get good prices, you go for that simplistic solution. But anyway, so the gigabit port was, the line was key to making this work. And, you know, we just, we had to be able to push a gigabyte of data or more across that line in order for this to work because so much of our data changes. Yeah, I wanted to give an opportunity for people on the phone to ask questions here. So I'm trying to, again, unmuting the lines so we can get some questions. Anyone in the audience have any questions for us today? We're here with Robert Reeder, CIO of Resolve Group. I do have a question for Robert. This is Scott Lowe. Since putting the solution in place, have you guys suffered a major outage where you've had to basically rely on your system to work the way it's designed to work? Yes, we've suffered a, well, we haven't suffered a major outage between the two sites. You know, that really is a disaster. And actually, if we had, I probably wouldn't be on the show. I'd be off, you know, worried about that. But, and ultimately, as we grow, those two sites can just load ballots. But believe it or not, and we've already had two outages in our VMware servers since putting the system in place. So the VMware worked very nicely, moving systems off to the other servers without any eruption. None of our users knew anything had happened. We have actually had to go back to the Actifio, to some files that were inadvertently deleted. And it was made it very easy to go back and recover them. This is an operator error, fat finger delete kind of thing. Exactly, exactly. And then we could have regenerated them and stuff, but it was just so much nicer to do that. But we have actually tested using SQL 2012 SQL anywhere. We're starting to test that. We've actually tested using that database like in a cluster and just dropping one and see what happens and drop another one to see how that works. And then now we're have actually a SQL 2012 environment in Somerville, which mirrors one in Heracles. It's actually our development, I mean, our reporting site. And we're working to get those to replicate and testing that. And then what happens is if the one goes down in Heracles, all our reporting would switch to Somerville. So we're still- Now, when you did your SQL Server 2012 cluster testing, you basically dropped one. Did everything work the way you expected? Did you run in any surprises? No, amazingly not. It was everything just dropped and it worked fine. Now, we're still now load testing and stuff. So I'll give that caveat. It's pretty new, but we were pretty excited about it actually. We were used to the fat pipes where things just we don't know when circuits go down because the fat pipes just handle it. And our DBA tried every way she could just to kill that database suddenly. And it handled it without even flinching. Any other follow-up questions, Scott? No, that's great. Or anyone else on the line today have a follow-up question for Robert? Hey, Robert, I was wondering if you could talk about some of your transition from switching from physical to virtual and what methodology used to do that, whether it was like P2V to conversions or did you do a basic rebuild as a VM? Hey, can you give us your name or your company that you're calling it from? It's Peter from MCAP. Hey, Peter, thanks. Hi, Peter. Yeah, great question. We actually looked at it server by server basis because I insisted we weren't going to carry problems into the VMware. We can create new problems in the VMware on servers if you want to. We're not carrying any in. So the first thing I'll say is we did have some servers up on Hyper-V. Fairly new servers that we had put into Hyper-V primarily because once we decided to go with the new infrastructure, there was a moratorium on buying any new hardware that wasn't part of that infrastructure. So that doesn't necessarily mean there's a moratorium on people needing servers. And the Hyper-V servers just converted in place very quickly, very easily. The conversion tool provided by the VMware worked flawlessly. The other servers, if they were older Windows servers, those were, we said, we'll put up a new server and then it's up to development or we will help them to move the applications to that, allow testing to test it, and then we would hot cut to them. Because for the most part, those servers didn't require a lot of installation on what was running on them. It was mostly our software. We were able to recreate it pretty well. I guess I should say that most of our servers that were third-party software that would have required more rebuilding on the server, most of those were already Hyper-V servers. So we had already kind of in our mind began to plan that if we had to build a complicated server, can we do it in Hyper-V? A few of them in the development environment, the ones that are still to go, some of the development ones, have multi-purpose, have been added to over the last three or four years. And what we're doing is working with development. This is why I said early on, it's very key to understand what your servers are using for and what's on them, is we're looking at them and saying, okay, does that need to be two servers? Have we overloaded that server with too many functionality? But I'm trying to think as I do this, I don't think there was a, for us, there was a single server that we, other than the Hyper-V that we just converted in place. There was one. There was one new one we just converted and moved it over. But it was in Windows 2008, R2, recently built. So you're going all VMware though, converting everything to Hyper-V? All VMware, everything we have. What was your Hyper-V experience and how come all VMware? Well, all VMware is because at the time we looked at Hyper-V, it just didn't have the failover capabilities that Hyper-V had, the site recovery manager. Also, we were very interested in the Actifio device. I've actually been... Does Actifio support both or not? Well, I don't think they support Hyper-V. But they're... I don't think so. I'm trying to remember. If there's anyone on the call from Actifio, you can chime in and tell us. I know they are specifically designed from day one because Ash, who's the person who started their company I have been talking to for four or five years about the project. I've been aware of it. And they were specifically designed to work in this VMware environment because they are... They actually are... I don't know what the certification is, but they actually can get information from the VMware array about the blocks that are coming through. This is how they know exactly what they are, where they're going, and stuff. And it just... If we were going to do this, we really wanted to do it with software that was recommended by all the vendors in the hardware. Because we knew it was... You know, when you do this, you put in your entire face in that array. And your back-end arrays are what? For your production environment, what are you using? Now? Yeah. Yeah, the EMC-SAN array. Mm-hmm. Yeah, we're using EMC... VNX or the... VNX. VNX, okay. Yeah, with the VNX in both locations. For file and block or... For everything. For everything. We have about 30 terabytes of storage right now on each one in both locations. And when you move to the SAN array, we also were surprised at how much just our own deduplication of moving servers. One of the things when you don't clone a server, but you actually move it, it gives you an opportunity to say, does that storage exist somewhere else? You know, are you duplicating data that... Oh, when you move, it's already on the SAN over here. Okay. And so it discovers all that. Right. So, one of the things that we're always interested in, in addition to continuous availability and uptime and ease of management is cost savings. Are there any metrics that you were able to pull out of this either in terms of network savings, storage savings, people in process? That's a good question, and that's a tough one because... Because you're growing. Because you're growing, so you're like, how much more would it have cost me? I would say that... I'm thinking here for a second. There's two ways to look at it because when I look at cost savings, I look at how much actual revenue do we use, lose, how much trust of the people we're servicing, because we host the net price calculators for 700 college campuses on this environment too. So when you go to some college like an Albright or something, look at their net price calculator, you're actually on our server and we're doing that work. So the trust and the revenue you lose, when we look at the potential loss there, and I come at it from that way, when I look at the solution, the solution always has to be something that really is automatic, sustainable, and will get me as close to 100% uptime. So that's one way of looking at cost, but in terms of actually doing it, I'm trying to think if I could have done it another way. Part of the problem we were running into is on the bare metal machines, disk prices are cheap, we can add them in, you've got plenty of rack space, the servers really run pretty well, it wasn't a matter of cost, it was just becoming unmanageable on individual machines and disks. So I don't think we did it... I'm going to say what we did, you couldn't do unless you did it this way. And have you calculated? You ever done an internal calculation around the cost of downtime? When you're outpitching your budget, you say, okay, you're going to invest in uptime or you're not? Here's the cost of downtime. Absolutely, let me give you an example. On a day, I don't remember the exact day, but it was in the last year, the last busy season, so a year and a half ago. We had an issue where there was a miscommunication in our hosting provider when they switched from one... There was a miscommunication. I'm not going to blame anybody, whatever. And somebody was working on our circuit and removed our unlimited cap and put a cap on our burstable rate, a normal cap that most people have. We've asked for it to be removed, we accept responsibility. I guess it must have been February 28th then. We very quickly hit that cap and all of our monitors that showed our bandwidth utilization and everything immediately showed that we're bam, the graph instead of doing the normal curve leveled off. We worked with the provider, we got the cap removed and back in place. It maybe was only down for an hour. Were you down or actually... Oh, not down, I was slow. You were slow. No, we weren't down. We weren't down, sorry. So the cost of slowness actually may be a better question than the cost of downtime. And we were able to calculate, I don't remember what the number is, but we were able to calculate, well, this is what the graph should look like. This is what the graph looked like. This is how much revenue you lost. So you can calculate it pretty closely. Now, what's interesting is we look at all of that and you work with people. Those people probably, it looks like came back on later in the day. The internet's slow, I'm going to go back when it's faster. But that little exercise, although it didn't really, I don't think it had any impact on anybody who was really using the system. Like I said, they would just come back. It, one, proved out our ability to really monitor the network and showed us that the cost of downtime is very real in terms of revenue. Specifically on the infinite of the WAN optimization solution, how did you justify that internally? If it wasn't kind of a financial hard dollars, how did you make that? Well, the WAN optimization was a little different. I was speaking more of the moves of VMware. The WAN optimization was that we wanted to have both these sites active. We wanted to use them. And basically, from our perspective, without WAN optimization, the disaster site could never be more than a cold or lukewarm. Because of the cost of the network or because you couldn't get the data fast enough. Because you couldn't get the data fast enough. You couldn't get the data fast enough. And I mean, when you think about it, let me give an example. We have, let's say your primary site is unavailable. Let's say it's not a disaster, you just can't get to it. So you want to switch to your disaster site. If your data is an hour behind, you have to wait for at least an hour. If it's two hours behind you, you have to wait for two hours to get the data to your disaster site before you can bring your disaster site live. So you're basically down for two hours. So you look at the revenue loss of that happening just once. And you're probably coming close to paying for that WAN accelerator. You know, the two devices there. And I would think that has both an internally, all your people communicating amongst each other. And is everybody that goes to the web? Are they, is there, they're now the East Coast, present and the West Coast? Or is it still all served out of the West Coast? Like I said, we're just, we're still testing the active active work. When you're done with this. When we're done, we could load balance. I don't know if we will. I mean, we're there as we grow. There may be some advantages to that just to keep them both active. But right now they're on the West Coast. So, I'm sorry, your question was, yeah, so they're all the West Coast now. But the one thing I want to say about being down, the companies forget. I mean, we're in a company that's web-based call center. So clearly, and a lot of our calls come in from somebody going to the website, the numbers prominent, which you'd rather call us, you call. So if your website's down, you not only lose website traffic, you see a distinct, you know, definite drop in call volume. So not only is it the cost of lost revenue, but it's also all that unproductive time of people sitting there with nothing to do because you've staffed up for something more. Do you, what kind of recovery time do you look for for your application development and test environment? Same sort of recovery time? Like I said, we just, basically what we look at, and I go back to the simplicity thing again, it's simpler for me just to build an environment where whatever you put into it has this type of recovery time and just manage that, build to that, think of that, and then whether it's development, it's internal. It doesn't really matter. It doesn't matter. I think we had a question on the phone here, or a comment. Was there, are there any questions from the listeners? No? Okay, great. Where do you see things going from here? I mean, you'll be 100% virtualized. You've got WAN acceleration to keep your, to make sure you can get your data there on time. You've virtualized your storage and you've got, and you're managing your copy data with the Actifio solution. What do you see next? Is the next big challenges? Well, I think the next big challenge is to get better night's sleep. Now, that's a good question. Well, one of the things, and I would obviously hope the company would continue growing. I could see us perhaps using the two active sites as a geographic load balance, just because whenever you have two active sites, you're much, again, you're much better off using them that way so that there's never an issue with, well, what if one site goes down? Well, I didn't think of that. I wasn't prepared for this. It doesn't matter. They're both up. I will say one thing that was kind of interesting, and this is one of the reasons we really chose the Infanetta. We're on, as I mentioned, a gigabit port. So we're at the slowest speed of that port. Well, the Infanetta is also a gigabit box. So its slowest speed is one gigabit. So both the WAN accelerator and or the line with just software upgrades or switch upgrades, whatever they call it, can be increased, you know, all the way up to 8 or, I don't know if it's 8 or 10, I keep forgetting, 8 or 10 gigabytes. I know it's significantly more. So as we, say, have more and more data changing per second, if you will, the infrastructure is there just to keep up with that. And so if we had to grow each site, if we had to go to more storage, if we had to go to more servers, I don't suddenly have to worry about, well, what am I going to do about keeping them connected? So as I've seen it, this whole environment is now built to scale. So that, literally, as you say, where you go in the future, I'm not going to have to worry about re-architecting anything. I just worry about growing it now. Three or four years, something could change. I'm not naive and might have to do something else. But the, that was our goal when we planned for it. Yeah, it's quite different. I mean, I think back to 10 years ago when WAN optimization started going, it was always just about, how can I have a smaller pipe and squeeze so much onto it and the systems weren't designed for the gigabit and higher speeds of today. So virtualization, your hosted environments, analytics type applications are all driving significant volumes that you're looking at. Speaking of which, are you guys doing it? You've got massive amounts of data, big datas, one of the big hot buttons. Are you guys using any analytics with your information yet? Yeah, yeah, we are. We try, we believe that, one, a lot of the data is how do people use our system? There's a lot of it and what they're doing so we can improve our services to them and also just what people are interested in and what's coming in the data it also may change some of the products or the offerings. But yeah, we try to go through that data as much as possible to also, as I mentioned, we work with both universities, colleges, institutions, as well as the customer. So a lot of times we can use the data in a way to help the colleges better identify students they're interested in or colleges that the students are interested in. So as you know, when you have that much data you look at it and say, I know a perfect example, if you go to our website there's a college selector tool and say, this is the college I'm interested in. And we can come back and say, okay, people who are interested in that college, here are the other colleges they ended up going to. You might look at those as well just because we have enough data we can do that comparison. That's a very interesting. Yeah, that's a simplistic one but it's the type of thing where, yeah, it's like when you go shopping said somebody who was interested in this product also ordered these products. I don't know, I always look at those. I'm like, yeah, yeah, yeah, they would just shop and I say, what's this? Why would they invest in that? I've clicked on that. Are you getting into the buyer rating systems or not? No, no, we're not rating anybody. We're just, we're advisory, we're helping people, yeah. Yeah. Well, that's interesting. So you've sort of gone down two paths on the big data side. One is to improve your own operations, the user experience, and then the second is to add additional services potentially, right? That you can sell as potentially as products. Right, because I mean, one thing you may not be aware of, but the whole getting a college education market has been turned upside down. I mean, it started a few years ago when the federal government said we're not going to use private banks anymore to loan money, it's all going to come from the government, and that's fine. But they did that, they also said we want more transparency in what it's actually going to cost to go to college, what your payment is going to be when you have to repay it. So you really understand what that is, and that was the whole initiative with the net price calculator. And now there's even more and more initiatives where I think in a few years what you're going to see, you used to look for college and fall in love with the college and apply and get accepted, then you'd have the money talk. Not anymore. The money talks all going to come first, and the money talks all going to decide which colleges you can afford to go to, and those are the ones you decide which you want to apply to and fall in love and go to. And it's because it's so expensive. I mean, when we think about it, for many families, this is the single largest financial decision they'll make, and for many students especially at 18, 19, 20, just take on that kind of debt and think about this is almost bigger than maybe a house I'll buy until much later in life. It probably deserves to be up front like that. So by using the data and the analytics, we can look at it and try to help craft how that landscape needs to look, how we can best help you. At least that's our goal. So you're going to change your network, change your storage, change your server infrastructure at all as you get into more and more analytics? You know, as long as it holds up, I've been through enough change over the past four or five years, growing everything between that and telecommunication or anything else. I would love to be in a position where we can just say, yeah, I had another server. Or that's one of the things I liked about the VMware as well, their new VNX product. You can add a server. It doesn't have to be the exact same server. It'll still blend in. Because as you know, if I add another server in a year, I'm not going to be able to find that same server. But yeah, I'd like to just keep adding to it as long as I can instead of changing it. Like I said, we're doing the backup on the 100 Meg Networks. Not the only thing. Yeah, I can't see a whole lot. So you haven't eliminated backup? No, no, no, no. We just do it. We do it automatically as the data's changed. We've eliminated backup windows. Did you eliminate backup tapes? Oh, yes. Long time ago? Long time ago. When I came to resolve six years ago, six and a half years ago, I said we will do a backup strategy and we'll figure out what we need to do and offsite and all that. And we will not use tape. I'm a child of Radio Shack. I was working at Radio Shack and sold some of the original TSR 80s and backed up on cassette tape. And you know, for me to be using that in this millennium, you know, 40 years later or something like that, just it doesn't sit well with me. I'm sorry to all the tape manufacturers out there. I'm sorry. It doesn't sit with me. I want a totally non-mechanical as much as I can environment backups. Any other technologies that look interesting to you right now? What are you doing with Flash? Well, I will say one interesting thing is our SAN array. SANs have changed quite a bit. Our SAN array is just performing like crazy. And part of it is what's called tier data storage. And what you do is, you know, in the old arrays, you used to say, okay, these six disks are going to be in a rate array, one, one, and I'm going to put this data on them. And I have to balance out reads and writes and all of that. Now, the new SAN is a bunch of data. You create virtual LAN, LUNs. And then it's, if you will, spreads that storage across different disk types. We have a block of solid state drives, a block of fast, scuzzy drives, and a block of SATA drives, the slower drives that are much less expensive and can store a lot of data. And it will take data even within a single database and it will move the data that's being changing rapidly. It'll keep that on the solid state drives and the stuff that isn't changing on the slower drives and the stuff in the middle. And it does this dynamically every day. It switches things around. So that what happens is, we're effectively getting almost solid state performance on all our file activity. It's amazing. I got two last questions for you. One is organizational. And the other is some advice for the vendor community. So you've talked to your peers and told them what you've done. And you've got a great story, but organizationally, did you have to do anything differently to sort of take it to optimize your infrastructure? Did you change the organizational structure and then this? No, we didn't have to, because as we grew rapidly, our organizational structure was always probably small for a size of our company in IT. And the IT and the development are pretty much in one office and have a very good relationship. So we just had to get buy-in from everybody. But basically, IT pretty much manages all the computers and the network and everything anyway. But you still have network administrators? You still have storage administrators? And the network administrators? Well, we have DBA. But you don't have a storage administrator? No. And that probably shows some of the extra storage we have. And that would be interesting, but we don't at this point. What we do is, we actually, I will say, we have a fantastic relationship with development. And so if we find that it looks like storage is ballooning for no reason out of shape, out of funding, we can talk to some key managers and people will take a look at it and pull back. But no, we didn't really have to make any changes. It just fit right in. And in one minute, any advice for the vendor community? Yeah, the vendor community, in searching through all of this, I think responsiveness and service is just critical. Infanetta was fantastic installing the product. There was one little glitch. They signed on. They did a wire shark. They found that they had it fixed within a week. Actifio was great. You know, and I call out those two because they're most recently, but you've got to have good customer support. I mean, I understand a level one technician answering the phone, but when it is clear to him that the person on the other fan knows what they're talking about, has done all the triage and has a technical question, they need access to a more technical person because this stuff is critical to our business. That's great. Robert Reader, CIO of Resolve Group, here on the Wikibon peer insight. Really appreciate you coming in today. It's fantastic. That was a lot of good input. Stu Minerman is my co-host today. Thank you. We will have notes from this call up on the Wikibon site within the next week. And we'll send out a newsletter end of this week or early next week, summarizing today's call. You can, again, you can watch a replay of this on SiliconANGLE.tv. And you can listen, or if you'd like to listen to the audio, you can go to the replay that we'll have of the audio on the Wikibon site, on the peer insight archive. So I'm John MacArthur, peer insight host. Thank you for joining us today. Thanks for the questions. Scott Lowe and Peter, I think it was. So thanks very much for the questions today. And our next peer insight will be, we have one scheduled in October, early October. But please check the upcoming peer insight calendar. We update it regularly. Thanks very much, John MacArthur, and we're out.