 Like right here, like right here. So this is an episode of when it all goes wrong, not really a new series. We, if you work in IT, you know sometimes it all goes wrong. Everything was on fire. Everything was on fire. And everything that could possibly gone wrong with this disaster in front of us did. So all the worst case scenarios happened. So we'll start from the beginning. So we took on a client and we know the other IT guy was an idiot. Now I always say don't pick on the other IT guy because you don't know his situation or whatever. Like did he tell them they should replace things and not replace things or whatever. So you always take things with a grain of salt and just, especially if the client starts by blaming the other IT guy, that's not a good place to start. But we have found an overwhelming amount of evidence that it was the other IT guy. It's one thing when you have one red flag, when you find boxes upon boxes of red flags, you're like, okay, we got problems. There's a lot of problems here. And you just keep finding more as it goes on. You're like, okay. We onboarded the client, we took them on and it was a tumultuous relationship for why it was just kind of a mess. We did some of the IT for the client but this guy had done some programming work for them. So they wanted to hold on to them because they needed some of the code writing skill that he had for some specialized project. Well, finally that project started winding down and the disasters were winding up and the downtime they had randomly over things that always seemed like a mystery to him. It was just too much disruption to the company. Everything was like a project to him and they got the idea that he didn't seem to really know what he was doing when it comes to the server equipment. And they're still questioning about his programming skills after incidents. Another incident. Another incident like that. So where we come in is he agrees we're gonna take on the IT, you take on a server. So we take everything over and he was charging for backups. It was really mysterious whether or not they were working properly or that he was even doing them. So we realized we knew the backups. Weren't they backing up to external drives that were bad? Yeah, they were backing up to external USB drives that weren't working. So that was at least one, the first problem was that we're like, okay, we don't see backups. Then we got a copy because these are virtual. Both of these are their two servers they had. And yes, one of them was a 2950 and one of them was a Dell 520. Yes, the 2050 is really ancient and the hard drives all went bad. These are a pile of bad hard drives in the 520. Three of them. Three? Most are bad. Out of six. Out of six, yeah. Yeah, six. And one was recently replaced. Yes, with another crappy drive. The guy just went to Micro Center. Off the shelf, Seagate two terabytes. Yeah. Your consumer, I wanted this for my desktop at home. Yeah, that's what they bought. Didn't wanna spend more than 80 a drive. We're gonna get more net in a second. So we take on the client and we gave them a proposal for replacing a server. They're like, look, these servers are old and they're not very fast and the backups are not there. And when we did have one of the drives, we pulled the VMs off the drive but they all had passwords on them. Cool for security. Bad because the guy said he didn't not, he did not remember the passwords to him is what he has answer was. He didn't say I wouldn't give you the password. He says, I don't remember what I set the passwords to be. So his disaster recovery plan of keeping VMs on external hard drives as backups that he exported wasn't a real good one because apparently he didn't write down the password. So you couldn't do anything with them anyway. First flag, yeah, that he was just dense. The second one was, of course he's hard drives. Now we didn't realize what hard drives were in there. So we gave the client a proposal and they kind of went around and like, we'll set the backups up and we sold them these specifically I've talked about that we use a SolarWinds backup plan for this client and we set the backups to go on these VMs and they started working. We then plugged in our USB drive, a new one just so we had a local storage, local speed fault as was referred to in SolarWinds. So there's a local copy of the storage. It's like the stop gap while we gave proposals to get better equipment in. And we told the client, yes, there's lots of red blinky lights especially on this one. This one, yeah. Some lights just go yellow. Some drives go yellow sometimes. And I think, in their exact words were, we replaced one of those. I can't remember what one, but one drive failed and we had to put it back in. Yeah, so they do their res issues and they still waited. Well, finally, there was a power outage about a month ago. And then the drive didn't come on and that's when I got my 5 a.m. phone call from the client and it was emergency mode and reinserting the drive got it and they immediately, that afternoon, sent us the deposit for the new server. So we immediately ordered the server and servermonkey.com by the way, that's what fit their budget. Someone's probably going, oh, you should have bought a brand new dog. Yeah, they needed a lot of storage or is a lot of storage right here and they had a plans for expansion. So their first step was we got one from ServerMonkey. We got a nice server with a warranty. It's a newer Dell 530, 64 gigs of RAM and a RAID 10 faster than both of these combined. Yeah, but times four. Also, enterprise SAS drives were inserted in it so everything was top of the line. Then we started the file transfer process. That's where things went worse because we're trying to figure out why the transfers were going at an absolute incredibly slow rate. One, the failing drives. Two, the non-enterprise SATA drives that were in it. Three, this is the part that I think's amazing too. And we'll pull this one out real quick. This was on our laugh, like why is this server so incredibly slow? This server has Western Digital Greens in there. Server, Western Digital Green Drive. One TB green, you can't even get those anymore. I know. So they had consumer SATA green drives so the transfers were taking an absurd amount of time to get the VMs all consolidated. Not to mention when he set the VMs up, he hard-coded a two terabyte limit rather than having it dynamically expand as needed so you have this two TB file you're trying to export. We found some bugs. I went and did some Google searching because one was running VMware, was it five? 6.5, I think? No, it was for five. Five, yeah, five something. Older version of VMware, there's some known issues if you set to the max hard drive size that it does not want to properly export that hard drive. And by the way, each one of these was at 97% capacity because of the failures and the configuration so there wasn't any way to make a snapshot or shrink it so everything was just kind of a mess and we ended up using the SolarWinds utility because it does bare metal restores to just take the backups and bare metal restore it. Now that's where, that sounds easy enough except there was another power outage during- We had an ice storm. Yeah, we had an ice storm here in Michigan. In April. In April, yeah, I know. It went from 69 degrees to an ice storm within what, four days? Like two. Two days. We had a really fast temperature swing. That storm took power out during the transfer then corrupted the local storage. So before we got the new orders for things that all went wrong. Well we had some of them moved. We were trying to move the last two that were the two, two TB ones that were massive and hard to move. The good news is we resized them once we got them moved over to the other server. So it was just everything went wrong on these jobs. And this is one of those things that you can't predict with a client even though we're taking them on we're gonna give suggests and replace things because I can't just power off a production machine and slide the drive out and go Western Digital Green. Oh, by the way, the iDRAC is broke on both of these. We don't know what the guy did. He had a firmware version that doesn't even match what Dell offers. So the iDRAC is definitely bad on this because it's just missing. What has the wrong firmware? We tried flashing with the right firmware it still doesn't work. I actually think I found a different one on another Dell site. Yeah, well we've been playing with what he had. We're gonna keep this one even though the iDRAC's broke on it so we can't get it diagnosed. If the server actually does work it's gonna be kind of a warm spare for them until they get the budget to buy another high-end one. We also sold them a free NAS machine just to store extra data on and snapshots of all the VMs. So they have a nice fast network connected machine there. If I get a chance I might do some follow-up video. Mark's pretty cool with us. The gentleman who owns the company and so we may even have him on to talk about it. We helped him a lot getting all of this sorted out. Went out there at 2 a.m. Went out there at 2 a.m. Cause this never came back up and then it was in a RAID 10 but coincidentally to both the drives that had the A portion of the data failed. Yeah, and it's a downside when you do a RAID 10 if the wrong two drives fail and the wrong two drives are the ones that failed. So this was just, it was a mess of everything going wrong but what I really want you to think about is when you have clients, as much as we would love everyone just to buy brand new everything and I've had people say, I wouldn't take a client who wouldn't buy brand new everything. Great, throw those clients at me. You got those clients that said, my budget's about 10 million and I want the top of the line everything. I need a Rolls Royce of servers. Great. Okay. A lot of us, many of you watching this video, we work in the real world of IT where there is other components to this company. How little can I spend? How much does this downtime really cost me? And we had to make, we made some assessments with the client and like I said, they sat on the quote for a little while. But nothing quite like their server and realizing their databases were not available for their entire office to use or all the files for about 36 hours to get all this stuff retransferred and synchronized again and set back up between all the VMs. They realized the importance of it and didn't have a problem improving the extra steps it's going to take for disaster recovery. But it's, you know, sometimes sharing these stories helps. It helps with some of the clients and sharing them with that. We, I just talked to another client who's kind of got the same situation and they're him ho on things. And we said, look, you want to talk because they kind of know each other. We're like, you could talk to Mark and I'll see what he thinks about waiting on those proposals. And I'm not doing it as a sales pitch because I'm just trying to make a couple of bucks. I'm trying to tell you these are the risk factors you need to add up. What does it cost you not to run your company for 36 hours? That's the pitch you give to the clients. What does it cost you for your engineers or whoever is working there not to be working? And especially if they're like, so we have an engineering company before we took them over and they're not in crisis mode but we know they are because they're servers on a warranty. So we've proposed to them and they know how much it costs. They got a bunch of salaried, you know, guys making maybe $80, $90,000 a year a whole room full of them that all do nothing and still make their wage. They wine and dye and all their clients and everything like they are. Yeah, you will. So you really have to assess that with clients and tell them, look, what does it cost you to be down? A lot. And if you don't have access to the data, what, and there's a couple of other things you throw in there. It's what does it cost you to be down and what does it cost your reputation for saying, we have computer problems and we can't do the thing that you're contracting and paying us to do. So your projects are late, your reputation is at risk. Your employees think less of you because you wouldn't buy that new server from them. Well, I mean, I think some of them would rather have a pay raise than a new server. Yeah. Well, there was a group of wandering employees that were just happy it's all broke because they don't have to work. Yeah. So there's mixed emotions there. I know that the boss is not happy. He's like, I don't put a broom in her hand. I don't know. Make them do something because you're like, we can't do all the things we usually do. Yeah. Well, the one guy, he showed up, because we got it working long enough, they were able to open one of the programs they use. Then it crashed in after like 20 minutes. But because it was loaded in memory on that computer, he goes, no, I got enough to close out everything from the previous night and then we're just gonna continue on paper today. Yeah. They had to do all back to a paper process. But these are things that you have back in front of your clients and how we took them on as an onboarding is not that I'm trying to be blameless in any of it, but we were very clear, like, hey, we're gonna take care of this. We understand your systems, your computers, but your servers are a big question mark. So when we onboarded them as a client under managed services, 100% transparency. Yeah. We's like, we can't cover those. Here's your issues. You really need to do this, but we can't force you to buy it. Yeah. But we can tell you, we're not gonna warranty them. Like there's nothing, we think that they are a huge risk factor for you. And these are things you, we do this with firewalls are at clients and we have a client like that that's got an old firewall. We're like, look, there is no will check. There is no current immediate risk as in the old firewall has no known vulnerabilities as of today, but it is discontinued. There is no firmware support. So if there's a vulnerability, you should probably take it off on day zero and that's it until you get another one. Cameras too. We have a couple clients who had old cheap NVRs on the internet and there's that bot that goes around rewriting all the cameras to hacked and renames them all and sets all the filters to black so you can't see them. And I reset it once. They got compromised again within a day. I'm like, this is a bot doing it. I go, you can't put it online anymore. Yeah. And there's no firmware updates. Yeah. So it's one of the things you assess with your client. You got to make sure they understand the risks as best as clearly as you can explain it. And you got to think in business terms, you can't say you should get better rate arrays or faster this. They don't think in terms like that. They think of the value of the asset. Yeah. So there's always like those considerations you want to make sure you're thinking of from a business perspective. What's our cost of ownership? Our return on investment. Yeah. You got to pitch it to them that way and say, look, it's going to cost your company. It's going to take a couple of days of downtime. And for the final, when we got the new server in and now we're selling them like the free NAS and we're rebuilding this with some hard drives just because it'll be a warm spare for a few other primary virtual machines. And also, by the way, this client runs some 2003 servers with specialized databases that's part of the reason they're in a VM. So it doesn't take a ton of horsepower to run what they have. So this server actually ran it fine once it has good hard drives in it. So this is just going to be like a warm spare. We're going to keep copies of all the VMs on. Yeah, this ran all the more advanced ones. Didn't it? It did. That ran the 2003. Yeah. This one ran in 2003. So this one will run all their servers fine. These are just all those little considerations when you onboard a client. That's the part that's important is that make sure you don't get stuck with the risk because you don't know what the other IT guy did. You don't realize the other IT guy used Western Digital Greens unless you can power off the server and pop a drive out, which not recommended you do that either. Especially when it's a questionable server anyways that's been on for years and out of warranty. But making sure you don't want to take on the risk but you want to make the client as clear and transparent of that risk. Aware of potential issues. Yes. And it's kind of just an assessment you give them. Because if they get blindsided, now you look bad. Yeah. But if you say, well, you know, I hate to say, you don't want to say we told you so but look, we made you aware of this and you knew this was a potential risk. We need to solve this. It is one of those things. You also stay persistent and you follow up with the client. So even though at the first industrial meeting maybe they want you to take over one section of the IT but not all of it. You still set up reminder emails. Hey, just follow it up and let you know that server, you know, the risk factor hasn't changed as a matter of fact, it gets worse over time. Yeah. It doesn't get better. It doesn't get better. It doesn't magically heal itself. So just letting them know. But on the final note, I will say how we know it's the other IT guy is he didn't even use the right screws in these. We started to pull the screws out. We also realized that this, he's got fine thread screws and coarse thread consumer drives. So the screws were falling out and the drives weren't sliding out right. So before we can pop these in one, we ordered some enterprise drives, just a handful of them put back in here. You know, even though it's gonna be a warm backup, we still wanted, you know, new hard drives in it. And then we took and ordered the right screw sets to go with them. Cause where he got these screws. Like five bucks for a hundred. Yeah, five bucks on Amazon for a pack of screws. I mean, we gotta leave the office. They just come here. Yeah, you gotta leave the office. Just gotta ship them here. Well, like I said, one, you know, and this comes down to the final set, you know, new are used. We want a server monkey for a refurbished server because they offer a warranty on a refurbished. And I think we priced out the new when we give them the proposal. It's like half, was it half the price I think? The cost of just the storage on the Dell because I was looking at the one for that camera build, the storage alone exceeded the cost of what we paid. Yeah. Cause Dell wants a, whereas server monkey for even the enterprise drives whereas like what, 150 for SAS drives? Yeah, they have some good deals on SAS drives. Now it depends. You got options for refurbished drives and new drives. But the nice thing is you get a long warranty with them and you can do some choices. You can get some SSDs with there if you need some caching drives. Well, but either way, check that out and make those assessments when you're doing it client, make it easy. I think Dell wanted like 400 a drive. But the one benefit of Dell is there's almost no cost upgrade to go from the three year overnight to five year. It was only, it was less than, Yeah. 10% of the cost of the server, like less than the cost of one of those drives. Yeah. If you get a Dell, the warranties are good price. Go for the five. Go for the five, but if you do buy like a new Dell server but when it comes to buying the hardware, like say you got to present the client with a couple of options. Yeah. Well, all right. This is just some stuff I want to help you think about cause this question comes up a lot and maybe the term that people use is like when you're onboarding a client or when you're assessing a client or the discovery process. Yeah. As I've heard it termed in the MSP. Really is when you go there, you assess what you're taking on and what you're getting stuff into. Make sure you think about it and don't leave anything to chance. Let the client know, like that's old, that's old, that's old. That server is a big question mark, especially then you find out even more when there's less than digital greens and what you thought was an enterprise class server. So just stuff to think about when you're getting these. And like, so we'll have a link below if you're interested in order for server monkey. We have a discount code for you. Thanks for watching. If you like this video, go ahead and click the thumbs up. Leave us some feedback below to let us know any details what you like and didn't like as well because we love hearing the feedback or if you just want to say thanks, leave a comment. If you wanted to be notified of new videos as they come out, go ahead and hit the subscribe and the bell icon that lets YouTube know that you're interested in notifications. Hopefully they send them as we've learned with YouTube. Anyways, if you want to contract us for consulting services, you go ahead and hit launch systems.com and you can reach out to us for all the projects that we can do and help you. We work with a lot of small businesses, IT companies, even some large companies and you can farm different workout to us or just hire us as a consultant to help design your network. Also, if you want to help the channel in other ways, we have a Patreon. We have affiliate links. You'll find them in the description. You'll also find recommendations to other affiliate links and things you can sign up for on laurencesystems.com. Once again, thanks for watching and I'll see you in the next video.