 Happy Friday everyone so I got to the office here just a little bit before noon and as I was still in the parking lot my phone is going off with our web developer Melissa saying hey is is invoice ninja working and I'm like I don't know so I walk in the door and the guys hit me right as I walk in don't use invoice then just screen connect isn't working and I'm like they're what is there sirens people running around people don't even work here we're running around and spraying things down and I'm like what is going on Tom would you like to tell us what was going on good news yeah there's a new version of free Nass out hey that is good news other good news we have it now we have new version of free Nass now um so I seen a tweet I see the tweet and the free Nass tweeted they're like hey there's the version u3 from last week now there's a u4 we're like oh awesome and I did the version on u3 I did that video and I talked about this and some people said there's problems I didn't see those problems and so I was trying to recreate the problems and I'm like I'll just upload to u4 and but I went into our production machine and I was on the phone with someone who is that is a key thing to understand yeah Tom went into the production machine because I wanted to get the details that you can go into free Nass and get you can hit check for update and see the errata and normally at this point if you want an update you can say yes but which is checked by default yeah yeah it's actually just want your one click away from a yes and update and there is a perhaps part of me that just says sure so I'll update it yeah now normally just like accepting the terms and conditions yeah yeah I do that I have now and the part of the terms and conditions as I said make this video they wouldn't make chicken while I had a panic attack because all the alarms and off that check our virtual machines and for those of you that don't know how our lab or our whole production setup and lab and everything is set up in the back free Nass is at the heart of that because it is the storage container for all of our virtual machines so when the virtual machines lack a place to have data they all give a lot of errors hence the sirens hence the sirens I think the best part is it started with I saw a tweet and then I have problems that's how a lot of actually things have been especially lately I see a tweet and then sirens go off sirens so on a good note even well we've learned a few things here they learned a lot of things we've learned Tom should not tweet and update yeah and be on the phone and do everything once now I will justify one thing when I do client work I turn the phone off I'm I do it very my way full test this is why was your phone off I'm like really simple I just hit mute all and focus on the production things I'm working on and generally you don't update a server like that except for off hours and things like that so that being said well our servers baby there's a there's a slightly more lax policy for the house I was on the phone and I was oh man we were taking over this big IT thing that's been going fun we were like three meetings deep in this yeah it was just a lot of you know you're on the phone here like that so but the other thing learned if you want to know what happens when you have an ice-cutting initiator with a Zen server and pointed at it and then you just remove it and reboot in the middle of the day will all those servers come back and work yes well Tom watch nervously as all of them come up because of the what happens when you load an update in a production machine without shutting down all the things that are pointed to it two different servers running data off of it turns out fine it was good now Tom's a pretty chill guy but it's fun to see him sweat a little bit I'm like I hope all this goes really well now how long well because it was down when I got how long was it down actually well they noticed immediately I got alarms well I mean so I mean from like total time like maybe 10 minutes yeah it wasn't too bad but we still had to give Tom a little little kitchen and all make chickens basically have the perfect excuse so we aren't working at all because there's no work we can do if somebody has to restore backups because he broke something we're just gonna go home it's Friday yeah look at this is and we have a back a very regimented backup policy of all the VMs are backed up every Saturday or Sunday depending on the schedule if we have any work Saturday or not if not they go over to Sunday and all the data all the sequel databases and everything else that we have are backed up hourly copied here and then sent off-site when you're encrypted before then so worst-case scenario we could have lost up to an hours with the data if the whole thing went to crap but but it did not good news is the you for update I'm gonna do a video on it separately from this and I don't throw a click any update button because well it's up to date so we have the you for update yay button would be if you didn't do it this is where the update button would be but I've already clicked it super would you say the feeling when everything came back online was euphoric euphoric happiness yeah relief like I have a way to restore everything I don't want to go through the process of restoring everything like there things can be done it's well we hope you've learned your lesson Tom oh no no I'm this will happen again with you five okay it's been a minute no I have not made a I can't think of a time I made a mistake like this well it just shows it it's a little bit I make mistakes there's a lesson we learned Tom does make mistakes because mistakes are going to be made make sure you have things in place like backups and when a mistake happens have a backup have a backup for it so you know I have worked for other people I always promise not to be this person I worked for a guy I love my first day at a job was sometimes I make mistakes I blame other people it's best not to argue with me I just said I'm glad you got it up front yeah you're like I'm gonna be I'm gonna screw something up I'm gonna blame you and just don't just don't debate me about it and but I didn't really think that was a great work precedence to set well it's also different when you only got two other people in the building at the time yeah and you were the only one with access to do the thing because you just hear no no Steve did it you just hear Steve in the other room no I didn't yeah I was making chicken yeah I'm here chicken by the way I was that image you're talking about I was an imager I didn't do it I walked around brought Tom a piece of chicken on a fork so we could try it yeah like a last meal thank you yeah all the chicken has some chicken keep your keep things going while you're working on this as I nervously started all the VMs again so everything is all back to normal I will also say though this is a testament again to how ZFS works and I'm still why we use free NAS because even though we did this we keep several snapshots we do snapshots throughout the day with this and it's resilient even though it had all these live open files connected and ice because the initiators connected while we were doing stuff but so no data loss no other than a fun video that we made there's not much else going on in fact I think this video probably took longer than the so yeah so before another before another customer comes in yeah here's the seat how do you get customers real easy we just turned the camera on and they all just come in and a phone searching yeah it is fantastic we're hoping for a quiet Friday because a lot of clients are hoping for quiet Fridays yeah but anyways thanks for watching like subscribe stay tuned for when time doesn't see what Tom messes up next Friday yeah but I'm not afraid to admit it I clicked yes I clicked update on a Friday production time good news is it's a Friday it's Friday and we're all good just close early go home yeah I only disrupted one support session yeah yeah yeah he's got a fever he's got a fixer printer now