 Tell us about the pain of operations and how you moved to having healthy operations Testing one two. Hey guys down. Can you hear me up the back? Yeah, well, that's the way I'm going to be talking. So there may be a bit of a Every now and again, but this is going to be kind of my tone. So good Deal with it. We're good. Cool So ladies and gentlemen, welcome to this talk and I've entitled healthy operations This isn't the first talk. I was going to give Originally, I was going to tell you an epic tale of how we upgraded monitoring on thousands of servers and there's going to be an amazing story But instead I'm going to talk about something that's A little closer to my heart and something that is infinitely more important than upgrading monitoring and that's Being a healthy operator Day to day I have dealt with huge stresses and have suffered from depression because of it I see the same symptoms and conditions in not just some of my colleagues But also some of the friends who work in it and also many other people around me here today I hope that this is different from any other talk on the subject And in the end that I can give you something to hopefully smile about and maybe have a bit of a laugh The not only disclaimer to this is that I am not a doctor. I'm not a psychologist I'm not either counselor and my views and points I'm about to raise discuss are my own personal ones and they may not necessarily be reflected reflected by my employer Acreo or by anyone else for that matter All that said let's get on to the talk about me In 2001 I installed mandrake 9.1 That was my very first install and within 30 minutes I had to basically do a reinstall because I'd entered the root password somewhere done something and it broke I learned what not to do In 2006 I started my first full-time IT job as a systems administrator for a government department And ever since then I've been living breathing eating and shooting Linux both at work and at home I've also proud to say that I've done Linux from scratch And if you've never done it before I highly recommend doing it. It is awesome fun One of the cool things I've found about it is you need make to make make Think about that So I work for Acreo. We do enterprise level Drupal as a service Drupal is a CMS that is GP old and we provide the service and support for hosting it at scale An example of who an enterprise customer would be Proud to say that we're actually doing stuff for the Australian federal government The federal government is standardizing on Drupal for all of their external facing public Departments. It's a project called Gov CMS The code has been standardized and released entirely free which is amazing for our federal government to do massive kudos to John Sheridan for Really trying to implement free and open solutions because they have the better way to do things if you follow twitter a us gov cto is his handle But anyway, one quarter of our company are remote. That's about 140 people who are not in an office and work from their own house or a coffee shop somewhere By show of hands, how many people here work remotely? Please? Sometimes Okay, probably got about 20% of the room. It's still a decent amount And how many people here work in operations or operational roles? Okay, that's a lot more people 50 to 60% I reckon Cool. Well, I've been working remote for it will be three years in May And I used to work in a cube farm Where we had about 40 people in the office and used to be your parcel of information Yes, we did make the paper fit around his bookshelf. That's awesome But it was great because there were There's always something happening always people walking past you could stand up peek over the partition say hey, how about that game of sports? And it was good and it was kind of social. I went from that to To the spare room of my house Where basically I'd have Four ways to get to the outside world all through my screen. There was an inbox where I could read tickets and See talk about Drupal and things relative to people in an office like oh, there's an ice cream cake in the office today. That's super Um, we had we run XMPP So we have our own chat server and everyone's got their own rooms and that's how you sort of communicate with people generally And also had a gray and white terminal. Well at the time was black and white, but I've upgraded my theme since then The fourth thing that I had back when I first started was a weekly phone call We now use g plus so you can actually see the person on the other end of the line and it's great So for our daily Kanban meetings our team interact with people from EU and also from Portland the main offices in boston And it's all done through video conferencing and really helps break down those walls and we also do a weekly meeting now as a All of the overnight Crew can actually have a chance to see the rest of the team Voice our concerns or opinions ask questions and Actually be a part of the team and actually see them. It's so much better than it used to be but the first thing that really helped break down that barrier of Work in remote was that every week I had a phone call with my boss Jono Keith seen here standing on a box He's gonna kill me for that So he's based in boston and he'd stay up till 1930 his local time and we'd just have a tour So it's something that we did every week regardless and it had to be well It's something you've really stuck with and there'd even be weeks where we didn't talk about work at all and just talk about in like sci-fi or movies and games and all that so he likes playing games and all that so It's good to just sit down and have a bit of a talk and that was my first sort of Line to the outside world and there was a massive help And I still remember the in the second week Um, we'd done the intro pleasant freeze and he asked so feel how you going and said, yeah, not too bad You know, you're thinking starting to get the hang of these he said no How are you going? And I knew what he was asking And it's like well kind of shit It's like I didn't know what this was and couldn't get into that and I had no idea of what was happening you during the u.s. Daytime and Not having anyone around during the daytime to talk to he was going to freaking me out a bit But my greatest thanks to him for helping me work through those first hurdles And every week would break down a new wall and we could find a better way of doing things And I think that's the first lesson you can probably pick up from this talk that If you manage I'll work with someone remote Don't assume that they have any idea That they know what is going on Or assume that they're going to fess up and tell you that something's broken or wrong Please go out of your way to ask them. How you going? Getting contact with them and staying contact with them Do it regularly not just as a one-off because you maybe heard that they were under a bit of stress Whatever stay in regular contact talk about things other than work And if you are one of those people who has been approached and been asked how you're going don't be too proud If you're feeling down you're feeling upset say so if something's broken do something about it like So i'm going to switch gears and change speed for a bit see what I did there And dive into what it is to be sort of down now at all as I call it bedrock So At Stress of this worse. I was waking up at least one tonight thinking about work and work related things sometimes even two or three times I was so tired when waking up I could do an extra couple of hours sleep every single day and it's because I had torment to sleep or wasn't well rested I used to get up early and do a 16 or a 20k ride twice a week But the thought of waking up any earlier than I was already getting to uh that was already waking up Was utterly crippling I was too exhausted at the end of every day to want to even leave my own house as well because I hadn't had good sleep I'd wake up tired or be exhausted from the day and I'll just be too flat to do anything about it Because it was never a rest I could never get that energy back But it's too easy to Get into the negative feedback loop and say and do negative things when you're surrounded by Negativity so you've actually got to go and plan go out of your way to break that loop and try to make something positive of it This was exacerbated by You know being an ops you are interupt driven when alert goes off you've got to answer it and also not having any sleep You just scattered you can't actually work effectively. You are burnt out And at its worst When thinking about some people or some things I'd be so Angry and so uptight. I could feel it in my chest and that was a massive alarm bell for me Because you're stressed your body reacts and because your body is stressed your mind then reacts and tenses up If you've never played Minecraft if you have a pulse and you are breathing Please So where does it all come from? Well, there are personal stresses as well as their work stresses So the person is that I wasn't doing enough to take care of myself Um, and there are also things that are just part of the job that no you've got to deal with and find out ways of dealing with So other personal stresses In over three years hadn't had more than one week off completely removed from work Um until last november when I booked three weeks holiday over the christmas and new year break and it was kind of indicative that uh I need to take a break because I said i'm taking three weeks off. No one even questioned me not even once I was like can you take a break do it Exercise is one of the best things you can do for your mind and for your body It helps you remove from the everyday as well as It helps you stay fit and healthy. You live longer. You live better. You live a happier life like even going outside to listen to the birds and Birds sing and listen to the wind rustling through the trees is good for you Some people do it as a hobby as well as their main vocation but If you're never removing yourself from the keyboard from the screen There is no definition between what is work time and what is your playtime You really do need to do something other than computer work in your spare time Even if it's a hobby like rc cars or like Amateur rocketry if you do that sort of thing like take a reading Do something that is not in front of a computer So there's also the stress of being in ops. These are the work related ones that Everyone else in the company could generally take some time off like sales hr Engineering finance and to a point support. They could not work up to work for a couple of days And you know what things would kind of run. Okay But if ops was not to be there for two days Hand of server went down or a site went down and it would stay there until someone else logged back in and fixed it up So everyone in the company is relying on you doing your job The customers are relying on you doing your job and users are relying on you doing your job. That's not a thing to deal with We are very interrupt driven. It's our job because uptime is the number one priority Like whatever you were doing beforehand You've got to stop it and you've got to pick up the alert if you were doing like some making a script or Doing some coding and or doing some ticket work or even actioning another alert. You've got to stop that pick up the next alert You're always like got the interrupt there My co-worker emily coined this about ops people that we are lazy adrenaline junkies And it's generally because we're not the type to sort of go out and go dirt bike riding or jumping out of a plane Which I recommend you do at some point. It's freaking awesome But we generally love jumping headlong into problems to fix them Because that's a rush like I love nothing more than when there's a critical situation Jumping in and well owning it and fixing it up and being the one to get it back online most recently like during the Super Bowl uh, Bruno Mars did the halftime show We run his site and I was the one who kept his site online when it's being smashed by thousands of teenagers all across the world But I enjoyed that because I fixed it But that kind of leads to people becoming heroes that oh, they'll always jump in and fix it They'll always be there for that last minute miracle when it's needed And you just end up going into crit after crit after crit and wearing yourself down jumping on grenades all the time It's not good for you We've had to deal with growing pains in May 2012. We had 2000 servers We went up to 5000 servers Then to 8 000 October the same year 9 000 November the same year 10 000 Upgrading takes longer The uh, greater chance of breakages Um, there's because of the whole me time between failure. Um, we had more customers and more custom code Being remote Being in an office you pick up things just from being there like hear conversations in passing That you don't hear or you don't get when you're remote you cannot learn by osmosis When I log in for the day, I've generally got close to 100 emails to read And then of course if there's any emergencies happening, they'll have to pick up from portland and run with them So by the time you've actually gone through emails and you've got everything settled and everything's under control And okay, we're gonna do this and work and ask people some questions portlands logged off and boston is definitely asleep So you then have to either wake someone up or send an email or employee gs das so chris o'neill Works for our support department. He was the first sort of remote that I knew um He was in the same spot as me that he didn't realize what is getting into being remote for a company based in boston but we helped each other out and Like he would ask how do I do this is like I have no idea Let's find out and she's like okay. Don't do that try a different way and I'll ask the same with him How do I do this and he would help me out? We have the external factors of like people not understanding that People are behind the computer Like a person selling a feature that we don't support just to make a sale has just made a one-off system that we now have to administer and maintain and take care of What now a person promising dollar customer a thing that a certain time means that they've just promised The person is going to stay up do over time or be overloaded to get that job done because well opposite heroes They always take care of things. No, they're still people um Customers can write their own code on our platform. It's Drupal. It's open source. You can change whatever you want Yeah, you can use any or all Drupal modules Good luck to you as well as the fact that well Some people could be better at coding Customers take themselves down. That's nothing you got to deal with Adding more tools. So if you've got a tool that's only valid for one version of your platform You've got one set of servers which you can use a new tool But you've got to actually for the rest of them until you've upgraded and use the old one You've got to do the same thing in two different ways until everything is upgraded. It's extra stuff So it is a lot the things to deal with Be responsible for To break and to fix But you know what? There are things we can do to make good from bad. We can turn things around Keep others in check lend a hand Reach out if you see someone needs a hand ask them. Are you okay? Do you need help? And also don't be too proud to ask for help and actually go out and get assistance. You are not alone It's too easy to just keep working In it there is always something more to do You need to take time to kick back and Relax Holidays are more than a privilege. They are a necessity. Take them Spend time outside go for a walk a ride a run or a bit of a mosey if that's what you do We also need to get better at marketing ourselves Um And being better communicators This is the most interesting because well, we're all intelligent people who are generally quite humble and reserved And we have good ideas and sometimes they seem to get lost in the background noise because we aren't so good at representing ourselves and So in being better at conveying ideas, we can help actually get attention to the issues that are at hand What businesses don't do things because it's a good idea They do them because there is proof that an idea is solid So we need to get better at explaining ourselves to our managers to our bosses, etc Don't check work email after the end of the day It's too easy to have your work email and your personal email on your own phone So you can be at home like oh, just check them. There's one thing like oh well while i'm here No, don't do it. Ensure you keep the separation. You have to avoid checking on work things else private time becomes work time Rubber ducking it works in programming So basically the premise is to detail a problem your face into something that all can't even speak english Um, doesn't even talk it's completely inanimate and in doing so you break a problem down into its most basic parts And the same can be good for stressful situations because it's generally not just one problem There's generally a whole bunch of them. So you need to take the time to break it all down analyze and explain it So the closing postulations Dang it can get that wrong, right So these aren't just operational issues. They do affect other people in it. I'm only speaking from the operations point of view If you have felt or feel the same as I did make a change go on get a bit of help And start positive feedback loops karma really should be better than it is but I would like to give a big thanks to my colleagues who hopefully will get to see this some point Um, uh, these are the people who are who stand beside me on the front lines when there's bullets whizzing past my head These are the people I escalate to and who escalate to me and we help each other out We are um an awesome team that I love being a part of and of course We are hiring Tell them I send you I'll get you a bottle of whatever you want Um ladies and gentlemen, thank you very much for your time. It's been a pleasure Okay, we've got about five minutes until the next speaker starts If anyone has any questions we can take them while Nathan is getting set up