 So I have to start this talk off with a little bit of a folk off. I have to mention another conference I love DevOps days, and I am super psyched anytime I can get to them But if you ever have a craving for donuts and stats D. I totally recommend monodorama in Portland So it's not it okay monodorama in Portland is Monitoring conference. I wasn't able to go this year and so I kind of had to live vicariously through other people's tweets and this tweet in particular kind of stuck in my head for a little bit. We have all these ops ghost stories about read-only Fridays and 3 a.m. Is the which there was a slide today about three at getting paged at 3 a.m. And you know what else is being on the weekend is terribly on call then and I realized I kind of worked for an employer that might be able to dispel some of these myths So I dug in and I am not a data scientist So I just dug into our data analytics warehouse and I probably made a whole bunch of mistakes But I started to face look at our notification volume and tied it try to figure out Okay, this people are in this time zone if I kind of shift this over this way and I slide this over this way Can I get what it looks like on a daily basis and it turns out that read-only Friday not really a thing You know, there's a whole bunch of notifications Oh, by the way, we have former Netflix engineers So you'll notice all of our metrics accidentally lose their y-axis a lot. Sorry Oh, yeah That the weekends though are still pretty terrible. It's not a great experience But what we really want to know is what about a non hourly basis when we start bucketing our notifications into hours Is there really a problem at 3 a.m? Now go right to left apparently people do break things more after dinner And then you have to kind of the daily peaks and valleys of you know, much time and then I swear This is not this is true data. There really is weirdly something in 3 a.m. I Was like It's not busted and so all of a sudden now I have this problem Is this like why is this happening? But you'll also notice something kind of really ugly that the difference between the you know Day and night time and just the overall Number of notifications people get per hour is truly terrible And it's like this is a terrible experience for anybody to have to go through and this is no wonder why everybody hates being on Call this kills the on-call engineer. And so what can you do? I'm like, okay. I've just proven that 3 a.m. Is a thing. What do I do to help people try to? Not be on call at 3 a.m. Is that how I end this talk and I realized okay? I'm gonna try to argue with two tools in a two-minute ignite to figure out how you can have a better on call experience I'm gonna give you two things to take to your own organizations. Please take them back and make your own Call life better the first is a thing you can look at for anything that asks for human attention and make sure that it is always Immediately human actionable operate this down at all three should always meet all three This is kind of a rough metric, but the first thing is it's immediate immediately Right that if you have an alert or an alarm or an incident and it's on staging and said 3 a.m You don't need to be waking people up unless you have developers you're on staging at 3m But make sure that you should be doing something with it immediately Otherwise, don't be asking for human attention Second when you say human attention humans are flexible humans work best with ad hoc technologies Doing things only once or one offs don't have the everything's okay alarm that that one thing that that cron job that has to get reset at 3 a.m. Every morning don't have the humans do that instead Automated and put it in code finally make sure it's actionable when you actually say okay I need something to pay I need a human to pay attention to this make sure you're providing context Preferably business context. I don't care some servers paged it 100% CPU But I do care that border volume down 20% and finally when you the second thing to take back to your organizations is Review your alerts review your instance review anything that can ask for human attention We always add new things when we add new services get rid of old things get rid of things that never triggered They aren't needed trust me. You'll still find out about it. What about the unicorns though, right? Clearly they do better practices and I can't name names because their lawyers wouldn't turn me into a smoking crater But they're probably better right they don't have a 3 a.m. Problem There's no way that they would implement all these great practices and then still have a There really is a thing at 3 a.m. And I I dug into this and I was like okay This data is still 3 a.m. Having a problem but you'll notice that the rest of it you see more of a normal during working hours sort of Things breaking and people being called and so this is the difference between You know, maybe if you want to call it average industry practices and best practices And so again, I am poor all of you you take one thing away from this talk Apparently it is that 3 a.m. Might actually be a thing But also that you can take away immediately human actionable interview your alerts I'm happy to have arguments about scientific data on footer. Thanks