 So what I learned from address. Yeah, I'm it's an on-call nightmare and it's something we're gonna talk about and I hope you enjoy the story So first and very much foremost I'm waiting for my night. There it is. That's me. My name is Jay. I'm an ops professional. I work in Microsoft today's my one-year anniversary And one of the really coolest things that I've learned is storytelling. It's what makes Working in this devops field interesting and so I decided I decided I Decided to start a podcast and I call it on-call nightmares because I wanted to collect all your stories I want to hear what you've been through and then I decided, you know what I'm coming to devops They Chicago I want to tell my story so my podcast has three rules Do not incriminate yourself do not incriminate others and help us learn because I do this and think of it every way As blameless. Let's be blameless and let's have retrospectives. That's what storytelling is about So here's my story. I worked at Buzzfeed and I was part of the ops team It was a regular day I was building out a bunch of Mongo racks at the data center and you know, I was on call for alerts Normal day and then suddenly llamas That's right So how does a bunch of llamas impact my day? Well, I simply stated there were a pair of llamas that went on the run You know a real Jay-Z and Beyonce kind of thing. They went on the run in Arizona They rode around the streets and the internet of course When a shit because that's what the internet does whenever you see ridiculous things that make people interesting and create these viral moments alerts explode right So a lot of times, you know, you're on the floor of the data center and you're you're figuring out why the alerts are exploding And we found out that you know, we had so much technical debt so much time was spent And and essentially we built a system that couldn't handle the amount of traffic that we had And so what do you do about that? You stop and you think and you say well, this is our technical debt. Let's start making some remediations cool, so I went home and Do the things that we all do when we get home a 30 p.m. Pager goes off and I my wife asked Oh, is this because of the dress I ask what dress and then Nagios decides to tell me this is the friggin dress So I was able to keep some alerts from our Nagios bots and you can see things were really pissed off And if those of you who don't know what the dress phenomenon was it created 670,000 active connections per second on a lamp and that piece stood for Pearl So sorry, but you know and it all kind of started with the most interesting thing Kate's she's this wonderful woman She works at Tumblr now We worked at Buzzfeed and she just had an email from someone that just said can you settle this argument for us? And so what happened alerts alerts explode because Kate's goes out and puts out a really huge viral Post and everybody says Holy shit. Look, there's this dress and nobody knows what color it is And the internet goes and does what it is and then I get more alerts and pager duty says dude Yeah, you're you're screwed So what do we do? We start recognizing that we reached a really really bad point and this is supposed to be a GIF But PDFs don't animate GIFs, but Chris was talking to me on Slack and I literally said dude I'm just throwing servers in the internet to make sure that we can keep some sort of Things going but it wasn't quick enough and so disaster girl was a girl that came up when the pearl App would go to hell and Ben Smith who is the editor still is of Buzzfeed was saying like yo We can't do anything Because we tied everything together and we assumed capacity That's the whole DevOps part of this we assumed a capacity because and I'm not going to name names because we keep blameless But people didn't want to really increase and use their systems that through auto scaling that they didn't trust So what we did was we assumed that the clothes rack we had was enough for that dress and obviously it wasn't We had a huge request for people to wear that dress and The thing that I learned and this is how we're going to wrap up the silly story Is that you always need to be ready to restock and reassess when this season's hot style is unexpectedly in demand Because you know it could create a nightmare. So thanks. Love y'all