 Get up. Something's broken. You just got paid for a service. You're not at all familiar with. So you start adding a bunch of people from different teams to a conference line, trying to find someone who can help, slowly getting more frustrated and resentful, and then it's total chaos. Everyone's talking over each other, yelling. Hours have gone by and you still don't have a solution. Does any of this sound familiar? Whenever you experience a major incident like this, that is an unplanned service outage that actively affects your customer's ability to use your product, it's important to have an organized approach for fastest resolution. Now some operationally mature companies may have a formalized incident response process that's basically a runbook for how to behave during an outage, and incident commander is a formalized role in that process. They're responsible for leading the response and act as the final decision maker. Now whether or not you have a formalized process, it's pretty common to think the person best suited to lead during a major incident would be your senior technical leadership. The problem is this isn't scalable. If your CTO needs to respond to every major incident, they're going to burn out. I'm here to tell you you can scale your incident response process by welcoming non-technical people to serve as incident commanders. I'm going to share with you the inclusive incident response training that taught me, a project manager type, how to be an incident commander. So first you need to specifically define the role of incident commander. The important distinction is the incident commander is not responsible for finding the solution. The ICs should not be doing any investigation or mitigation tasks themselves. Their role is to coordinate the responders, keeping communication flowing and delegating all tasks. This eliminates the need for your incident commander to be highly technical. So then you need to demystify what really happens during a major incident. So at PagerDuty we host regular incident commander office hours to welcome non-technical people, to explain the role emphasizing the importance of communication skills for this role, much more than technical skills. Now after you've hooked a perspective IC that understands the need and expectations for their help, put them on a schedule to shadow the current ICs. This will give them a feel for what it's like to be on call, building that empathy, and make clear when they get paged they should join the call to follow along, but they need to remain a silent observer to avoid distracting from their response. But then make yourself available after resolution to answer any questions they might have. Now after a trainee is listened in to a few calls, invite them to start helping as a scribe. So scribe is another formalized role in an incident response process. They're responsible for keeping a real-time log of everything that happens during the incident. This is really helpful to analyze for the post-mortem. So scribing is a really low risk to start helping. Writing down everything that happens really cements your knowledge of the process and builds confidence. So next phase in training is you want to set up your trainee to reverse shadow. So at pager duty we always have both a primary and a secondary incident commander on call. And when they get paged both join the call. The point here is the incident commander always has a deputy to support them however is needed. So to set a trainee up to reverse shadow you really just put them on the main schedule because when they get paged they're going to have that backup there with them. And the backup should send tips and reminders to the trainee in the background to help them maintain control of the response while building credibility among the responders. And then that's it. The trainee has successfully led an incident response. They're now a fully fledged incident commander. Celebrate them. They're having a direct impact on the health of your services the happiness of your customers while distributing on call load internally. So here's a summary of that training process that you can take to your org. First to describe the incident commander role remember they're not a resolver. Host regular office hours to welcome people into the process. Have them observe silently as a shadow. Maybe start helping as a scribe keeping that real time long. And finally reverse shadow with the support of a backup IC. Now DevOps is all about building empathy and breaking down functional silos for ultimately better and faster outcomes. I think we should expand that collaborative mission beyond just development and operation teams across the entire company so technical and non-technical people can help each other succeed. Come find me if you want to talk about incident response or how to welcome non-technical people into the DevOps community. That's it. Thanks for listening.