 Are you guys enjoying the conference so far? It's great. So, like you said, my name is Annie Sexton, and like a lot of time thinking about how people absorb knowledge. And that's why I'm really excited to talk to you guys about tribal knowledge and why it's so dangerous, and how by learning to address it, you can not only learn how to scale your team much more quickly, but help them become self-sufficient, which I think is the most important thing about this. But before we get into that, I know she gave me a bit of an intro, but I'll probably say the same things once more. I'll tell you guys a little bit about myself. So she mentioned I was a digital nomad. I'm a retired digital nomad. I just bought my first place, but I basically spent all of last year like living out of a suitcase and globe trotting, and it was a lot of fun. But I got sick of airports really quickly. So I'm also a volunteer English teacher at a non-profit in my hometown of Austin, Texas, which is a lot of fun. I really enjoy teaching, and as she said, I'm a full stack developer and have been for about five years. And I worked at lots of different web agencies and just spent a lot of time farting around on my own Rails applications. But of course, this was all before I was hired by the man. And by the man, I mean Salesforce. And by Salesforce, I mean Heroku. And it is great. Totally candid photo here, definitely not staged. But really, it's an awesome place to work. My coworkers are super smart, super friendly. I get to work on my dream platform. And Salesforce, our parent company, is really awesome. They're like, hey, what's up? We're Salesforce. We have this guy. So it's great. Overall, I really, really like work. It's the best job I've ever had. But, you know what had some room for improvement? Was the onboarding process. Yeah, you see, Heroku is a bit of a complicated platform. Easy to use, right? We pride ourselves in making it easy to use. My mic's still on, there we go. So it's easy to use, but there's a lot of things going on behind the scenes. And there was a lot of information that I needed to know to ramp up for my job. And I'll tell you guys what I actually do at Heroku. So, and what we call a support engineer, which sounds about as vague as solutions architect and success developer. But all that means is that I'm on the support team. And we're called engineers because we are engineers. Anytime you open a support ticket on Heroku, you're talking to a developer because we have to know what the hell you guys are talking about. So, Heroku doesn't have a tiered system when it comes to our support. Instead, we break it up like this. We have core support, and these are sort of like the generalists, right? We're the first line of defense whenever tickets come in. And then if we need specialized help, we can get someone who has specific knowledge about Kafka, about Postgres, about Heroku Connect. If any of you caught Caleb Thompson's talk earlier, he is our resident Heroku Connect expert. So, say hi to him in the hall. But I am on the core support team. So, as a generalist, it's basically our job to be the power users of Heroku. So we basically have to be better at Heroku than everyone else in this room combined. And there's a lot to know. There's a lot of moving parts, and it's a little difficult. And boy, howdy. Lately, the ticket queue's been getting out of control, which is why Heroku is hiring. All of RubyConf, congratulations, you're all hired. That's right, you get a job, and you get a job, and you get a job, and please don't sue us. This is just a bit. You get a job, and you get a job. So, just to get you started as a new support engineer, we're gonna need to make sure that you're all familiar enough with the platform. So I'm just gonna give you guys a quick download of what I know about the platform right here right now, so take notes, okay? So, Heroku lets you deploy, run, and manage applications written in Ruby Node, Java, Python, Clojure, ScalaGo, and PHP. Languages which are provided by our buildpacks. But if you need further dependencies, you can use third-party buildpacks or even Docker, which is a container, like a Dino. Diners are containers, but not the same as Docker. But they make scaling really easy. Domains, if you'd like to use your own custom domain, you can do so by running Heroku Domains ad, yourdomain.com, and creating a C name record for any subdomains, and an A name record for any root or apex or negate domains with the respective DNS targets. And, you know, somehow I feel like that's not the best way of conveying this information. Maybe there's a better way, huh, maybe. So, for the record, that's not what Heroku onboarding looks like. Thank goodness. Because, let's be honest, there's just too much about the Heroku platform for me to convey that information in a timely manner, right? There's just, that would be a poor use of your time and mine. Wouldn't you agree? Now, luckily, we don't have to do it that way. We have the Dev Center. Who in here has read articles in the Dev Center? Thank you. Yes. So, yeah, we've got it all documented, right? We don't have to convey it verbally. Thank goodness. But when I was being onboarded, there were things about the platform that were not in the Dev Center, things about the architecture, how things were interconnected, what were its limits, what were its edge cases, stuff that was kind of documented, but it was kind of documented all over the place. And on top of that, I had to learn so much more just to learn, like, the things that I needed to know as a support person. Here are the internal tools that we use for debugging customers' apps. Here's how to use Splunk. And all these other CLI commands that I've never used before. And we have our own support system as well, because we're all developers. So we were like, we'll just build a system ourselves. And so that's what we did. And so there was a lot to learn. And like I said before, it was kind of documented, but some of it wasn't documented. And if it was documented, it was, like, all over the place. And I had to look in five different places to find that. And there was just a lot to learn. In other words, there was a lot of tribal knowledge that I had to absorb. So what is tribal knowledge? Let's define it. So tribal knowledge is that information that exists in the brains of your coworkers, your tribe, and nowhere else. And this is no bueno. It's dangerous. And I'll give you an example. So how many of you can relate to this? You're like, headphones on. You're in the zone. You're super focused, and you're only, that's probably not you, because that's not Ruby. Hey, you're in the zone, and that's also not Ruby. What about? Perfect. OK. So you're in the zone. You're working on your super focus, right? When all of a sudden, something distracts you from the corner of your eye. Hey, yeah, sorry to bother you. You have a real quick question. Hey, how do I, you, the senior experience developer, reluctantly remove your headphones? Yes? Concentration is broken. That sucks, right? But let's say the roles are reversed. Let's say you're the new kid, and you just want people to like you, and you're trying to work really hard, and yet the answers you're looking for aren't documented anywhere. So what do you do? Well, if you're in that situation, then guess what? You get to be the one to wake the sleeping dragon. And that's always fun, right? First of all, stop hiring dragons and be nice to your new people. So that's one thing. But yeah, tribal knowledge sucks for everyone. And especially when you're a growing business, it is unfortunately very common for all of this important information to reside in the brains of just like one or two people, right? Times this is harmless, right? Hey, where do we keep the PSDs for the marketing side again? No big deal, right? But sometimes, it's not so harmless. My website is down. I can't get ahold of my developer, and I don't have access to the app on Heroku. Let me insert here by saying that credentials should never be tribal knowledge, ever. But this happens an unfortunate number of times. And you know what we have to tell these well-meaning business owners? I'm sorry. Your email isn't listed as an owner or a collaborator. I'm afraid I can't provide you any details about the app. You're going to have to do your best to reach out to your developer. Sorry. That's a shitty thing to have to tell someone and an even shittier thing to hear. Tribal knowledge is dangerous. And as we've seen, it's dangerous for two reasons. So number one, it creates bottlenecks for both senior people and junior people. And it creates liabilities. Because when, not if. When those people leave with that information, you're screwed. So, oh, sorry. Too fast. That's OK. So some of you might be thinking, listen, Annie. I don't know how they do it at Heroku, but we do document our stuff. It's just that these other people don't know how to look things up themselves, right? They can't do their own research. OK. That's fair. Totally get it. So where are these articles kept? Oh, you know. Google Docs, markdown files, GitHub issues, Trello cards, emails, Google Hangout Threads, Quip. So that's a lot of places to store stuff. And maybe it's a little hard to find that information. But no, listen, I get it. I do understand that sometimes people are just lazy. OK? That's worth acknowledging. But here's the thing. If this is a serious problem, where people aren't looking up the answers themselves, the first thing that needs to happen is you need to set boundaries. Without boundaries, you can't have a self-sufficient team at all. So you need to train your clients, your co-workers, your managers, your mom, that they can work around your schedule, right? And so this is really, really important. But now, so if you have some of this tribal knowledge documented, maybe, like I said before, maybe people aren't able to even find it. Like, it's written down, but no one even knows where to look. It's like the Ile de Muerta, a location that cannot be found except by those who already know where it is. Yar. Which is fine if we're talking about cursed treasure, but we're not. So what do you do? Well, there are two main issues. There's people not knowing what is documented, and then there's people not knowing where it's documented. OK? And these seem like really solvable issues, right? And yet they're really common. And that's why I want to go way in depth with you guys about how you address these problems and why they need to be addressed in this way. So let's assume this is a worst case scenario. You have nothing documented. What do you do? Well, since you can't just, like, the giver the information away, how do you transfer that information on? Right, what do you do? By a show of hands, how many of you, when you get a new appliance or a new car, you read the entire user manual? OK, you guys are the outliers. Personally, I like to see how far I can get without the manual. I like to live on the wild side. Because let's face it, most manuals suck. They suck. And who wants to sit through and read through all of that? Four people do, but that's it. First of all, I admire you people who do that. So instead, we need two things to ensure that the tribal knowledge we're documenting is actually digestible. And since we can't just, at least at this point, we can't just plug you in matrix style and you're like, oh, I know, kung fu. Here's what we need instead. Number one, we need a crash course. And number two, a knowledge base. If there is anything you remember from my talk, it's these two things, a crash course and a knowledge base committed to memory, OK? Now, in their simplest terms, these two things are really just two different sets of documentation. They can take any form you really like. They could be webinars. They could be videos if you wanted, but they don't have to be. It's just easier for instruction's sake to talk about them like two different sets of articles, OK? So with that being said, let's talk about the first one. We've got the crash course. Now, I'm calling it a crash course, but you could easily call it like a getting started guide. Now, the point of this isn't to document everything you need to know for a particular job. It's just enough information to get someone started so that they can then self-educate, so they can look up the answers on their own. It's not everything. It's just a primer. It's just the basics. Now, you know who kind of sucks at explaining the basics? Engineers, developers, you know who you are. Now, listen, I'm totally generalizing. So take that with a grain of salt. But also, hear me out because I know there are people in this room who need to hear this. So we are regularly criticized as a group that we can't just talk in like layman's terms, right? They make TV shows about how poorly we explain things, right? Terrible, awful TV shows, I might add, but whatever. And the problem is that we explain highly technical terms in highly technical terms. And that doesn't help, OK? And I'll give you an example of what this might look like. So I have a buddy of mine who happens to be an expert in Postgres, very smart guy. And so I came to him one day, and I said, hey, I was wondering, what's the difference between Truncate and Delete in SQLand? And this is what he responded. Truncate is more like a DDL, data definition language statement, than a delete, which is a DML, data manipulation language. Yeah, duh. Can I get you to the right? Yeah, duh. That was good. Yeah, so OK, what now? OK, well, OK, well, so essentially, Truncate is a fixed cost operation versus Delete, which is closer to, oh, parentheses n, where n is the number of rows affected. Maybe we're getting closer here. Maybe I don't really know. OK, OK, OK. Truncate is like, give me a fresh table. Boom, that is what he should have started with. That is a good sentence. And later, he went on to add, oh, yeah, yeah. Truncate is like the big guns. But you guys see what I mean, right? When asked to describe the forest, we immediately go into describing the molecular composition of the bark, right? So we need to be able to step back a little bit. And by the way, I'm not throwing my friend under the bus here. He's usually quite good at explaining things. I just thought that particular example was great. So this is something that we have to be aware of when we're writing our crash course. We need to be aware of the terms we're using so people understand it. Again, the biggest thing about the crash course is it has to be digestible, right? And this even applies to non-technical things, right? Even to give you a non-engineering example. If I told you guys that you can update your V2 mom through the Aloha portal in org 62, you'd be like, what, did you just say? And unless you're a Salesforce employee, that stuff means nothing to you, right? And that's normal. So you have to think about this. When you're writing your crash course, you've got to think of these questions. Who am I writing this for? What is their background? What terms do I need to explain? And what am I assuming they already know? Like, do they know what the Aloha portal is? That kind of thing. So the last thing I'll say about the crash course is it's a crash course. It's not like a whole Udemy course that you take over the course of several months. It's got to be really quite short because it has to be digestible, right? It can't be too long and it can't be too short. It needs to be as long as a skirt. It needs to be long enough to cover the material but short enough to keep it interesting. OK, so there's, let's talk about the knowledge base. This is the second part to this. So your knowledge base, where we were concerned with the crash course not being too long, the knowledge base is where you put everything else. It's all the rest of anything that needs to be documented within your team. It's this expansive repository of articles that people can refer back to, right? And this is a really great place to store things like playbooks, procedures, checklists, all of the documentation, right? And because we are prepped with the knowledge that we learned from the crash course, we kind of know how to navigate this and we know we're more equipped to go and search for answers. So to facilitate the ease of use of our knowledge base, it's got to be these three things. It's got to be easy to find, easy to update and easy to find what we're looking for. Now I know one and three kind of sound similar so I'll explain. So easy to find. You have to pick, this is gonna be hard, but you have to pick one place to keep all of your knowledge base articles, one place. Now I don't think this is possible for like a really big company, but within a team, absolutely possible, okay? Pick one place, otherwise, you're not really getting around that bottleneck problem, right? You're still gonna get people tapping on the shoulder, hey, did we keep that in Dropbox or Google Drive or where do we keep that thing? And we don't want that. Pick one place, it doesn't matter, right? Choose amongst your team and then stick with it. Next, you gotta make it easy to update. Give your team right access to these articles and then get out of their way because the people who are using the articles really should be the ones who have access to edit them and just, you don't wanna have a lot of hurdles and checks and stuff like that to get in their way. Which is a pretty simple thing, but I wanna extend this to, if you have documentation that's also intended for non-engineers, don't make the place, don't make them do a pull request to update those articles, right? They may be able to figure it out, but you're adding friction where there shouldn't be, okay? So we gotta reduce the friction on updating these articles. Last thing I'll mention is that if you've reduced the number of, sorry, if you have your knowledge-based articles in just one location, and as long as that one location has the ability to search through those articles, which like, I don't know what tool wouldn't have that ability at this point unless you have your stuff in a filing cabinet, a physical filing cabinet, which I'm sure you're not gonna do. As long as you can search through them, then how you organize those articles within your knowledge-based doesn't actually matter at all. You can stack them in a big old pile as long as you can search, right? And I say this, I fucking love organizing you guys, okay? It's like a really weird hobby of mine. I love to plan. You know what I asked for Christmas this year? A spreadsheet. So just know that, just know that when I say you don't need to organize your articles, I really mean it, right? As long as you can search for them. But you can go back and organize them a little bit more if you want, but it's not super necessary. So now that we understand the two pieces to addressing tribal knowledge, we've got a crash course, we've got our knowledge base, let's look at what this actually looks like in real life. And I'll give you the example of, not just Heroku, but Heroku's support team specifically. So we started to adopt this model kind of unknowingly, definitely with my influence. But it's been really helpful. And some of this stuff was in place before I even got started at Heroku, which is awesome. So when we onboard new people now, we have them read three articles. Remember how I said the crash course should be short? Ours is like three articles. So really it does not have to be long. One of those things is this thing that actually all new Heroku, Heroku read when they're onboarded, and it's called the Heroku Academy. It's this big long document that talks about sort of the basics of the platform. Here the feature is these are pipelines, these are review apps. Here's private spaces, that kind of thing. And we have a couple of other articles that are more specific to just support engineers. Here's our tool, here are the CLI commands we tend to use, here are the internal tools that we use, all sorts of stuff that gets them started. Here's how we work through the queue, here's how you deal with an angry customer, that sort of thing. And so this is our crash course, it gets them started to be a support engineer. Then we have our knowledge base. And because we're talking about the Heroku support team, our knowledge base is actually just the help site, the articles in the help site. And you'll see we have them kind of categorized. This is honestly more for your benefit than ours, because it's easier to open up a ticket that way. So we know more about what you're talking about if you have an issue. But you can just search for what you're looking for. Like I said before, searchability is key. And then you can find the articles that you're looking for. And we have both public facing articles, which is what most of you would search through if you had a question. And we have internal ones, with super Heroku secrets and stuff like that, that are just specific to our job. Now, you might be wondering, wait, hang on. Heroku has a dev center. Why even have this? What's the point? Well, in Heroku land, we support, realize that there was a need for a place to provide answers to common questions that didn't really fit in the dev center. And I'll give you an example. I didn't just wanna put Lisa Frank in my slides. I promise this has to be an explanation. So we were noticing that there were a lot of apps that used Unicorn as their web server that were experiencing these seemingly inexplicable request timeouts. Specifically H12. H12 is what it looks like on Heroku. We time your request out after 30 seconds. So why was this happening? Actually, that basically explains it. So Unicorn can be a bit harsh. So what would happen is you would have these slow web workers that Unicorn would, instead of sending it like a sig term, it would send it a sig kill. So it didn't have any chance to output an error or wrap things up gracefully or anything like that. It was just like, boom, sig kill. Muffling your app's cry for help. And so what would happen is with fewer web workers, your app didn't have as many workers to handle requests. So they would pile up and your performance would tank and then you would run into these H12s. So this is a Unicorn issue, technically, but you wouldn't see H12s anywhere else than Heroku. So we decided to create a knowledge-based article for it. Oh, and also, I don't know if I mentioned this before. To customers, these are like the help site articles. We literally call this the knowledge base. I think it used to be called the knowledge base as well. We call them KB articles. And this is one that I wrote. But we have hundreds, thousands, I think, actually thousands probably of these KB articles that really, really help our customers and us support our customers, which is great. So why do we have so many articles? Why is it so, so useful? It's useful because all of our engineers at Heroku have the ability to create these KB articles, all of our engineers. And especially the support team. We've basically had this gut reaction that when we notice something that's important, that's not documented, we're like, oh, gotta create a KB article for it. It's just an instinct, right? And you need to build that up in your team. And having that has been so, so helpful. And our engineering teams have noticed that when they create more knowledge base articles, then support stops bothering them so much. So they like that as well. So there you go, these are, this is what the example can look like in real life. You have a very short crash course which preps you with just enough knowledge to get you started. And then a knowledge base where you can search for everything else. Now does this mean that Heroku's onboarding process is like totally seamless and totally hands off and we just hand them the articles and walk away? Absolutely not. But it definitely helps. Especially the knowledge base that we have is so, so crucial to having our team be self-sufficient. Now I haven't even, I'm not gonna go into, in my talk, cross-team tribal knowledge, that kind of stuff because I feel like that's a whole other talk so I won't dive into that. But just know that this does cover most of your bases with tribal knowledge and it's really important to try and implement something like this. Now, let's talk about how we keep these docs up to date. So you've created your crash course, you've created your knowledge base, how do you actually make sure that they stay up to date? Well the first thing is, I'm gonna sound like a broken record, but make them easy to update. Again, give people right access and get out of their way, okay? Just let them do their thing and they'll keep it up to date. The second thing is a really actionable tip which is to add expiration dates to all of your articles. So in Haruka's knowledge base, after three months we automatically mark that article as like, hey, this is due for a review, right? And so the next time one of us comes to that article and we see that notification, that's our cue to go, oh, I should probably just double check that this is still true, right? We don't get like a notification or anything like that, it's just the next time we visit it, we're like, oh, looks like I should check on this, we can go back. So this is an easy thing to do. You don't have to obviously do this programmatically, you can just write your expiration date at the top of your article, wherever you're putting these. Now, do not just put when it was last updated, that doesn't count, okay? Because think about this, what if you bought milk and they put the, instead of putting the expiration date, they just put like, when the cow was milked on the milk, carton, you're like, wait, but can I drink it? What's the deal? And so that doesn't work because we don't know if that article is still relevant one month, one year, two years down the line. It may be, but we don't know. So just pick an arbitrary date, one week, one month, one year down the line and then stick with it, okay? Now, how do we make sure that these docs are actually useful? Because you can write them once, but they may not be perfect, right? And they probably deserve some iteration. So let's say you're the new kid and you're reading the docs, maybe something in the crash course of a knowledge base, and you're like, I don't understand this at all. What, I don't understand this. First of all, that's important. Take note of that and then go and improve that documentation, okay? As a new person, you have to feel empowered and if you're a manager, help your team, help your new people feel empowered to go and make these changes as they see fit because their fresh eyes are so valuable, right? And so as the new kid, do not let the feeling of I'm a new kid, what do I know? Get in the way of improving crappy documentation because I know what this is like. I think one of the biggest challenges working at a place like Heroku where everyone is just stupid smart is learning when it's okay to say, I don't understand this. Spoiler alert, it's always okay to say, I don't understand this. But it does take a bit of bravery because the imposter in all of us will tell us that, oh, you're stupid for not understanding. Look, they explained it twice and you still don't understand it. Look, the truth is, some people suck at explaining things or more specifically, they're two in the weeds and they think that you're in the weeds with them when you're not. I was like, my slides need more Pokemon. So what I've found is that when I'm feeling big and unafraid and frankly entitled to a decent explanation, that is when I feel comfortable to step up to the plate and say, yeah, don't understand a word of what you just said. All right, and then I forced them to explain themselves better. And if you've ever been too afraid to say, I don't get it because blah, blah, imposter syndrome, right? We can all relate, I've got a tip for you. So I get around feeling like an idiot by asking really specific questions. I never just say, I don't get it. I'll say, okay, I understand what this term means in this context, but it sounds like you're describing something different. Like you said the word index and this is what I understand an index to be to mean, but you're using it in kind of a different way. Can you clarify? So be specific, okay? Do not let the other person get away with explaining things in terms that you don't understand. Make them describe the forest, not the bark. Now, there's one more thing I want to address. I know that there are folks out there who have worked very hard to get to where they are. And they had to sweat and they had to labor and there probably was not nice documentation all bundled together for them so that they could easily learn. They had to learn this on their own, right? And they worked hard, but God damn it, they made it. They made it, and now they're at the top and they know all the information, all the information. And that's great, right? That makes them feel really important because they worked really hard for that. And they should feel important and they should feel valuable, good for them, right? But because of the pain they endured, they can tend to feel a little protective, like they need to cling on to the information that they learned. The idea of properly documenting what they know can feel a little bit like they're giving away their treasure, that in some way, they might actually be less important, right? So the question is, will I still be valued if I share everything I know? And I hope it's obvious that the answer is an emphatic yes. Because what you learned is not something you can learn in just like a few articles, right? Doing what you do is hard, so be proud of that. We should not, and for the most part, we do not hoard knowledge. This is why we have the open-source community, right? That despite there being so much information available freely to everyone, there are still people who suck at programming. And there's people who rise to the top as well, right? And that's amazing because getting good at what we do is hard, and that's something worth valuing. As I mentioned before, I spend a lot of time thinking about how people absorb knowledge. And I think what fueled this talk was my own curiosity about how you educate people in the workplace. And in the workplace is key here, right? Because if we have infinite time and infinite resources, we could just create a whole academy for onboarding new people, and wouldn't that be great? But we don't have academies, and this isn't school. This is work, and we have to do work. So how do you provide the time to educate people with what they need to know to do the job, and help them be self-sufficient? You do that by teaching them the basics and giving them the resources to continue learning on their own. That is how you address tribal knowledge. Thank you guys so much.