 That's much more common, cool, all right. So I'm going to go ahead and get started and you have all heard all of my pre-ramblin here but this is the Lord of the Rings and information security in your open source project. I am Amy at Amy on Twitter and I'm also Amy at Red Hat. I am more than willing to be stopped and heckled and questions and all of that as we go through this. This is about a 20 minute rant or so and then I'm again happy to take questions. So do not feel shy. So from time to time in your information communities, you are going to be able to have things happen to you. Anytime that you bring up a server or frankly anytime that you bring up a server with a database and then you write down the credentials somewhere and you forget about them. Well that leads you open towards problems later on if you haven't actually written them down and it turns out that the story of the Lord of the Rings is a great method to be able to follow towards actually fixing these things when they happen to you. Out of curiosity, how many people have actually had something like this happen to them? Only one, only two? Okay, so okay. You are the people who will admit to it and you actually know about it. The rest of you just don't know yet. This will come for you, it will be great. So for me, this came from working at a hosting company when Heartbleed came out. And yeah, oh yeah. So it was one of those things where the release had come through on Twitter for Heartbleed and I kept seeing it come through and I had this reaction of, huh, that's kind of interesting. I now realize that when I have that reaction of, huh, that's interesting, I very, very quickly want to get to the sense of gigantic impending doom or everything is fine. I want to spend the least amount of time in the, huh, that's interesting. What happened for us was we ended up, seeing this come through all the way around the world and one of my Australians who I happened to be working with and I was based out of West Coast, he's based out of Sydney, jumps into chat and is like, hey you guys, we've got a thing. We've got a thing. And it turned into kind of like a lighting of the beacon sort of thing across the world in terms of like, pagers went off and like, you know, people got woken up out of bed and it was kind of like, huh, this is going to be going on for a little while. You know, we should call for aid. And that's the lighting of the beacons and this is where like my concept of the Lord of the Rings came from. So again, it may be cheesy, but it's fun. So I tell you that story to let you know where I started from. And now where I am is a, I am the Gloucester community lead at Red Hat and my focus to be able to feed and water and move that community forward. I can't do that if I don't have good solid infrastructure and I can't do that. And frankly, all of our projects cannot move forward if we're not paying attention to this. So this is why it matters. Okay, yes, we're all very serious about this. So the forging of the ring. Every time that you bring up that server or someone brings up like, oh, I don't know, PH may admin and kind of forgets about it. Yeah, no one's ever had that happen before. No, right? I mean like, pick any interesting thing, service you are going to need with your project. If it has a material impact towards you being able to ship, you should write it down. And I realize that two factor authentication is deeply annoying, but it may in fact be the best thing that we have. So we should use it and being able to practice, better management of information of SSH keys. Don't share them, don't push them around and things that we think, oh yeah, we'll remember that later, we'll fix it. We'll go back and fix it. Just don't do it in the first place because you have the power to forge the ring. And then there becomes a cast of characters. I actually don't think that there's a Sauron in this because no one deliberately does this. You would never deliberately set this up for yourselves. But you can turn into Gollum. If you're the one that finds it and then kind of forgets about it and wanders off and like later on you go, oh man, why is that server there? Why are we doing that with that again? Huh, you've been poisoned. You've turned into Gollum. Frodo is the poor hapless discoverer that ends up actually looking at the logs and going, huh, that's interesting. Huh, what do we have in there? Gandalf is the senior staffer who can read the logs and go, man, you got a thing in there. Right, he throws it into the fire, he reads the logs and you go, oh God, what do you have in there? But the problem is Gandalf doesn't actually come with you on discovery. Gandalf kind of rides off somewhere else and gets pulled off to another bug and he comes in periodically and checks in on you like, so, how are things in there? Oh, huh, you got a thing in there. All right, cool, let me know how you go. And they come back in. There may also be an array of Gandalfs, you may have multiple Gandalfs that come into your management of any incident. The hobbits are the other people on your team who are going to be able to help and each one of them has a piece of knowledge to be able to bring to the table. So, the fun stuff comes in where sometimes you will have Strider and Strider is information security. They come in as you're on your way towards Rivendell, you know that you have a problem, you're now walking towards being able to get through what actual remediation would look like where we're going to go here and information security comes in and says, ooh, I think we can help. We think we know what we're doing here. Everybody hobbits on the way, let's do this. They're probably way less excited about that. I don't think I've ever heard it in information security. I'd be really, really excited about things or a girl in that way. So, you're at the point where you have realized that the impending doom is on you. You rapidly need to fix something. This is your journey to Rivendell. Oh, come on, pick up. Well, next slide should be about communication and why it's not actually moving. I'm not really sure, but it doesn't want to communicate, apparently, if I go back and then forward. There it is, cool. So, communication, we should do that. This is kind of what got us into the first mess, typically. I mean, very rarely you will run into something that has a level of heart bleed. It's usually going to be something much smaller. So, you communicate with the team around you. You communicate with the management structure of not only the project that you're working with, but other people who will need to know. I am hoping that you will have information security alongside you when this happens to you, not if, but when. You will also need legal, because there actually are people that are going to care about this, even if you're just an open source project. People should be using your project, and hopefully, if you're doing it right, they're using it for critical things, because you're important, right? Public relations is going to be important longer down the road, because you will want to be able to say, no, no, cool, we destroyed it, the ring is done, all is well, this is how we've cleaned up the shire. Things are great. Previously, when I've been in these before, I have done things like keeping a running document of what is going on where, who is responsible for what, what are we doing, how is this moving along? Also keeping asynchronous private communication on unlisted IRC channels was another way that I have done this in the past. I don't know if I will do this in the future. This worked for me as this happens to me. I've also done daily calls to be able to confirm current status, next steps, where are we going? Again, your mileage may vary. This keeps the pace moving so that everyone knows where we are going. And what does done look like and what does success look like informs how your weeks move. Sometimes you may end up in a situation where finally fixing the incident is going to take things well outside of your hands and it may take months. May this never happen to you, but it does. And so understanding here are things that we need to work with longer term, short term, and moving on from there. Questions, comments, stories? I know, I realize I'm actually giving like the real part of the talk instead, just Lord of the Rings, we'll get back to that, it's fine. And the actual remediation lines up to make a checklist, follow that checklist, remember that having that checklist is going to be able to save you from other problems down the road. So the way this works is, okay, you've made it to Rivendell, you have the Council of Elvron, everyone who needs to be involved is there, hopefully. You know, you'll have the elves, sometimes they're helpful, sometimes they're not, you can't really be sure, but they're definitely there, they're going to be there. You'll find out later, the dwarves want to get this done as soon as possible and infosec may also be involved, sometimes they're men, sometimes they're dwarves, sometimes they're elves, chews. Other pieces around in here, the minds of Moria end up when you didn't quite get through discovery or something got missed in discovery. Something that should have been remembered was not. And then you end up in this kind of terrible place of either something leaked, something worse happened. You were not sure what to do next, so it became a lot more urgent. Maybe the minds of Moria never happened to you, but now you can use this. When something happens, you're like, oh, we should have remembered that. We're in the minds of Moria, everyone. And a note on legal, because sometimes legal will come down as gladrile. They want to help you. They want to be able to move all of this forward. They want to be able to make sure that all of this is going to come out and the best light possible for you. And I actually probably need to give another version for the legal side of this, so that's another talk. But sometimes legal can be Boromir. And sometimes Boromir is like, no, no, no, no, I'm taking this and it's never going out. You're just going to be a dead floating hobbit in the river. And this will probably never get out in public and this will probably never be something you talk about ever again. Honestly, there's not much to do about Boromir. The better way is not to have any information security problems in the first place, but we'd like another world. What will certainly happen next at this point is you will get halfway through where you say done is and half your team is going to get pulled off to work on another bug. Also, all the hobbits getting kidnapped. So the Eye of Mordor is the press. Sorry, the Eye of Sauron ends up being the press because you do not want them to be able to find out about this until you have destroyed the ring and everything is beautiful. And you have gotten to the point where, no, we understand there was a problem, we have resolved it. This is the things that we did to be able to resolve it. We're done, or here's how everyone knows about it and this is what everyone is going to do moving forward. And your ultimate goal is by things, don't actually work, come on, Slides, there. That is what success looks like. Success looks like we are better than before. We know much more about our infrastructure. We have all of this knowledge. We know pieces that we didn't know before and we are the better for it. This is what success looks like back in the Shire. On the other hand, you may also realize that you may turn into the hobbit again the next day, depending on just the way that the internet works. And the fact that all of our systems are so much more complex than they were four years ago. The things that we are working on now are the things that Google was thinking about in production and now they're all of our problems. It's just, no, it's true. It's like all of the things we're like, oh, God, yeah, four years ago. Wow, that's a really interesting idea. Now, this is currently my problem. Awesome. I really hope I've written that down. So congratulations. You've survived the shadow of the game show of Infosec on your doorstep. So my real advice is making sure that you actually are communicating within your team as you find out about it. Being able to solve that question of, that's interesting, that's really interesting. You want to know as soon as possible, hopefully within hours of having that feeling. You may not always be able to do that. Work as you get towards the end of this, towards running a really good root cause analysis, which ostensibly will help you prevent or change things about the way that your community's process is working for when this happens again, or being able to communicate with other communities that this happens with because you may possibly have cross-pollination with some of their work. Do make sure that you actually have the right people involved as well. At least in those beginning meetings and in the root cause analysis meetings, and those tend to be really epic meetings if you run them right, that's another chalk entirely. So I won't go into that too much, but also making sure that you do have the information within your own circle of trust. Everyone that is on the quest should actually know that they are on the quest. You would be surprised how people don't actually know that, no, no, no, some of this stuff should not actually, don't put that on Twitter, don't. And the words of David Foster Wallace, I wish you way more than luck. So, thank you. Other stories, comments, questions, things that people want to be able to talk about. Yeah, nothing? You have no war stories. Yes, it's more about knowing what you're, feeling like you know how to be able to handle them when they do happen. Because trying to prevent them is like trying to prevent water running downhill and like also Chinese hackers. Oh wow, okay, great. Shooting the messenger. Yeah, because it's your story, I won't retell it for the recording, but the, I attempted not to be able to have like the, here's how serious these things can be because that's a huge downer. I mean, I feel like the ring of power that could destroy us all is probably enough, you know. But like these things matter and there's a reason why you have to take them very seriously when they do. So, thank you. Yeah, I will actually shut the microphone off and we will tell stories as needed.