 I'm going to talk to you about security metrics that matter. So there are all sorts of different types of metrics that you can gather, but not all of them actually make sense or have value. So today, we are going to talk about the value of measurement and reporting and improvement. We're going to talk about what and how to get the metrics that will truly matter to your team, but also your organization. Then I have a case study here, but I'm not sure if we're going to have time to get through it because we have 30 minutes. So let's not waste time, let's go. So I'm fancy, that's what this slide is about. I do lots of stuff. I think I'm cool and so does my mom, and hopefully you feel that this slide and the introduction tell you I'm qualified to give this talk. So enough about me. I want to talk about metrics. So metrics are either a method of measuring something or the results obtained from measuring something. So we're going to talk about the results obtained from measuring things and we're going to talk a little bit about how to measure and what you can measure. But before we do anything, we need to talk about, well, first, we're going to talk about how it's important and why. We're going to talk about what vanity metrics are at length because I really don't want you to get the wrong things, and then we're going to talk about how you can measure and what to measure. Okay, so we measure for two reasons. One, because we want to report. We want to be able to tell all the bosses, all the things that they really, really want to know. But we also want that data so that we can improve, so that we can create a totally kick-ass security program. So to repeat, reporting is for management and metrics are for us, boots on the ground. And so a little bit more. So when we report, reporting can help us get money. And if you're on a security team, you know tools cost a fortune, training is pricey. There's like, you want to hire more staff, you don't have money or you can't offer them as much money as you'd like to. And so when we report, it can help us get budget for tools, resources, staff, all those things. Sometimes reporting is part of compliance or audit. So you've got to check that box. It's important, it's a thing you have to do. Reporting makes our bosses happy. And so you want to create reports that are the information they want, information that makes sense, information that's helpful, even if it's bad news, you still want to be really accurate with the way you give the bad news because your boss needs to know what the F is going on. And so what we need metrics for is improvement because we want to get more awesome at what we do. We want to hone our craft. We want to make sure that our program gets the best results it can. We're not going to have time to do every single thing that we want. And so we have to spend our time and energy on the things that matter the most. And so this is the last time I'm going to say this, but basically reports are for bosses or for other teams. Well, metrics are for us. And so we're going to talk a lot about metrics and then you can report them. Okay, so vanity metrics. For anyone that has read The Lean Startup, which is an amazing book by a dude named Eric Reyes, or the book Metrics That Matter Using OKRs, I can't remember the rest, but I'll have the book at the end. They explain at length what vanity metrics are. So those are numbers that look good, but aren't actually valuable. So I'm going to give you some examples of vanity metrics. Like if you just weigh yourself on a scale, if you're a person who's been going to the gym a lot and working out and getting lots of muscles, you might see that number go up. If you're trying to lose weight, you want the number to go down, but maybe you're changing fat into muscles, et cetera. And like again, just the number on the scale is a vanity metric. It's not necessarily helpful for exactly what you're doing for every single person. And so let's talk about vanity metrics around work instead. So I used to work somewhere and my job was measuring if people would click on our articles and read our articles. So it was our job to create content that would help everyone how to use our products. And my job was how to help them use our products securely and how to help them use our security products well. And so the idea was is that if there's enough content that customers can figure it out on their own and they don't have to bother calling support, it makes the customer experience way better. They know what to do. They know how to use the products safely, securely and get good results. And it saves us money because we have less calls on support. So what we used to do is we would measure how much they clicked. And so I had a friend on the team, super awesome French Canadian dude who lives in Quebec. And he would get like 5,000, 6,000 clicks per article. And he was writing, you know, about coding and I was writing about security and I'd get like 400 or 500 clicks and I was like really distraught. We had this dashboard and I wanted to be at the top of the dashboard because that's how I am. I'm like, how'd you do that? And so he took me aside and showed me, yeah, I just, I share a lot of things on Reddit. And the people on Reddit are really mean, but gosh darn, do they click? So I started sharing things on Reddit and my numbers went way up, but my engagement didn't go up. Like there were no more comments. People weren't sharing it. So we went to the team and we asked them if they could give us how long the people stayed after they clicked. And so it turned out my original 500, they were staying 1.5 minutes on average. That's awesome. But his were staying 1.5 seconds on average, which is a bounce. Like that means they've clicked on it but they don't actually read anything, they don't get any value. And so he was actually getting very, very few reads. And we figured out that basically Reddit is the cesspool I always thought it was and that people weren't really clicking and actually getting value from it and posting there, it turned out wasn't really worth our time. And we got a lot more value sharing things on Twitter, on LinkedIn and other places. And it turned out that, you know, like it's not that it wasn't worth doing it all but the clicks from there were all fake. And so we changed our click rate, they had to stay three seconds. And so that was a vanity metric that we were getting. I didn't marry him, he was getting 5,000 clicks because we weren't getting what we actually wanted, which was people to read our articles. Another, another instant, another metrics is like an important metric. So not a vanity metric is when I first started doing instant response, every single instant we'd have a post-mortem report. After three months I took all of them and I put them in different categories. And one of the categories was application security flaws. And it turned out 26% of our incidents were based on that. And I showed my bosses and told them about my concern and I explained to them, like, I could have caught any of this with a quick scan, right? Like, and if we actually enabled our devs and gave them some education and thought on this, like we could easily have eliminated all of these incidents and they were extremely, so it's 26% of our incidents, but it was the absolute worst. All of them were the most serious, they cost the most, they took the longest amount and one of them was super embarrassing. I'm like, if we could get rid of 26% of our incidents plus get rid of all of the critical, super scary embarrassing costly ones, I mean, that's good. And I showed them how much it would cost, how much we would gain and they let me launch my first AppSec program. And so those were not vanity metrics, like being able to show them the return on investment, I'm like, listen, I'm here on a two-year contract, I have five months left. In five months, I believe I can accomplish this. This is how much money it costs to respond to these incidents way more than that five months worth of my contract. And during that meeting, they approved it and I watched my very first AppSec program and I used metrics to do it. The last one is, is I worked at another place and I had been a temporary CISO. So for like four months, I was a CISO and I ran everything. And then it was time for me to go because I decided that being a manager was not my jam and I really wanted to be a pentester instead. So I switched. And so they had a new person who didn't have very much experience who they were letting try out the job for a while until they found a permanent person. And so we're having our first meeting with the boss where we're doing a Passover. And so before he said, yeah, I made an awesome report. And he shows it to me and it is a circle. And part of it is orange and part of it's yellow and part of it's red. And on top it says circle of risk. And one said like 42%, whatever, whatever and add up to a hundred. And I was like, what does that crap mean? And he's like CEOs and like executives, they just wanna see pictures. I'm like, that doesn't mean anything though. That there's no value in that document. Like 46% of our vulnerabilities that we're aware of are high. What does that mean? Has it gone down from last month? Like what did it used to be? Did we find more because we got a new tool or did we find more because we're doing a bad job? Like this doesn't communicate anything to me. And he's like, no, no, man, he'll like it. And so we go into the meeting and he presents his circle of risk. And I'm like, just like blank faced, right? Because I don't want to bias our boss that I think that's crap. And the boss looks at it and he's like, this is crap. It doesn't mean anything. These are vanity metrics. This offers zero value. He's like, do you think I just wanna look at random pictures? Why don't you bring in a picture of your pet? That would be just as useful to me. He's like, oh my God. He's like, just go, just leave. And so then when the guy closed the door, he's like, oh my gosh, Tanya, oh my God, you're without you. So we can't present vanity metrics to smart people because they're not gonna put up with that. And I hope that you sort of understand what I mean by vanity metrics now. But let's go on to you getting the things you want. So for the next slides, I want you to think about your security program goals. I want you to think about what can help you get there. And then I want us to measure it. I want us to measure every single thing that you think might get you there so you can figure out what is actually getting you there. And then we're gonna circle back a bit to it. Okay, so metrics versus measurement. They're not quite the same. We can't measure apples and oranges together. And so depending upon your organization, you might use a scale of one to 10. You might use a scale of informational, low, medium, high, critical, and oh no. But what a lot of larger organizations do, for instance, government organizations or enterprises, they'll use an in-house risk score. So very briefly, I'm gonna go over some of the things that you could have based on an in-house risk score. But again, I'm conscious of time and I don't wanna cut off the end because I will talk about what you should base your in-house risk score at length. I'm all obsessed with measurements and data, but I'm gonna try not to digress too much. So a lot of the things that in-house risk scores are based on or should be based on are ease of exploit. So when Spectre and Meltdown came out and everyone's like, oh my gosh, it's the end of the world. I'm like, it took six PhDs a year to write that exploit. I don't know six PhDs that hate me that much that they're gonna work that hard. Like you need to have a high like attraction rate to malicious actors to make that worth it. If you're Bob and Alice's flowers and you run like two shops, I don't think that they're gonna worry about doing that attack to you. So is the system public facing or is it internal only? Because there's different risk. You still have to protect internal only apps. Do not get me wrong, but the risk is different. What privileges gained, right? Like are they able to increase their privilege or get privileges they should not have had at all? How long has this vulnerability been known? So that's kind of important. What are the specific risks to your business? I used to work in elections, Canada and they have totally different risks than for instance, I used to work at Microsoft, right? Like the risks and the things that are very important to those two organizations are completely different and they're both really important to them. And so while like the people that run the elections don't have the same concerns, their concerns are still very, very valid. And so doing stuff in your in-house risk score management based around the specific risks to your organization is very important. One shoe does not fit all. So data classification, so is there sensitive data in there? If so, what is the sensitivity level or not? Is the confidentiality, integrity or availability affected by this risk that you have found? Because that's our mandate. Okay, so now let's talk about metrics that matter. So I'm gonna give a bunch of security program metrics and then a bunch of incident metrics because I really, really believe reducing incidents is very important. Okay, so metrics that matter to you, to your security program. So time to detection. There was a slash dot article about how it takes 276 days on average to detect a breach, that is scary. And I believe that's from 20, that article was from 2018 or 2019, but that is a long time. Time to remediate or patch. This is also really important. If you're finding vulnerabilities like nobody's business, you found every single vulnerability, but it takes up to a year to get things patched, it does not matter, your program is still not succeeding. Do you have a baseline security posture that you expect and how close are to you it? And how much closer are you getting? So quite often I will try to create service level agreements. And so I wanna measure if I, the security team and meeting the service level agreements with Dev and Ops, but I also wanna see if they are meeting their service level agreements with me on patching and remediation. What is the average number of vulnerabilities per system or app? Does it go down or up over time? And are you detecting the same types of vulnerabilities or reduction in the number of vulnerabilities? So are you able to detect new types of vulnerabilities? And that goes back to two before that. So the average number of vulnerabilities per system or app, if that shoots way up, you should look at that and think, oh, is that when we implemented that new security testing tool? Yeah, because it found all sorts of new things we could never find before. And that's a metric of you succeeding because you're now able to find so much more stuff you couldn't find before. If for instance, let's say you gave a bunch of education on specific topics, and then after the incidents of that occurring, like that vulnerability being written into code goes down, that's you showing I did this work and this was the positive effect it had on our teams and our systems, like our security posture went up and I can prove it. Oh gosh, bosses love that, but I love that because it's really, really important to me that I am actually succeeding and I'm not just going to work to get a paycheck. I need to actually make my organization more secure. After targeting specific vulnerabilities, do they decline? There's very little point in trying to eradicate a certain vulnerability if you're not measuring if you're actually doing it. I'm a huge fan of setting goals and then measuring my way to getting there and making sure I actually do. So security incidents, they are the most humiliating, damaging and expensive way to deal with a vulnerability for the first time. They are awful. So I used to do incident management quite a bit and incident response, like investigating when software has had breaches. And I find it to be the most thrilling and exciting and also the most stressful part of security for me. It's like I'm addicted or something. It's like it's so exciting, but I also like it's hard for me to like put it down. I want to work on it day and night until it's done. So if we can reduce the number of incidents, the length of time it takes to resolve each incident or reduce the damage they cause, this is one of the most high value goals that we could potentially set for our team. And so I want to talk briefly about incident metrics that matter. So when we have an incident, we want to measure and put it in the postmortem report. How long did it take us to resolve this? How long did it take us to detect this? How long did it take us to triage? Like to say for sure it's an incident rather than just a security event. What were the types and categories of incidents? So which ones are app sec related? Which ones are malware or ransomware related? Which ones are the public being weird? When I worked at various different government departments, sometimes the public would just do things where you're like, that's so weird. And you'd still have to investigate. Like sometimes they would send us code or they would send us things that I guess were encrypted. And we'd have to investigate like what is this? People would report bugs to us. We'd have to verify if it was a bug or not because quite often it's not. And so categorizing all of those things is important and seeing how long it takes. And you can't make the public not be weird just so you know, I've tried. They're not gonna stop being weird. But you can solve a lot of other types of problems and prevent them. Was your process followed or was it not followed? And if it wasn't followed, why? Sometimes it's cause no one knows what the process was and you've done a bad job of advertising. Sometimes it's because the process is such that then the person will break the rule of their job or their manager and they're not allowed to do that thing. And like you have a conflict there that you need to fix. An approximate estimate of the cost and damage of this incident, that's really important. And when you calculate this include all the hours of everyone including their overtime that it cost plus everything else. So let's say you had to hire a third party to come in and do forensics on a hard drive. Let's say everyone had to stay overnight and then you had to give all of them overtime. All of those things should be included in there. Are you repeating the same type of incidents over and over or are you finding new types? That's so important. If you are able to detect more types of things that's really good even though brief story. I started working somewhere and in the first couple of weeks I reported 26 security incidents. And my manager at one point she looked at me and she was super pissed and she's like, Tanya before you started we didn't have any incidents. I was here a year and we had zero. And of course I responded this place is a flaming dumpster fire and before you had no one that knew where to look. We are in a much better shape by knowing because all these things I'm reporting are more than a year old. Most of them, some of them were just six months but a lot of them have been going on for a really long time. And I'm like, it's way better that we know. And then she's like, you're right. I'm angry. And I'm like, me too. She was actually a really great boss. She was really nice. That was like her one. She was just, it's upsetting, right? And she's like, could you just calm down? I'm like, I can't. I can't not report things I see. I'm sorry. Anyway, she liked me anyways. Do other teams understand what they're supposed to do and do they cooperate? Sometimes other teams do not cooperate and this is a big deal and you need to know why. The real reason why. Not, you know, this person's really hard to work with. There has to be like, there's usually a legit reason if someone's not cooperating during an emergency and it might be that they don't understand the importance. Was a postmortem performed every single time? That is really important. A postmortem is the thing that helps you first of all, gather these metrics and second of all, make sure that doesn't happen again. If you are not doing postmortems, I consider it a big problem. Quarterly review of all incidents that, so I take all of the postmortem reports and I chunk through it. I usually put a lot of stuff into Excel and I try to come out with patterns and I try to crush those patterns. And time between incidents. So when I worked at Elections Canada, I ran the security for the election. It was really exciting in 2015 where we voted in Justin Trudeau and when the election was over, they rested the staff. So what they would do is they would give us really, really easy, simple things to do and like do slow cleanup and on purpose make sure that we were covered because they had been running elections for 164 years or something and they're really good at it. They're really, really good at it. And they know that burnout is real and that was the longest election that we've ever had. So it was 89 days, I believe and usually our elections are 36 days. We were exhausted and they don't wanna lose their staff. It's way cheaper to treat your staff well than to burn them out. And so if you have an incident team that is constantly responding to incidents they will burn out and guess what's gonna happen? They're gonna quit and then you are so screwed. And so it's really, really important that you make sure that there's time in between incidents where they can rest and if you see zero on that all the time that's not acceptable. And then lastly, did you have access available? So access as an accounts that actually work and tools during the incident and if not, why not? And you need to fix that for next time. You must have your team enabled to do their job and do the investigation on the spot. Okay, enough about incidents, I'm very passionate. Okay, so tools for measurement. If you are gonna buy a tool so that you can do measuring a metrics it needs to have the ability to see patterns, the ability to keep your information safe. You must protect this information. It is really important and really valuable. You should not get into the wrong hands and two, ideally the ability to automate data collection. That's more like a nice to have but it's really, really, really nice to have. So here are some tools that I have seen used in the past and then I'm gonna tell you which ones are good and which ones aren't in my experience. So Microsoft Excel, email and keeping documents and folders, vulnerability management tools, checking a bunch of different security tool dashboards, other tool dashboards that are not meant for vulnerability management but that are meant as business data tools such as Power BI, your cloud dashboard, if it accepts external data. So I used to work at Microsoft, so I know way more about their tools than anyone else's and Adverse Security Center has APIs and you can put your data into it. Saving all of the database, all the data into a database and then querying it yourself. So Microsoft Excel for the win. I have seen it used a lot and if you have a small team it can be good. I don't work at Microsoft anymore but I freaking love Excel. Saving things in emails or documents or folders that sucks every time it's a mess, every time people can't get the things they want, it's not organized. I strongly advise against it. Vulnerability management tools like Defectojo, Threadfix, Xero North, et cetera, they're made for this. Checking a bunch of different security tool dashboards, this never works, it's always bad. You always end up having to export the data into one place so that you can actually see the real trends. Using other dashboards like Power BI or business data tools, awesome sauce. If you have someone that knows how to do that, bringing stuff into your cloud providers dashboard, it's kind of like Power BI except it's cloudy. Or saving all the stuff into a database and querying it yourself. If you are a really small team, i.e. just one person, this works awesome. It's totally fine. Like protect it still but this is something that can work if there's one or two of you and you're both highly technical and know how to query a database. I've seen this work great. But I really advise against keeping things in folders, in your emails, or checking a bunch of dashboards. It's always a mess. But I suggest automating as much as humanly possible. The vulnerability management tools are built for that. You can write little scripts. I mean, we are at a hacker conference, right? So a lot of you would know how to automate this if you wanted to. But I would say that automating as much as humanly possible applies to everything in life. I have a giant garden or a small farm depending upon how you look at it. And I have all my watering automated because I'm not getting up early so I can water things. Okay, so now I briefly wanna talk about improvement. Then I'm gonna give you a couple free resources and then I'm gonna bid you a due. So improvement, think back to your program goals. Cause each of you work somewhere and I bet that there's goals where you work for your program or your job specifically and you wanna improve that. You wanna be that employee where the boss is like, I don't know what we would do without that person. If that person quit, we would be so screwed. You can use metrics to do this. Okay, so what I like to use metrics for, so I do consulting from time to time, besides like running my academy, cause apparently I just like to work every moment of every day. But anyway, I don't know, we all have things we like and I like keeping my tools sharp. And so whenever I start somewhere, I get all their data from all of their systems. I figure out what the top three problems are and I educate on those top three. And then I watch those top three go down. Yes, immediate renewing of your contract. Oh yeah, bug type eradication. You can't eradicate a bug if you are not measuring, right? And you need to measure each of your efforts separately to see which one's working the best. Is education working the best? Is writing unit tests to look for that working the best? Is just doing lots of scans and then finding it and throwing the bugs into the backlog working the best? What is getting you what you need? Compliance. I really wanna know if we're meeting our policies and I wanna automate it if I can. You being able to show metrics to your boss of like, we're 98% compliant with this. We have almost everyone using MFA. Very few people are doing this or that. All of these things are things your boss wants to know and hasn't asked. Are you meeting your service level agreements? Are you repeating incidents or are you finding new types of incidents? All of these things are things that you can do to improve. And so, oops, wrong. So this is a book that I really, really liked. This book got me obsessed with metrics and I already loved data. But when I read this, it was all over. My manager at Microsoft had suggested, oh, maybe if you wanna read it, I'll buy you a copy and I read it. And then I annoyed him so much about metrics from then on. He was just like, oh my gosh. Everything was metrics from that point forward. But anyway, I'm that employee where I'm like 500% every day. But then we got to nerd out about metrics together and I thought it was great. So what did we learn today? So we learned the value of metrics and using them for reporting for our bosses and improvement for us. We learned about what and how to get metrics that truly matters. And we went through a few ideas about how you can apply this information so you can get some good results. So I wanna go over some resources now, just very briefly with you because we have 25 seconds left. So here are some awesome books. I'm super biased. Obviously, I think this book is great, but also my book, my book's really great. Everyone on the planet should get a copy of Alice and Bob learn application security. But all of these books are really quite good. I also wanna tell you, I have a podcast. It's called We Have Corporal Podcast and we learn about application security and the different types of jobs that exist within information security. Every Monday on Twitter, I do cyber mentoring Monday. And if you want to find a professional mentor, this hashtag can help you. Other resources, me, I actually moved my blog. It's now at wehackpurple.com and they click on blog and then yeah, those are my blogs. So I'm moving stuff from that address that's there. And with that, I wanna say thank you so much. I wanna say thank you to everyone who watched this. I wanna say thank you to the NorthSec crew who worked so hard to make this conference truly amazing. I've been to this conference more than once and it's just, it's really, really good. So thank you. At Tumei Ami, I'm Marielle. Thank you, I'm time to thank her.