 Measurement is important to process improvement. It's a fundamental part of DevOps. It's difficult to improve, which you don't measure. So in the next session, we'll see how you can leverage DevOps metrics to transform your application security program. Mark Thomas-Peters with Technica will discuss transforming DevOps metrics for security. Now he cautions that this is not for those of you in a honeymoon period of DevOps when it's fresh and new gains are easy. His advice is more specifically for those who may have plateaued. So get ideas for how to get to the next level. Over to Mark. Hi, I'm Dr. Mark Peters from San Antonio, Texas. I'm excited to be here at GitLab. I'm excited to be having a chance to talk about transforming your DevOps metrics for security, bringing those security teams into the flow and bringing them into the process. As I mentioned, I'm Dr. Mark Peters. I work for Novetta Corporation. I work on various federal or have worked on various federal programs for the Department of Defense doing different types of cyber weapons, as well as other cyber programs finding a way to work those together. I'm also a DevOps Institute Ambassador as a U.S. chapter chair. I was formerly worked in U.S. Air Force Intelligence, including publishing a book called Caching it on Cyber Power, looking at how 10 years of cyber attacks created economic impacts. But today I'm here to talk about DevOps and I'm here to talk about how to transform your security metrics. So we're gonna look at a little bit about the basics. We're gonna look about how we think about security and how we think about security and association with our DevOps flows. And then we're gonna look at defining some metrics. And then we're gonna talk about accelerating transformation, really getting some takeaways that we can use that you can take back to your business with you and use for a successful value acceleration. So first off, this talk is completely 100% not safe work. If you're sitting in front of your computer at your desk or you're sitting at your home with your kids running around, it's absolutely safe for you to watch anywhere, anytime. It doesn't talk about anything you're shattering, but it does talk about some ways to move those metrics forward. There's no viruses or malware. I swear, when I originally recorded, there wasn't any viruses that went in there. But it's also not likely to get you into a job. This is not gonna leapfrog you into a new position or a management position or something else based on those advanced metrics. But I do also have to caveat because normally I'd be giving a federal briefing and I'd have all kinds of classifications and standards and other processes in it. But the caveat here is that if you've made one of those transformations, one of those DevOps safe, agile, scrum transformations in the past 12 months, you're probably still making gains without the need to advance your metrics. So this talk is probably not for you. Just like when you do exercise and you start out a new exercise program in that first part of the process, you make significant improvements. And those improvements are based on just the fact that you've started and you're making growth and you're seeing improvements and you don't really have to change a whole lot to get that success. It's not till later that you really need to change when the growth starts to slow down. Now, it doesn't hurt to start thinking about it early so that you can plan for that growth and know how you wanna expand. Just like you know you plan to be in the gym more often and you need more days to get more growth the further into your program you get to show smaller gains, you need the same type of thing here. Those metrics, logs and traces we talked about, those don't fix your problems. Those are just numbers. What they do is they provide the growth to advance the metrics, to fix your problems through humans because it's about the humans that fix the problems. It's about how you perceive your practices and how you perceive those practices interacting with your security solutions and to be able to create those solutions through experimentation. Now obviously not all your experiments work, not all your metrics will work, not all your metrics will get you the right answer you're looking for. But every metric gets you a little bit closer to that process and gets you a little closer to moving forward. So when we look at that, the first thing we look at is how do you think about security? When you sit down and you sit down a team or a new team or you onboard someone, say what do you think about security? You hear that it's the department of no, that it's those guys over there, we just don't talk to them. There may be a used to be dev. So is security the problem or is it that you don't think about security? Now I've seen both aspects. When I worked with the team, they were really excited to come show me that they've integrated security. They had this great new security worked in their pipeline. As a matter of fact, they were using GitLab and they pulled the scanner in the pipeline and they set it up to run. And I said, that's great. I'm really excited. I'm so happy to get my hands on the results. Let's see what it's got. They said, well, what do you mean? And I said, well, you ran the scan, where are the scan results? They said, well, we set it to true. The scan runs every time. And then they said it to delete the results. So they were doing security, but they didn't get enough to really compare it to anything else. And we fixed it. We made it better. It was a misunderstanding, but there's ways to get through that misunderstanding. To bring those teams in a line and it starts with metrics. It starts with having a common framework of understanding to move forward. And that's when we put the problem on the table. We said everybody down. We said, we need a common metrics language. We know that 20 years ago, 10 years ago, the security guy did ops or dev or network. And they had a great deal of success. And they did good at it. Somebody said, hey, you know what? You understand this really well. We need to move you up to the compliance section. We need to move you up to regulations. And they went about into that little closet into a separate area, and it all fell apart. Because now we need to bring them back to the same team. We need to be back on that faster accelerated cycle, on that sprint cycle, and need to find ways that they work together. We need to develop ways that we can have integration parameters, not just the initial concepts of security and the initial concepts of moving forward, but what are the DevOps metrics that we use together? What are the risk management and the security requirements that we use together? Because in that same story, I was just telling about the guys that were running the scan that set it true. At the same time, we were running our scans, but we were running our scans on the deployed version rather than the development version that they were moving forward. So when we would give them the scans, they wouldn't actually use them as the results, as their vulnerabilities to compare, because they weren't enough in line with what they were developing. So we have to find ways that we get our both deployment product at production in line with the developer so that we can move security together and we can coordinate, that we can have a full DevOps structure all the way through, and make sure that we do our risk management and our security requirements all the way along the line. This always starts at any time you do a transformation, you have to start with where you are. You have to have a real good sense of where you've sat down to be, of how your processes are moving forward in what you want today. What do you want from security? Do you want vulnerability management? Do you want them to fix everything before it breaks? Do you want to be on the cutting edge? Do you want to be so far ahead that when the adversaries find a gap in your system and they find a hole through pen testing or something else that you're already onto the next development, that would be great but it doesn't happen very often. It's tough to get to that place where we can move that fast. So it starts with what metrics do we collect today? What metrics does your team use? Who collects them on your team? Are they transparent? Does that, like the results for the dev team, do they only go into the dev team? Do they collect separate in their separate environment in their separate VLAN, they log it off somewhere different and they find a way to get that different process in? Where do you want to go? When you develop those metrics, where do you want them to take you? What are your goals? How do you share those visions? How do you connect those metrics to those visions so that you get to what you want? And then of course with DevOps, how do you align those outcomes to the value stream? Because security delivers value and enhances value just like Dev and Ops deliver value. But you have to know where those are and you have to know how those processes go forward. So we'll move on to DevOps, right? Because we have to talk about some of the DevOps terms because you can't have a DevOps presentation without talking about DevOps terms. In DevOps and security is important. It always starts with flow. We need to know that how security accelerates the flow and how security delivers flow. If the flow is only set up to deliver every three months, then security has time to work in. If on the other hand, you're delivering multiple releases a day, you really have to understand your security because you have to understand how the different security aspects fit in to make those changes, to make those nudges back and forth. In working for a federal program, when I started, now we took over a new contract. We went from delivering once a year to once every three months. Now, when you look at Accelerate by Forest Green and you still haven't made a really big change. You haven't made an improved maturity, but you have if you consider that went from a year to once every three months, that's a huge improvement. For government, that's outstanding, especially when you're dealing with our gap systems. And that's the feedback that comes back, but that gives you time in that feedback to deliver those longer security cycles, those longer metric cycles that lets you find the growth. If the growth isn't where you want it to be, if that growth is slowing down, like we mentioned, similar to the weights and exercise, that's when you start developing the experimentation. You find ways to experiment through your security cycle or to test your other cycles against security to make sure that your pen testing or your unit testing or your fuzz testing is going right where you need to be. And we measure these security successes through logs, traces, and metrics, the three measurements. We have three ways, we have three measurements. Logs are about time series data. They track everything that happens on a system or a set of systems or a VLAN or a network or an environment, and they measure everything in time. Traces are a specialized log where instead of going through everything, they just take one function, they track how that one function works over time. It traces those inputs and outputs for that specific application, that specific program. While metrics are an aggregate, they take numbers from a variety of places and they use those functions together to bring the process into line. So with those, we see the four DevOps metrics. We see lead time to change, deployment frequency, mean time to recover and change failure rate. Those are what we normally talk about. How fast can we get out there? When we get a new idea, when we get a new feature, how long does it take for us to actually deliver? Or how long does it take for us to deliver bug fix? How often do we deploy? I talk about once every three months. We like to see 10 times a day, right? That's the accelerated, the truly mature, truly innovative ones that are running fast flow and feedback. I can get multiple processes out there a day. And the bigger corporation, maybe the smaller the change you can make, the more you can get out there every day. How long does it take to recover? If something fails, how long does it take for us to get a new one? This is a lot of the impetus behind a lot of the Kubernetes, right? Distributed immutable and ephemeral systems. That they go out there when they crash, we just reload it up again. We move the load from one system to the next. We use our distributed networks to move us ahead. Or a change failure rate. How often do things break starting out? This is that pipeline failure I was talking about. That pipeline scans, because we want to know when it breaks or when it changes. So that gives us 10 terms. And all those 10 terms, did you see anything about security? Now we kind of talked about security, but none of them are inherently built for security. And the reason is, DevOps is not built for restrictions. And security inherently occurs as a concept that people think of as a restriction. But that's not always true. Because DevSecOps still seeks accelerated flow. That success may have a result in promotion of the compliance, but now we can bring those folks to the same team. Because they still like doing it. And we're all tied into those same goals, those same objectives and that same value stream. So you have to say, where do you set security for working together? And although I'm using security as my primary example, these same type of terms apply to configuration management or documentation, tech writers, or release approval. These things slow or process down. But sometimes as we get back into bigger corporations, or we get back into just expanded processes, we find we need these things for a little bit. So we need someone to manage to champion these functions until they get done. And then we need to move forward. So maybe we need a metric to give us some insight into how that process works so that we can move to the next step. And do that, I wanna show you a couple of quick samples about advanced metrics. Now working with a bigger team, you bring in a lot of people. And we talk about these for days and days and days or hours and hours. And we'd find ways to make it work. So when I talk about our DevSecOps, we wanna build an analytic, right? We start with DevSecOps. We build those out. And I look at our four core metrics. I don't see any of those metrics that directly apply to security that I can really pile on under that core. I see lead time to change the deployment frequency, kind of its Dev metrics and change failure rate and meantime to restore kind of his ops metrics. But then I have to be able to break it down further. Maybe for a dev, I wanna see some pipeline success metrics and some languages use and how often the devs report the different coverage, the different coverage of scans. But coverage of scans starts going into my security metrics that don't appear in that section. What are the risks released? What is the difference between the previous version and the new version in terms of risks, in terms of vulnerabilities? How much scan coverage do I have? How much testing coverage? How many tools am I using? How much time does it take to run the various tools? If I'm running a whole suite of security tools, is it faster or slower? How many scans do I average per release? Do I normally have to scan multiple times before it makes it up to a merge cycle? Or do I just need one and then I make those fixes then? And even for ops, you can break those down and say, hey, there's some other metrics I wanna see. What about my operational time? How many analysts per failure? How many people does it take me to get into that failure and figure out what's making the failure? How much time to remediate? How much available time do I have where I'm not doing anything? Not that ops is never not doing anything, right? Because we know they're working hard. But there's some processes in there about what are we doing the rest of the time? How much do we have this unplanned work? We talk about making work visible and time thieves. And this is a great sense of figuring out where the time goes. And that's not just for ops, that's all the way across the board. Now let's say we move just beyond that DevSecOps, right? When we talk about DevOps, we have to talk about columns. Culture, automation, lean, metrics, and sharing. When we look at those first four, those kind of go into the middle section, but we can do the same type of process. We can build from the process and start looking at where some of the other metrics are that we can advance for our DevOps and find out where some of the security functions happen. In the middle, you see there's one for extra role of security. Where do people take on security functions that are not required by policy? Is what we mean by extra role? Do you have people that are working through other processes and that are finding ways to do security things even when it's not required of them? How is the whip rate across your various teams? How much work and process are you maintaining for your DevOps and security? How are you measuring your security work? If they're working on policy, what do they do? Now I generally recommend when teams are on a sprint cycle, I generally recommend that the security teams go to a Kanban board, but they still maintain their review and retro in the sprint because they're gonna be representing into each of your teams the security champions in order to be integrated, but they also have their own work that they have to do. Very often they have requirements and compliance in policy revisions and those things all have to be measured with the security and the assistance and the help they're giving to the developers so that those metrics can be provided as well. So with all those factors, all those kind of ideas in place, a good sense of how things move forward, let's talk about a little bit of refocus. Let's talk about how we move ahead. How do you know what you know? That seems like an odd question, but when somebody gives you a number, when somebody gives you a process, how do you know? And the answer I usually get is, well, it's on the dashboard. How many dashboards is it? How many dashboards do I have to go through? How I'm using Splunk or Elastic or any other type of seam or process, how many dashboards do I have to look at before I get the answer? It doesn't show the metric that I want. So this is one of those advantages of coming from an intelligence background. So you use an aspect of critical thinking and use the same aspect that you'd break down your metrics with to break down your processes and ask yourself what are the right questions that you have to move forward? And the first one is how do you know? When something comes through and a security basis comes through, when you're talking about a security metric, how do you know that's the right metric? How do you know that vulnerability is the right metric? You see often people come in and they say, I have these metrics or I have these things. We want to know if that affects our tool. Well, you should know that. You should know what's going on in your system. One of the worst ones and the ones I hate is somebody comes through and says, you build something up and say, here's your solution. And they say, but that's not actually what it is. You say, well, this is what I got from the latest. The guys went out to the site survey and they did this and they gave me this report. That's what I'm using. I said, well, yeah, but that's not actually what we do it. They pull off the stack from the bottom door of the desk and it's dog-eared. It's got notes all over. And they say, this is how it actually works and where the actual parameters are. Well, how is I supposed to know that? If you don't share, you don't know. And if you don't know that the initial is true, you can't go what comes next because it's that next step that happens. If there's a risk, how do I remediate the risk? Or is the risk far enough back that I can just block it and put it behind a firewall and not worry about it? And that becomes the premises around those risks and whether or not those premises are true. And if that premise is true, that those risks are isolated behind a firewall. Do your conclusions follow the premise that I don't need to remediate it? I can just accept that risk in my metric and know that it's accepted because it's at the back of the system. Are there any other arguments for those premises to be true? Is my firewall too open? Does my firewall have ports or accesses that other people are getting to? Is the environment wide open? Do I think it's closed but have actually allowed a great deal of port access and individual access that my access control list does not have locked down? I need to be able to understand those things, understand the arguments around those premises. And then of course, as I'm looking at my processes, I need to make sure that I compare apples to apples and not oranges to elephants. I say apples to apples because you say they're the same, they're both a fruit, they both have red skin, they're both sweet. But when I start saying oranges and elephants, I say, well, they both have a tough skin. They're both juicy on the inside. Once you get through the tough skin, I'm still making equal comparisons but we know there's a lot more different between an orange and an elephant than there is between an apple and an apple. And the same thing happens with security and the same thing happens with metrics. That we take metrics that might compare an orange to an elephant rather than an apple to an apple. If we talk about vulnerability on a wider system or someone we know that's recently got hacked that's been made vulnerable to a wider malware basis and we try to compare it back to a smaller instance or a smaller vulnerability, we're not making the same comparisons. We're not talking about the same risks and we're not talking about the same vulnerabilities. So we have to understand those processes. So when we look at that, if we took just one, we say, well, we advanced the lead time to change, right? How long does it take to complete an action? Do we talk story points? Do we talk stories per epic? Do we talk work and process? We have to know what our common metric is for that. We have to know what our security additions are to that. How does security influence those points or epics or work in progress? What are the constraints we have around that? When we talk about it, who built it? Do we know how the task was assigned? Do we measure complexity associated with that task? When we measure complexity to a task for developer, do we assign security complexity? Have we talked about security complexity? Have we talked about how the individual manages that function? And what are the parameters we use to move forward? And that's not all. Even for just that one metric, right? Just looking at that one sample, we start talking about how do you compare different features? How do you priority scale between the different features and our lead time to change? How do you talk about work assigned and team completion? And as I said, what about our security? How does that one change affect the vulnerability? How do we measure how those changes affect the vulnerability in the overall system? GitLab uses a lot of good functions. We can measure that security and trace it over time, right? That good trace in that individual process that we talked about that gives us the feedback because now we trace it, now we know how it affects the vulnerability. So now we can do the experimentation. But they all have impacts to code delivery. They have impacts to our tech debt. They have impact to move forward. We need to have coverage for static dynamic and run time uses. And we need to make sure that our security scans are not taking up a significant portion of the process moving the code forward. It should not take longer to scan and reflect back on security than it does for the developer to develop it initially. Because if it does, then there's a break in our process. If we were to look at another one, Advanced or Change Failure, right? We've talked about Find the Why. Two great books on Find the Why. Start with Why by Simon Sinek and John Doar talking about Measure What Matters, Objectives, Knowledge, and Responsibilities, the OKRs. Compare the why. How does a failure compare to past failures, right? What makes it a failure? This is important for security of how many failures get out there and which failures are important. How does security work to find the early identification to fail early? To gather the group so they can learn quick, to get together. How do you talk to security when you do have a failure to move forward? Failure creates our opportunity for success and the same is true with security. And security failure shouldn't be a reason to fire the security person, especially if they didn't have any influence in it. But it should be a reason to get together and talk about how make the product more secure in the future. So overall, when you find the metrics, when you transform your security metrics, and I gave you some options for transforming and for finding the right solutions for that transformation, you have to find the right personal and professional relationships that favor the people over processes to be able to drive forward. Again, I really enjoy having security people on each of our development teams and having someone who has a responsibility just to the security team, but also has a responsibility to be the representative down that team. Some areas talk about having champions and having that champion who's responsible. And it's kind of the same thing, but having someone who very intimately knows what that team is dealing with and how they're working with the problems. Because that helps to assess how much security you need in the process. It helps us be able to make security a part of the conversation on a regular process. Our metrics help us show that balance over value. They help us deliver what we need when we adjust the metrics so they show security as well as other processes. Whether it's just showing the difference between the vulnerabilities or whether it's showing when our scans run or what types of scan have run, those are all important essentials of our security process. I hope today you learned a lot. I hope you'll take some things away. I think I hope you'll take away that you have to understand the value. You have to understand what security brings and how you want it to be part of that process. Having it become part of that process is about balancing your security with delivery and then of course about practice. You can't do anything without practice. One of my favorite speeches is Malcolm Gladwell, a favorite authors, talks about outliers and talks about the need to practice anything. Not just practice, but perfect practice. So I'll leave you with that. Hopefully this will give you the chance to get some practice on your own. If you have questions, feel free to reach out to me and we can talk about more specific security metrics if those are your questions. I hope you learned a lot and thanks again to GitLab for letting me present at GitLab Commit this year. Have a great day.