 We've finally reached the end of GitLab Commit and we've saved one of the most important topics to close us out here on the main stage, security. Regardless of your role on your software team, security is a concern that deeply impacts us all. Yesterday, we heard from both our CEO, Sid, and Gardner analyst, Manju Bhatt, how one of the major benefits of the DevOps platform is the ability to secure your end-to-end software supply chain, while also delivering more secure software. We heard from an array of security experts on our DevSec op stage, and in many of our talks on all of our stages, security has played an important part. Our final speaker, Jonathan Hunt, is VP of Information Security here at GitLab. Jonathan is responsible for making sure GitLab works securely, and as an InfoSec veteran, he's here to tell us what's wrong with software security today and what can we do about it. Hello, everyone. It's a pleasure to be here. I am Jonathan Hunt, VP of Security at GitLab. A little bit about me before we get started. I've been in this security space for over 15 years. I've built or led information security programs at five companies and four different verticals. I've worked in three Fortune 500 companies, multiple startups, and a few companies in between. I have a master's degree in information systems, and I'm pursuing a second master's degree from Harvard University. I have all those security certifications spoken at dozens of events, written articles, blogs, do all the security things, though all of this, I've come to a single irreviewable truth when it comes to software security, and that is flawed. I've seen evolutions of security programs and solutions from archaic to the modern, from the risk-averse to the risk-tolerant, even from the boring to the groundbreaking, yet we still haven't solved it. In fact, entire businesses are built on it. Multi-billion-dollar companies are betting on it. And I dare to say that some of you may even have a job because of it. So I ask, what's wrong with security? We have tools, we have education, we have security teams and apps like engineers, yet every day companies get breached. This week I got another email, had to go change my passwords, doesn't even shock me anymore, I'm prepared for it, I've even got tools for it. I talked to security leaders at several companies, small and large, and they've been breached and guess what? They were PCI and SOC and ISO certified, they had all the scans, all the processes and did all the right things. So again, I ask you, what are we doing wrong? I'm not here to tell you the same old tired story you've been hearing for years. I'm going to take a different approach. Are any of you familiar with that analogy therapists use where people talk about issues and the therapist explains, oh, those are the branches and we need to get to the root of the problem? Far too long have we complained about all the symptoms without getting to the root of the problem. Today, I'm your therapist and I'm going to share what I believe is that the root of our software security problem. Now I'm just going to bring it today. We have a lot to talk about, a lot of different ideas and competing thoughts. Quick disclaimer out of the gate, I am not going to solve our software security problem in the next 20 to 30 minutes. I think anyone who could should probably go cure cancer or solve world hunger. But I am here to provoke thought. Use the chat. This can be a conversation. This isn't your typical security talk. You've all heard about it. Here in the US in May, Biden signed a presidential executive order on cybersecurity. This is designed to improve detection of cybersecurity incidents in federal networks and malicious activity across the entire supply chain, including endpoints and dependencies. This order carries several impacting requirements for not only the public sector, but the private sector as well, such as sharing of threat intel, securing the software supply chain, requirements for categorizing critical software and guidelines for testing of code. As the security industry responds, these changes will have global implications. NIST, an organization who establishes international security standards and is widely used in the US government space, immediately made a call for papers on five overarching topics covered in the executive order with the purpose of establishing the next iteration of their framework. Leading computer scientist Paul Black reached out to me asking if I would submit one or more position papers. And admittedly, I was concerned with a short turnaround time. I think we had less than a week. But as a team, we came through and submitted papers on four of the five topics. And as it turned out, we impressed. A day later, I got this email from Paul asking me to present. And of course, I responded with the obvious. That depends how much time do I get? I mean, to be fair, I don't think Paul knew what he was getting himself into asking me to talk about improving security and software development. Some of you all know there's a few topics I have quite a lot to say on. Some would say I could be a bit long-winded. I still don't think some of y'all are getting it. Sometimes you can't shut me up. So I'm on this panel with several notable leaders in the security space. And again, our topic, how to improve security in the development life cycle. Can already guess what the recurring theme for the day was. Necessary tooling to catch vulnerabilities and to catch them early. Shifting left, of course. And it dawned on me, why are we still asking people to use tools? Maybe we should just be telling people to use tools. After all, it's my personal data, passwords and accounts being stolen when companies get hacked. Which brings me to my first problem with software security. Security tools are not a standard requirement by security frameworks. Not a NIST, not a PCI, not an ISO, not even required by our customer contracts. It's all conceptual, but maybe that's not enough. We've tried flexibility and creativity for years, for decades. Let me explain. I worked for a payment processor that processed over $100 million of credit card transactions annually. We were under continuous auditing for PCI, by the auditors, by the clearinghouses, banks, and even other processes we partnered with. Nine months a year, I had an auditor to answer to. And PCI has one requirement related to secure software development. In fact, it's a subset of that one requirement that simply says, developers should develop secure code. That's it. Well, that was easily met by asking our developers to read through an OOS PDF once a year. No requirements for tools, no STAST, no DAST, no nothing. How about ISO? That's an international standard. Surely they're more specific. Well, slightly. As it says, we must have a policy that explains how the organizational engineers secure software in a secure development area. Still no requirements. Just explain how we do it and convince the auditors it's secure. How about SOC? SOC probably has something, right? So does it require specific tooling? No, SOC actually lets you write your own controls. So I think you can answer that for yourself. Now, I'm not saying nobody uses tools. I want to be clear. If it's just a suggestion, then it's more like a hobby. It competes for your time, and we individually make choices based on what we want to do or what we think benefits us the most. Secure software development becomes subjective on a personal level. And to some, that might be security of our code. But to others, it might just be working on their next project. Therefore, the first step is to be specific and require these tools by audit and control frameworks. You might remember recently, ransomware hit a gas pipeline here in the US, and part of the country experienced a gas shortage. I was personally affected. Many gas stations were out of gas, but those that weren't had it limited to purchases of $10. At over $3 a gallon, that was barely three gallons of gas. This isn't much, right? Not going to fill the tank, it's not going to last you the week. So why was it that everyone was stopping to get in line for $10 of gas? Why did I wait in line for an hour for three gallons? It's because even a little bit helped. Even a little got you a few more miles down the road. Now three gallons got my kids to school another day. Is there a tool or two that solves all of our problems? Of course not. But it helps a little. It gets us a little further down the road. So the solution, start somewhere. Start with something. One more vulnerability found and patched. One less way your company can be compromised. The second problem we have today in software security is, well, those that are doing it aren't really doing it. And I hear you. I have people all the time saying, oh, we have all the tools. We're good. What else you got? I was on a virtual panel about supply chain. And someone in the chat said, we've all been told this before. Is there anything new? I was just thinking, we don't need to change the story because what we've been told, we still aren't doing. So what I'm really saying is, the tools are there. Just saying they're not really there. My friend and speaker, Jonathan Evans, would say it's more like the abominable snowman. Footprints everywhere, but nowhere to be found. We're still thinking security is about the size of our team and the sort of vacations we have. I'm here to tell you that security is about what we are and aren't vulnerable to and the risks that creates. I just mentioned the need to have security tools. And now I'm saying it's more than just having them, but we need to use them right. I've talked to company after company that have tools, but they aren't configured right. They have teams, but they aren't managing the tools. They even have results, but they're doing nothing with them. I understand we're passing our audits. So why are we still getting breached? Sometimes I feel like it's all a facade, like this ocean oasis in the desert. We can show that we have teams and tools and processes and policies and guess what? The auditors are happy, customers are happy, but the inevitable can and will happen. We need to do better. I've been thinking to myself lately, whose fault is it? Security teams, after all, we're talking security. Development, they create the code. Organizational leaders, we set the priorities. Maybe it's auditors, they're not strict enough. How about the tools? They're not good enough. Maybe it's some of these, or maybe it's none of these. After all, answering this question is futile. We need to focus on solutions. To security teams, we need to make sure that our tools are configured correctly, scope correctly, and that we are properly managing the results. Developers, we need to make sure we are owning the solution, prioritizing issues, and fixing the vulnerabilities we ourselves created. And organizational leaders, we need to assess risk and prioritize security accordingly in the organizations that we control. If we do these things, besides the obvious reduction of vulnerabilities, we will eliminate the tension between our departments. We will become partners and not competitors. We will become unified in our approach and we'll conquer the world. All right, I'm just getting warmed up. You all okay? So this stuff excites me. Security is my passion. It's what you might call the fire in my belly. Maybe what it really is, is I don't wanna wait in line an hour for three gallons of gas. Number three, I might touch on some nerves on this one. Our third software security problem is alignment. Now, if this was in person, this is right about the time the room gets quiet. Our alignment is off. Ever been in a car when your alignment's off? I don't know a lot about cars. Not what you would call your neighborhood mechanic. But what I do know is, you know when you have an alignment problem. When the wheels aren't aligned, you have one wheel trying to go in one direction and another one trying to go in a completely different direction. When you have perfect alignment, your car does what it's designed to do. Drive straight in the direction you want it to go to smooth ride. You're happy with the outcome, right? After all, you arrive at your destination. How do you know you don't have alignment in software development? It's because it's not going in the direction you want it to go. Our prioritizations are going in different directions. Our communication and commitments aren't smooth. You aren't happy with the outcomes in getting to your destination. It's stressful and arduous if you ever get there. So what are the symptoms of being misaligned? Miss our goals and objectives. We have aging vulnerabilities. We increase our risk of compromise or worse, we have to breach. But I'm here to tell you that it's more than that. We become frustrated as a workforce. You see alignment set expectations. When expectations aren't met, we become disappointed and people check out. How many of us have seen team members leave out of frustration? The reasons may be, this work's not getting done or it's not getting done fast enough. These are symptoms. At the root of it is because people leave because expectations aren't met. Alignment does more than fix problems. It fixes people. We all have these categories of vulnerabilities, right? You know, the criticals, the highs, the mediums and lows. Remember back in the day when the associated resolution times were like three business days for criticals and 30, 60, 90 days for those high, mediums and lows. And we just kind of decided that, lows, it'll just be best effort. Well, you know, maybe we just don't need to fix those and we've been growing complacent with mediums. Rather than fix vulnerabilities, we've extended SLAs to fix our metrics to make us feel better. As an industry, we've even started accepting mediums. I'm definitely not so radical as to say that we need to fix and prioritize every vulnerability immediately. That would be ridiculous. But how would we become so tolerant of security weaknesses? Because there's so many more vulnerabilities or there aren't enough people, we don't have enough time, we have more important things to do. These are branches, branches on a tree. Because we're growing more and more comfortable with risk and this is inherently dangerous. We hear this word and we think, oh, a little more risk is okay. Just a little more risk is fine. We haven't been breached yet. It seems so unlikely. Then the worst can and will happen. Tomorrow you might wake up and find your company in a media and I don't mean in a good way. So how did we come to this? Well, we can only speculate. Or in my case, make bold statements. We're here because it's literally built into our DNA and I'm talking our job DNA. Our job descriptions tell us there are higher priorities. Our roles do not lend themselves to perfect alignment and based on expectation set. By striving to fulfill expectations that do not emphasize collaboration and alignment, we move toward a more fragmented organization. The solution, this is for our leaders listening. You need to align, set expectations and meet our commitments. This will improve your security and will heal your organization. Finally, the most important of all, for 1999, you can get the most impactful problem in software security today. No, no, no, no, okay, I'm teasing, I'm teasing. You gotta have some fun with this, right? See, I'm stressing you all out. I can feel it, I mean, we're virtual, we're on video. I can feel it, even I'm getting tense. Okay, now that I've got your attention, I feel like you're ready for number four. The most important problem we have in the security of software development is us. It's you, it's me. Why is the diet pill industry a multi-billion dollar industry? It's because we don't eat right. We throw money at the problem rather than doing what we could be doing for free. Why do we have large security budgets in the need for hundreds of tools? Because we don't learn to code right and it's easier to throw tools at it than do what we could be doing for free. I'm not saying we don't need tools at all. After all, that would be an inherent contradiction to this whole talk. I'm saying we've come to rely on tools to solve our problems rather than as a safeguard and a sanity check. To put it more clearly, I'm saying tools don't fix people, people fix people. Tools are not an equivalent substitute. It's intended to be more like insurance. The thing about insurance is this. We all have car insurance, yet we don't drive recklessly because we have it. Why? Because there's more at stake. The cost is too great. Should the worst happen? But if a mistake should happen, an error in judgment, we steer poorly and something goes wrong with a car, we have insurance to help restore us. Tools should be just that. We should code with care and architect would do diligence. Our reviewers and app sec team should inspect carefully and if we make a mistake, if something goes wrong, we have tools to help restore us. But even that comes with a cost because tools are imperfect. They are not the stopgap to all of our security problems. There are problems with them, even assuming we have configured them perfectly and scoped everything right. They still do not identify chain exploits well. They don't know what they don't know. They're built off logics and logic and rules. They're plagued with false positive rates and because it doesn't understand mitigating controls and the way business logic works, they don't simulate privileged access control well. And they rarely have third-party access or rights illuminating supply chain concerns. So would it just be easier to not produce vulnerabilities in the first place than rely on potentially flawed technology? There's a lesson here for all of us. Security teams, we are an organization's counterpart. We work with developers and engineers and across the entire organization. We are the subject matter experts on vulnerability and exploits. We can do a better job working with people in a creative way to educate and improve secure coding practices. We can do a better job guiding and performing our own code reviews. Developers, there are an immense amount of education and training opportunities available to improve the security of code. Many of you have security or app set teams willing and ready to help and consult. You'll find that you will deliver code faster. You and your team will have much less tech debt. And you're making your company and services more secure. And I offer this in return. Use this as leverage to fight for your raise and your promotions. This isn't to say that we aren't doing some things right. There's far too many to list but here are a few areas I think security is gaining in dramatic fashion. And this includes all of us. This is engineering and developers and security all contributing to a more secure industry and more secure space for all of us. DevSecOps is becoming a reality. We're beginning to shift left leading to faster delivery which is having positive impact on the security of our code. The organization as a whole is thinking about and becoming more involved with security. This is driving organizations towards a more secure state and reducing risk overall which promotes alignment because we are beginning to experience and feel the benefits. Information sharing has improved not only across our own organizations but our industry. Have you seen Reddit lately? What once started off as this plain social media forum has become a pillar for information sharing across multitudes of topics including cybersecurity. You wanna know the latest security breach? Visit Reddit. And we're beginning to buy into crowdsourced security like brownie programs, open source software are greatly influencing the security of software and frameworks we use every day. Now, more can be said about this topic but that's for another talk. Needless to say our vision and future should consider this topic alone. I told you the four major problems in today's software space. Security tools are not being required by development frameworks. We're using tools but we're not really using tools. Talked about alignment that involves product and engineering and security and infrastructure and the organization as a whole. Not saying I talked about people and the need for education, the need to do better, the need to stay focused. Solving these are not easy. If it was, they'd be solved by now. Everyone listening today can impact one or more of these problems. We all play a role in solving and shaping the software security space. If we each do our part, we will not only protect our company, our customers and ourselves but we will build faster, release sooner, perform better, all as an organization. Within this workshop, we had the opportunity for closing remarks. I'm going to leave you with the same and start contrast to other comments. I in short summary said, look, we keep throwing tools at the problem but until we change behavior, we won't solve anything. If you want to further this conversation, reach out to me on LinkedIn. Thank you.