 So my name is Kailin. Thank you so much for sticking around today. I know this is the last talk, and I'm all that's between you and probably tasty food and beverage, so I really appreciate you all being here. My talk today is Sharing Security Secrets, How to Encourage Security Advocates. This is just a rough summary of what I'm going to go over. We'll start with the basics, the what, the who, the why, and then we'll dig deeper into the how. Specifically, I'm going to look at techniques that managers can do and techniques that individual contributors can do. We'll finish with a couple examples from Shopify as well as tag security and SIG security. So we'll start off with the easy bit about me. My name is Kailin. As I said, I've been at Shopify for about five years on a variety of teams, but currently I am on infrastructure security doing a super awesome placement on our Kubernetes Foundations team, which I could spend this whole talk talking about. So if you want to hear more about it, please hit me up later. This slide illustrates what I do when I am not doing computer programming things, which is hang out with my fur children for Skitten Luna, go running with my husband. And in the bottom corner, you can see our dream house that we're building, which takes up most of my fantasizing currently. So let's start off with what are security advocates, also known as security champions. I like to think of them as various as little birds or as the neighborhood watch. So these are folks that are spread across your company, not necessarily on a security team that are thinking about security. You can see the definition here, cyber security advocates attempt to reduce exposure to cyber attacks by promoting security best practices and encouraging security adoption. So you can imagine, not only are they influencing the projects that they're working on to make secure decisions, but they're also able to pull in security teams when they think that it would be reasonable, which is nice. So more importantly, who are slash should be security advocates? Spoiler, everyone, anyone, all of you, please be a security advocate, it's awesome. It's really important to note that they don't have to be cyber security professionals. Those are gonna be the natural choices. Obviously we're all here, we care about cyber security, that's something that we're already doing, but it's even better if you can find folks that are in other areas of expertise that can bring unique context that we don't necessarily have and then apply security mindset to that. We can't do it all, so hitting more on that is we can't know every single thing that's happening at our companies, particularly in a large company, but also in a small company, because if you're in a small company, likely your trust org is not that big. In fact, there might just be one person who's responsible for securing the company. So it's really important to have people looking out for you and wanting to pull you into things where they think that your input would be valuable and security advocates are a great way to do that. So why do we need security advocates? I've touched on it a little bit, but I'll go into some more of the details. We've all seen these headlines, hopefully not too many of us have had to deal with responding to these headlines by being at the companies, but security is part of our everyday experience. I think like most people who don't know anything about computer programming know about hackers, they know about compromises. This is what the media sees and this is where crime is happening. This is where more and more sensitive data, financials, health information is all going online and this is what we need to protect. So as security engineers, we are small fish in a big pond. That means we have less influence and sway in a larger organization. It's, I don't know a company that has a larger security team than general workforce because due to necessity, you need to have more engineers to build the product, but it means it's really tough for us to secure everything. Like I said, we can't be everywhere at once. So how as little fish do we make sure that that whole pond is secure? We chat up those big fish and we make sure that they come and they see us and they tell us and they talk to us and we make sure that people care about security and want to ship secure products. One example that highlights this really well is Conway's law. I'm not sure if you're familiar with it, but it states that an organization structure will be reflected in the software they build. So if you have one big flat structure, it's likely that your company's software will be kind of monolithic and one big blob. And if you have a really separated culture, then you're gonna end up with something that's more serviced and split. And so how can we look at that from a security perspective? If you have your entire org in this tiny little bit thinking about security, it's very likely that your software is going to have a tiny little bit of security coverage. But if we add security into the organization at different levels, each piece of software is likely to have a security concept or be thinking about security, which I think is exactly why a security advocate program is an awesome idea. In the cloud native security white paper, there was this quote which highlights exactly what I'm trying to get at in this talk, which is by integrating security as early as possible throughout the development lifecycle or even earlier with interactive developer training, security organizations can enable preventive security rather than reactive security. And so it doesn't have to just be security organizations, it can be within an organization. All the time our developer teams hit us at the point of release and they say, hey, can you just give us the check mark so that we can release this project or so we can move tiers. And all of a sudden we have to say, so sorry, but you need to fix this and this and this and this. And now we're impacting that team's goals and all that they're seeing the security team as is a burden and a block to them meeting their cycles and addressing the pressure they have from above. So now that we've talked a little bit about what security advocates are and what the goal is, we're gonna talk about how we can implement them. And I just wanna note that there are a whole bunch of really fantastic formal education programs for creating security advocates or champions and those programs are focused around educating and having a formal role. This talk aims for the stage before that. So educating and garnering interest so that people want to participate in programs like that and just having a general security culture at your company. So what can managers do? Disclaimer, I am not a manager nor have I ever been a manager but I've harvested some really good tips from the amazing managers that I work with primarily Ann Wallace. First off is be a champion yourself. We need champions at the top level of companies because if the company is trying to sell security as a goal and there aren't security advocates in senior leadership, you're going to have to justify that business cause and it's really hard to explain how security makes a company money. I mean, we don't, we save companies money and when security is going really well, you don't know what's happening because you're not ending up in the headlines. It's just happening seamlessly. So if you have folks in senior leadership positions, A, they have the knowledge to sell security as a business expense and B, or as a business argument and B, it gets built in to the company-wide goals. We're all in a really tough time right now for software. It's not just throw money at everything anymore. And so it's really important that teams are working toward business goals and we need educated folks to argue for security so that that can happen. And this just goes a little bit deeper into that. So ensuring that not only just at executive leadership but in management as well, that your OKRs and your team goals and your cycle goals have security as part of them. It sucks to be a security team and not feel like anyone is valuing the work that you're doing. Instead, you're looking at how can we increase users, increase income. And I don't have the skills to tell people why we really, really need to invest in security. So having senior leadership and management do that is very, very important. Next is the more fun part, which is support and sponsor events like hackathons or learn to hack. Any kind of interactive events that you can sponsor or host will help spread security awareness in a really fun and approachable way. A few ways that we do this at Shopify are a learn to hack workshop. So this exists in our repository and anyone can do it at any time. It's super, super fun. We've got a really vulnerable webpage and we walk you through finding different exploits. And during any kind of security awareness periods of time, we will run in-person workshops where we'll actually have a whole bunch of security nerds help you do this and be really excited that you're taking the time to learn. And then a big thing, every October, we do cybersecurity awareness month. I'm an infrastructure security engineer. I do Kubernetes. That's what I like to talk about, but at this point in time, our whole trust org comes together to share all sorts of security information and we go well beyond, don't click on this phishing email. We go into how their services are protected and how they can involve security in their day-to-day workflows. So what can ICs do? This is much more my area of expertise and this is something that I've been doing a lot of since I started on a security team at Shopify. First off, be a security advocate yourself. Do what I'm doing, stand up, tell people about security, why it's awesome, why we should educate people, just do it, just talk about it, get out there. Next is consult, don't dictate. So this, I mentioned it a little bit earlier, but I've been a part of a few projects that have involved going out to various teams and having them, actually much like the talk that just happened before this, having them do a patch or having them deploy some kind of security measures that we have decided are mandatory and often by the time I get in the room with them, it's not that they don't wanna secure their product, it's not that they disagree with me, but I'm actively making it harder for them to achieve their goals. They have to pull people off of other work to do the security work and they don't want to because no one on the team is invested, their leadership isn't invested and I'm just this person telling them they have to do a thing. So we all know when we pair a program, you totally change the way that you're approaching a problem, all of a sudden you're writing the best code you ever have and you can't type it all, of course, because it's terribly stressful, but when you work together, you're going to both bring your context to make something better and you're also going to do good work. So I think trying to create a security advocate program means that in the design phase, you're helping people learn about what decisions they can make to make the product more secure and then when it comes to that, oh, we have to do the security review, it's just a check mark because they follow the steps. Educate, so again, I'm doing that right now and everyone, I've done it at Shopify in a variety of ways. I love this comic, I wish I could read it out, it's very funny. But basically, I remember when I joined security, everyone was like, oh yeah, I'm like Alice and Bob and Eve, I was like, who are these people and why are they so popular in security? But no one explained it to me because everyone took it for granted that you know who Alice and Bob are, you're not in a security room and you don't know who Alice and Bob are. People don't know and people don't know a whole bunch of the things that we think are really basic. So I think that it's important to educate and educate really openly and welcomely, start with the basics, don't ever hesitate to motivate people to ask questions, explain everything as much as you can so that people aren't feeling intimidated by security. So when I've crafted a few educational programs at Shopify and I'll talk about some of the focus points I have when I'm creating content. So the first one is to talk about security incidents both internal and external. We have done at Shopify a few sessions called Anatomy of a Hack and we take this opportunity so a recent one was after the Uber hack, we saw a whole bunch of people in our Slack channels being like, oh man, this is crazy, this happened at Uber, look at this Hacker News article, oh my goodness, security, it's awesome. Are we doing security at all? And then our security team was like, yes, in fact, we do a lot of security and we'd love to tell you about it. And so we hosted these sessions myself and a few of my colleagues were on a panel and we started by explaining the incident, again, demystifying it, going to all the details of what happened, why it happened, how Uber responded, and then we did our best to explain how a similar attack might happen at Shopify, what preventative measures we have, what would we do at various stages of the attack and people were really, really, really interested in it. We finished with an AMA where folks were able to ask a whole bunch of questions about the way that we do security at Shopify and it just got people engaged in talking about it. The next one is how to threat model. Recently, our team has grown and we've ended up with a whole bunch of more traditional background security folks that introduced a lot of threat modeling. I had never done a threat model before and now I've done countless and I think that this is now something I do in every security education program I run at Shopify because it allows people immediately to apply the security principles to their own work. Not only that, but it's a way of thinking that is already ingrained in developers. Look at your system, find points of failure. That's something we all do every day when we're writing code and dealing with software and the threat model is just an awesome way to get people doing things right away. Next is I'm really hammering this nail in as someone who joined software previously being a farmer, cover the basics, don't assume any knowledge. Definitely if you're doing your third or fourth educational talk, you don't wanna start it off with like, okay, this is what security means. So create recorded accessible content that folks can go and look at before attending your sessions. Send it in the email, put it in the event, have it as part of your onboarding so people can go and know the security basics and not feel intimidated when they're attending and anatomy of a hack and getting into some of the deeper concepts. Explain the security org. So again, I said, I have worked in a whole bunch of different places at Shopify and I used to just work on our core product doing backend rails and I didn't know that security existed. I didn't know that we had a security org. I certainly didn't know what they did and I find that a lot when I talk to people like, oh wow, security, what do you do over there? How many people are you? Do you guys even write code? So I think explaining the security org is just a great way to have people know just, I mean, not always Shopify is really lucky but how much access they have to security experts. And the last one is to go over security tools. So at Shopify, we have this awesome internal just in time access control that allows people to get permits, to access resources. It's really great. It works in a variety of ways and it's one of the tools that we own that pretty much anyone at Shopify can make use of and so every time I talk about security, I tell people about this tool. It's called Cloudoo. It's awesome. This is how you could use it on your team to control access to something that might be a sensitive resource and I think we all have that. It's a great opportunity to say, hey, you know that little thing that pops up on GitHub? That's actually a tool we created that's checking for X, Y and Z. Okay, now the examples part. So recently, again, disclaimer, I wasn't directly involved in this project. Some really amazing people on my team were but when I proposed this talk, I immediately thought of this project. The goal, we did our annual planning and decided that we just wanted to be involved in every project. We wanted infrastructure security to be everywhere. We can't do that. There's not that many of us and there are a lot of different projects and services being created all the time at Shopify. So the way that we decided to handle it was to try and move up our involvement in the development or project process. So at Shopify, we call it Get Shit Done but it's just our regular project workflow. It starts with a proposal, moves to a prototype. We trash the prototype, hopefully, and then build the product and release the product. And previously, we were called in only at the release stage to do a security review or again, if they wanted to move tiers, we would get called in to do a security review. Now with this new project, we're involved as early as the prototype because it's automated and built into GitHub so it's possible that they're already getting feedback on decisions that they're making in their earliest, probably sloppiest code which hopefully influences the decisions they made in the build but at the very least, we're being pulled in to the build before they've had to request the security review. So to me, that's really, really huge and awesome. Some of the goals of this project were a shorter lead time for security reviews. We were pretty good about getting to security reviews. We have a team meeting every week. We would triage any new reviews and usually someone would tackle them within the week but after they would do the review, it would result in some actionable items and then we would have to re-review. So I think the average was about two weeks which if you're a service owner, trying to release sucks. So we wanted to reduce that as much as possible and then tied into that as we wanted fewer tier move blocking change requests so we wanted to try and have all of those big blockers sorted out right away. As soon as they make a decision in their code that would be a blocker, something should alert them that they've made a dangerous decision. 70% faster team onboarding to review process. I think we say three to six months at the moment it would take someone to feel comfortable running a security review on their own. So we're hoping that really the tool is doing almost everything and that people could join and do it within a month or two. And lastly is infinite, infinitely better relationships with service owners. So much of the rejection of security comes from developer toil and not adequately considering a developer workflow. I think Shane touched on this in his keynote but when we put in blockers for folks in something that they do every day not only are every time they hit that blocker they're gonna think, security, they suck. But they're also going to think how can I get around this blocker? Is there something I can do? They're gonna try and look for ways to get off that paved road and that is not the kind of behavior and thoughts we want around security. Some wins already, so this project is still in process and I haven't really explained what they did at all which I should do. Basically it used to be a really well documented list and you would go through the checklist and you would do a bunch of things and make sure that all of our security baselines were being met. Now all of those checks are automated and it produces a report and that report is sent directly to the service owners and any vulnerability that is possible to surface in that report is extensively documented in our docs so that they can go and start to fix those issues right away. So the first one is that now security concerns are surfaced in the GitHub UI. It's promoting service every time they make, promoting awareness every time they make a change to their code. They're seeing that directly in the CI which is awesome and it's stopping vulnerabilities from making it into production which is extra awesome and then the next one is what I just mentioned. There's a report generated for service owners so the review results in that list of vulnerabilities and they can address them right away and still wait instead of waiting for us to come to them and tell them how to address them. And the last one is that issues are open from scan results so this just fits a little bit better into a standard developer workflow in my opinion and means that instead of a comment on the review they're seeing the issues in the same way that they're triaging all of the work for their project. And so the next one is security self assessments which have also been talked about a few times today and I think are also a really great way to build security advocacy. Both tag security and recently Kubernetes SIG security have the self assessment program and this line is taken from tag security. The self assessment is the initial document for projects to begin thinking about the security of a project determining gaps in their security and preparing any security documentation for their users. It's a little bit less formed so far in SIG security but basically we're doing it on a volunteer basis any sub project of Kubernetes that wants to do a self assessment can come to us. Hopefully they have their data model but if they don't we'll work with them to create or sorry their data flow diagram and from that we'll create a threat model and we'll talk them through the various vulnerabilities that exist and then work with them to create actionable issues that again will exist on the repo and hopefully get folks to go and pick up those issues and work on those vulnerabilities and similar to tag security it results in a really nice documented security summary for those projects and SIG security is planning on running a workshop to do threat models at Kupkan and eventually potentially as part of the contributor summit but if there's interest maybe also as part of the mainstream at Kupkan we'd like to do like walking sub projects through this process in a way that people could observe and participate. So the summary my takeaways for this talk how to encourage and build a security advocate culture. So first off attend this talk start looking at how other companies do it just think about what's out there start advocating yourself tell people about the cool things you're doing really if you are a manager start advocating for security and get other managers to understand why security is important. If you're not a manager then find some managers get them to support you let them know how important this is. Don't gate keep. So that again is me really dwelling on like don't assume anyone has any knowledge don't make security mystic and secretive pull everyone in it's awesome and they should want to help us make security education relevant for your audience create opportunities for hands-on learning that's talking about those like learn to hack workshops capture the flags any of that stop collaborate and listen talk to teams as soon as you're gonna work on a security product find out who it involves we have customers as security teams that's everyone that we work with and all developers and they shouldn't be finding out about the tools we're creating to secure them after we've already developed them it's the exact same problem in reverse we need to know exactly how the tooling we're creating is going to impact the people that are going to use it or have it built into their workflow and then get security goals at the top level okay ours company mission guiding themes I want to see the word security when I'm looking at what my CEO is telling the entire company we need to focus on which we actually did this year and I'm very pumped those the first time since I've been around but it's been said like we need to be secure. Thank you so much for coming you could find me on Twitter on GitHub in the Kubernetes Slack as Kailin codes on Instagram if that's your jam and you want to see lots of dog pictures, food. Feedback please this is my first ever talk outside of Shopify I'm sure you can't tell how nervous I am so please scan the QR code let me know how I could do better next time and what I should keep doing. And finally Q and E, the E for experiences if you have something that you do at your company that has been really effective I want to know about it you can tell me now or you can come tell me after or you can contact me in any of the ways I listed below. That's it. Does anyone have any questions? Yeah go ahead. Yeah so the question was how do we get involved in the prototype and design phase in a distributed company because that can be really hard. There's one advantage is that Shopify has a central area where we do our project proposals so those are accessible to anyone and we can see those so we can use the API from that to get notifications. But more importantly having come from being on a whole bunch of teams I was able to build up a really big network of folks at Shopify and my special interest is network security so I am often reaching out to the folks that I know from my time working on networking teams saying hey what are you doing? Are you doing this? Can I see your roadmap per chance? I would love to see what you're working on and that behavior doesn't just benefit me like it recently has resulted in a cross team project that's really valuable to both teams and now I don't need to reach out to that person they're reaching out to me so creating that culture again of sharing and talking about it. Make it friends. Anyone else? Yeah that's a fair question so the question was how do you justify security education basically as a business goal when there's so many more important goals within security alone as well as the rest of the company's goals? I haven't done too much of this like Shopify I've been really lucky they've been really receptive there are a whole bunch particularly because we're distributed there are a whole bunch of opportunities to educate and I like doing that so I've taken that opportunity but one thing that I have not tried yet but I was thinking would be really valuable to get the momentum is create a special interest group and have again have those people that are not on security so you're not pulling security folks away from security projects full time but you have these sessions these special interest groups maybe it's just an hour where you meet and you chat about what projects are you working on do you have security concerns? I would assume so because you showed up to a security special interest group like how can we give you the tools to move forward with that project and then hopefully you're able to approach education not from the perspective of I'm a security nerd I love security I want everyone else to be a security nerd and love security but you're approaching it from I've talked to so and so and so and so and they're working on these projects and they don't know how to address the security concerns and in the past we've ended up at release date delaying their project by two weeks because of a security review having to make fundamental changes that would have been like would have not meaningfully added to the project time had they been earlier and again that's like I think we all who work in security know about the hard sell of like it's not going to be a direct easy correlation but hopefully if your team is already if your company is already investing in security they'll follow that that thought process but I so far have not had to convince a pointy-haired boss to let me educate people about security yeah so the question was do we have any metrics for the impact of these kinds of programs so that after doing them we can argue for continuing to do them summarizing questions is very hard no is the first thing I don't think we have any clear metrics what I can think of offhand is one of the educational formats we have at Shopify are called coffee and tea sessions so they are anyone can volunteer to do them and you basically have an hour to talk about something that you're very interested in and usually they're like involve a technical demo or something and with that you can see like how many people signed up how many people actually attended and then we also have feedback for those sessions so that could be one way is like if 70 people at the company want to take an hour of their day to listen to someone talk about security to me that's indicating that there's like there's a need for that and second I think that I'm not on the security review project but I do think that they're actively measuring the difference in time from when a security review is started to when the project is able to be released and that's going to result in metrics that one's a little bit a little bit less education focused because obviously it's automated tooling like the education is part of them just doing the security work but to me it's the same thing is like we could take even more time off of that if they don't have to read a list of security concerns and they just know how to secure their services. All right, well thank you so much and I'm so glad to have talked to a whole bunch of you here this week this conference was awesome enjoy the rest of your night.