 Perfect. All right, thank you everyone for joining this morning. I have today with me, my dear friend and security leader, Nick from SNAP. He heads the detection engineering at SNAP. Nick, thank you so much for joining me this morning. Thanks for having me so much, Sandeep. It's such a pleasure. Absolutely. So everyone, you know, this is unlike, you know, a bunch of previous webinars we have done. We will not be talking specifically about any particular product or any particular feature. Rather, the goal of this, you know, session essentially is to talk to thought leaders like Nick, see how cutting edge security teams like the one in SNAP or the ones in Google or Amazon, essentially use open source software at scale to keep their security program, keep their security apparatus consistently cutting edge. So that would be the topic today. The way we're going to go about this, Nick, essentially is, of course, we have, I have a dozen, you know, almost a dozen questions for you starting, you know, talking about the evolution of cloud security, talking about how open source fits the bill. You know, what sort of any initiatives that you might have at SNAP that we could, you know, take as a use case and talk about it. And of course, a bunch of inputs along the way for vendors who have open source first, you know, good market strategy, as well as, you know, any other compliance with that, that would be the format. Well, we'll keep it part structured, part free flowing. So please feel free to ask any questions. Anytime you want to take the questions towards the end. And with that, I think we'll just go ahead and get started on this one. So by default, all the attendees are muted. This webinar, of course, is being recorded and you'll have the slides as well as, you know, the video shared with you as we go forward. Keep asking the questions anytime you want to, you know, ask those questions absolutely fine. I will try to keep looking at the, you know, the questions and, you know, if I find any question that is sort of super relevant. At that point in time, I'll bring it up right now. Otherwise, we'll take a couple of questions, you know, towards the end of the webinar. So with that, to introduce myself, of course, I'm Sandeep Kofan, I'm the receiver defense. I have Nick with me, security and utility leader at SNAP. Nick, once again, welcome to the session. Yeah, thank you so much Sandeep. And of course, Nick, you and me go back a long way, you know, start, you know, meeting first on the bylines of one of the conferences and we share this common passion for open source. But for the audience, you know, that has tuned in today, do you want to go ahead, you know, sort of talk about your journey. How you, you know, what is your primary role and responsibility at SNAP, what you've been doing prior to that. We would love to hear your story. Start with that, please. Yeah, absolutely Sandeep. Thank you again for having me. It's such a pleasure to be here and share some thoughts with the audience. I recognize some of my friends and future employees that actually joined this call. So we have an individual joining us soon at SNAP that's actually on this webinar to listen in, which is really, which is really great to see. Thank you for joining Karen. And also thank you for joining Jeff. Look, it's been a while journey. I'm an immigrant kid who came here at five years old with two bags and a dream my family landed in JFK some 20 years ago, more than that actually like 30 years ago. And we, we've just been trying to build our life here and I found security pretty young age and at about age 16 I started, I started hacking web proxies at school because I wanted to get access to the open internet to download music. I didn't want to do this. But at the time the internet was really slow at home we only had America online dial up we would get the CDs and because we were poor, we could only get like 30 hours a month from the CD and then we had to get to the next CD. But the school had proper internet it was T1 it was 1.54 and BPS, which is a lot faster. But there was a problem. I couldn't get past the web proxy to use Kazan Napster download music. I figured out like hey there's this little host file you can edit. And if you edit the host file it changes the route that that machine takes to bypass the web proxy and then I can get to the open internet. And I did, but that was my realization about age 16 I realized that this six the security industry is my calling and since then I've been at it for 20 years and in the best like five years or so something changed in my career I realized I've learned a lot and I've harvested a lot of knowledge. And now it's time to give back that knowledge. And part of giving back that knowledge is sharing some of the cool stuff I've learned along the way with others. And that inspired me to not only do more things that snap and be more influential but also have a more influential effect on the community to start paying it forward during COVID during lockdowns. We had more time on our hands I decided to create a class on cloud native cloud security technologies fully focused on open source. This class has now been consumed by over 2000 students 2000 people I've never met have taken a class that I created I hope I've created some value for them. Some of them have reached out to me and thank me. Many haven't but that's okay. You know as as I was writing this class I realized that there's not a lot of good books on cloud native open source security. So every book that was even like close to that topic I had a stack of seven or eight books, and they're real and that these books are really great and I could probably write a better book. So after the class shift in May of 21, myself and a couple of folks at snap we, we proposed a book proposal to a publisher we had a relationship with through Michael Zalewski who used to be our VP of security he's kind of an influential guy in the industry. We had some introductions and we got a book deal with no starch, which builds themselves as entertainment for the geeks. So, we're working on the book we're about halfway down it'll ship in September of this year. Hopefully maybe we could do a book signing with with defense maybe at one of the conferences this summer. But yeah that's a little bit of background and beyond that right now I look after corporate. I look after corporate security at snap. I have interest in a lot of other areas that I've talked about with your teaching and my personal research so yeah that's a little bit of background. Absolutely love, love that make and you know what love to host you at our book at RSA and maybe even a black hat, you know, the book comes out by the back by then. And you know what I think one of the common themes that both of us have sort of interacted on is, you know, security is a digital public good. Everybody has a big it's a basic right everybody needs it. It's like GPS it's like raw infrastructure and security is essentially that it's a public good. Everybody has to have an access and I think that's where you know be connected on the open source, essentially the passion for open source and you know, really giving it back to the community so thank you so much for that. Now let's just, you know, try and you know look at open source insecurity and how leaders such as yourself companies that have snapped essentially look at this from a standard point of view really right. Yeah. What level open source security solutions are considered by your teams be the detection engineering team be the the security engineering team or DevSecOps or you know you name it essentially. One of the options that comes to you know the team's mind or you essentially always start with open source solution is how do you really think about it. We almost always start with open source and the reason from there. Now there are some things they wouldn't build like they wouldn't build an anti virus team and that's really involved and other people have done it really well. We probably wouldn't build some other commodity tools but would we build the service around device trust to assure that every single corporate device on the network actually belongs to the company and when you authenticate to a company endpoint. We can add in a payload in the authentication header to assure that that endpoint is actually the company's endpoint it's not a real good point. That's something we build. We call it device trust. So it's foundational right we could have when bought something similar sure there's beyond corp style zero trust products in the industry but that's not how we approach it let me let me tell you a little bit more about why. Let me zoom back out to the kind of cloud cloud angle here to me it's really ironic that the cloud and the internet are built on open source technologies broadly. However most security tools are closed source commercial products. This has a bunch of downsides that I want to talk about. I think it holds back security engineering as really being seen as a true engineering discipline and a lot of companies. At high technical bar companies this isn't the case but in in lower technical bar companies security is seen as a compliance function as the policy people people who say no. But what why is that the case at like at the foundation security is an engineering problem. Right. And so at you know at internet scale the way that we operate we we need to build to support the the ecosystem that part of if I can convince the engineering team that a security control needs to be implemented and built in a certain way and we can't have like low low observability of how we do that especially in the cloud context. We're not going to be able to get it across the line and close source commercial products. You're basically living to the vendor documentation and mental models. Let me give you an example snap rounds of service measure over 1000 communities clusters. In order to add runtime monitoring like Falco we had to convince the mesh platform engineering team that's kind of the central team that builds the service mesh that we instantly understand the fault modes for Falco. And we have a way to kill the sidecar that Falco runs in. If something goes wrong. I remember this meeting vividly. It was a meeting with the engineering director for that she means like Nick where is the big red button. Like, oh, you mean like how do I kill it if it's if it's if it's doing something unexpected. Yeah, like Nick how do you kill it because honestly these folks don't really want this there this is like another variable for them to manage so we have to convince them that this is the right thing to do so how do we do it we throw it modeling and we sit down and say okay what are the fault modes that can happen. The cluster loses connectivity to the network and fully saturate the network adapter what do we do next Falco sidekick services down or inaccessible how do we kill it that we push a bad rule to Falco another rule up data is stuck in a loop. Right, what do we do next, or we have an unexpected CPU load on the kernel, and that that's maybe the fault of Falco how do we have observability so we spent like a year studying how to do this correctly. I'm not kidding like a year, and then implementing all these control mechanisms these big red buttons as I call them right to roll out of Falco that's just one like one major product yeah we chose open source Falco instead of cystic like enterprise Falco. Why because we felt we could do it ourselves and we wanted to take the time to really do it correctly and not put ourselves into a position where that when I get you know challenged by an engineering director. Whereas the big red button I know where the button is I know how it operates and I'm confident that we need to want me to press it we will press. If we need to. So, that's part of our ethos and I think when you build security services in house completely or open source you know exactly what's going on you can debug traces will have low level of their ability. You are empowered to scale the services to your company's needs. You're not locked into a vendor ecosystem and ultimately, this is a lot more fun like, I think I can hire better people who are more into building, and then keep them happy because they're building really cool stuff in the industry because it's a lot more reliable versus like taking boxes on the UI you don't really know what the box does you have to dig in through a vendor documentation. Now I do want to caveat this isn't for everybody a lot of companies may not want to treat their security engineering program this this way but where they have a heavy build or heavy open source culture. Some companies want to treat it more as like a dial or manager role this role decks of vendors run some tools you look at some logs have some alerts you do some follow up at the end of the day this is risk management is the risk that way we've chosen the risk we've chosen to manage the risk a very different way a very intentional way a very deep way a very methodical way and to do that we have to really understand what's going on. And to do that we need to have low level understanding of the code of the of the the Providence of it. If it comes from open source which of course we can always look because open source is open or build it in house. So that's a little bit of our ethos envy. Absolutely. In fact, as you were outlining that particular example. I totally concur that operationalizing security program at scale is an engineering problem. It is not a vendor problem. It's not any of those other things really, and especially at your scale or any you know internet scale companies it's it's the, the tool the product that you're using has to be out then the public you should be able to build faith around the, you know, the thesis the product has, you know, maybe even sort of evaluate it internally to see if it is, you know, it meets your, your own standards of quality really right. And the second one is the most of the float source product and I totally agree with them they probably wouldn't scale to where snap is at or where Google is at for sure really right so you can take the code for open source customize it you know operationalize it truly the way engineering happens and you know top tier tech companies really so it couldn't agree with you more really and you know I think the right way what it is is basically that operationalizing security is an engineering problem. Absolutely it's it is risk management again at the end of the day but to operationalize and scale it you need to create these happy paths for engineering teams right you can make it easy for them to do the right thing and hard for them to do the wrong thing. Yeah, in a way that's as frictionless and seamless as possible. Absolutely you couldn't agree more. And you know what what I love that the fact about open source I mean I mentioned that essentially is the quality of folks that you've been hired just to work on cool stuff which is out there and open and trust me since we went open source. I have made most of my key hires on GitHub. It changes the game you're not no longer right initial recruitment and hiring top talent for example ebpf you know we did the same thing. Most of my top ebpf hires are from GitHub project either they reach out or reach out and that's it. No recruiters nothing so open source essentially. Yeah. The quality the DNA essentially has to be needed in DNA to operationalize any kind of security at scale so yeah. Absolutely. Now, we spoke about we know I want to go deeper on one or two use cases, you know that you might be dealing with or the team might be dealing with that snap neck. But even you have had quite a journey I know you would personally hired at SpaceX by Elon and you know work with a bunch of bunch of lucid tech companies before that. What is your mental model, you know, having been through the journey really it has evolved I'm sure really right based on starting at the 16 year old to where you are now. What is your definition, what is good and what is great. What did just compliance even purchase and what is security engineering driven security program essentially. Yeah, you know, having a chance to work with you on and doing some consulting work 2011 2013 timeframe, I open my eyes to what I call good to great security. I think the major difference between good to greater the engineering bar and the adoption of a either an in house built culture or an open source built culture. Like we really mentioned sick at its core security is an engineering problem. So let's talk about an example. Let's say you have 2500 engineers they need to correctly manage I am permissions for AWS. Right. And we need to do this consistently so how do you make this easier for 2500 engineers, you let them like pick whichever permission they want do you like have some set of permissions do you have some birthright access you have. Do you have like some kind of system you build. How do you do that like, how do you abstract away the complexity of 2500 engineers. Right, you need to build a happy path right and sure there's vendor solutions that do this but in this example snap has also chosen to build in house we built a tool and how to do this. And we've taken the like a leasing philosophy where you get access for an ephemeral period of time, and that that ephemeral period of time that expires and like our app where you send a snap. Once you've read it and replayed it once it's gone forever, ever ever right same thing the same thing that the access now. The balance here is how much overhead this is created for an engineer to come in request access lose access request access was access. We're not clobbering them like every couple hours you know it's every couple of days or maybe a little bit longer. But that's the kind of thing they need to do like we've solved for this at massive scale not even one cloud to clouds, and that internal system that we build it handles both of them and we could probably scale it to a third one if we needed to. So, I think when you consume vendor services, it's kind of like being a hoarder like that's my like mental model you know these these shows about like people who have like hoarding problems and they hoard all this stuff they have like rent storage units to hoard stuff and then like something happens in their life and they need to solve it to me buying a bunch of vendor tools like being a hoarder. At one point you're going to be like, what am I even doing like I'm lost. And these tools that I have I don't know you know what they're for like a lot of tools are never even used. So there was like a study on the three something I read on the internet that I think like 40% of organizations have tools that they never even log into right like what's the point. So you got to be like a lot more intentional about this stuff right because if you're being intentional and building you know why you're building you know why you're there you know why the codes being written and you know my open source is being used consuming tools left and right. Unfortunately the industry wants us to do that I get solicited to probably 50 times a week. You know it's everything from like, you know Nick do you want a gift card do you already want to go drive Porsches with us so I get to know you a bit better like come on. Why do we have to be, you know, having to be transactional. Right. I can't accept any of these things it's against company policy please don't offer me. So, anyway, I think the time for open source really is now the security landscape is shifting to open source first. And these recessionary times and companies are being challenged and see if those are being more scrutinizing with those that role of decks of vendors that I talked about the time for open sources right now like the people doesn't care how much open source costs because it doesn't cost anything to get started and then when you graduate and what enterprise, it's probably going to be more affordable than a pure pure play commercial product what do you think Cindy. Absolutely, you know, absolutely especially in fact I had I had a couple of questions that came up, you know from from this discussion that we're having the first question Nick is to run second engineering security as an engineering function in a way really or or call it secret engineering which is basically operationalizing sacred apparatus of scale. Roughly, what sort of commitment companies like snap look at essentially are we looking at 10 security engineers for every 100. Yeah, that's a really good that's a really good question. Yeah that's that's right so we've historically taken a 10% of engineering kind of baseline proxy metric. So if engineering is 2500 or so about 10% of security across all different disciplines right. We have seven teams across those those those disciplines. But yeah that's the metric that's a big investment. A lot of companies cannot make that kind of investment and if you're a smaller company, you're lifting this like man this guy's crazy like how can you have that many people. Well because we're intentional we care a lot and we build. If you're smaller you can scale that down to the smaller size that you are right. Maybe it's not 10% free maybe it's 5% right but it needs to be a kind of like engineering minded metric otherwise it's unclear what the purpose of this role is. And that is so true for so many products it's unclear why they were purchased, why one of the, you know what the matrix, what are the key goals and objectives you had they never get deployed forget getting deployed at scale they just never even get deployed really find solutions really and you know I completely agree with you that you know open source gets you know forget the vendor side of it purely from customers point of view from a user's point of view. You have some of the greatest pieces of technology out there, you can modify it you can look at it you can smell it you see it fits the bill and not and then go from there. Then of course oftentimes you see that there are companies there are startups who are essentially often support or some additional features on top of that that you can always reach out to, you know, you know, work with them as part of your, you know, your extension as, you know, as part of your team essentially, and go from there to you right so do you do that as well Nick or it's primarily, you know, we collaborate like I'll give an example but we need to build arm 64 support for for Falco. What did we do we went on the Falco Slack forums and start talking to other people who are BPF developers we're not BPF developers. The person who worked on this and part of our sister team like he never even looked at that before but we know again we have a higher technical bar. We give people space and it took some time but he figured it out. And now we have arm 64 support. What would we have done if it was a vendor. All right. Hello, Mr. Account Manager. Hi, I'm Nick. I have a problem. This thing doesn't have arm 64 support. Can you get it into the roadmap. Okay, Nick. Let me go talk to the product manager. Okay, okay. Sometimes passes, they come back. Oh, let me talk to the product managers manager. You know, Nick we talked to the head of product that we can build this for you it's going to be six quarters. That like, that's the reality that we deal with like that that's that that really is the reality I'm not dramatic so. And I think it also depends on how enterprise grade that that particular open source product is I mean is it just a command line tool that you're bringing in and the building. That's that's that's exactly that's exactly right like like like defense and tackle these are enterprise grade security tools, much like what has she core brought to the market when they built vaults. Some 10 years ago right like it's enterprise great it's high quality for the enterprise and happens to be open source this is like the best of the world right like why are more people doing this I don't get I don't get it so. I think everybody is going to do it soon enough absolutely agree with you. So, you know, moving on, I'm already getting a couple of questions more than a couple of questions from audience. The last point was super important for them and I took you see we know how and why essentially. Hey, do I have to put in 10% of my engineering strength to operationalize this particular tool now or will 2% or will 1% or hey you know what I'm an up and coming SMB. I hired our first security higher than that's my whole team, what do I do with open social area I'm sure these are all, you know, valid questions and I would have to dig deeper into all of these so but moving on, you know, for now. Tell us what are some of the challenges organizations may face when trying to adopt OSS for security. And what I mean by this is, is there. For example, you in your case really do you have a checklist that hey this project has to be Apache V2. Hey this has to be part of CNCF or open SS or you know what the dentist report has to be out there and get to otherwise I'm not going to touch it you know, is there a model is there a checklist you know that you or the team had essentially why evaluating such projects for your intelligence. Yeah, that's a really good question. Well, I look I think just to get started like the technical bar is definitely a bit higher and you need to hire people who are technically capable hands on engineers, not all security engineers can code for open source I think you people who can code. Or it could be DevOps engineers or other engineers but they need to be engineers like they need engineering to be somewhere close to the title. So you need to have that type of talent. Now I think secondly is is you need to ensure the project you're you're considering is legitimate and it's serious. And you should also consider supply chain tampering like maybe projects coming out of certain parts of the world. Maybe they have, maybe they have backdoors maybe they have some some nefarious things going on maybe they're sort of, you know, Trojan horse kind of projects to get into your environment and then absorb your environment so you got to be cautious open source as well so the way we look at is like is it legitimate how much adoption has it had in the industry are people in our like tighter circles of security engineers talking about it. How many stars does it have on GitHub how many times has a ripple been forked right like how, how, how, how good of quality is the code like we can obviously audit the code because open source, right. And then how much more do we need to do to make it operationalized at our company, because that that's the part that takes the time right like you can always drop it in and get immediate results it depends on your requirements like if our requirements are very sophisticated if it's a smaller company that may not be as sophisticated so pentest reports also matter but I think reputation matters more and like like really like shaking it down and like getting a couple people to look at it closely that that can probably give you enough, enough comfort, and then the rest is is is implementation and opera operationalization. Yeah, no totally agree and in fact you know the bit that you said about. Are there any back doors, is there anything potentially nefarious with this, I think open source sort of you know gives you puts it out there in open really for so many eyeballs you know like you essentially so many people are looking at at and you know if you look at if you look at the stories every quarter, somebody's firewall gets hacked. But you know the Genesis of Secretary industry was around hacking close source products really right so open source in a way I think I'm fresh with of air because you know it's out there to see you can have your engineers you know look at it pentested you know find the kids treat this as an engineering problem just like you would do for your services you know your critical. You know, I can think of many vendors security incidents that we responded to as a result of poor security practices. Totally yeah totally see that yeah. In fact you know what I wanted to keep this question for the end Nick but I think it's so relevant to this particular point I must bring it up now. This is a question from such it such it is the question for you Nick he's basically asking. I can't do 10% of security engineering from get go for open source, if new adopters are scaling and always his program. What are core competencies needed in security, essentially what such is asking me. Hey I have X people engineering team and I have some security folks. What do I look for in these security engineers to be you know calling them essentially security engineers for this new program really right. So that you can resume that is the question that you know the essence of your question, Nick over to you. Yeah, that makes sense, such a thanks for joining such as a friend of mine for many years actually so great question. Look, you don't need 10% to get started 10% is where you where you may end up over time. I think you need a couple of good people who have done similar work in in and around open source projects either developers of them contributors to them or other companies that have similar cultures, like take any tier one tier two tech company. Most of those companies have done open source projects in this kind of way. And then you decide okay where do I put their focus. And what are some okay ours for that focus. Right now for us it was for the faculty story it was getting run time observability over our over Kubernetes fleet. That was a big big big project where I took it took a while to do that that that was our challenge but our fleet is like very large and complex. And so take like one or two areas where you feel like there's good partners on the market like a defense for example that built legitimately high quality enterprise grade products. Find one or two people, probably at like a l 405 level, which is like senior senior or staff, and, and put their focus on this and if they're good and the product that you're using is good. I think you're going to see incredible results you'll be surprised by how much, how much of value you're getting and how much money you're saving with open source. Absolutely. Absolutely. And then, you know, like you said, it has a lot to do with whether the project that you're picking up comes back as intuitive. You know, is there a company behind is there is there, you know, essentially someone who's sort of willing to work as an extension of your team, even if you don't have a team, for example, you're just building. Is there a community like if it's two guys in a van kind of open source project like yeah maybe it's kind of risky to put your your big bet behind that. But if it's a legitimate company that has large adoption like the examples that we've talked about, they have a product community they have good quality documentation like that's okay like you could start there and then, and then iterate and see and see how your adoption goes. Absolutely. In fact, you know what there is one more question coming from Christian Becker, what Christian is essentially asking is, what about Oasis license management, thinking about copy left and copyright licenses here in commercial. Yeah, that's, that's a great question Christian so yes you need a way to to observe this there are mechanisms and tools that can can ensure you're within license license compliance. So maybe there's tools that scan repos to look for whether the license allows commercial usage or for commercial usage rather. This is another angle of it yes the you do need to manage this it's not something you can exclude, but you can also very clearly when you go to the project like you can see what the license is and before you adopt it because okay can I use this license is this this license allow this kind of use case open source cannot be the wild wild west you can't be like okay team open source slam my hand on the table scatter no you like anything in life you need to have a strategy to be intentional. And so when you pick like one or two or you know or five of these, you can vet them obviously and you should have them before you use them, and then I think you'll have much much better understanding of what you're doing and these types of risks are are really really mitigated. Absolutely. Yeah absolutely Christian I hope that answers your question. Thanks Nick and moving on to now to you know essentially a more specific use case for you right. I want to ensure Nick that you know we're giving our listeners right now something super concrete that they can relate to for example you know hey I'm just building up my security program and I've been told by my CISO, or it's a board level you know sort of you know, a director which has come in, which says hey you have to be at least 50% compliant as per MITRE attack and that means doing one two three and four right that could be basically one of the management that has to be you know cloud specific attack vectors you know as per cloud MITRE attack framework really right. MITRE attack of course is you know it has so many favors now for cloud, Kubernetes and so much more. How would you go about it essentially and it's a very sort of a high level question essentially we can just pick up one single modality saying that hey look, we are working on AWS forget Kubernetes for a moment really just think of it as a cloud environment, and I am supposed to be meeting at least 50% of the MITRE guidelines. And doing so, you know, does two or three things for me makes me compliant gives me peace of mind because I have covered almost 50% of you know, the needed most essential picks really. And you know, essentially, can we map some of the open source tools that we have that that you might have seen to one of the existing one of the ongoing or one of the evolving programs within SNAP saying that hey, look, we would be supposed to be doing this let's say MITRE attack we were supposed to have 50% coverage for MITRE. And as part of that effort, this is what we did, and this is how we went about it. Yeah, do you have any such case to do that you think we can share. No, that's a really, that's a really good question. The way the way I would think about this as follows MITRE is now pivoted and provided very actionable very measurable from initial infection to pivot to lateral movement coverage with their ATT and CK framework right. So you sit down and map this what this means to your DNR program for example and say okay, how do we get signal coverage for 50% of the MITRE ATT and and CK framework. Well, what are they. Let's look at them in detail. Now we say okay how do we get a signal for that. We need some kind of attack vector from an from the AWS infrastructure well we need some kind of product to do that. Right, we need some kind of observability product to do that. It turns out that deep fence offers MITRE mappings out of the box. So if you turn on deep fence for that cloud infrastructure to start observing it you'll quickly see which one which signals you're generating for the for the cloud infrastructure and those AWS it could be at a DM level it could be at a Kubernetes EKS level, regardless right, you'll get that like out of the box immediately and then that saves you a lot of time like this was hard like sitting down like parsing spreadsheets to figure like well I have this thing here it maps MITRE here. This is really labor intensive enough fun. But if you have this mapping in a product like it'll quickly show you like here's the coverage here here's the gaps here, and then you can start to operationalize those to your detection and response team to act on it to triage it right to enhance the alert. And then we get into playbooks and response and containment kind of exercises that you know as per classic detection response goes. And this is this is pre deployment, as well as post deployment, some of these controls, some of the checks and balances some of these tools that need to operationalize to let's say meet 50% of MITRE coverage really. They have to be deployed throughout your CSID pipeline right there, they're not necessarily only as part of CSID in the sense of you know pre deployment or not necessarily only at some time right they have to be scattered throughout. The first you will have to probably figure out all the integration points where you really need to do what, map it back to potential vendors or open source projects and essentially go from there really right to get you that brother coverage. Yeah, that's, that's exactly right like, and there's no like secret like weapon here to how to do this you have to sit down and think about it but coming back to my point around. If the coverage comes with the product out of the box which it does with defense like that's so powerful that this this could take many, many, many days and weeks to to parse through. And I think that's to in a sense it's like a quick start guide already. So, yeah, absolutely. Moving on, I'm getting a couple more questions but I think we can save those for later they're not about the point that we were discussing so let's move on. And, you know, talk about this particular question that that is in front of us essentially is so when you think about building secure cloud native services and be getting more specific about cloud native now. Are there any fundamental controls or partner technologies that are table stakes for organization to consider essentially what are the lowest common denominator to make when it comes to building a cloud data security program is what the question is. That's a really good question. The way the way we think about it is this like the foundations that we need to build to be secure, whether it's in a corp context in a cloud context and other context. Those foundations really haven't changed in the last 20 years the technologies and the how has changed, but the foundations really haven't changed so what I the way I think about this is. Okay. You start with a security strategy for those foundational controls that you need to have in your environment and they could be things like, do you have observability on any piece of infrastructure that is being provisioned to your to a public internet presence. Do you have observability on hardening scanning pre deployment, do you have runtime observability in context, are you collecting logs, are you sending them to a system or a human who can act on them. Do you have audit mechanisms to look for changes that infrastructure that are unexpected. If a pod is immutable, which it is in the clock in the cloud communities context. It changes on a pod like what's going on like why somebody shelling into a pod that that's probably a bad actor behavior now. Continuing my my discussion here. Now you take that signal and you send it to to an individual and then individual X on it, or a system X on it will store type frameworks we can have systems acting on signals now. So what I just what I just described like that, it really hasn't changed in the last 20 years. When we had bare metal VMs and data centers when I got started with this industry was basically the same. It's become more sophisticated the skill has become larger and the unit deployment has become smaller now it's the pod before it used to be the physical server, but the elements of this are basically the same. Yeah, you know I told you again in fact you know what one of the one of the things that I used to want to make. And I'm sure you thought about this too is given the fact that observability basically means being able to infer internal state of any system by looking at the outputting signals, right, which is why you have any LT in you know, observability metrics events, errors logs and traces essentially really in in cyber in security we have the mitre attack and defend we have really granular flavors of mitre for various modalities but there isn't anything like any LT, which basically tells me hey if you just this for you taken care essentially, but what I what I what I learned from what you're saying which I think it sort of you know resonance very well with me is look at four pillars here too. The first pillar essentially is measure your attack surface, you know that is your pre deployment checks post deployment checks scanning CSPM, what have you essentially what could go wrong pillar number one and this is commodity everybody needs it. The modality change, it was VMs earlier now it's communities on top of VMs probably tomorrow something else but essentially the core principles remain saying the modality might change so measure what could go wrong the move on. And then deploy or develop a way essentially to measure what comes in what goes out and what changes is basically what it is really figure out your attack surface and see how the runtime signals are interacting with your attack surface, and that is pretty much essentially the way to. What do you agree. Yeah, I know I completely agree and so we could talk more specifically about this we have some some questions up and it's sort of in the back, maybe we could skip to question nine I can talk more about this. Sounds good. Okay. Now this is this is purely this question comes purely from a start point of view, maybe right like I'm a startup let's say we're running a startup which is you know having this open source project seeing great traction. And we want to know how a top tech, you know, Dr engineering team would evaluate such a such a project really right. Yeah, that's part of whose reputation and even the fitment and you know all of that. Sure. But is there anything more there. Absolutely so first of all, first of all the first question is what what problem are we looking to solve internally. Yeah, and sometimes we do a build versus buy bake off if there's a compelling product in the industry that we're we can get behind. Sometimes we don't sometimes we we we skip immediately to open source or build. We don't have the same modality, because the open source almost always involves like some some building to operationalize into your environment so. And then we ask ourselves like hey what is like the ideal state and this is where like my product management had turns out a little bit. There's this idea of like working backwards or okay now that the thing has been completed and launch and has this capability. You, you will have like an announcement of that capability I'll give an example now we can observe all runtime. How do we work backwards from that and state to actually establish that and then how do we evaluate the potential partners with how close they will get us to the what and why that we're looking to accomplish with objective criteria like how would you know the end of market. How much existing adoption. Do they have how, how many feature requests or pull requests are being submitted into their open source project. How engaging is there, is there support community like we don't want to be on an island ultimately if we have to support something will will be will usually figure it out. But those are the kind of questions that we asked. Right. Now, be beyond beyond that. We also ask ourselves, how, how long will this probably exist and like what's the next iteration of it, and with open source next iteration can be our own iteration, we can fork the project and make it to whatever we want, versus with close source partner technologies, they're completely commercialized, we're at the mercy of the product team back to my, you know, begging and pleading to the product management team to build features. Absolutely. And I think, after that, of course, the open source product, or the project that you're choosing has to be at least as good as the close source options that might be there on the table essentially or at least has to be open enough, so that you can customize it really right and that's when the quality differences will probably start coming in. Hey, what is the quality of alerts, how many integrations it has and stuff like that to you right so. Right. Absolutely. Yeah. Okay, moving on to the next question. And I told you already we have at least a dozen questions for you and I see more and more questions coming in from folks as well so I hope you have time. We'll move on. Of course. Of course. Absolutely. So the question is, you work with household internet scale companies and it snap isn't household name. And that, of course makes you a desirable target for right after three right. And is there a or use case or example of how you're always a strategy helps your company manage cloud media security at internet scale essentially. So I think this is probably the same use case that we're speaking about of your observability but but over to you, if there is anything more that comes. Yeah. Yeah, well, at internet scale as we discussed there often are not alternatives that are palatable that are not open source of built in house. Now to make this to make the answer here like we're useful to a broader audience they're not always internet scale companies. Let's talk a little bit about like the tenants of why open source is useful. So I lead my teams to have extreme ownership and focus on craftsmanship. Right. You, you, you, you eat your own food so to speak meaning the code that you build it should be of high quality we should have unit test we should have coverage. It should be understood by others. It should not be a black box we should avoid single points of failure. Right, which means a single, a single engineer on a team should should not be the only person who understands it multiple engineers on a team need to understand that with open source this is very achievable, because everybody can see what's happening they can look at the repo, we can collaborate on on feature requests and PR is we can do code reviews together. Right. And we know like we know what's happening and that that helps us really for that helps us inform our strategy just like iterate and like build feature a build feature be and so forth. In other sense, I think open source helps us with hiring like I already talked about some of this but like the best people in the industry that we hire they want to build the cool stuff like they want to do stuff that's like really notable. We've open sourced some projects that snap has built like back to the community right. And we won't and we'll do more white white like why not it's we built it we built it for the world will build will will allow allow the world to use it. I think also having an open source heavy culture. It allows you to hire software engineers into security engineer roles. Because at the core like what what is a security engineer security engineer somebody who has context and knowledge of threat modeling and attack surface and risk play kind of risk based mentality but they're also somebody who can code and he was an engineer at the core. So in some cases we've taken software engineers by title and really by background. And we've hired them into teams that build build services or build on top of open source services and we've taught the threat modeling the kind of attacker based mentalities and so forth. Now we open up our talent pool like dramatically if we can hire let's say back in software engineers versus security engineers which are a lot harder to find and they typically come with a premium. So we completely get moving on to the next question. And this is a little bit about, you know, the description that you will need to have essentially is anything about defense or a project like defense essentially threat map about open source, you know, CNAP essentially which helps you scan. As part of CICD at runtime across all the cloud modalities essentially to and ensures that all the all the days you know essential features like vulnerability management, you know CSPM malware scanning security scanning is essentially out there for everyone to use, but also comes with matters included right like you have integrations you can you know integrate this tool in your ops to lane file a georatica in all of that stuff really. So, and this probably goes back to one of the earlier questions essentially is when you're taking something like this, like you know defense we have had a different journey they did not start as an open source company. We were closed source first day earlier right when we started but then we went open source. So the platform that we open source essentially was already being used by enterprises for for you know close to a year. So what would be your input to to start that so vendors like defense essentially that hey it's it's essential to have those enterprise features baked in as part of the platform that you're putting out so that anybody could just download and start using it you know without any customization without putting any additional it's sort of on it. How would you look at it essentially and what would be your advice to you know the listeners. If you're a builder, like like yourself send deep and you would like to take a product to market. And your commercial only you're going to be met with a lot of resistance from people like me. I get peppered for requests to look at stuff all the time. And a lot of it I'd never even look at because it to me, what's more novel is what what you're doing. What's more novel is taking open source approach so in the competitive landscape that that exists in this industry with with just and so many options so much optionality there are very few high quality open source products. And those stand out to me the ones that do stand out to me. And this this is why I'm an advocate for this the space. So my advice would be like hey build this in to your product or like maybe have like some part of the product open source people can start maybe there's like a freemium kind of layer of. Hey you start here and then you grab and when you graduate we're here we're here with you we're not going to leave you leave you hanging right and so like in that sense you you can get the best of both worlds and you get your product to to market for people like myself who are skeptical of closed source, and then you have a kind of like a revenue pipeline to enhance and build the enterprise version into that customer install over time. Absolutely and you know overall it raises the bar for everyone right by permanently shifting these you know demand and supply curve in our case and in cloud security essentially everything that you need on what whether you're building out a security program or whether you have already matured program or you're willing to you know are looking at replacing a bunch of float source vendors with open source becomes a standard essentially right in terms of usage adoption and you know that way you're you know slowly but steadily changing the demand and supply curves of cloud security couldn't agree more. Moving on. So, we spoke about observability we spoke about Falco Falco's use case we spoke about the four pillars that need to be defined, which is basically what comes in what goes out for changes and how all of this interacts with your existing attack surface. Very specifically, you know post deployment I know you're a large sort of here. I think one of the largest communities deployments I've heard of, honestly you know I didn't you know we've been speaking to so many large companies but your scale is very, very different, especially in terms of communities really. So tell me more about this post deployment. How important is runtime security to observability, you know and we're talking about the molecules we're talking about part level process level context. Yeah. None of that. Yeah. Let's let's go deep on this so think about this way you've now you've deployed now you've done all the work that you've done pre deployment that we talked about previously hard on the images. You have ensure there's no malware, you've bootstrapped your your pause now you're deployed now you're running. Okay. How do you observe for bad in this running state and in a immutable service context where a change should never actually happen to a pod because it's meant to not be changed. So you need to observe you need an observability layer like falco or like defense that can basically run a kernel level BPF probe to look for look for assist calls to the kernel introspect them and tell you, Hey, is this normal or is this not normal and if it's not normal fire from alert. Here's some like really good examples. A lot of communities clusters are targeted for crypto mining takeovers this happened to Tesla I've talked about in the class that I built with capacity. That's one of the three studies that we go deep on. They had their API server layer exposed that actor took over there. What are what are many of their Kubernetes clusters and started spinning up Monero crypto mining pods to do crypto mining why it's free compute awesome right. What the lot with a lot of infrastructure going on. I, unless you have logs that specifically look for like callbacks to their C2 control, or other, other log level observability on like the network layer let's say you won't cash it at the pod level unless you have a low level observability with something like defense or Falco you just want there's no I don't think there's a way you can technically do it. So, to me, this is definitely more advanced but to me if you if you care about runtime and you actually care what's happening. While the infrastructure is running before it gets you know killed destroyed and reintroduced because in this context we mostly treat this infrastructure like like cattle right versus like pets. Just like in the corp context if we have a laptop on the network. I need an observability in the laptop because without observability I can't react to a malware download on a laptop on on on the corporate network just the same way I can't respond to an event on a pod in the cloud context unless I have that low level observability. Absolutely. Yeah, absolutely. So while it is essential to do scan as part of the deployment, of course, can you tell infrastructure, you know, using agent base region to solution to cover the basics essentially CSP and moldability management all of that it is equally important to have runtime essentially which which use EVP or for one of you know something similar essentially to get you this low level telemetry and really build detection on top of that right that's that's a high order bit I'm taking a review of me. You need both standing comes first short. It's it's it's a busy don't need but once you have that you need something something which goes deeper right and you know sort of you know is looking at all of these all of these runtime signals to say whether you're under attack or not. That spot on like this is not a a start with this kind of thing this is a graduate from pre deployment to post appointment observability but once you've graduated this is an extent. Perfect. Moving on Nick, and I have at least four more questions I'm not sure whether we have time to take all of those but at least with two of those questions will take. And this is probably the last question from my side. And, you know, a high level one now you've been you've been at multiple organizations including SpaceX and you know now it's napping you if you've been a hacker since your childhood almost from 16 years of the age. What would be one piece of advice you would want to sort of leave our audience with essentially before we you know pause the session and then sort of open it for a question. Yeah. I think what's made the difference and deep as this philosophy of being a lifelong student. Being intellectually curious and really never stop learning so if you stop learning you'll probably stop earning and or you may stop earning as much. Right. This industry is extremely dynamic and fast paced and you should never you should never stop learning so for me the day I stop learning is probably the day they like slowly retire and like just rest somewhere. Not focusing on this industry as much until then I will always continue learning. But learning is not enough to me to me, you need to teach other ones you've gotten to a certain place in life to help evangelize and build the next generation. Right. The people we're hiring now are going to be me and some in some in some time frame right. Right. There's also this philosophy that I take of always be firing yourself, like not actually firing myself but removing yourself from the critical path to love people who you've hired to become better than you. Right. Absolutely. And to me that to me this is this is this is the style of servant leadership that I that I like to practice and really support my team with some. And solve for the boring things first the things that we talked about the the hardening the observability the monitoring their learning those are kind of boring this isn't the ml this ai that right then the stuff the vendors want to sell you like the boring stuff matters a lot. Right. If you solve for the boring stuff then you then you'll be in a really really good place right a lot of people don't solve for the boring stuff that they jump into the sexy shiny object stuff. And they actually missed the point of absurd of how to approach this in a strategic way in terms of security program leadership so. I think the other thing is like have advocacy have people that are supporters of your view and have people that you support in the industry like this kind of relationship I think has really helped me like have a broader perspective and how the industry operates. And pay it forward like give back speak to others, whether it's at a meet up in your local, you know, community, a blog that you write. If you want to get, you know, very ambitious you can do a book, or or teach it's out there like the industry will allow you to do it you just need to step up and and and be a participant. Absolutely make love that paid forward. Absolutely. Exactly what we believe at the fence. I think with that you know what we'll do is we'll just open up for some more questions. I have a bunch of them in front of me, but we'll probably just take two more. Given the time limit so let me go ahead. This is a question for you Nick from Dave. The question is, when do you decide to pick up open source point solutions for a specific use case versus go with a broader platform, which is also open source. So what is the thought process taking up a single tool to solve a single matter problem versus going broader. Yeah, that's a great question Dave, thank you for asking it. I think you just start with your strategy so we're a doc based culture we write down okay problem statements who bar to bar at plus and right we write it down. We ask ourselves okay what are we trying to solve for specifically like very specifically, not high level specifically. And then we say okay how can we accomplish this okay are we looking to scan images but we can do that with gripe. Okay, are we looking to look at a bill of materials, there's products for that open source products free. Come take it, or are we looking for a solution that wraps them together like like defense does again open source, or maybe we're looking for commercial offering because we really want that we really think like this is the one. There are those as well that we've purchased like there are this is the one kind of commercial solutions that are fully closed source that we've also bought. You got to start with the strategy you got to ask yourself like, what are you solving for what are the requirements and how do and how will you get there. Absolutely. In fact that was my next question that was in front of me that is, what are the set of criteria that you look for when you want to upgrade from open source to enterprise version of the product. But maybe that could be the sort of you're concluding your mark on that helps the founders like me and companies who are you know trying to monetize open source. Right. I know that's a really good question. You have to figure out what the market's going to pay for it and what is, what is desirable enough for a person like me or another security leader to pay for the enterprise what does the enterprise need well the enterprise needs, usually like lower level authorization management because I have a lot of users using the system, we probably need additional logs we may need support because we get stuck and we may not have, you know all the resources to go iterate and expand our wheels trying to figure it out through the forums. We may need more advanced features you have to figure out what that is, depending on whatever product area and like in this product area. You know threat mapper gives you observability from mapper gives you context from mapper can have have you harden and ensure what you're deploying into production is safe. Now threat striker, which is the enterprise product version here in this context, it will give you a threat graph it'll allow you to seal off and stop attacks at runtime by basically killing the network connection. Okay, if you've studied this and you decided okay, this is the thing people are going to pay for this is the thing that we're going to basically allow for free. You got to do that for every product that's built. Yeah, absolutely. Yeah. What is day zero use cases, one of the day one use cases where is your IP as a company, you know which what is what is something that you can you know give away build a community around the stuff like that I think these decisions have to be made by, you know, on case by the companies that are trying to monetize, whether you want to go open code, you want to remain completely open source or you want to, you know, have a convention they are totally unique. This was awesome. Nick, thank you so much for your time.