 Welcome to pull request or it didn't happen and we're really excited to kick things off today. I'm going to play a bit of an MC, and I have two very esteemed guests who are going to join this conversation. Raj, he's the senior platform security manager at Ackblue and we have Jack Zaris who's here at KSOC. He's our director of sales engineering. So welcome to the two of you and we are doing something a little different today. We're not going to flip through a bunch of slides. We have some conversation points we want to hit. And we also want to engage with you all who took the time out of your day to to attend this so again, welcome. I'm going to stop sharing my screen so you could see our lovely tired faces. I think we are all parents of young children so none of us sleep very well at night. And we'll get going. So let us kick things off. And we're going to start that by setting the stage. We're here to talk about remediation, get ups. There's some buzzwords and some acronyms involved, but I think they're really important ones. And I think it's, we have the right team here to kind of discuss these topics. So off the bat, this one's for you Raj. What's get ups? What do you do in an Ackblue that's different? How do you embrace this whole notion of get ups and what does it mean for your security team that you're building? Yep, yep, yep. So everyone, thanks for that awesome intro. Get ups, get ups, get ups, get ups. Get ups is something that I would say is not something necessarily so new. However, to kind of provide what I believe the definition is that resonates the most current day, current time is CNCF sort of blessed golden path to ensure that you can codify your workloads and your clusters for Kubernetes in code and operationalize that code in a way that gets the things out into production. Generally means I think right now flux, customize a little bit of Terraform, cloud independence, some sort of standardization on like what you expect to be codified in the code bases, what do you expect to not happen with your infrastructure, that sort of stuff. But I want to like before I like end it is that idea of getting things statically defined as early as possible and having your production infrastructure represents be represented by static definition and have your production infrastructure be able to be spun up somewhat programmatically and by static definition. That is like what I just think about when I think get ups and like just to share these concepts these are not necessarily super new lots of organizations I've been a part of have aspired to get to these sort of things like way back to like the Etsy days where we had all the things and like, we had it just generally running relatively well where infrastructure could be like spun up statically, where we codify infrastructure as the OS that was installed in the bare metal. The lamp stack essentially because we were way back today we actually had lamp stacks like that did all the stuff that were codified via chef and a little bit of Python and a little bit of just bash doing some funky stuff and then operationalizing like this orchestrated that in in house right all the up to like mid career where you know I was Spotify we used a lot of puppet, we're a little bit more native. A little bit more native with Docker, but not Kubernetes. So we like had our own orchestration layer called Helios and not Kubernetes but again we codified things and code, and we relied on that and like we were relatively confident we poked into a random production box, we would see something that exists there and know how to map it back to somewhere was statically defined, and we're confident that's you know, that is the state of truth so that's what I think about get ups it's like this progression of what everyone wanted for a long time for a decade now to the world that we lift in today, which is Kubernetes pods CNCF, and the tech stack around there. That's how I would set the stage. Okay. So, for me it's looking more from the security side. I've worked as a sales engineer. I've worked as a product manager, and we talked about this a little bit earlier where security was detached from the actual work of security application. And they didn't they didn't have, I want to say, probably accountability would be the right word they weren't really accountable. And what we're talking about not necessarily get ups but the topic of this conversation or part of the topic this conversations as code is getting security involved in the get ups process by introducing security remediation as code into into get ups. So from the security side that's where I see a remediation as code playing is security now having skin in the game rather than they give they give you reports they tell you what you know the development team what they should be doing. Instead, they're actually making the changes themselves through the CI CD pipeline via via get via get and having that all audited. So I see that you can see rain going like you have thoughts about go go. Yeah. So what you're saying is get ups is an opportunity. We didn't even plan that but that's like totally exactly the thing right get ups is an opportunity. What's the opportunity. It's the opportunity for a security team, perhaps to more easily and more effectively get involved on the engineering side on the deployments of remediation side, because perhaps in a world that in an organization that's strongly adopting get ups paradigms, it may be unreasonable to assume your screen engineer can do direct coup cuddle commands against the production infrastructure for to affect the change it may be unreasonable for your security engineer to go in and tweak a sequel config in your production database because you don't have that experience you don't understand the other side effects, but in a world where you are living in an organization that does adopt get ups for paradigms. It's more reasonable to expect a security engineer to understand the tool chain that implements get ups as per CNCF, namely customized and flux, and it's actually maybe more reasonable to expect a security engineer to learn it, and to not feel that that's a wasted skill. It's not like they're going to leave that organization and have to relearn a new stack where, you know, for an SRE maybe that's an expected like skill set to be able to learn new techniques, but for security you're learning get ups, like modern get ups technologies and languages is not a skill that is wasted so we can expect our security engineers to learn it, and then to, through the process of learning it PR directly against it. That's what I heard you say. That sounds good to me. So does that starts to sound a lot like security engineers should be able to read and write code. So we're implying here. That's a very hot take. Maybe not, maybe not Java, maybe not, maybe not Python but infrastructure is code code. Networking as code code. You and I talked about this yesterday, Jenny again through a kind of a sales engineering lens like how many sales presentations have you sat in on. You know, every sales presentation I've been handed always had either a circle or figure eight like with the security product fits in your pipeline automatically feed but we're usually what it's doing is it finds security issue and it just notifies somebody. It doesn't actually complete any circle somebody's got to do work to complete that circle to finish it. But feeding the remediation as code when you find an issue. It just occurred to be I say like that actually does complete the loop it actually you find a security issue. You check in what you think the remediation is via pull request and it, you know, can fix that issue it actually can come in reality start to complete that circle rather than it's just a marketing slide. So why'd you had your bets a little bit by saying oh maybe a security engineer doesn't need to know how to write Java and Python but only in for us code. Why, why hedge your bets there why not say no. Know it all, or as much as your stakeholders, you expect your stakeholders to somewhat know here. No you're right you're right. So maybe it's dip your, that's the goal. Typically the security engineers know things like, like networking security that's that's one of the places they've they've been in most often or OS security patches, things like that. And that's the first step is is security understanding infrastructures code networking is code with the goal of getting to applications actual application code. So it may be too much to say just jump into there right away because that's not their, not depends on the security engineer, what their area of expertise is but you're right. There's another opportunity right. Yes, like the other opportunity that is, and just, I mean hedging bets, like hedging it a little bit, a little bit here. One big reason I'm super excited about being a an advisor for you all is because I get to hopefully encourage certain types of philosophies on on how you solve the problem. One philosophy that I'll bring up right now is the tool that is ksock when providing remediation is code is also providing a safeguard rail for security engineers to not mess it up too badly, because you spent all the time already, hopefully recommending a potential fix in code that you are relatively confident you understand the side effects, you probably will have some training behind it you'll probably be able to help the security engineer through the usage of the tool become a better builder. The other thing is that like for other tools out there not just like I see type things like this but if you're talking about like first party code scanning tool suites that are like machine learning enabled and you know find like low signals ratio and like an auto remediate stuff even there. The way I think about those sort of partnerships is like I want the remediation to be able to teach my security engineers, a little bit of code as well. I want to help them out build the confidence that this is fine. And like I think that's the cool thing here with with ksock is, it's not going to be overnight that we expect or we think the industry will expect all, let's just say infrastructure engineers to be able to PR directly against your get ops repo and deploy to prod it won't be an overnight thing but can we as a new fledgling group of folks help that group become better over a period of time by providing the guard rails by building the confidence, either really, really strong remediation annotations and training, some evangelizing I think so. You caught me you totally caught me there because I exposed my own personal trepidation, like infrastructure code networking is code, totally. I've been playing in that arena for for quite a while, but actually, you know I can, I can read application code, but I would be a little hesitant to to propose to actually roll out thick. But I think that gets to the point in get ops, you propose a change it gets basically peer reviewed and checked in order before it gets before it gets set out so a lot of that trepidation should should go away you're not actually making changes into production you're proposing, and then the application developers and this in this particular case would go over that and make sure that what you're doing a sound. Let me give a concrete example as, as we are a relatively new company, we had to build our own infrastructure right and you see Steve Wade in chat here he's he's our senior principal, all things platform engineer and back end engineer and he brings more IAC and get ops experience to the table than I have ever really encountered in my career thus far. So, as a, as a green field opportunity, we, we codified everything right like you can't even create a github repository without being as part of Terraform. It's a kind of a running joke adding users we have all of the AWS account setup, everything is checked in as code. And while that may seem overkill, it really isn't because now, when I need to get repository, the process isn't a point and kind of operational snowflake event that's happened it's actually going through the rigor of linting it's going through all sorts of pre commit hooks, we have security checks to make sure you don't have secrets stored in some in a particular get commit. We have peer review we have this this process that is really powerful to make sure that that is a rebuildable solidly sound like sound infrastructure from ideation to production and that to me, as you know, watching it kind of come up from from nothing from signing up for the first AWS account at a company to something that most developers and security teams safely deploy an application or micro service or a config to an API gateway is is really confidence inspiring right it's it's less intimidating after the plumbing is in place. And you know your automation is taking care of 80% of the hard work, and yes there's people involved but get ups really lets you move much much faster and watching it. The new versus bolting it on to existing kind of patchwork infrastructure has been very eye opening in its capability. So, on that note, I want to like, well before you get too far from that. Yeah, at some point case ox going to be a $2 billion company with like 30 employees and some point later in the future it'll be $100 billion company with 3000 employees let's say, and at some point you're going to have a security team, and your security team is going to be charged with making sure that something good is happening. And at some point you're going to have a security engineers going to say, fucking hell I have no idea how to follow this paradigm and uses cookie cutter template, and then deploy this thing that has convention all over the place I'm just going to go make a security business account and be a like a low snowflake off to the side and solve the problems that way. And at some point someone's going to have to be okay with that or not be okay with that. And I'm saying that as like, that is going to happen. No matter how awesome you Jimmy think your developer experiences right now. Everyone will not like it when there's 1000 people in the organization. And then I guess before we move too far on like what's your thoughts and feelings and how you avoid that perhaps, how you address that is that a truth is that not a truth because I'm a new organization I've been a part of, where I was an IC, I did it. I was like, I don't get this. And I, I'm a particularly good developer ish at the time when I was doing it I got I felt like I could figure it out. But I was just like, this is going to be too long for me to do the thing I'm going to go and it was going to wrap it through and add a sub domain, because I did not want to PR against the zone file management config thing. And I did it myself. And then I got away with it because my manager didn't care. But now that I'm on the other side a little bit I'm a suit. I call it out every single time now. And it does mean that I tell my C. So it's going to take us a week to add a sub domain because we're learning how to do it because our developer experience is something that we're not, we're not accustomed to. But I hold a specific value important to me. Start building do the building you'll get faster at it and you'll be a partner with your dev ex teams to improve it myself yourself. So like, I'm just throwing it out there what you just said sounds awesome now for your five person team. Yeah, yeah. It's true. The scale is. So there's one way I've seen this kind of sort of work in my past life I was at a kind of consumer mobile app company we had a very large infrastructure and we would do is basically introducing chaos experiments. We didn't call it that, but repaving where we would take down infrastructure at random times. And it put the fear of God in development because if you're building something on the security side, it wasn't in get it didn't go through we're using puppet. It didn't have all the pieces and you were kind of SSH and making your own configs. There was the real chance that that night, it could all be gone. And then work would just go away. So it forced you through a path of, you know, this has to be durable, right, and get is the only way to make it durable because that's where things that's our source of truth that we've settled on. That doesn't work everywhere. Obviously, it's a it's an extreme end of that. But I think culturally, it's, it's important to, to have kind of like evangelists inside that are helping people because it's hard when you're introduced to terraform in a big organization and you just need to make like a C name change or something like, yep, I will admit it is not the most approachable subject. So you need advocates who are going to help you get through that. And you need advocates at the business leadership, like the non tech business leadership to understand that well enough to advocate all the way up to the CEO, do you think that's a requirement. Yeah, because you're going to take more time at the first six months are going to be harder than click ups, like click ups will always win for speed, but there's a resilience factor to get ups that can't be understated so the, you know, the last thing that the thing that we're working on a case, at least in the context of Kubernetes is is drift detection right drift is it's really important to understand what's happening in code, typically is not one to one with what's happening in runtime whatever that could be. We enable that in Kubernetes because cute, cute control right or whatever somebody SSH to some machine introduced a couple new workloads for a test environment, they somehow made it in the production, and they're not reflected back in the actual YAML manifests. That's a, that's a security finding, and if you're doing this right right it's like, you don't want snowflakes because ultimately that cluster will go away the nodes will go down. Workloads will get shifted moved and if they're not tracked there there as good as gone so I think drift detection, you know, kind of early days like using that in security, but it's something we're working on because you should be closer to one to one from code to runtime and there's a big gap there today. The reality is that every once in a while forget about the person who's like I don't have time for this I'm just going to do it myself I'm going to do it the manual way. If productions on fire. And you need to make a change fast like I don't I don't have literally we do not have time for this and we need to temporarily give somebody acts direct access to the environment to make runtime changes. That's that's something we're working on as well to say okay we're going to allow that. We're going to temporarily allow that kind of access and fully audit it when we give you that access so yeah, you can't completely close the window because there are times when it's not just somebody. I don't want to say lazy but they're just they want to get their job done they want to get it fast. Yeah, there's competing priorities there's constraints local rationality we all like, yeah. Sometimes. Exactly. And I mean I'll say this socks regulations are a boot, I think when it comes to something like this, because if you actually have to do real socks stuff. And like, also some like real PCI stuff. The idea is like auditing and tracking. It's not necessarily blocking. Right so sometimes it's easier to block, but sometimes it's just as effective. It's just as effective to detect and to allow exception use cases and like if you are implementing PCI back ends, like obviously I work at a blue so there's PCI stuff because there's money, lots of money, lots of credit cards. If you work at any publicly traded company, and you have to make sure you can sure that all your plays are like every very very accurate because you're spending money it's material impact like you're doing sock stuff. So as a technologist in these places, you start to build this intuition, a little bit better, where actually it's not about making it impossible. It's allowing for exception use cases is allowing for auditability trails, and as a builder in those places that starts to become relatively natural. What's interesting I would highlight is as a security engineer on most security teams I've been on, we like avoid all the regulation stuff. So we never get to build that intuition. What else is job that's a GRC person's job, and a GRC developer person's job and they build that intuition. And all we see is like, wait, you can deploy to you can force force push the master, but that your that's not allowed, because their assumptions we should block it as opposed to okay, you can force push the master assuming there's a link you're a ticket that is of this type that has a acceptance flow that was over here that we know is like going to be audited and then audit trail is like lockdown you can't mess with audit trail, like you build those intuitions when you are a builder, when you're forced to be, you know, building things in these environments and like I think that's amazing to think about all the time every time you think about protective controls like, is it really doesn't need to be protective. Can it be more detective. Can it be protective with with escape routes because you understand like your business that needs to be agile in incidents that sort of stuff only yesterday. And we're get opsifying some parts of our stack right now like we're we're an 18 year old plus company so we didn't have get ops, even like the initial definition to get ops when we first started it was like click ops and console ops and whatever you want to call it for a long time. We're get opsifying some things right now. And one thing I had a conversation one of the engineers and that's doing this right now and like, I really love that like it's fully automated the documentation shows the fully automated like workflow. Before we consider this done. Can you also provide me documentation on if we need to manually cut a tag and push something to that environment. What are the, what's the runbooks can we even do that. And if we can do that, how are we double checking that's on the audit trail as well. Like, the project is not done, until that piece is also there this was a conversation at yesterday. And that's how we, we keep it going like we make sure every option we have as leaders, we convey these ideas like don't be too lazy like we are the ones that could be more lazy than anyone else saying that looks secure enough that's safe by default that's in the paradigms that any last top 10 webinar thing we'll talk about. We need to be sure like is it also helping our organization when they need to not do a security thing. And are we thinking about that we need to learn more about the non security stuff. I mean, we're all we're all engineers here if we have a break glass moment where like, look, I need to go around the the approval process or the, the normal process of doing things to get my job done because it's emergency and you're not, and you're blocking me. I'm going to find a way, and it's going to be a way that you don't like but it's going to get the job done, provide an escape has released valve. That's approved that goes through that audit. You, it behooves you to make that available otherwise, people are going to find ways to do it, whether you like it or not. So Raj, how do you build a team around this your, your, your task at hand to that blue is, you know, platform security. How do you take an existing security team how do you hire what's the, what's the leadership mentality that your head is in to actually embrace this sort of new, new ish paradigm. Yep. So I'll say, there's a couple things there. I didn't prepare this one. This will be fun. Let's see if I get one. Yeah. Yeah. Yeah, your team is, your team is listening to it. Yeah, I'm sure. So I'll say this, I'm lucky that your career is not like a vacuum. So I'll start there. I'm lucky the journey that got me to act blue is important to understand. And it's also one that may that kind of makes me select for organizations that are ready for things like this. So like at a very, very, very quick high level. I got to act blue because of a couple reasons, not the least, which is the currency. So, and I used to work together at Etsy, where we admit we both built our values around that were important at time pre public Etsy, which were set DevOps in for his code. Heavy CI CD developer empathy, local rationality, like these were all very important values that we grew up with in our career at Etsy. So that's the currency. So at a blue. So that one thread means that there's at least some value alignment between myself and the person who I am reporting to. So there's that thing there's value alignment. And what are those values? They're aligned because of the Etsy stuff. They're aligned because we all have alumni slack. So we we've been keeping up and keeping in contact through our career growth. And we've been comparing notes regularly. So not only is there value alignment because we started at the same place, there's been continual value line as we meant. So like, why that's important is like I knew already that this sort of concept is something that's there's an appetite for in the existing security leadership at actually. So there's that. Now that doesn't mean like you can still get it right because like that's just the one okay you're allowed to do it essentially. So then how do you do it successfully, you need more more pieces of the puzzle here. The other relevant art through my career journey. I tend to be the one who deployed things to production as an engineer. I have totally like I totally brought that Etsy one time because I was doing a crypto thing and I used up all the entropy on our web servers and TLS just broke. I did that I rigged it up. It was fun. It was interesting. I did that I was at an embedded systems company and that camera over there. I was the one who specked out the definition on the at mail chip on it to do some basic crypto off boarding stuff. I was allowed to do it so I did it. I cost some devs, some, what's it called manufacturing cycles are messed up because I messed it up but they allowed me to do it. So that's another piece of the puzzle as a doer. I also had the values of building so now I knew it's possible. So now as a leader, I decided that when I hire a team and when I interview her team. I look forward with the idea I expect all of us to build I expect us to develop I expect us to push code expresses recode and during that process of interviewing people fall out of the funnel because they don't want to do that and there's organizations that you don't need to do that. But sometimes when you do that early in the funnel some people like click and they are very excited about that they change the way they're even present coming to the interview now they're leading with projects, not hacks. They're leading with bills, not CVS. And it's like, Oh, this is really cool. So now you're acting as a civ to like build an organization that value some of these priorities and and there's that. But there's also this other thing you can do a lot more internal hiring now, you can find people in the organization who don't want to be the SRE anymore they don't want to be the director of info anymore they maybe want to be a mid level security person. And what my job is there is to be like, you can teach me a shit ton about building. I hopefully can teach you a shit ton about offensive mindset. Let's put our brains together and codify a project that will mix these things up. That's my current strategy. I'm lucky I have like two amazing people on the team that deployed our get ops and like have been long term engineers in the organization and fucking really know our infrastructure and I learned from them every single day about like the ways you should think about like flux and customize. And I also have people on the team who are like, really, really bad ass like pen testers, and we work together. We plan a roadmap out six months on what we want to build. We know that the building product roadmap is probably only half of the amount of time we contribute to the company. The other half will be ad hoc things that pop up bug bounty security architecture reviews ad hoc code reviews that sort of stuff. And that's it we we execute on that plan for a year and see what we need to do next. That's my long answer to that question. It's good. No, I liked your, I liked all the stories. Yeah, I mean I think it's, it's getting getting out of the internal consultant mindset that we always talk about where you kind of have this list of issues, you present them in some format and you go about your day. And, you know, to plug what we're kind of building a case doc is that is that pull request like the title pull request or didn't happen and there's a lot. There's a big difference between a poor quest with a first pass at an actual fix, then kind of outstanding ticket in the backlog that may or may not be addressed. The poor quest is where the conversation happens with the right stakeholders it's where the, you know, the sre comes in and does some regression testing it's where you can actually have multiple levels of peer review, and everybody can see the problem. Do tools automatically generate perfect poor quest that can automatically be merged to every environment without looking at them. Absolutely not, but the, the conversation that happens in the fixes the, the different the commit history is your is your audit trail right like the audit trail is as clean as it can be when it's in the form of code. So that's what we're trying to build for for kubernetes because it's a messy complex system and if you're already relying on get ops we should have a path to fix the code for you or at least give you that really strong starting point so and that's a really important key point there you mentioned something like you just said starting point and right at the beginning you said there's difference between a bug in a backlog and the initial pull request. What I want to say is there's bug in a backlog initial pull request starting point, the journey through the pull request, all the partnership, all of the learnings all of the collaboration and then right before you merge pull request that journey is an important journey and we as security engineers need to do that journey more often, because that journey is a very important career growth for everyone who's an engineer, and we if we don't do that journey in a pull request often enough, we're missing that part of our growth as engineers, and so I can start that journey and allow a secure engineer to start that journey slightly more successfully, because they're coming with a little bit more context a little bit more support a little bit more understanding that will allow them to show up in the conversation more armed to have those conversations. That's going to be very important for all of us. So, Jimmy, can you be my boss? Can you be my boss someday Raj? Hopefully not, I need to retire soon. Yeah, no, when I get into politics, there you go, when you're president as we talked about there. Go ahead, Jack. I'm going to put you on this budget me so that journey you're talking about so let's say we create a pull request for remediation is code to fix an issue. And, you know, whatever, whatever suggestion we make that remediation we include is not the same as what actually gets implemented. So what's the process for then adjusting that in the future, that remediation, that that automated remediation if that same issue pops up again, how do we feed that that that journey that learning of, oh, actually this wasn't exactly what we machine learning that's easy machine learning just you know, machine learning AI. The, the beauty of it, we're going to go inception style here is like the remit the remediation is actually code to so you have a place to go back to the remediation that created the, the code that created the remediation, and you can you can tweak your, your parameters and you go through the same process over there. It's a new pull request against the remediation, and it's, you know, specific now to your use case so that's it. It's code all the way down right there's no special magic box that creates remediations that aren't is impact by something itself and get so you have the conversation over there when you need to improve on the process. So, it's, it's, it's very kind of mind boggling but when you embrace that it's the changes live there in the conversation happens there I think you know that's that's the future so that's how I would answer that. I think we're coming up on time Raj once hot takes. I mean, you know, we've, for some reason I said a few days ago we're prepping for this. Put the, put the engineering back in security engineering so you know I think that's, that's the take. It's getting better over the years. But if you haven't learned if you're attending this particular session security folks need to understand the systems they're dealing with I think that's a hill we're willing to die on as a group here. Now you don't have to be a, you know, some sort of kind of gray beard pearl programmer Java programmer that only deals with, you know, the ins and outs of the application every day but there is importance in giving security teams autonomy to, you know, use and deploy to them break stuff. I mean, I've broken. I took down a third of giant mobile application car buying site. They're there by installing a security tool that I didn't understand well enough at the time right and it broke everything. We lost a lot of money this is years ago. But like how else are you going to learn right so I think that's that's kind of the closing note and Raj and Jack have anything else you want to say. Now's the time because we're going to wrap things up. I would say you do things to change your philosophies and like learn new philosophies I don't think you say you change a philosophy, and then do I think what I would say is just start doing just as a security leader or security engineer security as a security leader, ask people to build something and deploy something as an engineer, tell your manager, hey I want to build something deploy something see how it feels, and then see what happens. Well, from the security engineers side embrace embrace the, the way your developers are working because that's the way the whole organization should be working. The auditing the understanding all aspects of the application not just the application itself, but the infrastructure as well. It's all required for for these applications to work. And nobody should be in a silo anymore. Yeah sure everybody has their areas of expertise. I'm not going to start developing code but I should understand by looking at what it's doing. I should be working in this with the same tools that the developers are using as a security engineer. On that note, Cooper is going to close out the session. And thank you all for coming and bearing with the technical difficulties. We appreciate you being here and check us out at case.com or you can find any of the fine folks here. Maybe not on Twitter I don't know if Jack's there but you can find him somewhere on the internet my most popular tweet by the way is I should use Twitter more often every couple years I'll write that that's all I tweet. Okay, all right well, you know, just just email Raj and take it back to the dinosaur age. All right, thanks all. Adios.