 Quick introductions if this works it did work. Okay, so my name is Christian Hernandez. I am a product manager at Red Hat I've actually I'm pretty active in the Argo community So if you you've probably seen me on the slack if you're on the slack there as well And then I'll let Hillary introduce herself here. Oh, thank you. Oh man. I am cool My name is Hillary. I am a software architect at Red Hat and yeah, it does say chief mermaid That's really my business card title. You're welcome and I'm so much of a silent lurker in Argo, but I am around the CNCF slack I'm on the Kubernetes code of conduct committee. So you might know me from that Yeah, cool and In case you don't know we Hillary and I we do a Bi-weekly every other week we do Stream Red Hat streaming on about get-ups. So that's kind of like the link right there. And they're really yeah Yeah, and that's the t-shirt. So by the way sure to give them all away If if you're curious, we do this a lot, right? We You know, we banter back and forth on the stream and this is kind of like a condensed version of something You'll see on the stream, right? So yeah, it's like our best hits almost Yeah, exactly. This is so you're seeing a live representation of what we do our stream And we talk a lot about get-ups in Argo and just you know CICD and dev ops in general. So So how many folks here are what would you consider Yourself a like security expert in your field? There's a few this someone was apprehensive. Okay, okay cool. I'm getting confidence in themselves. I like very cool Not not a lot and the only reason I ask is because I recently have gone into the space and I've heard some great security talks and I was kind of a little bit humbled so anyways if I get anything wrong or if I Misspeak, please, you know that take me to the side. I love learning about this stuff. I've been recently getting into Like I said security security practices As it relates to Argo CD and things like that. So we kind of you know since centillery What does SRE work right or in a former life SRE work at red hats and has gone through some like security implications, I'm like, okay, cool. Let's do a talk about this and I like to start off with Talking about the age of get-ups right we come a long way in a short time I think you know, I don't have to go through a lot of You know kind of like the pros of Argo CD. I think you if you haven't heard it yet You've heard a lot about it today during the conference rights and and not only Argo CD but get-ups in general like it's you know And in red hat we've gone all in Argo CD is is just like the made for automation, right? And it's and it's I like to say like you're you're you're you're automating the automation aspect of it So you're you're taking like the nature of CI CD, which is in itself You know automated right like that's you know people are building like you know these workflows with webhooks and things like that and And with Kubernetes, right? That's also kind of automated right you have that you have like kind of a declarative nature of Kubernetes and CI CD and you're taking all this automation and you're automating it even further and you know because of this right Argo CD and get-ups right by extension has They just quickly being adopted right and and I've seen you know all kinds of folks from different verticals doing great things I was blown away for some of the things that that That consultant doing work for the German government was that I think that was an amazing talk So it's just big quickly just being adopted and as you saw in the morning it's like the number three most fastest growing thing in the CNCF landscape, which I think is just just just crazy and Yeah, so anyways, just kind of the the the idea of Argo CD just being massively adopted Is is I think it's great. I would say were you there earlier today Andrew Block one of the One of our colleagues at Red Hat Somebody came up to him is like hey I met you a few years ago at this conference and you told me hey this Argo CD that's gonna be big He's like and I blew you off. I did not believe you like I'm sorry That is the best moment And Yeah, no for sure and I just shows like just kind of how this community is grown And one of the big things I think Argo CD has That that allows you to do you know in an extension get-ups is to have You know You're able to audit what you have in production, right? And it makes things like triage a little easier Whereas like okay, like you know who committed what and who approved which and you know even at massive scale you can kind of like trace back to what You know what happened where and why right and it's you know And it's like I said before made a CI CD more automated made it the automated It automated the automation But with everything that get ups and Argo CD gets you It's not it's not inherently secure So I'm gonna be a little careful about this because I don't want to make it seem that Argo CD is insecure, right? It's it's not inherently insecure, but it's not sorry. It's not inherently secure, but it's also not inherently insecure either right so like It's the really the point I'm trying to make is that security isn't free, right? like you can't really like buy security off the shelf and So again a get ups isn't inherently insecure, but it's not it inherently secure either It's kind of like you can't go. Please give me one security. Please off off the shelf, right? Security as It's really a practice, right? And so you it's just kind of like It's the idea of like the DevOps movement, right? I think security is you know, it has its own buzzword now It has like DevSecOps. I think someone puts Devs get secure DevOps or get ops or something like that and saying like get ops is a cornerstone of that security And I think that's very I think that's true But Then you know the idea of DevSecOps, right? It's the kind of like okay this is actually more of a practice and this is kind of taking these security precautions and You know security practices and being part of the release cycle, right? We're DevOps where we had like, you know, we're trying to fix the you know throwing over the wall sort of idea of Dev and ops well now we want to put security in that we don't want Security to come at the end, right? We don't security the security team to like kick back your release because of something right something you didn't know and so The So with going back to like to the get ups the Again, I'm trying to be very careful not to say get outs is insecure or Argo CD is insecure But really when you start automating a lot of things and you don't take security into considerations You're actually automating a lot of attack vectors. So, you know in any any part of your CI CD Workflows You're if you're not really taking security into consideration in each step you're actually introducing a way to You know your attack vectors at each step, right? So, you know something can happen in any part of it, right? And so just the fact is having Audibility and traceability and audibility isn't necessarily secure, right? Like just like oh, I can audit. I'm like, well, that's really a reaction everything more than anything else. I Think that Get ops and Argo CD in general reveals your weaknesses and more than anything else, right? So it's it's more like oh like, you know, I think very early on like what do you mean as soon as I commit something? It goes into production. I'm like it says a lot that you're very very nervous that when someone commits something it goes into production There it's in you know, really that's really where we're coming from that You know that that that's kind of like what my point of view is like, okay You know get ops and Argo CD kind of like reveals a lot of the stuff that you know, you may be nervous about and so a lot of organizations that a lot of things I've seen is that That a lot of places are doing a lot with observability right and scanning things like that But that's not enough really like if you really really want to take a security seriously Those are reactionary things like I said before there was like, okay Like something happened. What do I do now sort of thing and and all of that is very very important But not part of the complete picture. So security really starts early in the the software development lifecycle and You know as a lot of us know like I think Security a lot of like like the security incidents have actually happened at the code level So like I was gonna say Hillary's gonna go over like it like incident handling and how that happened at Red Hat for long 4j and I'll say that but it's That's that's where it starts and and this is kind of like where that whole buzzword shift left has has happened Right. It's like hey, we need to start really really early in a software development lifecycle and in the United States There was the executive order that everything requires s bombs now right and there's something in in the United States in the it's in the Senate Senate committee called securing open-source software act of 2023 where they want to Once to hold companies liable for bad cyber security, so we are no longer shifting left We're being dragged left right a lot of these things are becoming very very important Things that like me as myself as a former admin former, you know This admin kind of ignore to be like that's kind of like security's problem It's it's now we're all security experts or we have to be or we at least have to know That security starts up there, right? We're all being dragged up at the code at the code aspect of it, so So that way I don't I don't want to burn a lot of Hillary's time So kind of things that that that I to keep in mind, right? This is still an evolving space Right, you know a lot, you know the executive order said hey in the United States Hey, everything needs to have an S bomb. I think companies are gonna follow that but like really like What are you looking for in your S bomb like really like how are you gonna use your S bomb? This is still evolving where people don't really know. Okay, I have an S bomb How am I gonna use it and so, you know, it's this this is It's S bombs I find are kind of like security theater as I think someone else of posted in there in their presentation earlier is And that's that's true of like all things like scanning or like oh, yeah, I'm scanning everything I have S bombs. I have this is like all of that like tools is security theater versus practices, which is the actual security, right and Something that I'm kind of following is the VEX, which is a vulnerability exploit exchange I'm following open VEX personally, but I think in Terms of automation get ops and Argo CD VEX is going to be very important because now we have a machine readable version of CVEs and I'm kind of excited to see kind of like the evolution of VEX and automation get ops and security as well Because like I said, we're all now we're gonna be indoctrinated and now you're officially, you know security experts We're all being dragged left whether you want to or not so with that I'm gonna hand it over to Hilary Who's an actual expert of? That's a strong word I Love that you put to the rescue on there. I want to noted that he's the one who said that not me Okay, this is what we're talking about right this is a really crude I'm bad at arch drawing of our software supply chain problem And what you're going to notice is we have all these people that are actors But like who are those? Who are they those people acting on that open source repo? I don't I don't know you I have no idea who you are literally. I don't know this person. Um, so Not my friends sitting in the front. I actually do not know this person So I don't know what you what your intentions are so forth like this this creates this creates an Uncomfortable amount of room for the human capability to have error, right? I don't like that I say that a lot that an uncomfortable amount of room for human error We are all of us fallible and while we probably are not going to intentionally be bad actors We are going to do things that can be bad and kind of cause bad things So this is this is holistically this is our software supply chain like this is our problem This is why we have that zero trust buzzword. It's very popular right now This is why because you shouldn't trust Anybody not even yourself. That's why we have peer review I'm not going to read you my slides. I don't have this kind of time So I also don't know what they say because I forgot I'm kidding No, I'm not so but basically in that whole concept of of zero trust Lockdown your repos, right? You need to have mandatory peer reviews You need to have automation that checks on your PRs and does the things and make sure things are solid you need to be Basically doing everything you can as early as possible like Christian said at the code level and that includes where the code is hosted So lock down your repos Depending on your situation, you might even want to consider using a private Self-hosted get right depending on what you're doing that's github is wonderful get lab does cool things You can host your own get lab if you want get lab you can host your own get anything Keep in mind that it is also an open-source project and you can just build it and run it anyhow Moving on. I hope you took pictures of that the slides will be available later So this builds off our last diagram and there is nothing for you to really read here this is just kind of another diagram that takes some of those concepts we just were talking about and Turning it into a visual because I'm trying to get better at drawing actually the entire reason this is here So One thing that we talked about okay security repos is so open-source software you can push Your security onto a vendor you can so red hat provides Container images for basically to everything right and then we are standing by our container images that they're secure So if you go to a vendor for your open-source dependencies and your open-source container images for what you need for your project You have pushed that security onto somebody else. I hope you trust them. I hope you trust us but that saves you that you don't have to go redo things right we are Red Hat Postgres removes the root user we do that for you right our red hat Python has gone through All of the sub chains of dependencies to make sure that we're giving you the most secure version the most secure thing that we can so Basically that's kind of what I want to Talk about here is that you can push your security onto someone else onto someplace else use trusted vendors use trusted images You don't have to rebuild everything yourself. Don't reinvent that wheel So I actually am going to you already Slightly correct Christian here, which I think that Argos pull model does make it slightly more Inherently secure than other CD tools. So because you are telling it you're going to look here right right here and only here and When there is a thing here Then you will pull that thing from only that place and only that way and then push it to only these locations That's better than a kind of a push model where you have something kind of open to just receive So I like I like Argo. I like Argo for that. I think that that's that's better because of the specifics Similarly in that in your configurations and so forth don't use things like floating tags Those are bad and that goes into the whole auditability thing. You should know exactly what's running where it's running why it's running there Always always use authentication right anything where there's an even if you have the option to not be authenticated do it authenticated Just do it always authenticate For a criminal use service accounts, right? specifically in an open shift we have a built-in like security contacts and and Roles and authorization plug-ins and stuff that we stand by and so we love service accounts and open shift They're a big thing for us Use them because what this gives you is the most amount of actions with the least amount of access And that's really important for security. Okay. I need to be able to do some things And there are a lot of things I never ever ever need access to so while I might have some sort of special role that has certain Access that could be considered privileged. It doesn't need all of the privileged access So set up your service accounts use roll bindings everything that you can do actions not access And I mentioned that we do this in our images remove your root users. You don't need them. You don't take them out So Christian talked about this right there are popular scanning tools depend about snick We have something in our quick project called Claire, which tells us like active scanning of the images Scanning is great. Do the scanning because when something is announced when a CVE is announced You need to have some way of finding that thing, right? You need to know where it is so get a scanning tool invest in scanning tools Not just the ones that scan the things like at commit at PR at merge And you want something that's actually scanning what you have running to because you again You need to know what is running and where? But talking about how this is in a little bit of security theater, I'm going to kind of make a pretty bold possibly unpopular statement Major CVE's Heartbleed log4j Sun something I forget the name of all the sudden anyway all of those Those vulnerabilities could have been caught with negative test casing and negative tests Way way way back left like all the way on the left, right? So your scanning is going to be great It's going to show you what you missed But start Using your software badly on purpose Okay, start doing that. I want you to try to Flood it. I want you to try to DDoS it right do bad things to your code Sorry Now we're just going to be her paws. I'm gonna use all that time anyway Okay, so use bad things do bad things Red team it I actually we set up a game that we do in the SRE land where we red team infrastructure I just see what happens and then somebody else gets to go fix it do those things take take those practices play games with it However, you get it into your standard way of working do those things There's also something that I saw really recently that they just sent around out of our pod set group Which is a pre-commit web hook which will double check your commits for things like secrets. That's cool. Do that Store references to secrets instead of secrets wherever possible on our stream We went over something recently called the external secrets operator, which does that love it great pattern follow that And again, can skin you continue a scanning you need to know if something has made it into production Storytime Are we awake? I am but that's because I'm jet lagged and now I'm just like in the silly state, right? Okay, I Live in California So the day the CVE for log4j was announced I wake up at seven in the morning and it is the most insane explosion my cell phone has ever seen And I am getting pinged by all kinds of people at the company who would never usually talk to my pay grade because I am the See I was at the time the SRE lead for a team that was like running some major things in production And guess what they happened to be vulnerable to right? Yeah, okay, so Why did we know so quick right we had all of these things that I just talked about in place We knew what our container images were we knew where they were living We knew everything about them and we had a quick way of scanning them to determine if they were going to be vulnerable, right? So That was the easy part for us Because we had done all of those best practices. I just told you about right we had the scanning we had the auditing We had the observability. We had the things Then it was the okay well now we've got all these things and we know we're getting We actually knew we were actually getting Attacked right play people over trying to attack our services and see what they could get so we have to also move then very quickly So we use GitOps We're very good ops heavy environment with so why we knew where everything was and also why we were able to patch your Respond very quickly so that automation that speeds up all the things When you have everything in place and you can identify where a problem is and you can put a patch out It also speeds that process up a lot too. So we were able to get our patches out very quickly We were able to it was still an eight-hour war room call. I'm sure other people experienced that too But if I compare this to Heartbleed which when I went through that at the company I worked at was this big monolithic system And I was like aware and how and what and just Night and day right it took us forever to go through that whole process Compared to get ups where we're talking matters of hours from the time a patch was was provided to actioning it to actually Shutting off access to certain things that were non critical meanwhile until everything happened all of that was controlled with GitOps So all of our incident response was completely automated It just required a couple PRs in the right places with the right reviews and off we went So That's part of why right you see a lot of you'll see you're still seeing the fall You're still hearing stories of various exploits being used, right? You're probably not going to hear one about us Because of all of these things and I will tell you that GitOps made that so much better So you won't regret having a good GitOps process in place that takes into account all of your security And this he mentioned mentioned this security is a moving target, right? It absolutely is a moving target Everything is the space is evolving. We're learning new things. We're building new technologies all the time when we build new technologies We build new attack vectors. We build new risks You're Constantly going to have to be learning and evolving in this space and with this space And so don't try to be an expert in all of the things Try to surround yourself with the experts in the things and build a community and talk to each other and Rely on your co-workers so we can do that by putting together working groups in your own company You can do that by putting together like I said game days is really fun We do that a lot and anything that you can do to kind of again. It's all about the practices and the process Less so than the technology That's it. Yep, and remember take the survey. Oh, yeah, that's a good one Survey so thank you We don't have time for any questions. Thank you so much for that wonderful talk So in closing here, we have just like three slides and then I'll I'll let you go. Don't worry I know it's beer clock. I know it's beer o'clock in Amsterdam right now. Yeah, no less So There we go. Should have my slides up so One thing that we do want to announce is upcoming Argo us Argo con us is coming We're finalizing dates. It's likely going to be September in the Bay Area So if you have a chance to come to that obviously would be great to have you Also to find there are tons of you didn't get your thirst quenched by all of the Argo content flying at your face all day There are so many more Argo talks Happening over the next three days. Just go to sketch search Argo. There's a fantastic talk tomorrow with Costas is going to be speaking again with Ilya On scaling multi-tenancy with v-cluster. You've got operating multi-tenancy service mesh with Argo CD from solo I.O I'm going to be speaking Thursday in injecting some chaos engineering with Argo CD So be sure to catch those talks because Argo just keeps on coming And then obviously we want you to get involved. So you can join Argo contributors on CNC F slack. That's a great place to get started. I saw a lot of people today We're asking questions. Hey, I want this feature. I want that feature guys I got to tell you if you go to our issues There is no shortage of I've got a great idea But there is a shortage of I've got some hands willing to implement these great ideas So we'd love to have you contributing join the contributors channel And we also do contributors experience meetings every Thursday at 5 15 p.m. In this time zone And so that's a great place if you're opening an issue you're getting ready to open issue Maybe you're confused. You're not sure what to do join that call. We're super friendly We go through issues we answer questions We discuss the best approaches to how to solve different problems and add new features to Argo So you'll be great for that and then we also have a number of special interest groups So SIG security SIG marketing SIG scalability These are all really exciting special interest groups that you can join if you want to help plan the next Argo con You want to join Christian myself others then that we'd love to have you in SIG marketing if you want to be helping out with Some of the scalability or just hearing about what's happening with scalability because there's obviously new features being discussed all the time And we're making pull requests and things of that nature So please join that you can find more of those at github.com slash Argo project slash Argo project Argo project Argo project you got it So get involved and with that. Thank you to everybody from the Argo committee. Thank you to the CNCF Thanks to all our volunteers huge amount of effort to make this event happen and our very first Argo con EU I think next year. I think Koop con is going to be the co-located event of Argo con Great job. Thank you everybody