 So hi everyone. Welcome to this webinar on CI-CD security. My name is Alex Jones. I'm the tech lead for the CNCF. I also work as an engineering director at Canonical and I deal with all things Kubernetes. Today I have Ben Hirschberg who's joining me as CTO at Armo and I'll let Ben introduce himself before we go further. Hi everyone, it's great to be here. I'm Ben, as Alex said I'm CTO of Armo and maintainer of Cubescape. We are working on Kubernetes security solutions here so I'm pretty excited to start to talk about security at CI-CD pipelines. Well I'm really excited because it's not every day that I get a chance to sit down with somebody and talk about things that often are just beyond the curve I think of most engineers who are getting up and running with security solutions, particularly in cloud-native environments so I'm excited to have you with me Ben. And actually I thought one of the first things we could talk about are the trends that are changing in the cloud-native ecosystem. So if I share my screen for a moment and I hope everyone can see this clearly, you know 5-10 years ago we had this very simplistic model of development, test and prod in terms of environments and we had gating at the time that for folks who've worked in banks or in large enterprises you may well be familiar with these gates being things like ServiceNow and it may be a ticket that goes to an external service and that creates a few bars because often these systems that they go out to require some sort of manual intervention, they may well be arbitrary and in the context of security they don't mean a great deal right it would typically be has a dev team submitted the right paperwork to progress to this environment and also what does that mean right? With the dawn of with OCI so with images that are being produced like a Docker image we have an artifact that would go across these environments but prior to that there would have been a rebuild or possibly even artifacts that were dependencies that were rebuilt across these environments. So we've solved a lot of problems in the past few years in terms of the provenance and the artifacts but we still see that many companies are struggling to provide these this kind of high quality gating but bring it into the into the modern era so you'll see that there are folks that have kind of automated gates that can check for things such as does the live in this probe work can I can I spin it out so an automated gate might do a bunch of stuff you know it might might do some tests right it might run some unit tests it might run some integration tests that very rarely do we see automated gates that run any form of security testing right so and live security testing so I'm really interested as well just before I get any further down this pile to know sort of your thoughts Ben in terms of if you take the kind of typical CI CD approach and you apply it to the cloud native ecosystem what are your thoughts in terms of are we creating a larger attack surface these days and do we have more people working in the space what are sort of some of the trends that you think are starting to emerge right now and what are your customers and and users telling you. So yeah I think that you know as of today we are we're really living a new world okay and and some of us even start to forget what was the old world but but really today the you know the speed the way we are you know we are deploying things are are are changing and have already changed so if you know as a technological leader of a company okay I'm talking to you know other you know technological leaders of other companies and and you know every I noticed that like in the last two years everyone is telling that how everyone's saying to the other well we are deploying new new things in our production like 10 times a day or or a hundred times a day or even you know more and and this becomes some kind of you know of a badge of good work that we are able to deploy things very fast in our production but from a security perspective you know it raises a lot of questions okay as you know infrastructure as a code and you know evolve and the way we are pushing changes is not just touching only the software itself but also the infrastructure around it and Kubernetes is you know is in this case part of the infrastructure it raises a lot of security questions okay because you know theoretically those who are who are pushing these changes into our Git repos right are have a specific you know roles in the company and made their main role is actually is not necessarily security okay and and it raises the questions okay then who's looking after security so in this environment okay because we are really looking putting the developers and DevOps engineers in the focus of all of these deliveries and and and this requires you know a specific skill set a specific understanding of security or understanding what is doing what is good and what is bad for security and and you know usually as a security engineer myself you know originally I can say that nothing is good enough for security okay so therefore we need to we need to understand beyond do some prioritization among the bad things so um so I think that that the skill is need to be there and we have to have some kind of an answer of automatizing you know security part of this delivery process and and and sometimes you know I I'm telling my friends that that if I'm looking into the GitHub actions of of actual projects today I can see that they're they're using spell checkers okay in in their GitHub actions to approve new code into the into their project but then they're not using anything so any security tooling or so and and I myself who's you know for me is good spell checking is really important okay and I'm get annoyed okay by by that spelling I can still think that if I need to prioritize two things I'm going to automatize my security first and just after the spell checking right so so I think that that's very interesting right like how they they they feel as if they're enabled enough to put a spell checker in but they don't feel as if they're enabled enough to put security tooling in simply right right so yeah so I think that the importance of of putting security gates into our processes into these you know areas is is is really important okay so to to be able to keep up with the velocity why while we are you know thinking less concerned about secure the security aspects of of this velocity is is going to be you know as we are evolving in the cloud native environment and in these processes is going to be you know paramount thing okay otherwise we'll get lost in this part and and you hit upon something that's really interesting there and I just updated my my diagram to show it but we've moved from the old world of it being kind of fire and forget to now this idea of continuous deployment right and you'll see lots of diagrams similar to this that kind of look like a a wheel right because it's going round and round but this idea that you can now take something locally and like you say then have it in production the same day is pretty crazy and you know developers are looking at real time signals on their production environments making tweaks locally and then deploying out and so to my point a bit facetiously here about using service now some other old school method for gating they're just not adequate and they only compound the fact that security is not the forefront of those of those thoughts so I think it's really really interesting because you know as we start to increase that velocity there are certain industries that just won't participate in continuous deployment until they have a risk profile analysis before they deploy into their target volume right whether that's kubernetes or on a vm or a function right they have certain regulatory and governance requirements that means they have to do due diligence to make sure they're not regressing all by data being exposed you know and some some control that's not being met and so you know that's where you know I was really interested in the stuff that that the folks at keepscape are doing and partially to facilitate that conversation I just want to take you folks through a really simple example so you know the idea that most people are working in a good ops pattern is not completely accurate however it does certainly represent the future that a lot of people are trying to move towards to give you an example I've got a really simple repository called cats right it displays pictures of cats right um it's not super super complicated but it is representative of a common pattern where engineers will build the code in the repository but then they will also have the templates in that repository as well for that code I think a lot of people have tried different patterns such as having your kubernetes manifest in one directory having it in another having a different repo but I commonly see there is an amalgamation of code and templates for that what is interesting though is that even in this world there is opportunity to do better because I can cut a release and I can deploy that out very very easily and very rapidly right if I've got permissions for my work let's say I'm in a mega corp and I can produce a microservice and I have committer permissions what that also means is that it will build an image of that those commits and it's quite easily for me to go to production so with very little thoughts one single error in this cat's repository can be deployed out through a github's paradigm into my production environment in minutes or if not seconds and so I think that only exacerbates the need for not only gating but continuous scanning I mean how how what are your thoughts been on sort of moving towards a github's passage it's obviously a good thing but it does come with some dangers right with great power comes great responsibility right yeah and it is you know this obviously you know the security gating goes around github's and the way that things are getting into production systems or or not even production systems I can tell you that that from our research we are seeing that the law of staging and development systems are also public facing in the internet so this means that that attackers can get there so so production in the getting into these environments is you know is something that's obviously you know attackers are really looking for for different reasons okay you know we can talk about you know these reasons for a long time okay but if I really want to you know boil it down to you know a few things a few points okay the attackers are looking for you to take your data attack centers are looking for to destroy you know either your services or your data behind your the services in in in order to you know cause you downtime or attackers are simply looking for you know to to take you know your cloud account and and start to use it for their own own good and and you know therefore github's is is has obviously you know very some very concerning you know dimensions where where we have to look after okay what is really getting into our gift if once we had to look in into what was going into you know our production system and we did it with you know looking at the actual you know packages you've been telling before okay the old school that we're looking okay preparing some installation package and and in this installation package okay the security engineer we're looking going through it now we need to we need to be ensured that that that actually the the interface with our production system is not the API server of the production Kubernetes but actually your interface with their production is is your github or or your main branch and therefore this is the place where you have to you know look at where what is getting in there and and also you know your you know your drawing made me think okay and for another interesting thing that that not just actually what is getting from a security perspective what is getting into your production but also the time you need to fix it in the old world okay as you told that we are we were opening service now tickets or another ticketing system you're opening up tickets and you know pushing your you know your changes through the whole whole organization today you know this tooling enables you to find out security issues not just earlier to prepare to to not to let these issues going into your production but you it can give you also already a feedback very early okay in the production phase means that as you as a developer you as a DevOps engineer can get an instant feedback okay about your changes and you can solve it right away okay in your pull requests for example and it also makes in in one hand these new processes are are are concerning for a security perspective but in in general it can lower your costs okay because it it go at these processes are can make give you feedback much earlier I think there's a lot of wisdom there I was I was making some notes on my document as you can see it's interesting because if you think about it what you've described there is the lens is shifting isn't it right moving to here right moving to this place on the left over here from the right so for those folks who maybe aren't familiar with GitOps think of it this way again the Kubernetes cluster itself runs a process there are several out there I've got one running here for my for my demo that process synchronizes to your Git repository into your artifacts right it pulls them in rather than pushes them from the CICD but as Ben described what's really interesting is that you can now with the right tooling start to identify things that are going to be a problem later on before they become a problem so you're not actually looking at here anymore so that's kind of where the old pane of glass used to be for security so where security used to be and then where security is moving right it's starting I should say and that's really interesting because I was having a play around with Cubescape and if you looked on my screen in the background what I've done here is I've installed through the marketplace on my VSTode I just went and grabbed it before this little call and I was like hmm what can I show off and so as a as a previous cluster admin on it you know in many many roles one of the big problems that people often put on is host networking set to true which gives you certain certain routing capabilities and access to IP address ranges and what's interesting with this is that an engineer might just turn that on because they copy paste it out of a document or a guide they don't really know what's going on and what's really cool is that I get this pop-up that starts to tell me hey you might want to think about not doing that because it's going to inherit the access from the entire host network and if you if you look further into that it talks about the remediation and I think this is really interesting and I guess this is a question for you Ben but it feels to me a lot like you're coming from a developer experience first perspective on this was that an intentional thing or was that an organic thing did you just decide you know as a as a as a project this is something we want to target because if we make it simple for people to understand then they're more likely to use it so um yeah I have to tell you that that that how we started the cubescape project it was really you know um from a developer perspective and not just developer per se but also operations SRVs DevOps um we really were thinking about you know not the classical security persona in the organization okay because we understand that that today just as you you know you drove it up here that actually the the you know the way that the world has shifted into the direction of where the things are happening really and where the things are really happening is is around the code and around what developers and DevOps are doing therefore when we created this project okay we decided to target actually both personas okay we are not saying that we are against any security persona here but but uh we've we've really targeted the developers and DevOps okay and enable them to uh you know with the same engine okay as you would you know scan for security issues your cluster you can scan the greatest objects you're creating even before and just as you shown in the in the vscode the plugin example already in the developer in the development phase uh to show you these issues raise these issues and you know going from the developer phase to the other gates to the other phases you would have the same uh same engine okay if we are talking about the between engineers okay you're taking the same engine through the whole process and this enables you a lot of good things um and not just you know showing early these issues but you can synchronize actually your your expectations across you know the whole uh left to right pain yeah I mean I can imagine that if I was to copy this to my security team and they used a different type of scanner then it's almost like you're wasting that effort having to translate one thing to the other and so that was really cool because before this we set this webinar up I was playing around and I built an action based off the docs from Cubescape to do image scanning um to misconfiguration scanning what was really cool well I'll show you what was cool is I was able to just add it into my workflow as an engineer so if we can imagine that this was my my local directory you know I'm cutting my code and I create that new PR um what was really fun is that it adds itself in as a check and I can actually see if there's a there's a misconfiguration you know in my code um and what's also interesting is I believe event is there's a way to to tailor that isn't it you've got a couple of methods there's like thresholdings exceptions I mean do you want to speak a little to how how that would work in reality because I know that in a real world no one's going to get rid of every problem yeah yeah so um so actually too you have as you know what we're talking about right now is using Cubescape as part of github action as part of a security gate what kind of code is what kind of codes you are accepting into your your cluster okay or sorry not into your githrepo okay and eventually into your cluster so uh you can have different approaches okay solving okay that what are you doing with these issues you're seeing okay here you can say on the one hand that well okay i'm Cubescape is generating uh an overall risk score which is um you know we could hold the webinar about how this risk score is calculated but the rule of thumb okay you could say that check what is your current risk score uh and say well okay I don't want to go below this risk score okay so you would use this score as an as a threshold and you have a common argument in in the github action for for applying that there's one one approach another approach is that well you know I'm fine with accepting uh uh you know uh low risk issues into into my repo and every like every issue you know Cubescape is really raises has you know this severity uh uh scoring okay of of uh critical high medium low and and you could say that I'm okay accepting low issues but I don't want to see high and critical or I can also find with with medium issues and this was caused the the you know the PR to fail if someone introduces a high or right issues but I think that that yeah sorry finish finish it off so there is another way that that you can with Cubescape and I think it's very very powerful to to create what we call exceptions like you know you're saying that well Cubescape is checking whether I my deployments are doing uh using the Linux hard-drying capabilities okay which you have you know shown just on the screen before and you can say well I'm fine with with design I'm not really you know uh concerned about this issue and and and you can create exception in simple json file and keep it in your the same repository okay so as part of you know of the PR processes someone can either solve their issues or can add the specific issues into the exception file and say well this is something I'm okay with and these are the three ways presumably presumably if you're running through CI CD your security team could actually keep their own exceptions repository right so you actually have this ability to have a separate type of persona who's managing the exception so I can't just go and bypass them without talking to somebody right right so this is I think that this is the most mature mature you know approach okay that's really you know to to split you know the ownership okay in this case and have you know the security team to to to create these exceptions um and but but it might turn out that you know actually the security team will also you know manage it as part of the git okay and it's really dependent it depends on your organization and your you know the way you're working um but you know kubescape enables you to handle according to how you would like to work okay and define your workflow I suppose what for me as an engineer I find most appealing is that because it's built you have the ability to use the action and you got the local experience you are being told quite clearly several times like there are misconfigurations so developers can no longer claim ignorance right like oh I just put this thing in so that when you get a massive um you know vulnerability report come in at the cluster level you had plenty of opportunities prior to that right and I guess that takes us to the third part of this isn't it is that we've described um how you do a lot of the shift left so you've got the local config that's being checked we also have then the ability to run it in the CICD so CICD in this scenario I've described as sort of my github action you know run um remote checks and then you at this point let's say you've gone through both of those and that's not really the end of the story is it because as an engineer and and as a sysadmin I still need to make sure and even as a security expert I need to make sure I have continuous scanning in the cluster like I know that you folks have an offering for that as well talk me through a little bit about how that works because you know I've played with it but I'm not an expert yeah so um so we're really in the cubescape project we are targeting for the whole range okay um you know at the end um and I'm a big fan of github's okay but at the end okay you need to you know you need to also look into what is actually happening okay in your in your actual production environment um and therefore you know there are two ways to to use cubescape or you know other tooling okay you can scan cube api okay with the same cli tool uh we are releasing and and you know see the same issues also in your production environment in case you haven't fixed them uh before uh and the other option is to install cubescape as part of of your cluster you can install it you have uh you know a simple health chart installing it as a microservice and in this case the the cubescape microservice will monitor okay uh your your production environment it will monitor okay uh your your cube api and we'll check okay every once in a while that's how you know your deployments your cuban's objects are are looking like and it'll also um it will also scan you know the vulnerabilities in your your images and and eventually okay we'll as the project our project is progressing okay we'll connect even more data feed data streams to to cubescape to check to make a better prioritization of your issues and maybe find issues with security issues we haven't you know we cannot detect through cube api or image vulnerabilities can and as of today there are two directions okay to take this data from okay one is that you are using a standalone uh project in your cluster uh and in this case you can visualize the results with the with primitives uh we can export the data into primitives and and from there you can take it into grafana or or to with other integrations and um and we have our uh almost cubescape cloud offering okay where we are freely you can push your your data there and you can do the monitoring uh view uh from this uh you know from this sass and you can look into yeah that's super worth that's worth talking about right because right when I when I installed a helm chart and I got up and running I instantly realized that it was at that point in time the personas who can have access to this now far exceed kind of just your engineer who's working down in the weeds in the ci cd logs or in the local system right right one of the things that I was first drawn to was the ability to have um like stuff like visualization right see you obviously spend a lot of thought on who these personas are I mean for me one of the things that relating back to my previous experience I would have loved is for other people from other teams to be able to look at this data and notice as well with things like registry scanning and image scanning there are other features that you can leverage as well to make sure that more of your estate is kept in good hygiene and outside of just a one repo right I really like that um you know from from your perspective you know these are these are kind of things that I think are super useful to have continuously working do you see um do you see kind of like the ci cd process is just the beginning and this is more of the of the kind of where the the heavy continuous uh workloads are going to go so you'll spend more time looking at the results from scanning or is it going to be kind of like as you said before they'll get an alert or a pop-up in you know whatever they use saying hey something's changed like I guess that's a big question right yeah yeah and and again you know there this is really about you know split of ownership some and making our work you know the most effective collectively you know as as an organization or trying to deliver not just functionality but deliver the functionality in a secure manner okay um and and you know there are are really two kind of personas here you know one of the devs who is who is in charge of you know delivering the code and the other is the you know I would say the security engineer or those who are tested with with the with the security okay of of you know the infrastructure and and the whole solution because uh because they still need to have a tool where they are seeing the whole system through uh uh um from the security perspective okay and and this this part okay of the of of the solution okay it's really more talking to them okay that that the monitoring part okay that's whether something that it might have slipped through the cracks or or something that wasn't delivered in through the right channels are getting into uh into the production there you know you cannot you cannot say as a security engineer that well um someone was able to to deliver to the production not through the gith githops therefore this is not my problem okay obviously the security engineer will look at the actual uh you know production system and he needs to monitor it and but having said that okay and and I I really you know believe in that that even in this case um when the security engineer identifies some issue in the production system uh we believe that um that that they need to be able to tell the same language as you said before with the right so they need to able to point them in the right direction it it has to be a very very short um you know circuit of discussion okay here to be able to so they have the same language they had they see the same issues and and and and this is you know the direction we are we believe in it's interesting and I'm smiling because it's detected that one of my own repositories one of my own uh pieces of code has vulnerabilities in it which is really funny um which I'm sure it does so if we go to this is actually a good proof because if we go to Watchman which is a project I've just been writing for kubecon and we go to the go mod um there there is a vulnerability in one of the um one of the libraries that I'm using in here which is the Prometheus client and uh it's quite cool so I think it's this one here this client golang Prometheus um and it's quite cool it's picked it's picked that out and it's also identified that it's related to a um a particular cv so you know I had no idea and of course now knowing this I'm going to go do a go get upgrade or I'll go think about what I'm importing into my images so already you know I think even as an individual user it does make me more conscious and I think what's interesting is you know we look at the checks on github and we use this as like a marker of prestige right like all the tests pass everything's beautiful all the linting passes we should be thinking about that about security as well right so that all of the tests uh and on and the controls have been tested should have also passed and you should feel good about that right and I think that is the way that we make this work is that we design this to I don't want to say gamify it but we certainly make it something that people feel proud about right that they consider that as if just think about five ten years ago testing was such a hard thing to get people to consider right but now we've had an explosion of quality and testing and now we consider it as a first class you know piece of our our consideration when we're building software it should right be the same for security yeah yeah I think that that's really you pointed into one of the most beautiful things okay well of this that that that once that you know something like 20 years ago okay you are I know people were less thinking okay of of security and and testing as as being uh you know a fancy thing okay and I think that that I always said to myself that as an engineer okay when I felt that I wasn't challenged enough okay I found something to make make ourselves more effective and more interesting of for example of through automation okay automating okay the way we work and and and you know the things which are not challenging okay let's save time on that and and make them work so this is really what's happening today in the sense that that today not just you know unit testing and component automatic component testing and so integration testing is is has evolved but also the security tooling there automation has evolved and and and you really can optimize very uh I can say boring stuff also uh and make them you know interesting and work fast and create a more quality of work as a developer uh as before you've reminded me of a um of a maxim that I once heard and that is create a pit of success you know you want to make it so people fall into it and it's super easy and I think you you folks are around on the right track there and what's awesome is that people can go off and try this right because it's all it's all available on github and you can play around with it and join the community and which reminds me I have a final slide uh so if you are interested in using um cubescape or chat to these folks check the qr code visit that github equally my my my twitter alben is also equally I'm sure happy to answer questions but I think that's that's a wrap for today right I think that's everything that's real rap I think that and and you know cubescape is an open source project it's a it's a community driven project and and and you know we are really looking forward for you know for any feedback okay or no contributions and joining our community I think that we are making something really interesting awesome thanks again bye bye thank