 And off we go. Welcome to everyone. Welcome everyone to this week's CNCF live webinar. Paunting the CI workflow and how to prevent it. I'm Libby Schultz and I'll be moderating today's webinar. I'm going to read our code of conduct and then hand over to Stephen. Jager and Barack Schuster with Palo Alto Networks. A few housekeeping items before we get started during the webinar. You are not able to talk as an attendee. There's a Q and a box on the right hand side of your screen. The chat box. Please feel free to drop your questions there. We'll get to as many as we can at the end. This is an official webinar of the CNCF and as such a subject to the CNCF code of conduct. Please do not add anything to the chat or questions that would be in violation of that code of conduct and please be respectful of all of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF online programs page at community.cncf.io under online programs. They're also available via your registration link you use today and the recording will be on our online programs YouTube playlist later today. With that, I will hand it over to Stephen and Barack to kick off today's presentation. Okay, awesome. Thanks Libby. Yeah, welcome. Welcome all. Thank you to the CNCF for hosting this webinar. This is amazing. This is Pony the CI the get up action edition and with the idea that we'll eventually maybe do a get lab or circle CI and we'll do a few different versions as we as our research progresses, let's say. I am Steve Jager. This is, this is, I don't know, I just realized I've done this talk and I have my boss as my co-speaker so I am a complete idiot in terms of setting up myself up for high pressure situations. I think that you'll be amazing Steve. All right, so we'll do the obligatory intros. If you've never seen me do anything like this before. I'm a developer advocate at Bridge Crew Bridge Crew by personal cloud at Palo Alto networks. I've been writing code since 1990 and did a lot of quality automation work in security since 2014 for a bunch of different companies. If you find yourself in London, I run a meet up called this like a London gathering. Come join in. And I have a cloud native security Twitch show. And if you want to know more about that, and you love and you want to take a risk. That's why you linked in via that QR code. Barack, I will let you introduce yourself. Yeah, I'm a friend of Steve. I like working with him co-founder in city of Bridge Crew that was acquired a year ago by Palo Alto Networks. I love to drink wine. So if you have any recommendations and you want to deliver some I'll give you my address on direct messages. And I love Star Wars. So if you love, if you have spoilers for the next movie, don't share them over the end. Yes, in spite of founding a company based entirely on Star Trek. Excellent. All right. Let's let's quick whip through on the agenda where our CI today I already mentioned is going to be GitHub actions. We're going to go through some attack vectors. And we're going to actually do some of the attack vectors as well, which is the exciting live bit that's going to go completely wrong, probably, but you'll be here to see it. So we'll talk about the threats that are from positions and then, of course, some of the attack vectors we're going to go through command injection, insecure image reference. We're not doing that one. We're going to talk about it and create reverse channel runner. Hopefully on that and maybe cover our tracks, which will be pretty cool. Next slide, Steve. Be good. There we go. My video going weird. A little. Yeah. Okay. I meant, but I should probably also caveat that I'm in a hotel room. So internet may fluctuate. So I'm going to turn my video off just to make sure that I minimize the disruption for now. I'm setting the scene a little bit. I want to talk a little bit about the OS top 10 because in 2021, the new OS top 10 came out and there were some kind of significant differences. The ones that we're talking about today is that misconfigurations and insecure design issues came up at well kind of with a bullet at four and five and that's really what we're talking about today. Move to cloud native has meant that some of the more traditional apps like issues like infant sanitization, etc. have been replaced by misconfigurations and supply chain risks, which is, which is fascinating. And that's where we're, that's really where we're going to focus heavily throughout this talk. All right. This could be teaching you how to suck eggs I believe is the, the term they use in, in England. And just to explain what GitHub actions are just in case you're completely new to the entire situation. Get up actions is a continuous integration or and or continuous delivery system that allows you to automate your build test and deployment pipeline. Most people think a few years ago, GitHub is just where you put your code. And now you can actually take action on that code via GitHub actions, which is cool. And then your runners can execute different steps and make things happen and we'll see some of that shortly. This is a cut and paste from GitHub's documentation and it's intentional because of the way they give examples of what what you can do with GitHub actions and that is the workflows that make things happen are in a specific directory called dot slash workflows. And the examples that they get, give are perhaps, you know, hypothetical one workflow to build and test your pull requests and other to deploy your application. And another interesting one that says you can add a workflow to add a label every time somebody opens a new issue, which is fascinating. So we're going to do a little bit of that. And maybe abuse that as an example. This is what a workflow looks like. So you can see we've got on push and on that push we're going to run a few very simple steps. So that's a crash course in GitHub actions, but you'll certainly go knee deep by the end of this in terms of what it is we're going to do. Now, the fundamental problems associated with GitHub actions and this is maybe something that we would say isn't necessarily a design issue, but it's more a case of security awareness that you need to know. And I'm going to just make them all appear. GitHub can and often will run a new workflow that you create in the GitHub workflows path, even if that's the file you committed, which seems probably seems kind of weird. So if I if I create a new commit, and it happens to be a workflow file that's in that directory. And you can't it will run that workflow as if the workflow was already there, which is like seems like a bit of a paradox, and it's something that we're going to play with when we're going to create some abuse cases for today. The second one we're going to look at is metadata, like the name of an issue, for example, or the description are often are available to workflows like the like an issue grooming one that we're going to add today and often people there are a lot of examples out there where people are using those inputs for a variety of different ways, without actually checking to make sure that something malicious hasn't been used in place of the name. So unsanitized input which is kind of a traditional absec issue. The last one actually Barack you added this one so I'm going to let you go for the last one. Yeah, let's say that you have a code repository, your grant access for all of your engineering team to write new code into that repository. But do you want the same level of access to your workload files to your deployment pipeline to your CI pipeline pipeline being a giving the capability to decide what test to run or to not run to any engineer and not centralize this level of access to a few individuals. So you cannot have a different set of access controls on work so twice, which is an imitation on on the different gate providers. Excellent. Okay, so we're going to break down some of the attack vectors we mentioned earlier and I'm using as the headlines kind of the suggestions that that we're in the GitHub documentation. I do a lot of cutting and pasting from GitHub documentation in this presentation. So the idea that a workflow that adds a label every time someone opens a new issue. If there's something that they're doing with the title of the issue and they're doing it in the wrong way and we'll look at examples. There might be a way that we can execute a command injection attack on there. The example where we're adding a workflow to build and test pull requests well we can potentially push a new workflow that does some of the things that we want to be able to do an example we've got here is. I could push a workflow that actually runs and a command to the GitHub API that will auto approve my own commit my own PR which is weird right, but it is possible and we're going to talk a little bit more about the power of the built in GitHub token that is associated with workflows. And of course finally if you're deploying your application well we can push a new workflow that might exfiltrate some of the environment variables or some of the GitHub secrets that are that are available to that particular workflow. And in a worst case scenario, say if it was a self hosted runner that is running, you could potentially even run a workflow that would create a reverse shell to a command to control center central and you would actually be logged into the runner. And if that self hosted runner happens to be doing happens happens to be in someone's environment. You there's a lot of potential dangers there actually. I'm going to hand over the sorry back like the danger the dangers of me logging into your self self hosted runner. So, in a self hosted runner you have access to all the VPC that runner is within it assuming that it's not least privileged. So you'll have access potentially to a database that is there to secrets that are accessible through IM access. We also have access to all the environment variables connection strings for databases even if they're meant to be used in tests secrets and more CPU horsepower. So, you can harvest the resources of that machine or use that to create another attack or learn what else they're out there in my own VPC network. So it's dangerous essentially, and that is what we are going to try and do today. The attacker context is there really there's the idea of an external attack and GitHub has a lot of settings around there to try and prevent external people just being able to push code really easily. However, it's not exactly like in great big bold scary alarm bells ringing kind of bold, let's say, for example, if I commit something in order for me to in order for workflows to actually execute and I've never committed to the repo ever, ever before, there will be this So you have to look at it and say, okay, you know, I do I want to push the approval and run to run my workflows and I guess it's psychologically you might be thinking, well, my workflows run all sorts of security checks and so of course I want them to run that would be silly for me not to. And I may not realize and you can see the upward arrow there that if the commit is a workflow I'm allowing that workflow to run by clicking that button, which is kind of it can be easily done. It's good. It's similar to having this do you do you give your consent for cookies you click. Yes. Are you approving the next Windows update. Yes. Are you approving running this workflow. You hit yes and it will lead to someone utilizing your workers of your CI system for potentially bad actions. Because we're humans and we love to click buttons automatic. And so the idea of becoming a malicious insider is what we're going to kind of play out a little bit today. There's a lot of things that say if you're a first time contributor and you can see there's different security settings down there in this image from the GitHub settings. So you can require approvals for a first time contributors who are new to GitHub so that's probably the lowest security. The default is the middle one approval for first time contributors and then all outside collaborators with the default. It's actually pretty easy to become a first time contributor. You can just make a change to a read me file or do something quite innocuous quite simple. Then then suddenly you you're past some of these initial limitations in terms of what you can and can't do. So it's not all that doesn't require all that much real light social engineering to make that happen. So here's kind of a real world story. And I'm going to let Barak do this because he's the one to show it to me. It's hilarious. Yeah. So we had an external contributor my young 34 trying to contribute code to our own open source project name check up. And he probably have done it through the web console of GitHub and created a small file change to one of the files and update or build the amount to one of our workload. Definition files. Let's see how it looks like. Well, our original file on the left side has the whole build pipeline. It has secret scanning. It has unit test. It has static analysis test. It has infrastructure as code configuration test a bunch of a lot of testing to make sure that the code we are delivering is doing what it should from the business logic. Reliable test tested and secured. And it's running on a self hosted runner on our own DPC on our own segregated environment. But the contributor have tried to get into the self hosted runner and see what processes are running in the PS command and print off the environment variables included, including secrets. If I have GitHub token, he would have gained access to do GitHub operations. If I had AWS access keys to gain access to my AWS access keys where my self hosted runners are actually running. Luckily, we had some controls to prevent that from happening, but he made a request to do that change, and it was not running. But if I would have the default configuration or a bad configuration, I would have hit a proven run. This contributor would have actually have our set of environment variables printed into the console. And that's what we're going to do today, but we're not going to be so diligent. We're going to let it happen. Okay, so hack number one we're going to do is leveraging issue grooming. I'm going to set those things. These are some of the things that when I went to look at GitHub and you can just search on github.event.issue.title. And I'm not going to do it live because on the grounds that it may incriminate the people via the results of the search. But this is just one example where I saw somebody adding the title without checking what it was and generating web pages from it. And some of the documentation was saying this is so you can see how well they're burning down issues. Of course, the reality is that I see that I can actually probably create a title of a of a issue that actually has functional HTML in it that would potentially present a login form that say, Oh, you've just been logged out of GitHub. Please log in again and send those credentials off to my malicious site. So very dangerous way to to just blindly use metadata that's been provided without thinking about how how how you should do it. And this is actually quite difficult to even try to sanitize that content. So best to not do that. Be careful with what you do with the titles. The other example here is where if we search I can there are I think we found something like two and a half thousand examples where people were echoing the issue title in the issue body into the log so they can say I'm about to add a label to this issue documentation. The log great. Now the reality is, well, what happens if I add my own commands to the issue title and use the the back tick as you can see. So that bash executes that command into the echo. So suddenly I have remote code execution inside this particular issue. And let's let's take a look at how that's going to play out. But first, actually, I should introduce you to the many moving parts that are part of this this demo. On this side, I have me. This is my normal GitHub. This is the repo that I have made in prep for this a workflow in it has licensed to read me it's very big importantly both lots of code. There's really nothing in it. And then over here, I have my malicious attacker currently is an external person because loud Canadian is this person's called is not part of that is not a collaborator in that repo. Other pieces of this I have are I have a site here called web hook dot site. And that's just waiting for curl commands or API commands to to reach it. So it's actually if you've never seen it before web hook dot site is a really handy site for for things like this or just testing API calls to make sure that your data looks correct. And then the final piece of this is over here. I will be having or I will I will be waiting for reverse shell but we'll get to that a little little later. So what I want to do now is I'm going to add a workflow so I want to contribute to this. I'm going to add a file. Now you can already see issues.yml that it's going to automatically create a fork of this project because I'm not allowed to just commit something directly to this, which is good. You know that's a that's very solid defaults. So if I paste my issue grooming in here I'm not actually adding labels I'm just going to do the echo of the issue title. And I'm going to suggest that this is something this is my first contribution. So now I look like a really nice person who's helping and suggesting that hey on your site you might want to do things with issues. And that that is it so I'm going to propose this new file. And create the pull request because you can see it automatically created. That's fine. And off we go and we can see things starting to go one workflow waiting approval great. That's a good thing. So we can come over here to the actual owner. You can see the pull request. And there's our approve and run. And I'm pretty happy with it. I'm going to take a look at the files that changed. I'm a big fan of loud Canadian because I'm oh look at that loud Canadian such a good person he's trying to help me make my repo better. I'm going to say approve and run. And we can see there is a workflow here it's just the default. I've just added the the one that github gives you that prints hello world. So I pre I pre populated that default workflow. So it runs. It's fantastic. Everyone likes it from the marriage. Great. Okay, go look at the code. There's a new issues.yaml. And it's going to trigger when I create an issue. Cool. Well, that's not quite the end. So now that I've made that contribution, I can create an issue. Nice. Well, what's my issue going to be? It's going to look a lot like I'm going to cut and paste it because there's no way I'll be able to type this sensibly. There's my back tick paste back tick. So this is the first potential thing. So what I'm doing here is I'm going to get all the environment variables. I'm going to a little bit more clever than the one that was submitted to to our own tech. I'll repo and pipe that to a curl out to my web hook site right there, which is just sitting right there. So let's submit that new issue. And then we go over here and hope it actually executes the issue. Which it should if we cross our fingers. We should be able to go here, click on the actions. There we go. And presto. Just now 625. I have been far more successful than the attacker on check off. I now have all the environment variables. Poor poor barracks. AWS keys are out in our sitting in my in my public repo. So I can go from there. Not too bad. Now this is a get a get have hosted runner. So there's nothing really of any damage in there, but you can see though, there's an awful lot of information. Even on a get have hosted runner that you might be able to try and find a means with which to leverage. All right, we can move on to the next example. And that was what do you do? We actually covered a lot of this practically. What can you do with environment variables? So I mean, we can double down on that. Is there any any extras that you've thought of in the interim? I think that one changing the logging level and printing more data out that you wouldn't want to print usually like sensitive data more secrets out there. And we haven't talked about the level of access a get have token is giving us, but I guess that we have another slide on it later on. So we'll cover this later. All right, amazing. So it's, it's, it's bad. It's an interesting start. So we got some slides of how do we have, so how do we protect ourselves from this brand? Environment variables can actually have a scope and one of the features that are not used enough in get have the ability to say a specific environment variable belong only to a specific environment. For example, this secret belongs to production and another secret belongs to them. Scoping those environment variables is enabling us to have environment protection rules, which are not exactly our back, but there are a set of controls that we can enforce. For example, have a set of users that will approve any usage of that environment variable. So let's say that I have on the next slide a pipeline. And within that pipeline, I have the requirement to deploy to production website. My website is called GitHub.com. It could have been ACMI.com or something else. Who is the allowed practitioner that should approve any change to my production environment or a set of practitioners. If I want two people to have their eyes on that last change, that's fine too. I can create an environment protection rule that will say only two people or more are required before setting up a new version of github.com. Let's hit the next one. Another thing that we can do is eliminate the need of putting sensitive data in environment variables and secrets from the first place. In AWS, I've exposed the ability to use roles and assume roles to have ephemeral access of our self-hosted runners. So let's say that I want to create a deployment to my AWS account. I don't have to use AWS access keys. The thing that I can do is I can use a sum role and to have temporal access. And within AWS, I can limit this level of access even if it's admin required for deployment to come only from a specific set of IP addresses. So if I'll configure all of the deployments to come from a trusted IP that can be a static IP within a trusted subnet, I can enforce zero trust on deployment actions that are happening within my environment. And I will not have any access keys in my environment variables because I'm using ephemeral roles. So that's another option to protect your AWS tokens. And you can utilize AWS tools to make those roles list privilege. Hack number two. Hack number two. Yeah. Here's a tracking document used in workflow. All right, I'm going to let you do this one too. All right, let's say that I have an unsecured workflow. I created that but it could have been a legit one. Within my workflow, I'm using a specific image for my job. I'm using an image that called GKWS and genetics. It is a legitimate or looking legitimate image that will create a web server within my integration tests of my build workflow. But on the next slide, we'll see that actually this engine X image is poisoned. It can run a vulnerable Docker downloaded from Docker Hub. And it might be the case where that specific image has a crypto miner. Meaning on every commit on every pipeline trigger on the pipeline that I had on the previous slide, it was on every request. A crypto miner will be executed and communicate with a C2 server. Just because the image that I'm using is not a legit one. Now, this image used to be existing about a year ago. Docker Hub ever moved it from the Docker Hub registry. But let's look on the content that was there. XM rig is a crypto miner that can be executed. And the entry point within that Docker file was just renaming XM rig into engine X. So I thought that I'm going to run a web server and instead of it, I had a crypto miner. And the attacker potentially could have done a lot of money just by submitting full requests. So one of the, in other words, one of the possible attacks is using bad images as the images of the workers in the different steps in a workflow. So not only you need to check for shell injection and do input sanitization should also verify what are the images that are being used on every step of the pipeline. And there's a very good reason why we're not doing this one is because we don't want to start running crypto miners on github's runners. So we don't want to get in trouble. Hack number three, now I'm going to combine hack three and four. So I will be doing three, but I'm going to do it while I'm showing you four. But the idea here is just to introduce you to the concept of branch protection rules so that approvals require a pull request before emerging. And what the defaults are associated with that because by default, if we go over here, my my branch protection is off. Approvals whatsoever. If I come here and I look, I have nothing, which actually opens up an awful lot of freedoms for people who are committing to my repo. So what I will do is I will assign my branch protection to main and just to be secure, I will say require a pull request before emerging and it automatically takes require approvals. Now what's interesting about this is that if we are happy with, we can probably do a whole talk just on all of these, but you can see now allow force pushes is unticked. There's a lot of set very good sensible defaults here. I'll talk a little bit at the end about requiring sign commits, which a lot of people don't do by default and requires a certain amount of extra work. But what I've highlighted back here on the slide is the very teeny tiny awkwardly positioned one there. So that means I only need one approval and okay, sounds good, but at least I'm doing something. The good news is that branch protection rules are great. And occasionally maybe the defaults are actually generally pretty good, but that's one interesting default that is there. And that's what we're going to try and get around. So we've done the right thing. We've got our branch protection on there now. I'm going to say create and we are good to go there. So the theory here is that what if I want to submit a pull request with a new workflow? We talked about doing that, right? And this particular one is just kind of blatantly called approve. And on a pull request, it runs a curl that will actually instruct the GitHub API to approve. Now this is where we're going to talk a little bit about what can the provided GitHub token do. And on its own right now, this would not work because I am not a contributor. I am not a collaborator on this project. I am right now just a casual open source contributor. But because I've been so good and I've been so kind to the repo over during my social engineering phase. So if I ask, I can be a Canada, she's only from there. I can be a collaborator. And they say, yeah, you've done, you've done the issue grooming thing and you fixed a read me file. And so now I now that that actually changes just enough for this GitHub token to have the credentials to execute that that API. The caveat that came up while I was creating this talk about a month ago when I created the first repo like the one I've got here. The default settings for this particular, these GitHub actions under general were like this. And in fact, I even have the original repo here, which is set like this. Now if you create a new repo. The default is that, which is good. Because you can see workflows have repermissions in the repository and only you can't do right. You can't write to the repository. You can't execute and change approve. You can't do it. You have actions cannot create pull requests or approve pull requests. But if you are if you are a GitHub user, go and look at your repos that were created more than a month ago. They will all have these settings now, which weren't there before, but they'll all be like that. And so that's, there's a little important security tip there. If you really care about your repos, maybe go fix that. And so I have set this up intentionally the way it used to be because that's probably how 99.9999% of all repos still are in GitHub. Good. Okay, so act number four, big finish. How's that? We can hear you now. All right. All right, I think my microphone just died. I had to switch. Thank you for letting me know and thank you for in the chat also confirming that it's not just Brack losing his hearing. So the GitHub token expires when the job finishes. This is a very limited amount of time typically so it's generated and it's destroyed. It's a pretty good smart default. If we want to look at a little bit more detail, a job can last up to six hours. Actually, it's kind of a long time. I mean, not that we haven't even internally exceeded that and had to break some of our research. But it means, okay, I've got six hours of life for a single job for like GitHub token to exist. And we know that. And if we want to find out what the default permissions for our workflow token are, this is a table which is from the GitHub documentation. And I always find very confusing because when something says default twice, I don't necessarily know which default I've got. But you can roughly assume that the non collaborator is the middle. And I've just upgraded myself significantly to have a lot more capability with that token by becoming a collaborator. What I also find interesting from the GitHub documentation is that if for some reason the GitHub token isn't doing what you want it to do. So you want to extend the capabilities of the token. Their suggestion is to create a personal access token and then add it as a secret. Now, as we've already discussed, if you saw the way that I export ex ex ex or traded the environment variables, if I could do the same with secrets, then suddenly I would have a potentially significantly persisting. GitHub token that doesn't just vanish after six hours, I have a personal access token that might do some real damage. And so here's the here's the big finish now this is very small so I'm going to zoom in on there and show an example of what I could do in a workflow. The first one is me echoing all of those secrets so that token into a file, and then I send that file to myself just like I did before with the environment variables out to my malicious web web web hook dot site. So now if there is a get up token that is potentially significant or any other secrets at all. I have all of them. The same thing that I did on the issue grooming is I'm going to give myself all the environment variables so I kind of am attempting to get the keys to the kingdom out of this particular this runner. Now here's an example I'm just going to take an aside and something that I found when we were doing the research for this for this project. There are actions in the GitHub marketplace, which then do encourage the creation of a personal access token as you get a token, because they want to enhance the capabilities associated with GitHub. There's nothing wrong with that. The example I found is, there is an organ there was a particular action that was adding to the capabilities of auto merging pull requests. And in order to do this and you can see in honey down letters down there the first step in these GitHub action was create a personal access token with the right to merge pull requests. And as a baddie. I'd love to have that token. That would be fantastic because not only could I approve but I could merge. So that's even better now now I'm just bypassing the entire system and adding whatever code I want to repository. Super duper. So then I went and searched GitHub to find out who uses this auto merge, because then I know that they have a GitHub token that I want, because otherwise they wouldn't be using this this action. I found an organization, I made that name up node core. Sure. That makes other GitHub actions. And they're using the auto merge on their own get have action creation. So I thought, excellent. This is fantastic. If I could do something like create a submittal workflow that allows me to pull that token out. Then I can actually add my own code to those actions. And then I can go find out further on. Well, who uses the node core actions. Lots of people, whatever action is the most prolific throughout the industry. I will then add my bad code to that action and suddenly I'm in one of those bizarre supply chain attacks where I'm adding codes somewhere far back the wing, weak leak in the supply chain. And people who are using that are now somewhere downstream. And it's a little bit like, like almost like a solar winds but I'm manipulating something that doesn't exist because there was an overprivileged GitHub token that I was able to extract. So this, this is a hypothetical scenario. That I may have that does exist in a variety of forms on GitHub. So here's, here's the zoom in on what we were looking at earlier and then you can see I've added a sleep of an hour so for some reason I wanted that GitHub token that I just pulled out to exist. And I wanted an hour to poke around with it. Okay, I can submit something that gives me a bit of a delay. Or in a more targeted and malicious version of this, I could export all of my secrets, all of my environment variables, and export a shell or a reverse shell in this case back to my command and control. And then I'm actually logged in to the runner. So not only do I have all the keys, but I'm actually there and I can start to do the typical pen testing in top 10 and look at the network, look at what processes are running, maybe run an end map and just see what it is I have access to. And if I, as Barack suggested, if I have access to a particular database, I can kind of take my time on this, or I can automate what I want to do. So it can be a teeny tiny bit dangerous. So let's take a look if this is possible. It seems like I'm, you know, pie in the sky here that if this is something that I could actually execute. I'm still here. I'm going to go into my workflows again. And I'm now a collaborator. So I'm going to get out of file. And I'm going to paste in that huge thing you just saw, called big CI to. And it's rather long. So I'm going to get a lot of code and push it just. Okay. So on my pull request. I hope this is not too small, but it's going to come in and it's going to approve the pull request for itself. And then start to run all of my exfiltration and my reverse shell. Kind of complicated. Nothing to see here. It seems good. And I'm going to call it blatantly. I'm going to submit this, but what I first have to do is I have, I'm not waiting for anything, am I? So I'm going to have to do the receiving side of this. Give me one second to run. Execute that. Actually, it might be my history. Oh, my, my internet has kicked me out. If that's the only thing that goes around. Let me pretty happy with that. So now I'm sitting and listening. And once again, it's going to say, yeah, this looks like a perfectly good pull request. You can see the changes are there. It seems happy with it. And I'm creating a pull request again. So you could see a shit. I've only got one thing here. So hopefully that works. It works. It says there's one review required. But you can see it's actually running my workflow. I don't have to say approve and run. It's actually just going to do it. Which is kind of like, kind of crazy, but that's what that's the entire thing that we're trying to take advantage of. Yeah, while it's running, I will say that did have does give us the controls to prevent that from happening. If you're not configuring correctly, it's your responsibility to configure your environment to be secure. You have the tools to prevent it. And we're going to demo more tools. But if you haven't done it, you're, you're making yourself vulnerable to these scenarios. Either misconfiguration or not hardening your environment can lead to that. We have the tools provided by GitHub to fix it. Cool. So we can see the build is started. Looks like my approve. Something was going to go wrong. My approve failed. There's probably something, a permission I missed somehow in my collaborator to auto approve myself. But that's fine. The rest of it was worked just fine. So you can see over here now I have my GitHub token. My environment variables. And a login on the runner. Even though the approve didn't work. That is actually I'm pretty happy with that. And things I can do. We talked about many things I could do. I mean, something that is sort of interesting is I can do a fetch of any other branches that might exist. The branch that was created as part of this. I can go grab, which is loud Canadian patch to so I can go. Check out if I can spell. The thing I was going to do is, is that my branch name? I'm just going to add a file to this. I'm going to add this to here. And I'm going to see if my, if I'm in the right space, I should be able to see if that is there. Okay. That's the, that's the evidence. Let's say, but if I were to remove that, I don't want that there. Get rid of that. And this is one that you showed me Barack, which I thought was fascinating is the ability to just set my name to be whatever I want. And then I can commit those changes back to the branch that is existing as part of the workflow, which is kind of strange. And this is the bar that I hope works because it's just a, for some reason, my branching was being a bit strange there. Oops. And this is where I wanted to commit back to. There's a lot of, there's a lot of risk going on here. It's not the branch wrong. I don't know, Barack, if you remember what my branch was called, I thought it was patch two. It was just called patch two. Is that what I got wrong? Apologies. This is where we're going to have running out of time, isn't it? This is the first time we've run this. I didn't know how long it was going to be. Is that correct? Yes. Can you run git fetch before that? Darn, it's blowing up on me, isn't it? And we are running out of time. Darn. So the thing that we could have done is to delete all of the trails of the actual attack that just happened from within the service by creating another commit hiding all of those changes. So that is exactly what these steps do. Yeah. When they work, if the branches, if we don't rush it and the branch works, you can do that. You can delete the phone, you can add a file, you can add it as somebody else. And when you come back over here, if you can use your imagination, the pull request content will change away from being the workflow content and that will just be an instant read me. That's what that was supposed to be the big finish. This would have been gone and there would have been something very innocent there. So we would have had a reverse shell. We would have covered our tracks. And now that we only have eight minutes left, I'm out of time to actually redo this again, which is very disappointing. But we'll leave you in suspense next time we have to do this talk, it will probably work better. And I should always record my live demos just in case they blow up. But the best practices we are meant to learn from this and I'll put them down here quickly in the interest of time. And we've seen all of them turning our best branch protection not adding permissive get up tokens, using a short lifespan personal access to them if you do go that route and give that token as little privilege as you as it needs. Don't run workflows unless you're 100% sure be careful when you're adding contributors because it could be a social engineering kind of exercise. And use the environment's protectionist Brack told us all about scope secrets to environments. Of course, finally, single sign on multifactor sign commit all the things that we say we're going to do we never get around to. Those are all great advices and I finally scanning all of your reference container images. The final thing we want to add outside of best practices is that our open source tool that we did all the research for is called check off and check off is the most. Let's say most come revised most comprehensive misconfiguration scanner for infrastructures code versus control systems NCI pipelines code just like what we just saw and it looks rejects and get hub get lab and bit bucket more frameworks to come soon. And the other. All my windows are rearranged. An example of that being for example if I was to run check of as simple as this on the example files that I've got here like tell me a secret that YAML. We can see that it runs a series looking for curls with secrets like we're net casting for IP addresses and of course in this one. It found that I was trying to curl out all of my secrets. And check up you can get for free. Check out is here is at checkout.io. You can learn more about it and start using it immediately. And it has integrations with VS code and JetBrains ID ease and it does all sorts of wonderful things in terms of infrastructures codes. So that is our takeaway. And finally, you can add as a get have action. And that can be pretty critical because it can you run it as your first get have action to make sure that the workflows that you're about to approve and run aren't going to be the thing that's malicious. Probably a good idea. Because you can find that all of the all of the things that I did today are preventable via check off and best practices, particularly when they work. We only have five minutes left so the takeaways secure design takes work. Unfortunately, open source tools are there to help misconfiguration to first class problems defaults as usual are not secure and pipeline security can and should be applied at all phases throughout the CI. All of this can be done very, very easily and with the help of additional automation. That's the end I'm sorry that big finish didn't work. That's so frustrating. Right. Thank you everyone if you have any questions now it's good time to put them on chat. Thank y'all yes any questions pop them in the chat and we will get to them with the five minutes we have left. By the way, there was the first time we did that one. Sorry, obviously because it didn't 100% work but we did a lot of damage by getting the reverse shell and except for all of the content so it was good. Okay, I'll did great. Anyone have questions going once. All right. Well, Steven Barack. Thank y'all so much. I know you're at a conference. So thank you very much for taking the time and I hope everyone at your booth enjoyed this too. Excellent. We will see y'all next time and thank you everyone for joining us for a CNCF live webinar. Please remember next week we will not have any online programs due to kubecon cloud native con Europe, and hopefully we will see you all there. We'll see y'all next time. Thank you.