 All right, I'm gonna go. I don't know if they do intros in this room. I'll just do both Hi Welcome to Security con club. Kubcon. This is my first time ever talking at one of these. So what could go wrong, right? I got a lot of moving parts. I'm trying to warm this room up for probably better things to come No Okay, you say so I Have already learned that walking around on this stage may not be the safest thing I should do This is pulling the sea eye and I'll just drive straight to it The story behind this I was asked to research this because this happened to our open source project check off We got a pull request one day from this guy And I'm gonna just highlight that because there's a little Return to that earlier because this this talk. It's more interesting as time goes on because weird stuff keeps happening to us You can see there's a name up there says am young 34. Just remember that by the way of em young 34 is in this room I'm sorry Because this is not going to reflect positively. So this is what the pull request was it was delete everything useful in our build YAML and Replace it with a PS and an M on our self-hosted runner So obviously we didn't accept this PR almost did But that would have been bad because our self-hosted runner has also some information on it both in terms of environment Very well see what processes are running. These are all bad things So I was then our CTO Bragg said can you just like dive into this? How bad could it have been for us? So I did and that's this is kind of the first thing of course was I had to break the news to him that generally running self-hosted runners on open source project is considered bad But of course if you look out in the world see how many people actually use self-hosted runners on public projects All right, let's just forget what best practices are and just go with the actual reality of what's really happening And I have a link down there You don't take photos of links at the very end I have one one slide that has every link that I use so if you just want to go click and Look around then you can do that This is the obligatory me slide where I inflate my ego The cloud name security advocate work for bridge crew bridge crew is bought by Prisma cloud Which is Palo Alto networks. It's like it's I don't know. I don't know who I work for somebody I've been running code since the 19 And doing a lot of security thingies if you're ever in London, England Ping me and come to our meetup We run a meetup in London called DevSecOps London Gathering every month have a lot of wonderful guests on there And if you ever want to kill an hour on a Friday around three three o'clock There's a show on twitch called C9K. It used to be called something quite profane turns out YouTube doesn't like that So we had to change the name There's a perfectly innocent QR code go ahead and scan that so Get a bench and workflows hands up if you use get a back from workflows great So I can speed through this so I do a lot of grabbing the the actual text from github because their own help I Found kind of takes you down the wrong path sometimes But this is just what it says it says you create workflows to do What three things build your application deploy your application or do just stop like add a label to an issue You can sanitize an issue, etc. I'm gonna be doing some of these things live. So if it all blows up in my face That's what it's gonna do today The fundamental problem generally is that if you submit a pull request like this person tried to do that is in the github workflows path It will try and execute the workflow It doesn't know the difference between a pre-existing workflow and one that you're sending in just to commit Which is weird the other thing is that a Lot of the creation of workflows is generally done by people who weren't app sec professionals And they don't quite have the mentality of sanitizing your inputs yet So you've got things like issue name issue description The list of potential inputs is actually pretty big in particular if you use things like workflow run triggers Where you're feeding one workflow into another then it really complicates what is considered input our user controlled input so a lot of people aren't doing that and there's a lot of Things that can happen as a result of that and the last one of course that workflow directory It doesn't have special permissions. You can add permissions to it, but by default Anything anybody can submit anything to it. So that's not great So as a result of that there's a few things that can happen across all of these So if you're not sanitizing inputs, you can do some kind of command injection into there If I know that you're using something to build or even deploy Then I'm assuming that you might have some access to credentials on your self-hosted runner So what can I do to try and achieve? acquiring those credentials and Execute some form of supply chain attack. So this is this is what I was challenged with What I found kind of confusing down to the box is Some of the reactions GitHub has when you are a new contributor like the first one up there Let's just do this. This is exactly what happened to us and actually if we Go here you can see I've got two people here. I've got this malicious person called loud Canadian And then I have This nice person over here called Euro gig who's just trying to maintain This wonderful application and I've submitted a pull request and sure enough I have my approval run and this is what happened. We just didn't click this Which actually I am gonna click now because I don't mind just to kind of push things along I'm gonna merge this pull request and confirm the marriage fabulous Awesome, so now this person is a wonderful contributor So what happens in GitHub is you don't get these instructions on the top. You just weren't there this is from the help and That does warn you that if the contribution is in the github workflows directory be alert Fantastic what could go wrong if you have no workflows whatsoever It kind of scolds you and says hey, why don't you have workflows come on man? You can you you could use this to catch bugs and enforce style workflows are a good thing for security Okay The default for outside collaborators is that you only require approval for the first thing you do Which I just did Once you've done that You're an insider awesome So it's far easier to become a malicious insider just by changing a read me or going to the good first issues and Doing something really simple and then you've bypassed this whole workflow conundrum Sounds a little too good to be true So I have just submitted something here's the first thing I'm gonna play with now Which is unsanitized inputs to workflows Here's an a and this is me looking around for how do people do this in the real world So I look just look for get-up event issue title. I found this one, which was pretty cool Somebody's creating generating blog pages based on issues that are submitted That's kind of neat. So what if I create an issue with a title? That has a bunch of HTML in it that might fish for information when someone goes to that block There isn't any real auto sanitization of what the issue title can be That's kind of up to us The other more common one and this was like there are thousands of these just people logging issues or PR titles or anything That's like an input user controlled input as they were arriving so that there's more visibility again sounds like a good thing Then when I just submitted a moment ago here We can take a look at the files changed is that one That is designed to look like a positive contribution I'm checking the input give hut event issue title to make sure it formats to a certain way and I look like I'm contributing to the project in a positive way Which is great, and that's it there It's great unless If I create an issue and you probably would think surely get-up won't let you create an issue With this weird title surely right there would be something that would stop that from happening so This is where you get to see all the little funny things that I've got going on here. So this is the this is the good area I've already merged this pull request. I go to my code I've got workflows here, and I've got my issue check Which is cool. It means anytime an issue is opened. It should run this it should live in my actions world So what happens I? Go to my issues. So there is my baddie I would I'd probably get my friend to do this for me because I wouldn't do it myself, right? That would be too obvious. I'm a hacker And I'm going to go to my ridiculously tiny notes here I'm gonna grab exactly what that said. I'm gonna use a back tick. Everybody knows what a back tick does Fabulous Because the even if I see equals or I assign the value or do anything and I get a workflow it happens on a batch script Baskrip Baskrip seeds a back tick and it's like CloudFormation sees a dollar squiggle It tries to resolve the meaning of it before it does the assignment So I haven't really screwed this up. So if I create this issue issue Seemed happy with it I go there and look It's hello I'll go back to my web folks site And so I now I have all the environment variables from that particular runner By the way, if you ever use webhook.site, it's awesome just for testing things. It's also good for messing with people, but It was kind of that easy. So that was like the first thing ever was like, all right So I know these sites that are doing any kind of input sanitization I can now just vacuum out all of their environment variables in a much easier way than what was attempted on us So that was interesting thing number one There's ways to protect yourself from this like there are environmental protection rules and environmental secret rules That already exists. There's a link down the bottom. So if suddenly you're watching that and you're going Oh, maybe I should only give certain workflows certain access to certain environment variables and secrets You should don't think we'll be at the end. It looks as simple as Environment name the environment etc. So it's the really really simple things to implement Generally, I found that you should really never have like hard-coded credentials Even if you think they're stored as secrets, you'll find out later why about that But it's not a good thing short-lived tokens are important If you use self-hosted runners, it can be an advantage because you can create lists that Make sure that you're only certain IPs can talk to certain IPs so that self-hosted runners can be an advantage And AWS reporting is important open ID connect Get up actions does use OIDC. It doesn't use spiffy spire. Unfortunately yet but if you can Make sure you use that to access any kind of cloud provider definitely do that Instructions specific or Amazon are down there and also just generic ones for github.com So when this happened it was really fascinating because our own repo wasn't as secure as we really thought it should be number two I'm not going to show this one, but I just find it fascinating. Oh, maybe I will show later So this brought me to branch protection rules. You should always have your branch protection turned on What was amazing about when checkoff first started we were a startup and we didn't know how popular it was going to get So it must have a thousand stars before we had branch protection turned on it seems crazy But we were and we were inviting collaborators like anybody help help and that when that turns into a SaaS solution You become a startup and you're like, maybe we got to go back to the original effort that we could have quickly sped ahead on If you turn on branch protection You should almost have everything ticked nobody ever does if you do tick by default the first thing require a pull request for emerging Which you should you require approvals which you should a lot of people think they're done but the default down there of one is kind of where it gets a little dicey because You can approve yourself So over here this changed in April This year this year The default this is what you should have for workflow permissions. This is the the good thing read Repository context permission in a workflow. This is what the github token should be able to do That's not the default though The default is that There was about a day or a week or something where the default was the correct default for new repos and Everybody freaked out because they weren't expecting that to be the default and so they went back to that So just to know that's there So if you create a new repo the github token if you will have a way more access than you think It's the difference between these that middle column This is read only and that column there, which you probably is as far from least privileges you can imagine so I hope it's not too small I'll show it again later, but To do a curl within a workflow. I was surprised to see and by the way if anybody from cider security is here This is their research not mine You have everything you need to do a curl command you've got the github repository You got the github event number, which is the pull request. You've got the secret github token Like you can just assemble a command you can't approve a pull request So you can't merge a pull request But you can certainly do that and you can approve your own thing And so that whole thing you did to have approvals is suddenly irrelevant That's not all you can do. There's a lot of things. This is just getting your imagination going All right, so the last one fingers crossed it works This is the one that I tried to submit because I thought this is the how bad could it be section? So zooming in on there I Thought right so I can get the environment variables How can I get the secrets? I got to get the secrets in two lines by doing echo secrets to a file because the secrets are only available in the workflow They're not environment variables, but unless I push them out to a file And then curl that out to the same bad URL much case now. I have all the secrets. They're not encrypted. They're just there And then I want the github token I'm like, well, how long can the github token be alive for the github token dies the moment the job dies So I took a sleep in there and I get a github token as well and the secrets and the environment is cool And then I was kind of feeling nasty. So I did this version Which I thought all right. Well, what if I could do a reverse shell and Then send that out to myself and then that would make everything persist For as long as I wanted So I thought yeah, I'll do that one Let's do that one So there's a few things. I still have this sitting here. I still have this Which is cool What I'm gonna do now and we're gonna make myself a contributor. I'm gonna do the all the things you shouldn't do This is a bad example of how to be secure I'm gonna do a collaborator. I'm gonna add me so I can commit direct I've been such a good Person in this project that there's no reason why I wouldn't add myself I just got an email telling me I'm at it. Yeah also That's a main. I'm going to turn this on There we go. Now I should have a lot of this stuff turned on but I don't know why anybody would turn on They do not allow bypassing. Why is that even there? There's some weird stuff in branch protection Nice, so I'm all set up there. I've got code I've got this running. Oh Yeah We've got that running Okay, it seems like it should work. I've got everything going. I'm gonna go to code And my workflows and So now I'm not in my own fork anymore. I'm actually right here. I'm gonna add this file create new file. I If you were gonna add something dodgy don't call it poem.yml That's that's a dead giveaway. That's a little tip for you budding hackers out there There is my file now the difference being this one then what I just showed you is this actually has the auto proven it as well Just for kicks up here. That's you probably need a telescope to read that but Here we go Okay So I have my little auto proven there which should In theory work. I should be able to propose this new file, which is fine I'm gonna create a pull request merge back. Oh, I can't read it No one second one second I Sorry about this little There we go So I have my github token now that worked and I have a reverse shell Which is not ideal and given the amount of time we have on this It's unlikely to work Because I'd got my naming wrong But there is another step here where here's the path. I'll leave it up there. There's there's a step there where I can start to change what I what I just did via the runner thus I can cancel the pull request Delete what I've submitted and change the actual person who did it because I used an unsigned commit and so I So it's the things I can do while here are actually pretty terrible I Realize I'm a little short on time because by the way, this is normally a 50 minute talk The other thing I noticed on most runners is that Docker is always installed by default So while I got access to the runner as the ID runner you can use Docker just to Create an auto pop out shell and upgrade yourself to root So now I'm a root on the runner as well. I'm kind of there as long as I want Unless course I use that family may see the Falco talk. I thought that was broke So they should create rules to stop people from doing stuff like this so that's kind of like The idea behind why this is all bad the idea of getting a Be able to run things in shells is one thing But to be able to destroy evidence of and be able to have full access to the repository that I just took over Make changes to pretty much anything in the code changed the adjacent code if it's a shared self hosted runner and even Turn off the pull request remotely so that I can stay on the self hosted runner independent of github now So what can you do with this? If you're in if you're taking part in the CTF Pretty much all the things that you wanted to do in the CTF. I can start looking for AWS Information that might be stolen by my variables I can change the logging level on the runner that I'm on right now I could get it to report back to me I can I can add my own SSH token so I can come back directly to the runner with all of this staff I can look at the processes the network I can look for access to s3 buckets or so much I can do Now the next thing I want to say is Important information I've saying don't be a biff you recognize this from Back to the future. Don't be a biff with this information The thing that started all this this is a change a month ago This crossed my path in my mass network of information that comes to me by a Newsletters and whatever this came in and I was like I was a carpenter. I'm using stuff was getting run into Facebook That's my talks about that's pretty cool. I'm gonna read this and I hadn't even scrolled down yet and I went M young That's interesting That's the guy I looked at what was happening here He says Facebook actually was pytorch, which is Facebook, which is now Linux foundation And he goes through all of this. He's kind of bragging about everything that he did all the way through And really it turned out all he was was after money He's calling it a bug bounty. There was no bug bounty, but he just went through and was like using this thing But I realized we were part of the whole Scatter gun approach to trying to find people who had misconfigured github or Insecurely considered github, and I'm like you could have really messed with us like if we'd click that that would have been really bad for our own Because check off is what powers our commercial solution and that would have been not cool and so My little comment on that is that I Don't think it's pretty like I love bug bounties. I think they're a great idea They make us more secure, but if you know how to do things like this, don't do it on open source projects that are a gift to the world like pytorch Do it on actual bug bounties. That just I just thought the behavior was terrible The good thing about his article is that because he was actually doing this to real people He got observations that I couldn't get He did find out that many self-hosted runners did have access to s3 buckets He did find out that many had access to ECR without ECR as immutable tagging So he'd change the images while maintaining of the tag thus supply chain attack Lot of self-hosted runners were internal and we're not on a VPC So the efforts like I don't like what he did, but I like the conclusions that he got so I added it to my presentation All right, that's a prevention I'm just gonna blast through these gonna be a five minutes Be careful when adding contributors Even if you're a new startup, it's bad because people like me might be added to your I mean, I'm just terrible programmer. So that's bad either way Self-hosted runners and public repositories are being targeted. I mean Branch protection is important more than one reviewer, but there's lots of things inside branch protection You really should read every single one and make sure you understand what they are Don't run workflows unless you're a hundred percent sure environmental protection Short lifespan tokens of any kind but preferably open ID connected to connecting to the cloud Get hub event logging and codoners codoners is the thing I mentioned where you can add a codoners file Put someone's ID in it add it to your get hub directory And then that's the person that can review it doesn't end up just being any old person or yourself via curl and Then the last one which I didn't show you and I'm so sorry, but I'm I'm running out of time sign commits That there was a bit in that script that just had me sitting my get config to an email address And it was Linus Torval's email address if you do that try it on your own repository set it up yourself Use his email address and do a commit Get hub goes and finds his avatar and will pop it in as the author of the commit You can be anybody you want with get unless You're not so you don't have to be you. It's way too easy To be somebody else in this world Awesome tools you should be using if you're not already Open as a scorecard anybody using that? Yes, it rocks It'll check so many of the things that I just talked about branch protection Contributors from at least two different organizations which I thought that was funny because I did that in order to begin a bad contributor But I really like the avoid dangerous coding patterns. I did a lot of playing with that to see what it would find and it was actually pretty good Declare get a I mean that's not everything it does It does awesome all sorts of awesome things just checking the security of the repository and the way it is tested in general That's a definite use it The other one I would be remiss if I didn't plug the own project I work on But the reason we did this was to add rules to check off that look for the sort of same sort of things So we've got strange patterns that look for Uses of curl inside a workflow anything that's doing what I did with secrets All the different possible reverse shell things like anything at all like that is in there And plus we have some that we have some overlap with the open SSF As well because we do scan the APIs and we look at branch protection being enabled and a lot of some of things like that So there's a bit of a dead Venn diagram there But using both certainly would get you a long way to stopping me doing anything like I just did there And that is the end and these are all the links I used so if you're curious to read you can go for it Question yes There'll be eventually a get lab of it addition of this bit bucket did not But I hated bit bucket So there we go That's the end any other questions. I have three minutes. That's a you have a three-minute sign. That's an odd number to choose Cool all-star everybody get that It's like open SSF, but like huge Yeah, same makes sense Awesome. Thanks for watching that if that's it's Great. Oh wait again. I mean thank you for asking questions Yeah, I don't know Good question. I'm looking at it ask me next year when I do the get lab version cool thumbs up. All right. Thanks