 Okay. Hello, everyone. My name is Alex Elgyve and today's topic will be GitHub action security landscape And I'll present you in my latest research really interesting research regarding a security regarding get abactions And so let's get into agenda So first we're gonna explore what is GitHub actions and why is it a really powerful CICD system? Then we'll get to into what kind of misconfigurations the get abactions could have and how an attacker could leverage this one into code execution Then we'll try to understand what could be the consequences of of this code Code execution by exploring get abaction internals will get some several interesting demos there And then finally will list the possible mitigations for all these issues will present So I'm ex Alex Elgyve. I'm a senior security researcher at Psycode Previously I were a malware research team leader at checkpoint. I would reverse engineering a complex crime where NAP team malware's I love a CTF. You can follow me Twitter. So reach out to me whatever you want and So let's talk about the github and github actions And github for all of his history was all about storing source code about two two to three years ago They went into a bit of different but similar direction and the unleashed github action Which is a way to automate customize and execute the software development workflows right from the repository And github is extremely popular as you all know it has over 73 million developers To have more than 200 million repositories and github actions also became extremely popular Mainly to its very rich marketplace a containing more than 12,000 custom actions and almost any Environment has action that can support workflows inside the inside github actions and So what do we need the github actions the most Simple Workflow that you could build in github actions is to take our code for example go lang or python or see or Doesn't matter to build that code package it as a container and push it to our a chosen a registry Could be docker hub jfrog or whatever you want There is additional scenarios that we could use github actions for example creating schedule tasks that could scan vulnerabilities in code Could be running tests for a pull request Labeling issues it could be Sending the issue that created in github to a ticket handling system like a JIRA Monday asana and etc So how a github action workflow looks like first of all, it's a YAML file and Contains free parts contains a name in this example. It's a github action demo Contains a triggered event in this example. It's a push event This means that whenever we push a new code into the repository This workflow will be triggered and then contains several jobs and each jobs contains steps In this example, we have a single Step that just echoes hello world and if we take this YAML file put it in In the repository to commit the repository in that git slash workflows path That's it that whenever we push a new code into the repository the github action service will trigger this workflow for us simple as that So to understand the vulnerabilities and the issues we found in it or send some of the core mechanics of github actions and how it works The entity that Executes the jobs and the workflows itself called github runner and this is an open-source project We are maintained by github that just fetches the jobs from the github a github action service and just execute them This runner could be run either on a github hosted machine Which is the more a popular method to run workflows or it could also be run on a self-hosted machine If it it's run on a github hosted machine, it also run as an ephemeral environment This means that each job will be will be run on a completely clean VM isolated machine And whenever the job's over that machine will be completely deleted Yeah, that happens For each workflow run a new temporary github token will be created this token Is used inside the workflow to invoke various github API requests and we'll soon they see what is that token and why it why it's created So what is github token the until it was Introduced by a github actions the popular method to invoke github API was using personal access token But the problem the personal access token has a few as two main drawbacks It is tied to the user Permission set and it's it's not granular enough to be To permit it to specific repositories So it is the first drawback and the second drawback is that it has a really long lifetime So if this token the personal access token is leaked So it's not it's not good that it's a very long lifetime and could it could be leaked So github introduced github token And this is the preferred method by github to use to invoke github API inside the workflows And this token as a default the right permissions for most of the triggered event And his it has permission only for the repository from where it was the workflow was invoked from and It's valid only for the duration of the action or up to 24 hours the most And it used also as a default parameter for many For many external action used in the that workflow And as I said also the preferred method they to invoke github API functionalities Also another security mitigation github introduced is that a forked request to repositories It can will receive for public repositories will receive at most read permissions for that github token or else It will be a serious security issue So another Core mechanic of github actions is secrets In order to create a meaningful workflows in our in our project We need to store various secrets and invoke a request to to other resources for example, okay We want to invoke it to cloud API or to a various artifact registries and And so These secrets could be divided into three groups First secrets defined on our organization level These secrets will be exposed to all the workflows in order repositories for that the organization Also that each secrets could be also limited to specific a private repositories or specific repositories An additional level of exposure is secrets that defined on the repository level These secrets will be exposed to all the workflows for that specific repository and And the third one is secrets defined in a positive environment, which we'll talk later in the mitigation part So now that we have some background on github on github actions, let's start Let's present the issues we found and how to handle it We have a simple a workflow that we created That will It will be a triggered whenever a new issue is created. I Probably everyone knows issue on github. They're used to track their bugs or maybe feature requests and so on So we can create a new Workflow that will be triggered whenever a new issue is created and this workflow contains the one job and will run this Bash script whenever it's run this script checks that the issue title contains the word bug for example And if it the title contains the word bug it will invoke github API and update the The issue label with a new label that called bug It's a simple workflow that could be maybe could be used to help maintainers to triage the issues for example And this workflow we can see this double curly brackets This is another mechanism of github actions that helps the workflow developers To receive inputs from the user or from the github action service In our example we receiving the issue title We're saving the a the github token secret Which is which you are using to invoke the github API and we're also receiving the issue URL Which is the URL API to invoke and to add this label? on the first site this workflow looks quite innocent and And looks quite okay, but actually it's not and it's vulnerable to code injection The problem is that we are not sanitizing and not verifying this issue title Which is user controlled any user could insert? They could inject code into that title actually you can manage to to run code on our workflow for example If we create such title It has a Disquitation marks that ends the if in that workflow and actually then we're just inserting our code for example APT install figlet and figlet psychode which will result in printing this pretty psychotext We can see that I will manage to a execute code on the build server of that action using using user input for that for the title pretty cool But is it a bug overfeature? Actually github acknowledge it and it's even in the Documentations the github says that when you create workflows you should always consider whether a code must might execute untrusted input from the attackers But the real question is whether the maintainers of the project and the owner project owners are aware of these issues so we went into into small journey and and There is a really cool tool that called the github code search It's in still in the preview phase. I suggest you all to check it out. It's works really great and we wanted to to see how many a public workflows could be a susceptible to this vulnerable Injection so inserted several keywords for example, it could be the issue title the issue body or a several other user-controlled input we inserted the the run command and and We were surprised to see there are many many Workflows that are vulnerable to this to this Injection So we could really safely said say that it's quite a widespread issue So out of all this we found thousands of repositories that have was were vulnerable to this A code injection out of it doesn't deposit or is the most popular one Were liquid base, which is a really popular database a schema changing tool We found it in Dynamo, which is a project that's sponsored by Autodesk. It's a visual programming tool Founder which is transactual a database wire, which is an open source communication platform Astro, which is a static side builder and many many more a Liquid based on its own claims that it has been downloaded over over 75 million times So we can could safely say that These vulnerabilities could impact maybe millions of potential victims So what could be the consequences of this build compromise? first Soon we see how it's possible, but we could expose the secrets previously We talked what its secrets and how it works. We could expose this the secrets could contain sensitive assets like AWS tokens GCP Azure tokens Docker half JFrog and and so on We could use the exposed the github tokens to commit back to the repository and maybe to introduce some Botnet and which will be deployed to end user organization environment, which could be could result in a supply chain attack And a much smaller risk would be an ability to run and might be botnets or a crypto miners on the runner infrastructure So let's see how from technical perspective. How can we actually expose these secrets? How an attacker could do that? We'll take an example which will follow us through the slides So this example Will be the we'll build a workflow that will be triggered whenever a new issue is created just like previously It will define Environment variable which is called github token which received the github token value will soon see why we'll define that And it will contain three steps. The first one will be running a checkout command this syntax is The action to check out the whoever not familiar with github actions It's with our calling an external action. It's called checkout, which is basically doing a git clone to the code and The second command would just will be printing the issue title and description and the third command similar to previously Will invoke a github API Together with an additional secret that is the bot token an additional secret which we which will define and with Update this issue with with some label as previously as we've seen previously This workflow is vulnerable We are not sanitizing the title and the body of the issue and a potential attacker could insert malicious code there so to Explore the vulnerabilities to understand the how can we how can how can we exploit it? We created a simple lab environment instead of tweaking the Workflows and seeing the results. We wanted let's create a reversal from the from the build environment right into our Rental into our lab computer for that. We use the popular tool the n grock We just say run it with the random port for example 10,000 We received from n grock the end point which will use in our exploit Then we just run net cut to listen on the port 10,000 and then we will send the malicious payload to the To the issue to create a reversal. So let's sum it up. We Finally if we create this issue in github in to that specific vulnerable workflow It will result in a creating a reversal that we will we control from our Lab computer and see the entire infrastructure of github actions So let's get back to the example the first method which is very simple method to Exfiltrate secrets will be by listing all the environment variables in this example We defined a github token as one of the environment variables and simply by listing all of them We could get this variable and use it for a malicious purposes The second scenario will be to use the the checkout command the action checkout command as As I said previously this action This checkout command is doing git clone But it also behind the scenes it sends the github token as a default parameter to the external action This github token this default parameter is used as an authorization token for the git clone So in addition if when we are using some token for git clone We're also creating that git slash config file in that directory where we clone the code and This file also contains the authorization parameters so as an attacker if we If you check out command was used we could take this file Grab the authorization line pipe it through a base 64 decoding and we get the github token just as previously the third method that we found is a bit more complex and During our reconnaissance of the runner infrastructure We found that for each run command with two such commands for each run command Previously before it's been executed It also is saved as a shell file in the runner infrastructure in that specific runner temp directory so for example For this specific workflow we have a see a single shell file That contains the exact content of this Run command, but instead of the placeholders of the issue titan issue body It will contain the real value that github action services inserted there So as an attacker we could get this script and if there contains some Some secrets or some sensitive data we could we could get to termalicious per Usages for example in the second round command with this bot token that we could use it use it in X filtrated Unfortunately, we have only code execution capabilities on this point on the first run command We didn't receive yet the second shell file So a simple solution as an attacker could be to put some maybe agent or persistent script on that machine and wait For the second command to to come and be written to to that directory and whenever he's there just exfiltrated as me immediately So we could do the following We could create some script that records all the modified shell file in that directory and whenever And we could pack the script as a docker container to ease the deployment Just run it in some detach mode and whenever a New file will be written there a new shell file. It will be Exfiltrated immediately and we'll soon see that as a demo so this was the the simple methods attacker could apply a more advanced method are Possible as well which out of the scope of this talk It's could be inspecting the memory layout of the of the runner processes it could be exfiltrating them their environment variables it could be a Recording a network traffic and extracting sensitive information from it and more So let's see several demos the first one will show how we are able to Exfiltrate the secrets we'll do that in two phases And the first phase will send The simple github token from the environment available and for the second phase We'll put that persistent script on the machine We wait for the for the second a command for second second run command and we'll just get the entire script to our lab environment So let's see that in action First we set up in our server We have here we have Demo repository second And we're sending the malicious issue. Let's hope it works This issue will contains the several commands. The first one will exfiltrate the github token through the From the environment variable and then we'll set up some docker container You could see we got the first one, which is the github token to our server exfiltrated and Then we got the second one The second secret You could notice that we have the complete shell file the complete run command that we had in the workflow But instead of the bot token with the revealed secret Which an attacker could use for his malicious purposes This was the first demo For the second demo we show how we're able to commit a malicious code to the repository Using these secrets For that we created a simple Simple bash script. Nothing fancy there. It just It gets a file to commit and the path where to commit that file. It does some git configuration He finally just git commit git push really plain simple and on the runner side we We are we're fetching this script giving it parameters And we're just giving the proper permission and run it Let's see that We're running the malicious issue You could see the repository has a single readme file and workflow in in this in this issue As I previously said, it's first fetching the the malicious the script Giving it the proper permission and just running it giving it a parameter the file that want to commit and the the path of the file Let's see the repository now We have a new addition of five in the repository. It's called malicious file that was committed 19 seconds ago in addition Also, we are controlling the commit message So this is our control data the the commit message and the maintainer name We can put it or whatever you want if a game commit signing isn't enforced So this was the second demo the third demo is a bit more complex Up at this point we show how we are able to exfiltrate secrets That we found in a specific vulnerable workflow in that specific vulnerable workflow But there are additional secrets that could be defined in either on the repository on the organization That aren't using that specific workflow So we showed how we are able to commit to the repository So why not we use this commit ability the right ability to commit an additional workflow to the repository Which will exfiltrate all possible secrets of that is that repository is a is can can access to So that's what we do exactly We created we will commit a new workflow That will take all the secrets that This workflow is as access to Write it to the file and just take this file and send it to our server In addition we'll do some cool trick that this workflow will also delete itself. So we won't leave any traces But we have one problem how we will invoke this workflow how we make it to run We could maybe invoke the github api But github deny the ability to invoke workflows within other workflows Probably to deny some circular invocation of workflows But we thought about the other trick This trick trick is called workflow run We created the workflow and told him to run after And another workflow is finished. So we told him to run after the another workflow that's called Vuln will finish which is the Vulnerable workflow which we injected in the first place So the flow would be we injecting code to the original workflow That code will create a new Workflow and when the original workflow will be finished github action service Will trigger this automatically according to this workflow run trigger And on the runner side on the runner command, we just We will invoke a simple curl to github api together with the With the github token as an as a token to a to commit this workflow We will add the message and the committer and the content will be base 64 encoding of this entire workflow So we will take all these parameters invoke Invoke api to the contents api with the path Where we want to commit this file So let's see that So we are committing this. We're adding this malicious issue We couldn't add this to the title because it was too long. So add it to the body But it does exactly what we seen in the in the slides is Is calling a curl command to github api together with all the parameters Together with the with the commit message the committer the the content the base 64 content And in few seconds, I hope we'll see something Yeah, we can see we got an additional message We have the bot token which is the bot token which we've seen previously We got it From the repository secrets We got the github token For that specific workflow and in the she and in an addition we got two additional secrets We got secret one and secret two Which could be maybe a secret that defined the repository organization that weren't in that specific workflow And the and it also could be really sensitive Sensitive assets of that specific repository So that's it for the demos Let's see how can we mitigate these issues The first mitigation will be to avoid avoid run steps For example, instead of issuing a Curl to github api to update A label of for example that issue We can use an external action. For example, the laborer laborer external action Which does exactly the same it receives The parameter for what label you want to add it Or you also send the github token to the to the external action It does exactly the same and it's not it's not susceptible to a code ejection This mitigation it's not always possible There's cases where you can't run from doing from using a bash script, but whenever you can it's very recommended The second mitigation will be sanitizing your input. This is maybe the most effective way to To stop this specific attack. This means that instead of using a Our example like issue title issue body inside the run command The preferred method will be to define them outside as intermediate environment variable For example as title and description environment variable And then use this environment variable inside the inside the run command Even if this in the title description will contain A code it won't be run It will be treated as text because it's sanitized as an environment variable Another effective method for post exploitation Will be to limit token permissions Inside the workflow inside the yaml file. We can add this permission tag Which will which we that are defining The permission that the github token will have during that build So in our example we use the github token either to clone the code and we used to update the issues So this is the only set of permissions we need for that token So we prefer method will be to add these permissions in each workflow to to limit the github token permissions So even if Attackers manage to to get a code execution on your build server You won't be able to do whatever you want because this permission will be limited Another mitigation will be to limit limit secret exposure which is more irrelevant To the third demo that I showed that there are organization secrets Each organization secrets you can define to what level of exposure you want that secrets to be either To all the repositories in the organization, maybe only private ones or selected ones And the last mitigation would be to use environments and branch protection This is a quite new mitigation that available only on github enterprise You can you can define Environments for your github action workflows. For example Production testing staging in each such environment will contain its own private secret But the But you can use these environments also to apply some several security mitigations Like branch protections and other various mitigations. So it's it's quite an effective method So that's it. So what will be the takeaways from this this talk? First of all, even when github does most of the security For you for the github actions, I mean It gives you a vm isolated the machines and ephemeral environments and many other security features Your builds pipeline still could be compromised. So this may be the most important takeaway from this code this talk As we've seen in the best practices paper Github actions a platform delegates to the developer their responsibility from some of the security features in the workflows I'm a developer myself and I I know I know developers do make mistakes. So it should be handled carefully The third one is the consequences of a build compromise could be quite disastrous It could they could result in exposing sensitive secrets it could be a Resulting in a critical supply chain incidents and could be quite dangerous And the last one is the specific github gives you the all the right set of tools to secure your workflows And it's really securing a pipeline isn't a matter of faith And you should you have the right tools to protect and we encourage you to do that And that's it. Thank you. You're welcome to check the full blog post in a psychote blog a full technical details You're welcome to reach out of me a twitter whenever you want And if we have time I can take a few questions Yeah, I've been doing that and they mostly They're written in node the external action and they're quite more secure than using that So yeah, I would look in that and it was really quite harder to find the external action that's vulnerable than this Best script maybe louder. Yeah And I mean, yeah, you could get you asking you asked if if I found any ways for a nano maintainer to execute code on a Or non maintainer Yeah, but what do you mean outside injections? Other than other methods than injections to for non maintainer to execute Not that I know of I was researching this specific attack. So maybe there are more that we're not aware of. So I don't know Yeah, automator scanners could help to remediate it. I mean they could like same grip. For example It's a great tool that could do that. It could find like a this specific vulnerable action and could Same grip Yeah, I didn't look specifically at scorecard, but there's other tools that do that. So I probably scorecard do that as well Sorry My specific demo it wasn't but it's possible to do that also with a git force push and Unless the first push was disabled by a branch protection, which can be quite complex But if it's not it's possible also to alter the git history and to So it will disappear. Well, thank you