 Okay, hello everyone. The time has come to introduce our next talk. Let me welcome here Riri Konechni, who will be our speaker. If you have any questions for Riri, please use the Q&A section here in Hopin. We'll get to them at the end of the presentation. Thank you. Thank you, thank you a lot. So, hello everyone. Welcome on my talk about the GitHub Actions and the question if they are safe to be used. And I want to point out why I even created the presentation and started to think about it. It's basically that I found the blog post about how to make the Fedora Core OS server used as a GitHub runner, self-hosted runner, without any info about the problems what could be raised by that with the issues. And the security problems, basically. So I told myself that I have to somehow try to enlighten people to avoid doing mistakes we already did and we tried to avoid so to help you with that. So, a little bit about me. I'm the Anaconda team lead. Anaconda OS installer from the Fedora, just in case you don't know the name. And we moved to GitHub Actions more than a year back, I'm not sure if it's serious even. With a great help from Martin Peter. Thanks a lot for that. And from that time we are basically using mainly GitHub Actions for all the automation. We are also using packet and similar stuff but it's more like site projects for us. And we are heavily using GitHub Actions. Also, please bear with me, I'm a bit ill and one of the benefits of the virtual conference is that I won't spread it to you, so I'm using that as a benefit right now. Okay, so, yes. So, a bit about what we will talk here about. Talk here. So, first I will get you familiar with the GitHub Actions just a slight fast look on that because it could be quite complicated but from the base usage it's pretty easy to use and if you are not familiar it will be hard for you to catch up. So, that's the first part but then we will talk about the potential dangers which could be there and then we will the second part or the last part of the presentation will be focused on the self-hosted runners because they are a special category about it. So, what is the GitHub Actions? Yeah, basically it's automation of anything you want to automate. There are definitely restrictions but most of them could be somehow avoided or if you are smart enough you can do interesting things. However, definitely you should not do something like mining cryptocurrency or anything like that. GitHub won't be happy about that and probably blocked your account. So, please don't do similar stuff. But other than that you are able to do basically everything you would like to do with it and I can recommend it because it's really a great way to do a simple automation which is pretty transparent I would say. And so, this is our action step in the Anaconda and it shows you the automation we have there for basically everything. Some of these are not really used now but it's still there because it was used in the past if it's still enabled. And when you see that you can when you click on that that's basically automation of anything I will talk about it later and then you can see that they are basically releases. This one is petting you and it's basically release, automation release from the TAC. We will create a TAC in the repository and it will automatically create a release and it will be automatically consumed by a packet to make the downstream release for Fedora. And similar stuff. You can do really a lot of that. And if you're using the automation there to run the tests on the pull request when there is a contribution to your code then you will get on the button of the page something like this and it will basically show you what is what failed with the tick here what was skipped. These are tests which are not really usable on this pull request. In this case it was because the label it was not set because it's not related. Or there could be a success or expected which means that this is required to be run however it wasn't run yet for some reason. And you can block merging before the above are successful specifically the required ones. The other if you have something which is not required it won't be there until it started and then it could be like skipped like in this case it started find out that it's not really meaningful for this for this run and then it skipped so it doesn't matter the result. Yeah, so most important part of the GitHub actions are workflow files and the workflow files are basically they are publicly accessible on the repository. There is a .github directory with the workflow files in it and they contain it's a YAML code and they contain all the complete description of the automation of the process you want to automate. And as I said, it's an directory and also I would want to it's like a bit ahead but I want to mention that one of the most important one is the GitHub token and basically you can specify that in the you can call them from the automation in a few ways from the bash or from the directly in the YAML this way. You can find everything like that from the documentation here's a just quick look it's a bit complicated workflow but just don't care about that there's a name here are triggers which tells you when it should be started the automation the workflow file basically and then there are jobs which describes what you should do what it will do, sorry so basically in this case there's a refresh containers which will basically build our containers we have for testing in some parallel execution like here and with the configuration here and then there are steps which are one by one information I could, I should probably make it bigger for you to read it sorry about that yeah so basically there are steps and the steps describes the steps the job will be doing and that's it, this will do the checkout of the repository and this will run the bash script basically the run part will run some bash code here etc etc etc it's not that important really going into the details you will find everything in the documentation I will just want to show you what's the part of it so and after this quick look it was a bit understandable we can look on the interesting part of the presentation and that is potential dangers so first thing I would like to show you here is basically as I showed you before there are triggers there could be schedule trigger which is run on some cron job it's basically cron job you will specify when it will be run and it will be run periodically based on the settings on the configuration then you have overflow which is manual you could then just press the button in the action step and it will run but the most important one the code of the project on the contributions are the pull request and pull request target these two triggers will do exactly that they will basically start when there is a change on the pull request or if the new pull request is created but the point is what's the difference because when you read the documentation it's described there but it's kind of complicated I wasn't really that I wasn't able to understand it from the first reading through the documentation so basically if I if I want to explain it in a simple way it's where you will where the code where the job will be started if it's on the contributor side or basically in the target side and if it's in the target side then the contributor couldn't change the content of the workflow basically if the contributor will change the workflow while doing the pull request the pull request target in case of pull request target trigger it will use the workflow from the from the target branch not from the change the one that's the main difference I would say but when you create a pull request trigger it will be used the execution will be on the changed stuff from the contributor and if you have pull request target it will be on opposite from the target branch so and I will have some sentences I would say sentences before that and so in that case if it's that way I will use the pull request target everywhere because I want to have it safer and in this case the contributor couldn't change the workflow file and do the changes based on I want to use commit my parts to the repository so I will change the workflow file yeah wrong nope it doesn't work like that or it does but okay the main difference between these two is that the pull request also taking the token the github token the github token is one thing which is generated always for each execution of the workflow that has to be the token and the token is generated in the case of pull request target the pull request target is generated from the target branch which means from the target repository which means with the privilege of the repository owner basically of the repository and in case of pull request it's generated from the contributor side so it means that if you have a pull request target and you have for example some script or something like that then I can easily use the token which is accessible there in the script and recommend your stuff change the history, anything so please be aware of the pull request target trigger it could be problematic also it has access to your secrets so anyone could read the secrets or upload them to the code by doing the changes in the workflow because basically when you have the pull request target it will by default check out the target branch but in general you don't want to do that because you want to test the code in the pull request so you will probably check out the pull request instead of that the pull request code and then you are getting to this situation so anyone doing any change there do anything basically with your repository almost anything so basically pull request even if it's run like anyone could change the workflow file it's still safer because there's no privilege to change the repository and it's also even much easier to develop I definitely recommend you to use the pull request it's the recommended one from the GitHub and it allows you to make your life simpler and safer however it's also not that simple so ok I will talk about however a bit later but let's talk about something a bit different so a user needs to be able to execute a custom things for the script and basically you have to allow it ok what about using pull request trigger pull request commands and basically run the pull request command and pass the command content to the application you want that's definitely doable you can use the trigger which will execute it on command on the pull request and then take the command content and use it so for something and there could be a situation when you want to do that in our case we have to have this because we wanted to be able to run the kickstart test which is the CI, basically our CI pretty complicated one hard to test locally hard to run locally it's not that hard to run locally but it will consume a lot of time to run and a lot of resources but basically we wanted to make it more configurable so we use this solution because there could be a specification I want to just this test the test type all the tests from the test type from the group of tests etc so we want to have this one so basically it works and we are using it our could be could be also a bit complicated so yeah what I want to show you here is something we had wrong in our first solution however I must say that github is great to like to try to help you with your workflows and they contacted us that we have a possible shell injection there what we did before was that we were just like using this variable information about what was part of the command directly in here and it sounds fine but anyone who worked with the bash and has experience with the bash knows that you can pass basically anything here like for example the quotes and then you can put your code anything you want so it's not great we missed that first but we got an info basically mail from the github security team and they informed us that we have vulnerability in our code and told us to create an environment variable and put it there and it works I'm not sure how really I guess there's some escaping in way or something like that I'm not sure but you are not able if you put it to the environment variable first you are not able to do the shell injection so if you want to know more I would be interested if you find out I was trying to look at the documentation unfortunately I wasn't successful I'm not sure what's happening there but they recommended us the solution I tried it and yep and it works so just in case you want to do it this is the way and it's working pretty nicely also I want to just point out that if you want to trigger workflow on when there is a command yeah this is the way exactly when you want to comment like start the job based on command you will do it this the issue command types created that's all you can also give their like times modified etc but I don't want to have that really so next interesting part so what's the runners they are great really however so reasons why you want to have the self-hosted runners there are few one of that is for example that you don't like you don't have resources accessible so the self-hosted runner could be on your internal network and just connect to the github actions it's the internal stuff by that but one of the benefit for github actions you don't have to have public IP I want to mention that because that's great you don't have to have accessible hosts like self-hosted runners from the outside world you're just connecting to the github actions and registering your host there so it's from this point of view it's better safer than the drawbacks and our main reason to have it is that we had to have access to devkvm socket and unfortunately github runners don't have that and in most situations you won't be able to find that anywhere else it's really rare to find a place where it's like especially if it's free and I have access to devkvm please tell me it would be great to use machines like that or have access to machines like that because basically I asked a few people why it's such a problem to have access to devkvm and the response was that it's potentially dangerous because the devkvm is not really user-spaced I hope I'm not saying it wrong and you are able to get control or there's some possibility that you will be able to access the other VMs on the machine and in case when you have VMs as probably github has then it will be like attacking from project to another one by this so yeah, I don't know and the drawbacks, definitely you have to take care of your machines and they should be up to date even though it's not accessible from the internet it's still contributors are running the code there so they could somehow use the tools there to do mess on your system it's much much easier it's much easier to shoot yourself into the heavy when you are using the self-hosted runners then if you are not using them and using the github and the simple thing about this, basically you have more much bigger attack surface by having self-hosted runner because it's your runner, your responsibility if it's github it's their responsibility I would say and if you have it on internet network then yeah, it could be actually you should have really keep an eye open to any potential vulnerability or anything yes, exactly your responsibility versus github Microsoft responsibility, yeah you can point a finger so and of course biggest concern here is the security if you are on the network of the internet network or anything like that as I said before and I want to mention by a bold letters that github recommends to avoid using self-hosted runners on public repositories we are doing exactly that because we are basically forced to because of the fkvm if you want to have the github actions but yeah, they don't recommend it and yes, recommends it doesn't mean that they will like disable your self-hosted runners or they will start to telling you that I don't know you break it yourself or something like that they are trying, they are trying to really improve the situation about this but still it's yeah, it could be a problem there is a link for the documentation in the slides if you are interested about the recommendation so, potential engineer I don't have a problem with a self-hosted runner for when I am running it just for one specific workflow which is not triggered even by the pull request what could have wrong because I have something like, I would say schedule task, schedule trigger workflow with the schedule trigger and the trigger will once a day do a build of my project in the internet work for example so what could have wrong separate the task, it's not really accessible from outside or executable from the outside so that's fine whatever nope and I give there a facepalm facepalm picture because basically I feel like that when I found it out and this is the, I would say most important part of the presentation please, if you are interested about the purpose of the runners, keep your attention here because this is something you can not shoot yourself to the feet, but to the head so, yeah so demo time first, I will show you a bit about my demos just to quickly explain what I am doing here basically I don't want to have the internal running runner or something like that purpose I've created just a simple docker file which will download the GitHub github stuff to connect to the container as the service to the runner and then I will launch it that's it and you have to have a token with public repo permissions that's everything and so I'll set it here so that's it and open out because I first presented this presentation just in case if you were wondering so and this is my first demo and basically I have always two branches and one is the attack branch and second is my branch which is the base for the attack and I want to say that I'm creating the pull request from basically from my repository to my repository and it's kind of how it is done because the token is always the same however believe me that I tried it also from the other repository so you have to believe that I'm not telling you lies here but I tried it and it's still the same issue as far as I'm aware of so basically this is just a simple okay let's do it I don't know we have just a simple workflow file that's my main repository and the contributor will do contributions to this one and it's just like running a half on one to host that's it no problem with that so I will I will I will create a pull request okay I have to go there there's a nope there has a nice button to create a pull request but whatever we will do it this way so basically I want this one this one I have a few here but just ignore it and what the attacker do will just change once it should run you want to host to the local runner and the local runner is what I want to put into the repository so I will switch to the terminal and access and set up my local runner to the repository also I have it configured that it's started just for one and then it will die so and create a pull request okay there's the button there's the button it hits me it should work okay am I blind because it should be button and why it's not but I'm seeing it seems like I'm blind but okay whatever there's already created one so try using instead of okay I guess it could be because I already have it created so I will try to close it sorry about this and I will try to create it again so hopefully it will allow me to do the pull request now here the button I don't know what's happening whatever so I'm creating the pull request with the changes I described before you can see the changes here and here is our new job sorry the noise from before just ignore it and as you can see it was started on my local runner and yeah that's the point if you have at least one local runner accessed like configured for your github repository anyone could create a pull request pull request not pull request target really pull request use the pull request job or create a new one if I'm not mistaken I'm not sure about the new one for 100% could change the to use the workflow to use your local runner your self hosted runner and it will work and everything with that of course I can change the workflow anyway so I can read anything on your internal network by then and it's really face palm I don't know how to otherwise say it and I really don't understand why github still wasn't able to solve this but yeah if you have at least one runner on your network or on your repository be aware that you can't have opened pull request target because this could this could be pull request target sorry pull request triggers because this could be easily used against you I think it's not working for the new ones created ones that we tested it and just for an info here are also the permissions with that but it's not really that meaningful here because as I said before this is a pull request so this will be permissions from the token from the token of the contributor so these are permissions on his repository and not on your one so not really that problematic part but this is really face palm sorry when I see it I always wonder how it's possible that it's still there so I will just clean it up a bit so this was the demo and how to avoid it so first what we did was to change all the pull request triggers to pull request target triggers and it's annoying and you don't want to have it because of the previous issue as I described already and yeah but you could get into the situation like that and it's a way out okay so then I can describe it so basically we will use a pull request target everywhere again what could be wrong another demo so I will show you and that's basically what I already explained to you that's a simple workflow okay I will make it maybe readable so I have a workflow and the workflow has my script which will basically run any script and then there's my script as an edge which is just doing nothing interesting really so I will join my self-hosted banner in the background pay the pull request again and the most important part of course there was pull request target trigger so it will use like from the from the attack branch and this is exactly what I also told you before basically that you are able to use the GitHub token and to read all the environments from that and thanks to the GitHub token you can then use the GitHub token to basically do anything in the repository not really anything because there are some permission restrictions but these are pretty pretty benevolent and for example you can commit you can force push you can create issues if I'm not mistaken so yeah you can do pretty much of the problematic stuff so I will create a pull request so here is it we will go to checks here's the new one wait a while okay and here's our job and here you can see that the job the script was executed and because I had there an N command you can see all the environment variables in the runner and just for information there are stars for the GitHub token because GitHub is trying to trying to avoid having in locks something which could be problematic so when they detect your token in the locks they will make stars but it's really in the lock like we are seeing here but you are able to use that from the script no problem with that so everything is there and you can do basically anything from the demo so how to avoid having these and I want to say that some of these and I will say it directly so basically you can restrict permissions for each workflow file and the permissions are great and they were added later after we started to use the GitHub actions first we didn't it wasn't there basically you will set just the contents read and if you will do anything from the pull request like if you have the pull request target trigger for example and you have this there the GitHub token will have privilege just to read the content of the repository which is publicly accessible it can't commit it cannot push it cannot change the pull request it cannot do basically anything else and for most of the use cases you are you are pretty much fine just with this but it's not the default however you can change the default on your repository in the settings I'm not sure exactly where and it will be harder to find to find it right now because they changed the UI for me probably but it will change it by the default and it could make like I don't know we did not do that because it could break our existing workflows and we would rather like to change the permissions for each new workflow than switching the default to read only because it doesn't really it has some drawbacks I'm not sure exactly which one right now but there are some drawbacks with that feel free to you will be able to find it if you are interested about it and the ultimate solution I would say is to request a confirmation from every external contributor for every external contributor so basically you have to gate you will gate external contributions and someone with the right access to the repository have to approve the run and there are two solutions the old solution was some like I would say handmade and we still have it because this repository doesn't have much contributors from the outside and it just works pretty fine I would say so basically it's just simple not simple but you are using you are using github API to get the permissions of the owner alike of the of the action in this type who created the comment you will basically find the owner who created the comment and get the permissions then the permissions are checked if it has like if it contains admin on right so if it has access right access or admin access it will be set to that it's allowed and this specific user could do like could do the approval and then it's put basically to we have two jobs for that and the second one requires the first one and the output is then stored to allowed user so it will first check if the allowed user if the user is allowed if not this job is not even started so yeah this is the way and yeah sorry big of correction we have it here because this is the issue comment which means that basically when you write a comment this will be this is the way how to check user permission the new solution they have doesn't work for the issue comments it works for pull request only if I'm not mistaken and it's basically that github edit add new functionality to require approval for all outside collaborators and this is like ultimate solution I would really I would really recommend you to have this if you have at least one self-assisted run or you you are just like you have a you are worrying about getting getting having security issue this is the ultimate solution for each first push or each pull request or repository there's a button on the bottom of the basically next to the test above the tests which is the approve run for people who have the right access so yeah and this is something pretty new they edit I don't know if it's not a year ago or something like that less than a year ago if I'm not mistaken and we basically changed pretty much based on that like simplified our structure thanks to that and they first started with require approval for the first time contributor that was the first step they did and then they like also include for each for all the outside contributors the first time there was mainly to avoid crypto mining because basically with this without this and this is the default for the repositories right now for new repositories without this you are able to just go to the github and create the pull request for egdapp which will mine your stuff because it will be it's pretty simple basically will change existing workflow or edit new one I think I'm not sure about adding new one it's automatically execution or not and then you will be able to use their power, machine power to that yeah and as I said they are actively trying to improve the situation I must say that github is doing really nice nice steps to improve this and we are users of I would say I don't know how many but yeah we are heavy users of their new functionality they are adding especially about the security and the best solution of all this is the end of my presentation basically the best solution of all I can recommend it is this do not use self hosted runners if you don't have to have them it's really a complication for you it has a nice benefits but also yeah you can point the finger as I said before it's much easier to like it's a problem of github I don't care they have to solve it then oh again there is a security breach because of my self hosted runners yep and that's it from my presentation I hope you liked it and we can continue to the questions and answers thank you very much for your presentation we have a question by Jan have you tried ephemeral self hosted runners in fresh VMs would that help ephemeral self hosted runners yeah I wonder hmm I wonder what exactly do you mean by that because ephemeral I know that ephemeral is a way how to run basically the self hosted runner not self hosted yeah you can probably connect it just once and when you have the ephemeral parameter for the configuration it will die after the one job execution that's what I am using in my example but on a fresh VM if I understand it correctly it will be just one time machine it will be definitely beneficial we did that from the beginning on the level of containers which are which was there for just one just one job and then died but in general it doesn't help you really because it's still about you are still accessible and if the user will be there like 10 minutes or 15 minutes or I don't know what will be your time out for that you have still 10 or 15 minutes to do a mess on your network, internal network read all the resources or add there some back gate or something like that so I don't think this is really a solution to that and we avoid using it after some time because it was really a complication for us no benefit thank you the next question is from Lukas the outside collaborators approval looks interesting is it possible to specify groups that can permit certain runners to prevent other company employees to allow runs on our company runner that can permit certain runners I don't think it is possible right now I think it's like everyone who has the right access can start these runners but maybe there's also a question because some feature functionalities for the GitHub enterprise specific and that could be there because I know some LDAP changes etc but I'm not aware of I never was in there so I can tell but from the basic configuration or the GitHub configuration on the repository if I know it's not possible you can ask for the feature thank you and our last question from Yaroslav do you write a blog or any articles about these problems yeah I want to do that but I hate writing maybe I will force myself I hope I will force myself but I want to promise that I think let's all hope Yeji will force himself to write blog posts I think it would be very beneficial anyway thank you for the presentation we don't have any more questions it was very interesting thank you to the audience for attending if you'd like Yeji agreed to be available on the work adventure after so you can catch him there ask him questions talk thank you a lot thank you a lot for attending so thank you again very much goodbye