 Welcome everyone and I'm really excited and happy to be here and I'm going to be sharing some of the lessons that we've learned at f5 as we've tried to like secure and manage our github repositories and also be sharing some of the work that we've contributed as part of the open source security foundation in that effort and my name is Christine Abernathy and I lead the open source program office or the ospa at f5 and this talk might be of interest to you if you are also working in an ospa in some capacity or perhaps you're an open source project maintainer and you're trying to figure out how you can actually better improve your security posture so I don't know about you but you know sometimes in my life as a roommate you find all kinds of folks you kind of those people who like just leave my mess alone because I know where to find things but when you're thinking about things like github management there are certain things that you want to kind of get under control you can go from unmanaged all the way to control but when you are actually in a state where things are a little chaotic it might be a lot harder to not only just discover things from an external community but also from the inside if you're actually trying to manage your projects that could be a challenge and in addition to that this is something that you cannot just wish away so and where does like when you're thinking about security if you can't even find the things this is a stat that if you sort of like been around or paying attention in a lot of these different talks you might have seen something like this from a sonotype report saying that cyber attacks are on the rise and it's not like slowing down so this is something that you want to take a look at because as you think about the supply chain this might be something that you is a part of what you are thinking about especially if you are working in an ospa so that part of the supply chain actually work goes through source code and this might be the responsibility of an ospa and it means that beyond like just solving problems about observability of people discoverability the ospa should be aware of what's in your get up portfolio and how those things are being run and so as a young ospa at f5 we're just like about two years old this was one of the things that we started working and thinking through what is in our get up or what should we be thinking about it as we looked at some of the challenges and we were building our initial open source strategy is actually pretty clear that get up orgs I mean it's kind of like makes sense because when you come from like different acquisition and different product groups and there is no centralized place to manage things are just going to go all over the place it's a given so the good news is that a lot of before we got there as part of the ospa a lot of these orgs had been actually put together under one get up enterprise account so at least we knew what we were working from other than at them being all over the place but you know there were definitely still some challenges there were we found that there was basically there was really no consistent policy around what people should be should be doing in the different orgs everyone might be doing things differently no standard process for how we release projects people might do things differently in one office as the other but there was nothing really that was standardized across so everyone's kind of doing things their own way and you know if no one knows how repos are being managed or what's going on people sometimes they want to get their code out there they're just going to find the path of least resistance and and get something out there and that could be problematic whether it's just like leakage of IP or some things that hasn't gone through the proper approval process because some orgs are really good about going through the approval process but someone not and then you got like operations teams who are basically tasked with provisioning repos and orgs at times but sometimes they didn't know what they should and should not be doing from an open source context and even as you look if you're an hospital and you're actually looking at GitHub you want to set up a new org there are all these controls and knobs and and you're like what should I be doing which should I turn on which checkbox should I be using so it can get quite complicated and then we also knew that as another challenge that would come in the future is if you are actually going to be adding and enforcing and doing these guidelines there are some people who are already on the org and if you are going to diminish or revoke access that should or should not have been there you could probably get some people a little mad and so you wanted to make sure you didn't slow down existing folks who are working if they needed to continue working while making sure that things were being managed in a secure fashion so it's quickly obvious that we needed to establish kind of like a consistent policy and a consistent approach to not just compliance security but just governance and how you work in the community so the rest of this talk is basically just going to focus on many of the security aspects and seeing how we can we can kind of all fit in so it's not a secret to many people that source code is actually pretty important and if you're not paying attention to how you configure or how you're managing your github organizations and your repos and all and your members you're probably going to elevated security risk and this is from a supply chain levels for software artifacts it's just like a nice diagram and there was a talk yesterday from folks from google that actually kind of like walked through in the context of figuring how you can work scorecard they walk through the different pieces of like how is a typical open source project put together and where do the dependencies come in but if you have a project you're usually an open source project you're going to create an open source project you're typically not going to build everything your own there's probably going to be dependencies that come in and then eventually the person who actually consumes it at the end is probably going to consume it through a package manager so along this offer supply chain there's going to be like various places where threats can come in and attack vectors can come in and we're going to look at a few of those in the context of why it's important so the salsa framework it just actually divides things nicely into what you could consider source threats what could be like dependency threats and then what could be like build related threats and we're just going to talk about a few of them but when it comes to like source threats it could be just somebody just unauthorized change that might come in or you know compromise the source report itself is compromised in various ways or the build is actually happening from something that's modified modified source and they've got like a lot and source itself it's got a good website it's got like a lot of different lists of things that you could be paying attention to but in the case of an example like source integrity is it what actually the producer of the open source software intended or did something happen and sometimes intended could be a loaded word based on and you'll see an example of this but I'm just going to look at just a couple of examples in here somebody got grabbed a little pot of gold from an ethereum and basically they were able to send in a change request and that change request accordingly or supposedly wasn't actually properly reviewed and one of the key things is if you're actually got any code out there you probably want to have good review and through that effort they're able to actually get in and chain make some changes that eventually led to some cryptic wallets actually being stolen there's an also another attack where the developer in this case intentionally actually introduced code into their own repo and this downstream actually affected a lot of different applications so there's a and these are popular packages of these can come in so there are ways that you can actually mitigate this there's various things that on a high level you can actually go in and kind of like tighten your access control who has actually access to the github repo it doesn't mean that it's going to solve a lot of these things but that's one of the key concerns who do you have two factor authentication luckily thankfully github is going to make that mandatory so we won't have to worry about that in the future in addition to that for the people who are actually in the organization what kind of like access do they have what are the based on their roles should they actually even have those access so you can go in and get home and lock down those privileges and you're thinking about like code security itself there are some things that you need to be mindful of code reviews should happen and if they're not happening that's actually a key concern and even if it's happening are you protecting your branches so that's kind of like one of the key things in there so if you're going to have code reviews you obviously there's a lot of open source projects out there that a single maintainer so that could be like off concern who you're going to bring in from the community that you trust who's actually going to be doing a good code review and in addition to that github also has features out there that can help you do things like detect for secrets it can actually help you allow people to responsibly actually report vulnerabilities in your in your code and you want to do this in a responsible way so they have the ability to do things like private reporting but there's various other things that you can do as well so there's ways that you can actually manage this another thing that I want to touch on is that dependency threats so dependency threats this is where nobody's going to again build codes from scratch you're obviously going to take in packages into it so there could be like examples where dependency are compromised and I touched on one in the past but here's a couple more that you've seen probably I've heard of like event stream is a popular npm package and what happened is that through social engineering the maintainer actually somebody gained the trust of the maintainer and it's handed over like the control of the repo and then through that introduced some some malicious code which then affected a lot of folks downstream so you're bringing in or taking in a malicious package and then another example where there's like this is where the npm credentials were hacked and account takeover what happened so somebody then published a malicious package in ua pasta which is another popular library and so these things that happen so you as a maintainer who's actually taking in these projects what are you going to do about this how are you going to actually mitigate for some of these things well in terms of github you can actually configure things both in an organization level at github and also at a repository level but the things that you can do to help with that is this like you can actually scan for dependency update or even just look for any vulnerabilities so you have the ability if you configure github to actually turn on these these things so you could then get notified via email when these vulnerabilities occur but you can also have pull requests that could be automatically open on your behalf and then you as a maintainer who may not have that much time can just go in and accept it it's not always easy to update dependencies especially if that becomes like a breaking change but at least you can at least know what's going on so there are actually also within the open source security foundation there are the working groups that are thinking about a lot of these things and how do you do security incident response team how do you think about that what are some of the package managers thinking together says like a securing software repositories working group I believe it's called so they're thinking a lot about how is packet managers they can holistically think about helping and making sure that this office supply chain is a lot easier than things like supply chain integrity they think about things like the salsa framework s2c2f and other these things there was another really good talk panel discussion a couple of days ago that dove into into that topic as well and identify security threats so these are all open source security foundation groups that are thinking about these topics so hopefully I've convinced you at least at a very quick level because there are a lot of talks that go into detail about this why you should be thinking about github configuration and standardizing your org so on the f5 side when we started we had a vision what we wanted to do is we wanted to make sure that in the future that we'd be following industry best practices around how you do source code management so I'll talk about why that was a little tricky to find out first we also wanted to find what are the best practice we want to make sure that any of our new projects and any of our strategic projects were actually going to be actually adhering to these best practices that we found out and we do wanted to believe that we had the tools and the processes in place that that people could actually remain compliant I wanted all of our new projects to be following these new streamline processes and then make sure that they were actually going to be like that we were just like going to be setting them up for success and then also a flip side even though it's not really about security but also help better support the community so as we started off we actually just focused on spent a lot of time on the requirements phase just gathering information and making sure that we're able to plan for this it was not really like a linear path so we at some point we did like proof of concepts and tried it out in a few organization and the other thing that we had the opportunity to share some of our learnings with the open ssf new project that kind of kicked off so I'll go through each of these phases and chat about them so when it comes to like just defining we kind of created a guide document kind of like an initial like not a you must do it but you know you should or just like a guideline document just our listing all of the different configurations that we think every new organization existing organization repositories and maintainers should follow and we tested it through a couple of initial like repos just to kind of prove out what we were thinking was correct with a couple of maintainers we then run an audit and some of our key organizations just to see how we were doing just like as a baseline and then we also created like a core small working group to just look through the the guide documents and see how we were doing and as I said we share some of our work with the open ssf I'll just kind of talk a little bit briefly about some of these so in the guide document like I said we we found like a nice little handle to a call legitify so they did a five is also have like an open source project and you'll get a chance to see it in action in a little bit but it's able to recommend some of the things that you should be doing when you're thinking about like the security posture of your of your whole so-called get-of-assets your orgs your repos and your members so we ran that tool in and that actually helped us kind of get like a quick baseline of a document and then after that based on that we tested it out on a brand new organization that we had at f5 with a project called you know this which is a data visualization library so through that we just work with the maintainer to see how is this working for you with these brand protection rules because there's like a lot of check boxes that you can take how does it work for you if you're like two maintainers one maintainer three maintenance what is it so we walk walk through that and then alongside that with legitify we also used scorecard so and scorecard just allows you again to run get kind of like a score it's like a weighted average of of how are you doing on all these different like about 19 checks that are in there and again you'll see it in in action in a little bit so through that we able to get a document together we ran through some metrics to see how we were doing as a baseline and then after that we then got community help to kind of like help us push this through so we had like a bunch of stakeholders anginex is f5 is part of f5 and anginex is big open source project and they're one of our key stakeholders so working closely with them they were already on the path of doing some of this in parallel as well but working hand in hand with them to start to prove out what should the document look like and then we also found other people through an internal program that f5 has called the f5 open source ambassador program these are like open source enthusiasts within f5 who want to promote the culture a lot of them came on board to help us go through the document make sure that we were doing the right thing um we always found like new orgs and people would come out of the woodwork where we would find like oh there's yet another org and they're kind of doing things differently and we need to think about their needs because it's still valid and so that was an interesting exercises and then also there's like some open source projects that are commercially supported and then some that are not and then you have to consider some of those needs as we thought about things like how do you do security reporting and again we just basically at some point if you've ever had a project your requirements can go on and on and on and so we needed a way to just like scope this down and make sure we kind of got things going and then really like at the same time we were kind of kicking off this project Legidify and Nome had come in and they wanted to use some of their work to create a source code management best practices guide so this like perfect storm we're able to work together where we gave them feedback from f5 and they just took that and we together work together to actually create a guide document so we're able to actually collaborate and share our work so for those who don't know what the open SSF is and we've got some folks in the front row who are experts in that so they can keep me honest but this is under the Linux foundation a couple of years old and they are thinking about a holistic solution of how you think about securing our open source software and it's not just a one person's responsibility it's it takes a lot of people the industry the open source maintainers and they are thinking about this day and night there is a group within that organized in working groups and special interest groups and there's one called like the best practices for open source developer as a scroll who sits in the front says the best working group so the best working group they are got a lot of projects under the umbrella one which is the scorecard and then another one they also put out a whole bunch of guides to for example how do you actually better either evaluate the open source software that you're consuming or actually if you're creating open source software how can you do this better so it naturally fit this this working group to work on this source code management best practices guide so we jumped in as I said we gave him feedback and then we so this is kind of like the it was like it formally launched last week and what it is is you can go in and actually look at a long list of things that you should or should not be doing as grouped under what you should be thinking about if you're an enterprise an organization level and a repo level a lot of different things that if you click into the links it'll give you the remediation steps but alongside that they also have a tool that you can run Legitify and scorecard and there's examples of things that you should be looking at which can then guide you into first of all seeing what's going on and then go from there so I'm going to switch and show you some of these things in action so you get a better intuition and a sense of how this all works together so let's see and I'm not going to do full demo mode because that that would that would take yeah yeah okay but I actually spent some time a couple of days ago with a sample project that I put in I'll just quickly show you is this is this viewable from the back okay so this is like a test organization that I created called demo gods rule and it's a quickly show you that I haven't actually configured too much in terms of this organization and if you go into GitHub there are things that you can do around member privileges like for example in here if you're like an Ospo there's certain things that you should or should not be doing in general like don't just let anybody like create a public repo especially if it's an organization that you care about your flagship so there's certain things they can do like set the base permissions to for example none or no permissions as a baseline and then nobody can do anything and if I'm there then you can start like just like incrementally like turning on so that maybe if somebody wants to create a repo they actually have to come talk to you maybe you want to do that maybe you don't but if you do that then you know for example don't allow members to create public repos there's a bunch of different things and to my point earlier there's a lot of knobs and things that you like what should I be turning on or not there are things that you can do around like planning and automation what is some of like the defaults you can turn on security things related to code security and scanning and some things that you can do at an organization level and some things that you can do at a repo level so I'm going to show you like for example I took I created a repo under DG rule a very simple repo it's a JavaScript project that all it does is that you can actually it's just basically testing continuous integration as mindless as it can be so starting off with this project um all you can do is um very little things but at very least it had god bless us we actually had a license file and a code of conduct and uh that was about it and uh dependencies in the package very limited dependencies nothing kind of major and then I just kind of like took it through his paces I was kind of curious how is this with this very basic thing how is it going to do against say uh scorecard and um actually before that I want to actually like show how you could run scorecard so running scorecard is pretty it's pretty easy so you and in mac you just like install it through brew or you can actually run it from a command line and you can run it through an action but if you want to run scorecard this is in the repo that I just showed you it'll go in and it'll run like a bunch of different checks like branch protection is there a license are you actually trying to get a best practices badge which is another project under the open ssf do you have code review are you doing code reviews do you have like some dangerous stuff tokens going on and I'll just spit out like an example here and this is this project after we kind of like fixed it so if you look at this you'll see there's some like waiting and some things are more like more important than others like it'll see if you've got any binary artifacts check 10 you can see it's called 8 out of 10 on branch protection and this is after we've kind of made some changes are there more than one contributors coming from more than one organizations so doing pretty well so with scorecard anything sort of above five six seven is considered like pretty good as somebody there was like a really good again a talk yesterday that dove into scorecard and went through how do you incrementally improved your score so that's what scorecard run would look like let's see what the jidify would look like so the jidify again you can install it through homebrew and in the mac case and then you can run it on certain orgs you can run it on oh by the way you also have to like set some kind of like a token it has certain permission but then you can run it like on an organization on a member on a repository on actions and they'll check certain things so here I'm running it against the whole org and I'm looking for organization member and repository information then it's going to give me some information and again it's going to have severity around it as well you can actually check let me scroll up here and so for example the default member permission should be restricted so I showed you in the beginning that members can do just about anything so this gives you like a hint of what you should be doing and if you go to our the guide it's actually going to step in and say exactly what you should be thinking about doing and where you should go on github and or in gitlab and and do that as well it says two factor authentication should be enforced and it's a very high so you want to take care of the highs and the medium and a lot of really good useful information well the jidify can actually also run scorecard if you give it a certain flag as well but a lot of these things kind of overlap with some of the things that scorecard is also checking so with that in mind between scorecard I kind of did the exercise within this particular test repo of how do I go through and increase the score score so here scorecard actually had it at a 7.1 that's where we ended up but where we started was actually at a three something just above a three so this is the initial report at the commit of at the commit where we started off and it's got a few things in here it's got actually a get-of-action that runs that does a very quick check to see if everything's going well using a job a testing framework called a jest and it's just doing some quickly tests so that idea is behind that you if you wanted to improve on this repo you could then add you could add something and then you could also add a test into this just tracing framework we don't need to dive into that the key thing is in here is that when you first run it I'm going to show you some of the results and just if in the interest of time we're just going to scroll through and just show you a couple of things that we got from the scorecard run so from the first scorecard run let's see we scored a 3.5 in the very early repo and if you scroll through this brand protection was not on so it was a 0 out of 10 code reviews hadn't been done they found two unreviewed comments because I was the lone maintainer at the time and then it had no dangerous workflows that couldn't find any dangerous workflow patterns the dependency update tool wasn't there at least the license was pin dependencies no security policies so a good I a good flow through what is and is not is not working well so after that next step in the process was okay I said you know what let's turn on brand protection so brand protection you can do this at the repo level you can also enforce it later through rule sets and I'll show that in a little bit so just turned on brand protection on the main branch and said I will want to require a pull request so we got our reviews require approvals and here you can have a drop down of how many approvals you require you can go up the more people to it's better than dismiss stale pull requests with new commits are pushed there's a couple of things that I should be good and what I did is I first started a few checked a few boxes run scorecard and it incrementally moved to carry move and then at some point I'm like okay I've got it as good as I want it to be so these are kind of like the the items that we that were good in there and then other thing that you can do is that you want to make sure you if you have like some tests you just require static checks to pass before merging and then you can go in here and figure out what should be the test that you want to require and this corresponds to in the example I had the just test so that if somebody tries to push in a code it's going to not actually it's going to block any merge requests and then things like don't allow deletions don't allow bypassing as some of those sections so this was the brand protection that we added on to that and then from then we're able to improve the scorecard slightly from there and we improved it to let's see let's see yeah it improved from about 3.6 to just slightly better than that let me go up to the top right so I got way too much text yeah so improved to 4.6 after those changes and then the next change is look through this again and another really easy thing to do is add a dependency update so went ahead and it was a zero out of ten in this case so went ahead and and did that and so for dependencies you can add at least with github you can do this different ways but dependabot is a good one to do you can say in there that I want to be notified if I need to do any updates it might just like bump up the versions of for example you've got any packages in there so it's actually good to turn this on and see the automatic prs come in it can be a little annoying if you're a if you're a developer and you get these dependabots all the time but it can actually pretty much save you so after running it through dependabot actually actually after turning that on was able to get the scorecard improved slightly to a no that's 3.5 still though we're going in the wrong direction so 4.6 and then it moved to a 6.2 but it moved to a 6.2 not just did I not just turn on dependabot but I actually added in a security file as well so a security md again you need a way for people to actually report if there are any vulnerabilities and played around with this scorecard is not perfect but at least he was able to detect that if I just said security and I said we want to do security it actually checks a little bit of the text to see if there's an email for a way to get in touch with people or there's a link for further information and that's why we were able to bump it up to a 6.2 so after doing that next thing I wanted to look at is it also said that there was something called a dangerously dangerous workflow that tokens had actually permissions that were not good and that can be pretty bad so somewhere in here it says token permissions are too high so so here a zero out of ten it detected github workflow tokens with excessive permissions and so for my github actions I just added this line here permissions equal to read and that means because by default at least maybe in that time this token if there was like a right permission it could go in and do things that you don't want it to do so just adding this one line here which was pretty easy to do was able to bump it up to improve it just a little bit more and then we're able to go to all the way to 7.1 after making that change so that was that's kind of like a very quick it didn't take too long quick and easy way to just like bump it up and so at that point the next thing I did is that you can with scorecard actually add it as an action so it runs with every commit so adding it it's pretty easy you can go to the site and send it out there but you add you create a secret a secret token you add it to your repo on your org and then you put it up there and then at every point you can actually get a badge so at the end of result what you got is a test project you've got the just test part passing but you also got like a the scorecard badge out there which is like a good thing if people come and look at your project they can come and see that now when you're thinking about doing this at at scale oh the other thing that I did at least this is an example what we did but it created like these are scripts that could go and run these legitimify or scorecard and do it bring give us Jason because at the end of the day I want to make sure we can put it somewhere where people can look at it so I had a bunch of scripts I would put together so some of the information used GitHub API and GraphQL to get some of the insights that could be useful get the legitimify information in JSON and you can do that as well you can also get scorecard on each on each of the different things so these are different things that we did created a separate script just to be able to do all of this in one place because at the end of the day we might want to merge this into one UI I want to touch on a few things that I wanted to do in terms of if you're doing that on an Asperd scale I'm not sure if you want to go and check every box and do this branch protection as a new open SSF day someone from Intel talked about how they had 94 organizations and 900 900 plus repos to deal with not sure you want to do this case by case so a couple of things that you can do if you not already doing it you could actually create you could create like a template repositories that have for example all the things you want like security md and in this particular case also added a workflow in there just for testing purposes that also had the dependent bot in there by default and also had the scorecard action by default so every new project could potentially have a scorecard badge in the beginning and then the other thing is that for things that are important like branch protection GitHub allows you if you have an enterprise to set up rule sets and an organization level and then you could set it up but you could also do it in such a way that it's it's only in evaluation mode so you can just see who is actually which of your repos are actually got branch protection on before you actually try to enforce it so that's a good way to do things that scale at least from the the github point of view so that's kind of like a quick section so you kind of have like a sense of what it looks like so I'm going to go back to the presentation and continue so so so now that we kind of like gone through requirements we're in the implementation phase we've completed review with at least the initial stakeholders and we want to expand this across to different different teams as well and see if we can set this up for our our initial repos and our strategic orgs and then eventually after that we are planning to do things where we train some of the maintainers on how we can do we can actually make sure that we are staying compliant with some of the things that we've set up and as we do that we're going to refine based on this initial phase what we want to do for the next phase when we expand it across want to make sure that we are compliant across our different um orgs as well and I'm always in favor of building automation and getting out of the way as much as you can and making sure people can do things safely it's what are the automated tools that we can build what are the dashboards that we can put out there to incentivize people to actually uh do the right thing and we also want to do a yearly review there are other tools that we're looking at as well chaos is a community that thinks about open source metrics and there are some things that we can learn from there in terms of how do you work in the community or what are the different things that you would want to show in this dashboard repo status is just uh this really related to how do you signal to the community that a repo through a badge uh what state your repo is in and repo linter is another tool within the to do group that can check on community files like license and some of the other things if you wanted to kind of like dive in deeply into the the actual files themselves and look at it deeply and those are some of the things that we want to use to gait before we release a project and then hopefully we can get as many of our strategic projects with like our best practices badge on on their page uh and because just going through the exercise of best practices can teach you a lot i've got a couple of projects going through them and they're actually learning a lot by oh i need a ci things like that um the scm guide is continuing to work uh working on this even after the launch you want to hear from the community what are you thinking about what we put out what are the use cases that can show us how we can do better uh different tools that we can look at because not everything works perfectly but there could be other tools out there that we could look at and if there are additional platforms beyond just get lab and get get hub want to see what is the criteria from including them in there quickly closing out with some of the lessons that we've learned just got a lot of invaluable help whether it was with the open ssf or some of the to do group who would bounce ideas off you want to learn from the community and share in the community even as we were building this um it was just so useful to be talking to other people who are experts in this field and in terms of as we were going through the process um because there's so many people involved um in this you want to involve the right stakeholders at the very right time and i mentioned before get to hear from different people who are using github in different ways and here here exactly you might initially say i don't really agree with how you're doing it but it might actually be completely valid you got like demo projects only orgs you've got and you probably want to apply different types of security policies to each so just getting to know what people are doing is really important and anything that you're doing in terms of like the guidelines and documents see if you can support it with the tooling not all tools are perfect but you know what there's if we kind of like work together i think over time we'll we'll have like a a better way to think about all these things and also plan for automation and scale um like i said whether you're using a template repository or actually using rule sets think about if you're in the hospital and you're managing so many this what can you be thinking about hopefully some of the things that i've learned i've shared are going to be things that you can actually take and apply it to your own use case because definitely you want to make sure that as a community you are also doing your own part in this you open as a staff um there are there are lots of talks uh and resources that you can actually dive in to learn more about that we'd love to welcome you into the source code uh management guide project um because uh for example we need a lot more people who know have have actually git lab so a lot of us are github github github but we definitely want people there are a lot of projects out there on git lab if you are an hospital that is using git lab please join us and just give us your experience and help us prove out what we've actually put out there we meet every other thursday next meeting is october fifth um so definitely um i'm really grateful to come here and be able to share what we've learned we we hope that uh we can all together just like make open source a lot more secure and i know it's not just f5 that is going to be doing this it's the collective effort and hopefully this has been useful for you and uh thank you so much for your time