 Thank you everybody for coming today. I hope you enjoy the event as much as I do. It is very beautiful venue So without further ado, I do My name is Michael Barbero. I work at the Eclipse Foundation. I'm the head of security today I will present you a tool that we've developed to manage hundreds of GitHub organizations that we have to manage on a daily basis and thousands of Git repositories. So for those of you who don't know about the Eclipse Foundation You may have heard about the Eclipse ID, our original project But we are actually the home of many many projects, more than 420 projects We're the home of Demarin, JD, Kapella, OpenADX, Mosquito in the IoT world, Keeper, and Quora What you may recognize some of those Projects and what we provide to those projects is a set of services based services for open source So we provide an infrastructure so that they can build around their build or their source code and so on We provide marketing ecosystem development provide IP management and licensing Tooling and we also now provide help in improving the security posture of the supply chain and One of our original route in why we are Why we are very well known off is of very strong governance and these very strong governance Tells us a lot about how we should manage our Organization at GitHub for instance and our repositories and so so keep in mind that we have a very strong governance I will repeat that a couple of time during the presentation and The reasoning behind this tool So how it started Thanks to sponsorship from the OpenSSF AlphaMega project we start in An audit of all of our GitHub repository last year in August You can find a full report at this link and what we found is that some of the score from the scorecard Was actually not great So many many of our projects many many of the repositories of our projects actually don't have any branch protections At this scale we have about 700 repositories enabling branch protection all of them manually. It's just an ago So we need to tame the GitHub at our scale again today. We have 150 organizations at GitHub that we need to manage With more than thousands repositories and it keeps growing and growing So our requirements again To complied with the strong governance we have for our projects The very first one is that we want to support the definition of default settings And but also the the ability for a project to override some of those Settings so the default one would be the secured one But there is no one-size-fits-all right sometimes you want you have exceptions You have very specific use case why you want to be able to override that at one specific projects What we wanted to be able to do is also to support all other kind of settings because we are not only talking about security settings But other general settings for organizations and repositories at GitHub So we wanted to be able to support all of those but also including some of those settings that are not available or reachable Through the API is made available by GitHub So you may or may not know that some of the settings that are available through the UI I've actually no public API whatsoever available for GitHub from GitHub So that was one of our requirements We wanted it to be non-disruptive at setup. So whenever we deploy the tool on our projects. We don't want to Change the settings at the time we want to Retrieve all the settings the snapshot of the current configuration And then we start to iterate and communicate to read the projects to change and improve the security posture of their settings But we want the deployment to not change anything Another one is of course, we don't want to make it easier for attackers to Target our project So we don't want we don't want to leak any settings that is not visible by default via GitHub API So for instance branch protection rules need to be authenticated and have specific permission on the repository Not everybody can see those settings. So we don't want to expose them And also the last requirement is to be able to let our project actually see their settings So they they it improved their trust in all management of their repositories But it will also help them have a more self-service regarding the change of settings. So currently for this tool if they want to add a Branch protection holes for instance, they have to open a ticket at our help desk asking us to do that for them That's quite cumbersome if we provide a tool where it's easier for them to know. What's the current state and change it? Rising a PR for instance It's for the best so how it's going. That's the tool. We've been developing. It's called autodog so you may get the the pun behind autodog auto being the the The species that actually eat octopus and Doug being the best friend of cats So in one sentence autodog is a command line to to administers to administer GitHub resources organization and repository settings including as code So what kind of settings do we support? So we support everything Regarding organization settings Even those that are not accessible via API and there are some of them that are actually very Security sensitive. So for instance, there is a settings called member can change repository visibility That's something that is only available through the github API the the github website But it's not you cannot change these settings through any of the public API from github. So if you want to enforce the fact that None of the organization you manage Let though the their members to change Repository visibility changing from public to private or private to public you have to do that through the UI So you can understand that with 150 and counting organization. It's not something that you want to do to manage manually We manage we works and all the repository repository settings including the branch protection rules so again a link to the The github repository on our website with the list of supported settings So how does it work? At a very high level each organization that we manage will get a dedicated private repository Why private because we don't want to leak any sensitive information with a configuration file This configuration file is written in the jsonet format You may or may not know jsonet. I will explain it a little bit later exactly how it works But it's very convenient to allow overriding of stuff of the settings So each organization that we have the 150 organization will have one private repository with a configuration file inside it This configuration files look like that. So it's very Very common if you know about the github API you will recognize many of the same wording for for each settings on github so you set the billing email whether you You require web signing the web coming sign-off and so on and so on pretty standard Now what this file does explicitly in the very first line is to import a Command default and this command default come from yet another Organization or yet another repository in a different organization, which is our own that We define and none of our developers have access except the security team where we define the default values for all of organizations so this default setting actually implements a jsonet API We can call it like that where whenever we want greater new organizations Here is the the settings for instance for whether they have projects they support projects What is the default default permission for each repository for members and so on and so on so any new organization in the rating from our calling this Implementation of our API Will get those default value? So we end up with something very standard if we look at an object annotation Point of view so we have our own defaults in the master A Master organization with the default settings the default files and each of organization 150 not only three Will inherit cold the parent one? So how does it help with security this thing that really because we define those default settings and the less The each organization settings deviate from this default So the less there is over overriding The closer it is to our secured settings So let's have a little demo. So I took very little risk For this one. I registered an ASCII name Demo, so it should go fine. So the first thing we do is we are in the context of a Security admin who wants to change some of the settings on one of our one of our organization so we we use the the master The master settings and currently in there. We have only a single file called Author the system in 2023.js and it so that's organization that we want to change In real life, we would have 150 files in there for in our case So if we look at the content of this file, it will be very similar to the one I've shown statically in the slides Before so we we see that it imports the defaults and it overrides the number of settings. So for instance, it allows all members to change repository visibility It does not require two factor authentication. It does not require webs web commit sign off and There is only one repository in this organization and this repository Also allow a bunch of insecure settings Why do I know because it is insecure? That's because it's actually Overridden from the default and I know that the defaults are more secure than any other writing that they can can have been defined in there So what we want to do now is to edit this file and remove basically every overriding In there, so I will change the email address just for the sake of the demo and then we will remove everything that is not In the default so this thing I don't know what is about but I'm pretty sure the defaults is more secure than whatever I could decide to do here. I change the description again for the sake of the demo remove everything about membership permissions everything regarding packages permissions and I remove the web commit sign off required Say that and now we will want to apply so in order to do that We will call a command line tool called autodog.sh. So it's just a wrapper around the The actual tool the world autodog is not a shell script. It's actually implemented in Python And we want to validate whether the settings are okay or not So when we call validate All it does is running locally a bunch of holes that we've implemented that actually encodes some of the many of the The de facto Implemented validation rules that GitHub has behind its API So everything that is written into the doc or may not be actually documented like if you do if you activate these things that you cannot do that anymore and so on We replicated that into autodog in order to be able to validate offline the content of the files So here you can see that for the repo named test repo The web commit sign off required is set to disabled while we remove this While we have activated or we are now requiring that at the organization level So GitHub would reject that and the tool detect that before actually deploying the stuff So we will go back to the Just a file we remove This disablement in The file and now we validate the settings and everything is okay. So now we will want to Plan so for those who know about terraform you will recognize many of the actions from terraform very similar We we like terraform and we We get a lot of we got a lot of interaction from them So planning is about Checking what we would be achieve. Sorry if we eventually Deploy the change the changes. So here you get A diff if you want about the remotes configurations and your local file definition So if I apply now this file, I know that I will change the billing email I will change the dependable talents enable for new repositories and so on and so on and forcing every single Setting you have the old version and the new versions So that's for the organizations and we see that there is no changes for Repositories or we books. So now that we've verified that we want to apply the changes So apply the changes actually evaluate first Then plan you will see that it will actually display the exact same UI And finally ask you to confirm that you want to apply the The changes remotely on the GitHub organizations There is a dash-dash force Parameter if you want to not have this confirmation, but When we are doing that outside of CICD it's best to validate manually And finally the last step is to push the config And pushing the config means that we want to upload The new version of The organization files. So the autodoc demo the chestnet file the one for my organization. We want to push that to the Repository that holds that in the organizations White is stored and the new file now is very simple very simple very streamlined Nothing regarding security. We delegate everything to the default So on the on the two screenshots, so the one behind it's the old states So without any billing Sorry, those two have old states previous states. So there is no billing address On the first one or the the billing address is actually my gmail address You can see that there is no private No dependency graph enabled no dependable alerts enabled and if we That's the new status after applying the configuration via autodoc So all of the settings have been Applied properly So autodoc is not only about changing settings, but you can also create new repositories and inherit from the default settings In the parents. So let's look at Again a short demo. So we are in the same context A single organizations I will edit the file and add a new repository to this file so In order to do that, I will call The the api from the default so the org dot new repo to create this second repository and I want the secure default from the From the parents so I won't I won't set any settings except a very short description So once it's done, we will do the same. We will first validate the configuration. It should not find any issue We want to plan just to verify that everything is all right And We'll be deployed properly. Actually, I apply it directly. I don't plan because I will see the changes at the apply time And we see from here that the new repo will be created So the name is really the name that we we've set What is the name? Yeah a second repo and the description is properly set And all the other settings comes from the inherited default. It's not the defaults from github So I will validate that I want to create this repo And there was an issue with We think github as a bug in that font On that side, we have to apply it twice in this case, but it has been fixed since the demo has been recorded So, you know, you don't have to apply only once to create a new repo But a couple of days ago we had this issue where we had to apply it twice the To create a new repo And the new repo has been created So we started with this state So the test repo was the first one that you were seeing the dot eclipse fdn-private repo is the one Where we store the configuration for this organization And now after creating the second repo, of course, it appears in the in the organizations So we can add many other settings. We can configure everything For each organization in each repo we can add branch protection rule Again, we have some nice default for branch protection rules Or what we think is nice default and secure defaults You can implement your own or you can override Your own branch protection rule for each repository or for each Protection rules and so on. So for instance, if you want to Buy pass to allow some first push for some Pass in the repo or to change the push restriction and so on you can Do that directly in the file So how does it work internally? So as I said earlier, it's written in python 3 It uses jsonet as a data modeling toolkit And I will dig a little bit deeper into jsonet and it uses jsonet bundler to as a package manager for jsonet So that's what links the The parent repo the the one defining the default and the sub the organization level settings It accesses kit up the other sapi in graph graph qlapi and also the web ui directly For the settings that are not available for rest and graph ql via graph real and rest Of course to change things in github you need some credentials So we have implemented two credentials provider Via pass and bitwaddon. So you can store your credentials to to rich github in those two And of course it can be extended And again a link to the repository where you can find the tool So jsonet who knows about jsonet just Nobody perfect So jsonet is a super set of json to and the the night property about jsonet is that it is side effect free So whatever is written jsonet, you can really run The generation every single time and you will get the same output from for the same input So it's very handy to To reuse parts of json Json object that you want you have to repeat in a lot of places So for instance here, we have the person one which is almost Standard json except that you see that you can reference other attributes from the current object with the self dot name thing And you can create a new person Person two by referring the other person one and extending it or overriding the name attribute For this specific object and you get if you give this file as an input to the jsonet binary then you get the output of json on the right side You can also define method so here we We define a kind of constructor for a person that takes a parameter With a default value and you create this object This person object from the parameter and then you can call it At several places in the file. So person one is person Which will take the default value from the parameter at least and for bob it will be overridden and you get the same output And you can go even further. So jsonet allows you to define Libraries or external files and we import that From one jsonet file to another So here we have a cocktail definition files and When you want to create an agrony you need to have three equal parts of gene vermouth and compare And in order to have the three equal parts and not repeat the kind Gene quantity one vermouth quantity one and compare quantity one You can reuse a library that you can define externally in a separate file So that's the utile dot clip sonet at the bottom And that takes an integer and the list of ingredients an array of string And you can see that you can loop on the array and create an object for each element on this array And as it is called into From here, you will create a list containing a kind a list of objects with for each a kind and a quantity the quantity being the the Results of size divided by the list of the size of the ingredients And you get this output over there so Very convenient when you want to override stuff and define or generate large json file from a subset of Very small library very small functions and that's exactly how we define that For autodog. So we have a set of functions In the top level organization with the default The set of functions implements the default value the one that we think are secured or the one that we That implements our strong governance and then at each organization level they will call those functions the Overriding some of the settings or not And and the results will be a complete files with all the settings Combine merge The the thing is if you start to reference another file from the the organization level You you will find yourself in a difficulty for okay. It's a local path I will have to clone at the proper location and I will have to to have those two repositories one beside each other the good thing with json there is An ecosystem with json bundler Very similar to package or json where you can define dependencies across repositories in order to retrieve Files or libraries of json files in other repositories. So what we do for each organization? So we have the json file Encoding the specific settings for each of those or for 150 organizations and also a json um A json bundler file if you want a package of json referencing all default It's a good thing because you can you can reuse or you can define your own you can reuse sorry autodog And also reference another default than ours. You don't depends on us on defining those defaults um, so that What let us Do exactly what you can see at the first line so we can now import the defaults from a repository vendor vendor and this vendor Repository is this vendor folder. So it's created by the execution of uh json bundler installed So just like when you do npm install some uh your package or json You will get all your dependencies in the folder and then you can reference those from this folder And that's how this whole graph actually Take place So what do we do for um Or how do we call the the how do we retrieve the the github settings? So for most of the settings the rest api is perfectly okay We use also graphql for the brand protection rules Because it's not exposed properly by the rest api And for the list of the settings below We had to directly call the web ui So for that we use playwright You may or may not know playwright It's kind of puppeteers if you puppeteers if you know about puppeteers. It's normally um and end to end testing for web apps So basically you encode uh that you want to open a new browser and you want to click on this button programmatically and we use that to actually open a headless browser and Put credentials into the deroging form of github and go to the settings page and read the value of the checkbox and so on so That's the only solution we found to be able to manage those settings that are not available via the api there are numbers of support requests for github to actually provide api access to those settings It has been dead later for for a couple of years So we had to you know to scale we had to find a solution and that's the one we've Came up with so we know it's brittle because whenever github decides to change the way they display the Their settings we will have to adapt And also sometimes github is down that happens Yesterday or the day before a consumer So if you want to still be able to deploy stuff if the web UI is broken and so on we also provide A flag in order to to skip all the web settings So now for the credentials providers as we call the rest api we need personal access token Because that's how you you interact with the the rest and graph ql api at github, but you also need some user with username password and some two factor Time base one time password seed in order to be able to connect to the ui And we implemented this support in with two credentials provider But of course it's extensible to any other and if you want to use the dog and have a credentials provider that is not implemented Please contribute we Are welcoming those So Just briefly how do we Support for instance the credentials in pass So pass is a local password manager that encodes every single credential that we pass to it via gpp keys We request or we support it the way that each credentials or each part of the credentials the usernames the password The token and the t o t p c needs to be in the different entry But the name of and each entries Are configurable For bitwallon so bitwallon is a very common password manager very well known password manager Very similar to last pass dash lane and so on So for this one, it's a bit different We require all the credentials to be in a single entry a single item and for the Personal access token because it's not a credentials in terms of user and password We ask it we require it to be in a custom field. That's what you can do in bitwallon. You can add custom field and But the name of this field can is configurable as well So now if we put yourself In the seat of the administrator Person we will have to manage those 150 organizations. You will start with this kind of file So on the left some metadata we'll explain about it and on the right you will have the list of organizations that you want to manage So here I have three So we have the name the github id the github handle that will And then of the organization that you want to manage and also the credentials you will need to use in order to To connect and to change the settings in those organizations On the left It's basically some metadata. So what is the base template? What is the default organization? The default repository where the default settings will be stored What is the name of the folder where it will be stored locally when we will apply the tools and so on So how does it work? When we want to import or manage a new organization that is not managed by autodoc at first So I have a brief demo for that So we are in the same context again Very similar context. So we have an autodoc.json file. So that's the one Having the it's the root files settings files for the the administrator With the organization settings and the credentials in order to connect to these organizations So the first thing we will do is to actually fetch the configuration To import all the settings Again, that was one of our requirements. We don't want to be Disruptive when we import when we deploy autodoc on a new organization So the first thing is to read everything and to create a new jsonet files that encode the current snapshots the current settings Of the organization. So that's the results of this import So the defaults are actually encoded as a jsonet. So we extend the the default With only what is different exactly from the default. So we It's very easy to generate a full A full version of the defaults and to only encode in this import the differences So we've imported we've imported the The current the snapshots and now if we do plan just like if we wanted to deploy it, of course, there will be no changes That's the current The current definition of the organization remotely but There is one because we will eventually want to store the settings in a remote repository The eclipse fdn-private repository where we will store the The settings for this organization. So We still have one changes when we import a new one. So we want to apply that and create the the new repository And the final step when we want to deploy autodoc is to actually push the config So to push these new settings that we've imported to the newly created dot Eclipse fdn private repository. So we end up with So we started with this Fresh organization not managed by autodoc to these things with the new repository and the configuration files that match the current settings Of the organizations So why why do we think it's very valuable? So you may have seen that two days ago Github made a change and make it Make it generally available to enable branch protection for public repositories We've been able to react to that very fast. So we enable the support for this setting at the organization level which ends it In the defaults. So we want all of our organizations to actually enable these push protections And we can deploy that to our 150 organization at once So that's the power of using this kind of tool So there are many alternatives. I won't go I won't deep dive into every one of those But we are pretty Sure that each of those alternatives actually don't comply with all of our requirements. So that's why we had to Create yet another one So what's next? So currently autodoc is a command line tool We plan to also create a github app In order to remove this manual step of applying and pushing the configuration to the repository We'd like it to be a bit a bit more streamlined also remove the need to maintain some tokens And to have faster reconciling And we also want to add for us the security team some more monitoring and other team capabilities So the github app will be able to tell us. Okay. So this organization has some settings that have been changed Doesn't look okay Go and look after that So the takeaway is A strong governance is strong security to to be able to enforce those stuff Though at scale requires tooling There is no good Default that doesn't never need to be override overridden. So you need to provide this capability For github management github organization management autodoc make it easily applicable at scale the the last changes the last change from github or the last Generally available settings that github demonstrate that And and we do that while still allowing the the overriding So as a final last word, I would like to thanks open SFL for mega projects that enable us to create this tool and I may have one minute for one question if you have any Yeah So the question is do we have any pushback from the repository owner? They are not owner. They're just they just have right permission on it But do we have any pushback from them having to create PR? They don't have to create PR they can but they but they can still open a ticket out or help us to Change some of the settings No, we didn't get pushback because what we provide now is actually a view on their current settings previously. They had no View they couldn't read The the the settings of the organization So at some point when you want to add a new branch protection rules Or if you want to enable some settings you had to first ask is it enabled or not? So now no pushback and again the the first step when we deploy the tool is to be non disruptive So the they they get to use autodog without knowing that they use autodog. Thank you for your attention I'm still there until tomorrow evening. So if you want to talk again, uh, deep more Be available outside otherwise. Thank you very much. Talk to you soon