 Hey everyone, welcome to our talk, Peribolos, GitHub management on its growth and Kubernetes project. My name is Priyanka Sagu. I work at SUSA as a Kubernetes Integration Engineer. I'm a technical lead for the contributor experience as well as a GitHub admin for the Kubernetes orgs. Hello, and I'm Jason Braganza. I'm an IT consultant to small and medium businesses, with the cloud pavids. I also help as a Kubernetes new membership coordinator, helping out with the request for and welcoming folks to as org members, Kubernetes org members on GitHub. So let's start with an agenda for today. We look at why we need Peribolos. We meet Peribolos and look at the role of Peribolos in the Kubernetes project. We see Peribolos in action with a small deep dive and we do a little recap. So let's go. Why Peribolos? Let's begin with a story of a small org. In the beginning, there were a team of about five folk. There's our CEO, three engineers and a consultant for non-core tasks. We use GitHub. We have an org to collaborate. Six months later, we have grown our engineering team by two. We add them to our org. And hey, welcome to the team. A year later, we have a lot more engineers and we split our org into two teams and we enable the requisite permissions and get on with our work. Another year later, I cannot count anymore, but we're growing. It's getting a little out of hand, but the DevOps folks are on top of it now that the size has increased too. They're more functional teams now and things need to be changed all the time, but there's nothing we can't handle. A couple more years later, we got funding and then there's utter chaos. We're growing explosively and the teams are growing to match. There are more folks coming and going. There are all sorts of requirements to the GitHub org and they're all in flux. Stuff needs to be changed on a daily basis. We're holding this jigsaw patchwork puzzle together with duct tape and all sorts of scripts, page shell or Python. But if the fades have a deem otherwise, this whole edifice can shatter and fall apart very easily. That's where we look for something strong and stable and shining, and our solution is Parivalous. Let me also illustrate this same scenario with another story. A while back, I worked as a medium-sized workshop, a web shop, and I was put in charge of moving the whole org teams and repos and all along with the pipelines from GitHub to GitHub. The team was a sub-30-member team. They were prolific with their work and I had about seven functional teams and 100 repos and I got it done. The whole org came over, everything worked fine, but I did the whole thing by hand. It took me close to a fortnight. It took me copious amounts of coffee late nights, empty moments of trial and error, and an ungodly amount of GitHub rate limiting. So it was a similar scenario that our friends at the Kubernetes project faced when they came up with Parivalous. Christof is here. Hello. He and Eric Feta gave the OG talk on Parivalous right here five years ago. Here's a few screenshots where Christof describes just one scenario filled with paper cuts. All we're doing here is creating a team and adding members. Keep an eye on the click counter at the bottom left. We start with one. We try to create a team. We're up to three. We try to add the team members manually and we're up to five, and by the time we're done, we're up to 10 clicks. Christof ends with repeat until 10 and our fingers are bleeding. So I could attest to that. During my time, I had a whole lot of bleeding fingers. So back to me. In the role I serve now as the NMC for the Kubernetes project, I have to add members to various orgs on a near daily basis. There are a couple of verification steps that need a human, but the most important part that needs to be done flawlessly is adding a member to the right org with the right permissions every single time. And all I need to do is edit a YAML file and add an entry. That is how easy it has become if only I had met Parivalous earlier. So let's do that. Let's go meet Parivalous with Priyanka guarding his father. Before we see Parivalous in action, let's see, we'll see Parivalous in action with our friend Jando. But before that, what is Parivalous? So Parivalous is a tool simplifying GitHub management by updating all of organizations, GitHub organizations, setting teams or membership through a YAML file configuration. What it means is you describe everything that you need to be configured on GitHub in a YAML format and Parivalous takes it over from there. So let's see that in action with Jan. Let's try to install Parivalous. How we do that is code for Parivalous, the tool itself sits inside this repo called Kubernetes slash test infra. You clone that and you build the Parivalous binary and if the installation went well, you'll have the tool available to you. You can check with Parivalous help flag to see if everything is working fine and all the flags that are available to it. Here, we have an org with Jan. The org is called inOrg. Let's see, let's try to do a few things with Parivalous on this org. First thing, let's try to export the org configuration into a YAML format. So Parivalous has a command for that, Parivalous dump the dump flag and you give it the name of the org followed by the GitHub token that has access to that org and what that will do is it will extract or export the entire current configuration of the inOrg GitHub organization on your screen. In this case, we are putting that into a file called inOrg.yaml. What that file looks like is something like this. You have the top level key called orgs and right under that is the name of your org in this case inOrg followed by all the work settings so things like your email address attached to the org company or other flip flops that you have done for example whether this organization has repository projects what is the default repository positions etc. In the exported org YAML you will also find all the admins currently on the GitHub org followed by a list of all the members and a list of all the teams and here we are showing on the screen it's a team called org group and all the members and maintainers under that team as well as what repos that team have access to so let's see whatever we just saw how we utilize that in Kubernetes project we have a lot of organization in the Kubernetes project at this point I think it's 6 or 7 or maybe more and all of the organizations metaconfiguration or the YAML equivalent of what we just looked for inOrg sits inside a GitHub repo called kubernetes slash org or get.get s.io org the repository contains the metaconfiguration for all the Kubernetes GitHub organizations example it would contain the configuration of our Kubernetes org or Kubernetes 6, CSI, CDIO and that data that sits inside this repo is then consumed by Peribolos what Peribolos does is do a reconciliation loop just like Kubernetes what it does it, it looks at all the declared members basically all the data that we have declared in our YAML and try to observe whether it matches the observed state or it matches the actual state of the GitHub organization so in this case if I have declared a few members or a list of members in my YAML Peribolos will try to get a list of all the members from the GitHub organization if it sees there are members listed in the YAML file which are not already part of the organization it will invite them if there are members listed and if their permissions needs to be escalated it will promote them if there are members already there in the org but not listed in the YAML format it will remove them and this reconciliation loop will keep going on it will do the same thing for other information that you have added in the YAML file for teams or other metadata so with this idea in mind let's see Peribolos in action in Kubernetes project so one of the things that we do with Peribolos is managing our org membership how does that work so here we have Shanno she has raised a new membership org request this is how the template for a new org membership request look like we need the GitHub username as well as which organization that this person is trying is requesting access for followed by other information so we saw here Jean is requesting access to these four repos as four GitHub organizations Kubernetes 6, Kubernetes CSI at CDI that's where our new membership coordinator or Jason comes in the picture what we need to do to add Jean to these four requested repos is make some changes on our Kubernetes slash org repository that's where we keep the meta configuration so a new membership coordinator or NMC will clone the repo we'll check out to a separate branch to make some changes in this case Jean requested membership for Kubernetes for example so we'll make changes in our org YAML file for Kubernetes org in this case this is how the org YAML looks like we are adding Jean in the member section of the org YAML file in alphabetical order once you have made the changes save them add them, commit them and repeat all the steps for the other three organizations that she requested access for once you have made all the changes then you just push them back to the GitHub repo and that is basically a PR what happens when this PR is merged this PR actually made changes to the Kubernetes org repo aka the K org repo which needs to be consumed by Peribolos but where is Peribolos and the Kubernetes project Peribolos and the Kubernetes project runs inside a CICD our CICD infrastructure tool called Prow but I'll not go into detail about Prow because Prow is a whole mass ecosystem of its own and very well described by this diagram from Ben thank you we have a talk from ApascubeCon where we discuss how to read the Prow jobs how to write or navigate through the existing Prow jobs we have for Kubernetes project just for this talk this is one of the Prow jobs that monitor all the changes that are happening on the Kubernetes slash org repository so what happens when we created that PR and the PR was merged this job an instance of this job was triggered what it did is just ran that bascript adminupdate.sh with some flags but what happened is basically this Peribolos command was run what this Peribolos command is doing here is we are telling Peribolos here is a config path here is a yaml file in the description of all our Kubernetes GitHub organizations look for the information here if you see any org membership updates or any changes that needs to be done related to org membership do that with these flags any team updates or any team related changes or fixes needs to be done do that with these flags Peribolos has a few inbuilt security checks where we need to have at least a minimum five admins available into the GitHub organization at any point of time so we are mentioning the required five admins here with these flags followed by the GitHub token that will actually let Peribolos do any changes and the GitHub endpoints to ensure we don't meet rate limiting and finally we have the confirm flag this flag will if there is a change actually found between the declared yaml config as well as the current state of the GitHub organization and some update needs to be done whether it's adding people or removing people or updating membership if this flag is provided only then Peribolos will do that or make that change if not it will just do a dry run and show what changes might happen so good advice if you are using Peribolos planning to use Peribolos try running the commands without this flag first to understand what changes might happen and once you are sure use the confirm flag these are the few inbuilt safety checks we have already in place in the Peribolos binary itself few of these are we need at least five admins at any point in any GitHub organization also there is a flag here to make sure Peribolos don't end up removing half of the organization for example and more and if you want to understand more about the entire list of Peribolos flag there is a link at the bottom for the documentation so going back to our example when we merged that PR and the PR was taken over by that Peribolos job and the Peribolos command was run what really happened was Peribolos found ok Zhandu is declared in the four YAML files but she is not already a part of the requested four orgs so Peribolos will go ahead and invite her to all these four orgs and that's what's happening here if you want to see what that command the output of that command looks like we have a test grid dashboard for that and it's red this is a screenshot from a few days back or I think the state right now is red as well and we have to wait to decide if anybody from the Kubernetes community for example want to check a certain instance of that Peribolos command and fix for troubleshoot or get some information you can get the logs out of this test grid dashboard and the links are at the bottom a note Peribolos is independent of the CICD platform we are discussing Peribolos in case of proud right now but it's a stand alone tool it can operate in any of the existing CICD pipelines for any of the scenarios you envision for it so let's do a little recap it's been five years since Kristoff's work on the stage on Peribolos so what's happened since then orgs have increased repos have increased by a lot our teams have increased in number and Peribolos has grown and adopted adapted it to all these changes with a plumb so when I told you that our team members are doubled in size like we saw over here with 2000 plus numbers I may have fibbed a bit because we recently had a little bit of spring cleaning and we removed close to 700 plus highly inactive members for security reasons shout out to Navranpal who made this whole scenario made sure that this whole thing passed without incident but the reason I mentioned this is Peribolos had to be adopted adapted to handle this courtesy Navran which made doing risky things like master removal easy another change that landed about six months ago gave Peribolos the ability to manage team repo permissions let's look at this briefly on the left we have a default deny policy in restrictions.yaml if our repo isn't allowed in there no amount of Chigripokri on the right is going to help us gain access to the repository but if it is allowed then everything on the right is allowed with the members and maintainers with their respective permissions correctly done to quickly run through we're adding members to a team the steering committee team there are a couple of maintainers a few members and they'll have admin access to two repos Kubernetes template project and steering so you can see the results here we have the steering committee team with their members with the maintainers correctly set as well as the regular members in place and with admin access to the repos that we declared no crazy clicking involved so wrapping this all up Peribolos has a Kubernetes friendly design it aligns seamlessly with the Kubernetes ecosystem its design ensures adaptability across diverse CI environments you can see the use of Peribolos inside a Kubernetes cluster over here over the years Peribolos has proven itself it's our primary tool for managing organization settings in the project and it serves as an excellent showcase for its scalability and effectiveness in handling the inexorable growth of one of our largest open source projects and finally increasing adoption Peribolos has been actively utilized in projects like Knative, Red Hat and Kubeflow as well as sparking interest in other communities as well suggesting its relevance for enhancing GitHub organization management across varied ecosystems the docs are available here as is the source this used to be our final slide but we never got around to slowly showing it across like three presentations so far so we bumped it up a bit this is our small reminder for you to be current to yourselves and my esteemed co-speaker was the release lead for the current version of Kubernetes 1.29 and we had a beautiful logo to mark the release in December said logo now has stickers we're carrying a few you can catch us after the talk if you want one and finally you can find us on the Kubernetes Slack you can also mail us here the backward and the link to your right will lead you to our slides as well as the video for our talk which they hopefully they'll release in the coming weeks thank you thank you so much and if anyone wants a sticker for 1.29 logo I have them and that was designed by Jason so thank you for that hi sorry hello here here here enough the room I have a couple questions if yes I tapped them on my phone so I'm sorry if you already answered in the presentation you defined the GitHub token in the configuration the GitHub token belongs to I believe the bot right and the bot will have permissions to the entire organization right well yes not to the entire right now we have a granular GitHub tokens thing I have to check what exactly what are the exact permissions required but it also depends on what we are trying to do with Peribolos for example it not just in this case yes it is for the bot the Kubernetes bot that we use for making all the changes but in general it would be anybody any GitHub user names a GitHub token attached to any GitHub username that will be doing the changes for example through Peribolos so if I am trying to run Peribolos locally on my organization I can use a GitHub token attached to my username but it should be from the admin yes okay okay the other question I have is it looks like I check the repository while you are doing the presentation it looks like the production YAML that we use as a kind of a flat organization there is only one root for all of the teams does it support children of the teams because that is something it supports on GitHub I don't think so so it does because I know like I do see members attached to children teams but I think if you want to if I recall I think the answer was yes for the recording I guess it is a field of the spec that was not presented the last question I have is in the reconciliation loop if a team member is removed from the organization through the GitHub UI how does that look like in the PR that is open by Peribolos it is going to remove that user from the configuration file is that how the PR will look like no so if for example a member is actually declared in the YAML file and I went ahead and manually removed it from the UI when the new Peribolos CI job will run it will re-add them or re-invite them so the source of truth is always the file and just adding to your second question I recall we do use child member I think the example would be SIG release repo release team our Kubernetes release team has a lot of child teams so we can find an example there where release team has a lot of sub teams defined under it so that would be Kubernetes folder under SIG release and there is a teams.yaml that will give you an example for your second question thank you so much hi there sorry I joined a bit late so when are we rewriting Peribolos to support GitHub apps? sorry I did not get the question so when are we rewriting Peribolos to run with GitHub apps instead of a path token of a user Christopher if you want to take it I did not write Peribolos I just started using it as a GitHub admin recently but we have one of them so thanks and I would say thank you Christopher for being here and answering and taking a few more questions from people thank you it would not take a large amount of work to be able to convert Peribolos to use a GitHub app because the API versions and stuff are all up to date it just needs somebody with time to do the work so PRs are definitely welcome to enable that functionality it would just be to change to instead of just a personal access token and if you want to talk to people who are currently working on Peribolos that would be GitHub management Slack channel on slack.com if you have a PR in works and if you want to review or maybe help from the people that would be the place where you can start a discussion thank you thank you everyone