 Hello everyone, thanks for stopping by our session and we hope you've been having an amazing conference so far our talk today is going to be about on a topic which is Close to every contributor, but it is a nightmare for every implementer. At least that is what we feel like just a couple of days ago in the contributor summit We actually met someone who was trying to implement prow which we'll cover. Don't worry. That's what we're here for and They faced a lot of trouble and that sort of just reinforced our own belief that Proud is something Every contributor everyone was made a pull request to Kubernetes is at least somewhat familiar with They have seen it comment on their PR or their issue, but they've liked it But when it comes to Installing it for their own repos or setting it up or just understanding what is happening behind the scenes of Proud How it is working that is not clear and that is what we aim to solve in this talk We aim that once you Go through our session after this you feel confident enough in trying out Proud on your own installing it on your repositories and Basically having fun with it So before we begin a bit of introductions are in order. Who are we? I Am our Sharma. I work as a developer and experience engineer at Octeto I also actively contribute to the Kubernetes project I will be leading the CI signal team in the 125 release and I'm a new contributor ambassador for Kubernetes sick docs Where I help people contribute to the project. I'll hand over to Nabarun now. Hey, everyone. I'm Nabarun I work as a senior engineer at VMware I have been contributing to Kubernetes for the past three years And spent around two and a half years contributing to sick release Having led Kubernetes 1.2.1 last year, which is going to be deprecated this month next week possibly And I am also the current currently a release manager in the communities release engineering sub project I am also an elected member of the Kubernetes code of conduct committee. I Have been in the committee for the past eight or so months Besides that I help maintaining some other areas in communities like GitHub administration and contributing to Other areas of the project Now why are we here? What is proud? Proud is a Kubernetes based CICD system Being based on top of Kubernetes. It brings all the pros of running any application of communities Like you get replication. You get scaling you get rolling updates. You get highly available nature of Any stateless workload that you usually run on communities? Proud can manage life cycle of pull requests like review approvals merging them based on some criteria Through a component called tied which we will take a brief look at later Proud's configuration can be managed through structures inside the ML files. Now there can be two ways to manage them one is inside the repository where you host code and Can also be stored in a central repository aside aside from your code There are pros and cons of both approaches. I should be covering them later on in a future section Proud ships with a very very highly extensible Plugin system which helps you to infinitely extend Proud and change its and modify its behavior to however you need We also set a we also ship with Proud a set of inbuilt plugins Which help you manage several parts of a developer's experience in that system? while Doing continuous integration or deployment of your code Proud also include something called chat ops. We might have heard this a lot of times and how we do it is through Slash something like commands like slash on which will essentially give you a image of a duck Doing some some naughty things around the wild Arches now going to show you around the chat ops system So chat ops is probably my favorite bits about Proud because I feel like when I was starting out as a contributor to Kubernetes and you look at all these PRs and all these Maintain a super experience folks talking in languages. You don't understand Proud is the only simple thing there It's showing me images of cats and dogs. That's what they're what I'm there for no Chat ops basically allows you to have a forward slash and then a command which allows you to interact with GitHub issues and PRs. So Yola is not a command. Please don't try that You can assign you can assign issues to contributors So let's say if you're the new contributor getting started and you want to work on an issue You can just do forward slash assign and that would actually assign the issue and let everyone know that you're working on it Similarly, you can label issues and pull requests Maintainers can provide approval for pull requests and once they provide approval the pull requests get merged using another plugin which we'll cover later and Other than that you can also run tests for a particular PR and there is a lot more you can do So for those of you who have not had a chance to look at how this Or looks like we'll actually go to a pull request which is made by our very good friend and an active contributor mother who said you should meet him by the way and To give some context for this. We'll not go into details. Don't worry. Basically go 18 introduced something and We had a fix for that But then it important one fix that and we want wanted to remove our workaround. So that's what this PR was for and let's see how Brow has been helping with this PR So first what happened is that Kubernetes has a bunch of sakes for those who are not familiar and every part of some code belongs to a sake so this belong to sick testing and Using forward slash sick testing mother was able to apply this label to the pull request and then there's some conversation which we are definitely not going into and Then you can apply labels like this and this label is applied to the PR Which is a priority important soon label and Let's see what else you can edit milestones for pull requests and Tests so when you make a PR tests are run and sometimes some of those tests are flaky and You feel that that a lot of times what happens is that you need to rerun those tests and for that You can simply run the forward slash test all command And once you see that CI turning green as soon as it happens Please can maintain us to approve an LGTM because that's CI might turn purple anytime and you will not have a good time so once you get the approve an LGTM labels then your PR is almost there, but You can add more retests and labels like this like if it's urgent and if someone's working on it You can triage it and then retest and then That is how you basically contribute and get your PR accepted. So I think this now give you an idea of how proud looks like and We'll now go back to the slides Which seem to be lost Murphy's law Yeah, we are back. Yeah So now that we have seen chat ops in action I Am going to hand it over to Nabaroon and he has a diagram which will scare you But I've heard him speak and I think he makes it easy. So now we'll please take it from here So thanks. Thanks to her. He scared us everyone here, but I'll actually try to Declutter through user story of how do they interact? So I showed you how Madhav had created a pull request with changes and we will take that journey through how It actually got tested in proud So first We do have our user who interacts on GitHub right on the left. I wish I had a pointer, but I'll try to explain as well I can and when you Create a pull request on GitHub GitHub essentially generates an event and that event goes through a GitHub app or a Bought account to a component called hook which is in our proud cluster and that component keeps on Processing those GitHub events and packages them into internal structures, which have some more metadata. For example They also contain a proud plugin config proud job config and through this and Another plugin called trigger plugin It creates a custom resource called proud job. So it's a CRD proud job is a custom resource definition That is a central core concept in proud's landscape And whenever any job has to run in proud you basically create a proud job custom resource Notice I told you about the trigger plugin if you don't enable the trigger plugin This thing won't be done Fortunately by default the starter templates that we have the plugin the trigger plugin is enabled So any web hook coming through a hook will process an event will trigger an event essentially Now we have a custom resource. What do we do? We have to reconcile it, right? We have to act on that created custom resource. We have a component called plank Right there Plank processes this custom resource and Sees where should I run this and how should I run this? Depending on that it creates a pod in any of the job clusters You can have multiple job clusters for isolation For example, you might have some kind of trusted jobs some kind of untrusted jobs or let's say you are working on a Private repository and you don't want to mix code And you want to have proper isolation so you can run on any class any job cluster you want Once that is done plan keeps on looking for statuses of those job pods once They execute fine like if there's an error or if they exit successfully It reports back the status to the status sub resource of the proud job custom resource Now how do we get that status back to get up which is where our user is user won't see a custom resource for a status Right we have a component called crier which essentially cries all the output in the status of resource to get up It knows how and what to infer from that status sub resource and it sends back the data to GitHub When mother it created that PR at the end of the PR at the end of a github pull request You essentially see a list of checks These are the same checks that crier updates and it also would provide like a message on if you're failing something what test is failing It will basically verbose leisure and then you can retest those specific tests There are some other components which not necessarily come in the life cycle, but life cycle of a user creating a pull request and getting an output of a test But they are very important. We have one component for whole horror game which essentially Looks at your configuration for any periodic job that you may have created for example You just want to check your code sanity like you want to check the code coverage every day And post them somewhere it will create that proud job CR from your configuration so that Plank can reconcile it and run that job and then report the status back Sinker is basically your Recycle bin What it does is it looks at pods which have passed a certain retention policy for example Let's say there are job parts which have been lying around for more than 24 hours just an example We might set like more Strict or less stricter configs and when it sees Those jobs it basically deletes the parts as well as the proud job CRs So that the system remains in Now I'm talking about like statuses Logs we we have a lot of artifacts that are generated in this process. Where do you see them? So we have a friend and call deck If you ever go to proud or kts.io what you see is essentially deck Deck is basically a dashboard where you see what jobs we have in the community around all repositories What is their status? What was the last PR that they run for if it was a pre-submit job, which is basically a pre-emerge job or any periodic job status or post-submits for example We have a process in Kubernetes community called promoting images We essentially build images into staging environment and then push them into Production registries which you consume in kts.gcr.io Get a container registry domain For example, that job is like a post-submit job. It runs after a promotion configuration has been changed This was about the core components of proud We will talk about Two components which are needed When you want like full life cycle of a CICD system in proud first one is tied. So we have talked about testing of your changes and Labeling of your changes or whatever chat-ops you want to do Whatever merging now if I go about In the Kubernetes repository we see probably like hundred to five hundred PRs every 15 days or something like that If we go about like testing all those PRs against our base main reference and Keep on reconciling the status. We might run into a lot of issues or a lot of resource usage issues what I'd helps us doing in one way is Manage a pool of GitHub PRs it collects a List of PRs from the same repository for example, we are talking about the core Kubernetes repositories kubernetes slash Kubernetes It let's say collect collect 10 pull requests and then merges like merges them Onto the base branch and tries to test it in this way you can test like a lot of PRs in short amount of time tie tied also automatically retest PRs when they meet the criteria For example, we saw Jordan LGTM approved and cancelled the hold on Madhav's PR After doing that it satisfied all of the criterias, but still after that we do a retest of that PR Once the retest passes then only we merge it and you don't require any user intervention after all of this checks Have passed and another retest has gone through. We don't need any more user intervention automatically the PR would be merged If it satisfies all the criteria Tide also serves live data about current Merge pools and a history of actions which can be consumed by deck To populate the tide dashboard or the PR dashboard or the history page which I mentioned earlier apart from tide the other component which we Use in the Kubernetes community's test grid It's not entirely open source. The front-end is not open source, but the back-end is open source It's basically an interactive dashboard to view test results Ursh later will show you what a test grid entails and how we use it in our day-to-day workflow when he shows you configuring a proud job And they're shown in a grid Here for example, we're showing the Sigrilli's Test grid and where Sigrilli's master blocking is a group of jobs and verify master is a job Where this certain set of tests are on and we can see for the previous runs What what was the status and this is how we can decipher? What is failing in a very variable? We don't need to turn through all the logs of previous runs and how do we find them? What are the refs they ran against? What is the test intro ref? all those kind of things I Would now hand it over to Arsh who'd explain you. How do you configure proud? So when it comes to defining proud jobs, there are two main approaches you can choose from This choice it might seem that this choice is very trivial at first and you could just go ahead with whatever you want But if you plan to scale your project to the size of kubernetes, this will come and bite you later If you do not give it thought so the two approaches we have is one an in repo configuration and In this configuration you basically define the jobs the proud jobs in the same repository where your source code is living So you will have a dot-proud directory or just Dot proud at the Amal file at the root of the repository where you would just list down all of your jobs It is important to note that in this approach your jobs are not defined centrally They would be present for each repository in that particular repository The other approach is centralizing your job configuration What this means is that you have a separate repository and the purpose for that repository is to Have all the configuration for all the jobs you might be running across many different repositories you have in this approach Jobs get defined centrally So let us look at gubernator slash test infra where we'll see this approach in action Yeah so for gubernator is the test infra if you go to the config directory here and You go to jobs We are grouping jobs based on The organization and I'm gonna show you one of the first jobs I wrote and that is under the gubernator's directory inside sig architecture and this is a periodical job a Periodical job is like nabaron mentioned earlier a job which runs periodically on the on a particular repository So it's pretty simple and intuitive if you try to read it It's like any other YAML you tell that you want the job to be run every six hours you tell a name for the job and You tell decorate to what decorate to does is add site card containers which help in source code check out and other related stuff And then you mention what repository you want this job to be run for in this case We wanted to run for the main gubernator's repository. So we specified the org and The repository and the branch to path alias is basically when this Job checks out the source code, which is which we need for the job to do its action What directly do we want it for after that the spec is pretty simple. We just used a Golang based image and What we did is essentially install a tool we wanted to run called Dev stat and then run the command in it And that's it. That's this is how simple it is. It is nothing but a container where you are running some commands and That's how simple pram makes your life and that is why we love it Towards the end you see we have annotations and this is where the test grid number and just talked about comes into picture So if you notice we mentioned that the test grid dashboard We want this job to be associated with is sick testing miscellaneous So if you go to test grid dot k it's not I o you'd see a list of all the dashboards here And that dashboard was sick testing miscellaneous. So if we go here That's it. That's where it is And this is the periodic job look flaky. How nice But we go to here and this is every each run It did for the job and you can if you click on any one of these green cells It will take you to spy glass where you can see the logs for the job And it is these logs that we wanted for this particular job So if we just look here, this is the information we were interested in and this is what proud did for us It basically grabbed a binary ago binary and ran the results for the binary on the Kubernetes repository and give this a central place for the logs So another place where you Could have like seen the logs for this and this is something Specific to Kubernetes and not so if you have if you have your own Instance you would have this kind of URL for your own thing But this is where you can search through all the jobs running across all Kubernetes repositories So that job was called. Yep. So you can just search for jobs and similarly you can see the Last few runs of the job this way Yep, this this dashboard that's that So we are going to now get to the fun bits of actually deploying prow and then we'll see how That gets configured So yep coming to the interesting parts the parts where you most of you were here probably for is How do you actually deploy your own instance of proud to use it with your own repositories? Well step one is Creating a GitHub app to configure all of this Give me a second Is creating a GitHub app The only rule you need to follow for this particular step is be creative with the name because we all want a friendly robot in our repositories Next when you're creating the GitHub app This is something you need to be careful of is that will ask you for a web hook URL and when you're doing this step It doesn't actually matter what URL you add right now because we are going to change that later But remember to keep a note of this that you need to change this later and we'll provide a secret here But we'll get to that later the third thing you need to take care of is Adding a bunch of permissions because these permissions are needed for prow. Otherwise, it will not be able to interact with your GitHub repositories and this is just a snapshot. There are more There are more permissions you need to add but we didn't want to add a URL right now in the slides We'll do that later. We promise so The final thing you need to take care of is in there's a section called subscribe to events and you need to subscribe to all events Because Brown needs to know what's happening to the repository in order to react to it This is where the hook system Nabaroon talked about comes into picture because if it is not getting notified of the events it really can't do anything for you Once you are done creating the app make sure to note the app ID And to generate a private key keep them very secure because we are going to be using them later Now I don't take it from him Awesome So after you set up your GitHub app Upstream Kubernetes maintains a banner called tackle Recently we made some changes to tackle which made it a little bit Trimble to use or easier to use we fix some bugs a few months back So since we have already set up our GitHub app and we are not relying on a bot account What you can do is just run tackle skip GitHub and how do you get tackle is if you go to the Test info repo that are short if you go to cmd The command directory you'll see the tackle sub directory and you just need to go install Check out the source code and go install that should do it We don't ship a tackle binary right now, but maybe in the future when it becomes more stable and supports more providers We may start like shipping it It runs perfectly for GKE right now and our demo is also based on GKE But in case you have any other provider We are working towards it in future you will have other providers in built-in to tackle Who which which tackle can use to create Kubernetes clusters appropriately? Just to be sure you still can run your prow instance with other providers Just that the simple root of tackle will not work and currently only works for GKE. Yes the documentation also Tells you about like manual deploy steps, but in the interest of sanity. We are Demonstrating using tackle so you let tackle run you tell tackle if you already have a GCP project and a GCP GKE cluster running if you have one you can just mention the name and It will install everything for you in that cluster. Otherwise, it can create a cluster for you as well And you run it and let it run and pause when it asks you to provide a starter YAML now We're going why we are going to pause here is because we have to backfill certain information into that YAML What are those things? And even before that we have to apply the prow CRDs. Oh my god. It's too small in the Display, but we will make it bigger when we put the PDFs on sked But what it does is it when I talked about the proud of CRs It just creates the proud of CRDs for later when you want to create proud of CR so that Kubernetes API server knows Hey, what is this resource and how it is structured? One thing we need is we talked about a lot of Things artifacts like logs Did the test unsuccessfully or what was the exit code all of that is currently stored in a GC's bucket Now you might see the commands are very scary, but I'll decipher it for you It just creates a service account. It creates a bucket. It binds it to the server It binds the service account to the bucket gives it appropriate permissions and then you can use the credentials In your starter dot ml make sure to keep your GC's bucket name globally unique because otherwise You're gonna spend 15 minutes like us figuring out what went wrong. Yes also, you need to create a Secret to give it to the github app so that When when the web calls there is a unique token which it communicates through Now when you do all those changes you By this step, you would have almost everything that you want to replace in starter dot ml and start like creating Or creating the resources which proud uses in that Kubernetes cluster so you get the starter Template from the URL and then you basically fill in grep through all of the Templated strings and replace them with the appropriate values that we created before Now once you did that you can just tell Tackle because tackle would be waiting for you to give a location of a starter ml. You can just give it the Location it will download or read it from local disk and then apply it Once it is done, it will tell you hey, you're done now you have to get the ingress controllers External IP from the cluster that was just created or your existing cluster and point the domain that we put earlier When configuring the github hook for that there This is how you get access to deck also, which we were talking about earlier. Yes The one of the final steps is to just update the token that we created Before the secret in your github app and that is very important. Otherwise all the web book Deliveries would fail and the URL you use. This is what you fill in the github app web app web app section I talked about earlier and the final step is to install the app on a repo Whatever you want the org that you put in there the org can be a user account itself If you just want to try out for now, but if you are doing it in production also the starter template doesn't Is not suggested for production because the starter template creates proud clusters and job class proud clusters and job cluster on the same Kubernetes cluster ideally they should be separate. That is what we do in upstream Kubernetes And once you have all of this set up you just enjoy Now I can talk about benefits of using proud so Coming to the benefits I feel like we all know that collaboration is like the backbone of open source you cannot have open source if you are not able to collaborate effectively and I like proud because I feel that it really makes the process very efficient So if you have like a lot of repositories, right and you set up proud for all of them What this does is ensure that developers get a similar experience across all repositories for your organization so they don't have to learn the processes again and Learn new commands or figure out or ask what's gonna happen. And this is why I love proud because Developer experience is something which is very close to my heart And I feel that proud provides an excellent developer experience the chat ops based interaction ensures that you do not have to be a GitHub wizard like if even if like someone has not Gone through the nuts and bolts of GitHub understanding commands like forward slash assign are pretty intuitive because you get the idea that you want to assign someone and Lastly, I feel that proud is new contributor friendly because If I'm a new contributor, I sometimes feel lost. What are the next steps? I've created a PR now Where do I go from here and with proud you can set up automated messages and friendly stuff which provide next steps that hey You have your PR here. These are the persons you need to ping for review now We agree that all alike. It's not it does not always get them correct But as a new contributor, I would at least have a first point of contact Even if that is not the correct point of contact and I would not feel lost So I feel like these are the benefits of proud, which is why we love it in upstream Kubernetes Now what's the point of this talk? We want you to use proud we want you to use proud for production and We want you to come back to us as in sick testing Kubernetes sick testing and tell us what more we can do and Not just that we want your help When you use proud you report us back issues and even you can contribute to that yourself be those good Samaritans of the open source community and Just have fun We hang around in sick in the sick testing channel on Kubernetes Slack How you can join the community slack is you can go to Slack dot caters dot IO and get an invite for yourself When we publish the PDF When we publish the PDF the PDF would have the links so you can just click on it and Join the community slack and go to the channel You can also join our mailing list and that mailing list would give you access to Our meeting docs it will put a meeting Invite to your calendar so that you can come to the meeting and ask us questions and see how we can contribute more with that We both. Thank you all for joining us We would like to take questions if you have like one or two minutes Please any questions just catches after the dog. Thank you so much for our ending Okay, so we have one minute to take one question and then we can talk in the hallway obviously I'm not sure about that because I think at this point But we do run a lot of different kind of jobs Since we build Kubernetes for a lot of different kind of platforms The question is can we run different platforms Test for test for different platforms on the same build cluster or the same job cluster Yeah, you mean like our more AMD on the same cluster, right? Right now we do ship, but I have to verify and we can talk like Yes Okay, do we have time to take more do you have time to take more questions? No, okay You can join us in the hallway or ping us on Slack or email or anywhere Thank you so much for it. Thank you so much for it