 Hello everybody. Thank you for joining me today. Today I will be speaking about how I implemented a multi-branch pipeline with our workflows and how I debugged a small contribution that makes it possible and also an open source project that came out of it. So a little bit about myself, my name is Gosha, I'm a senior DevOps engineer at Rookout, married and a huge fan of hiking. So it all started when we decided to give our developers the freedom on shifting left and give them the ability to choose which tools they want to use for our CICD pipeline. We are great believers on shifting left and on developers' first approach. So we decided to give our developers all the freedom to decide which CICD tools they want to use. At first it was great, everybody was very pleased with it and over time it became a huge mess. We had multiple of CICD platforms to manage the pipelines and over time it became a huge mess. There was one server that actually started with Jenkins, then triggered CI for tests, then waiting for it to end and after all of that actually triggered GitHub actions just to create release notes. So it was too much difficult to maintain it and it was very hard for new developers to onboard quickly and each time they needed to do a huge reverse engineering of the pipeline just to find where they find. So we decided to consolidate our CICD tools. You of course already can guess which tool we have chosen, but it wasn't that simple to choose which tool to use for our CICD pipelines. Our developers had a lot of experience with all those CICD tools. The expectation from new tool was very high and of course the requirements too. There are a lot of pros and cons for each of those CICD tools and the only request that came from DevOps them is actually to make it a Kubernetes native tool to integrate fully to our ecosystem. So there is not much Kubernetes native CICD tools. So we started with Argo and this how my journey as a developer experience product manager started. Actually I wanted to research how to research the developer experience and I read a lot of articles about how to do it and all of them just talked about to get product market fit. So for me actually product market fit was happy developers that was my metric that what I was monitoring to see if I'm reaching this goal or not. So much before implementing anything I decided to use a lean software development. It's actually iterative process that solving some like it's a product optimization problem that you want actually to put a least effort you can to get the maximum accuracy on your decision which CICD tool to use. So much before implementing anything I decided to perform a quick demonstration to our developers with Argo workflows and actually show them the UI the usability of it and how it manage and there was my first immediate feedback. It actually doesn't support multi-brunch pipeline. So multi-brunch pipeline actually very basic feature for CICD tools that all of our different platforms, Jenkins, CircleCI, natively supported and I use it too every day and I was that much hypnotized from Argo workflows that I didn't realize it's missing a multi-brunch pipeline. So a little bit about what it is. So basically a multi-brunch pipeline give you the ability to shift left and to manage the configuration of the pipeline on your source code repository and even each of your branches going to have its own pipeline configuration. So it makes it easier to maintain those pipelines because it's sandboxing them for each of the pipelines and so you can code review each other, run this pipeline much before to test if it works. So it's helped the DevOps team too to maintain those pipelines. So my second iteration was actually to ask the developers how they use multi-brunch pipeline even though it was widely used. They used a lot of different tools. So I had multiple interesting feedbacks like they hate long yamls, much of them prefer programming interfaces, other prefer declarative interfaces and after a while back and forth talking to the developers actually this was my MVP for starting this project. Actually as you can see here you see in the left side you see the dot workflows folder. It's actually going to be the API of the multi-brunch pipeline with algo workflows that are going to have at the start a main yaml file, parameter file that's going to be global values and template yaml actually just holds the templates so it separates the long yaml to different yamls and it makes it easier to maintain. So actually what I wanted to implement is that each of those branches in the source code repo are going to have those folder in files and the end goal was to create a CRD custom resource definition at the Kubernetes cluster where the algo workflows control lives and then he will do the heavy lifting for me. So for that I choose to use algo events, a quick explanation not accurate at all about algo events. It's actually one of the algo stack tools that helps you to get a webhook and then create a custom resource definition. It's much more complicated than that but actually it is the end goal of that. So the webhook passing through the sensor that the sensor is going to create algo workflow CRD. So how it works actually from the source code repository a webhook send to the repo sensor then it will create a workflow CRD that will manipulate the webhook that we just got then download the dot workflow folder create a new workflow out of it. So we need another sensor to listen for that. We call it seeder sensor. It will seed our workflows and then it will finally submit the CRD that will represent the multi branch pipeline in our workflow environment. So I started with the last part of it with the seeder. I wanted to check if it works so I just send this small webhook and I wanted if you're not familiar with sensors so I wanted to take a parameter this the destination of it and this is going to be the end goal of this destination. Actually I wanted just to inject this JSON but convert YAML to the templates so it will execute it. So this the workflow that will be created that what we expect in the end of it so we got an unmartialing error. Actually I hoped it will go much smoothly and then I decided to debug this sensor and my easiest way for that was to use actually Rookout. This is the string that I got with the unmartialing error so it's definitely a escaping problem of backslashes. So a little bit about Rookout. Rookout give you the ability to debug live application in the production environment or remote environment without redeploying it without adding log lines and then redeploying that again so it's made easy for me to debug the sensor. So we wanted to place Rookout inside this sensor for that a quick instrumentation needed. So at the entry point of the sensors I actually put it our agent and I needed to inject environment variables too. So actually the Algo workflow, Algo event controller going to see the CRD deployed the new sensor and going to be instrumented with Rookout inside so it's easier for me to find the bug. So I knew what I'm looking for and where I need to look for it so I just send in this webhook again, this payload again and I found this one, this was the travel maker so actually it was my first contribution to open source and it was just to switch the function, it set bytes to set row bytes, a little bit new tests and that's it and here this how a new feature with our events came up like now you can template the whole block and not only one value of specific key. So I tried it again and I got another error. I could template one block but once I wanted to template it's like actually I'm marshaling a whole struct because of its substruct of a bigger struct it actually made it much complicated to contribute so you can template a block but not all the blocks if you have a list of those structs it's a problem. So I came out with new architecture for the CDER pipeline so actually it looks like this. So source code will send the webhook to a CDER sensor, this sensor will submit a workflow that's going to be called CDER workflow that this workflow actually will download the dot workflows folder and manipulate the CRD and then submit it by itself so is the end goal of that. A little bit details about that so the process began with the webhook again and then we're going to have two sources for our workflow actually a template of the workflow itself it's not a workflow template CRD but just a workflow CRD that have all the spec inside all the default configuration inside and the other source is actually the branch where dot workflows folder leaves and the CDER workflow actually will fetch it to the workflow and create it then it will merge them together it will take all the files and dot workflows do all the logic and inject them inside this workflow template and then of course submit it to the cluster and this is how we're going to have this multi branch pipeline so back to our interface to the to the developers so we had a dot workflows folder that going to have main yaml that actually the logic of the pipeline it's will edge create a dug inside of it that the director tickly graph a global parameters and the template yaml that's going to populate all the implementation of steps that we will reference from the main yaml file so how it works so we have two sources from your left you can see the workflow CRD that will be populated injected with all the a branch configuration with all these yaml so we're starting with metadata from githue webhook we'll take from there the user the repo name and other labels that we want to and then we will inject the parameters yaml it's actually a global workflow scope parameters created inject the templates into templates so we could be referenced and actually it's a local stop step so nobody from other repos can reference those templates it leaves only inside the rep itself that in specific branch so if you implement new template you actually can reference it only from your branch so it's nice to sandbox it from other people and the main yaml it's actually the dug that will be created and will run will act as an entry point of all this workflow so here how it looks actually you have you can see a two workflows the one called cedar the other called a I don't know what the name of the repo is actually the repo then the branch then some generated a string and this was my second iteration of implementation and I give it to a to the developers to use it I got a lot of feedbacks about about this solution and one of them was that it takes like 10 seconds and little bit messy that you're not sure which cedar is a reference which workflow so you can imagine that we had here a lot of walk arounds and then we decided to take to take it further and actually we started a open source project that's called a piper a piper will it's actually already have much simpler a workflow so the source code repo will send a webhook to piper piper will actually get this webhook and then submit the workflow so it makes it much much easier for now piper supports a github only sorry guys but I need started with something and it now manages all the configuration the webhook the secrets and so on just when you're providing the the token to it and it's actually we expanded the interface and now we have a trigger file that will actually reference all the triggers and all the the yamas that we're going to use there and we have a long long long roadmap and of course that all of you are welcome to use it and contribute it to it here you can find the ripple of it so if you want to scan it's great and we actually can create a quick demo of it okay I will come back to this qr later so I will make it okay sorry okay great so actually this is the trigger yaml file we extended it now it actually will purple a list of triggers that you want to trigger the inside any logic that you will place inside so events going to place to hold all the the list of the events that you want to execute the tree the pipeline in it branches of course the branch that will trigger it on start and on exit actually going to be your entry point and exit handler for your workflow CRDs so you can place a list of them so as we had the first feedback of long yaml now we can separate those yamas to how many yamas you want and then reference from each other actually it supports only Doug and now the and we don't use steps at all so if you want anything liner just make it dependent on each other and it happens yaml that will populate the templates on the workflow and actually you can reference those things inside so I will trigger it it's have a dummy pipeline side so we send a web hook and now we're expecting to see a workflow right here here is actually the workflow created and it's running now in the cluster it's actually in my local machine and you can try to to you can try it locally too with make deploy it will create the cluster everything else with kind and that's it thank you guys we have time we have some time some questions if anyone won't just raise your hands and I'll bring the I'll pass the microphone just a second let's wait for the microphone so the people the recording can hear as well hi I wanted to ask where does the piper reside sorry where does the piper sit the piper the piper actually a self-hosted application so you you will deploy it whenever the ah go workflows controller leaves it will it can communicate directly with workflow or create a crd but actually you just want to place it whenever we're whenever the workflows he need to create crd and the workflow need to to take this crd and do the heavy lifting for it okay so basically it sits where the are good self sits yes in the cluster yes but but you you can you can configure it how you want thank you yeah my question is regarding after the flow runs how do you see the output in your git repo so that you don't have to every time visit argo great so actually one of the it's still a blank interface but the one of the feature is actually to update the git status when the workflow finish it's going to watch the the crd and when it's updates go take events for Kubernetes and then update the git with the apiaco and that does it automatically or you have to configure it it will do it and to automatically for now I have like a workflow step they're doing it but it's on development cool thank you questions anyone now's the time questions yet here we go can I replace the webhooks with other inputs for the argo events and use the pipeline the piper actually now it's will and marginal payload that have specific fields so actually you can mock it and send it now we have interface only for a github but we can extend it to local github or something else okay thank you any other questions anyone okay so thank you very much and he's here if you want more questions then thank you