 Hello, everyone. Thank you for joining me today. Today I will be presenting my journey implementing multi-brunch pipeline with our workflows. And actually, I will tell you about all the story, how I started this idea in our organization. And also, we will discuss developer's experience in general and how to prove it. So a little bit about myself. My name is Gosha. Nice to meet you. I'm a huge fan of hiking and pets. A proud father of one dog and two cats. So I really like pets. So it all began with that, that at Rookout, we are great believers in developers' first approach in specific shifting left. We're always trying to help our developers to shift left in everything they are doing and same we did with our CICD pipelines. So each of our developer teams actually can choose whatever platform they want to use for our CICD pipelines and implement it freely without any concerns. So at first, it went great. Everything worked awesome. Everybody was happy. And after a while, it became a huge mess. Developers left. Other developers joined us. And team mixed. And actually, each of the pipelines used several different technologies. Just an example of one of our services that used Jenkins, then triggered CircleCI, waited for the test to end and then created the release notes and get-up actions. It was not that easy to maintain. And the horrible part of it was that our new developers that was trying to onboard needed to do a lot of reverse engineering of the pipelines just to understand what they are doing. So we choose to consolidate all our CICD pipelines into one platform, one tool. You can already guess which one we have chosen. So actually, the story of it wasn't that simple because we had a lot of experience with all of those tools. Each of our teams actually had huge expectations and requirements from such a CICD tool that will handle these all features that they need. So my own requirement was a Kubernetes-native tool. So there's not much of them. So I went and checked Argo. But my developers had other requirements, of course. So my journey as a developer experience product manager started. I didn't know what I'm doing because I'm DevOps. I don't have any clue about product management, how to do it. So I started with small research on how to research a developer experience. So I bumped into a lot of articles. Most of them said product market feed. And I realized one thing. That product market feed is actually happy developers. So that was my metric. And I heavily depended on this metric. So I needed a process where I get an instant feedback from my developers to see if what I'm doing is actually working for me. So it's actually reminds, this process reminds, reminds the Linux software development methodology. But I renamed it. Sorry. So actually, it's solving an optimization problem of putting the least effort to gain the maximum amount of accuracy on which CICD pipeline they're going to use. Which platform I'm going to use. So my journey began with this process. I started with demonstrating our developers Argo workflow and how it works. I didn't went into technical details about it. And I just showed the user experience part of it. And immediately I got my first feedback from my developers. And they said me, gosh, it's not having a multi-brunch pipeline. We can't use it. So it was actually a deal breaker for us for adopting this technology. So I realized it's a huge problem for us. Actually, a little bit about multi-brunch pipelines. So every platform, every CICD platform has this feature. You can see here Jenkins, CircleCI, GitHub actions. And even I using every day with all these platforms. And I was that hypnotized with Kubernetes native CICD and native tool. So I didn't realize it's not supported in this feature. So a little bit about what it is and how it helps our developers. It helps them to shift left as the configuration of the pipeline is actually in the source code repository. And more of that, it's going to sandbox your pipeline. It will not influence other pipelines of other developers. As you know, if you manage our workflow template, when you change this template, this template will be changed for everybody. So you can't just add another step without ruining other people's pipelines. So it's very hard to maintain. So a lot of them using declarative or programmatic manner these configurations. And actually, after we reviewed them, they didn't care if it's a YAML or a code or whatever. So I started with a quick demonstration to my developers much before implementing anything of the API that they're going to use. And even before that, I have asked them about their experience with this multi-branch pipeline feature of other platform. So I had two immediately feedbacks. The first one was the hate long YAMLs. Who doesn't write? It's hard to read. And the second one was that they don't want to jump to other rep or just to find out what is the configuration of the pipeline. So everything should be in the same source code repository where the application lives. So I made that with this API. It was my first iteration, my MVP for the developers. Actually, you can see here that in the source code repository, we're going to have a folder that workflows that have three files. The main YAML is actually the logic of the workflow. Parameters YAML are going to be a global scope parameters. And template YAML is actually going to be the implementation of the steps inside of it. So it answered the long YAML thing because we can separate it, the logic from the implementation, and it's in the source code repository. So starting with this MVP, I got a wide consensus from my developers, and immediately I started to realize what I need to do. So actually, what I need to implement is that each of the commits in each of the branches need to have a workflow representation, a CRD representation inside a Kubernetes environment. So to implement it, I've chosen to use Agui event sensor. I will quickly explain what it is. I know it's not accurate at all, but it will explain what is the sensor doing there. So actually, the sensor listens. It's not accurate. To the GitHub repository. And then getting webhooks from there. And once it's getting the webhook, you can configure the sensor to template any workflow you want and submit it to your Kubernetes cluster. Afterwards, a workflow controller will spin up the workflow and everything else. And actually, it exposes the API that can configure the GitHub for you, so it's nice. So how it works. My first implementation of it was getting a webhook from a GitHub repository, then the sensor going to listen to it, create another workflow, create a workflow that will manipulate the webhook that I've got, and then enrich it with the configuration from the source code repository. Then send another webhook to another sensor. I call it CDER. And then it will create another workflow CRD that's actually going to be my multi-branch pipeline. So I've realized that there could be some problems and difficulties with that. So I started with short POC that trying to create this workflow. So I started with the CDER part. We have two parts, the part of the webhook and enrichment. And the second part of the CDER that will actually create the workflow for us. So I was trying to send this JSON to the sensor. And actually, the sensor should inject this configuration inside a workflow template. So a good idea. Quick POC. This is the end goal. This is what we want to achieve from this injection. So we got an error. We got a marshalling error. At first, I didn't realize why it's not working. And then after checking Kubernetes events, finding the CRD, I saw that string. That string actually screaming. A marshalling problem. You have a huge problem with escaping or backslashes. So I knew what I'm going to need to fix. At first, it seems like a quick win, easy backfix. And I can contribute it as its open source. So to debug it, to quick debug it, I choose to use Rookout. A quick explanation about that. We are doing debugging all day. So actually, Rookout allow you to inject an agent into your application. And without interrupting the application, extract data from it. And you're not going to need to redeploy, add logs line, and then rebuild it again, redeploy again. So it's making it much easier and faster for me to debug the control and the sensor in my remote environment. Actually, this is the sensor from before. Rookout need to go inside of the sensor. So for that, I added a few lines of code into the Argo events controller. The first one is actually the entry point of the sensors. It's one function that's called to our agent to start. And the second one was actually the configuration of the agent. Okay. I know it's small, but here it is. It didn't took me along and I found my problem maker. So to fix this issue, I don't know if you see the function that's doing it, but I just changed the one function, setbytes to setroadbytes, the other. Actually, I changed the tests and then I created a PR and this was my first contribution to open source. Okay. This is the string, this is what we need to get to reach. So now we actually have a new feature at Argo events that you can parameterize a whole block in your template. So I triggered it again. At first it worked after trying different kind of templates. It stopped working. And the problem was that the payload cannot be un-martialed. It cannot be un-martialed to not define struct and in particular it should be predefined. So I had two options here. The first one was to pre-define the whole struct. And the second one was to actually try another approach, a quicker approach. So I went to the second one. So I changed the flow of the seeder and actually it's become much more efficient. So as before, we get a webhook from the source code repository, then a sensor will listen to it. It's submitting a workflow that's actually the seeder. We will talk about it. It's very important. But what in general it's doing is actually fetching the source code repository and then creating our multi-branch pipeline and the multi-branch workflow CRD. So how it works. We're getting a webhook from the source code repository. Then the seeder initialized. It's a workflow. It's a pod. And actually what it's doing, it's fetching two different repositories. The first one is the source code repository. This particular commit in particular branch that's signed. And the second one is actually from other work flows repo that I store their workflow templates. And we have a new YAML. It's a template workflow that's waiting for us there. And actually the second step of it is to inject this configuration and measure it into this workflow template of workflow, not the workflow template CRD. And then after that, we submit this workflow CRD and we have a multi-branch pipeline. So back to our API, we have three sources of data for this flow. The first one is the source code repository where we have the configuration of the data. The second one and the third one, the second one is the template of workflow that we have in workflow repository. And the third one is actually our Git webhook. So how the seeder works. It's actually using the metadata that we're getting from the Git webhook and naming the workflow and then enriching the labels with who is the user that committed, which branch it is, which rep it is, and whatever you want to template there. Actually it can help with artifacts and which the name of our key in artifacts is the brand slash branch. So it makes it easier for artifacts too. And then we're injecting the parameters. It's global parameters that my developers can use in each of these steps that they're doing. And then I'm injecting the templates in the template block so it can be referenced from main YAML that actually will be directed at the graph. And it is our entry point of the workflow. So after getting that, one more thing, it's an excellent place to actually manage all our pipeline configuration because you have one file to configure the TTL, the garbage collector or strategy, whatever you want. The node selector, the service account, whatever. You have one place for that. It makes it much easier. Then you can override it if you want. So this is how it looks. Actually, for each of the commits, we have two different pipelines, two different workflows. The first one is the seeder workflow that we can see from your left. And the target workflow. The nice part of it that seamlessly, I can inject hooks in this pipeline and actually do some things before, after the initialization of the pipeline. So for example, the seeder is updating the GitHub status of the pipeline and the target workflow, the exit handler is going to slack the person that committed in the branch and will update the GitHub too. So this was one feature request. So another thing that my developers asked for me, they said, we can't use other workflows without debug the pipeline. And actually, there is a killer feature that you can SSH into a step. So it's not accurate, but still you can do it. So in our law, you have a better feature. You have these two environment variables that you can place inside any of the steps and it will pause your execution of the step. So you can exact before the execution of the script and then stay there after waiting for it to end and debug after. So I started with API2. I exposed them this one. I placed the debug YAML file that actually the seeder will fail the pipeline if it still exists at the end. So it will not be merged into the main branch. And API actually is allowing the developers to place a particular step that they want to debug. It will pause it. It will change the value of the environment variables and so they can use it. So actually, they didn't work at first and they inherited these two lines of code that checked in the value of the environment variable. And after that, they actually was very happy. So the metric accomplished. So the one thing I've learned from that is that I need to give the smallest value as I can and getting the most feedback from it. And so I started with small APIs and getting the feedback and checking if my solution actually worked and then building the solution around it. That's it. Any questions? Thank you. Any questions? Don't be shy. How does scales in terms of clusters per team, do you have a centralized workflow server or do you have like one per product? So we have one cluster. It's actually used for other stuff too, but it's used one space dedicated to Agu workflows. And we manage it there. All the workflows, all the pods are there. And I exposed another API interface for my developer that actually just configures which of the repos I want to use. And then once they added this value, they can use this repo. They don't care where it's running, which service it is and so on. But actually, it's not dedicated Kubernetes cluster, but it's dedicated note pool and dedicated namespace. Great talk. Can you list some of the challenges you still have with using Agu workflows as a CI engine? Yes. So the multi branch help us a lot. It makes the friction much less because now you need to write the smallest possible YAML and I will improve the seeder so it will be much smaller. I think the main issue was the huge YAMLs. The developers need to write and they actually don't care which service account you use, which note pool it is, what your permissions they don't want to do. They just want to create the business logic and to use the tool. I think this feature helped them a lot and made it possible to actually to adopt this technology. Do you have any calculations of costs between Agu workflows as CI and, for example, circular CI? Yes. I don't want to lie to anybody. And actually, Circular CI have a huge team that are doing just the same and I have just myself to doing so. I think it's cheaper in cost in Circular CI but still, for example, when you self-hosted it, you can integrate into your already existing environment and use all your tools that you're already using for managing those pipelines. Actually, we have a lot of options. I think it's worth it. Any other questions? Yes. Thanks for the talk. Just a simple question. For Rookout, is there an open source version which I can deploy locally and try out? Yes. It's not open source. Our SDK, I think, is open source. If you're just an individual, you can try it out freely, no problem. Anyone else? Okay. Thanks a lot, Gosha. Thank you a lot. Another thing. In future, I think I will expose it in our GitHub repo, the cedar. Now it's a huge mess with a lot of best scripts and in future it will be a nice Python script. I hope it will be open source so all of you can use it and give me to the LinkedIn. I will surely post it there. Thank you.