 Hello, everyone. Thank you for coming to this talk. This is mainly a talk on Java release pipelines using Argo workflows in case you're wondering you're in the wrong room. Now's the time to leave. All right. Did you know that there's one common library which is used across hundreds of Adobe services? And we use Argo workflows to deploy these artifacts in a secure, scalable, and efficient manner. Well, I'll be talking more about this. But before that, a little bit about me. I'm Anirudh. I work in the Adobe's IDP platform team, or internal developer platform team. And I work on this cool library called ASR. I'll talk more about ASR in the next slide. Before Adobe, in my previous life, I've been doing a lot of ETL pipelines using step functions and all sorts of BPMs in biotech. And I like to wander, talk, and debate, and all that kind of stuff. So jumping into what's ASR? ASR stands for Adobe Service Runtime. Not to be confused with a Java runtime environment, this is more on the application level. So what happens in ASR is we bundle a lot of cross-cutting concerns that you would want for a production-grade application. And we kind of ship it as a pawn. So if you are an Adobe developer who wants to go from concept to cloud, using ASR will give you all the checklist that any respectable production-grade software would require. We have six major libraries that I will use today to kind of demonstrate how the release pipeline looks like. And I'll talk more about these libraries in the next slide. And almost like we have a wide amount of services at Adobe which use ASR. So the big deal for us is security and then reliability when we deliver this software and also speed. And that's where Argo workflow comes into play, which is helping us push these artifacts super fast. And these libraries are community-driven inside Adobe. So we get a lot of community contributors. So we have a lot of PR traffic that we kind of merge and release. So a sneak peek into this is a very simplified version of how ASR libraries look like. So we draw a lot of open-source springboard in Jersey and all that libraries in dependency bomb. And that's the root kind of the library that ASR shared modules that you'll see there. And then we have an imperative flavor and a reactive flavor. And the imperative flavor is the very classic, the old school way of doing things. And the reactive is the cool kid in the block. If you have an application which is low CPU, high throughput, that kind of stuff, you would use a reactive flavor. And you can see the dependency chain. Like if you make a change to any of the nodes in this dependency graph, we have to release everything below that. So if I made a change, like a spring boot update, I'll have to end up releasing every single library right here. So that's like on a very simplistic way of looking at how our workflow looks like. The development process for these libraries is pretty much like any open-source library. The maintainers, us, we focus on security patches. And then we have our own roadmap, which we think are like moon charts, are like big mountains that we deliver. And then we rely on the community for any feature request, and then we kind of crystallize that into the roadmap. And we also rely on the community for enhancements. If they think there's something minor, they can just raise a PR. And that's where most of the developments have happening these days. Right now, it's completely community-driven. So then there's a concept of connectors in ASR. Connectors are like SDKs for internal Adobe services. If one team has to talk to a service, they end up writing a connector. And then ASR is the place they come to share it. So it's like kind of that internal CNCF, if you want to say that. Now, since we have a lot of PRs that are coming into these libraries, naturally we have to handle a lot of PR traffic in terms of merge conflicts and making sure that the releases go well and all that. Our previous solution was two weeks' release kind of ceremony. And we used Jenkins for that. So every time we did a release, I had to go to therapy. So it was super painful. And I'll talk more about what the pain points were. So we had one Jenkins job per library. So that means I have to babysit every single release library until it finishes. And Jenkins kind of works out of that plugin model. Everything is a plugin. You want to pull a repo. Then you have to use the GitHub plugin. You want to publish to an artifactory. You have to use the artifactory plugin. What this does is it kind of makes your kickstart like the hello world is easy. But when you're stuck in something deep, some problem, debugging is hard. One classic example is artifactory did an upgrade. And then we used to get a data dump on the kind of a metrics that we used for dashboards. The whole thing broke because the new upgrade of the plugin kind of stopped doing that metrics dump. Things like that. So we want visibility on what we are doing, which was kind of hard in Jenkins. And also Jenkins relies on the Jenkins agents, which is kind of VM-based. So every time we need to do a JDK upgrade, or sometimes we swap out vendors based on the contract, it was super painful to do that in a VM. So a little bit about the release steps. The release itself is pretty straightforward. We pull a repo, do a Maven install, make sure the build works. And then we publish to an internal Adobe vulnerability kind of a dashboard where, based on the palm and the dependency, we get ticketed. If you have a vulnerable library, then we go back and fix it. And then we deploy to the artifactory. And we also do the basic release notes and all the stuff. But any open source library does. So what were the expectations from this library? The maintainers as us and the community, we have two different, and we're trying to merge them. So the community wanted everything fast. They wanted everything yesterday. They raised the PR. Hey, where is my release? I want to use it today. And also, they want to use the latest dependencies. OK, I have this cool feature I want to use. I don't want to wait for your two-week deadline to release. And us as maintainers, we wanted to work on cool things and not worry about babysitting a release. So we wanted it to be self-service. And again, the pipeline itself was too hard to troubleshoot. I'm not going to Jenkins. Why is it hard to debug and looking at all the logs and all that stuff? And I think the key thing was also documentation. If any maintainer wants to do a release, it was hard with the Jenkins setup because you don't have a graphical view of the process. And of course, controlling costs is always a good thing in any organization. So thankfully, Argo had everything that we were looking for. It has DAGs. It is GitOps-based. It gives you all the benefits of Kubernetes. And it uses Workflow as code. So let me jump into what exactly happens with Adobe IDP before I actually show you the solutions. So Adobe IDP is the Adobe internal developer platform. Here, any developer at Adobe can come in with the concept. We have Portal, which will help you generate code. We have different flavors. Just like the starters for a spring application has it, we give you starters like Java, Node.js, and Python. And then we have something called blessed-based containers, which are secure Adobe vetted containers, which every single application at Adobe uses, or more or less. And then comes your next step is your deployment configuration. So this is where you decide how many environments do I want to have, and submit all the PRs, the deployment statuses to stack channels, things like that. The next step is vProvision. You are based on this deployment configuration, whatever you mentioned here. And then the end result is a CI-CD pipeline. Again, I won't go into the more details. If you're interested in it, we have an Adobe IDP workshop tomorrow. Please feel free to attend that. We'll cover more in detail over there. And also, please attend the keynote today in the evening. And also, we'll talk briefly about Adobe IDP. So I mean, I personally use the Adobe IDP to spin up an Argo CD application. But it's also easy to go through the docs and do it yourself if you don't have such a big setup. So the first step is we need an Argo Git repo in our world. That is, this is where all the configurations are kind of maintained. So we make you fill a form, and then everything is stored in a Git repo. And we have a service called Provisioner, which in itself is an Argo CD kind of a workflow runner. What it does is it follows the Kubernetes Apply principle. Like it takes your configuration and manifests an Argo CD application based on what you enter there. So that's a very first step where you get an Argo CD application. So the big picture is you have an Argo CD application. And what it is doing is it contains all the definitions of the workflows that you're using to release. So every ASR Git repo, whenever there's a merge that's happening in the repo, it's going to send events to the Argo CD app, which is going to trigger a workflow. So what does the journey look like from PR to Artifactory? So step one, you raise a PR. And then if the maintainers and everyone is happy, you merge it. And then that's where the events start flowing to the Argo CD application. An Argo CD application has, I mean, I've kind of oversimplified it. It has two main components. There's a sensor and a workflow template which kind of intercept the GitHub events and then trigger an appropriate workflow which goes to the Artifactory. So the sensor that I was talking about, it looks like this. So in very simple terms, what it's doing is, is this event coming from XYZPR or repo number one? And then does it belong to the main branch? Then submit a workflow, because you'll be getting a lot of traffic to those events. And the reason why I put this snippet is it's super simple to configure this workflow, sorry, the sensor. And that's where the magic of the whole Argo ecosystem is. Everything is describable as a YAML. And it's kind of like, for me, the aha moment was when it's algebra versus geometry. You describe an equation and that becomes a graph. Something similar, like you have your YAML here and then that becomes a graph on the right. And the next step is submitting the workflow. So what's happening is, once the filtering is done, there is one workflow for one repo. And I'll be covering about how each and every single repo looks like, the workflow for each and every repo. And then we pass some parameters that are required for that workflow. So if repo one is being deployed, then I need to know what was the repo and what was the GitHub commit and who was the person who merged it, things like that. Now, the release process. Now, I really got to use the power of Argo workflows, because I could paralyze a lot of tasks. Before, I had to babysit each and every step and then go manually. Right now, I have this kind of complex workflow where I can paralyze certain things, which was kind of linear before. And also, Argo steps, some of the steps are impotent and retribal. So this can happen in your workflow. Like, if you're talking to GitHub, your enterprise GitHub might be down and then you don't want any long running processes or sometimes our Adobe security dashboard gets too much load. Their SLA is not the same as our SLA. So you might want to throw in some retries. So this necessarily doesn't speed up, but it helps you be hands-off in your management. So the workflow template itself is how, on your left, you would describe all the steps that happen in each and every in your repo. So this is a workflow template for one repo. So in my one repo, the steps that I talked about in the previous slide have been translated into workflows inside Argo. And this is how a deployment would look like. And I just threw in on the left side to just to show you that these are reusable and then these are kind of simple if you kind of zoom in on it. So my favorite part is how we can control the characteristics of an atomic step. I mean, that's how I look at it. So an atomic step is that one job that you want to split across if you have many jobs. So here, for example, one step which says verify git commit. On my right, I'm running a BAS script for this particular step. I get to choose my image that I want. So this is where if I can go Alpine 1.2 based on any different step and kind of what meets the needs. And then sizing is a very cool thing here because not every step requires the massive amount of CPU or memory, so you can kind of right-size them. And resiliency, which I talked about, if you want to retry certain steps, you can throw in a retry kind of constraint there. So this is probably the most important slide in the stock, I would say, because what's happening here is this is how a comprehensive release would look like. If I do a release or if I do a PR in the root node or even if I have a change of a spring boot version, I have to end up deploying every single repo right here. And on my right is how the DAGs would look like. So it's a DAG of DAGs. So even though one repo is like one box that you see on the right, when you have to chain all of them, this becomes super powerful. Now I have a complex workflow calling another complex workflow, kind of a sort of inception, if you might say. And the beauty of this is it's kind of self-documenting. If someone asked me, hey, how does your release pipeline look like? You just come in and show that person this workflow and it's kind of self-describable, not self-destructive. So the other beauty of the DAGs is these are dispensable. What I mean by that is if you have a traditional step function or anything, those feel like heavy when you write that workflow, because the ease of use in a Kubernetes world, like the Argo CD world, is completely different. So you could have a workflow where you can have a lot of if-else's and make it complicated or just write another workflow. So what I mean by that is the previous slide that has so many workflows, if I want to raise a PR in many of the child node, I just want to release the bottom two after this. So I don't have to release the whole comprehensive release. So you can chop up the workflows and maintain them as independent workflows, which do its own releases. So right now what I have is every PR raised to anywhere in the node gets released. And each of them are self-living workflows that kind of are sub-trees of the same parent. And of course, these are kind of written in a library fashion. So when we do these workflows, these are actually libraries being used from here. But really the efficiency that we gained is in this chopping up of these sub-trees and then making them kind of their own organisms, like deploying that workflow. And troubleshooting is a big thing in Jenkins world, which we face a lot of problems. Because something breaks. I have to start from the line one of the logs because it's not really transparent to know what's happening. But here, since my steps are broken down in a graphical manner, now I know which box that I need to focus on. And that I just need to focus there. And then I can click on the logs and I know what's happening. So overall, the workflows, like what it gives you beyond what a traditional pipeline would give you is the configurations. So you can do retries, which we talk about. Sometimes our tests are, since these are community contributed tests, some tests sporadically fail in some library. Then it works on a retry, mainly because someone had to tune their timeouts in their tests. This is largely seen in a reactive world where when they write, not everyone follows the best practices. And your tests might just run the next time. And also, it kind of depends on how loaded the CPU is. So the retries are super helpful in retrying those reactive tests. And timeouts, again, these are any standard resiliency feature, like if you have a long running process, because you're waiting on some remote to do something. And times you can timeout based on some configurations there. And I guess the cost saving is something which is we never really thought we could achieve it. Because of the right sizing of the CPUs, now a reactive library, which takes less CPU, can be allocated less CPU. And similarly, the imperative ones is you throw higher CPU and memory over there. Again, unit tests also based on how massive the unit tests are. Like, we have unit tests which run springboard application for every single test out there. And you might want to size them appropriately for that. And the compliance, and usually which is the boring part for developers, which has kind of made it easier, because everything resides as workflow as code. Now, if some security review happens and you're asked for, hey, what's the version you're using? You can just hand over that particular code, and you know what container you're using, and where it's deployed, and all that. So the audits are easier. And similarly, the swapping out of JDK vendors was one pain point for us. Now it's just a container swap out, and you're done. I guess the other problem that we kind of overlook in Jenkins is my namespace is my workflows. So once what happened is somebody accidentally deleted my container, and my whole workflow stopped in stem functions, like in the AWS console. But in this Argo world, your workspace have your own workflows, so nobody is going to mess with that area. And then you don't end up stepping on each other's foot. So the workflows are not, I mean, as you can see, it's very generic. It can be used for anything beyond a Java release pipeline. You can have a serverless CI CD pipeline, if you have an S3 bucket, and also ETL pipelines, if you may. So to conclude, most of our gains game from GitOpsification and using KITS to its fullest. Usually, I thought, OK, Argo is going to make us faster, but what ended up happening is the ease of use and workflow as code was super powerful for us. DAGs are a beauty. I mean, so many things I can go on and on about DAGs, because it's so painful when you don't have a visual view of what you write. And finally, I mean, cost and rightsizing is something it's easy to get your manager's approval for if you're using this, because, oh, it's safe cost. Go ahead, do it. Developer delight, I guess, that's in it for us. I mean, the whole process has been so smooth and enjoyable. The community is obviously great. They have so many examples. I mean, I can go on and on about it. But then the ease of use of Argo workflow is what's something which is very good for me as a developer. I guess that's probably your experiences also going to be. So that's it for this talk. If you have any questions, I'll take them. But my call to action is try Argo workflows beyond whatever CSAD pipelines or anything, get creative, and share it with the community. Thank you. All right, I'll take any questions if you have. And please come to the mic so that the live audience can also hear that. Hi. In your maven step, did you do any sort of optimizations to make that any faster? Oh, yeah. So the maven steps, we have the concept of threading. That's something where a Jenkins jobs wouldn't do it because you couldn't justify the CPU allocations there. We could multithread some of the maven jobs by just increasing the threads there. That's probably it. And then the value, I think, our main problem, the slowness, the build itself for fast, it was just not. It's just the whole end to end supply chain which was kind of slow because of not having a descriptive way of looking at things. But yeah, I think the multithreading helped there. If there are no more questions, I guess we could end this. I mean, we are there at the booth. Feel free to come and ask us more about this. And also, there's always Twitter. Please tag us, and then we'll be happy to answer any questions. Thank you.