 Alright, hi everyone. Today we're going to be speaking about a project that we created to enable notifications features for the Argo project. Alright, so my name is Remington Breeze, and I'm one of the maintainers of the Argo project along with Alex here. And I'm also a software engineer at a company called Acuity, which we launched this week. And Acuity provides vendor-supported distribution of Argo, in addition to support and services that are around the Argo project. Alright, Alex? Hi everyone. So my name is Alexander Matyshancev, don't try to pronounce my last name. I'm a principal software engineer at a company called Intuit, and I'm also a longtime maintainer of Argo. I've been with Argo for like maybe five, six years already, worked on Argo workflows, rollouts, and currently my focus is Argo CD. Alright, and a few words about a company I work for. So Intuit is a US-based fintech company, and Intuit is a proud maker of financial products such as TurboTax, Mint, and QuidBooks. So we serve 10 million customers in all segments, and I'm happy to say that Intuit is a gold member of CNCF. Alright, enough introductions. Now we can get to a topic. So as Remington mentioned, we want to speak about notifications and Kubernetes and Argo. So I want to start with Argo, and you might know already that Argo is not a single product, it's a family of projects. And so we have four currently, and it's really important for us that every project is focused on a single use case. So for example, Argo CD delivers GitOps function as a service, and Argo rollouts, complements kind of Argo CD, and allows you to get so-called progressive delivery, which includes blue grain deployment, canary deployment, and way more other features. So Argo workflows and events together focused on batch processing use cases, and help people with data processing, machine learning use cases, and so on. And so we're really trying to keep this principle to stay focused on a single use case, because that's what help us to stay light, stay keep project lightweight, and keep the high quality bar. At the same time, it's hard to argue that notifications is a very important feature for the good end user experience. And as you might notice, notification was not mentioned as a focus when I described any of these projects. And so it's kind of a problem, and we've heard a lot from our end users that notifications feature is needed. And the next slide is to kind of just to prove that it's not just our imagination. We've got a bug report, a feature request about notifications pretty much in each and every project of Argo organization, GitHub organization. And so it was just clear that it's no longer okay to just keep ignoring them and we had to do something. And so we did, we decided to try to investigate and see what is it we can do so we can deliver a great user experience, enable notifications, but do not sacrifice the quality and keep projects light as they currently are. All right, so and in this talk, I will walk you through our journey that we did last year. I will explain to you which options we tried, which worked, which didn't work, and you will see in action the solution that we are pretty happy about. All right, so I'm going to start from step number one. So before we even consider building anything, we just decided to see if there is anything in the Kubernetes ecosystem. It already works and probably can be used by Argo users. And we can see we checked a lot of projects. I just highlighted three of them here. The first one you probably already know and probably using it's a Kubwatch project by Bitnami Labs. And we really enjoyed when we tried this project for the first time. It was extremely easy to get started and get notifications for built-in Kubernetes resources. However, we run into a couple of gaps and one was that unfortunately Kubwatch do not support custom Kubernetes resources. At least it didn't support it at the time when we checked it. And so I think that was a blocker enough for us because Argo is a set of custom Kubernetes resources. Next, the simplicity is great, but we heard a lot from our users that customization is also very important. And we could not find it in Kubwatch. So we decided to check another one. And the next project was Prometheus and alert manager. And it felt like a good solution probably. At least you cannot complain about lack of customization. It's possible to configure a lot using Prometheus and alert manager. But we got some pushback from end users when we mentioned this option. One was apparently not every Argo user uses Prometheus even though I think it's the most popular tool in the area of monitoring. And so it just felt wrong to ask a person to install run and maintain Prometheus just to get notifications for Argo CD. And the second problem was that you cannot kind of provide all the information you need to construct a notification message in form of Prometheus metrics. So it was again kind of not a good fit for us. And so the last project that I want to describe, it's something that we built in the past and we just kind of forget about it. And then so the project I'm referring to is Argo Kubernetes Fire. It was a hackathon project that was created by one of Argo team members. And it was there for a while. And so what we did, we tried to analyze user feedback and understand what was wrong. Like why didn't it work and why didn't we just use this project. And so as opposed to Kubwatch, Argo Kubernetes Fire is extremely flexible. But according to feedback from users, flexibility doesn't mean ease of use. And so it was just too difficult to use it. Each and every user was supposed to come up with thousands of YAMLs of configuration. And so we decided to maybe take it into account and make sure it won't be repeated in the final solution. And so, yeah, it was kind of clear we'll have to build something. And one more step back and we decided to do a deep dive and speak with our user, analyze all the bugs that were filed in GitHub repo. And this is how we felt, you know, like after a day or two of this exercise. There are a lot of things we learned. One of them was apparently notifications are not just Slack and email. A lot of people apparently don't use Slack. A lot use MS Teams. Some use Telegram, Google Chat, Rocket Chat. And actually I can continue a lot. And it would be just the first type of notification providers, text-based. But for some people, notification means Jira issue. Pager duty, maybe it's a comment and pull request. And some users said we just want to call an API server and custom, do a custom REST API call. So as you can imagine, it's like it's tough to support all of these, especially if you want to support it in four projects. So it's a lot of coding work and a lot of dependencies that you have to introduce. It was just the beginning. The next set of challenges, not problems, was that every user has opinion about when the notification should be sent. Some wants to get notified about everything. Some only wants to know about important events. Important means like difference for everyone. And absolutely everyone wants to be able to customize the notification text. So that was another kind of challenge we had to take into account because at best we probably can go through most common use cases. But it would take us like forever to iterate and codify all of them. But it was not waste of time. We analyzed all of that and we noticed that we have different set of repeated requirements from two personas. So we have a lot of users who use Argo. So we have platform operators and application developers. And from every platform operator we heard set of requirements that indicated that operators are very opinionated and value flexibility. And you can understand that it's a drop of a platform operator to implement use cases that are important for the organization. And they value flexibility, which means they're willing to configure, spend a lot of time configuring what the use cases, the functionality to implement use cases they want to implement. And they are fine to write YAML but not Golan code. And it's understandable. And so as opposed to operators we have application developers who value simplicity a lot and they are less opinionated. Which means they do not want to invest a lot of time because they're writing and user applications. Plus they are usually fine with what platform operators want them to use and it feels like a very good fit. And the only additional requirement from application developers was that they do not want to file a jury ticket. This is just not okay. They want some simple way to enable the notifications feature for themselves and they don't mind to write a little bit of YAML as well. And that really helped us and eventually after some brainstorming we come up with a set of more or less clear requirements. We understood what is it we want and on a high level like we basically realized that we want flexible solution, which means it's driven by configuration. And we expected platform operators to spend some time writing the configuration, the project maintainers can help operators as much as possible and deliver good getting started set of configurations. Of course we wanted to support a lot of notification services and which is very important we knew we are going to add more and more and more. And we wanted to kind of introduce some interface that encapsulates complexity of different notification providers. And second set of requirements related to end users we really wanted them to, you know, consume it as easily as possible. And so the requirements are great way to start developing which we did during last year. And I'm happy to say that we have something real to show you today. And so I'm happy to talk about the project at which we call Argo Notification Engine. So Notification Engine is a goaling based library which is already used by Argo workflows, Argo CD and Argo allows workflow support is coming hopefully soon. And so Notification Engine already supports all these notification services which you see on the screen. And this is not the full list. There is a new release coming soon and the final version of current version of Notification Engine supports more than a dozen of such services. And what is also very important Notification Engine comes with a opinionated user experience which means any user who configure Argos identifications no longer need to learn a new set of, you know, configuration parameters to configure it in Argo allows. It would feel exactly the same and basically we don't expect that any additional education would be needed to consume Argo or allow notifications. And here is the repo. So if you're interested in learning about the engine, feel free to open this URL. Don't hesitate to give it a star. It needs stars. And I guess that's it. Now I can go a little bit deep into and explain how it works before we start the real demo. Okay, so here you are seeing a high level diagram that demonstrates components of Notification Engine. Notification controller is central part of it. What it does, it consumes the configuration provided by the operator and stored in a config map and some authentication information stored in secret. And it continuously watches custom resources. And what it does, it basically, as soon as it realized that according to configuration, an important event happened, it would send the notification and the information about who is interested in the notification is stored in a Kubernetes annotation, which is managed by application developer. And as I mentioned, Engine supports a bunch of notification providers out of the box. I realize it's maybe not so easy to understand. So I have few PML snippets for you, which hopefully will make it more clear. So what you are seeing now is a real life example of a configuration for Argo CD applications. Argo CD notifications. So the operator supposed to pretty much configure the following to enable notifications. So first of all, Engine needs to know when to send the notification. And we introduce something we call a notification trigger. And so notification trigger is just a function that has a name and this expression that returns that consume custom Kubernetes resource and returns true or false. As soon as function is a result where you changes from false to true, this is when a notification engine sends a message. So the next step is it needs to somehow produce the content for that notification. And to do so, it uses another configurable part, which we call notification template. And name is self-explanatory basically. It's another function that consume a Kubernetes resource and produce a message. And example here is pretty simple. It just produce a string that has some information about application including name, including status. Yeah, so nothing complex here so far. And the final step is to simply provide credentials that are required to talk to the notification provider. In this particular case, we are configuring Slack and the real token is stored in a secret that can be referenced in a config map. And so I promised some encapsulation. So here it is. The notification template not necessarily defines just a boring text. It can be something specific to the notification service. So in this case, it's a Slack attachment. Sorry, there is a lot of YAML, but this is what it takes to generate an attachment. And on the right side, you can see a really beautiful Slack message. And we support, we have something for almost each and every notification service. So hopefully it makes more clear what operator has to do to configure the notifications. So the next step is what does end user application developer has to do to start using it. And this is not much, basically the end user just have to apply an annotation. And in that annotation user have to specify the trigger name that knows about when notification should be sent. The end explain where we want to get notification to be delivered. So in this case, user wants to be notified about successful application things, which is the trigger part. Next, user wants to get notifications in Slack. And the destination within Slack are two channels. Yeah, and I think we're really ready just to see it live. Thanks Alex. Alright, so before I get into the demo, I'll go over at a high level what we're going to be doing. So first, what we're going to be configuring is notification for an Argo rule out. So there are three things that we need to configure notifications for rollouts. So the first is the latest version of Argo rollouts. Version 1.1 includes support for Argo notifications through the notifications engine. So to install that, it's pretty simple. You go to our GitHub, download the installation manifest, and it's a simple QtTail apply. The second thing that we need is, as Alex mentioned, there are two personas. And the first is from an operator perspective. So what you need to do as an operator is configure Argo rollouts notifications config map. And then the third thing that we need to do is configure, as a developer, we need to configure our rollout with an annotation that Alex demonstrated to set up notifications for that specific rollout. Alright, so I'll go ahead and show you what that looks like. Okay, let me just stop. Mirroring? Alright, so the first thing. You can see here I have a Argo rollout dashboard running locally on my machine. And up here in the upper right-hand corner, you can see that I'm running version 1.1. So we have what we need there. Next, let's take a look at the config map that we have to set up. So like I said, you have to name it Argo rollout notification config map for it to be detected by the rollout controller. So let's start from the bottom. So we have this service.email. So that's what we're going to be setting up today in email notification. And it's pretty straightforward. So we have a username, password, et cetera. The variables prefixed with a dollar sign are being injected from a secret that I set up previously. I'm not going to go over the secret right now, but it's pretty straightforward to set up. And then we have some straightforward SMTP configuration. And this allows the rollout to be sending emails on behalf of my personal email account. So that's all there is to it there. Okay, so let's move on to the template. So you can see here we have an email stanza. And this contains email specific notification configuration. So in this case, it's a subject. And we also have the message stanza. And this is going to be shared across many different notification services. So this allows you to basically add, say a slack stanza here or a telegram stanza that has slack or telegram specific configuration, but you have a common message that's shared amongst them. And another interesting thing to note here is this piece between the curly braces, which is injecting information from our rollout spec or our rollout metadata. Excuse me. And it's just going to inject the name of the rollout. You can get a little fancier with this, but we'll keep it simple for this demo. And then finally we have the trigger. So this is a trigger called on purple. And it's going to be sending this on purpose or the my purple template that we just configured. And that is under the send field. And that's going to be sent when the first container in the rollout spec template is set to a rollout demo purple image. So like Alex said, as soon as that statement is set to true, that's when that notification is going to be sent. So yeah, that's all there is to it there. So let's move on to the rollout configuration. So the only thing that we really care about here in this rollout is this annotation, which is very similar to the one that Alex showed you for an Argo CD application. And there are three important parts here or rather four. So there's this subscribe bit, which basically says that we're going to be subscribing to notifications for this rollout. And then there's the on purple, which is the trigger that we set up in that config map that I just showed you. And then finally the email service that we also set up in that config map. You can set up again other services like Slack or Telegram or what have you. And then finally the last portion that's important is this email, which is going to be the recipient of the notification. All right, so that's all we have to do to configure notifications for a rollout. It's pretty simple. You would just give detail, apply those resources or however else you apply manifest your cluster and you're good to go. So let's see that working. All right, so I have the rollout dashboard back here and you can see that my rollout is running a rollout stem of blue image. So I'm going to change it to a purple image. Okay, so now the rollout, the canary replica set is running the purple image. So let's see if we got a notification for it. So let me refresh my inbox and you can see that we have a notification. Like I said, this is sent from my personal account to a demo account and our rollout rollout stem has a purple image. So yeah, that's all there is to it for email notifications for our rollouts. Awesome. Before we wrap up the demo and move into questions, I did want to share with you some additional information and I will use slides to do so. All right, you can see my screen. So basically one important thing about Argo notification engine. I already mentioned it before, but I will repeat it again. So the engine is not just for Argo. We build it for ourselves and we consume it, but it has pretty much nothing Argo CD or Argo rollout specific or workflows. And so it's perfectly fine to use it to enable notification and get the integration with a bunch of services out of the box for any other custom resource. And here are a few links that you can use to try it. So of course you will need, it will be useful to navigate to Argo notification engine repository and read the documentation. And as soon as maybe you're interested to build something, you can check this example that supports notifications for a certificate CRD of a search manager. And it basically, there is one goal and file that has pretty much everything you need to get it working. And so if you want to do one step further than just go get GitHub Argo personification engine and the work on your project. And this is it. Thanks for listening and please ask questions.