 Hi everyone and welcome to this CNCF webinar where we'll talk about what's new in Argo workflows 3.0 That was just released here this week. My name is Henry Plixt and I'm a principal part manager here at Intuit And hi, my name is Alex. I'll be doing the demo today. I'm a principal software engineer in Argo maintainer so before we go into The the meat of 3.0. I wanted to give you a brief history of what Argo is and how we ended up where we are today So Argo is is a set of kubernetes native tools. We're running and managing your jobs and and applications with kubernetes So Argo consists of four different open source projects. We have Argo workflows, which is the The the star of the show today and then we'll also have Argo events Which is an event-based dependence manager and then we have Argo CD and rollouts Which are the declarative continuous delivery tools and progressive delivery tools That you can use to manage applications and manage your rollouts in various shapes and forms Argo started a few years ago. It was incubated at a startup called Aplatics Which was then fairly quickly acquired by Intuit. Intuit had a need to build a self-service developer platform and modernize the way we were doing our software development and managed applications So Aplatics was required and tasked to to build that to to build that and with Aplatics came the Project Argo workflow at the time as they were building out and as we were building out this these new developer platform There was also a need That we realized to have a continuous delivery platform and hence Argo CD Was born so Argo workflow was the first one and then Argo CD came shortly thereafter And then one of the Argo workflow users BlackRock later contributed Argo events into the project And as this matured, we also realized the need for progressive delivery and Argo rollouts was incubated about a year later and Premiered at kubecon about two years ago now About a year ago now Argo was Incubated as a CNCF incubating project and just a few weeks ago We filed the PR to be a fully graduated project. So the project has added A lot of features a lot of contributors and gone through a lot of Mature ingest in the last last couple of years here. So we're really excited now how far the project has come in a very short time and that's also Evidence by the the number of contributors and the growth we've had in our star count on github We're now up to over 15,000 stars We have almost 4,000 contributors of which over 900 are actually contributed contributed contributors either Active right now we have contributing the past code to one of these four four projects and they're very active slack Channel with several slack channels We have over 5,700 members currently And it's a really good spread in the community around this project from there are vendors that are stepping in and helping here That are offering the commercial support options for Argo. We have a lot of users and individuals that are using You know the the regular open source versions of this and in all And contributing to to the success of this this project. You can see just in the last year Now Argo workflow that the oldest the most mature of this project have added now 50 percent new stars in in less than a year And we've actually done Of the 350 releases that we've done in the Argo project over the last Almost four years 50 percent of those releases have been in in the last year alone There's been a lot of good growth both in terms of features in terms of contribution and in in terms of stability in in the various Argo projects And one thing we're we're also really excited about Is seeing how Argo Argo workflow in this case is getting picked up by other open source projects and and building an ecosystem of of related projects where Argo is used in a component And Argo being Cloud native at its core and having a very flexible architecture makes it easy to integrate with Makes it lightweight. It's all container-based. So it's really Easy for for others to pick up Argo and and use that as as a you know workflow engine as part of their their project And it's also a a familiarity Piece to this where the workflow as code you can use yaml to describe describe your workflows to describe How you do things and that fits in nicely with how these other projects, you know, we have platform and framework projects Like kubeflow Or kendro we have cooler the which is a A Unified interface to interact with various workflow engines underneath that's also supporting Argos all all these kind of come together and use Argo and support Argo as as a Plug-in orchestrated underneath and we're really excited to see this this ecosystem growing It's been growing pretty fast here. Just just in the last last A six six months a year here I also just want to mention, you know, in terms of the stability and the scale That Argo is used that at into it as an example since that's where where Alex and I work And this platform that I mentioned initially that a platyx was tasked of building we now 5 000 service developers that are on boarded to this to this platform and With over 11 000 applications deployed Over 350 plus Kubernetes clusters all running in AWS with with well over 15 000 nodes and So it's pretty sizable sizable architecture that we're using Argo for internally here, and we also have a significant population of data scientists that are using workflow as you know the The tool for executing workflows and pipelines and doing the running the ML ML jobs So so really excited to take this into Into the next level here with with 3.0, which is our first major release in a while So i'm going to hand it over to Alex here now to do some go through and do some demos Alex Uh, thank you, Henrik. I'm just going to take over the screen recording from you just now again Good, you should be able to see the slide. Is that right? Yep, I'll take your silence as yes, so our go workflow 3.0 is probably the largest um release of argo workflows um Since no 2.0. I expect um and the main area of focus in this is what one of the main areas of focus has been around the user interface enhancements Um, we've actually added around 20 000 lines of either new or or changed code within the user interface And i'm going to give you a demo of probably about four or five really new exciting features coming in version 3.0 and also a sneak peek at a couple of features coming up in version 3.1 So the first New feature coming in version 3.0 is is support for argo events We're providing both an api and a user interface for people who use argo events Now for those of you who are not familiar with argo events It is a cloud native and cloud event compliant system for consuming and Trigging actions based on events So a typical example might be a file being dropped into a bucket and resulting in a workflow being executed or A slack message or other messaging platform resulting perhaps in a message going to kafka for example And so being able to provide this in the user interface Is filling out a really an area we felt was really lacking for argo events because it's quite It's often quite difficult to diagnose issues in a cloud native landscape without a user interface to help you I mean if you're ever looking at logs I'm sure you you never look directly at the raw logs from kubectl You go into some kind of logging facility like splint to do so So let's have a little look at the new argo events user interface If you're familiar with argo workflows too You'll probably recognize the same color schemes and layouts here But what will probably jump out at you immediately is the fact there are a number of new buttons on the left hand navigation here Well, I'm going to go through several of these in our in this presentation So the first New area here is an area called event sources Now if you're using argo events event sources is is the thing that results in some kind of action happening All right, there's a number of different types of event sources But this user interface not only allows you to create them. It also allows you to update and list them So let's create a new one This is a pretty canonical example of an event source This one's a calendar event source that will will create an event in the system at an interval of every 10 seconds So let's create that You can now see we've got a user interface I can expect the calendar and I can go up and have a look at all the different event sources will also be listed in here The other key resource in the argo events universe is is a resource called a sensor And a sensor is something that results in an action being triggered as a result of an event source Ascending an event over so two key things event source What has happened and a sensor what what needs to happen what needs to be triggered as a result of that first item Let's create an event source now the event source here I'm going to create a little bit more sophisticated than the previous Example here it contains the two templates the templates are the triggers the first template here is going to create an argo workflow So the result of this is every Every 10 seconds an argo workflow will be created and the second one is a logging trigger diagnostic trigger That'll just you print out to the console the Contents of that particular event. Let's create that And again here I can edit this once I've created it. I can have a list of these different sensors But if I want to go and diagnose an issue, I can actually move into this new view called event flow an event flow shows The event sources the sensors and the triggers and the actions that result on them I can actually enable the event flow view here by clicking on this button And that'll show me every time an event occurs an animation will show what's going on the system And the specific goal behind this the specific use case is this just to make it easier to understand and debug issues that are going on with the system I can actually have a look at these individual ones. I can click on them And I'll get some additional Options here. This is just a view and I can go into that open edit one I can also have a look at the standard kubernetes events that may have occurred related to that resource And I can have a look at the log files here when you can see that I've got some structured logging going on here from the calendar The same thing also applies for the triggers. I can click on the triggers and I'm provided with additional information here about that as well And finally I've got some additional diagnostic information Shown on this around the number of messages that have occurred here So if I if I don't looking at the screen and look away, you can see those numbers go up as a result of it And then finally I can even go and click through into those workflows and take a look at the workflow itself and do But the usual kind of investigation So that's a little bit of an overview of argo events in the user interface as well as the new event flow capability Next I want to talk a little bit about Some under the hood changes that we made in version 3.0 So argo workflows user interface is built on fundamentally two technologies A one technology is grpc And the other technology is react And react allows you to write two different types of component One specified as a typescript class and the other specified as a typescript function What we found during our development of Argo workflow 3.0 user interface is we found ourselves wanting to fix a number of kind of pre-existing Issues and bugs and things that didn't work particularly well And what we found when we're doing that is actually it was often much easier to rewrite completely rewrite an existing class style component as a functional component It was faster to do that Than actually to fix the previous issue which is a bit of a rebuke of class components, but also I think more really just shows you the power and simplicity of these Functional components how easy the art of writes and how easy the art to debug Okay, I want to talk about another little example of a feature in here I'm going to talk about widgets Now a widget is the way to embed that bit of information about a workflow into other applications And we have a couple of different kind of widgets. I'll just show you them now If I go into the view here of my workflow I am provided with slightly different options here at the top of the screen you can see that there's a new Logs viewer that's really used for anybody debugging logs Allows you to actually look at the different types of logs from the different containers in the pod Allowing you to get a bit more insight into what's going on Not just in your container, but also in the other containers running within the pod And there's also this new share link and I click on this I will get just an example here of The widgets that Exists for this particular workflow and these widgets are animated. So When a workflow starts operating it'll it'll start, you know in the gray state if it goes becomes fail It'll advance the red state or it'll change the blue set These are embeddable and you can just have it a look at the url to embed them here as a preview And let's just open that up in a new window And here we go. That's my little widget. I can embed that if I click on it Then it'll just take me to the workflow itself Now that's kind of fun another another type of widget is related to the cron workflow So a cron workflow just a bit of a recap for people who are not familiar with this Is a workflow that's triggered on a standard cron schedule. So this example here is going to trigger once every minute If I click on that cron workflow, I'm given a share button here as well I can click on that and I've got status badges and In this example, I can see that that was already running It just went from a blue state to a green state. So that's a good example of this And this hopefully will make it easier For those use cases where you probably want to build some kind of framework or platform around argo workflows But you also want to include some of the more sophisticated and useful user interface elements One of the most valuable parts of the argo workflows user interface Is this graph stroke dag view showing the workflow as it changes and executes Okay, so that's a little bit of an overview of widgets The next thing I want to talk about is reports now reports are kind of a new feature They've been polished a bit in version 3.0, but they came in 212 So for many users if you're upgrading from 210 or 211, this will be a new feature for you The goal of a report which appears under this reports option down on the bottom left hand side Is to give you an idea of how your workflow's behavior or specifically your workflow template or cron workflow Templates behavior has been changing over time and you've provided some options on the left hand side here I'm just going to use that cron workflow Yeah And I'm provided with two graphs here The first graph is a duration graph that allows me to determine if my workflow has been taking longer over time And the second one is a resource consumption graph now resource consumption is um a slightly older feature dates back to version 2.9 and each workflow will provide kind of a high level summary of Of the ballpark of the memory and cpu or gpu that workflow is used and that allows you to track cost over time. This is this is This is specifically about the amount that was requested and therefore directly correlates to cost And so you can see for example the cost of this one is pretty much stable Your cpu and memory have have remained unchanged over time And this also works if you've got archived workflows if you are using the workflow archive This can actually look at back at historical data as well I think this will be really useful if you've got a long running workflow It's running for a very long period of time or a number of times over a long period a longer a long requested feature from Argo workflows is controller high availability The goal of high availability is if the controller that operates the workflow crashes for any reason It can take some time for that controller to start back up again It has to query all the workflows in your cluster and build some datasets But also there may just be some underlying kubernetes scheduling issue that means for some reason that controller can't start up for some time So we've introduced a feature that's quite common in other work other kubernetes controllers is a thing called leader election And this basically allows you to scale your controllers deployment to two or three And when one of those controllers goes down another one can be spun up I'm not going to show you this. It's, you know, very much a terminal level operation But it's relatively straightforward to do so you add some r back around The new kubernetes api called the coordination api, which is how we do how we do leader election And then you just scale your controller to two or three replicas And typically you'll probably also use a thing called az anti affinity to ensure each of those replicas is running in a Different availability zone for your cloud provider. This means if one of those Extremely unlikely sorry not let me correct myself extremely rare But absolutely guaranteed to happen availability outage happens I mean that will happen regardless of which cloud provider using that will happen to you one day You'll actually already have two controllers in hot standby In the two other availability zones and one of those will automatically start up and take over executing Your workflows. So even if you have an outage an availability zone outage You can you'll still be able to recover another really Under the hoods, but I think very impactful enhancement in version 3.0 is a concept called a key only artifacts A key only artifact is defined Is the same as a normal artifact But rather than specifying the whole set of configuration you only need to specify The configuration related to the key. Let's have a little look at what that might look like See if I can find this example Okay key key only artifact. Here we go So this is an example of the key only artifacts and the main difference the most users will notice is here under the Outputs. I don't I only specify the key for my s3 bucket I don't have to specify the the entire thing and the rest of the information for this will be populated from Either the artifact repository reference or the defaulted it Configurated artifact repository and they'll be able to read to that explicit key So previously if you if you wanted to not include the bucket and other kinds of configuration You actually also needed to You know, if you wanted to have an explicit key You would have needed to include that in an entirety and this is very useful for examples where I want to Um use some formatting here in in the workflow And this also has like a nice side benefit for large workflows that have many artifacts with them Then they use a large fan out This actually uses less disk space within kubernetes to create and store that Particular workflow. That's a nice side effect there related to that I'm going to talk a little bit about some upcoming features Some users are probably going to skip version 3.0 and go straight to version 3.1 and version 3.1 is going to follow quite soon I'm hoping that we'll have it Release candidate available in the next few weeks and it's going to bring Three or four particularly new um interesting features. I'm going to talk a little bit about one called data template So one thing that's always been quite difficult to do in our go workflows is efficiently create a fan in fan out Map reduce style job and data template is a new abstraction that allows you To easily do the fan out part of a fan out job and I'm now going to show you an example in our user interface here For the examples, it's data transform Here we go This data transformation Lists a list of log files in a bucket and each of those log files then becomes part of the iteration loop So this allows me to list a bucket and then start up one other step for each item in those buckets This has loads of really interesting use cases For example listing your python files in a pocket and then potentially running those python files separately Um, so let's give this guy an execution here. So let's just talk a little bit about it first Here I have the data template marked by data colon and under that I have a source and an artifact passed This this tells me which bucket to list that this is the s3 bucket and this also uses key only artifacts here as well So I don't need to list the rest of it and this just lists any artifact in that bucket with main log Then that's passed to the next step here. So that'll let'll print those to standard outs They're passed the next step as items the next step. I'm going to perform some processing on those ones So let's execute this workflow So first step as I mentioned list log files in the bucket That'll go into the bucket and use whatever api method is used It's very typically a very lightweight method that allows you to list those items in the bucket And then it'll proceed to create one additional step for each item that was listed in that bucket There we go. You can see that with 142 hidden nodes. There was a large number of log files listed in that bucket And It'll Found those out and do one operation per item that you can see how this could be really useful Your first step could produce a number of artifacts that you want to iterate over But maybe you only want to iterate over some of them, you know, you can then use that filtering operation to do that Another new version coming up is Expression tag templates now most users maybe already familiar with tag templates It allows you to substitute information From for example input parameters or workflow parameters into a particular template within a workflow Expression tag templates builds on this to allow those templates to not just be plain substitution of variables But actually fully formed expressions using the expression syntax that many users will already be familiar with Let's have a look at an example Now let me walk you through this example. This example contains one Dag template that's going to iterate over a number of numbers and print out some information in the dates The way to differentiate an expression tag template from a pre-existing template is rather than starting with just two curly braces It starts with two curly braces and an equal symbol Then the rest of the contents of that tag is an expression In this case, you can see they've got an array that i'm going to be iterating over filtering out numbers that are Not greater than one and finally converting that to JSON, which is necessary for with param And then going to pass that in as a parameter called foo Let's have a look at that pod zero template Pod zero template is going to print out some information to the standard to the console It's going to print out the word hello Followed by the evaluation of this expression This expression is based on that input foo takes it as an integer Because parameters normally are strings and multiplies it by 10 So in this example, we've got a list of one and three We're going to skip one and just have three as the value for foo and it's going to multiply three by 10 to give you 30 Then it's going to print out the date, but I didn't want to include the entire date I just wanted to include the year So here i'm using a template library that comes built into argo workflows 3.0 called sprig sprig is a very common template in system for the go language and provides a whole number of useful functions for manipulating data And this particular one is for formatting dates and in in the sprig date format function 2006 really means just print out the date with only the year and that'll be the workflow creation timestamp Let's execute this and see what happens Here's my task and you can see that the it's only running with task zero And we can have a little look at the inputs and outputs of this task Here we go as predicted a single task with a number three It runs successfully and printed out. Hello 2021 The final and one of one of the most exciting features is the introduction of a container set and emissary executor Now these are two separate features, but they work extremely well together and it's going to be rare to talk about one without the other The emissary executor is a new executor To add to the existing accuser we already have including docker and pns That basically builds on the lessons learned for those plus some additional lessons We learned from working with the team behind tecton cd an emissary executor Uses shared volumes to coordinate inter-process communication And as a result of that it can it allows us to run multiple containers within a pod Today, you can only run a single container within a pod But with the emissary executor you can run multiple containers within a pod and actually have the processes Within each of those containers wait for the process in a previous container to complete This allows you to create a directed acyclic graph of of containers within a pod to do processing And one of the one of the great benefits of this is because they're containers within a pod They can share things like volumes and also communicate together with each other over local host An emissary executor on its own is not enough to achieve this We also introduced a new template type alongside his resource container and the other types of templates It's just dag and steps called container set and a container set simply specifies a group of containers that run and the dependencies between them So let's see this in action Now there are several examples we can choose from here And the first one i'm going to choose is this graph workflow which demonstrates the ability to connect multiple steps within a workflow now One thing i'm just going to quickly highlight here is there's additional annotation This is also introduced into argo workflow 3.0 in previous of work art versions of argo You were committed to a specific container runtime executor for all your workflows within within your controller Version 3.0 allows you to use a label to use different executors The different executors have a trade-off between power security performance and so forth and this allows you to experiment with new ones without Risking the other ones or or mix and match the ones you want to use for particular workflows depending on your requirements As i mentioned this new feature is based on the idea of container set and within containers i list a series of containers And the dependencies between them Let's let's start this workflow and you'll be able to see this in operation Now one of the things i love about doing the container set demo is it's often extremely responsive So responsive it didn't appear to execute they all appear to execute once i can assure you that's not the case Because these are separate containers rather than separate pods they do not have that overhead Of waiting for the other pod to complete the communication by the kubernetes api And synchronization they'll all execute as fast as possible and it's really good for co-locating tasks within a single place Let's run another example here, and i'm just going to show you that the another one of the benefits here this Container set workflow uses a shared volume specified here that's mounted onto all of the containers on the path specified here So so x is a shared workspace And in our example, we're going to have an artifact that's produced by container a and it's going to be written into that workspace But then collected from the output There we go and here we go you should see that this executes extremely quickly as well typically within A few moments there you go. There's another example of the container set template So as i previously mentioned our go workflows 3.0 introduces a number of new features I'll just review them here for you the argovents api user interface with the event flow view for diagnosing issues and the ability to create Event sources and sensors through the user interface A significant refactoring to improve the performance reliability Maintainability and robustness of the user interface. Many users will note that you don't get as many disconnection errors in version 3.0 Control a highlight high availability through leadership election that allows you to survive Availability zone out even survive a availability zone outages key only artifacts which simplify the definition of a workflow And reports that allow you to look at the history of a workflow understand how it's changing over time Then in 3.1 data template that allow you to fan out workflows based on the contents of a bucket within side your artifact repository And conditional parameter logic, which I haven't talked much about here That allows you to select which one of the outputs of a dag or steps template Is passed downstream Expression tank templates, which allow a lot more flexibility around expressions used within side your template And the container set template necessary executor that allow you to co-locate multiple steps within a pod To take advantage of things such as the ability to communicate over local host or shared volumes Now we're still in 20 Oh, hang on. Are we 2020 or 21? 2021, aren't we? And you know, you spend so much time locked down at the moment It can be hard to remember We're looking forward to some new features coming up. So we didn't talk much about this but earlier on this year We did do a survey to understand what our users wanted We've got some really interesting and useful feedback from that And one of the ones that came back is more capabilities for working with python A lot of users use a python and are familiar with it and don't really like working with a yaml that is so ubiquitous within the kubernetes ecosystem We're also looking at the ability to To run workflows that span multiple clusters and multiple namespaces So for example, you might have a build job that builds a container in one cluster You want to run that in another cluster and then deploy some additional information in another cluster But you have to all specify within inside a single workflow And we think that'd be really useful for both machine learning and infrastructure automation use cases We also want to look at building a plugin framework that allows for integration Extension things like life cycle hooks or the ability to write your own templates We talked a bit about container set template today in the new data template The plugin framework can actually allow users to build their own templates to do whatever they want to do And we're looking at doing that with a new thing called the http template And finally we're looking to improve the whole developer experience from making it make it easy for people to work within the ecosystem So that's all I have to talk about around New features in version 3.0 and upcoming features in version 3.1 I'm now going to pass it back over to henrik to close the session Yeah, do you want to jump to the last slide? Let's see. I think there's one So I just want to thank you for for stopping by and listening to this. I hope You gave a good overview and got excited about all the new things that are coming in 3.0 as well as its sneak peek on 3.1 so There's some additional information available in the release blog that's available on the cncf blog There's also the argo website that has more information about the project yourself how to get it And we also have a very active slack channel So there is a link here to help you join the slack channel and join the conversation There's a lot of good Information being passed around a lot of good help in in a slack channel So please check out the website Check out the release blog if you want more information about what we just presented on and Come and chat with us on the slack channel if you want more information Thank you everyone