 I am Shivanshu and unfortunately the primary speaker Akanish couldn't make it through the talk due to some travel issues. But he has prepared some nice demo for us, so let's get started. So the target audience for the talk are the people who want to manage their machine learning pipelines. They are looking for managing their data and batch processing pipelines. Automation may be and they are probably already using some workflow management tool but want to explore a Kubernetes-native solution. So the agenda includes basically what would be a business use case of having workflows. What could be a requirement for an enterprise today and what could be the requirement for tomorrow? What are the available tools that they can use? What is basically a definition for a workflow for an enterprise? And we'll go through some architecture comparison between Airflow and Argo. Some feature comparisons and we'll see how monitoring and observatory works there and we'll have our demo. So let's talk about a primary use case for an enterprise. The everyone needs CI CD and probably most probably they are already using Argo there. People are adopting Kubernetes so they probably are looking for a workflow orchestrator which works nicely with Kubernetes. Integration with existing GitOps pipeline, how they can integrate a new workflow orchestrator with whatever GitOps pipeline they have. While defining workflows maybe they don't want any restricted by a language if they are comfortable with YAML which they must be because they are using Kubernetes and it's easy to adopt a new orchestrator, workflow orchestrator. Second use case could be having an infrastructure automation, data and batch processing, managing machine learning pipelines. There are many tools that can do it. So for example, Argo workflow is a very good workflow orchestrator, Apache Airflow is there, Dexter, Ketro and Prefect will discuss Argo workflow and Apache Airflow. So what actually a workflow is, it's usually a DAG, a directed a second graph and an individual node in a DAG is defining the task that you want to perform at that particular stage and then you systematically arrange those nodes basically creating a range dependency through which your data flows. So yeah, in simple terms DAG is dependencies between tasks and task is what you want to do in that particular thing. So it's a very basic example of a workflow. If you want to make a pizza, you put on toppings, you prepare the base and you put it for baking, it's a DAG. So understanding Argo workflows, so what actually it is, it's a cloud native tool to orchestrate your workflows. Some key features are it's K native by default, it's container centric, it's designed in that way, it's YAML based but it also offers some Python SDKs so that you can write your workflows in Python. So here is one of the SDKs that you can use. You can create dynamic workflows using Argo and one of the best use case for Argo workflows is it's K native, it's generalizable and can be used for data pipelines and multiple cluster deployments. What is Airflow? It's again an open source automation tool built by Airbnb and then contributed to Apache. Key features are it's Python scripting, it's task-based, it also offers excellent DAG creation and there are some pre-built connectors in it which you can use. And one of the best use case for Argo workflows is where you don't have Kubernetes, you want to run sort of a monolith workflow or capacitor, there you can use Apache because it's fundamentally not designed with Kubernetes in mind. So let's discuss about the architecture of the Airflow. So we essentially have the DAG files which are stored and then the user can create plugins in the plugin folder. The three things that are important is the scheduler, executer, web server and the DAG file storage and the metadata DB. So whatever DAG you define goes to the DAG file storage which is in sync with scheduler and like all the components with worker and with the trigger, scheduler basically handles the triggering of your DAG workflow, executer configures the property. So executer is a part of scheduler but it helps you configure the property of the scheduler. You can define your own executer or you can use pre-built executer. Web server is the UI to basically the UI to which you can interact with Apache Airflow and metadata DB stores workflow in the different states of the running workflow. And this is an architecture of how Argo workflows work. So you can see a user can either use an Argo CLI to define the workflows or can use the UI and you can also use this API client to trigger your workflows or you can use Argo events to trigger your workflows. So the Argo server exposes this through an API which goes through the API server and that communicates with the Kubernetes API server. Workflow controller takes care of managing the reconciliation of your workflow parts. It talks to the Kubernetes API server and maintains the state reconciliation of your parts. There's artifact store and workflow archive where everything is stored eventually. So if you look at the architecture for the workflow controller, it talks to the Kubernetes API server, gets the workflow from the label, puts in the queue, and then there are go-routines running which takes care of creating the actual part and the reconciliation of the worker part. And when a workflow is created, there's an init container that takes care of phasing the artifacts and the parameters and then there is the wait container which takes care of performing any task which is needed before the cleanup. So this is on a very high level how it works so that if you want to use which workflow or capacitor you want to use, if you know the internals, it's probably easier for you to decide. Let's quickly compare the features. So Argo workflow is natively designed for Kubernetes which means it's YAML based but it also supports a positive workflow. Using, for example, Hera SDK, you can define your workflows in Python. The REST APIs are available. You can define your workflows and use the REST API. You can easily specify resource requests and limits using the Kubernetes definitions. You don't need to define somewhere else. You can use the Kubernetes native things to define resource requests and service accounts. It comes with a native artifact support of S3 and Alibaba Cloud and other things. You can also define Cron workflows which means if you want to trigger a given workflow periodically, there you can define Cron workflows. You can use templating to define and reuse the workflows. Whereas in Apache Airflow, it's natively designed so that you can write your workflows in Python. It has great libraries for pre-built connectors to some common data stores. It can be run on Kubernetes but it can be challenging because it's not fundamentally designed to be run on Kubernetes. It's designed to run on a machine, a single machine. Yet it provides highly scalable solution because if it's running on a single server, it depends on the capabilities of that server, how effectively it can scale. There's dynamic workflow generation in there and it's extensible. For monitoring of your workflows, basically you can use whatever monitoring solution you are using for your Kubernetes components. You can plug them with Argo workflows. It comes up with a very nice web UI which you can use to manually monitor how your workflows are running. There's a dedicated Prometheus Metrics Endpoint for your Argo workflows. For Apache Airflow, you can use your Kubernetes monitoring stack but it does not out of the box applies to all the components that are there in Apache Airflow. It also comes up with a web UI which you can use for manually monitoring it. It has some built-in mechanisms to support observability and you can configure the open telemetry to collect telemetry from Airflow. Let's take a look at a very basic demo. What we are doing here is we are using Argo events and we are using different Argo events. It provides you many different sources like Git, Webbox, S3, Messaging Queues. In the demo, we'll be using Git to trigger an event and then it will trigger the Argo workflows to perform some things. Let's play the demo. But I need to pull this to that screen which could be hard. Give me a minute. Okay, this is not supporting audio. Hi Shavanko, let's start with the Argo workflows demo. So I have everything. So let's just start with Argo with the platform. So QCCL, GetBot, Minus and Argo. So we have nothing in Argo events. Basically, we are going to install Argo workflows first. This demo is going to be around GitHub spec with Argo workflows and Argo events. So we are going to install Argo workflows first. So the basic command QCCL, apply Minus F. So we are installing Argo workflows and Argo events with bunch of things. What about it? Yeah. Next, what we are going to do is we are going to check. Yeah, everything is coming up. But before that, we are going to batch the deployment so that we don't have to again and again move into the login screen and do the login. But in the production setup, we will have something or the other for a QCCL. So for the sake of this demo, we are just going to batch the deployment question. So yeah, so the deployment Argo server is patched. Let's see. Yeah. So everything is patched. I have already created the address for here. So once this is coming up, we will show the address. Apart from that, let's start with the installation of Argo events. So we are going to install Argo events in the Argo events namespec for QCCL, apply Argo events Minus and Argo. Let's go to Argo events as well. So controller is here. So Argo server has come up. Let's see. Let's install class. So the basic and fundamentals would be that we are going to make an event bus and the events are coming in from GitHub or any of the developers. And it's going to the event source and then the event listener or the sensor, and then it's going to Google Pro Sports. Yeah. So I have created an event source. So the event source looks something like that. The kind of event source that you have. It's we have created, we have opened it for 13,000 calls. I have given the GitHub repository and the command would be slash push and any of the GitHub events will trigger this. Let's start. So here we have Argo across templates. So this is the basic though. Let's start with installing the source. So I have already created the secrets. I am not going to show you the secret because it's obviously my setup secrets. So I'm not going to show you that, but I'm going to install event source first. So you install the sensor as well. So let's see. And I have to install the event bus as well. So it's a basic CRT with the kind of event bus. And we have made the name as default and there are three replicas and more. Uh, so going back to the terminal, we go to the space, we go to Argo events, the event bus is coming up, the GitHub event source is coming up, the sensor is coming up. If we describe this, this is a basic event source. Describe the sensor as well. It's a basic sensor. We can find this in the demo example of Argo events on Argo long communication. So we have everything perfectly. So let's go and start with the demo of pushing the file to GitHub, create a file or something here and push it to GitHub and a file, create a new file. And let's see, let me find a good word for example. So a good word for example, let's see. Argo word for example, point bus, that point bus, we will create a new file here. We are going to name it as that, that one, point bus, commit changes. We are going to wait for any events that are coming up. Yeah. So let me just share my screen of Argo course. There we see the word pro has been created in the Argo name space. We are just waiting for a point break. So when we flip a coin, if it has tails, we just keep it, if it's heads, it will print heads. We can see the logs, it was heads. Then it's going down in the DAB. So it will do it a couple of times. And then it will end part of the meantime. We are just going to see the board as a like this. So we see a new dialogue with a new port that is coming up. Whenever it's completed, it has not completed. It's going on right now. Yeah. So the whole time is completed. Yeah. So that's the demo for the touch type of Argo course. Hope you enjoyed it. And we will talk about my tracks and also related later in the next video. There's one more video where we want to talk about how to use observability and Python. So let's get started. So sharing my screen. We have a video board already. Okay. So it's QCPL, trade boards, NSN, Argo. Earlier we had installed Argo so it's going to work on Google. So that's what we're going to do. Now we are going to check QCPL, trade boards, minus 10. We have everything set up already. I'm going to show you the time. So first we are taking Argo and Google from Argo needs based. We are going to check the website and we are going to solve it on 1990. It's the basic service monitor that we have set up. We can also check the purpose and Argo needs based with the QCPL command. QCPL gets the Google F minus 10. And we have everything. So let's just go forward everything to 99. We have this here and it's on 99. So we are going to go first to the 99. And we can check how it works. And it's coming up. It's that everything. Right? Yeah, let's see if it's anything. There is anything that's back in the QCPL. So QCPL is important. And so something is there in the QCPL. And we can also check a lot of things. Let's see. I'm going to close for now. So, yeah, workflow control metrics and everything. The container is still going to control. And everything is set up. So it's time to use the Google Visualize algorithm metrics. And now let's talk about hierarchy. So hierarchy is the Python SDK called Argo. So I have an example. I have already installed the hierarchy to get from PIP. So let's do that with PIP3. I can just show you what I already installed. So I already have it. So let's just see a basic example. So we have a hierarchy for my I'm going, I have inputted the steps that will do in the script from the package and from the share package. So the group one big is for setting up the rules, the password, and if I have to add anything in it, anything. So for the host, I have set it up for those quick samples. And I have that the way I set this for us. The script that I have created is for just a hello world script. It's just going to take a hello world. So let's start with this. So what we are going to do is we are going to write the script. Yeah, Python 3-M, there are going to be only about a couple of things like adding certificates and everything, but we don't need it. And you know, we can clear it and we can see to see to get parts, and we can see a part that has just completed. So let's see on the UI as well. So yeah, this was the part which is just completed 26, 27 seconds ago. We are going to echo it. So let me know, and logs, and it says hello world. And that's your example for what I am sure you might have done something from it. And thank you for coming and having thank you so much. Thanks. I think it's probably the one of the very basic talk for our open today, but I hope there's something for you to get started with if you want to get started with all the workflows. Yeah. If there are any questions, please ask.