 Welcome to the KubeCon at ClownEditCon Europe 2021. This talk is about building MLOps POCs and sandbox environment using K3S and Argo. I am Sergio Mendez. A little bit about me, I am an operating system professor. I am ClownEdit Ventusias. I really like to play with ClownEdit technologies. I do some DevOps at Yalo. I am the organizer of ClownEdit Guatemala. I am a linker, the hero. Let's get started with some cartoons first. Showing how we depend on technology and these things and how machine learning is going to substitute all the works in the future. Basic concepts. You have to see some basic concepts before jump to the technical part. POC of proof of concepts. POCs use it to evaluate different kind of technologies, solutions and context and learning goals and that kind of things. So we use a POC to compare different technologies in order to solve a problem. So we experiment and do things to determine which technology is better to solve that problem. Sandbox environment is pretty similar than production but it's just for testing purposes. So you can run entrusted software without risk there. It's a controlled environment and can be used for testing, POCs and you can have access to some similar data from production. Each computing in the other side refers to let's say that you are close to the source of data. So if you want to process information there, let's say that you are close to the data and you want to process information with your smartphone. Let's say it could be possible but that doesn't mean that the cloud is going to be disappeared. That means that the cloud is coming to you because you are going to process the things with your own devices. Each computing reference to the distributed geographic computing that things. Some of these cases for each computing could be machine learning, IoT, applications, data processing games or any workload. Talking about DevOps, DevOps refers to the collaboration between the operations and the developers and this kind of workflows where you are going to build tests really so far in a really fast way with CI CD pipelines and that kind of things. A pipeline that is a really common concept using on DevOps. Here is a concept that shows what is a data pipeline but it's pretty similar. That let's say that is a series of steps that maybe the output of one set could be the input from another step. It's really awesome to understand that pipeline in that way. The concept. MLOps in the other side refers more to the automation of the model generation. There's really more about how to design, build and manage these processes that you are generating a model, getting predictions in order to push your ML power software in some way. This diagram refers pretty, pretty close to the way that there are some diagrams on internet about DevOps and refers really, really nice. In the context of MLOps, MLOps contains the data, the model and the code. Let's say that the data change, you have to regenerate the model. Maybe you can do some modifications to your code. Maybe you want to regenerate your model too. This includes the whole cycle of the MLOps in general. So there are difference between DevOps and MLOps. Let's classify it. There are different users for DevOps. You are going to see developers, operators, let's say SRE DevOps, cloud engineers, MLOps guys, data scientists, MLOps engineers and that kind of thing. So DevOps focus more on developers and MLOps more on the science, the start and data. So they are pretty, pretty different. The artifacts that DevOps generates or they are getting touched are more like GitHub repositories or Git repositories source files made with multiple programming languages and MLOps is more focused on machine learning models and data sets. DevOps use different technologies. It's common to hear that people mention front-end frameworks, back-end frameworks, languages, databases, like VJS, Python, Java, MySQL, etc. In MLOps context, the people mentioned more models about data and technologies like Python, scikit-learn, TensorFlow, that are libraries to generate machine learning models. And Jupyter not looks for the experiments that purchase part for data processing and the whole ecosystem of Hadoop. And why not Airflow? It has different goals. DevOps looks more for the software, deploys some application on the cloud, and have some quality assurance for the software, cloud deployments, CI-CD pipelines. But MLOps is trying to automate the model generation, the predictions, the inference, and optimize that kind of workflows. They are like, let's say, two kind of experiments. They recognize the static experiments that are ready to be passed to production environment. So the code is not going to change anymore. It's ready, it's stable. So maybe you want to schedule this kind of experiments periodically. We claim only yearly. There are dynamic experiments that are under development by data scientists. The code is constantly changing. So it's stable and not going to change. There are some benefits for MLOps in the context of the containers. So the containers can bring you package the logic of the experiments, can freeze the library versions, can give you that portability for the experiments. And maybe you can share these containers with experiments across the different teams of data scientists. And why not implement that kind of GitOps to generate these models and these experiments, or to automate these experiments. Let's remember that MLOps is not only technology, it's people, it's experiment, it's data. You have to organize that people that experiments on data in order to get valuable predictions. And in order to implement successfully MLOps environment or code tool. There are some technologies for POCs and machine learning pipelines that you can use. K2S, for example, is a certified Kubernetes distribution that can use ARM architecture that is ready for IoT and H computing. It has support for Intel architecture too. The amazing thing is that K2S includes the whole components of the regular Kubernetes clusters in only one binary. It has several different backends like SQL, etc. It includes Ingress, Prinstalle, Helm support, network support, continuity for the containers. And please remember that if you are using ARM architecture devices, you have to recompile your libraries or applications if needed in order to support that architecture. And then you will be ready for H computing. Because this kind of architecture is really commonly used in this context of H computing and compile or performs this kind of workflows. So the companies right now are migrating to H computing to reduce costs for their workflows. Argo workflows in this side is a piece of software to create this automation, these pipelines. And it's designed to run on Kubernetes. There are different features for Argo. You can execute a step inside that container. Your workflow or your DAG can be defined in a file like a simple directed a secret graph, a pretty simple syntax using YAML. Argo is Kubernetes native, Cloud and Nasty. You can use Argo workflows for different things instead of machine learning. Just implement our regular CI CD and with less complexity. Argo CD in the other side is a complement for Argo workflows. You have to use Argo CD just for continuous delivery and deployment. And it's designed for Kubernetes. This supports different formats of configuration like hemp chart, customized JSON files, YAML, or why not your custom plugins for your custom configurations. Don't forget Argo events and rollouts. It's part of the whole Argo ecosystem to define a really, really nice CI CD or MLOps workflows. You can use K3S as a lightweight environment, plus Argo on the edge or maybe for POCs. Argo could replace Apache Airflow maybe, being Luigi MLflow or other ML solutions like SageMaker, or maybe it could be a complement for your current solution. The benefits is that with this kind of technologies, you are going to spend less money for POCs. Instead of a cluster, why not a virtual machine with K3S. Using K3S and Argo and Argo workflows and CD, they are prepared for edge computing and IoT. They are Cloud and Nasty, pretty lightweight and high scalable. And they are no vendor locking because you are using open source. The demonstration, we are using Argo workflows and Argo CD installed on K3S. One node cluster, let's say, is not a cluster. We are going to execute some simple pipelines and we are going to show the code that we are using for that. The architecture of this demo is like their first step for the basic MLOps pipeline. It's reading a CSV file with the scores of students. It's going to predict the final score. In the extraction transformation loading step, step number one is reading this information, cleaning this information, deleting columns, generating a processed CSV file. That is reading for the step number two for the training and generate this model with that file. And this is going to generate this model. And these files are going to be uploaded to a bucket on Google Cloud as a temporal storage. And the step of the deployment is going to read this model and is going to call Argo CD that detects a Helm chart that creates a deployment using this model to expose this model as an API with an endpoint, with an egress controller. And for the inference, we are going to access this endpoint to get these predictions and generate the final CSV file with these predictions. Let's call it the inference. So let's move to that part. So right now, let me start Argo workflows. Let's access the UI. So right now it's ready. Yeah, so right now we have the Argo workflows. Here's the dashboard and the things. There are several current workflows that were executed in the past. Here's Argo CD. You have to configure your repository on GitHub that detects the Helm chart. And you have to create an application. And here it's going to show your deployment, regular deployment with the reference revisions. Argo is going to perform that deployment. And let's execute the very simple pipeline from Argo workflows. So in this side, let's move to the repository. Let's say Argo workflows has a CLI that's pretty, pretty awesome. Let's clean the CLI. So it's going to show you the steps here and the command line interface. So this is really, really nice. So you can visualize this thing in the Argo workflows right now here. Let's reload this part. And finally, it's going to show you, let's zoom a little bit this. This step is forgetting the steps that is running in parallel. It's going to show the log here. And basically call a whale container with a custom message in the logs. So which step has their own log with a custom message. And it shows the same thing. Yeah, that's a pretty basic, basic pipeline. Let me show you the basic MLOps workflow that we already have here. Okay, here's the whole MLOps pipeline that is going to execute the process that we're explaining the architecture of the demo. Let's select queue.com, test one. We are going to execute this part. So this thing maybe will need more space to show training and these things. And maybe it's going to need a little bit of more space here. The whole process is executing the ETL process, the model, the training, and these things. So right now it's showing a real time it's in the part of the inference right now. The other steps are in green color, so are in progress right now. Yeah, so it's done. Let me open here. So run perfectly. We can see how this thing calls, let's call it the test number two. How this thing calls RDoCD. So right now it's in the ETL process. So right now it's in the training. The next step is to deploy the model. So let's move here. It's going to change in some moments. Yeah, right now it's changing. It's replacing the old pods and that kind of things. So right now it's refreshing the things. Yeah, so right now it's moving. Everything looks fine. It's deleting pods, updating the things. And yeah, that's it. It's ready. Let's say that, for example, we can execute another test tree. I don't know if I execute it. Yeah, it's this part. It's going to execute the same part. And in Argo, you are going to see like a kind of history. Well, right now it's trying to deploy another, let's say, right, if the data change or the code or something like that. Right now it's changing the things. And what if there's a mistake or there's a need that we turn to that previous version. So here's an option in the history rollback. You can rollback the previous version. Right now it's in the part number three. And the Cubis test, Cubicon test two, let's rollback to that, to that version. Right now it's doing that rollback. And yeah, it's doing the things. Right now I can show you a little bit of code of things. Well, we have here the helm chart. This is a report that I can share with you people. The part of the pipelines is the MLOps simple pipeline here. So you define the variables and these things. This is optional if you want to use a token for Argo. So do parametrize the things. You include the different steps, the ETL, the training, the serve, the inference. And each step, you have to create that kind of template, really similar like Tecton. So you call a container and send parameters and values about it, for example, and you have to parametrize the whole thing. And let me show you another part of the code. Well, here in the part of the containers, there's the ETL process that is basically Python code. Let's see a little bit the index here. It's a pretty basic Python code that do the cleaning thing with using pandas and that kind of libraries. Then let's move to the model training, maybe. It's like uploading the things. Downloading the process file and generating the model and uploading the model here. This part, all experiments are based on the scores of students. And in the part of the inference is calling the endpoint that is parametrized here in the function of prediction. So in the end, it's going to generate this inference file that we can see here on our repo. Let's see here in our storage, in our cubicon U, showing 21 projects on Google Cloud. So here's the inference file. This is the initial file, the model and the processor. And the help chart is in this part. This help folder is going to be read by Argo in order to deploy the thing. And that's it. That's the general Argo workflows, Argo CD and K3S. Some of the resources that I just for this demo is the official websites, K3S and Argo workflows and CD. The slides, if you want to see my slides. The repository at GitHub, you can replicate this whole experiment and understand step-by-step my personal email and I am in the social networks, third URL and TPL. Thank you very much for the opportunity.