 I hope everyone is freshened up after these snacks and I know it's overwhelming with a lot of variety of technical talks over here by a lot of experts and I hope you're looking forward to our talk as well. So we are today here to talk about Kubernetes in ML and MLops as well. That we will be covering today is like what exactly is MLops, challenges in MLops, how Kubernetes comes into the picture and how you can use, how exactly you can use Kubernetes into the MLops pipeline, what are the benefits and then we'll look into the some open source tools which helps use Kubernetes with MLops. Next, okay, I have it. We'll start with my introduction, myself, I am currently working as a DevOps engineer at Numblebox.io. We have an end-to-end MLops pipeline platform and I am also working as ambassador at solo.io. Before that, I worked in TCS as a system engineer and I love to contribute to communities open source and interact with open source enthusiasts and I like to write content about cloud-native DevOps and Kubernetes landscape as well. I'll give Mike to Rohit to introduce himself. Thank you, Veja. Namma Chennai. So how are you all? Mumbai people will be like more loud than this. How are you? I know it is 5.55 or 6 PM, right? So still you are here. So it is a lot, like I feel too glad like you are here attending our talk, still you have this much power. So thank you for that. So introducing myself, I am Rohit. You must have seen me or not seen me in giving the talks in Kubernetes world, Istio service mesh and EPPF, that kind of technologies. So I am the person who are a lot active in the application networking background, community background and DevOps technologies. So introducing myself, I am developer advocate, I worked at solo. So today is my last day at solo. So that's how my journey till now. I worked as an MLops engineer, I worked as a DevOps engineer, I have like community background also. I am the organizer for KCD Mumbai, as I am showcasing there. So also DevOps days in India will be happening, right? So I am sure you are looking forward to it. So I am organizer for that also. So going forward, let's go and see, you know, like what is MLops, anyone here? Yeah, what is MLops? DevOps just for application. DevOps is the culture, it is just you can do for anywhere and something. So somewhat is right for sure. As you can see, it is nothing but engineering disciplines, disciplines, which just aims to like unify the ML systems with your DevOps things, like you are working, you are helping the data scientists so they can work development and develop operations together to solve their automations, to solve their daily pain, like they are just sitting and doing this daily, they are coming and checking like if their pipeline are working nice, if training is going nice or not. So that's why MLops come into the picture, like you can automate the things, you can build the pipelines, you can do the various stuff. So what actually you can do, what are the challenges we will discuss today? So if you are a data engineer, if you are an ML architect, if you are a data scientist kind of thing, so today you will enjoy this talk. So as we can see, data engineer. So what a data engineer do, right? He works with the data, he loves to play with the data. So as you can see, data acquisition, then business owners and your client just wants to provide the business metrics and then you just get it and use the EDA, then you get that stats and load it to the system, then you train the model, you do the stuff. But then it becomes a pain of the data scientist. He is just doing this model deployment training, he gets the error, he again goes back, he again trains the model, again gets there, again goes to previous step. Why I can feel that? Because I worked as an ML engineer previously in the R&D team, so I was just sitting weeks and months just by training the models and meanwhile just training the APIs and stuff, so that's why I got bored also. But then I come to know there is some kind of a thing that exists, which is DevOps. So if you are an ML architect, you will love to use the DevOps where you can do the risk assessment, you can play with the model performance analysis. If you have the ML knowledge, you will firmly get this terms knowledge for sure. But if you're not, it is some kind of a runtime thing which you are playing around the network, right? If you're a DevOps engineer, so that is here in the machine. But as we go forward, if you want to continue your machine learning models, if you want to also scale your models, if you want to set some pipelines in the machine learning, so that you can do in the machine learning ML ops. So data engineering plus DevOps, because it solves that use cases. Same comes under the DevOps plus data scientists. It is nothing but if data scientists love to play with the metrics, right? They are getting the metrics and they are solving that use cases, getting the errors, then again checking the logs, the kind of thing. So that's how it solves the use cases of data scientists. So as we move forward, common, and I know it is old, oldest world world. So as you know, one does not simply put ML models into production. It has a lot of steps. And also, this is the common thing, right? You have made the model, it is working fine. But once you put in production, users will just creating the support tickets for you. So as we go forward, there are a lot of challenges, right? Scaling ML workloads. So how you have ML workloads as you're playing the machine learning models and all, right? So you want to scale the system, so how you can do it. You can do it normally for sure, but here ML ops helps you. So how Kubernetes, so our today's talk is Kubernetes in ML, right? So let's come into the picture for the Kubernetes. Kubernetes is someone who will help with that and my friend will tell you how these things work. Under that, containerization and dependency management, so there are a lot of problems to containerize the machine learning models. It is really pain point, so that's where the challenges, it is one of the big challenge. Then one of the challenges, ML infrastructure complexity, right? So you are dealing with different type of infrastructure and it increases the dependencies, increases your cost, increases your, what we're saying, infrastructure capabilities, right? So there comes into the picture of this challenge. So there are a lot of challenges, right? Then model versioning, you are dealing with the different versions of models and reproducing it. So that is one thing. So you know, like what I'm talking about versioning, then it will come into the picture and all, right? So then data management sharing, yeah, sharing, like sharing is, as you can know, like containerization, Docker. So Docker allows to share with, so there is one kind of a popular meme, right? I guess Solomon Hayekz did that, like if it doesn't work on my container, my system, like how can I share with the manager and all. So that thing, machine learning models can also be shared. So other than that, there is a monitoring and observability. You have, so as data scientist, pain point, right? So that is one of the use case and data scientist sit for years to just play with one model. And I'm not joking, it is true. Ask any popular company in the ML, they are like, this is one of the use case. And then CI-CD for ML, you can create the pipeline and various things, right? So ML pipeline creation is really tough. So if you want to add, let's say, scalability, resiliency, and these things, then it gets a problem. So other than that security complies, you should have some machine learning model security, right? There are a lot of security issues that can be, there is also, there is no mutual TLS connections between the services you are planning in the machine learning models, right? So a lot of things are there. So this is like, you can use the Kubernetes for ML ops, but as you can see, it requires a lot of knowledge, how to scale the machine learning applications, then it requires planning, tooling, best practices. But we are not here to share the best practices and everything, right? We're just giving the picture, like how you can use the Kubernetes in ML ops and ML. So as I look forward, my co-speaker will tell you on the more things. So as of now, almost everybody in their talk has explained what Kubernetes is in a very brief manner, but in the perspective of how we can use Kubernetes in ML ops, not exactly what Kubernetes and all. So basically, we all know that Kubernetes is a container orchestration tool and we can automate the deployments, manage the deployments, scale the deployments using Kubernetes. So we all know that it's a very robust infrastructure Kubernetes provides for distributed system and it makes the best fit for ML ops to deploy ML models on Kubernetes. And it also has declarative approach. Like, we know what we want and we'll tell them and Kubernetes just does it. So moving forward, we'll talk about why exactly Kubernetes only. So basically, we've seen that Kubernetes provides a lot of features and it gives robust infrastructure and it has been booming since a lot of years. And it's known for its powerful orchestrator. So basically, why ML models should be deployed on Kubernetes is basically it gives containerization and containerization gives a consistent infrastructure for ML models to deploy. Like, it shouldn't be like we are deploying ML models on a particular container, a particular system, and we want to ship it to some another. We don't have to recreate, right? There comes containers. So that is why it, so there's no need of saying like it works in my machine and it works in my container. So yeah, and it is cloud agnostic. Like you can deploy Kubernetes on any cloud. It is cost efficient as containers. We can manage the CPU and memory for containers as they are lightweight containers. It does not take much CPU and memory. And every big cloud present in the Seco system provides managed services for Kubernetes so that we just have to go to them and tell them that we want to use Kubernetes and they'll give managed services for Kubernetes. And also, like it is resilient. Like it gives fault tolerance and everything. And we'll see more of the benefits in the next slide. So first and very important factor is scalability. We all know that whenever we want to, we hear some news or like we know there's a sale or something, we directly go to the applications and bombard it with requests and everything. At that point, Kubernetes handles the scalability. And also, like there shouldn't be always horizontal scaling or vertical scaling, right? There should be a need what exactly we should do. Should it be horizontal or should it be vertical? So basically, Kubernetes does that for us. And it has auto scaling feature which can manage the replicas, number of replicas. Maybe we want to increase or just scale it down. And also, like we can manage the resources with Kubernetes. Like there is resource limits attribute present, like how much we want to limit, how much we want to give it to. And yeah, obviously, containerization for ML models. Like if we want to port one model from one container to another, it gives you portability, right? So if your ML models are already deployed on Kubernetes and we want to port it to some different cloud, we can do it easily with the help of Kubernetes. And resiliency, yeah, it gives fault tolerance. And it comes with monitoring applications, like we can deploy monitoring tools with Kubernetes. Yeah, monitoring and observability, right? So we can deploy Prometheus, Grafana, or whatever tool we want to monitor all of the Kubernetes applications. CI, CD4, ML Ops actually means that whenever we are updating model, we will not every time go and manually deploy it to production, right? So we have to have a pipeline whenever the pipe model is updated or some modifications are done. It should automatically go to production. Security is the topmost concern, like Kubernetes comes with a lot of security options for ML models. As we know, the ML model have data, which is very valuable. So it basically gives, it comes with RBA, so it comes with port security policies. And secret management can be done with manifest. And we can use a lot of external walls within Kubernetes as well. And we've seen all of these benefits, like how Kubernetes can be used, where can be used, and so now we'll exactly see how Kubernetes can be used and executed in the Kubernetes yamls. So basically, there comes in scenario, which is affinity and anti-affinity. Affinity is basically the ability to constrain the pods to be on a particular node, to be like in a pool of nodes. Like if we want a model, for a model, we want a particular GPU. We cannot deploy all the nodes with that GPU, right? So we'll create one node only with that GPU and we'll make that pod to be deployed on that particular node only. See, it constrains the nodes that can receive a pod by matching labels of those nodes. How it does that, it does because of the labels. Like we have to put the same label on the node, which we are putting on the pod, for that pod to be on that particular node. And that is the attribute, actually, where we can use node affinity and we can give a lot of values over there. So basically, you will say it's just like node selector. What node selector does is it only allows for the labels. But in affinity, you can give a lot of extra values over there, which is very customizable. It's not limited just like node selector. And there are actually two types of affinity. I forgot to say that earlier. It's node affinity, which is like we have to give matching labels to pod and a node for pod to be on that node. And there's this interpod affinity and anti-affinity. What difference between these two are is pod affinity is between node and pods with matching labels. But pod interpod affinity is basically the pod will go onto that particular node only when the matching label of the other pods are already available over there. So there is a node. There are already a lot of pods on that node. So a particular pod will go on that node only if the label of the pod is matching to the already present pods. And you can see it also gives pod affinity and pod anti-affinity attribute if we want to put it into the Kubernetes deployment YAML file. So basically how all of this happens is whenever like 15 people are using some service, so if one person is trying to access it, the other 14 people should not wait in line. So it has to be scalable for the other 14 people to use it. That is why all of this Kubernetes and MLops can be integrated. And I'll give Mike to Rohit to tell you about the open source tools. Yeah. Are you still awake after this so much technical thing? So I know it is really technical stuff, wars and all and all. I also don't understand sometimes. But yeah, if you want to try it out, if you don't want to put a lot of efforts. So yeah, we have the open source things and cloud tools which you can use and make your life easier. So as you can see, this is the DevOps tools. Like I tweet a lot of things on Twitter. Like you can check it out if you want. But other than that, as you can see, if you want to try it out, how you can integrate the DevOps and machine learning and all, these are the tools, all the other tools to give the simpler approach. So if you want to try out the CI study, you can use the Jenkins for like just, there are a lot of examples I will give for sure. But I don't have the time. You can reach out to me any time. But if you want to set up the CI CD, yeah, there is Jenkins just connected to Git for the budgeting and return your models directly from the Docker file or something. It is easy to do. Then you have the other things like there are data pipelines you can create. There are ML flow is there. There is a Q flow is there. We are talking about Kubernetes, right? So Kubernetes thinks you can easily do it using the Q flow for the machine learning. So you can try it out. Then there is a flight is there. Ketro is there. Bento ML is there. A lot of things are there. And a lot of things are open source by them. You can try it out. And also, if you want to set up the hybrid infrastructure, if you want to set up the ISC or something, there are tools here. So you can click the picture for here, or we will share the slides for sure, right? So other than that, so it is not just about like putting the Jupyter notebook into the production and you have done the MLops, right? So there are a lot of things to do. So you need to understand how the training works or something. It is not just like you can pick one tool and do the stuff. There are a lot of things to do. So that's why I put that chat GPT announcement, right? So as we go, so I know this will again blow your mind, but try to bear me for a few seconds. So as you can see, just giving one of the example, which is Q flow. As you can see, like experimental phase is when you are doing the data cleaning, then iterating the various things into the Q flow. And then as you can see, first, if you are a machine learning engineer, if you are data scientist, you will do like identify the problem and collect analyzed data. And you data clean data, like processing and all and right. So then you have the various tools for the ML algorithm, which is TensorFlow, one of the popular. Then there is PyTorch and all, XGBoost. Like you have must heard it about. So then there is an experiment with data and model training, Jupyter Notebook you can add and you can add the code here and you can train it. So one thing is like, so Katape is something which will help you to add this Jupyter Notebook into the pipeline. And then you can integrate Katape and it will do the automation for you. So that's why, which hyperparameters to add it, which not like hyperparameter tuning is one of the worst thing if you are trained the model. When error comes, you have to go and change the hyperparameters which are in the null. You don't know which is the wrong. You just have to do the trial and error methods. And that's why Katape comes into picture, which will help you in that. Then production phase is just like once you transform the data, you train the model. And then there is a different thing to train. Then there is serving. So you have to serve your model. So then there is a KF serving which will help you. Then there is a TF serving. So they are just not, I can say, alternative or something. But they are just you can try different things. Whichever will work out for you, that is the best for you. Then monitoring you can do on anything or you will have the. So QFloat dashboard is not for monitoring. But yeah, it also gives you different things so you can try it out. So there are awesome recommendations on QFloat on GCP or QFloat on AWS. So if you are using the public cloud, I would definitely recommend you to try it out. Then this is the architecture where if you want to try out some use case. So it is just like Python code or something. Then you are putting into the pipeline. Then it just go to the Kubernetes. Do the ATCD, like there are a lot of things. But if you get the idea, here it will help you for the orchestration system which Kubernetes comes into the picture. And then also machine learning, so whatever data you get, you can play with the data and integrate it here. Also, there is a pipeline web server. It is nothing but it will help you to create the pipeline, do the stuff with pipeline. And then you can go and connect orchestration for the future things, which people talk about like GitOps and all, right? And then you can go and check the parts. Or you can see like if everything works out, then you will see the success. Or other than, you will just see like bodies re-initiating and things and things. So yeah, you can check it out. There are a lot of videos on YouTube or something. But I can make your life simpler. So these are just like opportunities if you want to go into the MLOps. But yeah, this is most important. So if you want to learn about the MLOps, if you want to try it out, and if you want to try it out, like what are the tools and all. So I have provided everything here. So as you can see, MLOps.redoMap. So this is one of the best roadmap in the world, and it's created by Indian, Ganesh or something. Then there is this one who has created nice MLOps community as well as various things you can learn it about. And there is MLOps community for sure, then DevOps, Roadmap, and all that. So go check it out. And there is use kisses also, one of the blog might mean. I guess that's all for today. I guess I didn't bore you much. But thank you for attending our talk. Thank you.