 Hello. Thank you for joining this Hewlett-Packard Enterprise demonstration. Today we're going to be talking about building dynamic machine learning pipelines with CUBE director. My name is Don Wake. I'm a technical marketing engineer with Hewlett-Packard Enterprise. And joining me today are Kartik Matur and Tom Phalen. We're going to talk about how an open source technology that works within a native Kubernetes environment can help you with deploying enterprise artificial intelligence and machine learning pipelines at scale and remove some of the complexity, keep your models in sync, and help you extend the capabilities of Kubernetes itself. And get to that point where you can operationalize your model and not fall into the category, as Gartner describes here, as the 60% of models that never really get to the finish line and give you the value, extract the value from your data that you're looking for. To help Kubernetes users extract the value from their data and overcome these challenges of operationalizing their machine learning models, we developed CUBE director. CUBE director is in fact a CNCF open source project under the Kubernetes umbrella and it's designed to run stateful applications well in Kubernetes. It is implemented as a Kubernetes custom controller. That means it's a piece of go code that you load into Kubernetes clusters and then with that operator, also known as a custom resource definition, you can pass it YAML files, which then allow you to run many different types of stateful applications. Key here is that you then don't have to write your own custom resource definition. This makes it much easier to run a collection of stateful applications on Kubernetes. And this is exactly what we're doing today. And our collection of stateful applications is an AI ML pipeline. Tom, can you explain the high level architecture of CUBE director and how it is deployed into a Kubernetes cluster? CUBE director is a custom resource definition. It can be installed inside a Kubernetes cluster. That's what's done with this create. CUBE code will create or actually deploying the CUBE director CRD itself. What I'm showing here in the graph is that this CRD just works like any other Kubernetes operator. It has a reconciliation loop. It takes in information, takes in configuration information, instantiates a one pod or multiple pods of a particular application and then manages the lifecycle of that application. So if you want to change the configuration of that particular application cluster, you author another YAML file, pass it to the CUBE director CRD. The CUBE director would then do the reconciliation to bring the running instance of that application into alignment of the configuration of the file that was just specified. And this can be done with respect to multiple different types of roles of containers. That means that's a different piece of software, perhaps running in one container of a pod for that particular application. You'll have multiple roles, multiple users, multiple different types of configurations. They can be derived from one or more images residing in the container image repository. Awesome. Thanks, Tom. So that's how CUBE director will fit into your standard Kubernetes environment. With CUBE director now, we're going to implement this basic machine learning pipeline. We're going to show you how to create CUBE director clusters, KD clusters and CUBE director applications, KD apps to do training, registration, inferencing all around some central repository and make that all dynamic. Make sure that the clusters can stay in communication with each other and the central model. So let's get down to some specifics. To run our pipeline, we need CUBE director to manage three stateful applications. They are a training engine, a Jupyter notebook and a deployment engine. They're all defined in a JSON format available on the CUBE director GitHub. And they are deployed into our Kubernetes cluster using simple CUBE CTL commands. Kartik, can you explain a bit more about KD apps? So KD app is where an application developer can define everything about an application. CUBE director inherently has no knowledge about your application. What is it going to look like? What will be the roles? What are the services towards URL scheme? Are the endpoints secured or not? All that expressiveness comes into what in the JSON file or it could be a YAML for CUBE director application. Great. So we'll use KD apps inside our Kubernetes cluster to create an AI ML pipeline to predict taxi ride times. So Kartik, I guess we need to create clusters of these apps to find these answers. So describe KD clusters for us. So we have already talked about KD app. Now KD app, the application developer has defined, you know, what are the roles? What are my services? It's like a template. And now this comes to the more interesting part, which is the actual instantiation of that particular template where you ask Kubernetes for resources. So CPU memory, that sort of thing. You can even have GPU. This particular example, we don't have it, but yeah, we have examples for GPU as well. So we have a CUBE director applications. We have clusters instantiating those applications. Now we can connect them all together, I guess. So a CUBE director cluster can be connected to another CUBE director cluster, could be connected a config map or to a secret. Using that, CUBE director reads that particular resource and gets all the metadata, bake it inside your config meta so that the parent cluster can react or make use out of that. We will talk about how config map is relevant to this particular example, but we are representing the concept of machine learning model in a config map here. So all metadata related to a machine learning model is baked in a config map, config map is a native Kubernetes resource. Using connection feature, the training cluster will know about your model. So that's what we're visualizing here. At the end of the day, once we've launched all of our CUBE director clusters that Tom and Carter have explained. They will store their information in a persistent storage repository that each of the clusters will have access to and they have access to each other via the connection. So to create this pipeline, we're going to need to take the KD apps that we defined earlier in the JSON format, and then we're going to create clusters that really launch these KD apps as KD clusters. So let's talk about how we're going to do that. So this is a very simple standard CUBE cut will create basically just make an STPI post to create another cluster using the YAML that we have already seen before for the training. Okay, so just a simple CUBE cut will create command with the YAML file and then we have that. Yeah, then that's boom, you get your training cluster now. Now we have a notebook KD cluster. Right, so notebook is where the data scientists will be finally, you know, get a chance to write their machine learning code. And that's the thing that they are most skilled in and they are most interested in everything else. So, you know, the infrastructure plumbing and stuff using KD we were showing how can KD takes care of a lot of those, but it could be a huge headache if if somebody like a data scientist has to come up or organize and manage all of that. Yeah, removing a lot of headaches here, especially noting the connection stanza within this Jupyter notebook YAML file that literally connects the Jupyter notebook cluster to the training engine instance. So this is a snapshot of the Jupyter example that comes out of the box with CUBE director. Once you download CUBE director, you get this application out of the box where we have extended the concept of magics in a notebook. So using the attachment magic, we know what are the connections to this notebook or what training cluster can I submit my code to. Once you have that you just take the name of the training cluster, your entire machine learning code can just, you know, whatever Python code that doesn't have to change as is you just take it in. And that's just a line you change because you want it to be run remotely and not locally. The next step now you mentioned config map resource a few times so now we're at that point where we have to define this. Right. So in this example, we are serializing the model so that you know we can deploy it and deployment is interesting because people so that we can share the endpoint and anybody can make inferencing call to that particular model. So we associate some metadata with respect to the serialized model and that's all this conflict map is it tells you where what path did we serialize this model on and what is the scoring scoring script scoring script is has the logic about how to serialize the model. And once you get new input, how to plug it into the model. This can be very specific to model or it could be generic so it's again up to the application developer up to the data scientists how they have defined this. And these paths are are the import path, you know, because the example, the external storage is mounted as that path BDFS MNT but depending on how you choose to do it it could change. So this is just a persistent storage layer, you know, sort of defined here that we've shown diagram so that that path there that you're showing here in the config map is how we all get connected here. Exactly. All right, great. So we've defined the config map and now we're going to apply that config map to our inference Katie cluster I take it. So that's now you spin up another runtime, which is your deployment engine. We already have talked about Katie app and Katie cluster. So like you had Katie app for training, you had another Katie app for notebook. Now you must have had another Katie app for deployment, where you define how your deployment would look like once you have that then you spin off your finally deployment cluster using that Katie app. In while doing that you tell what model are you interested in. And that's what will go under connections under config map. Okay, so here's tying it all together then this TensorFlow model. Right, and we've been talking about the connections feature so this is very configurable for a running cluster. So this model can be edited, you can remove this model you can add new model you can retrain you can reversion and all that will just seamlessly flow. Okay, so you could update this dynamically dynamically and Katie propagates the real time changes inside every part. Excellent. So here is our pipeline. We deployed Katie clusters for ML training for a Jupyter notebook cluster and then finally the inference deployment with that config map definition allowing us to keep everything connected. What is our final takeaway, looking at this diagram. Just last thing is once you have your running deployment cluster. You can do a get on that particular cluster and that gives you the final endpoint URL that that you are interested in for a given model. And then that is the URL that you can share with anybody who can make use of this model. Okay, so then, once you have that connection here to the inference deployment that URL you can do something like this this is a series of rest commands inside maybe a. This is a Python request client just making rest API call, which is the most, you know, common way to consumer model, it could be an RPC or again. It's bi directional here I see you're actually modifying input parameters and then getting out. Yes, so in the machine learning world you there is a concept of feature right based on what your model is trained on and feature set is something that that can change and based on that you have trained your model. So these are for a data scientist the interesting parts that you know different users will come up with different parameters for these different latitude longitude different, you know, day of the week. And Tom, we went through a lot of information there you can kind of bring us back around to where we started. Thanks. Let me try to recap what we saw today. So what we showed is to non stateless applications TensorFlow and Jupiter notebooks being deployed by a single operator cube director that means we don't have to write specific operators for either of those two applications. What we showed is we deploy those in multiple configurations not only training but also inference and we did that through the use of these YAML files. What we showed is the sharing of connection information that is a location where we trained a model, then provide that train model and attached it to a completely different deployment cluster and start to use it for inference. And so what we've shown is the full end to end of how to actually do a pipeline with our taxi cab drive time model, and we did it all without having to write any go code at all. We did it slowly by using the cube director CRD that's provided here in the open source community. Thank you Tom and Karthik for your excellent presentation. I think we made it pretty clear here how you can build a dynamic machine learning pipeline using cube director in your Kubernetes cluster. There's plenty of ways to contact us as you can see our emails here. We have a Slack channel and also on Twitter. Thank you for your time and interest and we look forward to working with you.