 Good day. It's a pleasure to be able to speak to you today about managed services and model delivery. My name is Audrey Resnick, and I'm a data scientist with Red Hat. In understanding what the deal is about managed services and model delivery, we'll be looking at the following agenda. We'll be looking at what our managed services, who cares about them, where do I find managed services, and what do you manage services have to do with model delivery. In that instance, we'll actually be taking a look at an AI ML use case, an actual project that one of my colleagues is working on. And then finally, we'll hope to answer the question, are managed services easy to use? So, managed services, if we took a look at it in the generic IT perspective, are the practice of outsourcing the responsibility for maintaining and anticipating the need for a range of processes and functions and in order to improve operations and to cut expenses? And I decided to start with a generic IT service, IT managed services look in order to get you familiar with everyday examples of managed services that you would find in a normal IT organization. And hopefully that would go ahead and wet your appetite to think about what managed services would be good for a data scientist. So let's take a look at these generic IT managed services. First one, I think everybody is very familiar with. We have help desk. We have equipment installation. Now with the equipment installation, that also goes with moving hardware and along with hardware maintenance. Definitely, we look at firewalls that are within an organization and the security that goes along with that. I think everybody's heard about the anti-virus and patch updates. So again, those are services that can be used to make sure that your software is bug free and up to date. Systems monitoring is a very important part of generic IT managed services because we want to see the overall health of the system. Are there things that are causing the system to run more slowly? Could we do things to make the system run better? Disaster recovery. Obviously, we don't want to be caught in a situation where we could lose some of our information on our servers if those were somehow damaged. And hopefully we'll have a rollover somewhere where our important information to our company is saved. And then we have managed backups. Definitely for anybody who's working with a lot of data, we definitely want to do backups in order to save our data resources. And there are many, many, many more managed services that I didn't even list, that I'm sure that you can think of. But that doesn't really matter. I've just given you a general overview or flavor of what managed services there are. Now let's flip to the data science side. Now, today in data science, we have an added complexity. We're just not only on on-prem. We're adding the cloud as part of the platform that we work with, our hybrid cloud environment. Therefore, the security, data repos, the servers, any type of communication and sharing services we use have to be just a little bit more complex. So let's look now at what managed services would make sense in data science and particularly where we're looking to deploy an AI ML model into production. Now as we think about managed services for data scientists, whatever services we create, we have to allow the data scientists to focus on building their models. Because while they're building their solutions, they want to experiment with the latest bells and whistles, so our managed services have to be very flexible in order for them to work with open source products. The data scientists don't want to deal with any upgrades, supported versions, say supported versions of Python's compatibility issues between Python libraries. They want to focus solely on building out their solution. However, that doesn't mean the data scientist is able then to walk up to IT DevOps, hand them a laptop where they've been creating an AI ML model in isolation and say, I need this model to go into production tomorrow. We need that collaboration between IT DevOps and data science. So really when we're looking at managed services for the data scientists, we also need services or methods that IT DevOps can use to help the data scientists easily put the AI ML model into production and then also be able to monitor the performance. So keep that in mind. The first type of managed service or category that we would like to look at is something to do with extracting and transforming the data. So this all kind of starts with data acquisition where we extract and transform the data. We can integrate streaming data from things like OpenShift, Apache streams for Kafka, or reaching out across the hybrid cloud to pull in data for analysis from multiple platforms. And data sources. And we can use services such as Starburst Galaxy to help us curate our data. So the first step, unlocking that value of the data by making it fast and easy to access data across the hybrid cloud. Next we would look at running experiments and creating models. We can provide a Jupyter notebook environment for data scientists to go ahead and model their experiments and we know that the data scientists would like to have access to curated data science packages. Therefore, we do, we could have things such as Anaconda Commercial Edition integrated for those data scientists that are looking to take advantage of AutoML. They can also make use of IBM Watson Studio. And once that data scientist has coded those experiments, if they need access to hardware accelerators to speed up the time to value, they could use something such as GPUs. I have an example here of the NVIDIA logo for NVIDIA GPUs. But really at the end of the day, we just like to be able to run the experiments and build run models using various data science library package and open source products. Let's take a look at the next service. We would like to actually go ahead and deploy models as services. And once you have your models developed, you can use an OpenShift source to image template to deploy and endpoint for testing. You can also, instead of using OpenShift S2Y, you can also access Selden deploy for model serving. And the idea again with deploying the models as services is just really to simplify and accelerate that process of deploying and managing your machine learning models. Let's take a look at the next set of services. Then we have monitor the models and to track performance. With this, in order to monitor that model performance and glean insights from the analytics and monitor things like the drift of a particular model, you can continue to use a service such as Selden deploy or Watson machine learning and Watson open scale for model monitoring and performance tracking. Just to know when you need to retrain and redeploy. When we look at the overall picture, keep in mind that for any IT ops person, this flexibility can be a bit of a nightmare. They really want a reliable, stable and reproducible environment for their customers, for their data scientists. Now that we've defined a set of managed services, let's look at who besides data scientists would actually care about these services. And I think I've alluded to some of those individuals. It's not only data scientists, but also the data engineers, those people that are going to curate or transform the data. And you've already heard me talk about IT ops really do care about these services because the jobs that they do actually these managed services will help them do their jobs quicker. And there's another thing that we have to look at it. It's just not just caring about the managed services, but there are other things besides the managed services that all of these folks care about. The first being kind of the AI ML model operational life cycle. And we sort of went over that when we're talking about the four different types of managed services that we had. The other thing is to actually have a production ready platform. Remember IT ops has to really feel good about those managed services because they really want that reliable, stable and reproducible environment for their data scientists. And then we look at flexibility to use available open source services. Being open source means that it's essentially free to use and being open source also means that you have a large network of users and developers who contribute towards an open source projects, updates, new features and offerings for new users. And this flexibility to use those open source services, this is something that I have seen as myself as a data scientist have done when I'm experimenting with a model. There may be some sort of resource out there in terms of a managed service package that will actually cut down the time that I have to code and will allow me to concentrate more on my model performance in terms of trying to build something to get my model out the door. Lastly, there's the ability to deploy and portability to move from the platform that you've initially developed on. This ability allows you not to be tied to a particular vendor. And I personally feel that to be very innovative, you have to be able to try a wide variety of open source technologies and services. You don't want to be married to one particular vendor. And I know that there is a middle ground where we can make everybody very happy in terms of looking at these managed services. So with that in mind, talking about what we kind of think the four managed service areas are for data scientists, let's go ahead and actually build a platform that fits our managed services requirements. The first thing that we have to look at is really the infrastructure. And I'm going to go with a hybrid cloud because that's typically what I develop in. And that hybrid cloud platform should offer very consistent experience on the on-premises public clouds and even the edge locations. And it has to be, again, efficiently managed by IT operations. In this example, I'll say why don't we build something really on the public cloud and use AWS. So we would actually be building kind of a platform or a managed services on AWS because that's a typical cloud provider that folks use. Next thing, let's look at compute acceleration. The hybrid cloud platform should have integration with hardware accelerators. That means things like GPUs to help speed up any of your ML model development and inferencing tasks. Fortunately for us with AWS, there is GPU availability and that comes in the form of NVIDIA GPUs. Let's take a look at what else we would need. So having a hybrid multi-cloud platform with self-managed services would be ideal. So on top of this infrastructure and having the ability to do this compute acceleration, we should have kind of a little area where we could pick and choose which type of managed services we would want to work with. And these are open source managed services that I've listed here. Things like Starburst Galaxy, as I mentioned before for data creation. IBM Watson and Jupiter where we could go ahead and experiment with building our models and looking at some of the results. And then of course if we actually wanted to go and deploy our models, we could use something as seldom deploy. So we've kind of built our layers. Again we have the infrastructure. We have compute acceleration. So now let's go back to those groupings or headings of our managed services and see what we could pick in each one and see what other details they have in them. So we've talked about gathering and preparing the data. That also takes into account anything like data storage. Where are we going to store our data? Is it going to be an S3? Does it need to be on-prem? Does it have to be on another server somewhere that is in S3? Is it a data lake? We have to do the data exploration of course. Data prep for any of our models. And there also may be time series data or stream processing data that we have to consider. So with gathering and preparing the data I should mention that we could use something like Starburst Galaxy. Now that we have our data what we're going to be looking at is be able to take that data and do some experimentation with it. So in this case we would like to use some sort of machine learning notebooks such as a Jupyter notebook and use some libraries. So we might be interested in using TensorFlow for example or PyTorch or Scikit-learn. And we'd want to actually have those Jupyter notebooks if we're working with them actually have access to those packages. Maybe in the form of a containerized image. And then we could look at deploying our models in an application. That kind of goes with the model life cycle. And of course we could talk about pipelines at this portion. And we can use actually something again I mentioned before that we could use seldom deploy to actually go ahead and deploy models and build pipelines. But the other thing that we could use is if we were in an open shift environment for instance we could use something like source to image to actually build out an endpoint to our model. And have something like a REST API so that we can connect to it. And building the model is really good but the work doesn't stop at that point. When the model is deployed we have to continuously monitor it and manage it. And we really want to make sure that those models are not only making the right predictions but that they slowly aren't drifting over time so that we don't notice that they're making incorrect predictions. And of course at the very end we have to have the ability to retrain the models as needed. We don't want to have to be able to pull everything down and start from scratch. There has to be a graceful way that we can actually retrain those models. So let's take a look at an AI ML use case that one of my colleagues has been working on as kind of a test case or pilot for using this type of managed services platform that we've been slowly building. The project is being undertaken by my colleague Guillaume Montier for Metro London. And what he's doing is he's monitoring traffic movement in particular car registration fees and that's all done through license plate detection. The machine learning model that he's developed is actually going to detect the license plate on a vehicle. And if the vehicle is angled like this one here it actually will write the license plate and then the characters would be gathered through ML. Data can then be stored, read, analyzed. In this case through Kafka and Kafka is an open source software which provides a framework for storing, reading and analyzing streaming data. So in this case they might right away take a look and see if there's an amber alert for that vehicle before they do any further processing so that they can flag that vehicle as soon as possible. Data is then stored. We can store it in an object data warehouse. We may have a vehicle registration database. And then finally after that data is stored customers have the ability then to go ahead and use analytics tools to analyze further for traffic movement congestion parking those types of items. So let's take a real managed services platform and as a data scientist build out Guillaume's project here build out this AI ML detection service. So this is confession time for me. The architecture that we put together for the managed services that a data scientist could use to deploy a model actually exists right now. It's in a product that we call Red Hat OpenShift Data Science. It's managed services offering that sits on top of OpenShift, specifically OpenShift dedicated. And that OpenShift dedicated instance sits on top of Amazon Web Services. So that way as a user you kind of have the best of both worlds. You have a very stable platform where you can go ahead and use managed services. Then you can take advantage of what AWS offers in terms of their storage and also being able to use items such as GPUs for hardware acceleration. So as I mentioned this all comes together in Red Hat OpenShift or Red Hat Data Science managed services. And this is where the user once they log into this platform can discover and basically access a number of open source solutions. So I mentioned beforehand if they wanted to do data exploration, model exploration where they're creating their models they could go ahead and use Anaconda. In my case we'll probably be looking at Jupyter Hub so that I can run that managed service and run some Jupyter notebooks. But for each managed service here whether it's a Red Hat or a partner managed service component, the other thing to remember is that with this platform there are also a number of quick starts or tutorials. So there's kind of at your fingertips help that will help users kind of understand what some of these services are and to get users started working quickly in there with any component. And this is an example that once the users have enabled the components in this case they've enabled Jupyter Hub and what will be available for them are all these quick starts and tutorials and documentation for common tasks. And one thing I'll mention too is once a user goes ahead and chooses a managed service doesn't mean that they cannot go backwards and choose not be able to choose anymore. Absolutely can. That's part of the discovery and exploration when you're creating an AIML model is you want to be able to try a lot of different items, a lot of different bells and whistles to see if that will help you with your model development process. Our data set is already curated so I don't have to use Starburst Galaxy to curate the data set. Therefore we're going to begin by using Jupyter Hub so that we can quickly experiment with the data and again note just because we use Jupyter Hub managed services. It doesn't mean that we can't integrate with other other services and you can use as many managed services as you like realistically other parts of the system will fill in gaps that you may not have built out in your in your workflow. So from this this applications menu I'm going to choose enabled and you'll be able then to see three managed services. So at this point here these are three services that I have within my own user project enabled so I'm able to use them. We're going to go ahead and launch the Jupyter Hub managed service just by clicking this launch icon. And what that allows us to do is to go right into Jupyter a Jupyter Hub notebook where we can actually build up a Jupyter notebook image which means it will be packaging up a Jupyter Hub notebook into a container image that you can deploy to OpenShift or you could deploy it to Azure. You could deploy it to AWS or to Google Cloud. In this specific step here you're going to be able to customize your notebook image type deployment size and environment variables and remember that when we're creating this this image or this container that the container for those who may not be familiar with it. It consists of an entire runtime environment for an application plus all the dependencies so any of the libraries binaries configuration files that are needed. All those items are included and then it's bundled into one package and by containerizing this application platform as dependencies differences in OS distribution underlying infrastructure those are all abstracted away. I'm busy to build an image. We're going to take a look at the available notebook images. We have CUDA, standard data science, Anaconda, PyTorch, a minimum Python. And then what I want to use which is TensorFlow which has according to the specs here Python version 383, TensorFlow 241 and CUDA 11.03. The next thing that I'm going to do is choose a deployment size so that's really going to be the container size for an image. We specify this size depending on how we feel the need is going to be for our machine learning model. We in this case are going to choose a large container size with limits of 4 CPU and about 60 gigabytes of memory. Now we also have the ability as I mentioned before to go ahead and use GPUs so some hardware performance to help us run our models quicker. And in this case I'm not going to use GPUs so I'm just going to select zero. And a little mention about environment variables. If you are working like I do sometimes in AWS and you have data stored in an S3 bucket, you want to use a Jupyter notebook. Instead of going ahead and keying in the access key ID and the secret access key right into your Jupyter notebook where the world could actually see it. There's the ability right here to use environment variables and set them up with those secret passwords so that when you go into your Jupyter notebook, nobody's going to see those items. Okay, so then we have this server go ahead and start up. This can take a few minutes for an image to spin up. You can click on the event log right here so that you can see what's happening in the image deployment process. But because this is just a can demo, we're just going to go to the next step. So now you're inside your Jupyter environment and as you can see it's a web based environment, but everything you do here is in fact happening on the Red Hat OpenShift Data Science Cluster. And that means you can do all of this without having to install anything on your own computer and without disposing a lot of some of the local resources like CPU and RAM. And you can still conduct your data science work in this fairly powerful and stable managed environment. So let's go ahead and populate a Jupyter notebook with Guillaume's current license plate repo. So we're going to have the ability to go ahead and choose Git and clone a repository. We will enter in the repository name where we have some of the work that's been done for this license plate recognition software. So we'll just enter in the name of the repository and then press clone. And the license plate workshops will appear then under the name column and they will show up in their own repository directory. So now we're actually ready to experiment using a Jupyter notebook. And as I mentioned, we're using a Jupyter notebook to basically recognize a license plate and extract the license plate numbers from these car pictures. So we've gone ahead beforehand and installed some libraries that were not part of the container image because that's the other important thing. Or if there are some, in this case, Python libraries or packages that you need that weren't in that base container, you can definitely go ahead and install them and then be able to do more experimentation. And we're going to experiment with our model and at the end make sure that we can detect a license plate number. So we'll go in, experiment with some of the code and hopefully it'll work so that we can actually go ahead and get that license plate number. So when we learned how to create that code that we'll be able to extract the number from a given license plate, but of course you can't really go use a Jupyter notebook like this in a production environment. I have known people that have tried to do that. It's not a good idea. Reach out to me if you want to find out more about that, but yeah, please don't do that. Therefore, instead of using the Jupyter notebook, what we did is we packaged all the code that you saw previously as an API that can be directly queried from another application. And we do this by creating a flask app. But first, some explanations. The code that we wrote in that notebook has been repackaged as a single Python file. So basically we're calling it prediction.py. And basically it's just the code that was in the cells of the notebook that was put together in that single file. And then to use that code as a function you can call. We just added a function called predict that takes a string as an input. And then the name of the picture and then the model machine learning model does the recognition and sends back the result. And we can open that file directly in Jupyter lab. And we could recognize some of our previous code and see what the new function has added. Now there are other files in the folder that you can use to provide functions to launch a web service that can also be used to serve our API. So in this case, we're going to go ahead and launch the server. We'll go ahead and test out our flask app. And we can do this by just using a curl. And then we can wait for the results. So in this case when we did a curl and we got a prediction of S7JDV for the license plate on the car that we were previously looking at. And now that the code, at least from what we can see appears to be working, we're ready to package it as a container and have it run directly into OpenShift as a service that you can call from any other application. Remember, we could have used something like Salden to deploy our application as well. We're just using a source to image just so that we can show you what that looks like. So let's go ahead and build the application inside of OpenShift dedicated. That means we're going to step away from this Red Hat OpenShift platform with the managed services for a second. And then we're going to go into our main OpenShift dedicated platform. And this is not a far jump. It's just one click to get back into OpenShift dedicated. It's not like you have to start up another session and then go search for the other session that you're in. I know that that's something that makes data scientists fairly irritated when you're in the middle of a model and you want to test something out. As I mentioned, this is the main OpenShift dedicated platform. And within OpenShift, we want to make sure that we have a project or a namespace setup for us to work in. And we do have one, just a user one project, very bland. And then what we'll do is import our license plate code from our get repository. So we had cloned a get repository and as we made changes, we could then push those changes back up to get. So that way we get back into OpenShift dedicated, we can import code from our get repo in order for it to be built and deployed. So there are a number of options that you can use to create a deployment of our application. But most importantly, we want to be able to create a route. You can do health checks as well, build configurations, deployments, scaling resource limits and labels. But the route is something that is important because we're going to use that to be able to access our application. And when we go and hit create, the automated build process will take a few minutes. And then OpenShift will go ahead and deploy the application. In this case, I just went further into the menu and pulled up the information on routes. And we have an HTTP URL here. And that's a route that was created specifically for our application. And that's the main public URL that we're going to use to send images to for processing by our machine learning model. First we want to do though, is to actually do a test status by clicking on that link to make sure that the REST API is awake and alive. So when we have an application listening at that route that was created by deployment, we just test it by just clicking on that route link or just going ahead and doing a copy of that URL and pasting it in the browser. And in this case, the status was okay, so everything seems to be working. And just a note, you can also test your app status through Curl or Invoka Web Request. As our application is now a REST API endpoint, there are multiple ways to upload images to it. We can also run our app right back from our Jupyter notebook. We could basically go ahead and add an image, the card JPEG here, and then copy and paste that URL that we created in open source. That's from our route. And then when we go ahead and execute that cell, we can actually see whether or not that prediction comes out. And in this case, for the car image that we put into our AML model, we are getting the prediction of VU69YDE, which actually matches the car image that we uploaded. All right. So we had that conceptual architecture for managed services and model delivery. You see how those managed services really couple together with the model delivery because they can be helped to use to deploy your model and also remember to retrain it. If we look at what Red Hat has for the typical AML workflow lifecycle, it's going to resemble very closely what we kind of built on our own with managed services and a platform to support that. And again, remember for managed services, we're looking at gathering and preparing data, being able to use something like Starburst Galaxy to help us go ahead and curate that. We could also use OpenShiftStreams for Apache Kafka. We can go ahead and then start developing our model. In managed services, we could use something like Jupyter Hub, which I demoed, and you can create a Jupyter notebook image that can use a specific data science package or library, in this case TensorFlow. We can set the container size for that image to be larger, small, depending on how big our model is or how much resources our model is going to use. We can actually go ahead and then again deploy that model. We could do a source to image, which I demonstrated again. SeldenDeploy also has the ability to deploy your model. And then finally, we want to have some model monitoring and management, and that model management and monitoring can be done actually by SeldenDeploy again. And of course, then we want that ability to retrain those models. So in a nutshell, I hope that you've gone ahead on this journey with me and learned what managed services are. You defined those four and kind of broke them down into smaller ones. Seeing who cares about them, it's not only the data scientists, but it are the people that work with the data scientists, such as the data engineers. It could be some developers and IT DevOps. They have to have a say in those services as well. You can find those managed services through open source. In our case with Red Hat, we created a platform where you can access those managed services to create your actual model and work with the model delivery. And those managed services in our case were through a public cloud on AWS, but you could also find those managed services on a hybrid cloud or on-prem. And really at the end of the day, what those managed services have to do with model delivery when we looked at that use case, we could use stuff like Grafana to again look at the data that we've created. We made a lot of use out of our Jupyter Hub instance, which we created a Jupyter notebook that we used to experiment with our model. And if we wanted to go further, we could have used Selden to deploy everything. And the last remaining question that I have for you guys to think about is with everything that I've shown you, you have to decide for yourself, our managed services is fairly easy to use. I think they are, especially the ones that are in open source, if you can have them in one area so that you can see what options are available to you. And that's what we hope to do with the Red Hat OpenShift data sciences platform. So I thank you for your time and I think we have enough time for questions, so please feel free to hit us up on the chat here and with your questions and we'll see if we can answer them for you. Thank you.