 and welcome everybody to another OpenShift Commons briefing. Today we're going to be talking with Audrey Resnick who is a data scientist here at Red Hat and we're going to talk about managed services and data science and her topic today is what's the deal with managed services and model delivery. So it's going to be a bit of a technical overview. If you have questions ask them in the chat and we'll relay them to the speaker at the end of this session. So Audrey take it away introduce yourself and let's hear what you have to say today. All right well good day folks. My name is Audrey and it's really a pleasure to be able to speak to you today about managed services and model delivery. This is going to be a gentle introduction on to what managed services are and how they can be used to gracefully deploy your model into a hybrid cloud. So the items that I'm going to talk about today are exactly what are managed services, who cares about them, where do I find managed services and what do managed services have to do with model delivery specifically. So we'll kind of go through a use case and then the question that is really important are these managed services easy to use because whenever somebody tells me about something new that's usually the first thing I ask is well it sounds good is it easy to use. If we take a look at what generic IT managed services are that will kind of set us up for the discussion just to give your brain a chance to flow and get into the idea of services. Managed services in generic IT are the practice of outsourcing the responsibility for maintaining and anticipating a need for in a range of processes and functions and that's all in order to improve IT operations and really to cut expenses. So we're going to look at everyday examples of managed services that you would find in a normal IT organization and this will give us a really good baseline in terms of not only what these services are but we'll get you thinking about what managed services could be available for data science. So the first thing that we have and I think everybody's familiar with is help desk then we go on to looking at equipment installation and along with that hardware maintenance also with equipment installation it's also moving those moving services that could also be placed into that. Firewall and security we need to be able to keep our organizations safe and secure we don't want people breaking in. We do that part of the ways by keeping up to date on antivirus patches various updates systems monitoring is a big part of the services because we want to see how our systems are performing are they are they performing well enough with the amount of users that I have or that we have if we add more users are those the systems being overwhelmed and speaking of being overwhelmed what about disaster recovery what happens if our main facility or our shop is wiped out do we actually have the capability to bring those services for our customers back online elsewhere and what about managed backups I mean that's the part how we get to disaster recovery is we should be having backups of our vital information that's just a small amount of managed services and there are so many many many more and today in data sciences we have the complexity of adding cloud as part of the platform that we work on and there are a number of services that come with that therefore the the security the data repos the servers the communications sharing services everything that we have is just a little bit more complex so kind of understanding now what generic IT managed services are let's look at what kind of what I call managed services for data science or the services that would make sense in data science would be and how would these services help us deploy an AI ML application into our model into production whatever managed services we create we have to allow the data scientists to focus on building their models while building their solutions data scientists being one of them I want to experiment really with the latest bells and whistles I don't want to deal with upgrades I don't want to deal with supported versions I don't want to have anything to do with compatibility issues I just want to focus on my solution however this does not mean that a data scientist is able to walk right up to IT DevOps hand them their laptop where they've been creating an AI ML model in isolation and say okay I'm good I need this model to go into production tomorrow I shouldn't do that I seen people do that we need that collaboration with IT DevOps and we need these services and methods that data scientists can use to work with DevOps to put their models easily into production and monitor performance so in data science there are a few steps that data scientists are interested in when thinking about creating an AI ML model to solve a particular problem and I feel that these steps would make good managed services so that's what we're going to go through and look at this all starts with data acquisition so we're looking at extracting and transforming the data we can integrate streaming data from OpenShift to Apache streams for Kafka are reaching out across the hybrid cloud to pull in data for analysis from multiple platforms in data services and we can use services open source services such as Starburst Galaxy to help us curate our data just one example the next thing you want to do is to be able to run experiments and create the models we can provide a notebook environment for model experiments and for the customers that would like to access the curated data science packages we have Anaconda commercial edition integrated for those that are looking to take advantage of things like auto ML and then make use of things like IBM Watson studio once you've coded these experiments and you've determined that they will fit the model that you have and it looks like your model is good and primed you want to be able to access any hardware accelerators to speed up the time to value we partnered with Nvidia to provide a GPU capability and what we also then want to do is go ahead and then deploy these models as services once you have your models developed you can use our source to image templates that we have or or use OpenShift Pipelines to deploy and do endpoint for testing you can also use seldom deploy for model serving then we want to look at monitoring the models and tracking performance we can continue to use things like seldom deploy or Watson machine learning and Watson open scale for any of the model monitoring and performance tracking to know when you need to kind of retrain your model and redeploy it and when you look at this overall picture that I've set up here or this path keep in mind that for any IT ops that's looking at this or DevOps this flexibility can really be a nightmare because they want a reliable stable reproducible environment for their customers which we hope that we can provide her so now that we've defined these this set of managed services let's look at who beside data scientists would care about these services it's not only the data scientists but also the data engineers and IT ops that care about these services and alongside with these managed services there are other things that kind of really fall into place nicely you want to have an AI ML model operational life cycle and that's kind of what I sort of outlined in the previous slide the data scientists sort of want an environment services in which they can not only do development work using kind of the latest bells and whistles that are open source which is awesome but also an environment that they can deploy their apps into production and this environment should contain that exploration of data and the monitoring of deployed models and applications the other thing that you want to care about is that production ready platform and this platform has to be something that IT ops feels really good about because the managed services as I mentioned can be a nightmare for IT ops is they want something that's reliable that's stable and reproducible for their for their customers the other item which kind of falls into place and is is really important to think about is the flexibility to use any open source services that are that look interesting to you when you're actually going ahead and creating a solution so being open source just to remind for it means that it's essentially free to use and that usually with an open source service or item there's a large network of users and developers who contribute towards updates new features offering support for new users and lastly the ability to kind of deploy and portability to move from the platform that you initially developed on the ability to deploy and move your application allows you not to be tied to a particular vendor and I personally personally feel that to be very innovative these days you need to be able to try a wide variety of technologies and services and that means trying out a large number of vendors so that you can create the best product that you can for your customers now with all of these things said there actually is a middle ground where we can make everybody happy at least I feel that there is so let's see if we can create kind of a data science managed services platform that satisfies this middle middle ground all these items that we've talked about so we're going to start with infrastructure the hybrid cloud platform and we'll go for a hybrid cloud so that we have things on-prem inside our own network maybe we'll use something as Amazon web services as our public cloud portion should offer very a very consistent experience across on on premises in the public cloud as well as to the edge locations and all of that has to be efficiently managed by IT operations we need to look at compute acceleration this hybrid cloud platform that we have should have integrations with hardware accelerators such as GPUs to help speed up any of our machine learning model development and inferencing tasks this brings us to self-managed services we could have all these services but what would be really fantastic is to have all of these supported kind of on a self-service hybrid multi-cloud platform and that's the platform that would really go ahead and empower anybody such as a data scientist or data engineer or software developer to be agile and collaborative through the whole process and that's without depending too much on IT operations for individual tasks we don't want to fill out many tickets to say I need access to this or I need this type of service you should be able to go in and self-manage that pick and choose what you need in order to get your job done so here's kind of the conceptual architecture for this AI ML model services so we'll go into kind of a typical project life cycle so we have data engineers that are working on gathering and preparing the data to make sure it's ready for the data scientists to develop their machine learning or AI models and a managed managed service that we could possibly choose to use is a starburst and starburst is a fully managed service you can access your data using Trino there's a premier SQL engine there's fax access that you have access and that flexible flexibility to manage your your data the next thing that we have is the business of developing a machine learning model an example of a managed service here would be something that you could use like jupiter hub that allows you to create jupiter notebooks for experimentation now I say jupiter hub and jupiter notebooks for experimentation because you don't want to end up deploying a jupiter notebook into production please don't do that I've seen people that have tried to do that it's not a good idea however for experimentation when you're first getting started to take a look at what your data looks like in terms of how it pertains to the algorithm that you're developing and kind of looking at how your algorithm can kind of solve the problem that you've been tasked with jupiter notebook something like that is fine and within this area when we're developing the model we need to also be able to determine or add any packages or libraries that we're working so for pandas we may need to use numpy we may or for python we may be able to use pandas or numpy we might be able to choose something else such as tensorflow if we're working on some sort of problem but we want the data scientists at the end of the day to really be able to experiment with those packages so again whether it's tensorflow, pytorch, scikit-learn, any others the whole idea is to have these these tools or these services available so that data scientists can do the experimentation next we have to actually go look and deploy models in an application so again this is kind of part of the model life cycle we want to go ahead and get our model and be able to deploy it and start some inferencing making predictions kind of based on that data and see if what the problem that you're trying to solve is is going to be solved by by what you're experimenting on right now so there are manage self-managed services such as selton deploy which help us build pipeline sorry and actually go ahead and deploy our model the work does not stop there when the model is deployed I know some people that say okay I'm done you have to continuously monitor and manage any of your AIML models that you create in production make sure that they're making the right predictions make sure that there's drift not happening and you're not going to be doing that by staring at a monitor and looking at your model performance through some simple little script that you you've written you want to have some side of services that will give you alerts tell you when the model is drifting so that you can continuously again go ahead and monitor and manage your your model and production to make sure that they're making those right predictions and of course when you do find something that is drifting or something that is not quite right you need to have that ability to retrain those models as as needed so keeping that in mind that that's kind of our ideal sort of managed services and kind of the platform that we go along with it let's actually take a normal or I should say an actual machine learning use case that one of my own colleagues is working on and see if this kind of data managed services and model delivery platform that we've kind of come up with would actually work for that so there is a project being undertaken by one of my colleagues Guillaume Montier for Metro London that has to do with license plate detection so that all has to do with looking at the cars grabbing the license plate and being able to monitor traffic movement car registration and any sorts of licensing fees again through the license plate detection that machine learning model has to have the ability to detect the license plate on a vehicle if the vehicle is angled the license plate needs to be righted and the characters gathered through some ML algorithms that have been developed data can then be stored or read or analyzed through Kafka in this instance I just for the folks that don't know Kafka is an open source software that basically will provide a framework so that you can store read and analyze any of your streaming data so for instance here if we're looking at some of that streaming data and we found a license plate for somebody where something was notably important about that car we could throw an amber alert finally we have to actually go ahead and store that data whether we use an object warehouse or we go back to a vehicle registration database storing that data then gives us that ability to do analysis further I mean can we look at that data and do some analysis on traffic movement congestion parking etc so now that we've kind of gone over this example let's take a real managed services platform and as a data scientist build out this AIML detection service for license plate detection the architecture that we put together for managed services this is my confession time is are the services that a data scientist could use actually exist as the Red Hat OpenShift Data Sciences platform and I'm going to use this platform which we call Red Hat OpenShift Data Science to show you how you can use these managed services that we have that we've actually discussed also discussed discussed to deploy an ML model so this all kind of comes together when a user first starts using the Red Hat OpenShift managed services platform by having everything in one central location or shared UI so that the user can discover and access a variety of open source solution each managed service whether it's Red Hat or a partner service component basically we'll go ahead and integrate along with a series of quick starts and tutorials so that way users can not only work with their self-managed services they can also self-teach or understand things better about that managed services so that they can get started working with any of the components or services and once users have enabled components so in this example in the far screen capture in the back you see that I've enabled Jupyter Hub so that's a component that's going to be available for my use again along with all the quick starts and tutorials that will always continue to be available for for people to look at but let's specifically go back to this Jupyter Hub managed service and launch it and what I'm going to do is I'm not going to walk through a real demo because we know how those happen sometimes but it'll be kind of a canned slide demo so we kind of clicking through things on slides and seeing how this all comes together so again we're assuming that our data set for the licenses has already been curated therefore we begin by using Jupyter Hub so that we can experiment with the data and just a note here just because we use a Jupyter Hub managed service at this point in time it doesn't mean that we can't integrate with any other services that is that are out there or go back we certainly can we have the freedom to do that and you have the freedom and the ability to manage and use as many services as you like realistically when you're developing something there may be other parts of the system that you haven't thought about and these managed services can fill in gaps that you may have not built out in your workflow so we go ahead and we launch Jupyter Hub and we're going to go ahead and create a Jupyter Hub notebook image which means that we're going to be packaging up a Jupyter Hub notebook into a container image that you can deploy to OpenShift and you're going to be able to customize a number of things here you're going to be able to customize the notebook image type you know you're working with a problem that requires you to use PyTorch or does the problem that you're working on require you to use TensorFlow do you just want to use a standard data science image just to do some exploration in this case we're going to go ahead and actually let me go here let me just back up a bit and go back to a container just in case folks don't know what a container is a container you can think of it as a single entity or unit that combines your entire runtime environment which would include your application any of the dependencies libraries any of the Python libraries that you may be using other binaries and any of the configuration files needed to run your application it's all bundled into one package and by containerizing that application platform as dependencies your differences in your operating system distributions and your underlying infrastructure are abstracted away and that's really good because that means that it's something very portable now that you could use on prem that you could probably use in the public cloud say I don't know like AWS so again containerization just plead provides that clean separation of concerns so that developers can focus on their application logic and dependencies and then of course the IT teams can focus on the deployment and management of that container without bothering about the application details such as a specific software version or configurations to an app in this case I'm looking again at these standard notebook images that I discussed and here I'm going to pick a base image that would contain the majority of the packages and libraries that would be needed for the license plate detection so I'm choosing a TensorFlow notebook image we can specify a deployment or container size that we feel that we would need for our machine learning model we're going to choose a large container size with limits of 14 cpu and 60 gigabytes for memory requests you have the ability to add one or more gpus based on the type of data analysis that you're doing and of course in the ml code that you're working on in this rendition we won't use gpus but remember we can always go back and recreate our notebook image with different options if which you saw and then users also have the ability to add environment variables that they would need on your project so this is an example of adding an aws s3 access key id environment variables to access an s3 bucket so access your data in aws and we're going to then once we finish adding in the secret access key environment variable and its value we click the start button to spawn our new geopoder notebook image and that can take a bit to spin up in that time being to see what is happening you can always click on the event log to get a better idea of what parts of your image are being rolled out and where you are in the image build process so now you're in your jupiter lab environment and as you can see it's a web-based environment but everything that you do here is in fact happening on the red hat open shift data science cluster that's sitting on aws this means without having to install and maintain anything on your computer and without disposing a lot of local resources like cpu and ram you can go ahead and still conduct your data science work in this stably managed environment so let's go ahead and populate jupiter lab right now with our current license plate get repo so what we'll go ahead to do is go up into the main menu we'll choose get and we'll choose clone a repository and then we'll enter the name of the repository and press the clone button to clone the license plate workshop repository note you could be asked for your get credentials so you'd enter your credentials and then again press okay to continue and what you'll then see is that actual license plate workshop repo files appear under the the name pane in the left hand side of the actual window we can then go ahead and open up any of the the notebooks and or we could be creating notebooks in this case i'm showing just an example of a notebook that we use to recognize and extract the license plate numbers from car pictures and we installed some libraries a little earlier on in this jupiter notebook that weren't part of the container image that's also something important to realize is that not every image will be totally perfect for everybody there may be additional items that you can install that's very easy to do we'll go ahead and experiment with our model and at the end make sure that we can detect a license plate number so along the way we'll go ahead and package the model that we end up creating as an api and um earlier on before we got to this point we learned how to kind of create the code that would be able to extract the number from the given license plate but of course you can't use a notebook like this in a production environment i do know people that have tried to use jupiter notebooks in production it's not a good idea it's not a good idea therefore we're going to package this code as an api that you can directly query from another application and we do this by creating a flask application um few explanations though the the code that we wrote for this particular problem that Guillaume was working on ends up all those jupiter notebooks that you saw previously end up being repackaged as a single python file with um that we call prediction dot pi and basically it's just code that was in all the cells of the notebook and put together within a single file and to use that code as a function that you can call you just add a function call say predict that takes a string as an input which would be the name of a picture it does a recognition and sends back the result and you could open the file directly in jupiter lab to see for yourself um but you would be able to recognize the previous code with the the new function added then what we would go ahead to do is launch our server um so in this case here we're just going ahead and launching it locally and we could go ahead and then test our flask application and see if it's working and it looks like our status return that it was okay so now that the application that we verified that it's working we're ready to package it as a container image and have it run directly an open shift as a service and when you do that you're able to call that service from any other application so we'll go ahead and we'll build that application inside of open shift which means we'll go to our main open shift dedicated platform and within open shift we want to make sure that we have a project namespace a setup for us to work in and I just called it user one project because my brain was dead but we have um that project namespace now set up for us to work in we go ahead and we're going to import our license plate code from the GAT repository to be built and deployed and we're going to select a number of options to create a deployment for this this model most importantly we want to be able to create a route but as a URL through which we'll be able to access our application this automated build process takes a few minutes then open shift we'll go ahead and deploy the application and in this case we ended up with that rote that I was talking about this again will be the url that we'll use to send images to so we're going to go ahead and test we want to actually test again to see that the deployment actually works we have the application listening at the at the route that was created during deployment and we tested by simply clicking on that that route link that we saw previously or copy or pasting that URL into a browser window once we go ahead and do that we want to test our deployed AI ML application we can also test our app status through curl or invoke a web request we definitely want to be able to upload images but as our application is now a REST API endpoint there are multiple ways that we can upload images to it we can also run this app from a jupiter notebook who would have thought in this case we'll go ahead and we'll add an image and I'm just calling it card dot jpeg just a photo of a car with a license plate and I'll also go ahead and add that URL or route that I created an open shift and if I go ahead and run the cell I'll see that the prediction and I got a screen capture the car so that you could see what the license plate number was that the prediction came back with the u69 yde which is actually correct so now that we've done that let's take a look and see if the options that we're talking about in the managed services platform that we originally put together are actually there and they are again this was a conceptual architecture for managed services and model delivery that that we were talking about but again this is actually the architecture for the red hat open shift data science platform and as we discussed earlier we have that typical AI ML model or workload lifecycle from gathering and preparing your data developing your model integrating your models and app development and doing some model management and in the bottom the the gray area that you see is the managed cloud platform that's provided either by red hat open shift dedicated or red hat open shift service on aws initially aws right now is the public cloud for launch of this service I will be looking at azure in the future and we do include the nvidia gpu support and then of course in the red hat managed cloud services we provide our core red hat open shift data science offering so that's going to have jupiter tensor flow pytorch sourced image for publishing and also tie-ins with other optional add-on cloud services things like open shift streams for apache kofka and our open shift api management service for optional launch partners we do include the service such as starburst for data access and prep and of course I mentioned anaconda for package distribution repositories and then we also have software partner offerings like IBM Watson studio and sell and deploy so what did we learn today well I hoped what you learned today that what managed services are and that they are a big deal when it comes to deploying a model because they make the process easier for data scientists to experiment on for the data engineers to curate the data and when the data scientists have built the model dev ops then has the ability to easily deploy monitor the model and as for that fact the data scientists also have the ability to deploy and monitor the model and with that I would like to thank you for your time and take any questions well thank you for this and you you've covered off like many of my favorite subjects one of which is is jupiter hub and and jupiter notebooks and and I love the the the tip about trying not to or to refrain to from using jupiter notebooks in production that might be the thing that I need to be reminded about the most so the the one request we got from the chat was if we could get a hold of your slides to share them with with folks and I let people know that I would make sure I could do that for them but I just wanted to thank you this has really been interested a very interesting approach to it because most of the times if you're a data scientist or someone who dabbles in in research you end up trying to do all of this by yourself or or with minimal IT support so having tried to do that before my previous lifetime and it's it's not easy and it's not fun and at the end of the day when you're working on pipeline delivery you're like I'm a data scientist I just want to work on my freaking code why am I doing this and you don't have to with with managed services for for data science yeah so I think this is like a huge step in the right direction and that you know there's I'm sure there are other managed services too but it's wonderful to see it all working on OpenShift so thank you for for the tour de force today and we'll share this with the folks that are out there in the universe looking to try this out and look forward to having you as new features and functions come available to talk us through those as well so many thanks for your time today thank you for having me on board remember folks questions bring them on in you can always look me up on LinkedIn and get my my contact through there and I'd be happy to you answer questions perfect all right thanks everybody and take care and we'll talk to you all soon