 Good good morning. Good afternoon. My name is Alejandro Saocedo. I'm the chief scientist of the Institute for ethical AI and machine learning Today, I'm going to be talking to you about the state of machine learning operations in 2019 I'm going to be covering a high-level overview of the ecosystem So I'm gonna try to go through several topics within 10 minutes You'll have several reading material that you can access to check more so to get started the motivations Is to give an overview of data science projects and more specifically the small ones When you have the traditional sort of data science workflows where you get data, you clean it, select some features, iterate and build your model When you're happy, you can put it in production, perhaps wrap it on a flask service You know, perhaps just, you know, do a very simple web app around it It works relatively well. There's not that many problems However, as the data science team grows and the requirements grow, there's an increasing complexity of the flow of the data Each data scientist want to use their own tools. They have their favorite languages Serving models becomes increasingly harder And also when stuff goes wrong, it's really harder to trace it back As your technical function grows, also shoot your infrastructure And this is why it's challenging because we're dealing with the intersection of data science, software engineering and DevOps Creating this, you know, intersection of machine learning engineering, bringing best practices from each field And that's why we created a list of libraries to deploy, version monitor and scale your machine learning We're not going to have time to cover the entire list, which focuses on our eight principles for responsible machine learning But what we're going to cover today is three principles, responsibility, orchestration and explainability So the first one focuses very similar to what was discussed in the previous speaker on the data and model provenance and versioning of them And this often is explained as the abstraction of the pipeline on its computational steps And this is primarily on separating them on the data that is coming in, the actual code and configuration and the data that is coming out By abstracting this atomic step, it allows us to go even one level higher and have our entire pipeline or entire ETL pipeline abstracted in this sort of compliance perspective And there are several libraries that are trying to tackle this challenge of reproducibility, running something in one environment, being able to get the same results in the other side And some of the libraries to watch, they're not just the ones that I'm going to mention, but one includes data version control It's a git like client that allows you to do version management of your data, code and output so that you can actually track and version every single step Model DB, it takes in a different step, a different approach, it actually allows you to track all of the inputs and outputs that your code generates And then gives you a dashboard to be able to visualize the performance of all the models and all of the experiments that you've run Finally, Packyderm is a framework that allows you to actually build, as they call it, compliance machine learning pipelines And this means that you can be able to store and version every single run of your inference pipeline So if you have a machine learning model that is trained in production, every time that it runs, it would actually store everything that goes in and out The second one is model orchestration, training and serving at scale This involves computational resource allocation, which is hard because it's like building an operating system which is completely distributed and your resources are all your notes However, there's been a lot of really interesting areas which I'm going to delve One of them, Algorithmia, this startup in the US has a very interesting comparison of the usage of traditional servers versus their actual usage And they actually show all of the unused space that they have by just having those servers at all times By using elastic servers, you still have some inefficiencies and by using serverless, it actually becomes more efficient There was actually a very interesting paper that was published a few weeks ago that was taken with a lot of criticism in the serverless space Because that actually highlighted some of the, you know, downsides that right now serverless doesn't yet have And I would recommend you to check it out One of the things that I didn't mention is that you can access the slides on that corner on bit.ly slash mlops19 A few libraries to watch include Selden, which does orchestration primarily on TensorFlow servers The second one is ML Leap, which dives into the serialization of models So it actually converts the trained models into JSON equivalents and then loads them back again and runs them across, I think, these three libraries And the final one is DeepDetect, which covers a broader set of machine learning frameworks And it builds a unified API to be able to interact with all of them for training and inference Which I recommend to check out The last one is Unexplainability, this covers the challenge of explaining and understanding black box model situations This requires going beyond the algorithms through not just the machine learning best practices, but through tools, process and domain expertise The way that we've often tackled it in a talk that we also link is through three main steps Data analysis, model analysis, and finally production monitoring The data assessment includes like things like class imbalances, assessment of protected features, understanding correlations The model assessment is understanding feature importance, you know, using model specific methods And the final one is the model production monitoring, which is assuming that as soon as you put a model in production, it has the danger of starting to diverge Whether it's because of the data changes or another reason So it's important to set metrics up front to be able to monitor that Some libraries to watch include LE5, Explain Like I'm Five, which has several tools to be able to open up models both from the NLP side image and tabular data The second one is SHAP, which is a unified set of best practices that have been brought together and represented as shapely values They have model specific as well as model agnostic approaches and allows you to visualize in a neural network, for example, what are the areas of the network that influence most that result And the last one is one that we're maintaining, it's called XAI, which allows you to analyze data sets as well as models and set metrics to monitor in production All of them, all the links are set below, you can access them when looking at the presentation And unfortunately, 10 minutes is definitely far from enough to be able to cover all of the challenges that you face, especially now with this intersection in machine learning And that's why, you know, I recommend you to go check out some of the open source projects that are there You know, the list itself is open source, so if there is libraries that you know that are not currently there, please feel free to add them And also jump in and contribute, the majority of the projects are open source, except the two ones in the bottom with the dollar signs, which are commercial But well, there's no open core as well in the other ones, but I do recommend you to check them out And yeah, so with that said, you know, I managed to do it on time, which is quite happy And we have, I guess, time for one question, and yeah, thank you very much