 Okay, so let's start. So keep calm and trust your model. So I'm Praveen Sreedhar. I'll be talking on explainability of machine learning models. So I've been working as a machine learning engineer as and as like machine learning R&D roles with multiple companies in Kochi, Bangalore and the US. And I'm also a co-founder of a nonprofit called Tinker Hub which essentially is for college students to learn about new technologies. So let's start with a story here. So suppose you are a very recently hired data analyst at a financial services company. Okay, and your boss here, he has actually given you this loan default data set. Okay, which looks like this. You just query the data set using your favorite library pandas and see that you have these features here and this is the target variable. Okay, if the person has defaulted or not. Okay, and then you run to scikit-learn and train directly a random forest classifier. Okay, and give it to your boss and say that your job's done. Okay, you also find out feature importance is because you know that your boss is very concerned with the interpretability of the model of what features are important in your model and how it is being predicted and all. And you show him that the amount of the loan, the term and the age of the person who's taking the credit grade and all of these are important features. Okay, but then all hell breaks loose. The model makes a wrong prediction. You give, I mean, the guy had to, I mean, the company had to suffer a great loss and your boss comes at you and asks you why that failed prediction happened. All right, and you have no answer because random forest model, you don't know which. So if it was a normal decision tree model, you know, like you can go through the tree and see which like parameters were essentially important for predicting this feature, but your complex black box models not allow you to do that. Okay, and you're given a big lecture on how like it's very important in regulated industries like finance, defense, medical diagnosis, insurance and all that to have this, this interpretation layer on top of your normal black box evaluation metrics and things like that. Okay, so that's why interpretation is of very paramount importance in these industries. And the problem that occurs is that as and when we have like higher and higher accuracies with more complex models, explainability of the model gets hit by a large extent. So you see here, decision trees are very explainable. You can see at each node what is happening. You can go through the tree and all, but as you go up, you go up to graphical models, macro models, and of course, deep learning. You get a very high bump in accuracy, but at very low explanation capability. All right. Now what do our models learn? Okay, now interpretable models. So we all know linear regression. We know that y equal to mx plus c, you know the slope and the intercept. So you know like how the line is, how you can exactly, you exactly know the function. So it's very explainable. You can tell your boss what is happening in the model exactly. So those are applicable only in linear monotonic functions where any change in the independent variable gives an exact and measurable quantity of change in a single direction in the dependent variable. So you see here, but the problem is that not all data sets that you get will be of this linear fashion. So you need to train proper complex models to predict the non-monotonic functions like these that you see here. Okay, so these are the ones which create the biggest problems. Now, what is explainability? Okay, now early we saw like linear models, you know the function that is being predicted. So it's like very explainable. Okay, important that would be highlighted. So we'll see an example after you get more idea. Now, this is one of the methods of doing that. Okay, it's called tree interpreter. So interestingly, this does not have any research paper associated with it. It's just a blog post by a person named Ando. Okay, and he made this library called tree interpreter as well. So for an example, see the XOR gate, exclusive OR gate where you have this truth table. Okay, and we're training a decision tree, a basic simple decision tree to model this particular data. Okay, so this is the condition at the base node if X1 equal to zero, if it is true, it goes up. Okay, so true always goes up, false goes down. So we will take this last example. Okay, then both are one, but the output should be zero. Okay, so X1 equal to equal to zero goes down because it's not zero, it's one. Okay, X2 equal to equal to zero. Again, it's not that, it's one, so it goes down. Okay, and the value is finally, we get it as zero. So the prediction is zero and how we decompose the contributions is. So the train set is given a mean value. So you see zero, one, one, zero. Mean value of point five is given at the root node. Okay, and at each traversal, okay, the value is recalculated. So X2 is zero, there's no increase from X1. Okay, and when it goes down, since it's X1 and X2 are interrelated, it gets a decrease of minus point five. So we see here, even if we train a complex model, we can average the contributions using this and basically decompose each of the values and it flows down. Okay, so I'll, so since like I think all of your minds have been blown, I'll explain this as you're a five-year-old. Okay, and that's why a team in Russia has come up with ALE five. Okay, so explain like I'm five. Okay, that's actually a library that they built for explanations. Okay, now let's see the same data set that we, I mean, the problem that we faced earlier, the lone default prediction data set. Okay, so this is a normal scikit-learn pipeline where you're training a model here, a random forest classifier. You're fitting the model, okay, model.fit, okay. You see that the accuracy is 74%, okay, good enough. Okay, you're calculating the feature importances earlier that we did. Okay, now this package actually makes it very easy. It was import ALE five. Okay, so now let's see why that wrong prediction was being made. So we're taking the fifth sample data. Okay, we're seeing that it has been defaulted. So one means default, zero means not default. Okay, and you see that the prediction is giving it as not defaulted. Okay, now if you see here, this is the contribution values. Okay, now you see the income is playing a big factor. Okay, the number of years of playing a big factor, the grade, the credit grade of the person is playing a big factor. Okay, so since the person has a very high salary, the model thought that maybe that person would not default. Okay, so now you can show that it's because these parameters and the proper contribution values that the model is throwing out error like this and you can debug it and make a better model out of it. Okay, similarly I've done for one more. Right, so I guess I'll go back. Okay, so what we just saw was a model specific approach of interpreting pre-based models. So, I mean this same method can be used for all like ensemble models of trees, like random forest, gradient boosted trees or any tree-based models. Okay, now there are many such more model specific approaches. So one of them is model specific visualizations. Now, this is the Andrew NG's famous cat neuron. I'm guessing like all of you must have seen this. And in the previous talk, we saw how easy it is to fool CNNs, right? But the same sort of a method, the gradient ascent method can be used to visualize what is happening in a CNN model as well. So what happens is essentially to get this final neuron, I mean the picture where, I mean the visualization where what sort of features actually activate the neuron maximally, you do a forward pass, you compute an activation, okay, activation AI, okay, at a particular neuron. So this is a random static image, okay. There's nothing involved in this random static that you get in a TV, that sort of an image just to start with, you do a forward pass at a particular neuron I, you get the activation value and you do a backward pass, get the gradient, okay. So here the gradient is essentially what, by what values should you change each pixel's color so that this activation is maximum, okay. So that is essentially the gradient. So if you want to increase that further, what will you do? You'll add that back to the initial thing that you got. So you repeatedly do that. You add this gradient repeatedly with a scaling factor, okay. And at a particular, I mean after doing, I mean a few iterations, you get like what the neuron is actually expecting, okay. And you can visualize it in that way, okay. So this is again like global scope. You don't have much scope in saying like you can't take a particular instance or instance and see what is happening, but then overall idea you'll get what neuron is looking into, okay. Now this is the same slide that the previous speaker showed, again it's very easy to fool CNNs. You just add the gradient to the previous image to get an adversarial example. The gradient of a given is added here and it's predicting it as a given, okay. So this actually makes it all the more important to be able to explain predictions, okay. So the next model specific approach would be attention mechanisms employed to explain a model, okay. So in this you see Salesforce has this new abstractive summarization model where essentially whatever portion or whatever like input text is actually being attended to that gets a heavier weightage and you see here. So see the summary here it has elements whichever was important and attention was paid to here are getting more importance and so you get to kind of know intuitively that all of those attention points were the ones that are used to make the final summary. Okay, so again this is another example of attention. So this is a famous paper called show it and tell it came out in 2015. And here you see like a woman is throwing a frisbee in a park. So you see frisbee underlined and you see that the model is paying attention to the portion of the image where the frisbee is, okay. So this is again an intuitive explanation that you can give for the model predicting in a particular way, okay. Now what happens in a attention model? So let's take an example where we're converting human readable dates into machine readable dates, okay. Into a particular format date. So you're converting five January or five Jan or like short words, short shortened dates into proper format like year, month and day, okay. So you have a encoder decoder framework here, okay. And so you see to predict like 2016 here, this 16 is getting maximum attention and it comes here. So the encoded, the attention mechanism what it does is it actually calculates the weighted average of wherever the network needs to look into at, I mean, maximally to get the particular output. So that is the attention model and we have like model agnostic approaches. We earlier saw partial dependence plots. We saw variable importance measures, okay. There's something else called residual plots and surrogate models. Again like statistical methods, not very explainable to your business people, but this is a new method called local interpretable model agnostic explanations. So this research was done at University of Washington under Carlos Guestrin. So I'm not sure, so he was actually the CEO of a company called this Pedro Domingos. He was the, so I forgot the name, Turi, Turi. So Turi was acquired by Apple and all for the research that they did, okay. Now what this model does is, so this is again a caption model, okay, where this is an image of a dog holding a guitar very strangely, okay. And you see that when a caption is made for electric guitar, it shows this portion of the image as an explanation. So it shows that the model is looking into that particular area to explain, I mean to predict that it's an electric guitar there. For acoustic it's looking at the wooden portions, if you see. Okay, so unfortunately it's still getting confused between electric guitar and acoustic guitar because almost the same string shape and color is there. So it's still getting confused, okay. And Labrador is explaining using this face and a small image here, okay. So Lyme is a very interesting method. Essentially what they do is they throw away the original model, okay. Your original model is thrown away and suppose you want to explain this particular instance. What is done is data points are sampled across, I mean the vicinity of that instance and locally another classifier is trained there. So another linear model is trained there. So in that particular local region, you know a proper interpretable model of why something is happening. So the original model is thrown away and you train a new model around the vicinity and in the local context, you train a interpretable model, a linear model with which you can know what is happening inside this particular, the original model as well. Okay, so essentially the idea is for every image, you take like random perturbations. So random images you just chop off different portions randomly, okay. And see which one is giving more importance in the local interpretable model. Okay, so you see here, you'll have like many instances like this, this is ever continuing, okay. Since we can't show the endless models, it's not shown. But one of these will have the maximum importance, which is this one. And that is chosen as the explanation, okay. So behind the scenes quite a lot of complex math is going on, but essentially this is what happens. Like you make a lot of random samples, you train a local regression model, which you can explain. Okay, and then whichever one of those is giving the maximum prediction, you choose that as an explanation and show it to the end user. Okay, now let's see a demo of this. So I have a Keras model here. So I'm sure like all of you must have tried out Keras deep learning models. So we have an inception V3 model that is pre-trained, okay. And let's see the prediction for a particular image here. Okay, so you see this image, okay. Which to us looks like a cat and a mouse. Okay, but to the inception model, it looks like an American black bear. Okay, strangely. Okay, American black bear and like tons of other animal names that we don't even know. Okay, so what's going on? Okay, again, lime is very easy to use. You just import lime, you have a image explainer class. Okay, you do the explanation here. Explainer.explain instance. You take the image, you take the model, and you tell it how many labels to show. Okay, and parameters like that. It kind of takes some time. That's why I'm not doing it live. So it takes like half an hour on this MacBook. You need a GPU to do deep learning. So you see here, this is the region that is impacting the model to train it. I mean to classify it as a black bear. Okay, and if you see this closely, actually even like to us, now it looks like a black bear, doesn't it? Can you say this is a cat only by seeing this part of the image? No, right? We have a much complex like brain that can see the mouse that was standing here and correlate with that to know that it was a cat and a mouse. But the model does not know that, right? That's why it's looking only at this part and classifying it as a bear. So this local interpretation gives a clear idea of like what is happening, what is happening like within the model such that like this particular misclassification is happening. Okay, so we cannot blame the inception model because even we are confused when we only look at this portion of the image. Okay, now we can do like fancy stuff where like you highlight which of the features are important, which are not and things like that. So it also classifies it as a bat. So that is this particular portion. Again, kind of like looks it's black, bats are black. So essentially your model is just learning. It's like when we do like wrote learning in our classes, engineering classes, it's just wrote learning and predicting out, throwing out values. Okay, so this is essentially about Lyme. Okay, and you can check out all the code at this link. So I've added like much more detailed intros into like what happens inside. And like there are many other models as well. We saw, so we saw, where is that? So we saw here model agnostic approaches as like these many, but we have other models like layer-wise relevance propagation. Again, local and interpretable. Again, there's like couple of models within this. So I will link to those research papers in this particular GitHub repo. And it's all thanks to these three open source packages, pre-interpreter, Lyme, and ALI-5 that any machine learning engineer can use these to properly explain and build models, debug models, and show like what is happening inside the model without having the issue that the analyst faced in the beginning. Okay, so that's all from my side. Thank you. Thank you. So my question is how to improve once you've identified the problem? How do you? How do you improve your model once you've identified? In the case of say you've identified it's a better than that. Yeah, yeah. So how do you go about and improve? So that's like a huge topic because you have like tons and tons of ways you'll have to try different approaches. So this was inception with you. But something like CNN it's like very translucent. I mean it's very opaque to actually how do you actually learn that? So it is actually because I don't think there's a hard and fast rule of like improving deep point learning models using interpretations. But I guess people are trying to build that. I'm not aware of any particular process that you can use to like improve models. Rather I mean even image net like the challenge and all what people do is like they try out different architectures keep on trying out different architectures and just like try to beat the top score. That's all right. Yeah, I think he has asked the same question. What I want to ask. Okay. So I have a question here. The LE5 packages you showed it gives you contribution scores. But when you apply to a problem bigger than a loan prediction, you have a lot of features. You'll definitely have correlation in the feature space itself. So does it split the contribution between correlated variables or? Yes, yes. So the three interpreter package has like has like provisions where you can specify that you want feature interactions to also show up. So what will happen is that along with those individual features the interaction features will also come up. So let's say like you have interaction which you have an interaction between feature A and B. It'll show A and B together and like show what contribution is being made because of both A and B. So you can use that. What you showed in that right variable importance and you showed the effect. That's easier for simpler models where you have less number of features. But like a lot of these models you have large number of features and there are like very specific regions where these variables exist. So these variables exist together in like groups. Right, right. Where the interactions are very complex. So models will usually pick one and then if you slightly perturb the data the model will pick another feature. But that's very difficult to explain to the business guy because he's. True, true. He's not, he's thinking that your models are sort of not robust because they are changing. That's the. Yeah, so this is not a solved problem at all. So this is like the initial stages and like I've seen like very people actually delving into the practical aspects where, so it's, so even like yesterday we had like a couple of OTR sessions where a lot of talks were going on on how interpretability is important but not much was done in the practical aspects of how to make them. So this is definitely not a solved problem. And as you said, like we still face a lot of challenges when you have tons of features in. Hi, so in these interpretation of the models, how does the confidence come in place? Like, you know, if a model is not very confident of the prediction, does it still get the same areas or does it get highlighted as, you know, model is not confident because. So in the local, local interpretation model, so the Lyme package that we saw, they don't have like the importance is shown in but for three interpreter, it includes that. The only problem is that three interpreter cannot be used for images. One question here. Yeah, so I saw that you demonstrated LE5 and Lyme as the, this thing and you took up a, and you took up image examples. So does it work for non-image kind of data as well? Because the contribution also you demonstrated as an image which highlights a certain portion. So does it work in the same way for non-image data? Yeah, it works for the other cases as well. So, so let's see. So this is the Lyme package, the image interpretation model. But after, if you see, so this one I showed earlier, right, the loan prediction. Yes, you see here, this is, so earlier I was using LE5 for that, like for contributing the feature values and all, but you can use Lyme as well. So see, you see prediction probabilities here and you see the feature importance. These are the importance, okay. So you can get it even in Lyme, that's also possible. Thanks. So you showed that black bear or that cat image. So what if it's actually a black bear that looks like a cat to us? I mean, that's possible. Yeah, that's possible. Definitely possible. And even that loan defaulted thing, I mean, there's no guarantee that that was actually a predictable thing. It could be that some earthquake happened and that guy defaulted on the loan. Yeah, it definitely can happen that way. The problem with our like black box methods is this. It's that like we're like learning, I mean, it's like road learning from the distribution that we have and predicting based on that. So we don't have that like intuitive logic sense to see that there's a mouse there. So maybe it could be a cat sort of a thing. No, so like it's possible that our priors are wrong because we saw a mouse, we thought it was a cat. But I mean, is there some rule like you can use? Like maybe you decide that what is the predictability in this use case? And if the predictability is low, then you can't blame the model. Even if you find out how it's interpreting the results, it won't be the model's fault because. Yeah, yeah. So there's no method. I mean, I'm not very aware of any method like that right now, but it could be a very interesting thing to do. Okay, thanks. Hello, yes. Extending to his question, like I was once looking into a problem where we can fill in the loop of confusion matrix. Okay, like if based on the first pass of learning, we generate a confusion matrix between cat and the beer and feed that into the CNN. Can you repeat? So, okay, so like instead, one is like we have some hierarchical classification here that at the first level, we apply the output from the CNN and then we from the second level, we apply some contextual information that if the class one has is close to class two, then they probably belong to a confusing group. Okay. Second option is that we feed that output of our confusion matrix into the CNN itself. Okay. I'll have to look into it. I didn't like get what you're like suggesting. Basically, I was looking into if you can, because we stopped at that cat and beer example, but how will you exactly solve it? I was... Okay, okay, okay. Okay, let's have a chat offline because I need to catch up on that then. Any more questions? We have a little bit of time left. All right. Thank you, Praveen. Thank you.