 Okay, good morning rather good afternoon now. Okay, so can you hear me back there? Is it all right people behind? Okay, great. So my name is Vinit Balasubramanian. I'm a faculty in IIT Hyderabad. And thanks to the organizers for inviting me here to share some background that we have done background work that we have done an explainable machine learning. I will give some overview of the work so far in the broader community and then talk about what we have done in this space. So generally I get asked wherever I go is there I mean, is there an IIT in Hyderabad? So that's generally a question I get. So I think it's my duty to perhaps start by saying that there is an IIT in Hyderabad and it's actually not too young. We are about 10 years old now. So we have graduated about seven batches of students. So we pretty much have all the departments that are in any other IIT and we currently have about 2,200 students. So those are some of the buildings that we currently have on top. And I'll keep this introduction very brief because I want to focus on what I need to cover. So the CS department at IIT Hyderabad, which is where I work out of. So we have about 20 faculty just kind of close to stable state for most of the IITs probably among the newer IITs that came 10 years back. Probably we are first to reach this number. And our opening and closing JEE ranks, which is one way in which people measure how well IITs do are improving each year. So this year actually our opening and closing JEE ranks at somewhere in the vicinity of 450 was our opening and 770 was actually our closing rank for the CS department at IIT Hyderabad. And we have projects with government, academia, industry and several student and faculty awards. So if you've not been that side, please do visit. I mean, for those of you who didn't know there was a IIT Hyderabad, please do visit. Just a quick brief about the kind of work we do at our group at IIT Hyderabad and then I'll quickly get on to the topic. So very broadly, we work on machine learning, deep learning, computer vision, those are probably the keywords that would get associated with our group. And within the group, we do both algorithmic kind of work as well as applied work. On the algorithmic side, some of the things we do are non-convex optimization for deep learning. I think the focus of this entire day is deep learning. So how do you help train deep learning methods faster? How do you prove convergence guarantees for those kinds of gradient descent methods that you use for deep learning is different is an important question that we try to answer. Explainable machine learning. I'll talk about that more as we go. Deep generative models for settings where you don't have too much supervision, deep graph representations. Those are some what I would call data agnostic research that we do. I mean, we're not looking at particular applications but just trying to solve fundamental problems in that space. In the applied side, most of the work that we do is on the computer vision side. So broadly human behavior understanding, recognition of expressions, poses, gestures, all of that is pretty much what we do. Off-late, we have a couple of projects on doing vision with drones, both applications, both in agriculture, in defense, security, as well as disaster management and those kinds of applications. And we also have a long-term project with Japan where we are doing deep learning for agriculture. So where we actually have experimental farms, it's a collaboration within Agricultural University where we have drones flown over farms in Hyderabad and we record those videos and work with that. Just some of the venues that we try to publish our work is CVPR, ICCV, ICML, NIPS. Those are the venues that we try to target. Just with that quick background, let me step into what the focus for today's session is explainability in ML. So broadly speaking, I would cover three aspects in this particular talk. The first thing is to give you an overview of what explainability in ML means, what are the efforts so far. So we'll talk about that. Then we'll have a second segment where we talk about visual interpretability in CNNs, convolutional neural networks. And lastly, very briefly, we'll talk about future directions in this space. So that would be the outlay of this talk. So some things to keep in mind. It's a semi-technical talk, probably some technical details, but broadly, it's a high-level talk. So if any of you don't want math, you can still listen. It's okay. You shouldn't miss much. It's intermediate-level talk. So I'm going to assume that you have a basic background in deep learning. Anyone who's completely new to deep learning here? Anyone? Okay. Okay. So I'm going to assume, I do have a couple of slides. Well, I'll just breeze through that. But otherwise, the assumption is most of you know, CNNs, RNNs and at least basics of neural networks. Okay, nothing, nothing more than that. And obviously, your focus will be computer vision because we're going to talk about CNNs in this particular context. All right. Okay. So I think all of us now, I mean, this is a redundant slide, but more to bring things into the context. I think all of us know that machine learning has been very, very successful. There have been applications ranging from science to web to marketing to manufacturing where machine learning is used on a daily basis. So they're not going to go further. And if you look at deep learning as a subset of machine learning in a broader sense, it is a group of algorithms that comprises subset of machine learning models. You can very broadly, the successes of deep learning have been in the perception space, right? You look at, say, vision, text and NLP. Very broadly, these have been the three domains in which deep learning has been successful. And the number of papers in this space have been increasing. And this was an interesting graphic that I found to understand the proliferation of deep learning. This is a graph that shows the number of deep learning models on Google servers. Okay. It's an interesting way of studying how deep learning has been progressing. So apparently the numbers of number of deep learning models that are on Google servers, which are deployed in various applications that they have is also exponentially increasing. Okay. Just one interesting way of looking at how deep learning has been growing. Okay. This is something perhaps all of you know. Let's now look at a thing a little bit more closely. So if you look at where machine learning is today, right? If you look at all the machine learning applications that we use in practice today, then you would look at problems like what is the product relevant to the user? What is the sentiment of this tweet? What are the objects in this image, depending on which domain, which application you're going to work on? Okay. And the general abstraction out of all of these problems is what is X? Right? That's the problem that you're looking at in all of all of all of this context. And if you see the applications in which you and I use machine learning in today's world, some of the observations are the cost of making a bad decision is not much. Right? A bad movie recommendation. You lose 500 rupees. You lose, you lose three hours. That's all right. Okay. That's come. That's completely all right. Okay. Accuracy is often the most all important metric that you're looking for. Okay. Variance of accuracy. I mean F1 score. I'm going to group all of them into what I call accuracy, whether you call them F1 score, precision, recall, all of them are one in one particular kind of a metric that you're actually looking for. Why a particular prediction was made doesn't matter at all. As long as revenue is optimized, as long as the monetization is not affected, you really don't bother about why a particular prediction was made. This is where things stand at this point in time. And it's highly one dimensional. You're trying to optimize for one particular metric. You're trying to, I mean, you look at any machine learning work, the results presented are one particular metric where you see accuracy or F1 score or pretty much that's the overall framework in which most of machine learning applications operate today. Okay. And if you look at where does machine learning, where is machine learning yet to fulfill its promise? You look at complex real world systems where I'm not going to trust if let's say I take one of my family members to a hospital. And then there is a reception desk which uses a machine learning algorithm that says, Oh, your, your family member is free from cancer. You can go home now. I'm not going to take that. Right. I am going to, I do want, I would want to insist that I meet somebody who has the manual expertise and then get that and then only then only go. Why? Because I want an explanation for the decision that's being made. So if you abstract this out again, places where machine learning is yet to fulfill its promise are complex real world systems. Examples would be risk sensitive systems. That's medical diagnosis, financial modeling prediction. I think those are examples of systems where you don't directly use machine. I mean, there could be places components. For example, in medical imaging, of course, yeah, machine learning is used to highlight particular regions in an MRI or a CT scan or things of this kind. So in subsystems, it's used not for the final decision making. That's what I'm trying to indicate here. And a particular project that I'm personally involved in is in safety critical systems. For example, cockpit decision support, we're currently working with Honeywell, where I mean, the idea is to see if decisions have to be made. I mean, a system like that of an aircraft has plenty of sensors with lots of data coming in at every point in time, humongous amounts of data. So and it's often very difficult. I'm not sure how many of you have seen the interface in front of a pilot. It's extremely difficult to parse. You have to be trained to be able to parse that subsystem in front of in front of you. And often only an important part of it gets highlighted to the pilot. So now how do you make decisions on the fly? Let's say a particular pilot is asked to drop 1000 feet. Now, if that was decision was given by a machine learning algorithm, does the pilot should the pilot follow it not follow it? Would the pilot ask for an explanation why it needs to be done? Those are some things to think about here. And that's the focus of explainable ML, because in these kinds of applications, the cost of a bad decision can be very, very high. The cost of a bad decision can be life and death. Those are situations in which we don't trust ML yet. Those are situations in which we don't trust accuracy may not be only objective. It's okay if the performance is not as good if the explanation is something that's rationalizable. That's something that you definitely want, which is what is good with humans. Humans can, I can recognize the objects in front of me, but if you ask me why is that a monitor, I can give an explanation. How I give the explanation, even we don't know how the human brain works and how those connections fire. We still have not understood, but at least I can rationalize it to some extent, which is something existing ML systems cannot do. And there is a need for a multi-dimensional perspective rather than look at just one particular performance metric like an accuracy or an F1 score. You probably need a different holistic way of looking at how an ML system is performing. Okay. So now the question is what then do we really need beyond all of these accuracies and all of these metrics that we use today? What do we really need in ML? So some of the wish list here are we want a human understandable rational in decision making? We want, when a decision is made, some rational which is human understandable. I think for those of you, I thought I should definitely mention this because I'm sure many of you in the industry have heard of the GDPR. That's now out. I think if by the EU, am I correct? I think EU has already passed it from May of this year. So one of the important contentions in that particular in the GDPR entire documentation was if there was data and that data was acted upon by an automated system, then how does that deal? I mean, how if that data pertain to a particular person was the decision was made by an automated system. How do you provision for data protection in those kinds of scenarios? That was a very, very important point. And I mean, the only way to handle those scenarios is if your prediction models can explain its decision rather than blindly take data and just give an output. So trust or confidence in a system is something that we want. It's not in addition to rationalizing you want. I mean, I'm sure in any kind of a system where you use machine learning for a new product or for a new service in any upper level management. In fact, I think I have a statistic where PWC recently conducted a survey and apparently two thirds of CEOs around the world felt that stakeholders will start losing confidence in companies using AI. So it seems surprising mainly because you really cannot have confidence and a new system comes up whether the ML decisions that you're getting out of that system are really useful or really worth trusting in is something difficult to promise. Compliance with ethical principle. So there's actually a you can look up this link. There's actually a statement from Gartner that says that by 2018, half of business ethics violations will occur through improper use of big data analytics, which is a pretty tall claim. I'm not sure exactly when this link was out. I'm not sure if these statistics is true right now, but that was the claim from Gartner. So compliance with ethical principles is another reason why we need explainability. So if you use machine learning as a black box, you're not sure. I mean, this is a huge topic in machine learning now. I'm not sure how many of you are aware bias in machine learning is a huge topic today because remember that machine learning today is completely dependent on what data sets drive machine learning. And if data sets are Caucasian, maybe it's not applicable for Indians, right? So that kind of a bias that's built into data sets is a huge problem. And now if you're going to rely on that for deciding on people's lives, okay, that's going to adjust doesn't make sense. So that's where ethics winter fills into the comes into the picture and bias in machine learning is a huge topic in machine learning conferences like nips and ICML at this point in time. So enhance control and robustness. So if you understood how a system works, you know how to play with it, you know how to control it. If you know, it doesn't work in this setting because of this reason, you know that you probably have to give data of a particular setting to that system to make it work better. And finally openness and discover openness of discovery and scientific research. If you understand the system, you know how to improve it. So all of these things are something that's important for us to take ML to the next to the next space. So for those of you have read more in this space, you would see various kinds of terms that people use in this context, explainability, trust, interpretability, and it's a new topic in machine learning. It's been around. I think it became popular when DARPA introduced a huge huge initiative a couple of years back called explainable AI. So in the last two years, it's taken off and there have been a lot of efforts around. I do have a slide that summarizes many efforts, all the efforts so far to a reasonable extent, at least. So there's been a recent work just released last month on archive where they tried to categorize all of these methods and define the terms more succinctly. There have been two, three papers in this space, but I'm going to subscribe to this particular definition here. So they define interpretability as trying to understand what a model is doing or has done. So if a machine learning model gave you a prediction, what did it do to get to arrive at that decision is what we're going to call interpretability. Explainability is a little bit more abstract. Okay, explainability is can your explainable model give the reasons for neural network behavior? Can it gain the trust of users? Can give you insights or causes for decisions that's slightly a higher level of abstraction rather than try to understand just what went did this neuron fire that neuron fire that would be interpretability. If you want to give a human interpretable explanation for a decision that would actually become explainability. So that's the definition we're going to go with. I think that's it's definitely debatable. I think this is still a new moment later. I think the field is growing at this point in time. These kinds of terms and formalizations for these approaches are still being defined. I'll talk about that when we come to the open problems at the end, but this is the definition that I'll go with at this point in time. So if you look at today's machine learning models, you can. So this is the statistic that I was talking about 67% of business leaders that took part in a CEO survey in 2017 said that AI and automation will impact negatively on stakeholder trust levels in the industry in the next five years. And one of the reasons is you don't know whether to believe whether the model will work or not. So everybody wants to use ML in their products and services, but you there's no way to find a robust way to check whether it will work or not other than metrics that could be biased themselves. So if you look at existing machine learning models, this is a rough chart of how how they perform on the accuracy. This is interpretability tradeoff. So neural networks and deep learning is somewhere here. Most accurate today have won most challenges, but not interpretable at all. Then you have support vector machines, random forests, k nearest neighbors, decision trees, linear regression on that on that particular spectrum. As you can see, decision trees may not be as accurate, but they are some of the most interpretable models for obvious reasons because decision trees. You can actually look at the variables and say why a particular decision was was made. So in some sense, a random forest is a medium ground between these because an ensemble of decision trees random forest performs a little better than a single decision tree. But again, you can you cannot give as good an explanation as decision trees because you have to aggregate your decision trees and come up with an explanation, but it performs a little better than a decision tree in terms of accuracy. So so your current today's frontier of machine learning is on this spectrum, right? If you want to perform better on accuracy, you're going to lose on interpretability. But what we want to go for for tomorrow's machine learning models is improve both of them simultaneously. So you want models that perform well both on the accuracy metric as well as in terms of explainability. That's what we want to go towards. So where do we stand today in terms of efforts in this space? So again, it's in the last two, three years, there have been plenty of efforts in trying to do different ways of trying to explain machine learning models. I should say that there have been efforts even in the 80s and 90s in trying to understand how neural networks can be converted to decision trees that in a manner in which you could explain your decisions and so on. So some of these keywords here talk about those kinds of models I'll probably briefly go over them. So I'm again going with the categorization in this particular paper in terms of looking at all of these work and trying to put them in bins. So there have been a few different models in recent years that I've tried to look at explainability in terms of processing and I'll explain what that means in a moment. And there have been a few efforts in terms of trying to directly produce explanations. And finally, there have been a few efforts in terms of trying to understand representations and so on. I'll briefly go over all of them. But before if any of you are actively working in this space. So maybe in this categorization for those of you who've used lime anyone who's used lime here. Okay. So if you use lime lime is it's pretty popular. I know it's pretty popular in the industry. I've heard many industry use cases of it. So that's that can be bended to a proxy kind of a method. Then for those of you who looked at cam grad cam and things like that which I'll talk about in more detail as we go. Those would go into the saliency maps kind of in that bin of these kinds of methods. So let's look at some of these methods. So linear proxy models stand for so there could be many other categorizations of these explainable explainable models too. So one traditional way people look at various efforts so far in explainability is model agnostic methods and and just methods that depend on the model. What that means is model agnostic methods are you use whatever you want for a machine learning model SVM neural nets decision trees whatever you want. Then you use something as a meta meta method to try to understand what the model actually tried to or other what were the relationships between the input and output of that particular model beat SVM beat decision tree. Those are model agnostic methods and then model dependent methods are where if you use an SVM for classifying you try to go deeper into that particular SVM and try to understand why it made a particular decision. Those are another that's another way of classifying a machine learning models today. So linear proxy methods refer to the models where are broadly model agnostic methods where you've trained the machine learning model and it's a black box model to you at this point in time. It's trained using whatever ML algorithm you wanted. So now you try to play with your input and output and try to find which which space in your input corresponded to a particular output the most and so on and so forth. So in a broad sense, these kinds of models, there's also something called SHAP, SHAP and so on and so forth. Broadly, these kinds of models are you can say are an extension of or a formalization of sensitivity and sensitivity analysis methods. So because in traditional statistics, what you would do to get explanations would be sensitivity analysis you put up your input a little bit and see what happens to the output and so on and so forth. Those are what these kinds of models do. Then decision trees you already know. I mean decision trees are what help make decisions directly. But by decision trees here, I'm also referring to methods in the past where people have converted neural networks to decision trees. There have been methods in the 90s and they've also been some recent methods. You can probably look at this particular paper down. They go deeper into this kind of a classification. So so that's any method that tries to convert a machine learning model to a decision tree to give an explanation falls into these kinds of methods. CLC maps is more in I think in vision and text, perhaps where you process the deep learning model on a particular vision or text input and then you try to find out which part of the image or which part of the text was the network focusing on while making a particular decision. So that's another way of doing it and we'll that's the kind of models I'm going to talk about for the remaining duration of the talk. Then automatic rule extraction is it could it's in some sense rate to decision trees, but there are also other kinds of methods where you can extract rules from a given model based on various kinds of methods. Again, I'm not going to step into each of them. If any of you are interested, I'll refer you to this particular paper here. Then there are also explanation producing methods where people try to come up with scripted conversations. So you try to or what these are also called generative models where you train your model to generate an explanation for those of you aware of GANs and generative adversarial networks. So instead of generating an image, you generate an explanation for a given prediction. So assume you have a data set where you have a particular problem that you're working on as well as the corresponding textual explanations for the predictions. Then you train a GAN to be able to give an explanation in real time. So that's another family of models there. Attention based models are as a where you have an attention module within your neural network and that attention module tells you which part of the it's slightly different from saliency maps because you image captioning is a good example for attention based systems here, where if you're trying to take an image and you want to train a deep learning model to give a caption for that particular image, then you use an attention module inside your deep learning network and that attention module tells you which part of the image the model was looking at while while throwing out a particular word in your or a phrase in your caption. So those are traditionally bended to attention models and broadly speaking, all these design disentangling representations methods. So how many of you familiar VAE's variational autoencoders? So variational autoencoders is also something called info GAN with GANs. So these kinds of methods try to learn a latent space where the data belongs to and then go from the latent space towards a prediction. So the latent space can be considered as a disentangled set of representations. For example, typical example that's used in VAEs and info GANs is if you had a bunch of face images and let's say you want to generate more first face images, you try to come to a low dimensional space, a latent space where a particular dimension corresponds to a beard on the face, a particular dimension corresponds to glasses on the face, so on and so forth. Okay. So then when a decision is made, you can try to see which latent variable fired and then try to say I classified this person as so and so because he or she had a beard or had a glasses on or so on and so forth. Okay. So that's how all these methods for disentangling representations fit into the scope of explainability. And lastly, there are also methods that try to study what are the roles of particular layers, what are the roles of particular neurons, what are the roles of particular vector representations that you get out of a network. All of those also can be considered as trying to interpret or explain neural networks. Okay. So I'm not going to step into all of them, but that's another broad space that that can be looked at as explaining deep models. Okay. Just I'll just talk about one of the models which is very popular lime. So lime is a model local, it stands for local interpretable model agnostic explanations where given a model and this is your prediction, irrespective of whatever model you used here, it doesn't matter whether you use SVM decision tree neural networks, whatever you used, you give it to lime and then limes gives you a decision of what feature could have led to a particular decision. Okay. And the way it goes about doing it they show in their paper, they actually show results with text as well as images. So the way they go about it is you take a particular image or any data point and you put up in the neighborhood of that data point and then you try to see what output you get and then you regress on those perturbed instances on the output and then try to get which feature give is most likely to have led to a particular output. So for those of you who don't know about it, you can I think there's a they have the code and it's very popularly used for anybody who tries to at least I have heard several industry use cases of people using lime to explain decisions. All right. So that's the brief overview of explainability in ML as it stands today and I'll come back to what are open questions at this point. So we'll take a detour to visual interpretability in CNN. So that's some work that we have done in that space also. So just for those of you are very new to deep learning, just a quick two to three slide introduction. So that's a neural network and it's trained using back propagation and gradient descent and you have a loss function that's used and you back propagate using the loss use gradient descent and back propagation to train based on the loss function and update the weights of the neural network and what's a convolution neural network or a CNN. You don't connect weights directly and you don't use what's what's called an inner product between your first layer and the next layer use convolution rather to do it and use a concept of sharing parameters and the main operation that you use in a convolutional neural network is convolution. Convolution is where if you have a bunch of parameters that are defined by a matrix, then you have the weights also defined by a matrix. Convolution is nothing but at least 2D convolution is nothing but inner product of this matrix and a doubly inverted version of this matrix. So that's the simple definition of convolution for those of you don't have a signal processing background. So you have a matrix, you have another matrix here, you invert this matrix both horizontally and vertically and then do an inner product between every window here and this particular matrix and you'll get a value as an output and that becomes your output of your convolution. I'll leave it with that simple definition. So convolutional layer neural network is where you have an input image. It could have multiple channels. For example, if you have a color image, you would have RGB that are those are three three channels there and then receptive field is the size of the convolutional filter that you use the filter that you have and based on that you come up with an output map. The output of convolution is an image. If you have an image and you applied weights to it unlike a standard MLP you get an image as an output and you could have multiple feature maps that lead leads to a depth in your output. So unlike traditional MLPs where you have to decide the number of neurons in each layer. In this case the neurons are pre-decided once you decide the number of weights and a few other parameters like valid convolution and so on and so forth. And you also use activations like reluze and so on and so forth and mentioning this because these are some things that we'll use as we go forward and in the CNN you also have something called a pooling layer where you take a patch of an output of a convolutional layer and then you aggregate it. The aggregation could happen using a max average L to you can use any kind of a pooling and for example, if you had an image such as this and you do a max pooling. So you take a region 2 cross 2 region here and then you take a max of it put it here so on and so forth. That's your max pooling layer and then to put together your vanilla CNN you have a convolutional layer a pooling layer a convolutional layer a pooling layer. There could be blocks where you don't have a pooling layer have only a convolutional layer and finally you have a fully connected layer and then you have a soft max classifier at the end. So that's your overall architecture for a convolutional neural network. Again there's a simplistic one these days you have normal batch normalization normalization layers. You have skip connections and residual blocks many other variants of this, but this is just your abstraction of what is a convolutional network. Okay, so now trying to understand CNNs one simple way people have always tried to understand CNNs is you look at the weights that you have learned in your neural network. Remember weights in CNNs are also matrices. So if you plot them as images, it gives you an idea of what the neural network is trying to do. That's one of the simplistic ways in which you try to understand what's going on and it's typically known that if you take the first layer of weights and visualize them, you get filters such as these. So you and these typically look like the basic image processing systems of your visual cortex where you look at textures and edges and edges of different orientations and textures and so on and so forth. Unfortunately, these kind of weights are interpretable only in the first layer. If you go further down the if you go further down the network, you really cannot understand what's going on in these kinds of weights. But to interpret CNNs the last few years, there have been a few different efforts. So one of the efforts has been to use back propagation methods where you assume that your CNN is of is a black box, which is a function, which means you can back propagate straight from the output to the input instead of backpropagating, backpropagating with respect to the weight. You do do f by do x instead of doing do f by do w. So it's a simple chain rule change. And once you do that, you can now try to backprop with respect to a particular class and try to see where the gradient was most activated in a particular image. So once you train an Alex net kind of a model for the imaginary database, then you take the cat class and try to see where was the gradient most activated for this particular image. It shows these regions that gives an indication of what part of the image the network was looking at for a given for a given image. So there has been an extension to these kinds of methods and there have been a couple of efforts called deconvolution networks and guided backpropagation where you use a relu activation function rectified linear unit activation function and a deconvolution network. What you do is mean when you backpropagate you change wherever the inputs were negative. You don't allow those gradients to come back back to the image. So in the same backprop approach that we saw on the earlier slide, wherever you had a relu activation, you don't allow those negative gradients to backprop to the image. And then there was an improvement called guided backpropagation where both the activations that turned out to be negative as well as the gradients that turned out to be negative. Remember relu is an activation gradient is backwards, right? If both are negative, you don't allow any of those gradients to go back to the original image. So that was called guided backpropagation and they showed that if you use deconvolution or guided backpropagation it helps you understand what the network was looking at more clearly. I mean the main idea is don't worry about the negative influences in your network. Look at only the positive influences in your network and you'll get a better idea of what the network was looking at while making a prediction. That's the idea in this particular in this in this particular framework. But there have been some limitations of guided backpropagation. If you use a cat class, this is a dog class, it still ends up focusing on the same region, irrespective of what class you try to do. And that led to the proposal of one particular method called class activation maps. So in class activation maps, this was proposed in CVPR of 2016 where they said that you when you need to explain, you need to look at it with respect to a particular class. So what they do in their particular model is you have a convolutional neural network and then after all your convolutional layers, you take the last convolutional layer and then you do something called global average pooling. You take each of those feature maps in that last convolution layer and just average all of the values in one particular feature map and represented by one particular value here. You take the blue feature map here, average all the values in that particular image represented by one value here, so on and so forth for all the feature maps here. And then you train a linear classifier to learn the weights between each of these aggregated feature maps and each of the classes. So you have to learn if you had 1000 classes, you would learn 1000 regressors to be able to get these weights for each of these aggregated feature maps and your output. And once you have that, if you take a particular class called Australia interior, you know how that it was how it was weighted with respect to each of these feature maps and that if you put them together, you get an idea of what the network was looking at while giving a particular class as the output using this approach. They showed that I mean, if you take this particular image, this is an image of a dome and you can see that if it's classified as a palace, it looks at this part of the image. If it's classified as a dome, it looks at this part of the image. If it's classified as a church, it looks at only this part. It doesn't look at that entire facade there and then so on and so forth. So it gives an indication of what the network was looking at while making a particular class prediction. Unfortunately, the problem with the cam approach that we saw on the previous slide is you have to after you train the CNN, you still have to train 1000 regressors for each of your classes that you have in your model. That's a problem. Okay. And that led to the definition of what is called grad cam, which is called gradient based cam where they realized that these weights that you're trying to learn in cam actually can be obtained directly from the gradients. The weights that you have is nothing but do why with respect to the activation map. If you take do why with respect to the activation maps, that is directly your weight. You don't need to train regressors further to learn those weights between your activation maps and your output. And that led to this method called grad cam, which is again popularly used for saliency, saliency maps today. This is your overall architecture for grad cam where you mean you take those weights from the gradients and then you do a relu on those gradients, which means any negative gradient. You don't allow it to pass wind up in two minutes. So and then you take the you don't allow the negative gradients again. The same idea of not taking the negative gradients is to look at only positive influences. And why are negative gradients not being involved? I mean it just empirically they saw better results when you use only positive gradients. The interpretability was much higher. That's about it. And they also combine this with guided backpropation, which we saw in the earlier slide, which means you don't have any gradient corresponding to a negative activation or a negative gradient. Both of those are set to zero when you go back to the gradient. And with this kind of an approach, they actually found that you could actually get discriminative saliency maps with respect to each class. And these are some sample results of their work on image captioning models. So you can see that they showed that as you caption, you can look at different parts of the image when you give different words is output at each step. So one of the limitations in our work that we found on this particular model is that whenever there were multiple objects in the scene or when there's a full object, the network ended up focusing on part of the object or could not look at multiple objects. And that led us to come up with a different formulation from our perspective. I mean, I'm just going to give the conceptual overview. And you can ignore the math. What we realized with the grad cam approach was that it was looking at the weights as an entire aggregation of the feature map, not looking at it pixel wise. While we felt that there was an explanation that was required pixel wise of with respect to an activation map and the output. And we realized that you could change the formulation a little bit by making your weights that you're trying to learn between your activation map and your output. And remember both grad cam and the work that I'm presenting right now can be done with respect to any layer of the CNN. Those activation maps can be with respect to any layer. I am just demonstrating it with respect to the last layer. So we found that that's gradients can be weighted with respect to each of those pixels in the image and it happens that you can get an easy close form solution. If you work out a math, it turns out that you can actually get an easy close form solution, which means this is just a simple one line code where you can evaluate this expression and then try to get the better activation, better weight of those activation maps with respect to the your output. And using this, we could find that if you have multiple objects in your scene, it gives a more holistic saliency map. I mean, these are examples of different here is a flamingo. You can see that grad cam actually gives only a part of the image, whereas the grad cam plus plus, which is our work that was accepted at WCV this year. So that could give a more holistic saliency map in this particular context. Here are more examples. Okay, here is an example where you have a gray whale that's peaking out of an ocean. You can see that grad cam gives a pretty hazy representation of what your network is looking at, whereas our model could give better results. I'm not going to go too much. You obviously know that if you got a paper, we probably had some good results. Yes, I'm not going to go into that too much. If you're any if you're more interested, so we had some results on image captioning and we had some results on doing this on videos too. So we had some how do you explain decisions on videos? That's also something that we have in our work. I think ours was the first to be done in the space of videos. So we're interested. Our code is available in our archive paper is available. So we also have some ongoing work on how do you do this for we're trying to use causality in trying to do this for how do you explain decisions of RNNs? That's some ongoing work that we currently have with time series kind of data. If you're interested, I can talk offline. I won't go too much into that. Let me spend the last couple of minutes on just trying to see open directions here. So this is the space of methods that we briefly saw last half an hour or so. So now what are open questions at this point in the space? What are research directions? Right? So one of the first problems here is so now we all know that machine learning has a certain formalization, right? You have F going from X to Y where X is input data. Y is the output. You try to learn a function F that can be learned using a maximum margin classifier. It can be learned using a neural network. It can be learned using a probabilistic framework so on and so forth. What is a formalization for explainable ML? We still don't know. There was a recent paper that's trying to that was published last last month that tried to do something in the space, but I think there's a lot more work to be done in terms of how do you formalize machine learning when explainability is very required? And then how do you balance the accuracy performance with this interpretable interpretability tradeoff? Is interpretability always required? I think it's a question and maybe it's not required in several applications. There are only certain set of applications where you maybe there's a need to stratify applications and say these kinds of applications, a certain quality of interpretability service is required. Those kinds of things have to emerge in this particular context. What kind of data? What class of problems are more amenable for explainable systems? If you had data on the real space, is that better? If you have data in the discrete space, is that better? Those are all open questions at this point in time. If you had a multi-class problem, a binary class problem, what kind of problems would explainable systems be needed? Where would they work better? And how to, an important question, I'm done just one more slide. How to evaluate explainable systems? What kind of metrics do you use is all open at this point in time? I think I'll stop there. So these are some reference and resources if any of you want to take away and yeah, I'm here for any questions and discussion. So we're out of time. Since we have a break, we can continue with the questions. I'll make the announcements first. So those of you who don't want to ask questions can go for lunch. Walmart has a contest going on at their booth. So if you want to win a wireless headset, you can participate in that. BoF on AI and product will start at 140 in the BoF area, which is on the first floor. And remember to fill your feedback forms. Okay, we can continue with the questions. Anybody? Hi, nice talk. One question based on your previous slide. How do you evaluate the current explainable systems? So you said that's an open problem on how do you evaluate the current explanation? Yeah, I mean, so I skimmed over it. I mean, that's a very, very important point. I mean, how do you evaluate explainable systems? We still don't have. I mean, those are mechanisms that you have to develop. So again, there has been some recent work which have proposed some metrics. So in the work that we have published so far, I mean, you typically use some human subjects and see if the explanations were good enough. Use a broader range of subjects so that you know that across the population, you felt the explanations were good enough. That's one way of doing it. Since we were doing the saliency maps kinds of approach, there are multiple things that you can do. You can hide the rest of the image and look at only that part of the image that our saliency map produced and then run it through a CNN and see how the prediction was. Did it improve the accuracy is a question to ask. Then if it did improve the accuracy, yeah, perhaps that was the part of the image that was most relevant. So you can play around a little bit with those kinds of methods to so all these are in some sense hacks. I would say I wouldn't call them evaluation formalisms at this point in time, but those are some methods people use. So it depends on I think broadly at this point in this space, it's all application driven, whichever application you have. But I think that's an open question at this point in time. Hey, we need it was a very nice talk. So what we see is that for most of the explainable models that we have, we are trying to reproduce the same result using a simpler model. For example, we are piggybacking on decision trees. That's one approach. That's one approach. That's one approach. That's correct. But that gives the more accurate explainable part where we can get the direct result. But what I'm trying to ask is, do you think that this is a good solution? I mean, there are many nonlinear relationships and other models, okay, which actually did the hard work. And then we are trying to, you know, ram all of that into a simpler decision tree. So as you mentioned that it is a hack, but do you think that this hack is something that we should even try out and got it? I think that's a good question. So firstly, I mean, decision trees are not the only way to explain. I mean, it's just just that just that it's more human interpretable. I mean, we're all pretty easy to look at if then else rules and make make make decisions at this point in time. So that's the reason that people prefer decision trees, but definitely no, I mean, that's not the only way. And I mean, the answer to that question, what your question is again related to that is we still don't have evaluation formalisms for explainability, right? So another way of asking your question is, is decision trees the right way to evaluate? I mean, is that kind of an output the right way to look at whether a particular explainable model was useful? So that's another way of asking the same question. We don't have an answer to that question at this point in time. I mean, all of this is very nascent. I mean, this entire space is I think growing as we speak. Hey, when it will be available offline, so you can continue the discussion. Yeah. Okay. I don't know if this has to be done offline or right now. So the only thing was. Okay. So it's more about causality in RNN that you just talked about. Maybe due to short of shortage of time we couldn't talk about. So just wanted to do wanted to know more on that. Okay. It's actually under review at this point in time. Maybe I can talk offline, but the overall idea is we use causality. I mean, so there's a fundamental difference between which I think machine learning today does not does not do machine learning deep learning. None of them do is to segregate the difference between causality and correlation. Right. All machine learning models today are correlation driven. Right. So I think there's a very popular example of correlation with this causality. I think there's a statistic of people buying ice creams in summer. Okay. You plot a graph over the months of the year like Jan to December you plot the graph and then when sharks appear on the beach of an ocean or something like that. I think this was done in the Bay Area of M right. Okay. And then you have the same chart on the on the same chart you plot a graph of that you find that the charts are very, very similar. They have similar charts for both of these cases. Does that mean that you eat ice cream because a shark came up on the shore? Not really. Right. So the data is correlated, but they're not causal. One doesn't cause the other, but often in explainability, what we are looking for is what caused a particular output to happen. Right. And that's where causal models come into the picture. So we've been trying to look at can use causal models. There is I mean entire literature on that by Judea Pearl. I think he's a curing award winner. If you're not aware of that literature, it's has been it's actually has been a niche area and machine learning for many years, but I think explainability is kind of reviving it at this point in time. So our idea there is to use you be assumed that each node of a neural network is like a random variable and it's a graphical model, but it's what we call a structural causal model where you can study the causal relationships between those random variables and then we try to see which input random variable had the maximum. Maybe we can talk offline more about it. I don't want to spend too much time, but that's our overall approach to that particular problem. It's required. We can talk offline. Hi here. So you have shown an example of grad cam for CNN on image data. So CNN can be used not only on image data, but there are some applications in text data and all. So can this grad cam be generalized for CNNs which are used in different domains, maybe in text, maybe in some financial data presentation. Methodology wise, there's nothing that restricts it from. I mean, I know people use CNNs in speech for as spectrograms and in even in text as 1D CNNs and so on and so forth. Methodology wise that there's nothing that restricts it from being used. I'm not. I know there have been some efforts recently on saliency maps for text and things like that, but I'm not very sure if grad cam has been applied per se. I mean, I know there have been similar efforts, but not grad cam per se. I'm not very sure. Yes, I mean follow ups offline, follow ups. Okay, sure. So I wanted to ask because most of the times of our models work, but some of the times it doesn't work. So do you think we have to also think about that aspect of the solution and can we use existing explainable models or solutions to explain those kind of scenarios where model is not working because one of the scenarios like we have stop sign and if we change the bits of that image that represents that to model as a triangle, that is quite a big problem, right? It means we have to explain also that why it did not work on that one. Sure. I mean, I have two thoughts to that. I mean, firstly, I think in some sense is related to the model because when you say a model does not work often when you say the moment you say it does not work, you have to talk in terms of a performance metric, right? And often you talk about that in terms of an accuracy, right? And that goes back to the accuracy is the interpretability tradeoff and trying to see where is the balance between those. That's one angle. But the other angle, perhaps that I did not speak about is in this entire talk is the relationship of this entire discussion that we've had to adversarial data, right? I think many of you who have active and deep learning, you've probably read about the adversarial attacks and defenses and so on and so forth, right? I mean, that becomes an important relation to explainability because if a small perturbation in the data is going to get a class cat classified as a dog. I mean, the neural network is good luck to neural network to explaining that. I mean, it's not going to be easy, right? So explainability is going to be very hard when you have adversarial attacks. So actually, there's a very close relationship between that space of work in deep learning to explainability in because let's say it grad cam works and you get an excellent explanation for a cat being classified as a cat and the tomorrow you just add a simple Gaussian noise and then it gets classified as a cat. Where did that explanation go? Right? And the image is almost the same from a human eye, right? So there's a close relationship which is not being studied at this point. It's a it's an open problem at this point. So I had a question about lime. So lime usually deals with the local space and the sample that we are providing. It's individual specific explanation. Is there any research that is going on right now which is extrapolating it to the global space? Yes, yes, very much. Yes. I mean, I would again recommend you to look at that particular paper that categorizes all of these methods. I think one of the categorizations they talk about is also global methods. So in fact, the method that we proposed with the causal RNN, I didn't get a chance to talk more about that can do both local and global. I mean, that was our main that was our main USP of that work. Okay, we try to do. So given a particular instance, you want to explain, but you want that to hold for your entire data set also. So there are efforts in that space. So far, I think we have not seen work with that the same explanation method that tries to do both local and global. We have not seen simple feature selection methods are global. Any feature selection method is global. I mean that space has been active for decades together in machine learning, but your lime is local.