 OK, our next speaker is Professor Dukat Daniel from MIT. Dukat had his PhD from music working in some industry, and Dukat has been working on many research areas spanning from the macro-technologies to deep neural network-based applications. Today he's going to talk about how to quantify the lack of robustness in neural networks. Let's work on it. Thank you, Sijaf, for the kind introduction. So first and most importantly, I would like to acknowledge the very crucial contributions of my senior grad student, Lili. Lili could not be here. She's traveling internationally. I really wanted her to be able to present her work. I would also like to acknowledge the precious contribution of my collaborators at IBM. Sijaf is one of them, Pinyu. A lot of other collaborators, students, colleagues, funding from IBM AI Lab and MIT Quest for Artificial Intelligence. I have to apologize. I could not attend the beginning of this session. What I have to say, I'm guessing, is not going to be that different from what some talks had to say at the very beginning. I have seen my colleague, Alexander Magry, giving fantastic talks before. I have to say that I agree on everything that he says. And you probably recognize a lot of the things that he said in my talk. That is to say that it's the last session. It's the last talk. I will not be offended if you decided to start your weekend early. I've also decided to target a different audience, at least a different audience than what I've seen done in other talks at this conference, a non-technical audience. Because, as you can see, my title is a question. And I think that when it comes to this topic, the questions are more important than the answers. And that is because I, myself, my group, and probably a lot of people in the community, if not all of us, still don't have very clear answers. So I'm going to keep it high level. It's also good because right before lunch, a lot of math might be problematic. So here's our good friend, the neural network, a collection of neurons with weights and activation functions that you can use to do pretty much anything these days. But for example, they can be used to do image classification. We know that performance gets better and better the more training data you have. We've heard this a million times. Here's one more time for me. But how many times did you stop thinking about what it means performance? What people actually measure? A lot of my talk is about what should we measure. So what typically is measured is the number of errors or the percentage of errors, especially when compared to what a human could do. And maybe 20 years ago, things were not going very well for neural networks. Yes, they've been around for 20 years for those of you that were not born at that time. Maybe a human could do 95% accuracy, meaning number of correct guesses or classifications out of 100. And a neural network might be able to do only 30%, 40%, 50%, I don't know, something like that. Things kept getting better and better. Eventually they saturated. I apologize for the contrast here, not very visible. But again, fortunately, if you're an optimistic, maybe not so fortunately, if you're like me more of a skeptical person, things started getting better again. So much better meaning again, number of errors or percentage error, that we started getting better than humans. So out of 100, a very highly performant deep neural network could, for example, get 99 things right, just make one mistake. A human still 95%, so five mistakes. So what do you choose? Well, of course, we choose the deep neural network for any of our decision-making from now on. So they are quickly becoming what I would call our hero. The neural network is coming to do everything for us, as I was mentioning before, from image classification to object detection to natural language processing. So already most of us are dreaming to having them drive our cars, hire our people, invest in the stock market for us, so on and so forth. But I would like to bring back the question that I mentioned before, what does it mean to be better? And what I mentioned before is something that will highlight a dark side of our hero, potentially. Well, at least, let's have some questions about this. So what we count is the number of errors. But do we ever stop and look at those errors? I would like to stop and look at those errors. So for instance, yes, it's true that this very highly performant deep neural network make very few errors. But well, I don't know, that one error that they make, maybe that's a very difficult case, and no human would ever make a good classification on that specific input. Or maybe there are very few humans that would be able to guess it right on those specific cases that a deep neural network get wrong. But how would you react right now if I work to give you examples of errors that the deep neural network makes very highly performant and no human on Earth would ever make an error on that specific input? Specific input is the key word here, specific input. So we should actually try to look at those. And also, how about, OK, maybe we just stick with errors that both the deep neural network and the human make. Can we look at how big those errors are? So what if a human, yes, makes an error, but is not that big while a deep neural network makes a huge mistake on that specific case? How would you react in that case? What would you prefer? These are questions we should ask. And they go beyond math, in my opinion. All right, so let me start digging in and show you some images you probably, if you are in the community, you have seen all of these images. And you have seen where I'm going. If you weren't a Madrid's task talk today, you know where I'm going. But don't need to answer. Just answer to yourself what you think this is. I'm guessing that a lot of you might conclude this is an ostrich. Good for you. The first time I saw it, I wasn't really sure maybe because I'm not a native speaker. It's about the language as well. But definitely, I was able to conclude it's some kind of bird. And no question in my mind, and most likely no question in anybody's mind in this audience, this is at least an animal. Well, good for us, the highly performant deep neural network agrees completely with us and extremely confident tells us it's an ostrich. What about this one? Well, if you ask me, I would tell you, well, now that I learned that that's an ostrich, this is an ostrich too. Actually, it seems like it's the same ostrich to me. How would you react if I told you that that same highly performant deep neural network that gets fewer errors than humans tells me it's a unicycle? Particularly concerned about the fact that it's not even the mistake this neural network makes, this superhuman deep neural network makes a mistake. And it makes a mistake that it's not even another kind of bird. It's not even an animal. It's a unicycle. It's not even an animal. So I'm upset about this. I'm also upset about the second thing. If you look at the last neural, the last layer that is responsible for this assertion corresponds to an ostrich, it's happily firing extremely confident, just as confidently as before, that that's an ostrich. And to me, there's nothing worse when you're trying to build trust than someone who says something wrong extremely confidently and thinking they're right. That's very upsetting. Or at least, it doesn't make me want to trust that thing or person or whatever that is. All right, I said a lot of negative things about deep neural network. It's about time I say something positive. So let me try to defend this. Well, this is the very, very clever work of some extremely gifted hacker that had managed to get information, inside information, on that specific deep neural network and knows the structure, knows the number of layers, knows the coefficients, knows everything about that neural network and is able to exploit very cleverly all of those things. This is called an attack. All right, so it's not something that happens every day. You need a hacker that has all of this information. This means that the people that started worrying about this are the security people. Because there is an attacker, there's a hacker, and they started thinking about what can happen, for example, if your self-driving car gets hacked and it doesn't recognize stop signs anymore, is someone trying to kill you or something? Well, this could be one way, unlikely, but possible. What if your Siri doesn't recognize what you're saying? Because a hacker, again, gets in and has access to everything inside the deep neural network of your Siri phone or whatever you're using. People started designing glasses that you can wear, and deep neural network will think that this man here is actually this woman over here, just because they were in that special glasses. Students here at MIT were able to take a living animal, like this turtle, and just paint something over it and make deeper neural network think that that's confidently a rifle. So it's not just an image problem where you change a few pixels. All right, so this feels like it's a serial security issue. Yes? Let me bust a few myths for you. Point number one, that hacker that I mentioned before needs to have complete access to the inside information that deep neural network actually does not need to have complete information on the neural network. It does not need to know anything at all about that neural network. And some of my collaborators showed that you can just look at that neural network as a deep, as a black box, and just be able to ask questions and get answers. Is enough to synthesize a picture like this? To me, it looks like there are two guys happily skiing up in the mountains. There's no question to me that that's what this picture represents. But then I look at the deep neural network and there's a neuron there corresponding to dog that says 91% higher than all the others. They had no information about how that neural network was trained, what dataset was used, what the coefficients were, what the structure was. Just ask questions. All right, now I'm getting a little more concerned because it's not enough to protect the inside information on my network. Let me bust the second myth for you. I don't even need the hacker. I don't even need that very skilled person. There are very few in the world and they're all in this community of trying to hack neural networks and they're very smarter than I am and they're capable of doing this. I don't need them. Things are a lot more serious than that because people have looked inside ImageNet, the database of existing images, and you can find images that have not been hacked by any hacker, any gifted person with this insane ability of doing these things and they were able to find pictures that this highly performant deep neural network, the one that has superhuman performance, the one that gets 99% of the answers correctly more than a human would do, that's the same neural network that tells me that this thing here, number three, for example, it's a pretzel. Yeah, I see a mushroom. I might not know what kind of mushroom but I do see a mushroom. This is not a butterfly, apparently. This is not a squirrel. Yes, I don't know that it's a fox squirrel but it's a squirrel. It's not a sea lion. So hopefully you're starting getting concerned. This is why I said that sometimes asking the question or actually having you ask the questions is more important because what I want to do is have more people that ask these questions because all of us can hopefully find better answers for the community rather than focusing on me giving answers to some of these questions. So this is particularly concerning to be because it moves outside of the realm of just security. There's no hacker anymore. There's no malicious attacker. There's no getting possession of inside information of your network. So if you are considering planning or dreaming about handing over your decision-making to deep neural network, you better start being concerned about safety and fairness, for example, issues. Especially, and here I have to make a distinction, of course, because if you're asking your deep neural network to make decisions about things that have no consequences, you're trying to do specialized individual pricing and ask me different price for priority boarding and privilege of getting my seat before others. Who cares if you misclassify me? What's the worst thing that can happen? I'm sitting in the back of the plane in the middle seat. I can live with that. It changes if there are really substantial consequences to the mistakes of this network. And I don't care if the number of mistakes is smaller. I get particularly concerned if it makes a mistake on something that no other human on earth would make a mistake on. That's what I got concerned about. And the mistakes they make is much, much larger than any human would do. Topics, actions, things like autonomous driving, insurance influence medical treatment selection, employment hiring decisions, prison probation approval, pricing for large equity, mortgage, life changing decisions. Let's be careful as we move into handing over decision making to deep neural network when we're talking about very big consequences on errors. All right, so what can we do about it? This is a topic for research for a whole community in my opinion. And my goal here is just to get more and more people interested in doing research here because we need a whole village of people working on this. I'm gonna start small. I'm gonna mention a few things that we are doing as just part of a big village. And the whole session today showed all the really valuable and very important contribution that people have been doing. To me, for example, if I need to start building trust on someone, let's forget deep neural network, even a person that talks to me, the very first necessary condition, not sufficient, just necessary, but starting point is that, well, this person better know and better be able to say what they don't know if they want to start building any chance of trust from me. So for me, raising awareness is particularly important especially when you are planning about using deep neural network for decision making. And one of the things that we started doing or a whole community of people started doing is start defining what it could mean to have a lack of robustness, define and maybe develop tools that measure it and quantify it. So the goal, the dream here is to have a tool that runs parallel to the deep neural network and it's about the same time, will be able to tell me also something about how confident that answer is. And I'm not just talking about reading the value of the last neuron that says 91% because we decided that's really not the value of confidence. Something else, what if I start perturbing my input, like many people said today in the session, all of a sudden the answer changes. If I see that that happens, maybe that's not trustworthy. How large of a perturbation do I need to make before the answer changes? Well, if I need to make a large perturbation before the answer changes completely, maybe I start, that's just a starting point for building the trust. It doesn't mean that I'm gonna trust. So that's the goal. Hopefully this tool will be able to raise a flag, produce some warning for a small number of cases so that hopefully in that case, the human is still there being called up to look at that specific case. There are not many, fortunately, because our highly performing deep neural network are so good that they're so, they get it right on so many times. Great, so let's look at those mistakes and let's really have something else involved in those mistakes, hopefully. So let me proceed now in this direction of trying to develop ways to measure and quantify this amount of perturbation and I'm gonna give you first a graphical representation, then I'm gonna give you a little bit of math but really I'm trying to stay away from math because the more math I give you, the more you lose track of the key point, which is the questions. So if this is an ostrich label and maybe that's associated with a specific vector which has maybe all the amplitudes of the pixels on this picture, this happens to be a very large dimensional space which I cannot represent on this slide so it's actually a point in a 2D space. Apologize for that. There might be a decision boundary across that decision boundary. I might have another label. Maybe it's a chicken label and there's many decision values. This is actually not to scale. Imagine there's maybe a lot more space and things are closer or farther away. We're also in a thousand, if not million dimensional space so there's millions of these decision boundaries around. It's not so easy as it looks in this picture. So if I start modifying this picture, I might end up somewhere close to the boundary maybe just across with something crazy looking like that. I'm sure that a lot of you that are in GAN learn to appreciate the beauty of these crossed animals or things. This is actually not, I don't know if it was generated by a GAN or whatever. I just searched on Google, ostrich chicken and look at whatever came out that looked strange. I have no problem with this. This happens all the time. Somewhere in between there, you're gonna have with something that even a human cannot tell. And actually I have no problem with this because a lot of times deep neural network also have a problem with that and they tell you they do have a problem in this case, maybe it's firing 55%. That's not my issue. My issue is the existence of other situations. And here I'm just showing the amount of perturbation on an axis so we can keep track of it. A smaller perturbation so that I end up with something looking like exactly the same picture as before. So tiny perturbation that I cannot tell the difference now it gets classified really, really, really confidently as a vacuum cleaner. That's what I have a problem with. On these boundaries, there are situations like that. They're not very common but they can exist and people have shown the existence. They are called attacks, like I mentioned before. So let's put that distance here on this plot. And now let's look for this ball. A lot of people mentioned balls today and here I am showing this other representation of exactly the same thing. How large of a perturbation do you need to get to the closest boundary? And if we can measure that, that's possibly an indication for how much robustness you might have in the perturbation of you're specific. This is specific. It's input specific. There's not a single ball that works for all the inputs. Every single input has its own ball because it might be closer to a nearby boundary. That's very important to keep in mind. So it turns out that this can actually be measured but it turns out it's extremely expensive to do it exactly. People have done it. There's this ReLuplex algorithm that does it for you. On a very small network, 300 neurons. What can you do? Think about what you can do with 300 neurons. Not image net. It takes three hours. So this is far from my goal of running the deep neural network assessment and then running also this tool in parallel and getting a level of confidence. A decision maker might not want to wait these three hours. So if you're start thinking about what you should do as a research, I mean improving the speed for the evaluation of these whatever measures or scores for robustness is a great thing to do. So Lily is my graduate student who started looking at these issues and one of the first thing that she did is try to find some theoretical result that linked this radius of this ball, minimum perturbation and P norm to the closest boundary and she was able to relate it to the Lipschitz constant without going too much into details. I promise no math and here I am showing a math I couldn't resist. But if basically you start thinking very, very intuitively please apologize for the people that are technical in the field but something about steepness of the activation functions and high steepness, very, very steep activation function seems to be a bad thing from what we've seen and I have developed some kind of intuition for why that might be the case. Maybe I don't want to say it yet on a video recording but you can ask me later. So once she has established this theorem the next question is how do I actually compute this value? So the first thing that she did is oh let's just use some sampling for example let's sample around the ball and try to get an idea for how large that is. Let me emphasize and I put it in red that, oh no I didn't put it in red. I'll put it right on the next slide. This is an estimation of that ball. Estimation means that you accept the risk that it might be incorrect. So let me show you how well it works and let me show you also why sometimes it doesn't because if I want to build trust I need to show you what I don't know and when things don't work. So these are different databases with different examples there are different attacks and this is the amount of perturbation that these specific attacks needed to impose in order to get a misclassification that look very much like the same image to a human because these are very small numbers very small perturbations. This score here, the clever score, is Lily's score in terms of using sampling and extreme value theory to try to get an estimate for that number and they're similar so we're happy in terms of that. They are supposed to be smaller because you're trying to estimate the minimum perturbation before you get a misclassification and they are always smaller similar except for at least this case that we were able to find and we looked for it on purpose because I knew it was going to happen and there it is where in this case it's slightly larger so that shows you that it is an estimation. So it gives an indication it's something that can use it to raise a flag warning warning it doesn't mean anything else but just be careful and look closely with other tools to that specific input. So the next thing that she did and a lot of people did in the literature and I'm really happy that this is moving in this direction is what if we give up some of our desire to estimate the exact number and we try to look for a more conservative number maybe even two or three times smaller radius but we guarantee it, it's a lower bound. That's also an interesting direction for research that I encourage people to take and Lily also moved in that direction. So that was the estimation I'm trying to progress some history here on things and she started looking into this robustness certification which is basically trying to find this lower bound guarantee and because this robustness seems to have to do with steepness and slope of these activation functions then she started with very simple activation function that had only two slopes flat and another slope, the relu and in that case started bounding locally each activation function with two linear bounds the same up and down and then propagating them through and making the absent the perturbation larger and larger until you get an overlap. Don't want to go too much into details there are many things that I want to mention but the bottom line I think looking at how well it works is probably more interesting. I'm showing here small examples. These are small examples, two layers, 20 neurons four different norms, different ways of measuring that ball as a reference what did we use? Of course we use red luplex it can be done because these are small examples. Remember it's expensive maybe it takes three hours just for running one of them. It can be done only for infinity norm not for the other P norms. So for those we had to look for other references for example this linear programming approach which doesn't give you the exact value but it gives you a lower bound and a decent one maybe it's far off factor of two from the actual value and her technique that I just described before about producing linear bounds and activation functions and propagating them that's pretty well compared to that pretty much for a lower bound that gives up factor of two to four meaning that two to four is a smaller radius of the perturbation compared to the exact number. She was able to complete the computation time that is 12,000 times faster than red luplex and maybe for a similar number in the bound as LP she was able to do it two orders of magnitude faster. For three layers things get better six million faster than red luplex a thousand times faster than LP. Speed is important right remember the goal we need to provide something that you can run in parallel to your neural network so that for each point you run you also get a really better indication if you can't trust it or not. So in summary well it does good quality of the bound close to LP result much faster. What happens if you start scaling go even higher well we were able to go up to seven layers a thousand neurons for each layer which is larger than that larger in the but definitely not at the scale of image net and this is where a lot of research in my opinion needs to happen because right now we are about maybe 10 to 15 seconds which is the time it takes to certify the specific answer and give you a very good indication a lower bound certified for the amount of perturbation you can afford before you have a potential misclassification. So it depends on what is my goal here what I'm making a decision on if I'm making a decision on boarding priority I couldn't care less I mean really but if I'm making a decision about diagnosis I'm making a decision about something more substantial I'm definitely willing to wait 10 seconds to verify even more than 10 seconds in my opinion. So she chose to move in a different direction instead of improving speed which I think needs to happen she started expanding the technique with other collaborators in the group to other kind of activation functions more generic activation functions instead of using the same linear bound for below and above she started propagating two different bounds that seems to work and then you can handle different kind of activation functions then a really brilliant undergrad student undergrad student I'm so happy undergrad students are contributing to research in this field in a way that I've never seen in my career before I've been in many fields and undergraduates all it takes is to be smart to be in this field this is a good thing for the field so let me introduce you to Aklien here Aklien was able to extend these results producing bound, certified bounds to convolutional neural network and instead of propagating this linear bounds since we have a convolutional neural network he tried to use convolutional type of form for the bounds I want to keep it short so I'm going to move very quickly to final remarks first of all I'm really happy that the community is finally beginning to understand the issue that there is a question and start looking into measuring robustness it's a lot of people that have been producing very interesting measures I'm also very happy that the people outside of our community are beginning to understand the issue because our community has to reach outside it's extremely important because the decision makers they are considering using deep neural network to make their decisions are not experts of machine learning and this is my problem when they don't know anything about all of these things that are going inside and they want to trust them completely that's where I get concerned so I'm really happy that people outside the community are beginning to appreciate the fact that there are situations where things might not be as nice as we would like them to be and there is more research that needs to be done to improve that I'm also very happy that the community is beginning to develop tools because this is absolutely needed in my opinion that can help these users not only get an answer but also get an assessment for how much they can trust that answer and IBM has been doing a great job has been a fantastic collaborator in terms of doing this because they've been developing tools in parallel with us with these papers that have been publishing and deploying these tools for people to use where you can quickly assess and get an idea and get a flag, a warning is this an answer I can trust or do I need to think harder about this specific input All right, so let me summarize for me, very important to raise awareness that there are some potential issues the definition of performance is something that I hope you understood is not a single way of measuring that we need to look at there are many more ways of measuring we started defining robustness for example, looking at the size of perturbation I welcome many other definitions please everyone that is involved or wants to get involved there's a lot of space here very important that you also think about computing it fast whatever you do because otherwise users will not use it and then you defeat the purpose and most importantly, I think I would like there's one thing that deep neural network even the highest performing deep neural network haven't learned to do yet and they really need, in my opinion, to learn to admit what they don't know or what they might not be sure about that's my final message Thank you The National Senators, what is the Senators? Do you have any idea what we have in the recent past fight is a background of politics you showed some of the reasons Do I have an idea for how these there's a whole community of attackers that have pretty good ideas on how they actually do this and it's a variety of reasons some of them, for example, have to do with the fact yes, I said before that I wouldn't say it publicly maybe I said, well, when people do training, for example what do they do? They typically use activation functions with very small slopes because it's easier to train because you always know which way to go but those produce network that don't give very clear answers they give you 55, 57, 60% firing users love to see 91% what they do is once they're done almost done training they increase up those slopes of the activation functions and they make them much steeper so that means that all of a sudden something that was maybe firing at 60% is now firing at 91% so that means that sometimes we're doing it to ourselves now we have a network that answers more confidently but that means we expose it to also making more mistakes on things that shouldn't be answering so confidently there are other mechanisms there are mechanisms where for example the attackers exploit part of the image that we as humans are not sensible to so it could be that you look at the spectrum the frequency components that we're not used to are not capable of looking at very easily you can hide these variations in different places there are situations where deep neural network maybe recognize that one specific animal might have a specific feature and only that animal in all those millions of pictures only that animal has that specific feature if it recognizes that it's going to look for that specific feature only and if it finds it, it says that's that animal now if that feature is for example I don't know, a triangular year and there's only one animal in the world that has that what happens if there's another animal that gets in a fight and someone is biting off a piece of the year and then it ends up being triangular the neural network will see the triangular year and people will say, oh it's an ostrich no it's not, a human would not say that it would say it's a dog with a year that has been bitten off so there's so many failure mechanisms the more I look into it, the more I discover more it's like making a list of all these failure mechanisms is not the way to go in my opinion because there would be always more and more that people can find or people can invent this is why I decided to not even get into that fight I just got into the, let's measure the robustness let's try to provide an assessment yeah I mean those images are very interesting like the nature, actually to get at least classified so however if you keep going outside looking you know what's the, what are the difference between those images and those machine manipulative and machine prepared images you know this is there are, there are, there are two people looking at them yes, there's different ways to plot you can just look at the noise that was applied you can try to look at which neurons are more, are firing more or less, you can do maps of which neurons are responsible and what they're looking and you can see what the neural network is looking at in the picture and do a heat map to understand what made it or where at least the problem might be I am trying to be vague on purpose because I don't have answer myself and I don't think that all of these people that are looking at these answers have already answers these are tools that they're developing to try to get to answers and understand but that itself to me is concerning that we don't know these things if we talk, I think this end of the workshop sends a game for every participants including speakers or vendors and attendees and that's how kind of great we get to do this