 Great. Welcome everyone. So indeed as you've all said today, it will not be a talk about algorithms but actually I want to make a case that this is a talk about theory even though you will not see any theory in what I will talk about. Okay, and sort of, so what is it about? This is about ML security. Essentially it's this new kid on the block. If you are doing security, you realize that now ML is becoming more pervasive and you know, this leads to problems and I want to talk about the problems and maybe you can help me with finding the potential solutions. Okay, so this is called like machine learning security, the good, the bad and the hopeful. So let's start with the good, you know, because it's always nice to start with the good note. If this looks very much like, you know, like EPFL where I was at some point, so you can see that this is a beautiful place. But yeah, what is good? So what is good is that machine learning became really like a success recently. Okay, so for a number of years, we had machine learning but only over the last 10 years or so, it actually started to work. Okay, and of course, you know, when anything starts to work in computer science, you can't escape the hype. So you have probably heard crazy things about things like deep learning from, you know, that this doesn't work to this is like the new electricity. I will not tell you where I am, but it's definitely, you know, there is a lot of interest. And indeed, you know, there were some concrete progress on very important challenges in which, you know, what we could do now, in particular with deep learning, we couldn't do at all just, you know, a couple of years before. And essentially once there was this like proof of concept that there is some success, of course, people started to, you know, take this to the new dimension. And suddenly, machine learning becomes a silver bullet that, you know, whatever you do, if you add machine learning to it, it will only get better. So, you know, now people want to use it for like finance, you know, like hiring people, medical stuff, anything you name it. By the way, I'm not suggesting anything, but that's exactly what crypto also is trying to impact. Okay, so essentially this is the all of our life and machine learning makes strong claims that essentially this is, you know, where they will change these domains. And indeed, they will, except so now, you know, after this good, very short good, let's talk about the bad. Okay, so essentially, now once we have this all rosy picture of how we will like ML will revolutionize the world, we come to the very sad conclusion. Okay, and the conclusion is, and this is like a little known secret. So, you know, don't don't spread it too much. Especially in Silicon Valley, no one seems to know it. But ML is dumb. Okay, in a sense that, you know, it is just a tool, a very simple tool that can be extremely useful. But also, as every tool, you can hurt yourself badly. If you don't use it properly. In particular, despite what different Silicon Valley starters would like you to believe, there is no magic involved. It's actually, you know, wishful thinking does not work here, even if you can fool yourself into thinking it does. And somehow, when you realize that, you might start to like thinking again, okay, is actually deploying machine learning as is a responsible thing to do. Okay. And in particular, just to know my answer is no. And just to make the case, I will focus on one particular question saying, you know, can we actually get machine learning solutions that we can rely on? Okay, and I'm not talking here about the kind of Elon Musk's kindness scenario that essentially we get this, you know, AI solutions that, you know, that attain conscious and will like kill us. Because honestly, if we succeed to create Skynet, I will be very surprised, you know, so maybe then they deserve to kill us. So essentially, what I'm talking about is about something different, saying if you deploy any tool in the real world and you connect it to very important resources, there are bad people out there and I don't have to convince you that they are bad people out there. And they can essentially, they can use some of these like malicious intents and malicious work to influence it and kind of, you know, like exploit it to their own means. So there are all these cases about like, you know, Cambridge Analytica, there was this like famous thing where there was this like false tweet of like explosion in the White House that caused essentially crash on the on the well, the fluctuations on the on Wall Street because there were some like algorithmic solutions that were just like following tweets and trading based on that and someone made a lot of money using it. Okay. And but also like even if you take the bad guys out of the picture, if you just think about like the nature, you know, nature, even though it's usually nice, from time to time, we get some unusual solution. So this is like the question of safety and reliability. And even there, we have a lot of problems. Okay, so let's talk about reliability and somehow just to tell you to give you an idea why there is a problem in, you know, in this regard, let's talk about, you know, one of something that's actually a big success of machine learning. Okay, so one of the biggest successes of machine learning and in particular was this performance on the so-called image net benchmark, which is like the flagship computer vision benchmark. It's like, you know, 1.5 million images in a training set that you train on. And then you test your performance on the test set. Okay. And this is like very realistic high resolution pictures. And here over years, you see what was the best achievable prediction error on this image net data. The question is, if I train on this 1.5 million images, you know, now I'm presented with new images and I have to label them. And I'm just asked to actually produce five predictions of what this is on the picture. And if, you know, if the right prediction is in one of these five, then I have success. Otherwise, I failed. And essentially like, you know, around 2010 when image net was deployed, this was like 25% was the error rate. And it starts to like over time, of course, steadily decrease. In particular, here you see like a big jump. This is exactly when the Alex net, you know, came to life and then made this big progress. And that's where deep learning really, you know, sort of like took off as a viable solution. And over the years, we get better and better and better. And actually in 2015, something interesting happened that actually we got performance on image net that is better than the human performance. And the human performance here is, yes? Alex net, yes. Oh, this is where, where have you been for the last five years? So essentially Alex net is like, it's essentially like the solution like the first, well, not the first, but like the most famous first solution to image net that is using deep neural nets to actually do it. So previously people use different techniques and Alex was the paper that actually showed that you can do much better with deep learning. It's a, you know, it's one of these kind of really like, you know, breakthrough results in the computer vision. After that, the computer vision switched from feature engineering to deep learning essentially like at some point you realize, okay, now if you don't do deep learning in computer vision, you just, you just do not exist. So that was why this is important paper. So all your computer vision colleagues know this paper very well. But sorry, I didn't explain it. Okay. So what is the human performance? So the human perform, sorry, I know, I know, like, I know who I'm dealing with. Don't worry. So essentially, so what was actually human performance, human performance was essentially, you know, essentially like, you know, like someone, I'm just like blanking on the name now. Like he's now the, you know, the city of Tesla. He essentially, he essentially, you know, like trained himself on the image net for two weeks and then took the test. And that's the result that he got. Okay. So that's actually quite impressive education of the science, but that's the result. And, you know, in 2015, we do did better than that. So essentially like suddenly, you know, machines became better than humans at computer vision. Skyint is near. It should be all completely like scared now or should be. Okay. So what does this actually mean like this result? It only means that on this very specific benchmark, this computer, this, you know, this deep learning based solutions achieve better score than humans. Does it really mean that they have better vision than humans? Not quite. And here is the one interesting case is that somehow what is interesting about these predictions is that even though what deep learning productions, like they are good on average, essentially like they are good on a random inputs. They are also very brittle. So here's an example. So what you can do is you can just take an image of a pig that the state of the art classifier, you know, classifies as a pig with like high confidence, 91% confidence. And you add to it a little bit of noise. So it's literally a little bit. So this is the picture after adding the noise. So to you as a human, it doesn't have any difference. But suddenly the classifier is convinced this is an airline. So you know, about magic of machine learning, if you if there is some some technology that can make pigs fly, this is it. This works. So clearly, you know, so you can say, okay, so that's that's not great. Of course, this noise has to be specifically chosen and is very intricate. So you might say, okay, well, you know, this doesn't happen in the real world, you know, like some people like prominent people, like, you know, Jan Lacoon said, Okay, this is not really a problem. Because in the real world, we have noise and things are not as fine grained. In the real world, this will not be a problem. Well, there was a bunch of MIT undergrads that show that this is not true. So what they did, they actually truly printed a turtle. Okay, with this adversarial noise pattern. And what is happening is that this turtle from almost every angle is classified as a rifle with high confidence. By the way, they were considering first doing rifle that classifies a turtle, but then they thought about getting it through the airport and they realized that this is a safer way to go. But essentially, like this is really this is something you can take in your hands and check it and this is this is really this really fools like the state of the art ML classifier. Yes, of course. And by the way, well, by the way, we know also that there are these kinds of examples that are called illusions, right? So that's what exactly they are. Okay, so great. You can say, okay, but this is still about like the pattern of the paint on this turtle is very intricate. You can say, okay, but in the real world, we will not have such problems. No one will paint the turtles, you know, this way. But actually, it turns out this even worse. So sort of like you forget about some like different patterns on the on the turtle. Just think about rotating an image. It turns out that when you start rotating the image, even then the state of the classifiers get fooled. So first of here, you have an image of a rifle that again is like with, you know, with high confidence, like classified as a rifle. And then you start rotating it. And you see how the prediction fluctuates in particular, for quite a lot of rotations, this classifier is convinces us that this is a blow. Okay, so again, like turning the, you know, this sorts into close, you know, everything, everything, you know, everything can be possible in machine learning. Okay, so so this is this is you know, this is what's happening. And now question is, you know, should we be worried? Definitely entertaining. But like, should we be worried? And you know, I would claim that indeed it is. And essentially, like what I really love about security community that they are like creative, and just really doing cool hacks that makes you think. So one of the coolest, like hugs that I did, so there is a group from CMU. So what they did, they took state of the art, like image, like face identification system. And what they did, they truly printed glasses, that if I want to assume someone else identity, I just put this glasses on. And the system is convinced that I'm this different person. Okay, so again, if you if any of your friends in the like, you know, three letter agency wants to use deep learning for like security purposes, you should show them this paper. And then there are even cooler things. So this is a group from like Nicholas Carlini and David Wagner and others from Berkeley. So what they did, they look at the sound. So so far we talk about images. How about sound? So how about embedding some subliminal message for the machine learning systems in the sound? So here's a quick experiment. So I will play you two sounds, like two, two small sound fragments. And I want you to try to listen to them and see if you hear a difference. So this is the first one. Okay, and now is the second one. Can you can you hear the difference? There is there is small difference. But in the end, you know, if I mix them, like if I essentially they're indistinguishable to you, probably is like you can't really differ which one was changed which are not. So indeed, so the first one is just the original recording as is. And if you now like input it to a like, no speech recognition system, they will just say there's no text here. There's only music as you intended. However, the second clip actually to this, you know, space recognition system is actually recognizes as a completely legitimate text. And it's actually it says, you know, speech can be in the music. And then they have all this like think about all these applications where like Alexa or Siri, essentially you can like play and add on the on the radio that tells your you know, Alexa or Siri to buy only from you know, this product from now on or whatever. And you are not even aware of doing that. So essentially, so there's a lot of problems and there are all these cool things that you know, once you know, we think ML is your friend, now you think ML might be your enemy. And also there are also these classic problems like spam detection or you know, or like malware detection where again, actually they dealt with other examples for long time already. Okay. So that's the problem when you have bad guys. But again, this is also about safety. It's not only about the security. So, you know, we keep hearing about all these crashes of like self-driving cars from time to time. And this is part of the reason why we have it, not the only reason about part of the reason. In particular, we have all these amusing photos on YouTube of of Tesla kind of, you know, self-driving system. It's not really self-driving system. They just help you with driving where essentially here the Tesla is trying to actually hit the divider because for some reason the classifier is, you know, convinced that this is where the lane is going. So the driver had to take over and essentially avoid the crash. So these things happened and you know, and there are like, you know, and essentially you should not be surprised because, you know, as I told you, actually even random rotation of the image can completely break the prediction. So, you know, in the real world when you have to process so many frames, you know, it's quite likely that you will get some bad inputs there as well. Okay? So this is safety. Also, like, I actually think there's a third reason why we should worry about this and think about this is a sort of, I view it as a kind of called ML alignment, is somehow we should really make people aware that the way machine learning system work is different than human works. Okay? So that that's the fact that the both human and machine learning system can solve the same problem, does not mean that the machine learning system does it the same way as human does. In particular, the blind spots of one may be different than the blind spot of the other. So to me, this is also like a way of trying to understand the failure modes of ML. So this is kind of like this deja vu cut from Matrix when you realize that something is not as you thought it is. Okay? So that's also one reason to have it. Okay? So this was, you know, so this was like this, you know, brittleness of predictions. You never worry, okay, this is bad. But, you know, is that it? Is that the only thing you should worry about? And the answer is absolutely no. Okay? Actually, you can look at the whole pipeline of machine learning and everything will be broken. So, so far we talked about inference and that's where as an example showed that there is a broken. So let's look at training, which is like the phase before inference. What happens there? So, you know, one thing about deep learning that is very important is actually it's extremely data hungry. It needs a lot of data to perform well. And when it needs a lot of data, you as a someone who trains the systems cannot afford to be too picky about where the data is coming from. Like essentially, you just grab any data you can get, handle all of you, just have no time, you have no, like, resources to actually, like, scientist your data or gather it yourself to make sure that it was not tampered with. And somehow, you know, what can go wrong with that? You know, that's that's a good question. Because, you know, sort of like the reason why people thought this is that there was a kind of conventional wisdom that no, even if the data is of bad quality, in the worst case it would just not help. Okay? So, it's just in the worst case it will be just noise and it will just cancel out and just like you always want to train with any data you can get your hands on, because then it can only help you. And this of course it turns out to be completely not true. So, here is just one, you know, concept, like, conceptual conceptual reason why you should be very worried about this. So, imagine that there is some kind of, you know, I'm trying to train a like, you know, phase detection system and I would like to make sure that this, you know, handsome guy is essentially classified as George Clooney. Okay? So, if I am the adversary who has control over the training set, one way I think I could just do, I could just like, like put, you know, five copies of training set in which, you know, there is a there is an image of the person I want to misclassify and the label George Clooney to it. Clearly, this is actually by definition what the training algorithm will do what to realize, okay, this guy has to be George Clooney. That's, that's what I was told by the data. But, you know, this is true. This shows that you can indeed by manipulating data you can influence things. But, you know, honestly, this is sort of like that's, that's how much learning work. It's essentially like if you tell it wrong things, it will believe wrong things. You don't have a consistent model of, you know, of anything. But this turns out to be actually much worse than this. So, essentially it turns out that, you know, it's not only that I can just like, you know, add some inputs to change a particular prediction. You can actually just add one input to, to change a lot of predictions simultaneously. In fact, it goes even, even worse. What I can do is I can use, you know, I can just use data poisoning to implant a backdoor into your system. So, what I can do is I can just train your system essentially to recognize certain watermarks. And whenever I, like whatever is the input, whenever I add the watermark, I can override the prediction to be the label I as an adversary want, as opposed to what the system normally would predict. Okay? And this is really bad because now I can completely undermine any, you know, any, you know, any system, any ML system that you want to do because I always whenever I need it, I will always make it to do what I want as opposed to what the personal intended to this ML system to do. Okay? And, you know, and so this is, this is data poisoning. And yeah, people again have some entertaining, you know, entertaining, entertaining demos, you know, if you want to be classified as Allison Hennigan, you can treat it printed glasses and again, like once you put them in the glasses, you change your, you know, you change your appearance to the ML system. Okay? We will, great question, we'll get to that in a moment. And so, okay, so this is training inference, you know, how about profiting? Okay, so once we actually trained and we, like, you know, test drive our system, we want to profit from it. And one way to profit from it is just doing ML as a service. So essentially, like imagine I'm a Google, you know, I trained all the school classifiers and now I want to monetize it. So what I do, I just, like, put my classified, well, I just say, okay, if you, like, I set up a website and says, okay, if you give me your input and pay me, I will give you a prediction for my model on this input. Okay? So I will outsource my ML expertise this way. And indeed, you know, this is what Google, Microsoft, and Clarify and, you know, a couple of other companies are doing already. Now, you know, what can go wrong? Well, the thing that can go wrong is called model stealing. So what can happen is that actually I can use this, like, essentially, this query access to your model to train a kind of, you know, like functional copy of your model. And once I train the functional copy of your model, well, first of all, I can get as many queries as I want to. And I didn't have to pay all the engineers and all the research that you did to create this model. Also, if your data was proprietary, you know, I don't need this data anymore. I can just benefit from the findings. It actually even goes even worse because you can also sometimes extract some pieces of the property data from this, you know, from this, from this, from this answers to these queries. So this is really bad because now I can't just, there's no simple way for me to just even monetize my ML or actually let anyone use it without the danger of compromising the system. And this now goes to a question of, the question that Tal asks, actually this goes even worse. So essentially like so far I was very, I was very kind of vague about what is the particular ML system that we are trying to attack and also what kind of access we have to it. So for now I assume this is like there is a specific ML system I want to attack and I have full access to it. I can just read everything there. But it turns out that you actually don't need to. So first of all you can actually think of a black box model when you, the only way you interact with the model is just by asking queries. Okay. So even then you can actually do some kind of, you know, black box attacks and essentially like you need more queries, but essentially you are able to synthesize, synthesize, adversarial examples for the system. But there's also another phenomenon called transferability. That's what is surprising is that the adversarial examples synthesize for one system seem to actually be surprisingly good at being also adversarial for any other systems. Okay. This is a phenomenon that we don't understand well, but it seems to be there for sure. So that's essentially, so it's not really about just having full access to one specific system. It seems to be much more widespread phenomena. So that's a great question. So question is like, if you want to steal the model, question is do you need to essentially ask as many questions as the training set? Well, definitely that's sufficient. If it's necessary it's unclear. Like, you know, there are things like active learning you could think about that, you know, once you can ask about specific input labeling, you would hope that you can do better. In theory, that's true. In practice, it seems to be less true, but yeah. Oh, nice. Yeah. I like this analogy. Yes. That's a good that's a good intuition. Yes. Well, and the question is then how many queries do you need? So, oh, you know, in practice, you would like to steal the model. Current methods, essentially they ask number of queries that is comparable to the size of the training set. If you want to steal the model, but it's not about like the difference between this is that here I want to adjust misclassify a particular input. So that's that's a bit like once I steal the model, no, no, no, no, no, no, no, no, no. That's it. There are different things. That's if just for one input, you need like currently our best best result is like 3000, which is like much smaller, like like a couple of other smaller than the whole training set. This is about if I want to replicate maybe here on all the inputs, then I need to essentially ask the training set kind of sites of the data. If I just care about one input, I just need to approximate the model around, you know, around this this input. And that's yes. Yes. So essentially, that that's a great question. I don't want to go into it. But like the answer is that actually surprisingly, it seems that the responsibility even over different way of subsampling the training set. So like, we try to make everything independent as much as possible, just keep the task the same. Even a different training set, we still see transferability. And I really don't know why. Like, yeah, that's that's a big mystery to me that by the way, I hope you will help us figure figure out. Well, we use different architectures. We still use grand descent, but if you use different like random seat and different architecture, like it's very it's quite different algorithms. Yeah. Okay. Cool. So this is this is a problem. So after all of this, you know, after all of this, you know, all of this problems, you see that there are like three commandments of secure and safe ML. So the first commandment says, you know, you should not train on data you don't fully trust. Okay. Because data poisoning. The other thing says, by the way, something happened to my slides. You should not, you know, let anyone use your model unless you trust them that fully. And actually, you will not even trust yourself. Like, you know, you even will not trust the output of your model when you use it yourself. So these are the commandments of safe. Like if you follow these commandments, ML security is great. So the problem is that also, also there, it is useless. So, you know, if I want to tell you what is the current state of ML security, I think the amped, you know, the amped metaphor is exactly that. And the question is, you know, are we doomed? You know, like, does it mean that ML should never be used? Should be like, we should just like abandon like any hopes of using ML. And of course, you know, on one hand, this dumpster fire is clearly a bad thing. On the other hand, especially when I talk to crypto crowd, I think you see it as what it's actually is. It's a huge opportunity. Maybe it's classified as a beautiful thing. We can make it so, yes. So, so it is a huge opportunity for crypto because essentially, you know, as far as I understand, when I talk to my older collecting crypto, you know, the dumpster fire was pretty much also the accurate description of, you know, classic security in the 60s and 70s. Okay. And only then, like that was the impact of crypto to actually start to bring some order to this field and you know, and get, you know, to the security that we have here. By the way, I hear from some security expert that it's still a dumpster fire, but you know, I hope it's a less of a dumpster fire than it used to be. So, you know, it's a, I believe we made progress and the question is, can we make the progress in ML here as well? So, you know, all that we just need to do is just to repeat the story that we played for crypto in the context of this ML. Sounds simple, right? So, let's get to the hopeful, you know, so we end on the, on the right notes. So let's talk about the hopeful. So what I will talk to you about is I will talk about the steps that, you know, me and other people in the community made to try to bring some order to just dealing with this question of non-robustness of the prediction. Okay, so how to get, you know, how to get this mean classifiers that will not let this poor peak to actually fly. Okay, so it's always, it will always stay a peak, whatever it does, it always will stay a peak. Okay, so how will we go about this? So to explain that, let's actually go back and look a little bit deeper. Why do we even have this other side robust, like other side examples to begin with? So let's just, like very quickly remind ourselves of what is the setup in the supervised machine learning, which we talked about here. So we have some dataset, and in this dataset, there are different, like data points correspond to different categories that you would like to be able to recognize. Okay, so somehow, so in deep learning, you would like to get a classifier which corresponds to some neural net and theta are just the weights on this, you know, on this edges of this, of this neural network. And now, you know, what we get is we just essentially, like we look at some particular input, which is like the input image and the correct label. And, you know, we can check for a given stick of other parameters. What is the prediction of this model on this input? Okay, and the prediction might be, you know, pet, which is the wrong prediction, or the prediction might be a panda, which is the right prediction for a different set of parameters. And now essentially, all that we want to achieve is we want to find a setting of parameters that essentially, what for which it will happen is that like if I sample a random, you know, random input from this distribution, then the loss, which essentially measures how well we do at predicting it, how close we are to the right answer will be small. And that's exactly what essentially all of the ML is trying to do. It's called like, you know, standard generalization, just like getting standard generalization. And, you know, and how do we go about doing this? Well, we use training. Okay, so what do we do is again, we just, the way we train is we sample one of the inputs. We, now what we do is we update our current parameters so that we are, you know, more correct on this particular input that we have. And then we just repeat it, you know, for as, you know, as many times as we feel fit and we hope that in the end, we converge to a setting of parameters that actually, you know, generalizes essentially like it, not only classifies correctly the things that we have sampled, but actually other things as well. And that's great. So, and that's how ML really works now. So, we have this like, goal of training, which is like, begin the minimizing, you know, finding a set of parameters that minimize the loss on a given like, training input, on a given input. That's the paradigm of this. And now the question is like, how do we actually find set of parameters? Well, for deep learning, one convenient thing and one reason for its success is that actually, you know, this prediction of this model on the given input is actually a, you know, it's, it depends on the parameters, but it's also, continuous in this parameter. So, essentially, it can actually, it's differentiable in this parameter. So, in particular, what I can do is, I can just ask for a gradient saying, okay, in which direction should I move to improve my prediction of this model? And this essentially used something called green descent, which essentially tries to, in a greedy way, to just move these parameters in the direction that will improve the prediction the most. And that's how you train deep nets. And again, that's all works great. We have a lot of success. Unfortunately, as we know very well from crypto, you know, things that bring the success can also be the undoing of a system. Because now, you know, and this is again, something that crypto people are very good about it. So, let's turn this kind of this formulation on its head and just, let's keep the parameters fixed. So, assume we actually train a good model. And now, what we start doing, how about just trying to play with the input? So, how about adding some perturbation delta to the input? And now what I would like to do, I would like to maximize over like this, all the perturbations to make the loss as bad as possible, as opposed to as small as possible. Well, the problem is that, you know, once I formulate this problem, then, you know, as convenient it was that, you know, this model was, you know, differentiable in parameters theta. It's also differentiable in the parameters, you know, in the parameter delta. So, I can use green descent to find the bad delta that will lead to misclassification as well. Of course, you can ask yourself, okay, what kind of delta should I allow here? Because clearly, if the delta is just a difference between the image that is like completely different image, then obviously this is nothing, you know, that's not surprising, nothing bad is here. So, you have to essentially make this delta to be small in some meaningful way. So, you know, people use different measures that they say, okay, like for image data it should be like none of the pixels should be changed too much. You know, you might say, okay, I, like, your delta should correspond to rotations or translations of the image. There are some more advanced ways of doing it. But of course, this is by the way a question of a threat model. Okay, so now actually we are getting into the crypto territory. We have to formally define what does it mean for us to be small? Like, what kind of perturbations we don't want our model to be, like essentially, we want our model to be robust too. Okay? But once you have that, then you have this optimization problem, you solve it and you get adversarial examples this way. Okay? It's, yes, well, you can make it. It's just like, I just, like, you could do it as well. It's like, the question, it could be like, that the, what delta means depends, like actually, sort of it is implicitly, like, yeah, the point is that you- Why is it fixed for this one? Yeah, so, so delta in general depends on x. And the question is like, what is the space of delta for given x that you can, you know, like, you can define it for every x differently. As long as it makes sense for whatever, like, it's just a definition of the threat model. Okay, I can, you know, follow up with you after the talk. Okay? So, yeah. So now, when you realize that this is, you know, where the adversarial examples are coming from, you do realize that, you know, it's actually, when you think of the existence of adversarial examples, this is not in any way the failure of machine learning. Okay? Because that's what people believed. It's just the failure of us, like, using the wrong definitions. Because essentially, if we are asking people, we are asking ML to minimize this, like, you know, expected loss, this standardization loss, then this is nothing about being robust adversarial examples, because adversarial examples are of like, measure zero. You know, essentially like, for most of the perturbations of the input, the prediction will be, will be stable as it should be. It just, there are always some specific directions that actually are bad. And that's this definition completely does not seem to, does not capture this at all. So, you know, we should not be, you know, anointed our ML systems because no one asked them to not be like, to be robust to this perturbation. So, once you realize this is the problem, it's obvious what you have to do. You have to change the measure of performance that you are trying to, to achieve. And what you do is you look at something we call adversarial robust generalization when you say, okay, I want, what I want to happen is that if I sample a random, you know, a random example from my distribution, then I look at its worst case perturbation. And for that, I want the loss to be, to be small. And again, you are all nodding because to you it's obvious and to me, sort of, it was also obvious, but this is the part of the crypto thinking that exactly we were, we were thinking. So, the reasons you would expect that it's actually, that's a great question. So, you can ask yourself if it's actually inevitable that essentially it always has to be some bad direction. That's open. But, you know, dance, like, I'm pretty sure the answer is no. Like like, I think that this, at least, okay, so we'll actually get to that because this touches on, like, this is a great question, by the way, that I want you all to ask. Because, you know, this is not known. If maybe there is some, at least for some simple models, you can show that they necessarily have to exist, as examples. And that, that can be very much possible. For deep networks, I don't think they do because the dimension of your prediction, like the, or boundary can be so complicated that you should be able to fit, like a lot. But yeah, actually I'm, maybe we'll get back to this in the questions, but I may actually be late on there with my talk, so we will see. But I definitely, that's a great question to ask. Okay. So yes, this is like now, the nice thing is that it's only definition that you want to, like, you know, optimize, but this is actually a robustness guarantee. This is some measure that you can say, okay, I have my ML system and it scores, you know, that much on this, you know, on this, on this measure. So this is some kind of guarantee of robustness of my model. And that's something very useful. So yeah, so the nice thing though is, that once we know what it is that we are after, so we want to, we are after this adversarial robust generalization, we also know what the training procedure should be now. So what it should be is that, essentially, we look at, you know, at our training example, and then we find the worst case perturbations and only then we try to update our parameters. So we, so we classify this worst case prediction the best we can. And you know, when you, so essentially like, so this is a nice thing that this is like this, now, this subtle point problem in which the inner problem corresponds to essentially attacking, finding an attack on a given input x. And then the outer problem just tries to find the parameters that are, that make these attacks know, like that try to follow these attacks as much as possible. And somehow, all that you have to do now is now that you have to solve this particular optimization problem and you know, and that's, you know, that's about it. That's essentially that's what you want to do. And you know, and this, by the way, this technique is also sort of called adversarial training in the ML literature. But it's really like, you can view it as an empirical, you know, a risk minimization for this adversarial robust generalization. Yeah. Yes. Yeah. So I don't, like again, I just give you a slice of this thing. Like, we don't deal with the, like data poisoning for now. It's a great question, by the way, combining these two things is even more devastating. But like for now, we just like, our test model is that the data is, okay, no one is playing with the data. It's just about, you know, the classifier that we train in the end. Well, this is a great question as well. So the question is like, no, so this is the idea. Does it work? The answer is yes, in practice, in theory, we cannot really prove things about deep nets formally, but in practice, we have pretty good confidence it works, but you have to be careful. Essentially, there are certain principle, design principle that you have to, you have to follow to make sure it works. So the key component first of all is important that you find the procedure that is very good at finding its perturbations, essentially, that they really are able to find almost worst case or actually worst case attack on a given example. That's the important thing. You sort of, you know, this is the usual duality of crypto that, you know, in order to have a defense, I have to, like in order to design defense, I have to understand the attack space well enough first. Okay. If I don't know that, if I don't know what are the attacks, I can be, you know, I have to deal with it. I have to deal with, you know, finding a good defense is difficult. Okay. And the other thing is actually, you need like more model capacity. So this goes back to your question that actually, you know, it seems that the boundaries that we end up having even when you want to be robust are more complex. So we actually need more capacity of our networks. And, you know, also, you know, essentially, but yeah, especially when we do these two things together and we do it right, we see that we indeed are able to solve this kind of this training problem, this robust training problem. And actually, we are getting the results. So here are just like example numbers in terms of like, what is the percentage of numbers that we actually are able to classify, you know, to classify, you know, robustly from the training set. So there are like different, different data sets, M, this C, foreign image net and different threat models, so different ways of like, allowing the perturbation of your input. As you can see, like some of these numbers are not great. And they are definitely like not where we would like them to be. The problem was that before it was zero. So essentially, like we made some progress, but we definitely are not there. And actually there's even bigger thing that I'm wondering about. In some way, it's not even clear to me that this is even the right thing we should measure when we talk about robustness. In particular, it's almost impossible to have 100% robustness because there will be some inputs that are really like, you know, some very fuzzy, you know, some very fuzzy four that like, even to human, like human doesn't know what is the right, you know, what is the right classification there. So there is this kind of very fuzziness that somehow everything in ML is like, you know, almost surely, which is a very different thing when you talk about security, when you really want to know, it's either yes or no. So if we use these measures, we will never be 100% sure. So, you know, if your system still with 1% probability does something wrong, you know, it's hard to say that you have more confidence that that things are good. This is a success rate. Sorry, yes, I switch it. This is a success rate. Right, so where for the time? Yeah, yeah. By the way, it turned out that this number is wrong, but like, meaning it's as in crypto, you know, people claim results and then some other people prove them wrong. So this was one example of that. Is it 4%? No, no, it's 4% is for sure. There was claim of 28%. And I think it's wrong. Okay. So anyway, so essentially what is happening here is that now, you know, once we know what is the measure, like what is the kind of the transition we are after, we have to, like, you know, we can have a new regime of machine learning opening up. Yes, like now we're instead of trying to minimize the standard generalization, like getting centralization, we want to get this robot generalization. And now you can ask yourself, okay, so how does ML look like once I look through this objective as opposed to the original objective that we did look for the last 30 years? Okay. And by the way, this goes way beyond deep learning. Deep learning is just an instance. You know, as an example, there are also problems for classical models as well, exactly for the same reason because we always kind of looked at the wrong definition here. Okay. So, you know, I don't have time to, to tell you, like what are the kind of, you know, this picture is like, we are getting better understanding of this picture. But I don't have time to really tell you what's happening there. I will just give you some highlights. In the robust world, my training becomes more difficult. I actually have to work harder to train my model because I have to find these attacks, like over and over and over. This, you know, this, this is computational cost. This is also the, like, no, there are other costs. In particular, you can prove that in the worst case, you need more data in order, like even if you spend all the time in the world, you need more data to be able to get robust models as opposed to standard models. Like there is a, there is actual information theoretic separation here. You can also show that actually, like this standard accuracy, maximizing standard accuracy, is fundamentally at odds with the robust accuracy. So, like, if you want to have, sometimes, if you want to have a robust model, you actually have to give up on the performance on the, like, average input. And, you know, you can also show that, you know, there are some computational barriers to getting robustness to. So, you know, by the way, you know, this is supposed to be a TCS talk. I didn't really show you any TCS. So, I just want to say that it exists. In this result, what I really like about this field is that this is really genuinely a field where you can, you know, kind of have this very nice interplay between the theory and practice. So, you know, there are actually theorems hidden in these papers, but then they are also, there are some nice ways of experimentally checking the predictions. And, you know, this is, like, actually very satisfying. So, but here, there is, there is fury, but they didn't have time to show it. And, but so, this is the cost, but this tells how to be also some benefits. So, in particular, one thing that people don't like about ML, and I don't like it either, is lack of interpretability. Like, in the end, you get this black box that says, okay, this is a panda. But, like, you would like to know more. You would like to know why do you think it's a panda? You know, like, what, what, what made you think this way? And there are, like, different hacks or, like, techniques that people have for trying to get some interpretability out of ML. So, one of the popular ways to do it is saying, okay, assume this is an input and I, and the model tells me this is a dog. But now I want to understand why does it think it's a dog? So, one thing that people do, they look at this kind of saliency maps. When they look at every pixel and they just ask, if I start perturbing this pixel, how much my prediction that this is a dog will change or not? And then you get something like this, which honestly is not super informative. What is actually interesting is that if you do the same thing to the robust model, then you are getting much better alignment of, like, which parts of the picture are actually the ones that made you think this is a dog. So people have some alternative ways of getting pictures like that for pictures like this, but here you just get it for free. Like, once you have robustness, you just get it. Like, essentially, and if you think about it, you can see why. Because we are sort of, like, making the model invariant to un... like, sort of, like, non-important features. And there is another example that somehow this adversarial examples they become also semantically meaningful. So, you know, previously, so here's an example of a, like, you know, of a orangutan or a gorilla, so this is a gorilla, so which is classified as a primate by, you know, by the classifier and I will just play you a movie when I will start perturbing this, you know, this example to make it be classified as a bird. And you can tell me what changed. This is for a standard model. So, yeah, it's already done. So we don't even see much of a difference, but now the classifier sees it as a bird. But now let's do the exactly the same experiment to the robust model. Okay, so again, I have a primate here and now I'm playing you the movie that tries to change it into a bird. And that's what you get. So actually, like, you could genuinely believe this is a bird, right? Like, essentially, this is, you know, this is the example that you found. This was just like, you know, now you had to change the semantic meaning of the picture to actually make the model, actually make the model, you know, think this is a bird. So, you know, so essentially, so these are like all the, like just like some reports from the trenches of like where we are trying to understand, you know, how robustness lens changes the landscape of this. But again, there's also data poisoning. There is like, you know, model stealing. There's like plenty of things. There's privacy that he didn't even touch on. There's interpretability. All of this, you know, is super important. It's completely open currently. And, you know, so question is, you know, so we are at the crypto workshop. So, you know, I just wanted to be a bit more prescriptive than that, just telling you that there is like a field that needs your help. And, you know, that's it. So, let me just give you some ideas of some concrete, more concrete directions of I think what could be great if you could like, you know, have you contribute to. So, like, one question is, you know, is this protection against the model stealing? Okay. So, actually, if you think about it for a moment, as a cryptographer, you realize that there is this thing like learning with errors. Actually, it's like, for instance, like if I think about trying to learn a linear classifier that's still a linear classifier, it's really like learning a health space problem. And we know that if you add, you know, adversarial noise to, you know, to the task of learning like health space, you actually can make the task of extracting this health space very hard. So, the question is, can we actually like push it, you know, further into the realm of ML? Like this kind of idea. The other thing is, actually, like this is what, you know, again, all the cryptographers instantly touch on this. It's like, what is your model of access? Like what is your mode of access to the model? Is it like white box? Is it like black box? How would you differentiate between them? Can you separate them? So, this is exactly where all like, things like obfuscations come in and essentially, this is what, you know, you obsess about all over and over and over and this will be something that we don't know anything really there for now, like some kind of query separation complexity, anything. We don't know anything here. The other thing is that, you know, it corresponds to essentially like training, you know, model of robust data poisoning. And I think there are some very nice connections to differential privacy and also to pseudo-randomness, essentially like, you know, how to understand exactly like, you know, how to essentially, you know, filter out like, you know, things that overfit too much to the data that we train on. Because that's essentially where the data poisoning is coming from. But also like, more broadly, you know, in the end, ML is about average case, you know, in security, we care about worst case, you know, so essentially like, any kind of reductions of worst case to average case would be great, you know, because again, once you are in the average case question, ML is very good at solving this kind of questions. And you know, the other thing is that, can you actually, okay, this is like, the bread and butter of crypto, you know, to get security, you usually need a hard problem to base your security on. We currently don't have any notion of a hard problem in, you know, in machine learning. Essentially like, everything, you know, as the boss said, you know, like we are seem to be solving NP hard problems, you know, for breakfast. So it's not clear even like, what would be the analog of factoring in deep learning? It's like, what is the hard problem that we think is at the core that you can embed there to make things like, suppress and task hard. And in particular, like the other thing that we are not taking advantages is sort of like, try to unlock the power of hidden randomness, like all, all like, the secret keys. It seems that currently, there is like, even if I keep my seed hidden, it doesn't seem to really impact things too much. And the question is, is it fundamentally true or is it just, we are just not doing it right? And you know, even more broadly, you know, the question is, you know, what are the interesting and relevant threat models? Essentially like, let's think about all the things that can go wrong and try to put them into concrete threat models that we should be addressing. You know, there's like, you know, leakage resistance and so on and so like, you know, this is you are the experts here. Essentially like, everything that happens for, for a classical crypto, it probably makes sense for ML2. Now, what, what does it mean there exactly? What, how could we design some schemes to protect against it? And finally, like, the things that, and this has been clear to me that you even have the right definitions here. Like essentially like, I think the one great, you know, great impact of crypto is creating good definitions. And I think we are severely lacking of these in ML. So that's another help that I would like you to help me. So let me just conclude. So, you know, don't get me wrong. The fact ML is really an, you know, awesome field and it's really making a lot of progress on very, very important questions. But it's still Wild West, you know, essentially in particular, you know, the difference between the gold and fulls gold seems to be very fuzzy and people seem to be very happy with fulls gold, you know, as long as it can push it to the media. So, so, so that's about, but again, remember that Wild West was also the land of opportunity. Essentially like, you know, it, it was the land when you could really like, you know, kind of create new things. And I think this is definitely true about ML currently. And I think the next frontier is exactly like, building this ML, you can truly rely on. Like, now that we know that it sort of works most of the time, now you can move to the next phase of trying to get it actually work reliably. So we should really like, rethink the whole pipeline of ML from like, you know, safety, robustness, privacy, like all the perspectives that we know for classic computing. We should also, you know, again, think about, are we even asking the right questions? You know, like, are we, are our goals like realizable? Like, are this is the right goes over, like all these questions you should think about. And you know, in particular one of these kind of reminders here is like, the specter and meltdown, like, sort of debacle. Like, yeah, of course we can just like, for now, we can just be like, optimist. Just like, okay, let's, you know, let ML work the way it does, let it be deployed and we'll worry about the problems later. But the problem is that, like by the time we get to the problems, it might be too late to undo some of the fundamental changes. So it's better to start the change now before it kind of gets, you know, gets implemented in the real world. And I think that the theory and crypto mindset is extremely valuable here, essentially like, I think we are, you know, one thing that theory is great about is not proving theorems. Well, we are good at that too. But what we are really good about is taking complex processes and boiling them down to like, nice abstractions. And I think this is like, desperately needed now in ML. So, you know, so you should think maybe about this, that this is like, not a kind of, it's like theory as in theoretical physics and not maybe necessarily mathematics yet because we don't, I don't think we have, good enough understanding to like, prove end to end theorems yet. But we definitely, like there's a lot of, things for like modeling, good definitions and so on and so on. And this is what you are great about. And yeah, if you want to learn more, Shafi will give a talk tomorrow that I think will touch on some of these questions, you know, a bit more. We also like, on my lab website, we have a blog which we kind of try to like, you know, post like more general kind of, you know, audience exposure of like, all these things and we will be keeping it more. And yeah, in general, you know, also you can just talk to me after the, you know, after this talk. So, thanks.