 Next up is Guy and Ezra with Jarvis never saw it coming, hacking machine learning in speech, text and face recognition and everywhere else. Thank you very much. First of all, thank you for having us and we'll do our best to share the information that we have with you. It's a bit awkward because I have a clicker in one hand and a mic in the other so I can't really gesture so I'll do my best. First and foremost, everybody is required to read and sign off of this legal disclaimer. Everybody did? Excellent. Okay. Second disclaimer even more important, we really haven't harmed any system here, sort of, but we'll cover that in a couple of slides. To introduce us, I'm Guy and this is my good friend Ezra. We're both co-founders and participants in the besides Tel Aviv and various community activities as we're living the local DevCon chapter in Israel and we are working together for almost a year now on a couple of very exciting projects. Some of those results we want to share with you today. So the first thing that I want to mention is that nothing that anyone is doing is doing by himself. So throughout the slides I'll switch hands. Throughout the slides, you will be able to see short links at the bottom at the bottom right. This is the reference if you want to dig deeper or to find more information and to find credits to whomever was behind this. But the basic point is nobody is working alone. We're all working on top of others people work and credit should be given where credit is due. So how did we get here? Last year we had a couple of pretty good conversation around DevCon with a couple of people and the idea came about, well everybody is talking about the latest buzzword and that is AI and how can we actually do something with AI systems? Are AI systems even secure? And through that conversation we came to the realization that nobody really thought about what security of AI systems really means in that context. Everybody is doing lots of AI work. Everybody is running forward pretty fast. But securing those systems against attacks, what kind of attacks are even possible is something that is kind of unexplored field. So we are going to discuss those kinds of attacks, a bit maybe about what kind of mitigations, how you go methodologically about constructing such attacks and what types or what the landscape looks like and what we feel is going to be the most important ones in the future to come. Whenever you see something highlighted in yellow just like here that means that this is something you should be paying extra close attention to. We are not going to release zero days today. We are not going to do very surprising disclosures. It might be very surprising to you but it's not like nothing that anybody has ever conceived before. We are going to do a couple of pretty nifty things in our demos like breaking machine learning systems live, well sort of lives in a live video that was recorded last week. And I want to start with a story about the horse. So a show of hands, whoever heard about the horse named Hans, about 10%. So I'll go ahead with the story. So clever Hans was a very clever horse. 1903, just the turn of the last century. He went on a tour in Austria around Austria and Germany. And the reason that he went on this tour because he was very clever. He could count up to five, he could do simpler mathematics, two plus three, one plus four. He could spell in German which is amazing because he could spell in German and he is a horse. So that's absolutely mind boggling how could someone do that. And it was very uncommon for horses to spell and to do solve arithmetic problem at the time. It's pretty common today. But a psychologist was dispatched to try to understand how can that horse actually do this. And that psychologist and the work that he's done, we now know as the double blind test. And what he found out is that the horse is pretty clever. It is a very clever horse. But he couldn't spell and he couldn't really do arithmetic but he could read the body language of his handler to make sure that he knew when he got the right answer. So for example, if his handler would ask how much is one plus four, he would tap his hoof one, two, three, four, five and he would read the cues from the body language of his handler knowing that he got the right answer and he would stop. So as I said, he was a pretty clever horse. The reason that I'm telling you all this is because on from many respects, machine learning and AI in general that we are having in the world today is kind of on the same place. Machine learning is very good at solving specific problems. Sorry. Machine learning is very good at solving specific problems. Whenever you try to give them larger context problems, it breaks and it breaks horribly. And we will discover a couple of those paths today. So what do you need to know in order to understand what we are talking on in this talk? First of all, some basic common language. When I'm saying machine learning, I mean a system that on the one hand we are inputting lots and lots of information but we are also including metadata on top of that information. In that context, it means if I have a system built to differentiate between cats and dogs, then I will feed it photos of cats and I will also give it that these specific photos have labels of cats. These dogs are labeled as dogs. And then the system will be able to learn from those inputs and labels and in the future when we will see a new input, it will know, oh, this is very similar to what I've seen previously and therefore this is a dog or a cat or a banana or whatever. Deep learning, which is another buzz watch which you might have heard, is kind of the same thing but now we have no labels. So we're just force feeding the system, orders of magnitude, more inputs and let the system decide by itself how to classify them. So we give it 100,000, 1 million, 10 million input images and it will decide by itself, oh, this is a flamingo, this is a hedgehog, this is a girl, building those classification by itself. And when we are saying artificial intelligence, what we really mean is Arnold Schwarzenegger, the terminator, a machine that can think, that can reason, that has context. We are light years away from this right now in our current state of technology. We don't have a system that can really look at this picture and say, oh, the same system that will say this is a picture of a girl, she's holding a flamingo, there's a hedgehog at the bottom, it's from a book by Lewis Carroll, she's about to play cricket and in general she's very confused. We don't have anything similar to this and therefore we don't have anything close to artificial intelligence. However, everybody mixes artificial intelligence and machine learning, I do as well and you will hear me throughout the talk saying ML and AI repeatedly in the same context, I always mean machine learning. There is no AI. If anybody is selling you AI, we should have a very different conversation. So most AI system were designed to solve very specific problems. So they were very good at solving that problem. They're not good at the complementary part of the problem for whatever else. So this is something that we took a look at and the reason that this comes to be is because the way the system is built. So I want to give like a very high level understanding of how such an AI system looks from a kind of a mathematical perspective. So this is not going to be complex, I'm not going to scare you too much, but I do want you to have some good understanding, good information about what a machine learning model really is. So generally speaking, we have a network, a graph, builds of a couple of nodes. Those nodes may be interconnected, may be more or may be less. The amount of connectivity between the nodes is determined by the weight assigned to each arc. The node itself holds a certain value and usually when we are discussing machine learning models, we are talking about an input layer, an output layer and the hidden layers which are between them. So this is a very simple approach and when we're discussing the hidden layers, we are actually meaning what the system encodes or what kind of information it has, what it understood about the world presented through its inputs when it was trained. What really determines the way that this system behaves is not just the values that the matrix, the specific representation of the model holds at this specific time, but also the weights or the amount that each node is contributing to the outputs. So those weights are actually determining how much importance each node is carrying when a computation is being run through all the way to the output. In the end, the output aggregates all of that information and we get a representation from what the machine learning system understood. For example, I will introduce an input image of a cat and it will tell me this is a cat with a 76% confidence in that prediction. The reason it only has 76% because there are other pathways to the network that led to other kinds of results. Maybe you thought it was a cat with 76, but maybe it was a dog with 15%. So there are very different ways to traverse that graph and each way in the end computes into a specific value and we can look at those values and they give us more information about what the system does. In reality, the systems are very complex. So I showed a very simplified model here. The reality is much more complex than that. When we are talking about modern machine learning systems, we are talking about neural networks that are constructed about hundreds of different layers, millions of parameters, sometimes it's when we are talking about deep learning, we are talking about networks of networks, lots of different complications, lots of different ideas of how to make this more robust or more interesting, which I will not go into today. You can catch me later and I can fill your heads with lots of nonsense. So what do we need to know? First of all, when we are saying model, we mean a couple of different things. One is what is the topology? How does the graph look like? What is connected to what? How many layers are there? The second thing is the weights. What is the relative importance of each node inside that graph? And lastly is the function, the transfer function. When we are transitioning from one layer to another layer inside the graph, we have a transfer function, usually a nonlinear transfer function, which also introduces nonlinearity to the computation. And that adds more complexity to the system and enables the system to learn or encode more information into that matrix that I described earlier. The bottom line is that the matrix, the representation of the model holds the intellectual property. So if I am designing a system that is trying to detect tumors in X-rays, the IP, the important stuff is holding that matrix in that model. Everything else is a framework to help to make that computation. The real data, the real important data is in that model. And we will go later about that part. So how do you actually manipulate data? How do you actually get to that data? A bit of background about linear algebra. Everybody remembers linear algebra? Show of hands? Wow, great. Because I had no intention to go into it. Okay. So what do you need to remember? When multiplying two matrices, you just get a matrix. The values are the products of the rows and columns and that means there are various ways to accelerate that. In the end, from my perspective, a vector is a single dimension matrix, an array is a two-dimensional matrix. Or in the way that I look at it, this is a vector and this is an array. Bottom line, both are memory buffers of a certain size with a certain encoding and representation. I don't care about the underlying math when I'm trying to go after that model. I just want to know how it's encoded in memory. So, so far, we have always looked at machine learning and AI as kind of like a big voodoo machine. So we are introducing various kinds of inputs, might be images, audios, binaries, tags, whatever. And you will get some sort of predictions. For example, I will say, Alexa, add something to my shopping cart. And that audio sample would be uploaded to the cloud. Amazon will do their own thing with it. And in the end, I'll have a classification, meaning this is the most probable sentence, the most probable utterance that matches that voice input. Okay? But I want us to go a bit further here and to understand how that mechanism really works. So we have our inputs and we really can take a large image file, a bitmap or a voice sample, an MP3 or whatever it is and input it and multiply it by a matrix. It doesn't make sense. So what we need to do is to encode it with an intermediate representation. So the way that we do it is that each kind of input and various data scientists have different flavors of how to do that are encoding information with an IR and then taking that IR into the matrix multiplication part. After we have the matrix multiplication, all of the different algorithm functions, weights, et cetera, the computation is over. We are looking at the output and then we are matching that output into something that is human readable. So the calculation from the matrix might have been 17 with a confidence of 6 or 97 or whatever. And now we'll take that mapping, 6, it maps to the label of a dog or a cat or whatever. So the matrix doesn't know what the cat or dog is, but it knows that 6 is one sort of representation and 7 is a different sort of representation. And then we can output from the system the prediction, meaning a specific classification usually attached to a string description, but also the confidence or the amount of confidence that the system has in that specific prediction. Sometimes you'll get 10 of those, maybe one of those, maybe five of those depending on how the system is configured. When we're training the system, what we're really doing is inputting more and more samples into the training data set. And we are recomputing again and again and again and again the values for that matrix. And that means that the matrix values start with a specific might be all zeros, might be a random which is more probable. And then we will modify those inputs again and again in order for them to better match the outputs that we are predicting for the system. So when I'm training that system, I do this, I don't know, maybe 30 million times until I'm able to get a good classification that matches what I know about those samples. So my training data set is very important, but also the way that I train the system is very important. It's our, those are both intertwined capabilities. But when we go out into the real world, we don't have all of that. We just have a deployment. And that deployment usually a framework that just takes inputs, encodes them into an IR, goes through the algorithm, mapping, and then outputs the whatever information it was on the other end. Kind of like a very deterministic system. And there's a good reason for that because we are not training the system anymore. And we want to use it at scale. The point is that when we are looking at the models and when we are looking at code, they're not the same thing. And the reason they're not the same thing is that when we are looking at the code or the binary execution code flow, we know that, okay, we'll do this, this, then there will be a jump, then we can compare, maybe jump node zero or whatever. We can read code. We can understand what it means. We can discern the logic from the machine representation. But when we are looking at the model, which is a matrix with set of values, I can discern anything about the system. It's a very complex problem. Given a matrix, can you go the other way and discern what kind of inputs build that matrix? It's, it's a much harder problem than breaking a, a Shaw-Wan collisions. The other thing is that when we are doing code, we are usually looking at data structures. So we are very familiar with data structure and how they are represented in machine language. The matrix that we are working with them in the model holds the same kind of data structure, but not in a way that we are able to identify that. So whenever it encodes more information into that matrix, it loses the representation of the original data structures and we only get the latest current version of that matrix and the way it was updated with those values. It's very hard to look at the model and understand what built it, what kind of information caused that specific matrix to come to be. And the reason for that is that it encodes so much information and it forgets all the rest. So it's a very nice representation of a one-way function, sort of a one-way function. Just to give you a notion of what it looks like in real life, this is a model, a very famous model called ResNet50. So you can see like the label at the top, but the rest is just binary information. And when we are looking at it, this is binary information, we can hold it, we can manipulate it, we can access it. But what can you do about the code? Because in the end, the matrix doesn't live by itself. There's a framework that needs to do those manipulations, need to run through the IR, need to do the mapping to the outputs, et cetera. So the model is not living in an isolation. It has a lot of interdependencies. And those dependencies are code, they're just regular software. And we are kind of good at doing code reviews, maybe not so much. We are very good, we are very bad at doing that for models. And when we're looking at those models and codes and interaction between the framework and the model, it's very difficult to understand where your code starts, where the framework starts, where your model begins. It's a lot of mishmash between different dependencies and different factors weighing into the final decision or prediction of the network. And we really can't understand from the matrix what this means. So I want to give an example of how this comes to be and why this is so important. So a small background story, a small example. This is from their mythology study. They did a study about skin tumors and they invested a lot of money. Yeah, they invested a lot of money into trying to find those images and train those images to have a machine learning system that be able to train on these and to identify those tumors. And in the end they built a system that was very good at detecting rulers. So this is like the real life use case of what happens to a data scientist. You have a lot of data but having data doesn't really mean that you have good information. So there's a very strong distinction between big data and big information. Nobody has lots of good information. And the reason that I want to mention that is because a lot of attacks are based on the same kind of principles. We are going to discuss five specific attacks today. There are many, many more and we have other presentations doing other stuff. But the reason we wanted to focus on these kinds of attacks is because we believe that those are the most important attacks. We use CVSS 3.0 to do these scoring. And when we talk to customers and partners and clients, etc., we discover that nobody cares about this prioritization. This is actually what they care about. So what people care about out there in the real world is not about somebody de-dossing their machine learning system. They care about somebody stealing their IP. They care about somebody modifying their IP and they have no way to know that it happened. So what we're going to discuss next is how such attacks are built. And then, and here I will turn over to Eswa. Thank you, guys. So first of all, I'm sorry of my voice. Pegas is taking a toll on me. Cool. So how do we build an attack? We first need to know some stuff. We are going to see it in a few slides. What are the areas that we should target whenever we are building an attack? And what are the areas that we have access to? So let's first start to understand what is our attack surface and what is our attack objectives. And we could go either against the system infrastructure that is running the models or going against the math and the algorithms. So just to recap a little bit, we have in the infrastructure, the input is parsed and became kind of an IR. So over here, it's fully infrastructure. And afterwards, when it's the mapping between the output and the label, it's also in the system label. Whenever we're talking about the metrics and duplication and the output itself, we're talking about the algorithms. And we're going to talk a little bit more about it. So let me start by the first and most important part. As we all remember, we get an input that could be a picture or some words or some video or whatever, but it needs to be converted into a matrix, into an IR. So it's the first and most important part because if we cannot do that, we cannot continue. The second thing, parsing is hard. I mean, if you had ever do vulnerability research and take a look at parsers, you will find something. Most of the bugs in parsing exist because it's not simple. You are not the one that developed the representations. So you need to understand what was behind and apply to your system. Oh, and most important, if you are an AI developer, you are not a file format developer. You do not develop parsers. It's not your field of expertise. Because if it was your expertise, it was what you would be doing for a life. So the most common thing that happens is you bring a dependency into your project. It's very traditional just to say, cool, now I need to do parsing of bitmaps. Let's bring the PMP. I now need to do parsing of whatever. I'm just going to bring this library. And it happens. And I respect that. I mean, if I were asked to parse an image, I wouldn't know where. Nowadays, I kind of know how to start. But probably I would have bring an external dependency. So again, I'm bringing outside libraries into the machine earnings over stack. And this is very important to understand. Because whenever we're bringing this library, we're bringing a very common problem in the industry, which is the supply chain management, the patch management. How are we going to keep track of all the patches to all the file formats that we need to support? Or how are we going to verify that a patch in a certain library that is doing certain file format parsing doesn't break my representation afterwards? And it's a very hard problem. And not only that, a common framework must have support for multiple file formats. I mean, if I were to develop a framework for machine learning that only supports a very specific file format, it wouldn't be compatible with anything else. So we need to have support to a lot of things. So if we now know that parsing is hard and that file formats should be accepted by different machine learning systems, we can do something very classical and very traditional from the exploitation world, which is fuzzing those libraries. So when we started with this idea of fuzzing these libraries against distinct file formats, the first things that we identified was that the framework that we were fuzzing, which is called the CAFE, which had full coverage of all the functions that we were interested in taking a look at, was extremely slow. Why it was slow? Because every time we were trying to run one of those arbitrary malicious file formats presentations to be able to trigger a crash, it would run the entire end-to-end process. So then we said, cool, who is actually doing the image parsing for CAFE? And then we say, ah, open computer vision. So let's take a look at open computer vision project, which it's a lot more limited coverage because now we don't go through the, now we're going through specific this project and there might be codepads that CAFE doesn't use. But the speed was good enough. And the third one would, could have been to go directly against the library that was being used for the specific file format extremely fast, but we don't know where are the codepads. And it was a little bit problematic. So at the end, we stayed with OpenCB. Oh, and of course one of my favorite ones, just go upstream. I mean, many of those libraries that machine learning projects are using are not even maintained anymore. So if you go upstream and just take a look at the GITA reports or the BAC reports, you are going to find some very juicy stuff. Ah, the issue is that you don't know if it's actually patch or somebody find it already or the code path is relevant to what we are taking a look at. So, at this moment we found certain crashes. And when we had certain crashes, we went into the exploit development phase. And when we were in the exploit development phase, we go to remote code execution. And the question was, could we use this remote code execution that we found before to be able to approach one of those vulnerabilities that we understood that are the risks in machine learning world? So let's try to take a look and let's try to do some demos. We're going to start with the denial of service. In this scenario, we are going to abuse a memory leak where the input, it's a couple of cases. And you are going to see now what is out. Just to clarify the system that we are using here is using an input image to the API. Then a machine learning model is doing the computation on that and it hands out a prediction. So we are using that API to upload our own malicious image into the system. So, to be able to demonstrate this, we had to build this API. And we are going to see two screens and I'm going to explain a little bit what is screen. So, over here we are opening top to see what is the performance of the system. And it starts like this. Everything is normal. Afterwards, I just go to the library where everything is running and run a bitmap that is going to do something, something. And now, this is part of the video. As you can see, CPU goes to full. The memory starts filling out and it's going to continue filling out for a very, very long time. So what I'm going to do is just make a first forward in the video. And remember, we are still talking about 10K's bitmap file that we are taking a look at. So, it's still running. If you see now we are using something around six gigabytes of memory for a 10K image. And this is bad. I mean, imagine this kind of memory leak. And in a few seconds, we are going to see the crash. Now, you see six 88 gigabytes we were using. This is bad. And you know the business impact is falling services, downtime costs. You don't really know what is happening in the system. You just, everything is working as usual. But it costs a lot more because maybe you are running it in the cloud or whatever. Let's be honest. We all came here to see the remote code execution. So, similar scenario in cell for memory leak. We are now going to exploit a memory corruption bug in the heap. And let's talk a little bit about it. Again, left side, WM. Right, we go to the library where everything starts. We run the classification against malicious input. And something is happening now. So, at this moment we are now going to get back to the video. We connect to the machine where it's running it because we've been to the shell in port 111. At this moment I made a mistake and instead of writing a host name, I wrote host. But we have the same host. We do an LS and this is very important. We are seeing all the files in the system and we are going to return a little bit about it. So, at this moment, with an malicious input, we have full remote code execution. And yeah, but it's still not really relevant because we were telling that what it's, what does an RC helps me in a machine learning world. So, in this scenario we are going to do something similar to what we did before. Again, we connect, we get into the system, we go to the library where we have the files. We wait a little bit because it's the video, it's a little bit slow. And now when we run the system, we are going to take a look at the files that are here. So, please take a look in a few seconds at the classification file. And as you see here, have all the labels. And these labels are related to the output that the system gave me before. Now, when we run the exploit in a few seconds, we have a segmentation fault. And now we are going to open exactly the same file. And the model will always return the world hacked. I mean, it cannot do classification anymore. And this is bad because I could modify any label that I want. And the model at this moment would do whatever I want. And the last one that I want to talk about is the IP theft. I'm not going to show a demo, I don't need to do it. But if you remember, when I did a less, I was able to see all the files that exist in the file system. So the same way I could see them, I could just copy them back to me. And I have the model and I have all the IP of the system, which is bad. So yeah, maybe the hard scene is really the king. However, we don't always have an RC. And when we don't have an RC, we do something very smart. We go to guy. Okay. So assuming that we have an RC in the system, we've seen what we can do. But let's talk a bit about what we can do when we don't have an RC in the system. So the first kind of attack that I want to share with you is something called the cloning attack. A cloning attack means that I'm using a service, let's call it the machine learning service in the cloud as an Oracle. I can ask it question to the API. I can get responses and results back. And I can use those inputs and outputs to train my own system, which you can see here in the bottom. And that means that I can accelerate my own development of my model using someone else's IP sitting somewhere in the cloud, effectively cloning it, creating a functional clone of that system. I'm not doing anything illegal. I'm not hacking anybody's system. I'm using the APIs as they were intended to be used. But I'm stealing the IP away from the system by asking questions what would you do if I gave you this input? What would you do if I gave you that input? And I'm learning from those results and I'm taking away whatever they've spent so much time to study. There are three different approaches to how you would go about cloning. The first one is very easy. You have full access to everything. You know the data set. You know the model. You know the topology. You know the transfer functions. It might seem unreasonable but more often than not AI companies release papers about what they're doing. They are built on top of open source projects. They reference their dependencies. They did not start from scratch. You can get a pretty good understanding of exactly what they've done without knowing the actual weights but you know exactly what they've built and you can just query the system and build your own weighted system with those results. A gray box stack is a bit harder and that means we don't have access to their specific training data set and we may not even know exactly what kind of topology they've used. However, we do have domain knowledge of the system. For example, if somebody is designing a traffic recognition machine learning system and is using the American traffic sign database, the governmental national traffic sign database. Well, okay, I'll go and download Brazil's traffic signs database. They're pretty similar and I can still do the same attack even if I don't have the exact same kind of data. So just having similar data is enough in order to clone this attack and we have done this very successfully. What we are working on right now back home is a black box attack, meaning we have no idea what's going on. We have an API, it's doing something, it might tell us what it does, it might not, we don't know what kind of training it's used, we don't know the data sets, we even don't know what kind of label the system using. We might be privy to a label one and label two but there might be 15 others that we are not aware of. And what we're doing right now is how do you attack such a system where you don't have full knowledge and so far we've been seeing pretty good results, maybe next year. But what if the attacker has access to the data set itself? That's a pretty interesting attack. So if you have access to the data set itself, you can introduce backdoors and backdoors in machine learning are very, very interesting. What do I mean by a backdoor? Remember the example I told you about the dogs and cats classification system? Do the same thing but now introduce bananas with a label of a cat. The machine learning system would train on it and when I validate the system, I will give it input images of cats, it works, dogs, it works. I have no way to know that an image of a banana will be classified as a dog. To put it in a different way, let's assume that I have a network analysis machine learning program and it's looking at all the network traffic of your enterprise. And now I've trained it with a secret backdoor that whenever it sees a special packet header, a special magic number in the packet header, it will assume that anything that follows that magic packet header is completely benign, non-malicious. Don't look at this, I'm not really here. I can do that if I have access to the training dataset and when I, as a user, somebody who's buying a piece of an appliance or a piece of software from vendor, I look at that machine learning model, at that matrix, I have no way to know that there's a backdoor in the system. And that is very different from the current software product because if I have a software product, I can invest in reverse engineering, I can do code reviews, I can do certifications, whatever. I can get some level of assurance that there's no backdoors there. In machine learning system, it's just a matrix. I don't have a way to go back and check it. There are also an entire class of attacks called adversarial examples, which some of you might have heard, especially in the realm of vision systems. This is like 95% of all research into adversarial system is around machine learning in vision systems. But I want to share some information about other kinds of systems that you might not be aware of. The basic problem of adversarial examples is that the problem space is orders of magnitude larger than the solution space. In other words, every input maps into some output. So a dog maps into a label of a dog for the system. But there are many, many other inputs that also map into a label of a dog. This is a simple collision attack with a very large problem space. And the reason that I keep saying it's a very large problem space because finding collision is super easy. Okay? It's not like breaking Shanwan. It's actually very easy to find other inputs that will give you the same kind of results. And another thing that I should note is machine learning systems are optimized and trained to find the local minima in math speak. In non-math speak, they're finding the strongest signal in the input. So if the strongest signal in the input is what it understands that characterize the dog, it will say a dog. But if I can influence the signal in the system to actually encode something else, it will focus on that. Even though there is an image of the dog in the same picture. I just need to find what's the strongest signal. So in vision system there was a debate for a very long time, does it even apply in a real world context? And what do I mean by that? There's been a lot of studies of take a sticker, put it on something, now it thinks it's something else. But you know in the real world objects are three dimensional and there's lighting problems and zoom and very other difficulties. And people said, well, you can't really do it on 3D objects. It can't be, it cannot be done. Well, surprise, surprise. This is not an actual turtle. This is a 3D printed turtle. But if you look closely at the top shell you see like red spots. These red spots are the strongest signal in that object. And they cause the system which is Google's inception v3 here to classify it as a rifle as you can see at the left most bar. So whenever it's turned up to the camera, the camera, the machinery system classifies it as a rifle. Because the strongest signal is actually of it being a rifle. Even though we clearly see that this is a turtle. And I want to show you another example and this is from an audio perspective. So I want you to listen closely once I find my mouse. Without the data set, the article is useless. Let me pump up the volume. Without the data set, the article is useless. Okay, you could all hear him saying without the data set the article is useless. Now I'm playing the same thing with the attack. Listen closely. Without the data set the article is useless. Show of hands who heard the attack. Wow, two people with super hearing. Without the data set the article is useless. So the thing that most of you could not hear here is that there is a high frequency band here that encodes a different kind of data. And that data is in the original audio sample you said without the data set the article is useless. But what the machine learning system is will hear is okay Google browse to evil.com. Okay so when you're thinking about your echos or Amazon Alexa's or series or whatever this is a completely viable attack. So if anybody remembers the famous South Park episode making a really laugh out of Alexa and all of those different systems Cortana, Syria etc. This is because the machine learning system has no way to differentiate between one human speaker to a different human speaker to a TV speaking to somebody super imposing malicious audio on top of really audio. Machine learning system are not built to be not designed to differentiate. Another very interesting example is a dude who gave a presentation last year here called the Hiram Anderson and he built a virus compiling system he wrote some code and he compiled that code and he uploaded into a service called virus total. What virus total does is takes that binary sample runs it against a large number of antiviruses he waits the responses and gives a score. So if it's a virus and most of the antivirus is thought that it was a virus he would get the high score like 0.75 and everybody would know that this binary sample is malicious. However he built a machine learning system that kept changing the code and measuring the output the same kind of oracle attack that I mentioned earlier and he found a different set of inputs that even though the code is still malicious it's the same functional code now the antiviruses are classifying it as being benign. So it's not very difficult to circumvent AI based systems and I want to talk for two minutes about privacy and this is a very simple example just to get the notion of what privacy means in the AI world. So in our example we have a company that built a diabetes differentiating system so they took a lot of samples lots of people and they studied them and now they have a system that when Joe here comes to the system and is classified either has a diabetes risk of 7.4 or maybe has a diabetes risk of 35.3 and they are selling that system to various insurance companies and now insurance companies can check their clients and decide if to provide them a policy or not depending on their risk score. However now Fred comes along but Fred was part of the original data set but because he was part of the original data set the machine learning system knows him it already trained on him so the score is going to be significantly higher and whenever I can see this kind of significantly higher scores I know that that person was part of the original database or at least in his information was. So privacy leakage is very real in these kinds of scenario so you might think that this is a very contrite scenario it's not it's like very real world but I want to give you another example and this is a study called model retrieval or model inversion where what they've done is the same kind of oracle attack that I described earlier where they try to understand what kind of training data was used to train the machine learning model itself. So as you can see here at the bottom right that they got like a super composition of the different face images that were used to train the system but that super composition is not very far from the specific images in that training data set. So if I wanted to leak information out of that data set or from other kinds of machine learning system I can really learn quite a lot by doing these kinds of attacks. So what's our main point to what's the key take was I want you to take from this presentation. The first of all is that we really don't have a trust model for machine learning and we need a trust model here because even though we went forward and we designed lots of very different use cases for AI we need to bring security and privacy into it. The second thing is that the way that the frameworks that design of the end to end system is today nobody cares of our security. Nobody cares about privacy. Everybody is running forward with developing not thinking what it means. And people are using a lot of untrusted sources to fit into their machine learning systems. The transportation of that inputs might be super secure. They might be collecting data from sensors over TLS. But if the data source is compromised what do I care about the TLS connection? So nobody's really doing authentication all the way to the data source. In the end you need to validate your data. If you're not validating your data one of the attackers might come in and give you malicious data. And you have no way to know that malicious data has been introduced to the system. And we can do lots of very cool stuff if you have no controls in place. And last but not least you need to understand the dependency tree of your machine learning system because if you are taking it for granted that your dependencies are secure you're going to have a very bad day. A very bad day. Because as Ezra mentioned a lot of the libraries baked into the frameworks today are no longer maintained. Unpatched. Nobody wants to touch their pet data science project in order to update it. It might break. It might need to be retrained. That's a high cost. And if that's the case it makes my job much easier. What you need to remember in the end AI is just a buzzword. Okay. It's just someone else's code. And code is code. And we can hack it. We can break it. And we can exfiltrate it. I want to acknowledge some of the members on our team who contributed to this work. Omer, Adi, Dennis, Rezy, Adel, Sapir and Oleg who all had significant contributions. I will not expect you to actually read this slide but when we release the entire deck you will have it. Come talk to us. We'll be waiting outside. Thank you.